Last month after having seen a few articles describing errors in traffic sources by analytics solutions that fail to correctly identify referring sites, I wrote about some informal tests I had been doing with my blog. Since then, I’ve had the time to perform a few controlled experiments with some unexpected and very surprising results.
To test the hypothesis that WordPress traffic reports fail to identify clicks coming from sites like Twitter and Facebook, I set up a separate test blog where I could do a few experiments inconspicuously without being misled by incoming traffic from other sources. I populated the blog with a few posts designed to test some specific features of WordPress stats and began sharing links to the posts on various sites sometimes via special accounts and pages set up for just this purpose.
In all, I made tests with Twitter, Bit.ly, Facebook, Google+, Google Reader, email and webdoc.com. In each case, for every visit resulting from a click from a referring site, I verified that the view was correctly recorded by WordPress and noted the resulting referrer information, if any. Mike Cane agreed to be my lab partner for this exercise. He helped me by performing clicks for a few tests I could not easily make myself, as well as serving as an independent control to make sure my own observations were repeatable under independent circumstances. We used several browsers to test the repeatability of the results: FireFox 5.0.1, Safari 5.1 and Opera 11.50, and performed tests with email, Echofon 2.2.2 FireFox plug-in and with the following iPhone apps: Twitter version 3.3.5, Facebook version 3430, and MobileRSS Free version 3.4.
All tests were performed on Wednesday/Thusday August 3rd/4th WEST. I’ve made the results available in a Google Spreadsheet. They represent the state of wordpress.com stats and referrer information at the time they were executed.
In all, 32 click tests were made. Of these, 29 resulted in views being logged at wordpress.com. Only 11 out of these 29 listed any referrer information, so that 62% of traffic was listed as direct even though virtually all the clicks came from shares on one of the tested social network sites or a share via email.
Closer examination of the data reveals the following:
- WordPress lists referral information for views only when the link is clicked to open the page in a new browser window. Clicking on a link from any of the sites to open a post in-place or in a new browser tab results in a view without any referral information, i.e. it shows up as direct traffic. Google Reader is the only exception to this rule.
- Views in Google Reader don’t show up in wordpress.com stats! I subscribed to the blog with Google Reader and opened two posts in the reader using my browser and one in the Free Mobile RSS app on my iPhone. Surprisingly NONE of these views showed up in my WordPress stats at all! However, opening the link from Google Reader by right-click or control-click to open the link in a new tab or window both showed up as referred views from Google Reader.
- 75% of views from Twitter show up as direct traffic. I made a variety of tests using bit.ly links as well as long links shortened by Twitter’s built-in shortener, both on Twitter.com, in the Echofon FireFox plugin and the Twitter iPhone app. Only 1 out of 4 clicks from Twitter were recorded with referral information, meaning that 75% of views resulting from Twitter clicks showed up as direct traffic.
- All the clicks I made on bit.ly links were correctly counted on the bit.ly dashboard.
- Clicks on search results from Google are correctly identified. I tested the search referral information by searching on google.com for a meaningless phrase in one of the posts: “Militant vegetables earn macaroons joyously still now.” Each of the three click tests performed on the search results was correctly identified by wordpress.com as coming from a search for this phrase.
The most serious problem I identified is the lack of view information coming from clicks on posts within Google Reader on the website and in the MobileRSS iPhone app. It is important to keep in mind that if my data are correct, none of the views performed within the reader by these subscribers will show up as blog traffic AT ALL, leading to an underestimation of traffic for posts viewed this way. For blogs with large numbers of subscribers, this means that total number of views could be underestimated significantly.
It is not clear to me why wordpress.com lists referrer information only when links are opened in a new browser window. I had expected to find exactly the opposite result. In other words, I expected that WordPress would show the referrer if the link were clicked to open in-place, i.e. in the current browser tab, and that the referrer data would be lost of it were opened in a new tab or a new window. If WordPress is able to identify the referrer when the link is opened in a new window, surely it should be able to do so if the link is opened on the same page or in a new tab. Clearly something is wrong here, because FireFox, Safari and Opera all demonstrate the same behavior. This does not seem logical to me, but I have not had time to look into a plausible explanation.
I am not able to explain my previous observations that traffic from bit.ly is between 6-8 times higher than accounted for on bit.ly’s click history for shortened links. The most plausible explanation I can think of to explain this is that links I’ve shared on Twitter using bit.ly are being reshared in a way that is not visible to me by people copying the link or shortening it with other tools.
Overall, the results of these tests provide a very clear explanation for the high percentage of direct traffic in the stats for the blogs I manage, all of which are hosted on wordpress.com. It would be interesting to see the same detailed click tests performed on a self-hosted site using other analytics packages, such as Google Analytics, to see in which cases the referrer information is recovered. Such tests might shed light on the question of the origin of the problem, whether it is a failure of conventional browser behavior to transmit this information to wordpress.com or a failure of WordPress to correctly record the data.