Why you should care about #missingtweets

More things keep happening to distract me. This time it’s Twitter (as if Twitter isn’t distracting enough).

First let me say that I’m a big Twitter fan. Maybe it’s the idealist in me, but I believe in the ethos of Twitter. Twitter has great power to bring people together. Its asymmetric friendship model more accurately reflects the way relationships work in real life than those of social networks like Facebook, and despite the overwhelming perception of Twitter as a broadcast network where everybody is talking and few are listening, it’s the one place where you can often make contact with people who are otherwise inaccessible.

Personally, I’ve benefited from Twitter in a number of ways–some of them small, some of them less so. I’ve discovered a lot of really interesting people on Twitter who’ve pointed me to information and who can be a source of inspiration and ideas. I’ve been mentioned on the radio in Portland by @caseorganic (I live in Switzerland), quoted in a research paper by @zephoria, and made a contact with a company when I needed help and their customer service email wasn’t responding–all because of Twitter. I don’t think it’s a stretch to say that the experience I gained on Twitter played a role in the new job I’ve landed. Twitter also gives me real time news information, which saves me an enormous amount of time.

I joined Twitter on May 28, 2008, although you’d never know that because Twitter doesn’t seem to display this information on your profile anymore, and it doesn’t provide a search engine that searches Tweets older than about 10 days. It used to be the only way to find out when I joined would be to scroll through my 4’468 tweets to the very first one. However, you can’t do that anymore because I’ve been visited by what users are calling the #TweetBurglar, and over 4’100 of those 4’468 tweets, all of them prior to Nov. 8th, 2010, went missing almost two weeks ago. Gone. Update: Since I started this draft yesterday, my tweets were restored during the night before I published this post. Thanks, Twitter! (In case you’re wondering,  I didn’t remember when I actually joined Twitter, but I was able to find that information in a partial backup of my timeline.)

I’m the victim of a problem that many users are experiencing. It’s explained on Twitter’s support page, except that page inaccurately says the problem is fixed, when in fact it isn’t. Dozens of Twitter users leave comments every day to say they’ve recently been affected, and many of them have had their tweets restored by Twitter, only to lose them again days later. So Twitter is experiencing a serious problem.

I’ve been tweeting about this situation daily since my tweets disappeared, and I’ve received several comments about it. Most advise me to use a backup solution to archive my Tweets because I shouldn’t trust Twitter to keep them. Some people suggest that because Twitter is free, I shouldn’t expect to have a high level of service. I’m not satisfied with either of those suggestions, and a Twitter discussion with @frederik_vl prompted me to write this post.

Here is part of it:

This made me think a little more closely about the real problem with Twitter’s #missingtweets, and so I tweeted this:

Twitter has been extolled for its ability to track the real-time pulse of public opinion. Recently it’s been used by researchers for applications as diverse as visualizing travel information and daily variations in moods to predicting elections. It’s been lauded for its ability to disseminate information widely and quickly, which has led to important changes in the way news is broadcast and consumed as well as a significantly increasing awareness of global issues.

So what happens when that information is unreliable because Twitter isn’t able to maintain it?  The value of Twitter is significantly reduced, because the information obtained from it may be incomplete or incorrect. It may be impossible to locate a particular tweet because it is no longer accessible, and depending on the extent of the problem, the combined result for missing tweets over a large group may have an effect on inferences made from analysis of the data. This situation is important not just to researchers, but also by extension to businesses, as they would increasingly like to use Twitter for data mining to understand the profiles of consumers. If that data is incomplete because users are missing thousands of tweets from their profile, then the understanding of their interests, likes and dislikes will be affected.

There’s another problem. Due to the threaded nature of Twitter, if one of my missing tweets contained an @ mention of another Twitter user, that tweet will be missing for them too, so the dialogue of conversations is corrupted and the authority scores Twitter keeps for users will be affected.

I’d like to mention here that I started using TweetBackup on Aug. 12, 2009 to archive my tweets. Unfortunately I started the backups too late to capture my very first tweets due to the 3’200 tweet limit of Twitter’s API, but throughout this time, the service had been backing up my tweets regularly until the #missingtweets issue struck. I haven’t verified the accuracy of the backups, but the latest one occurred on Nov. 25th, and the last backed-up tweet was recorded on Oct. 11th. I don’t know if the backups were affected by the #missingtweet issue directly, but I’ll be looking into that over the next few days as my account seems to be back to normal. I also note that TweetBackup makes a certain number of tweets available publicly, but not my entire backed-up timeline. This feature is useful, but far less so than having the tweets directly available within Twitter itself.

Ok, so that explains why a backup solution isn’t sufficient. Now, what about that argument that you can’t expect Twitter to keep your tweets because you don’t pay for it? Well, Twitter has been described as a microblogging platform, so let’s compare it to another type of blogging platform, say WordPress. [Blogger is probably a service that is more widely used, but I’d like to keep Google out of this discussion. I also like WordPress so I’d rather use it as an example here.]

You can get a free WordPress account and create your own blog. Over 15 million blogs are hosted on wordpress.com. What if you couldn’t trust WordPress to keep your old blogs available? What if they were randomly deleted? What if you complained about the problem and WordPress did nothing for weeks? Would you want to use WordPress for blogging? Or would you use something else?

Now you might think that Twitter has more data to deal with than WordPress, being real-time and all, so the problem they are facing must be more difficult, but no. After exchanging some tweets with @frederik_vl, I quickly looked up some figures. According to Barry on WordPress, at the end of last year, over 150 GB of content were uploaded daily to WordPress. Now, looking at Twitter’s About Us page, and Raffi Krikorian’s presentation “Twitter by the Numbers” I calculated that Twitter’s daily volume of 95 million tweets represents about 19GB (data from September). That’s just over 10% of the daily volume of data uploaded to WordPress.

So if WordPress is able to provide reliable storage, indexing and retrieval for all that data, able to provide searchable information from a blog’s very first to its very latest post, Twitter ought to be able to do the same for its users’ microblogs. Now, I admit that the real-time threaded nature of Twitter certainly places some constraints on the server infrastructure, the architecture of which may make the storage and retrieval of tweets a more complex problem than for a blogging service like WordPress. However, from a user standpoint, the reliability requirement should be the same.

The last news from Twitter on the #missingtweets issue was on Nov. 9th, when they Twitter updated their support page on Nov. 9th to say the problem had been fixed. Since then over 1760 comments have been left by users who are still experiencing the problem, and that’s not to mention those who may not have noticed because they haven’t paid attention or are only missing a few tweets.

I’d been waiting for one day short of two weeks before mine were restored, but there’s still no clear word from Twitter if they are working on the problem (although they most certainly are since mine came back), or why users are still experiencing it the problem even though they say it’s fixed. Meanwhile, some users are loosing confidence in Twitter and still others are starting to wonder what’s the point:

I hope Twitter will improve their transparency on this issue and continue to work on fixing the problem, both to restore user confidence and to ensure the platform lives up to it’s awesome potential.

Follow me on Twitter for continued updates on this subject.

Update Nov 28 19:30 CET:

Just noticed the following tweets from Twitter Support. I hope this means the problem will be definitively resolved for all users.

This entry was posted in Blogging, Social Media and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s