The other week I launched TWT on DemConnect. I’m excited about this new stats/data venture and what it means for understanding how people (government officials in this case) are interacting with Twitter. Here’s a short behind-the-scenes look at what I’m doing:
- First, most of this is automated. If you know me at all, and how I like to work, if there’s a way to automate a part of a job, I’ll attempt to do that. In this case, I’ve automated the data collection process. I’ve created software that grabs key data points daily for a selected group of users. Right now it’s grabbing government accounts, but it could in reality grab any public Twitter account. (This will probably feed into a future research project similar to TheFacebookStudy – which I’ve been sitting on for a while, I know.)
- Following my favorite ETL method, the automated kind, I pull the data into my big data machine, do some data cleansing (not changing, just cleaning up), and load it up for analysis.
- From there, the sky’s the limits. I have some automated data functions that return the type of data you see in TWT: follower and friend changes, tweets analysis, etc. I can pull data for any date range, so the flexibility of comparing different areas is great. Also, Twitter users are in groups – which makes combining and comparing data points much easier.
- Finally, I spend some time putting the data into the design you see published. I think the design can use a bit of work – there’s a lot of data there, and frankly I would like to put even more on there. I’m trying to figure out the best way to display a ton of data in a nice graphical format…that I can easily and quickly update on a weekly basis. That last part – regular updating, is key. The consistency you see online now makes it easy to read and follow from week to week.
That’s the short, behind-the-scenes look at The Weekly Tweet. As of this post I’m tracking the U.S. Senate, U.S. House of Representatives, and 2012 Presidential Candidates. Soon I’ll be adding the executive branch to the listings; probably in the next 2 weeks. I’m also trying to figure out a way to turn this information into a podcast as well. I’d prefer an audio podcast – but this much data much be tough to digest in an audio format. If anyone has suggestions – I’m all ears.
Researcher friends: any interest in doing some data mining and Twitter analysis? Precedent suggests I cover how college students use/don’t use Twitter.
Update: 8/6/2011: I am in the process of redesigning The Weekly Tweet and hope to relaunch it in the upcoming months. The new version will be an interactive version with more data and analysis.
