US to archive billions of tweets

2013-01-22 08:30
The US is to archive tweets. (Duncan Alfreds, News24)

The US is to archive tweets. (Duncan Alfreds, News24)

Multimedia   ·   User Galleries   ·   News in Pictures Send us your pictures  ·  Send us your stories

Washington - The Library of Congress, repository of the world's largest collection of books, has set for itself the enormous task of archiving something less weighty and far more ephemeral - Americans' billions of tweets.

The US institution is assembling all of the 400 million tweets sent by Americans each day, in the belief that each of the mini-messages reflect a small but important part of the national narrative.

"An element of our mission at the Library of Congress is to collect the story of America, and to acquire collections that will have research value," according to Gayle Osterberg, director of communications at the library.

The Library of Congress, located off the National Mall in Washington, houses millions of hard copy books and historic documents, and its online archives amass millions of additional works produced by Americans for more than two centuries.

Now it wants to be keeper of the nation's brief internet messages as well: Twitter in April 2010 signed a deal with the Library, giving it access to tweets dating back to the company's inception in 2006.

Challenge

Collecting the 140-character micro-missives, said Osterberg, is in keeping with the library's main goal "to collect the story of America and to acquire collections that will have research value".

One major challenge to the Library, however, is storing the messages from the popular social messaging site, which now number 170 billion. Twitter in December said the number of active users on the messaging platform has topped 200 million, most of whom are in the US.

Tweets that have been deleted or that are locked will not be among those gathered by the Library of Congress.

Among the messages to be preserved for posterity are the first-ever tweets sent by one of the company's founders, Jack Dorsey.

Also saved for all time is a famous tweet sent by US President Barack Obama after his historic November 2008 victory to claim the White House in his first term.

"We just made history. All of this happened because you gave your time, talent and passion. All of this happened because of you. Thanks," read the micro-message from the famously tech-savvy US president.

Unlike traditional bound books or even digital web pages, the real challenge of preserving tweets is keeping up with their number, which has continued to grow almost exponentially.

Researchers

There were 140 million tweets sent each day in February 2011, but more than three times as many - about a half billion - by October 2012.

The Library of Congress's tweets are being stored by Gnip, a social media aggregation company headquartered in Boulder, Colorado, which has put more than 133 000GB of storage space available.

Gnip says it is a particular challenge to gather tweets during "peak" times, such as news event watched the world over like the Japanese tsunami in March 2011, which generated many thousand tweets per second.

It has proven to be a Herculean challenge for Gnip to make tweets accessible to all those who wish to view them.

So far it has been unable to meet the demands of researchers worldwide who hope to access the archive. Even a search among the first four years of tweets, from 2006 to 2010, could take about 24 hours.

"It is clear that technology to allow for scholarship access to large data sets is lagging behind technology for creating and distributing such data," said a recent white paper published by the Library of Congress.

"This is an inadequate situation," the Library concluded, calling the massive archiving project "prohibitively costly".

And yet Lee Humphreys, a professor of communication at Cornell University in New York, said that the brief online messages can reveal volumes "about the culture where they were produced".
- SAPA

Read more on:    twitter  |  social networks
NEXT ON NEWS24X

SHARE:

Read News24’s Comments Policy

24.com publishes all comments posted on articles provided that they adhere to our Comments Policy. Should you wish to report a comment for editorial review, please do so by clicking the 'Report Comment' button to the right of each comment.

Comment on this story
0 comments
Add your comment
Comment 0 characters remaining
 

Inside News24

 
/News
 
Traffic
Lottery
 
  • Tuesday Parow - 08:40 AM
    Road name: N1 Inbound
    DELAYS between Giel Basson Drive and the N7 Highway
  • Tuesday Cape Town - 08:38 AM
    Road name: N1 Inbound
    DELAYS between Lower Church Street and the Waterfront exit via the Elevated Freeway
 
More traffic reports...
 

Jobs in Cape Town [change area]

Property [change area]

Travel - Look, Book, Go!

Magical Massinga

Spend 5 nights at the gorgeous Massinga Beach Lodge in Mozambique and only pay for 4 from R13 220 per person sharing. Includes return flights, accommodation, transfers and romantic turndown. Book now!

Kalahari.com - shop online today

2 TV Series for R250

Get 2 TV Series box sets for R250 and save up to R148! Offer valid while stocks last. Shop now!

Mind blowing deals on beauty & fragrances

Save up to 30% off beauty and fragrances. Offer valid while stocks last. Shop now!

Up to 50% off hair care products!

Save up to 50% on professional hair care products at kalahari.com. Offer valid while stocks last. Shop now!

30% off academic books

Score a mind blowing 30% off academic books! Offer valid while stocks last. Shop now!

Mind blowing deals on electronics!

Save up to 35% on electronics. Offer valid while stocks last. Shop now!

OLX Free Classifieds [change area]

Samsung Galaxy s4

Mobile, Cell Phones in South Africa, Western Cape, Cape Town. Date October 24

Best bargain in big bay

Real Estate, Houses - Apartments for Sale in South Africa, Western Cape, Cape Town. Date October 25

VW Golf 6, 1.6 Trendline (Excellent condition)

Vehicles, Cars in South Africa, Western Cape, Cape Town. Date October 25

Samsung Galaxy Y

The Samsung Y (Young) is tiny but powerful with Android...

From R1499.00

I'm shopping for:

Horoscopes
Aquarius
Aquarius

Something may come up to alert you of your security needs. It may be that you need to create a better structure for your finances...read more

There are new stories on the homepage. Click here to see them.
 
English
Afrikaans
isiZulu

Hello 

Create Profile

Creating your profile will enable you to submit photos and stories to get published on News24.


Please provide a username for your profile page:

This username must be unique, cannot be edited and will be used in the URL to your profile page across the entire 24.com network.

Settings

Location Settings

News24 allows you to edit the display of certain components based on a location. If you wish to personalise the page based on your preferences, please select a location for each component and click "Submit" in order for the changes to take affect.








Facebook Sign-In

Hi News addict,

Join the News24 Community to be involved in breaking the news.

Log in with Facebook to comment and personalise news, weather and listings.