We use cookies to provide you with a better experience. If you continue to use this site, we'll assume you're happy with this. Alternatively, click here to find out how to manage these cookies

hide cookie message
 
74,944 News Articles

Twitter search engine gets an upgrade

Tweet search now much faster and scalable

Twitter has overhauled the back-end infrastructure of its search engine, boosting its speed and capacity to index posts, process queries and deliver results, while making the system more stable and better suited for the addition of new features.

Twitter transferred its search engine to the new platform in recent weeks, after working on the new back-end system for about six months, according to the company.

Twitter's search engine ran on a system based on MySQL from Summize, a company Twitter acquired in mid-2008, but scaling up the system had become difficult.

The engineering team in charge of the project decided to do an extreme makeover of the search engine using a different technology: the open-source, text search engine Lucene, which is written in Java.

Twitter modified some aspects of Lucene, including its garbage collection, query termination, posting lists, and data structures and algorithms, and was left with an inverted, index-based search engine that scales much more and has better performance.

Twitter fields 12,000 search queries per second - or more than 1 billion per day - and "tweets" become part of its search index less than 10 seconds after they are posted.

"We estimate that we're only using about 5 percent of the available backend resources, which means we have a lot of headroom. Our new indexer could also index roughly 50 times more Tweets per second than we currently get," Twitter official Michael Busch wrote in a blog post.

Twitter will contribute back to the Lucene project the modifications and improvements it made to the code.

Although Twitter makes available its index of "tweets" to external search engines like Google and Microsoft's Bing, its internal search engine is a key component of its microblogging service.

In addition to being the preferred vehicle among private citizens, public figures and companies for broadcasting short status updates, Twitter has become an increasingly valued repository of real-time data, tapped for following news, trends and collective musings.

To maximise the value of this "tweet" repository, the company must have a search engine that is fast, comprehensive and scalable, and its massive revamping of its search technology shows the company recognizes the importance of its internal search capabilities.


IDG UK Sites

Netflix to introduce price increase: New subcribers to start with

IDG UK Sites

Apple financial results: iPhones, iPads & Macs sales for Apple's Q2 2014, plus shares to split

IDG UK Sites

Twitter - not news

IDG UK Sites

See Moo Studios' new animated advert for Blue Moon beer