Text Retrieval Conference - Conference Contributions

Conference Contributions

TREC claims that within the first six years of the workshops, the effectiveness of retrieval systems approximately doubled. The conference was also the first to hold large-scale evaluations of non-English documents, speech, video and retrieval across languages. Additionally, the challenges have inspired a large body of publications. Technology first developed in TREC is now included in many of the world's commercial search engines. An independent report by RTII found that "about one-third of the improvement in web search engines from 1999 to 2009 is attributable to TREC. Those enhancements likely saved up to 3 billion hours of time using web search engines. ... Additionally, the report showed that for every $1 that NIST and its partners invested in TREC, at least $3.35 to $5.07 in benefits were accrued to U.S. information retrieval researchers in both the private sector and academia."

While one study suggests that the state of the art for "ad-hoc" search has not advanced substantially in the past decade, it is referring just to search for topically relevant documents in small news and web collections of a few gigabytes. There have been advances in other types of ad-hoc search in the past decade. For example, test collections were created for known-item web search which found improvements from the use of anchor text, title weighting and url length, which were not useful techniques on the older ad-hoc test collections. In 2009, a new billion-page web collection was introduced, and spam filtering was found to be a useful technique for ad-hoc web search, unlike in past test collections.

The test collections developed at TREC are useful not just for (potentially) helping researchers advance the state of the art, but also for allowing developers of new (commercial) retrieval products to evaluate their effectiveness on standard tests. In the past decade, TREC has created new tests for enterprise e-mail search, genomics search, spam filtering, e-Discovery, and several other retrieval domains.

TREC systems often provide a baseline for further research. Examples include:

• Hal Varian, Chief Economist at Google, says Better data makes for better science. The history of information retrieval illustrates this principle well," and describes TREC's contribution.
• TREC's Legal track has influenced the e-Discovery community both in research and in evaluation of commercial vendors.
• The IBM researcher team building IBM Watson (aka DeepQA), which recently beat the world's best Jeopardy! players, used data and systems from TREC's QA Track as baseline performance measurements.