- Kevyn Collins-Thompson, Paul Ogilvie, Jamie Callan. 2004. Initial Results with Structured Queries and Language Models on Half a Terabyte of Text. Abstract: 3. INDEXING ENVIRONMENT We indexed all of the GOV2 corpus documents, submitting complete documents to the indexer. We did no special processing for titles or other document structure. After running each document through Indri’s HTML parser and stripping out tags, terms were normalized by removing punctuation, converting to lowercase, and performing Krovetz stemming [6]. Indri has a utility to extract anchor text, but a working version of this was not available at the time of our experiments, so no anchor text (or other linkderived features) was included. We did not use an indextime stop list or acronym list. We partitioned the GOV2 corpus into six roughly equalsized pieces and built a separate index for each subset. There were two reasons for doing this. First, it was our intent to distribute retrieval for the corpus across multiple machines: Indri has a facility for transparently federating retrieval results across the network. Second, it helped reduce the time to find and fix problems with interim Indri releases (typically, pathological HTML cases that would cause the parser to crash).
