BM25
210837588
Which BM25 Do You Mean? A Large Scale Reproducibility Study of Scoring Variants
When researchers speak of BM25, it is not entirely clear which variant they mean, since many tweaks to Robertson et al.'s original formulation have been proposed. When practitioners speak of BM25, they most likely refer to the implementation in the Lucene open source search library. Does this ambiguity "matter" We attempt to answer this question with a large scale reproducibility study of BM25, considering eight variants. Experiments on three newswire collections show that there are no significant effectiveness differences between them, including Lucene's often maligned approximation of document length. As an added benefit, our empirical approach takes advantage of databases for rapid IR prototyping, which validates both the feasibility and methodological advantages claimed in previous work.
ECIR
Chris Kamphuis Arjen P de Vries Leonid Boytsov Jimmy J Lin
220487143
BISON: BM25 weighted Self Attention Framework for Multi Fields Document Search
Recent breakthrough in natural language processing has advanced the information retrieval from keyword match to semantic vector search. To map query and documents into semantic vectors, self attention models are being widely used. However, typical self attention models, like Transformer, lack prior knowledge to distinguish the importance of different tokens, which has been proved to play a critical role in information retrieval tasks. In addition to this, when applying WordPiece tokenization, a rare word may be split into several different tokens. How to translate word level prior knowledge into WordPiece tokens becomes a new challenge for the semantic representation generation. Moreover, web documents usually have multiple fields. Due to the heterogeneity of different fields, simple combination is not a good choice. In this paper, We propose a novel BM25 weighted Self Attention framework (BISON) for web document search. By leveraging BM25 as prior weights, BISON learns weighted attention scores jointly with query matrix Q and key matrix K. We also present an efficient whole word weight sharing solution to mitigate prior knowledge discrepancy between words and WordPiece tokens. Furthermore, BISON effectively combines multiple fields by placing different fields into different segments. We demonstrate BISON is more efficient to capture the topical and semantic representation both in query and document. Intrinsic evaluation and experiments conducted on public data sets reveal BISON to be a general framework for document ranking task. It outperforms BERT and other modern models while retaining the same model complexity with BERT.
ArXiv
Xuan Shan Chuanjie Liu Yiqian Xia Qi Chen Yusi Zhang Angen Luo Y Luo
201892552
Sistem Perekomendasi Dosen Pembimbing berdasarkan Relevansi Topik Tugas Akhir menggunakan Metode Okapi BM25
Abstrak Sistem rekomendasi adalah sebuah perangkat lunak untuk memberikan rekomendasi kepada pengguna mengenai produk yang dapat digunakannya. Masalah administrasi di kantor jurusan P endidikan G uru S ekolah D asar Universitas Muhammadiyah Malang merupakan salah satu permasalahan yang selalu dihadapi oleh para staf TU dan part timer. Penggunaan sistem manual yang masih berjalan saat ini dinilai kurang efektif terhadap waktu, tempat, dan tenaga sehingga diperlukan adanya bantuan berupa sistem informasi. Pada perancangan sistem informasi ini akan menggunakan metode Okapi BM25 dimana metode ini merupakan fungsi peringkat yang digunakan oleh mesin pencari (search engine) untuk peringkat dokumen pencocokan sesuai relevansinya dengan permintaan pencarian yaitu berupa topik tugas akhir. BM25 memiliki fungsi yang sesuai dengan 3 prinsip pembobotan yang baik, yaitu memiliki inverse document frequency (idf) term frequency (tf) dan memiliki fungsi normalisasi dari panjang dokumen (document length normalization) Abstract The recommendation system is a software to provide recommendations to users about the products they can use. The administrative problem in the office of the Primary School Teacher Education department at the University of Muhammadiyah Malang is one of the problems faced by the Administration staff and part timers. The use of manual systems that are still running at this time is considered to be less effective against time, place, and energy, so that assistance in the form of information systems is needed. In designing this information system will use the Okapi BM25 method where this method is a ranking function used by search engines for matching document rankings according to their relevance to search queries, namely in the form of final assignment topics. BM25 has functions that are in accordance with the 3 principles of good weighting, which has an inverse document frequency (idf) term frequency (tf) and has a document length normalization function.
Meilina Agustina Yufiz Azhar Nur Hayatin
225047406
Term Recency for TF IDF, BM25 and USE Term Weighting
Effectiveness of a recommendation in an Information Retrieval (IR) system is determined by relevancy scores of retrieved results. Term weighting is responsible for computing the relevance scores and consequently differentiating between the terms in a document. However, current term weighting formula like TFIDF weigh terms only based on term frequency and inverse document frequency irrespective of other important factors. This results in uncertainty in cases when both TF and IDF values are same for more than one document, hence resulting in same term weight values. In this paper, we propose a modification of TFIDF and other term weighting schemes that weights terms additionally based on the recency of a term, i.e. the metric based on the year the term occurred for the first time and the document frequency. We modified the term weighting schemes TF IDF, BM25 and Universal Sentence Encoder (USE) to additionally consider the recency of a term and evaluated them on three datasets. Our modified TF IDF outperformed the standard TF IDF on all three datasets; the modified USE outperformed the standard USE on two of the three datasets; the modified BM25 did not outperform the standard BM25 term weighting scheme.
WOSP
Divyanshu Marwah Joeran Beel
220730252
JASSjr: The Minimalistic BM25 Search Engine for Teaching and Learning Information Retrieval
We present JASSjr, a minimalistic trec_eval compatible BM25 ranking search engine that can index small TREC data sets such as the Wall Street Journal collection. We do this for several reasons. First, to demonstrate how a term at a time (TAAT) search engine works. Second, to demonstrate that a straightforward and competitive search engine with indexer can be written in under 600 lines of documented code. Third, as a way of providing a simple code base for teaching Information Retrieval. We present two index compatible versions (one in C/C+ the other in Java) that compile and run on MacOS, Linux, and Windows. Our code is released under the 2 clause BSD licence, and we provide several suggestions for extensions which might be used as exercises in an Information Retrieval course.
SIGIR
Andrew Trotman Katinya Lilly
222461460
BM25 FIC: Information Content based Field Weighting for BM25F
BM25F has been shown to perform well on many multi field and multi modal retrieval tasks. However, one of its key challenges is finding appropriate field weights. This paper tackles the challenge by introducing a new analytical method for the automatic estimation of these weights. The method denoted BM25 FIC is based on field information content (FIC) calculated from term, collection and field statistics. The field weights are applied to each document separately rather than to the entire field, as normally done by BM25F where the field weights are constant across documents. The BM25 FIC outperforms the BM25F in terms of P@10, MAP and NDCG on a small test collection. Then the paper introduces an interactive information discovery model based on the field weights. The weights are used to compute a similarity score between a seed document and the retrieved documents. Overall, the BM25 FIC approach is an enhanced BM25F method that combines information oriented search and parameter estimation.
BIRDS@SIGIR
Tuomas Ketola Thomas Roelleke
226174595
Klasifikasi Berita Olahraga Berbahasa Indonesia Menggunakan Metode BM25 dan K Nearest Neighbor
Antara Sumbar merupakan sebuah portal berita yang bertujuan untuk memenuhi hak publik untuk mendapatkan informasi yang akurat dan lengkap secara seketika. Terdapat berbagai macam berita yang disajikan, salah satunya adalah berita olahraga. Dengan perkembangan informasi digital yang terus meningkat menyebabkan begitu cepatnya penyebaran dokumen berita sehingga dibutuhkan portal web yang dikelola secara profesional. Antara Sumbar merupakan portal berita yang telah dikelola secara profesional, namun ada beberapa kekurangan terhadap pengelompokkan berita olahraga, yang menyebabkan terganggunya pembaca dalam mendapatkan berita berdasarkan kategori olahraga yang ada. Kekurangan tersebut dapat diatasi menggunakan metode BM25 dan K Nearest Neighbor. Proses yang dilakukan untuk mengatasi kekurangan tersebut adalah melakukan preprocessing terhadap dokumen berita, melakukan pembobotan dan pemeringkatan menggunakan metode BM25, dan yang terakhir proses algoritme K Nearest Neighbor sebagai metode klasifikasi. Proses pengujian yang digunakan adalah metode k fold sebanyak 7 kali pengujian. Data yang digunakan pada tiap tiap pengujian yaitu sebanyak 240 dokumen latih dan 40 dokumen uji. Berdasarkan pengujian yang dilakukan, didapatkan hasil pengujian tertinggi ketika nilai ketetanggaan (k) sebesar 20, menghasilkan nilai precision 0,921577, nilai recall 0,914286 dan nilai f measure 0,917917. Proses klasifikasi dipengaruhi oleh jumlah dokumen yang digunakan serta besaran nilai k yang ditentukan.
Enggar Septrinas
201061637
Efficient Hyperparameter Tuning with Grid Search for Text Categorization using kNN Approach with BM25 Similarity
Abstract In machine learning, hyperparameter tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. Several approaches have been widely adopted for hyperparameter tuning, which is typically a time consuming process. We propose an efficient technique to speed up the process of hyperparameter tuning with Grid Search. We applied this technique on text categorization using kNN algorithm with BM25 similarity, where three hyperparameters need to be tuned. Our experiments show that our proposed technique is at least an order of magnitude faster than conventional tuning.
Open Comput. Sci.
Raji Ghawi Jurgen Pfeffer
171095870
Term Weighting for Feature Extraction on Twitter: A Comparison Between BM25 and TF IDF
Feature extraction is to transform a text document from any format into a list of features that can be easily processed by text classification techniques. Feature extraction is one of significant preprocessing techniques in data mining and text classification that computes features value in documents. Hence, efficient feature extraction techniques like the BM25 and term frequency inverse document frequency (TF IDF) techniques are normally utilized in term weighting. Nevertheless, BM25 is not a single function that is utilized to exceedingly correct very long documents. This problem cannot denote the helpfulness or importance of confident features, and decreases the efficiency of classification. This paper presents a comparative study of feature extraction techniques. Two techniques were evaluated BM25 and TF IDF to weight the terms on Twitter. In this paper, TF IDF feature extraction technique is presented to compare between the two techniques. The experiments show that TF IDF improves the performance evaluation of feature extraction according to the maximum value of F1 measure is 89.77 for TF IDF and 89.16 for BM25.
2019 International Conference on Advanced Science and Engineering (ICOASE)
Ammar Ismael Kadhim
229211715
Information retrieval system for silte language using BM25 weighting
The main aim of an information retrieval system is to extract appropriate information from an enormous collection of data based on users need. The basic concept of the information retrieval system is that when a user sends out a query, the system would try to generate a list of related documents ranked in order, according to their degree of relevance. Digital unstructured Silte text documents increase from time to time. The growth of digital text information makes the utilization and access of the right information difficult. Thus, developing an information retrieval system for Silte language allows searching and retrieving relevant documents that satisfy information need of users. In this research, we design probabilistic information retrieval system for Silte language. The system has both indexing and searching part was created. In these modules, different text operations such as tokenization, stemming, stop word removal and synonym is included.
ArXiv
Abdulmalik Johar