license:apache-2.0 finance legal region:us
---
license: apache-2.0
tags:
- finance
- legal
pretty_name: RAG Instruct Test Dataset - Basic - v0.1
---
# Dataset Card for RAG-Instruct-Test-Dataset

### Dataset Summary

This is a test dataset for basic "retrieval augmented generation" (RAG) use cases in the enterprise, especially for finance and legal.  This test dataset includes 100 samples with context passages pulled from common 'retrieval scenarios', e.g., financial news, earnings releases,
contracts, invoices, technical articles, general news and short texts.   The primary use case is to evaluate the effectiveness of an
instruct-fine-tuned LLM used in conjunction with closed-context, fact-based question-answering, key-value extraction, and summarization with bulletpoints.  The context passages are relatively short in this test-set ranging from ~100 tokens to ~500 tokens, and was designed for use with the
BLING series of models but is suitable for comparison evaluations of any LLM for basic RAG scenarios.   

### **PERFORMANCE on BASIC RAG TEST DATASET** 
							
| Model                 |  Params (B) |	Sourcing |	GPU/CPU	 | Output Tokens | Out as % of Input | Process Time (secs) | Score (0-100) |
| :----------           | :--------:  |  :----:  | :-----:   | :---------:   | :-------:         | :--------:          | :-------:     |
| gpt-4	                |   <=1000	  | Closed   | Multi-GPU | 2665	         | 10.53%	         | 183.8	           |    100        |
| gpt-3.5-turbo-instruct|	<=175	  | Closed	 | Multi-GPU |	2621	     | 11.49%	         | 62.7	               |    100        |
| claude-instant-v1	    |   <=50	  | Closed	 | Multi-GPU |	6337	     | 26.50%	         |  154	               |    100        |
| aib-read-gpt	        |   7	      | Closed   | GPU	     |  1964	     |  9.30%            |	114	               |     96        |
| bling_falcon-1b-0.1	|   1.3	      | Open	 | CPU	     |  3204	     | 14.55%	         |  696	               |     77        |
| bling_pythia-1.4b-0.1	|   1.4	      | Open	 | CPU	     |  2589	     | 11.75%	         |  593.5	           |     65        |
| bling_pythia-1b-0.1	|   1.0	      | Open     |	CPU	     | 2753	         | 12.49%	         |  428	               |     59        |
| bling_cerebras-1.3b   |	1.3	      | Open     |	CPU	     | 3202	         | 20.01%	         |  690.1	           |     52        |
| bling_pythia_410m	    |  0.41	      |  NA	     |  CPU	     |  2349	     |  10.66%	         |  189	               |     36        |
| bling_cerebras_590m	|  0.59	      |  NA	     |  CPU	     |  4407	     |   20.01%	         |  400.8	           |     30        |

Please check out our [BLOG](https://medium.com/@darrenoberst/evaluating-llm-performance-in-rag-instruct-use-cases-083dc272a31d) with more details, commentary and comparative results testing with this dataset.  

We will be enhancing the test dataset as well as creating more advanced test datasets in the future.    


### Languages

English

## Dataset Structure

100 JSONL samples with 4 keys - "query" | "context" | "answer" | "sample_number"

### Personal and Sensitive Information

The dataset samples were written bespoke for this objective, but do rely upon some public information, including major public figures and widely reported events.   
Any other names were created/masked and any overlap with real companies or people is coincidental. 

## Dataset Card Contact

Darren Oberst & llmware team

Please reach out anytime if you are interested in this project and would like to participate and work with us!

