Update README.md

README.md

@@ -9,7 +9,7 @@ license: llama3
 ---
 <a href="https://www.gradient.ai" target="_blank"><img src="https://cdn-uploads.huggingface.co/production/uploads/655bb613e8a8971e89944f3e/TSa3V8YpoVagnTYgxiLaO.png" width="200"/></a>
 
-# Llama-3 8B Gradient Instruct 4194K
+# Llama-3 8B Gradient Instruct 4194K (Work-in-progress)
 
 Join our custom agent and long context (262k-1M+) waitlist: https://forms.gle/L6TDY7dozx8TuoUv7
 
@@ -17,10 +17,10 @@ Gradient incorporates your data to deploy autonomous assistants that power criti
 
 For more info see our [End-to-end development service for custom LLMs and AI systems](https://gradient.ai/development-lab)
 
-This model extends LLama-3 8B's context length from 8k to 4194K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. For this stage, we trained on 185M tokens for this stage, and 1.6B tokens total for all stages, which is ~ 0.01% of Llama-3's original pre-training data.
+This model extends LLama-3 8B's context length from 8k to 4194K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). 
+It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. For this stage, we trained on 192M tokens for this stage, and 1.8B tokens total for all stages, which is ~ 0.01% of Llama-3's original pre-training data.
 
-
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/644fac0ce1d7a97f3b653ab1/V8mtPbVCWgwvxTRoUwiI3.png)
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/644fac0ce1d7a97f3b653ab1/01_d4UYPE47EHlFGyaG9X.png)
 
 **Approach:**
 
@@ -47,7 +47,7 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
 | RoPE theta             | 15.3 M    | 207.1 M   | 1.06B     | 2.80B     | 45.2B     |
 | Batch Size             | 1         | 1         | 16         | 16         | 2         |
 | Gradient Accumulation Steps | 32    | 16        | 1         | 1         | 2         |
-| Steps                  | 30        | 24        | 50        | 50        | 12        |
+| Steps                  | 30        | 24        | 50        | 50        | 12 (stopped early)       |
 | Total Tokens           | 62914560  | 100663296 | 419430400 | 838860800 | 201326592 |
 | Learning Rate          | 2.00E-05  | 2.00E-05  | 2.00E-05  | 2.00E-05  | 2.00E-05  |
 | # GPUs                 | 8         | 32        | 512       | 512       | 512       |
@@ -57,30 +57,19 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
 
 **Evaluation Details:**
 
-**[UPDATE THESE NUMBERS]**
-
 ```
-EVAL_MAX_CONTEXT_LENGTH=1040200
+EVAL_MAX_CONTEXT_LENGTH=4194200
 EVAL_MIN_CONTEXT_LENGTH=100
-EVAL_CONTEXT_INTERVAL=86675
+EVAL_CONTEXT_INTERVAL=260000
 EVAL_DEPTH_INTERVAL=0.2
 EVAL_RND_NUMBER_DIGITS=8
-
-HAYSTACK1:
-    EVAL_GENERATOR_TOKENS=25
-
-HAYSTACK2:
-    EVAL_CONTEXT_INTERVAL=173350
-    EVAL_GENERATOR_TOKENS=150000
-
-HAYSTACK3:
-    EVAL_GENERATOR_TOKENS=925000
 ```
 
 The haystack used is haystack #3, as detailed in this [blog post](https://gradient.ai/blog/the-haystack-matters-for-niah-evals).
 
 **Quants:**
 
+There are no currenty quants released. We advise to run the KV Cache in fp16 precision for higher accuracy.
 
 ## The Gradient AI Team