text
All You need To Know About Phi-3 (Technical Report Walkthrough)
All You need To Know About Phi-3 (Technical Report Walkthrough)
new_line
new_line
text
Summary of Summaries:
Summary of Summaries:
new_line
text
Phi-3-mini
Phi-3-mini
new_line
text
- Architecture specs: decoder-only transformer, ModelSize: 3.8 billion 
- Architecture specs: decoder-only transformer, ModelSize: 3.8 billion 
new_line
text
  parameters,  LongRope [ 128K Context length ], Vocab Size [ 32064 ], 
  parameters,  LongRope [ 128K Context length ], Vocab Size [ 32064 ], 
new_line
text
  trained on 3.3 trillion tokens. at bfloat16.
  trained on 3.3 trillion tokens. at bfloat16.
new_line
text
- Rivals performance to larger models like Mixtral 8x7B and GPT-3.5, 
- Rivals performance to larger models like Mixtral 8x7B and GPT-3.5, 
new_line
text
  capable of running locally on a smartphone.
  capable of running locally on a smartphone.
new_line
text
- Utilizes high quality training dataset heavily filtered from web data and 
- Utilizes high quality training dataset heavily filtered from web data and 
new_line
text
   llm-generated synthetic data.
   llm-generated synthetic data.
new_line
text
- Can be quantized to 4-bits, occupying ≈ 1.8GB of memory.
- Can be quantized to 4-bits, occupying ≈ 1.8GB of memory.
new_line
text
- Ran natively on iPhone 14 with A16 Bionic chip with inference speed of up 
- Ran natively on iPhone 14 with A16 Bionic chip with inference speed of up 
new_line
text
   to 12 tokens per second. 
   to 12 tokens per second. 
new_line
new_line
text
Phi-3-small
Phi-3-small
new_line
text
- Architecture specs: Also decoder-only, 7B parameters, Vocab size [ 100352 ], default context length [ 8k ], Context Length: 8K, Hidden Dimension: 4096, Number of Heads and Layers: Follows 7B class structure.
- Architecture specs: Also decoder-only, 7B parameters, Vocab size [ 100352 ], default context length [ 8k ], Context Length: 8K, Hidden Dimension: 4096, Number of Heads and Layers: Follows 7B class structure.
new_line
text
- Uses tiktoken tokenizer (for enhanced multilingual tokenization) 
- Uses tiktoken tokenizer (for enhanced multilingual tokenization) 
new_line
new_line
text
Phi-3-medium: 
Phi-3-medium: 
new_line
text
-  Architecture specs: Also decoder-only, Hidden Dimension: 5120, Number of Heads: 40, Number of Layers: 40, Tokenization: Consistent with other models, Training on 4.8 trillion tokens.
-  Architecture specs: Also decoder-only, Hidden Dimension: 5120, Number of Heads: 40, Number of Layers: 40, Tokenization: Consistent with other models, Training on 4.8 trillion tokens.
new_line
new_line
text
Training Methodology:
Training Methodology:
new_line
text
- Focuses on high-quality training data deviating from standard scaling laws.
- Focuses on high-quality training data deviating from standard scaling laws.
new_line
text
- The models undergo two-phase pre-training using a mix of web sources and synthetic data for general knowledge and logical reasoning skills.
- The models undergo two-phase pre-training using a mix of web sources and synthetic data for general knowledge and logical reasoning skills.
new_line
new_line
text
Performance:
Performance:
new_line
text
- Phi-3-mini achieves competitive scores on standard benchmarks like MMLU and MT-Bench, indicating strong reasoning capabilities.
- Phi-3-mini achieves competitive scores on standard benchmarks like MMLU and MT-Bench, indicating strong reasoning capabilities.
new_line
text
- Higher variants show even better performance, suggesting effective scaling with increased model size.
- Higher variants show even better performance, suggesting effective scaling with increased model size.
new_line
new_line
text
Limitations:
Limitations:
new_line
text
- phi-3-mini: limited by its smaller size in tasks requiring extensive factual knowledge, primarily  supports English.
- phi-3-mini: limited by its smaller size in tasks requiring extensive factual knowledge, primarily  supports English.
new_line
text
- phi-3-small limited multilingual support.
- phi-3-small limited multilingual support.
new_line
new_line
text
Hosting LLMs locally is a big win for OSS - private, secured inferencing on the go😎
Hosting LLMs locally is a big win for OSS - private, secured inferencing on the go😎
All You need To Know About Phi-3 (Technical Report Walkthrough)

Summary of Summaries:
Phi-3-mini
- Architecture specs: decoder-only transformer, ModelSize: 3.8 billion 
  parameters,  LongRope [ 128K Context length ], Vocab Size [ 32064 ], 
  trained on 3.3 trillion tokens. at bfloat16.
- Rivals performance to larger models like Mixtral 8x7B and GPT-3.5, 
  capable of running locally on a smartphone.
- Utilizes high quality training dataset heavily filtered from web data and 
   llm-generated synthetic data.
- Can be quantized to 4-bits, occupying ≈ 1.8GB of memory.
- Ran natively on iPhone 14 with A16 Bionic chip with inference speed of up 
   to 12 tokens per second. 

Phi-3-small
- Architecture specs: Also decoder-only, 7B parameters, Vocab size [ 100352 ], default context length [ 8k ], Context Length: 8K, Hidden Dimension: 4096, Number of Heads and Layers: Follows 7B class structure.
- Uses tiktoken tokenizer (for enhanced multilingual tokenization) 

Phi-3-medium: 
-  Architecture specs: Also decoder-only, Hidden Dimension: 5120, Number of Heads: 40, Number of Layers: 40, Tokenization: Consistent with other models, Training on 4.8 trillion tokens.

Training Methodology:
- Focuses on high-quality training data deviating from standard scaling laws.
- The models undergo two-phase pre-training using a mix of web sources and synthetic data for general knowledge and logical reasoning skills.

Performance:
- Phi-3-mini achieves competitive scores on standard benchmarks like MMLU and MT-Bench, indicating strong reasoning capabilities.
- Higher variants show even better performance, suggesting effective scaling with increased model size.

Limitations:
- phi-3-mini: limited by its smaller size in tasks requiring extensive factual knowledge, primarily  supports English.
- phi-3-small limited multilingual support.

Hosting LLMs locally is a big win for OSS - private, secured inferencing on the go😎
👍
LHPKAI heydrnik Tonic xiyuzhai Akhil2507 mrfakename Adriato MissKinziRoze victor rashakol19 AtAndDev youknowwhatAImean EddyGiusepe cuongtaongoc
🚀
StatsGary AtAndDev
/avatars/b2008910996680695d9107b8195a2ca2.svg
Sebastien
sebastienbo
user
https://cdn-avatars.huggingface.co/v1/production/uploads/6438a9027de34e8ea7e4b257/vib8QSd1AWMr_bR9ig_xJ.jpeg
Jaward Sesay
Jaward
user