mistralai/Mistral-7B-Instruct-v0.3
Text Generation
•
Updated
•
2.56M
•
•
1.26k
Gary Hutson
StatsGary
AI & ML interests
- Deep Learning
- Transformers for MLM
- Transformers for seq2seq classification
- NLP
- Computer Vision
Recent Activity
liked
a model
about 1 month ago
mistralai/Mistral-7B-Instruct-v0.3
liked
a Space
7 months ago
EmilyWitko/Hugging_Face_Values
liked
a Space
7 months ago
andito/Florence-2-DocVQA
Organizations
None yet
StatsGary's activity
liked
a
model
about 1 month ago
liked
2
Spaces
7 months ago
liked
a
Space
8 months ago
upvoted
an
article
8 months ago
Article
Getting Started With Embeddings
•
44
updated
a
model
8 months ago
liked
a
dataset
8 months ago
updated
a
model
8 months ago
liked
a
model
8 months ago
updated
a
collection
8 months ago
updated
a
dataset
9 months ago
liked
a
Space
9 months ago
updated
a
model
9 months ago
Enable inference API
2
#49 opened about 1 year ago
by
mrfakename
liked
a
model
9 months ago
liked
a
Space
9 months ago
reacted to
Jaward's
post with 🚀
9 months ago
Post
5347
All You need To Know About Phi-3 (Technical Report Walkthrough)
Summary of Summaries:
Phi-3-mini
- Architecture specs: decoder-only transformer, ModelSize: 3.8 billion
parameters, LongRope [ 128K Context length ], Vocab Size [ 32064 ],
trained on 3.3 trillion tokens. at bfloat16.
- Rivals performance to larger models like Mixtral 8x7B and GPT-3.5,
capable of running locally on a smartphone.
- Utilizes high quality training dataset heavily filtered from web data and
llm-generated synthetic data.
- Can be quantized to 4-bits, occupying ≈ 1.8GB of memory.
- Ran natively on iPhone 14 with A16 Bionic chip with inference speed of up
to 12 tokens per second.
Phi-3-small
- Architecture specs: Also decoder-only, 7B parameters, Vocab size [ 100352 ], default context length [ 8k ], Context Length: 8K, Hidden Dimension: 4096, Number of Heads and Layers: Follows 7B class structure.
- Uses tiktoken tokenizer (for enhanced multilingual tokenization)
Phi-3-medium:
- Architecture specs: Also decoder-only, Hidden Dimension: 5120, Number of Heads: 40, Number of Layers: 40, Tokenization: Consistent with other models, Training on 4.8 trillion tokens.
Training Methodology:
- Focuses on high-quality training data deviating from standard scaling laws.
- The models undergo two-phase pre-training using a mix of web sources and synthetic data for general knowledge and logical reasoning skills.
Performance:
- Phi-3-mini achieves competitive scores on standard benchmarks like MMLU and MT-Bench, indicating strong reasoning capabilities.
- Higher variants show even better performance, suggesting effective scaling with increased model size.
Limitations:
- phi-3-mini: limited by its smaller size in tasks requiring extensive factual knowledge, primarily supports English.
- phi-3-small limited multilingual support.
Hosting LLMs locally is a big win for OSS - private, secured inferencing on the go😎
Summary of Summaries:
Phi-3-mini
- Architecture specs: decoder-only transformer, ModelSize: 3.8 billion
parameters, LongRope [ 128K Context length ], Vocab Size [ 32064 ],
trained on 3.3 trillion tokens. at bfloat16.
- Rivals performance to larger models like Mixtral 8x7B and GPT-3.5,
capable of running locally on a smartphone.
- Utilizes high quality training dataset heavily filtered from web data and
llm-generated synthetic data.
- Can be quantized to 4-bits, occupying ≈ 1.8GB of memory.
- Ran natively on iPhone 14 with A16 Bionic chip with inference speed of up
to 12 tokens per second.
Phi-3-small
- Architecture specs: Also decoder-only, 7B parameters, Vocab size [ 100352 ], default context length [ 8k ], Context Length: 8K, Hidden Dimension: 4096, Number of Heads and Layers: Follows 7B class structure.
- Uses tiktoken tokenizer (for enhanced multilingual tokenization)
Phi-3-medium:
- Architecture specs: Also decoder-only, Hidden Dimension: 5120, Number of Heads: 40, Number of Layers: 40, Tokenization: Consistent with other models, Training on 4.8 trillion tokens.
Training Methodology:
- Focuses on high-quality training data deviating from standard scaling laws.
- The models undergo two-phase pre-training using a mix of web sources and synthetic data for general knowledge and logical reasoning skills.
Performance:
- Phi-3-mini achieves competitive scores on standard benchmarks like MMLU and MT-Bench, indicating strong reasoning capabilities.
- Higher variants show even better performance, suggesting effective scaling with increased model size.
Limitations:
- phi-3-mini: limited by its smaller size in tasks requiring extensive factual knowledge, primarily supports English.
- phi-3-small limited multilingual support.
Hosting LLMs locally is a big win for OSS - private, secured inferencing on the go😎
- 4 replies
liked
a
model
9 months ago
upvoted
an
article
9 months ago
Article
How to Finetune phi-3 on MacBook Pro
By
•
•
65