text
🔍 Meta teams use a fine-tuned Llama model to fix production issues in seconds
🔍 Meta teams use a fine-tuned Llama model to fix production issues in seconds
new_line
new_line
text
One of Meta's engineering teams shared how they use a fine-tuned small Llama (Llama-2-7B, so not even a very recent model) to identify the root cause of production issues with 42% accuracy.
One of Meta's engineering teams shared how they use a fine-tuned small Llama (Llama-2-7B, so not even a very recent model) to identify the root cause of production issues with 42% accuracy.
new_line
new_line
text
🤔 42%, is that not too low?
🤔 42%, is that not too low?
new_line
text
➡️ Usually, whenever there's an issue in production, engineers dive into recent code changes to find the offending commit. At Meta's scale (thousands of daily changes), this is like finding a needle in a haystack.
➡️ Usually, whenever there's an issue in production, engineers dive into recent code changes to find the offending commit. At Meta's scale (thousands of daily changes), this is like finding a needle in a haystack.
new_line
text
💡 So when the LLM-based suggestion is right, it cuts incident resolution time from hours to seconds!
💡 So when the LLM-based suggestion is right, it cuts incident resolution time from hours to seconds!
new_line
new_line
text
How did they do it?
How did they do it?
new_line
new_line
text
🔄 Two-step approach:
🔄 Two-step approach:
new_line
text
‣ Heuristics (code ownership, directory structure, runtime graphs) reduce thousands of potential changes to a manageable set
‣ Heuristics (code ownership, directory structure, runtime graphs) reduce thousands of potential changes to a manageable set
new_line
text
‣ Fine-tuned Llama 2 7B ranks the most likely culprits
‣ Fine-tuned Llama 2 7B ranks the most likely culprits
new_line
new_line
text
🎓 Training pipeline:
🎓 Training pipeline:
new_line
text
‣ Continued pre-training on Meta's internal docs and wikis
‣ Continued pre-training on Meta's internal docs and wikis
new_line
text
‣ Supervised fine-tuning on past incident investigations
‣ Supervised fine-tuning on past incident investigations
new_line
text
‣ Training data mimicked real-world constraints (2-20 potential changes per incident)
‣ Training data mimicked real-world constraints (2-20 potential changes per incident)
new_line
new_line
text
🔮 Now future developments await:
🔮 Now future developments await:
new_line
text
‣ Language models could handle more of the incident response workflow (runbooks, mitigation, post-mortems)
‣ Language models could handle more of the incident response workflow (runbooks, mitigation, post-mortems)
new_line
text
‣ Improvements in model reasoning should boost accuracy further
‣ Improvements in model reasoning should boost accuracy further
new_line
new_line
text
Read it in full 👉 
Read it in full 👉 
link
https://www.tryparity.com/blog/how-meta-uses-llms-to-improve-incident-response
https://www.tryparity.com/blog/how-meta-uses-llms-to-improve-incident-response
🔍 Meta teams use a fine-tuned Llama model to fix production issues in seconds

One of Meta's engineering teams shared how they use a fine-tuned small Llama (Llama-2-7B, so not even a very recent model) to identify the root cause of production issues with 42% accuracy.

🤔 42%, is that not too low?
➡️ Usually, whenever there's an issue in production, engineers dive into recent code changes to find the offending commit. At Meta's scale (thousands of daily changes), this is like finding a needle in a haystack.
💡 So when the LLM-based suggestion is right, it cuts incident resolution time from hours to seconds!

How did they do it?

🔄 Two-step approach:
‣ Heuristics (code ownership, directory structure, runtime graphs) reduce thousands of potential changes to a manageable set
‣ Fine-tuned Llama 2 7B ranks the most likely culprits

🎓 Training pipeline:
‣ Continued pre-training on Meta's internal docs and wikis
‣ Supervised fine-tuning on past incident investigations
‣ Training data mimicked real-world constraints (2-20 potential changes per incident)

🔮 Now future developments await:
‣ Language models could handle more of the incident response workflow (runbooks, mitigation, post-mortems)
‣ Improvements in model reasoning should boost accuracy further

Read it in full 👉 https://www.tryparity.com/blog/how-meta-uses-llms-to-improve-incident-response
👀
John6666
👍
rbgo