text
We recently worked with OpenAI to fine-tune gpt-4o and built the SOTA model for the 
We recently worked with OpenAI to fine-tune gpt-4o and built the SOTA model for the 
resource
https://huggingface.co/datasets/patched-codes/static-analysis-eval
dataset
patched-codes/static-analysis-eval
https://huggingface.co/datasets/patched-codes/static-analysis-eval
text
 benchmark. All the code and data 
 benchmark. All the code and data 
resource
https://huggingface.co/datasets/patched-codes/synth-vuln-fixes
dataset
patched-codes/synth-vuln-fixes
https://huggingface.co/datasets/patched-codes/synth-vuln-fixes
text
 on how we did it is available on their GitHub - 
 on how we did it is available on their GitHub - 
link
https://github.com/openai/build-hours/tree/main/5-4o_fine_tuning
https://github.com/openai/build-hours/tree/main/5-4o_fine_tuning
text
.
.
new_line
new_line
text
Here are some tips based on our experience:
Here are some tips based on our experience:
new_line
new_line
text
→ Establish baseline with "conditioning" / prompting
→ Establish baseline with "conditioning" / prompting
new_line
new_line
text
→ Task-specific datasets are ideal for PEFT; hard to beat gpt-4o on "broad" tasks
→ Task-specific datasets are ideal for PEFT; hard to beat gpt-4o on "broad" tasks
new_line
new_line
text
→ Add your best system prompt to each example
→ Add your best system prompt to each example
new_line
new_line
text
→ Ensure training data distribution is similar to inference data
→ Ensure training data distribution is similar to inference data
new_line
new_line
text
→ Shorten instructions with concise prompts; may require more examples.
→ Shorten instructions with concise prompts; may require more examples.
new_line
new_line
text
→ Define clear evaluation metrics (seriously, please eval!)
→ Define clear evaluation metrics (seriously, please eval!)
new_line
new_line
text
You can see more details on the benchmark and process here - 
You can see more details on the benchmark and process here - 
link
https://www.patched.codes/blog/the-static-analysis-evaluation-benchmark-measuring-llm-performance-in-fixing-software-vulnerabilities
https://www.patched.codes/blog/the-static-analysis-evaluation-benchmark-measuring-llm-performance-in-fixing-software-vulnerabilities
We recently worked with OpenAI to fine-tune gpt-4o and built the SOTA model for the https://huggingface.co/datasets/patched-codes/static-analysis-eval benchmark. All the code and data https://huggingface.co/datasets/patched-codes/synth-vuln-fixes on how we did it is available on their GitHub - https://github.com/openai/build-hours/tree/main/5-4o_fine_tuning.

Here are some tips based on our experience:

→ Establish baseline with "conditioning" / prompting

→ Task-specific datasets are ideal for PEFT; hard to beat gpt-4o on "broad" tasks

→ Add your best system prompt to each example

→ Ensure training data distribution is similar to inference data

→ Shorten instructions with concise prompts; may require more examples.

→ Define clear evaluation metrics (seriously, please eval!)

You can see more details on the benchmark and process here - https://www.patched.codes/blog/the-static-analysis-evaluation-benchmark-measuring-llm-performance-in-fixing-software-vulnerabilities
❤️
codelion rikhoffbauer2 Mackin7 KingNish
🔥
codelion KingNish Bruhn
👀
John6666 codelion
🚀
codelion
🤗
codelion
➕
codelion