text
Can language models replace developers? #RepoCod says “Not Yet”, because GPT-4o and other LLMs have <30% accuracy/pass@1 on real-world code generation tasks. 
Can language models replace developers? #RepoCod says “Not Yet”, because GPT-4o and other LLMs have <30% accuracy/pass@1 on real-world code generation tasks. 
new_line
text
- Leaderboard 
- Leaderboard 
link
https://lt-asset.github.io/REPOCOD/
https://lt-asset.github.io/REPOCOD/
new_line
text
- Dataset: 
- Dataset: 
resource
https://huggingface.co/datasets/lt-asset/REPOCOD
dataset
lt-asset/REPOCOD
https://huggingface.co/datasets/lt-asset/REPOCOD
new_line
mention
@jiang719
jiang719
text
mention
@shanchao
shanchao
text
mention
@Yiran-Hu1007
Yiran-Hu1007
text
new_line
text
Compared to #SWEBench, RepoCod tasks are
Compared to #SWEBench, RepoCod tasks are
new_line
text
- General code generation tasks, while SWE-Bench tasks resolve pull requests from GitHub issues.
- General code generation tasks, while SWE-Bench tasks resolve pull requests from GitHub issues.
new_line
text
- With 2.6X more tests per task (313.5 compared to SWE-Bench’s 120.8).
- With 2.6X more tests per task (313.5 compared to SWE-Bench’s 120.8).
new_line
new_line
text
Compared to #HumanEval, #MBPP, #CoderEval, and #ClassEval, RepoCod has 980 instances from 11 Python projects, with
Compared to #HumanEval, #MBPP, #CoderEval, and #ClassEval, RepoCod has 980 instances from 11 Python projects, with
new_line
text
- Whole function generation
- Whole function generation
new_line
text
- Repository-level context
- Repository-level context
new_line
text
- Validation with test cases, and 
- Validation with test cases, and 
new_line
text
- Real-world complex tasks: longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00)
- Real-world complex tasks: longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00)
new_line
new_line
text
Introducing hashtag #RepoCod-Lite 🐟 for faster evaluations: 200 of the toughest tasks from RepoCod with:
Introducing hashtag #RepoCod-Lite 🐟 for faster evaluations: 200 of the toughest tasks from RepoCod with:
new_line
text
- 67 repository-level, 67 file-level, and 66 self-contains tasks
- 67 repository-level, 67 file-level, and 66 self-contains tasks
new_line
text
- Detailed problem descriptions (967 tokens) and long canonical solutions (918 tokens)
- Detailed problem descriptions (967 tokens) and long canonical solutions (918 tokens)
new_line
text
- GPT-4o and other LLMs have < 10% accuracy/pass@1 on RepoCod-Lite tasks. 
- GPT-4o and other LLMs have < 10% accuracy/pass@1 on RepoCod-Lite tasks. 
new_line
text
- Dataset: 
- Dataset: 
resource
https://huggingface.co/datasets/lt-asset/REPOCOD_Lite
dataset
lt-asset/REPOCOD_Lite
https://huggingface.co/datasets/lt-asset/REPOCOD_Lite
new_line
new_line
text
#LLM4code #LLM #CodeGeneration #Security
#LLM4code #LLM #CodeGeneration #Security
Can language models replace developers? #RepoCod says “Not Yet”, because GPT-4o and other LLMs have <30% accuracy/pass@1 on real-world code generation tasks. 
- Leaderboard https://lt-asset.github.io/REPOCOD/
- Dataset: https://huggingface.co/datasets/lt-asset/REPOCOD
@jiang719 @shanchao @Yiran-Hu1007 
Compared to #SWEBench, RepoCod tasks are
- General code generation tasks, while SWE-Bench tasks resolve pull requests from GitHub issues.
- With 2.6X more tests per task (313.5 compared to SWE-Bench’s 120.8).

Compared to #HumanEval, #MBPP, #CoderEval, and #ClassEval, RepoCod has 980 instances from 11 Python projects, with
- Whole function generation
- Repository-level context
- Validation with test cases, and 
- Real-world complex tasks: longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00)

Introducing hashtag #RepoCod-Lite 🐟 for faster evaluations: 200 of the toughest tasks from RepoCod with:
- 67 repository-level, 67 file-level, and 66 self-contains tasks
- Detailed problem descriptions (967 tokens) and long canonical solutions (918 tokens)
- GPT-4o and other LLMs have < 10% accuracy/pass@1 on RepoCod-Lite tasks. 
- Dataset: https://huggingface.co/datasets/lt-asset/REPOCOD_Lite

#LLM4code #LLM #CodeGeneration #Security
🔥
John6666 AdinaY clem lin-tan shanchao bhadresh-savani iky1e
🤗
prithivMLmods lin-tan shanchao mvelic
https://cdn-avatars.huggingface.co/v1/production/uploads/649a54b896d5747b35e2163b/tdZmsov6fN1VHztaE5kX9.jpeg
Vezora
Vezora
user