Research on Hyperparameters for Fine Tuning
I fine-tuned the databricks/Dolly-v2-3b with the b-mc2/sql-create-context dataset in order to get the SQL queries for the given context in response, but after Fine Tuning the model even gave worse results instead of SQL queries it gave random statements as a response. And also in SQL queries it is missing the conditions.SELECT count(*)FROM headWHERE ageSo, how should we configure the Hyperparameters and what is the relation between Hyperparameters and the model and also what is the best approach to do Fine-Tuning?
2024-02-21T17:03:33Z
you should start with low learning rates and also the amount of dataset depends
2024-02-26T14:56:41Z
Hi@Pekka10, you can try usinghttps://huggingface.co/defog/sqlcoder-7b-2to see if fits your needs. We use this particular prompt format:sql-eval/prompts/prompt.md at main · defog-ai/sql-eval · GitHub.p.s. I work for defog and am aiming to improve our OSS model so feel free to send any bugs my way.