text
New #NVIDIA paper: Improving Hyperparameter Optimization with Checkpointed Model Weights
New #NVIDIA paper: Improving Hyperparameter Optimization with Checkpointed Model Weights
new_line
new_line
text
Hyperparameter optimization often dominates the cost of model design.  So, we want cheap surrogate functions that approximate model performance to guide our search.  Existing methods can train on optimization metadata – like a trajectory of losses – to build these surrogates.
Hyperparameter optimization often dominates the cost of model design.  So, we want cheap surrogate functions that approximate model performance to guide our search.  Existing methods can train on optimization metadata – like a trajectory of losses – to build these surrogates.
new_line
new_line
text
In our work, we add the ability to train our hyperparameter optimization surrogates on checkpointed model weights with a graph metanetwork.  This allows us to leverage a large, pre-existing source of information that can featurize the architecture, dataset, losses, and optimization procedure.
In our work, we add the ability to train our hyperparameter optimization surrogates on checkpointed model weights with a graph metanetwork.  This allows us to leverage a large, pre-existing source of information that can featurize the architecture, dataset, losses, and optimization procedure.
new_line
new_line
text
🔍Project page: 
🔍Project page: 
link
https://research.nvidia.com/labs/toronto-ai/FMS/
https://research.nvidia.com/labs/toronto-ai/FMS/
new_line
text
👨‍💻 Code for reproduction: 
👨‍💻 Code for reproduction: 
link
https://github.com/NVlabs/forecasting-model-search
https://github.com/NVlabs/forecasting-model-search
new_line
text
📄 Full Paper: 
📄 Full Paper: 
link
https://arxiv.org/abs/2406.18630
https://arxiv.org/abs/2406.18630
text
new_line
new_line
text
Our project was a collaboration between NVIDIA’s Toronto AI Lab and the TAO team.
Our project was a collaboration between NVIDIA’s Toronto AI Lab and the TAO team.
new_line
new_line
text
Check out more work from Toronto AI Lab here: 
Check out more work from Toronto AI Lab here: 
link
https://research.nvidia.com/labs/toronto-ai/
https://research.nvidia.com/labs/toronto-ai/
text
new_line
new_line
text
You can view the TAO toolkit here: 
You can view the TAO toolkit here: 
link
https://developer.nvidia.com/tao-toolkit
https://developer.nvidia.com/tao-toolkit
New #NVIDIA paper: Improving Hyperparameter Optimization with Checkpointed Model Weights

Hyperparameter optimization often dominates the cost of model design.  So, we want cheap surrogate functions that approximate model performance to guide our search.  Existing methods can train on optimization metadata – like a trajectory of losses – to build these surrogates.

In our work, we add the ability to train our hyperparameter optimization surrogates on checkpointed model weights with a graph metanetwork.  This allows us to leverage a large, pre-existing source of information that can featurize the architecture, dataset, losses, and optimization procedure.

🔍Project page: https://research.nvidia.com/labs/toronto-ai/FMS/
👨‍💻 Code for reproduction: https://github.com/NVlabs/forecasting-model-search
📄 Full Paper: https://arxiv.org/abs/2406.18630 

Our project was a collaboration between NVIDIA’s Toronto AI Lab and the TAO team.

Check out more work from Toronto AI Lab here: https://research.nvidia.com/labs/toronto-ai/ 

You can view the TAO toolkit here: https://developer.nvidia.com/tao-toolkit
🔥
yifei01 osanseviero Joseph717171 ioda-idg atasoglu lorraine2
🤗
yifei01 Joseph717171 John6666 lorraine2
🚀
yifei01 Joseph717171 lorraine2