text
How would you benchmark performance estimation algorithms vs data drift signals?
How would you benchmark performance estimation algorithms vs data drift signals?
new_line
new_line
text
I'm working on a benchmarking analysis, and I'm currently doing the following:
I'm working on a benchmarking analysis, and I'm currently doing the following:
new_line
new_line
text
-  Get univariate and multivariate drift signals and measure their correlation with realized performance.
-  Get univariate and multivariate drift signals and measure their correlation with realized performance.
new_line
text
- Use drift signals as features of a regression model to predict the model's performance.
- Use drift signals as features of a regression model to predict the model's performance.
new_line
text
- Use drift signals as features of a classification model to predict a performance drop.
- Use drift signals as features of a classification model to predict a performance drop.
new_line
text
- Compare all the above experiments with results from Performance Estimation algorithms.
- Compare all the above experiments with results from Performance Estimation algorithms.
new_line
new_line
text
Any other ideas?
Any other ideas?
How would you benchmark performance estimation algorithms vs data drift signals?

I'm working on a benchmarking analysis, and I'm currently doing the following:

-  Get univariate and multivariate drift signals and measure their correlation with realized performance.
- Use drift signals as features of a regression model to predict the model's performance.
- Use drift signals as features of a classification model to predict a performance drop.
- Compare all the above experiments with results from Performance Estimation algorithms.

Any other ideas?
🔥
osanseviero