Skip to Content
You are currently on the new version of our website. Access the old version .
  • Article
  • Open Access

18 November 2025

23 Pages

Model-Aware Automatic Benchmark Generation with Self-Error Instructions for Data-Driven Models

and
1
Research Center “Strong Artificial Intelligence in Industry”, ITMO University, Birzhevaya Liniya 14, Saint Petersburg 199034, Russia
2
Netherlands Institute of Advanced Studies, Korte Spinhuissteeg 3, 1012 CG Amsterdam, The Netherlands
*
Author to whom correspondence should be addressed.

Abstract

The growing number of domain-specific machine learning benchmarks has driven methodological progress, yet real-world deployments require a different evaluation approach. Model-aware synthetic benchmarks, designed to emphasize failure modes of existing models, are proposed to address this need. However, evaluating already well-performing models presents a significant challenge, as the limited number of high-quality data points where they exhibit errors makes it difficult to obtain statistically reliable estimates. To address this gap, we proposed a two-step benchmark construction process: (i) using a genetic algorithm to augment the data points where data-driven models exhibit poor prediction quality; (ii) using a generative model to approximate the distribution of these points. We established a general formulation for such benchmark construction, which can be adapted to non-classical machine learning models. Our experimental study demonstrates that our approach enables the accurate evaluation of data-driven models for both regression and classification problems.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.