Background/Objectives: Estimating energy and macronutrients from food images is clinically relevant yet challenging, and rigorous evaluation requires transparent accuracy metrics with uncertainty and clear acknowledgement of reference data limitations across heterogeneous sources. This study assessed ChatGPT-5, a general-purpose vision-language model, across four
[...] Read more.
Background/Objectives: Estimating energy and macronutrients from food images is clinically relevant yet challenging, and rigorous evaluation requires transparent accuracy metrics with uncertainty and clear acknowledgement of reference data limitations across heterogeneous sources. This study assessed ChatGPT-5, a general-purpose vision-language model, across four scenarios differing in the amount and type of contextual information provided, using a composite dataset to quantify accuracy for calories and macronutrients.
Methods: A total of 195 dishes were evaluated, sourced from Allrecipes.com, the SNAPMe dataset, and Home-prepared, weighed meals. Each dish was evaluated under Case 1 (image only), Case 2 (image plus standardized non-visual descriptors), Case 3 (image plus ingredient lists with amounts), and Case 4 (replicates Case 3 but excluding the image). The primary endpoint was kcal Mean Absolute Error (MAE); secondary endpoints included Median Absolute Error (MedAE) and Root Mean Square Error (RMSE) for kcal and macronutrients (protein, carbohydrates, and lipids), all reported with 95% Confidence Intervals (CIs) via dish-level bootstrap resampling and accompanied by absolute differences (Δ) between scenarios. Inference settings were standardized to support reproducibility and variance estimation. Source stratified analyses and quartile summaries were conducted to examine heterogeneity by curation level and nutrient ranges, with additional robustness checks for error complexity relationships.
Results and Discussion: Accuracy improved from Case 1 to Case 2 and further in Case 3 for energy and all macronutrients when summarized by MAE, MedAE, and RMSE with 95% CIs, with absolute reductions (Δ) indicating material gains as contextual information increased. In contrast to Case 3, estimation accuracy declined in Case 4, underscoring the contribution of visual cues. Gains were largest in the Home-prepared dietitian-weighed subset and smaller yet consistent for Allrecipes.com and SNAPMe, reflecting differences in reference curation and measurement fidelity across sources. Scenario-level trends were concordant across sources, and stratified and quartile analyses showed coherent patterns of decreasing absolute errors with the provision of structured non-visual information and detailed ingredient data.
Conclusions: ChatGPT-5 can deliver practically useful calorie and macronutrient estimates from food images, particularly when augmented with standardized nonvisual descriptors and detailed ingredients, as evidenced by reductions in MAE, MedAE, and RMSE with 95% CIs across scenarios. The decline in accuracy observed when the image was omitted, despite providing detailed ingredient information, indicates that visual cues contribute meaningfully to estimation performance and that improvements are not solely attributable to arithmetic from ingredient lists. Finally, to promote generalizability, it is recommended that future studies include repeated evaluations across diverse datasets, ensure public availability of prompts and outputs, and incorporate systematic comparisons with non-artificial-intelligence baselines.
Full article
