A Three-Stage Transformer-Based Approach for Food Mass Estimation ^†

Abstract

Accurate food mass estimation is a key component of automated calorie estimation tools, and there is growing interest in leveraging image analysis for this purpose due to its ease of use and scalability. However, current methods face important limitations. Some rely on 3D sensors for depth estimation, which are not widely accessible to all users, while others depend on camera intrinsic parameters to estimate volume, reducing their adaptability across different devices. Furthermore, AI-based approaches that bypass these parameters often struggle with generalizability when applied to images captured using diverse sensors or camera settings. To overcome these challenges, we introduce a three-stage, transformer-based method for estimating food mass from RGB images, balancing accuracy, computational efficiency, and scalability. The first stage applies the Segment Anything Model (SAM 2) to segment food items in images from the SUECFood dataset. Next, we use the Global-Local Path Network (GLPN) to perform monocular depth estimation (MDE) on the Nutrition5k dataset, inferring depth information from a single image. These outputs are then combined through alpha compositing to generate enhanced composite images with precise object boundaries. Finally, a Vision Transformer (ViT) model processes the composite images to estimate food mass by extracting relevant visual and spatial features. Our method achieves notable improvements in accuracy compared to previous approaches, with a mean squared error (MSE) of 5.61 and a mean absolute error (MAE) of 1.07. Notably, this pipeline does not require specialized hardware like depth sensors or multi-view imaging, making it well-suited for practical deployment. Future work will explore the integration of ingredient recognition to support a more comprehensive dietary assessment system.

1. Introduction

Food mass estimation (FME) is a fundamental step in determining the nutritional content of meals and is central to calorie estimation (CE) systems. Over the past decade, increasing research efforts have focused on developing image-based and mobile solutions for accurate estimation of food volume and mass, as well as analyzing the food nutritional value. This growing interest is motivated by the need to support individuals monitoring their dietary intake, healthcare professionals overseeing chronic disease management, and researchers advancing digital nutrition tools. Recent scoping reviews have highlighted both the technological evolution of calorie estimation methods [1,2] and their clinical relevance for populations with weight-related chronic conditions [3]. Several studies, ranging from manual methods to deep-learning-based (DL) systems, have been conducted on FME and food volume estimation (FVE) as preliminary steps to CE. According to Konstantakopoulos et al. [1], these studies can be grouped into categories, namely, 3D reconstruction, depth camera, perspective transformation, and DL-based methods.

Three-dimensional reconstruction techniques require multiple images of the food taken from different angles. Using feature point extraction algorithms such as scale-invariant feature transform (SIFT) and speeded-up robust features (SURF), the system constructs a 3D model of the food [4,5]. While effective, this approach necessitates careful image capture and multiple snapshots, as well as high computation power and processing time, making it unsuitable for real-time applications. On the other hand, depth cameras capture the 3D structure of food, allowing for precise FVE with lower computational expense. However, these devices are costly and not commonly used in commercial applications [6,7]. Perspective transformation involves removing projective distortion from a 2D image to reconstruct the shape and size of food. This method requires a reference object within the image for accurate volume calculation. Although perspective transformation can handle irregularly shaped foods, it is sensitive to image capture conditions [8]. DL-based methods utilize advanced models such as generative adversarial networks (GANs) and convolutional neural networks (CNNs) to estimate the depth of food items from RGB images. These methods create depth maps and estimate food volume without the need for specialized cameras or multiple snapshots [9]. While DL-based approaches are more suitable for real-time applications, less costly, and user-friendly, there is room for improvement in estimation performance by using more recent and robust models.

Our study addresses this gap by introducing a cost-effective novel, and robust DL approach for food mass estimation using only RGB images, eliminating the need for reference objects. Designed for real-time deployment, our method provides improved accuracy and adaptability over existing solutions. Our main contributions are as follows:

• We propose a three-stage, transformer-based approach for enhanced FME. The stages involve image segmentation to extract food regions, MDE to extract depth information, and FME to predict the food mass on plates.
• We use four variations of the SAM 2 model for image segmentation based on the SUECFood dataset. The GLPN model is employed for MDE using the Nutrition5k dataset, and the ViT model is utilized for FME.
• We combine the image segmentation and MDE results using the AC method to ensure better contour definition in the depth images and, consequently, more accurate mass estimation namely, an MSE of 5.61 and an MAE of 1.07.
• We conduct a comparative study with prior research using the same dataset and demonstrate the superiority of our approach.

The remainder of this paper is organized as follows. The literature review is presented in Section 2. In Section 3, we describe our approach. The experimental results are provided in Section 4. Concluding remarks and future work are presented in Section 5.

2. Literature Review

While our study focuses on food mass estimation (FME), we also examine prior research on food volume estimation (FVE), as both applications tackle similar technical challenges, such as accurate measurement and spatial analysis, and are fundamentally linked by their reliance on density to translate volume into mass.

Konstantakopoulos et al. [10] used the stereo vision technique, which relies on multiple camera views to capture food images from different angles. These images were then processed to reconstruct the 3D object of the food and calculate its volume. Jia et al. [11] introduced a method for estimating food volume by first reconstructing the dining bowl and then isolating the bowl’s geometry from the food. By subtracting the bowl’s empty volume from the total, FVE could be performed more accurately. The authors of [12] proposed a two-stage approach for reconstructing detailed 3D models of food. The first stage involves capturing images from different angles and using the structure from motion (SfM) method to analyze these images, identify key points, and reconstruct the camera’s positions and orientations. The second stage refines the 3D models by adding depth details to each surface point using the multi-view stereo (MVS) method. Naritomi et al. [13] presented an FVE approach that reconstructs 3D meshes from a single image of a dish. After reconstructing the 3D meshes, they separated the actual dish from the plate and used the result to estimate the food volume.

The authors in [14] developed a mobile app for CE based on FVE using depth cameras. They leveraged depth-sensing technology to accurately capture the volume of food items. Thames et al. [15] conducted experiments for CE, micronutrient estimation, and FME using a dataset they collected, which includes RGB images and depth maps produced by a depth camera. Their main contribution is the creation of this comprehensive dataset.

Pouladzadeh et al. [16] proposed a solution that requires the user to place their thumb next to the dish when capturing the image. By knowing the dimensions of the user’s thumb, the system can calculate the dimensions of each food item. Similarly, Okamoto and Yanai [17] used the dimensions of a reference object to compute the real size of the food regions. On the other hand, the authors in [18] introduced a method for estimating food portion sizes from smartphone images without requiring a fiducial marker for scale, making the process less intrusive for users.

Lo et al. [19] presented a method that leverages point clouds and view synthesis to estimate food volume. The view synthesis technique creates intermediate views of food items that may not have been directly captured by the camera, using DL models. Han et al. [20] proposed DPF-Nutrition, a two-stage approach that fuses the features of RGB and predicted depth images for CE, FME, and micronutrient estimation. They adopted the depth prediction transformer (DPT) to generate depth maps and designed a cross-modal attention block (CAB) to extract and integrate the complementary features of RGB and predicted depth images. Similarly, Shao, et al. [21] introduced a three-component approach consisting of the backbone network, the feature fusion module, and the nutrition prediction module for CE, FME, and micronutrient estimation.

Although prior research has explored various methods for FME and FVE, contributing with valuable solutions, many gaps remain. For instance, approaches involving 3D reconstruction, while highly accurate, demand extensive computational resources and processing time, making them impractical for real-time or mobile applications. Depth camera-based methods provide precise depth information but are limited by the high cost, restricting their accessibility to the public. Additionally, older techniques relying on traditional image processing and simpler models often lack sufficient accuracy due to their inability to capture the complex textures and shapes of food. Our approach stands out by leveraging RGB images and advanced transformer models, notably SAM 2 for image segmentation, GLPN for MDE, and ViT for FME. We employ the AC method to fuse the segmentation and MDE results, which plays a crucial role in enhancing the performance of our approach. This strategy enables us to achieve improvements in accuracy, computational efficiency, and cost-effectiveness.

3. Proposed Approach

Our approach consists of three stages. First, we segment the RGB images into regions while generating depth images from them. Second, we combine image segmentation and MDE results using the AC method. Finally, the resulting images are used for FME. The complete pipeline is illustrated in Figure 1.

Accordingly, the steps to implement our approach are detailed in the subsequent paragraphs.

3.1. Dataset

• MuseFood: The MuseFood dataset [22] is a large-scale Japanese food image dataset designed primarily for food segmentation tasks. The key features of the MuseFood dataset are as follows:

  ∘

  RGB Images: The dataset contains 31,395 images of various food items, providing a substantial resource for food image segmentation.

  ∘

  Masks: Each image in the dataset is accompanied by a pixel-level annotation, or mask.

• Nutrition5k: The Nutrition5k dataset [15] is a large-scale, diverse dataset designed specifically for food image recognition, mass estimation, and calorie estimation tasks in the context of dietary assessment. The key features of the Nutrition5k dataset are as follows:

  ∘

  RGB Images: The dataset includes 5000 unique food dishes, offering substantial diversity in food types and portion sizes. Each dish is captured from multiple angles to enhance the accuracy of 3D reconstruction and volume estimation techniques.

  ∘

  Depth Images: A depth map is provided for each dish, captured using a depth camera to represent the distance of the food from the camera at each pixel.

  ∘

  Annotations: The dataset includes detailed annotations, such as dish ingredients (e.g., eggplant, roasted potatoes, cauliflower), to facilitate classification tasks. It also provides mass per ingredient, total dish mass, caloric information per ingredient, total calories, as well as fat, carbohydrate, and protein content for each dish and ingredient.

3.2. Image Segmentation

For this task, we use SAM 2, a generalized transformer-based model developed by Meta for image and video segmentation tasks [23]. SAM 2 offers improved boundary precision, real-time processing speed, and scalability, making it an ideal choice for handling complex scenes in images and videos. Additionally, SAM 2’s few-shot, and zero-shot capabilities reduce annotation costs, making it efficient for deployment. We experiment with four variations: Hiera Base Plus (Hiera-B+), Hiera Large (Hiera-L), Hiera Small (Hiera-S), and Hiera Tiny (Hiera-T). These variations differ primarily in model size and capacity, with larger models (e.g., Hiera-L and Hiera-B+) containing more parameters for enhanced feature extraction, while smaller models (e.g., Hiera-S and Hiera-T) are more lightweight, offering faster processing at the potential cost of segmentation precision. The segmentation process is accomplished through three main steps The segmentation process is thus accomplished through three main steps:

• Fine-Tuning: Each segmentation model variant (Hiera-B+, Hiera-L, Hiera-S, and Hiera-T) was fine-tuned to partition images into distinct regions such as food ingredients versus background. This training was performed using RGB images and their corresponding segmentation masks from the SUECFood dataset. A total of 80% of the dataset was allocated for training purposes. The models were trained for 1000 epochs with a fixed learning rate of 0.0001.
• Testing: The evaluation phase was conducted using the remaining 20% of the dataset. The results from this testing phase are summarized in

  Table 1. Libraries and frameworks used in this study.

  Library	Version	Usage
  Torch	2.4.1	DL Framework
  Torchvision	0.19.1	Image Processing
  Transformers	4.45.2	Large Language Model Framework
  SAM 2	1.11.2	Image Segmentation
  Pillow	10.4.0	Image Manipulation
  Numpy	1.26.4	Numerical Computing

  .
• Inference: For inference, the best-performing checkpoint of each fine-tuned SAM 2 variant was applied to all RGB images in the Nutrition5k dataset. The goal of this inference step was to generate segmented versions of the images, ultimately facilitating mass estimation based on RGB inputs. This approach is critical because using the depth maps provided by the dataset directly would undermine the objective of the study, which emphasizes leveraging RGB images instead of depth camera data.

3.3. Monocular Depth Estimation

For the monocular depth estimation task, the GLPN transformer model was employed, as proposed by [24]. The GLPN model is particularly well-suited for this application due to its ability to capture both global contextual information and fine-grained local details from RGB inputs. The architecture of the model consists of two distinct pathways: a global pathway designed to extract large-scale structural information and a local pathway that focuses on detailed visual cues. These pathways are fused through attention mechanisms and feature aggregation modules, ensuring a balanced representation that maintains spatial coherence while preserving subtle depth variations.

Given the complexity and ingredient diversity in the dish images of the Nutrition5k dataset, GLPN is an appropriate choice for robust depth estimation. The depth estimation process consists of three primary stages:

• Fine-tuning: The model was trained to generate depth maps using RGB images paired with ground-truth depth images from the Nutrition5k dataset. As with segmentation, 80% of the dataset was used for training. The model was trained over 10 epochs with a learning rate of 0.0001 and a batch size of 512.
• Testing: The model was evaluated using the remaining 20% of the dataset to assess its generalization performance.
• Inference: The best checkpoint obtained during fine-tuning was used to infer depth images for the entire Nutrition5k dataset, using RGB inputs.

3.4. Alpha Compositing

In this stage, we use the AC method to combine the inferred images from the best segmentation variation with the inferred depth images [25]. The motivation for combining these results is that we observed the contours in the depth images provided by the Nutrition5k dataset are more defined, as illustrated in Figure 2. Moreover, images resulting from depth cameras are renowned for their superior performance compared to RGB images [1].

We observed that the contours in our segmented images were well-defined, unlike those in the depth images generated by the GLPN model. Therefore, we applied the AC method to overlay both images, using the depth image as the background with 0% transparency and the segmented image as the foreground with 39% transparency. This approach enhances the contours in the segmented regions while keeping the background depth image visible. The 39% value was determined empirically, as it ultimately delivered the most accurate results for FME during our experiments. The results are illustrated in Figure 3.

3.5. Food Mass Estimation

For this task, we use the ViT model proposed by [26]. ViT leverages the self-attention mechanism to capture long-range dependencies across an image. It divides images into patches (e.g., 16 × 16 pixels) and processes each patch as a token, enabling efficient parallelization. As a result, ViT is highly scalable for high-resolution images. Consequently, the FME process is carried out in two steps:

• Training: The model learns to estimate mass from the previously generated AC images. We train the model for 100 epochs, with a learning rate of 0.0001 and a batch size of 8 to avoid overloading the GPU. The data is split, with 80% used for training.
• Testing: Similarly to the previous stages, the model is evaluated on the remaining 20% of the dataset.

3.6. Experimental Settings

In this study, we use Python 3.11 as the programming language. Details on the versions and usage of frameworks and libraries are provided in Table 1. Furthermore, we use PyTorch 2.4.0 ’s CUDA autocast, a feature designed to optimize DL model performance by automatically applying mixed precision during training and inference. We also employ gradient accumulation to reduce the GPU memory load [27].

For computational resources, we utilize Compute Canada, a national high-performance computing system. Dynamic resource allocation is applied based on the submitted job requirements. Each job may execute several Python scripts in parallel, with each script assigned to a separate node. A node is a computational unit comprising 40 CPUs with two cores each (Intel Gold 6148 Skylake @ 2.4 GHz), four GPUs (NVIDIA V100 SXM2 with 16 GB of memory each), and 186 GB of RAM.

4. Experimental Results

In each stage, we conduct evaluations on the test sets to assess model performance. Accordingly, we structure our experimental results as follows:

4.1. Image Segmentation

Table 2. Image Segmentation Results.

SAM 2 Variation	MSE	IoU (%)
Hiera-T	0.0149	94.35
Hiera-S	0.0162	94.94
Hiera-B+	0.0196	94.97
Hiera-L	0.0086	95.30

presents the performance of the four variations mentioned in the approach. The reported metrics are MSE and intersection over union (IoU). MSE measures the difference between pixel values of the true and predicted masks, while IoU calculates the overlap between the predicted and the ground-truth region. Lower MSE values indicate better performance, while higher IoU percentages reflect more accurate overlap between the predicted and actual segmentation masks.

Hiera-L outperforms all other variations, achieving the lowest MSE (0.0086) and highest IoU (95.30%), indicating the most accurate segmentation and best overall performance. This is most likely due to Hiera-L’s larger architecture, which encompasses more parameters and allows for more robust feature extraction. Moreover, smaller models like Hiera-T or Hiera-S are more likely to underfit complex data, especially in cases where there is high variability within the images.

4.2. Monocular Depth Estimation

In this phase, we use the scale-invariant logarithmic (Silog) loss, which combines both depth differences and scale-invariance errors to calculate the distance between the logarithm of depth pixel values [28]. Lower Silog loss values indicate better performance.

The MDE Silog value is 1.1339, indicating that the depth maps generated from the RGB images using the GLPN model are moderately accurate. This is likely due to the intricacy of the depth images in the Nutrition5k dataset. Such intricacy makes it challenging for the GLPN model to achieve highly accurate results. Indeed, capturing finer details and subtle depth variations in food images is difficult, which can affect depth estimation accuracy.

Improving this result may require additional refinement, such as further fine-tuning or using other FME datasets, to better handle the complexity present in this dataset.

4.3. Food Mass Estimation

In this stage, we use MSE and MAE to evaluate the distance between the true and predicted masses based on the AC images, which combine depth and segmented images generated using Hiera-L, as it is the best alternative for image segmentation. The evaluation results are 5.61 for MSE and 1.07 for MAE.

The low error values demonstrate that combining depth and Hiera-L-based segmentation images using AC provides a comprehensive input representation for several reasons. First, the robustness of the Hiera-L variation allows it to capture complex patterns in food images due to its larger capacity. Second, complementary insights from MDE and image segmentation contribute to improved accuracy; depth images offer valuable spatial information about the distance and volume of food ingredients, while segmented images delineate the contours of food regions. Lastly, the AC method efficiently merges the depth and segmented images, making contours and boundaries more pronounced while preserving the spatial information provided by MDE.

We further conducted a comparative study with other research papers that used the RGB images of the Nutrition5k dataset. The results are shown in

Table 3. Comparative Study Results for FME.

Method	MAE
Naritomi et al. [13]	18.8
Lo et al. [19]	13.7
Yang et al. [18]	10.6
ours	1.07

.

The results presented in the table further demonstrate the superiority of our approach, as the reported MAE significantly outperforms that of previous studies.

5. Conclusions

In this study, we explored the use of advanced DL techniques to accurately estimate food mass. By leveraging state-of-the-art models such as Meta’s SAM 2, the GLPN transformer, and the ViT transformer, alongside the AC technique, the study aimed to address the challenges of FME using RGB images. AC was utilized to effectively combine segmented and depth images, enhancing the visibility of depth contours and improving accuracy in FME. The experimental results highlight the effectiveness of these models and techniques in achieving high accuracy compared to prior research. As a future direction, the next step in this study involves implementing ingredient recognition for each dish, which, when combined with FME, can significantly enhance the accuracy of calorie estimation. Additionally, ingredient recognition can be used to assess nutritional value in terms of fats, proteins, and carbohydrates, providing users with a more detailed analysis of their food intake.