2020/10/20 | Research | Artificial Intelligence

Single-View Dietary Assessment

Current computer vision approaches for food volume estimation require the input of multiple food images or additional information. The ARTORG Center lab for Artificial Intelligence in Health and Nutrition (AIHN) has developed a network architecture that only needs a single RGB food image to estimate food volumes and calculate nutrient information accordingly. The new approach to be presented at the 25th International Conference on Pattern Recognition in Milan, Italy, in January 2021 is the first to enable a full-pipeline single view dietary assessment.

The input of the proposed system is a single RGB food image; the outputs are the predicted depth map, the semantic prediction map and the table plane. The food 3D model is built based on these three outputs. ARTORG Center, Artificial Intelligence in Health and Nutrition

Food volume estimation is an essential step in the pipeline of dietary assessment by nutritionists and private persons wanting or needing to control their exact nutrition intake. Automatically assessing meal volumes and compositions via pictures taken with a smartphone remains complicated as systems need to receive several food pictures as well as additional information on the depth of the picture surface and food recipients. A particular challenge are large texture-less areas in the pictures which tend to deteriorate the accuracy of unsupervised depth estimation.

To tackle these challenges, the AIHN laboratory proposes to use a network architecture that jointly performs geometric understanding (i.e., depth prediction and 3D plane estimation) and semantic prediction on a single food image, enabling a robust and accurate food volume estimation regardless of the texture characteristics of the target plane. Tested on two separate image databases, the network outperformed the state-of-the-art unsupervised network approach, achieving results comparable to a fully-supervised method. The new approach will be be refined in further studies.


Link to the study preprint

Artificial Intelligence in Health and Nutrition