Decrypting Delicious Meals – “Artificial Intelligence and Computer Vision for Dietary Assessment”
Do you know how many calories are in a serving of steak, mashed potatoes and green beans? Do you know the carbohydrate content of a freshly baked apple pie? Most of us presumably do not know, and looking at the expanding waistlines around the world, most of us would prefer not to. But for most insulin-dependent diabetics, ignorance about their daily diet is not an option. With every meal, it is not the pleasure of having lovely food, but the immediate need to calculate the carbohydrate content and the corresponding insulin dose. This is vital to avoid the serious side effects of incorrect carbohydrate estimation and insulin dose. For most diabetics, negotiating this as part of work canteens, restaurants and holidays, can be a challenge.
To empower diabetic patients the Diabetes Technology Research (DTR) laboratory has developed GoCARB, a smartphone App that harnesses the synergy of artificial intelligence (AI) and computer vision, to give actionable data to users on carbs.
How does it work? The user places a credit card-sized reference object next to the dish, and takes two photographs from different points of view. One of the photos is used to detect, segment and recognize automatically the existing food items, while semiautomatic tools are also provided for correcting the results, if needed. By using both photos and the card, we build a 3D model of the food itself. With this 3D model of the different foods, you can calculate their volume. Once you know the volume and food type and using nutrient databases, you can calculate the carbohydrate content. In house clinical studies together with the Bern University Clinic for Diabetology, Endocrinology, Nutritional Medicine & Metabolism of Inselspital have shown that this is superior to getting the diabetic patient to estimate the carbohydrate content and that glucose control is then more precise.
A working GoCARB prototype is being used in a clinical pilot study: AI and computer vision for dietary monitoring and assessment, the result of 10 years of basic, applied and translational research. The team has reached an agreement with an international diabetes care company on the further development of the system for patient, hospital and use in clinical trials. Future work will include adapting GoCARB to monitor malnutrition in children and the elderly. For more information, refer to GoCARB.