Nutrinalyzer: Leveraging Multimodal LLMs for Enhanced Food Recognition and Dietary Analysis in Type 2 Diabetes Management
Abstract (300 words maximum)
This research project aims to benchmark the performance of various large language models (LLMs) such as GPT-4 and Claude 3 Opus in accurately analyzing nutritional content from food images to support blood sugar management in type 2 diabetes patients. The central question explores whether LLMs can match or surpass expert nutritionists in providing comprehensive dietary analysis from visual data, and how additional contextual information can refine these assessments. After initial nutritional estimation by the LLMs, we provide them with comprehensive multimodal data — such as food age, storage conditions, genetic modification status, preparation/cooking methods, refrigeration period, and user dietary restrictions — to determine if there is any improvement in nutritional analysis. Our methodology involves capturing a diverse dataset of food images under various conditions, prompting LLMs with different techniques (e.g., one-shot, zero-shot, instructional), and comparing their outputs against expert nutritionist and endocrinologist evaluations. We also conduct error analysis to identify systematic biases and assess confidence scoring to evaluate the correlation between LLM confidence and accuracy. Expected results include a robust understanding of LLM capabilities in nutritional estimation and a prototype application that allows users to manually input corrections for personalized dietary recommendations. The project will culminate in a small-scale trial with diabetes patients to evaluate the real-world applicability of an LLM-powered smartphone app for diabetic diet management. The research will also explore future applications, including the estimation of micronutrients, glycemic indices, and integration of other health data, such as physical activity levels and medication adherence to create a comprehensive dietary management tool that not only personalizes meal recommendations but also supports holistic health management for diabetes patients. Ethical considerations regarding biases and privacy will be critically examined. Ultimately, this study seeks to advance the usability of AI technologies in dietary management and improve health outcomes for individuals managing diabetes.
Academic department under which the project should be listed
CCSE - Computer Science
Primary Investigator (PI) Name
Maria Valero de Clemente
Nutrinalyzer: Leveraging Multimodal LLMs for Enhanced Food Recognition and Dietary Analysis in Type 2 Diabetes Management
This research project aims to benchmark the performance of various large language models (LLMs) such as GPT-4 and Claude 3 Opus in accurately analyzing nutritional content from food images to support blood sugar management in type 2 diabetes patients. The central question explores whether LLMs can match or surpass expert nutritionists in providing comprehensive dietary analysis from visual data, and how additional contextual information can refine these assessments. After initial nutritional estimation by the LLMs, we provide them with comprehensive multimodal data — such as food age, storage conditions, genetic modification status, preparation/cooking methods, refrigeration period, and user dietary restrictions — to determine if there is any improvement in nutritional analysis. Our methodology involves capturing a diverse dataset of food images under various conditions, prompting LLMs with different techniques (e.g., one-shot, zero-shot, instructional), and comparing their outputs against expert nutritionist and endocrinologist evaluations. We also conduct error analysis to identify systematic biases and assess confidence scoring to evaluate the correlation between LLM confidence and accuracy. Expected results include a robust understanding of LLM capabilities in nutritional estimation and a prototype application that allows users to manually input corrections for personalized dietary recommendations. The project will culminate in a small-scale trial with diabetes patients to evaluate the real-world applicability of an LLM-powered smartphone app for diabetic diet management. The research will also explore future applications, including the estimation of micronutrients, glycemic indices, and integration of other health data, such as physical activity levels and medication adherence to create a comprehensive dietary management tool that not only personalizes meal recommendations but also supports holistic health management for diabetes patients. Ethical considerations regarding biases and privacy will be critically examined. Ultimately, this study seeks to advance the usability of AI technologies in dietary management and improve health outcomes for individuals managing diabetes.