Food recognition and characterization using computer vision, intelligent sensors and artificial intelligence

It has been demonstrated the ability of the most recent computer vision and artificial intelligence methods to locate and recognize and characterize a limited set of foods if they are acquired in controlled contexts, and when the data quality is good (Ciocca et al. In Computer Vision and Image Understanding, 2018; Ciocca et al. IEEE Journal of Biomedical and Health aInformatics, 2017; Aslan et al. International Journal of Machine Learning and Cybernetics, 2020). In this project we want to extend this research by allowing systems to locate and recognize a much larger number of foods and, most challenging and important in real scenarios, to characterize them in terms of quantity and quality.

As part of the design and testing of interventions and strategies for healthy and sustainable eating 

• in public, corporate, school canteens, etc., and non-domestic settings such as hospitals and nursing homes where meals only usually served on trays;

• in public venues such as restaurants and fast food outlets, food halls, etc.

The planned research activity includes: the design, development and experimental validation of methods and algorithms based on Artificial Intelligence, Computer Vision and Intelligent Sensors techniques for localization, recognition and characterization in digital images and videos of prepared or packaged food. The methods and algorithms should be scalable and adaptable to different realities of the restaurant industry by providing for the use of artificial intelligence approaches based on incremental learning techniques. 

We intend to test/evaluate the developed algorithms and methods in prototype applications and in real use scenarios.

A prototype system for food recognition and characterization may include: 

• computational unit for processing, storage and communication;
• smart scale for food weight estimation;
• binocular/stereo camera for depth/3D image acquisition;
• additional cameras in non-visible spectrum;
• proximity sensors for food detection;
• radio frequency identification and communication devices;
• adaptive lighting system;
• touch displays for user-machine interaction.

Figure 1 shows a sketch of the intended prototype system.

• A smart system for monitoring food service with analysis of how much is served, how much is consumed, and how much is left over. On the user side, such a system will enable more accurate compilation of the food diary by allowing a more accurate estimate of the actual food consumed. Its use, on the side of those who prepare and dispense meals, will allow for improved galley management and food preparation, thereby minimizing food waste;

• From a Computer Vision and Artificial point of view: Novel algorithms and methods advancing the state-of-the-art in automatic food analysis tasks.