Pic2Recipe: The recipe in a photograph

Foodstagramming is one of the hottest trends around. Taking photographs of the dish we’ve ordered even before tasting it has become an important ingredient in our social media potpourri. This might have changed the way we perceive food, but researchers at the Massachusetts Institute of Technology’s Computer Science and Artificial Intelligence Laboratory (Csail) have developed a neural network that could actually change the way we cook and understand our dietary habits. Those hundreds of food images in your smartphone can finally be put to good use.

The network, called Recipe1M, has a database of a million recipes and images that the researchers selected from different food-related websites. This data was then used to train the Artificial Intelligence network to identify patterns and connections between food items, ingredients and recipes. The resulting portal is known as Pic2Recipe!.

Users can go to Pic2Recipe! (tuesday.csail.mit.edu on mobile browsers) and choose from one of the many preloaded images to get the corresponding recipe or upload their own image to find out how to cook the restaurant dish they’ve just tasted. While the website wasn’t too responsive, Pic2Recipe! works just fine on the mobile version. A YouTube video by Csail shows how Pic2Recipe! correctly identified eight out of the 11 ingredients needed to make sugar cookies. It missed out on details like the icing on the cookies.

We decided to go with one of the preloaded pictures. One of the dishes looked fairly nondescript at first glance, but the recipe turned out to be one for a rich and creamy pea soup, complete with split peas, shallots and carrots. At the moment, Pic2Recipe! doesn’t show a matching recipe for every preloaded image, but it identifies desserts quite accurately. An image of a frosty lemon chiffon pie took us to the recipe, where we learnt the key ingredients were generous portions of whipped cream and lemon curd.

The portal also has a link to the source of the recipe. The chiffon pie recipe, for instance, was sourced from the Food Network.

When we tried uploading some food images, the portal didn’t find any matching recipes. The network and portal are still in an early stage of development, but going forward the researchers plan to use food data more extensively. According to a release on the Csail website, “In the future, the team hopes to be able to improve the system so that it can understand food in even more detail…. The researchers are also interested in potentially developing the system into a ‘dinner aide’ that could figure out what to cook given a dietary preference and a list of items in the fridge."