Making smarter dietary choices

Background

I've been creating speculative concepts for Humane's recently unveiled AI-powered device for the past few months. Thus far, I've been writing about them in the order that I've posted them to Twitter, but I've decided to skip ahead to write about my most recently created one here because I find the ramifications of it to be extraordinarily exciting. Coincidentally, it also ties in with one of the use cases that was demoed during Imran Chaudhri's recent TED talk.

In this concept, I've decided to explore how a user could benefit from having a device that knows their dietary restrictions and preferences inside and out, and how they could be guided towards making healthier, and better choices in a restaurant setting.

I'm incredibly excited about this concept in particular, because I can see it having a major impact on my life and the lives of people close to me who are on restrictive diets.

The Opportunity

There are a few benefits to having a super-smart device help you make decisions on what to eat, and they range from nice-to-have, to completely life changing.

With a device that knows you and your preferences, you could have a more personalized experience when looking at restaurant menus. If you're ever overwhelmed by the amount of choices you have, something like this could make recommendations for what you may enjoy based on what you enjoyed from other restaurants in the past. It could also surface information on what items are popular at a restaurant, so you have additional information that can help you narrow down your choices. This kind of stuff is super nice to have.

On a more serious note, having to navigate the world of eating out when you have dietary restrictions can be challenging. It's difficult to know exactly what might trigger an allergy with limited information provided on menus, and for people who are accustomed to ordering with restrictions, this can lead to the need to have conversations with waiters or kitchen staff that can be time consuming. It's also a possibility that even the kitchen staff may not have all the information on what's included in something prior to ordering, leading you to making decisions you're not confident in, with the possibility for major health consequences if things go wrong.

An additional challenge occurs when you're new to having restrictions. Imagine being told by a doctor that you have celiac disease, when you've been eating an unrestriced diet for your entire life prior to this point. Imagine leaving an appointment with this diagnosis and the anxiety, fear, and uncertainty you'd feel about deciding what to eat. You might eat things that are bad for you without knowing it, or have trouble finding something that satisfies the restrictions you suddenly have to adhere to. You'd likely receive a pamphlet or be told what to avoid by your care provider, but it would be difficult to be confident about the decisions you're making right away.

Now imagine a device that could help you make suggestions on where to eat, and what to eat with this information in mind. In my concept, I imagine the device being able to project information on to a paper menu that would help to guide you to selections that make sense for you. The difference something like this could make in someone's life would be profound. Receiving a difficult health result like this is already stressful enough, but a device that knows exactly how to help you make the right decisions could enable you to confidently make healthy decisions from the moment you leave the doctor's office.

There are countless other examples of how this could help. Maybe you're recovering from a major surgery, and need to follow a strict diet temporarily. In this case, you could receive recommendations that only apply to certain recovery days, and not have to worry about learning what you can and cannot eat. This information would be presented by a non-intrusive, contextually relevant device that helps you confidently make the right decisions for yourself. Maybe you just want to try out a diet to see if you notice any benefits as a result, this type of technology could make it incredibly easy to do so without needing to study what you should or should not eat.

I recently had a close friend, and a family member receive strict dietary restrictions from a doctor and watched them struggle to navigate these new selections they needed to make immediately, and often. I understand how much something like this could help, and it's what makes me so drawn to the capabilities of Humane's device. This is only a small example of how it could help improve your health on a daily basis.

The Concept

There's definitely a need for this type of information to be surfaced in a more easily accessible way, and I started thinking about different ways this could be applied to every day selections. I tend to visit restaurants very frequently, so the first idea that came to mind for me was to apply this to menus in restaurants.

The debate over digital menus accessed on a phone via QR code and physical menus is ongoing, and though I personally prefer a physical menu, I decided to prototype both methods to showcase how the same information could be applied in two different ways. One where the device projects the menu in its entirety on a table, which could replace a digital menu, and the other where information is projected in an overlay on top of a physical menu.

Coincidentally, this concept ties in to a feature Imran Chaudhri recently demoed during his TED talk. In his example, a similar use case of helping a user identify safe foods is shown off, but the device visually identifies the food to let him know if it's safe to eat. Similar idea, just different way of applying it. (This was very validating to see, as I started working on this concept prior to the announcement.)

Designing & Prototyping

I go into detail on how I create these prototypes in my first blog post here. To summarize, I created these concepts using a small, battery-powered projector, which I connect to my computer and design for using Principle and Figma. When I get something I'm happy with, I take it out into the real world and start recording videos.

For this concept, I had to design a physical menu that I could use as a backdrop for one part of the projection. The idea is that your personalized information would be overlaid on top of a paper menu in a restaurant. I imagine the device could be intelligent enough to recognize a menu being held up in front of it, and could display these recommendations in situ.

The physical menu is based on the selections available at my favourite café in Toronto. Here you can see the separate layer of projected information, which I designed in tandem with the paper menu.

A couple variations of the digital menu launch screen shown here. I originally tried having the menu skewed to mimic a more real life version of a paper menu, before settling on a more straight forward presentation with content peeking out when a user hovers over it.

The Paper Menu Overlay

In the first part of this concept, I imagine Humane's device could intelligently scan a menu you're viewing, and overlay information on top of it to guide you to the right decisions.

In this specific illustration, I imagined the menu was being viewed by someone who didn't have strict dietary restrictions, but could still benefit from having guidance when choosing menu items. The device projects a star icon and "Popular" label to show what others have ordered at this restaurant, and a heart and "Recommended For You" label for items that this person may enjoy based on a previous understanding of what they like and dislike.

This overlay could be triggered with minimal interaction from the user. Perhaps a voice prompt of "show me what I'd like" would be enough to get the device to display the projection.

You can see how I imagine this would work in the video below:

The Digital Menu Replacement

As an alternative option, many restaurants have recently replaced paper menus with digital ones accessed via QR codes. While I personally prefer a physical menu, I understand there are benefits to digital ones. My feelings aside, I wanted to imagine how Humane's device could replace QR code menus on a phone by displaying the entire menu on the table in front of you.

Since you're likely to have a large surface to project on in front of you at a restaurant, I decided to display the menu and selections on the table rather than your hand for this concept. I created some interactions that were triggered on hover, such as the buttons growing and shrinking, and the menu peeking open before it's opened, to make the interface feel more dynamic.

I wanted to include the option to dismiss or open the menu initially, in case you're not interested in viewing it at all and you want it to be gone completely. I imagine the menu opening interaction would be optional if you were to summon this menu via a voice command, which in itself would signal that you want to view a menu and would open to the first page.

The benefit of having a menu displayed by a device that knows you and your preferences, is that it's able to make selections for you right off the bat without you needing to view the full menu. This is shown in the "For You" tab, where in this case, the user is shown a couple options that have been dynamically pulled out of the full menu based on what they might like. I imagine most users could make a selection from this page and be satisfied, but in case they're curious, or they're discussing options with the other members of their group, I included the rest of the menu in other tabs at the bottom.

I also wanted the initial state of the full menu to be options shown as the restaurant intended, but with the option to see your own preferences filtered to the top on each page. This is shown on the "pastries" page, where in this example the user is a vegan, and is shown options they would enjoy once they tap "Filter for me."

Below I've shown the concept both displayed on a table as it would be seen in reality, as well as my full-size Principle prototype I used to film the demo:

The recording from Principle that I used to create the video of the menu being displayed on the table.

The more realistic version, showing how a user could interact with the menu to view different options. Recorded in my favourite café in Toronto!

What I learned

It feels profound to think of the impact this specific concept could have on people in all types of situations, and that makes this concept feel like the most exciting one I've worked on so far. I'm constantly imagining more and more situations I could see people benefitting from this, and having discussions with friends and family members about how it could help in ways I hadn't even imagined.

For many, including myself, making the right choices around food is really tough to do, and it's something everyone is forced to do multiple times a day. To be able to access something that can help you without you needing to think about it, and give you peace of mind and confidence that you made the right decision, would be a game changer.

I'm excited that this was one of the demos shown off at TED, and I fully understand why. Beyond food, imagine the benefits this could have on health care in general. Imagine the device being able to keep track of your medications and dosages, reminding you and giving urgent notifications when it's time for a dose, or helping you count out the number of pills needed at any given time. Maybe I'll explore that in a future concept…

The ideas are coming to me faster than I can visualize them. Stay tuned for more!

-Michael