Jarvis AI assistant: Mark Zuckerberg’s Tony Stark moment

Taking a leaf out of Tony Stark’s (played by Robert Downey Jr in Iron Man) book, Mark Zuckerberg has developed a virtual assistant for his home. It is called Jarvis, and it is able to respond to text and voice commands which could involve asking it to play music, turn on the air conditioning, open doors and even recognize visitors. Jarvis is able to understand context as well, for better response to requests such as personal tastes in music.

Incidentally, Jarvis speaks in actor Morgan Freeman’s voice.

First things first, Zuckerberg had to ensure that all the existing gadgets are connected to the Internet. This included a Crestron system for lights, thermostat and doors, a Sonos system with Spotify for music, a Samsung Smart TV and a Nest camera. Since all these gadgets work on different protocols, it was important to make them talk to each other, while also be connected to the internet—there are many appliances that are not connected to the internet, which was also a challenge.

The next step was to write a code which would understand keywords and subsequently the spoken word. In this case, there needed to be variations in the task to be done, depending on who was saying the command, and that contextual understanding was important in a home environment. “The more context an AI has, the better it can handle open-ended requests. At this point, I mostly just ask Jarvis to ’play me some music’ and by looking at my past listening patterns, it mostly nails something I’d want to hear. If it gets the mood wrong, I can just tell it, for example, ‘that’s not light, play something light’, and it can both learn the classification for that song and adjust immediately," said Zuckerberg, in a blog post.

Then there is the matter of face recognition. While selecting between two objects is simpler, identifying two potentially similar faces may be a challenge. But Facebook is already quite good at identifying your friends, and that helps with Jarvis as well.

It isn’t a surprise that Zuckerberg is using the Messenger app to communicate. So, there is a Messenger Bot which allows Zuckerberg to communicate and command Jarvis from anywhere, via text.

This is just the start, because AI still needs a lot of work. At the same time, Zuckerberg predicts that in about 5-10 years, AI systems will be more accurate than people for each of the senses—vision, hearing and touch, as well as language.