As promised, Gemini AI is taking its first steps into the Google Home app, with Nest camera feeds and home automation among the first beneficiaries.

Starting this week, Gemini will be able to write descriptions of what’s happening in captured Nest video events. And soon, the LLM will start helping you create Google Home automations based on natural language queries.

Google is taking a cautious approach to adding AI features to the smart home, adding just a few convenience features for its Nest cameras and automation recipes before diving into more in-depth improvements, such as giving Google Assistant a more “fluent” conversation style. .

This news is part of TechHive’s in-depth coverage of the best security cameras.

The new Gemini features follow a extension for Gemini app which allows Google’s AI model to control your smart lights, thermostats, and other smart home gadgets.

For now, you will need to be a member of Google Public Preview Program to try out the new Gemini-based features, and you’ll also need to enable the “Experimental AI Features” setting.

Nest AI-powered video descriptions

Until now, video events in your Nest camera feeds were limited to generic text descriptions like “animal or activity detected.” However, with the help of Gemini, the Google Home app can now generate more specific labels based on what actually happens in the clip.

For example, the description generated by Gemini might say “the dog is digging in the garden” or “a pack of balloons.” According to Google, tapping the information icon will give you a more detailed description.

Google

In addition to just browsing your Gemini-tagged video streams, you’ll also be able to search on these descriptions.

For example, if you enter “Did I receive any packages today?” » In the search box, you might get a result generated by Gemini saying: “Someone left a package and balloons this morning, and 2 people with a small dog dropped off 3 packages a little later.”

Better yet, the description could include the name of a person detected in your “familiar faces” library if they are the only ones participating in the video event. If your “familiar” friend is in a group, their name won’t appear in the description, but you can still search for their name, Google says.

The new Gemini-based video descriptions still have some limitations, starting with the fact that they will only work on Nest cameras managed in the Google Home app. In other words, don’t expect AI descriptions with existing cameras that are still stuck in the old Nest app.

Additionally, the new descriptions are only available in English, and only in the United States at the moment.

Finally, it will be interesting to see if the descriptions generated by Gemini fall victim to the hallucinations that plague most AI models.

“Help me create” automations

Coming soon, for those participating in Google’s Public Preview program, another interesting feature based on Gemini: the ability to create home automation based on natural language queries.

For example, you can say “Lock the doors and turn off all the lights at bedtime” and Gemini will instantly create a Google home automation for you, which you can then view or modify as needed.

Google

If you’re not sure what kind of automation you want, you can just give Google Home the basics (like “Help kids remember to put their bikes away when they get home from school”) and Gemini will offer you some suggested triggers. and actions.

Such “help me create” automations will begin rolling out in public preview “later this year,” Google says.

Coming soon: a better Google Assistant

Perhaps the most dramatic new Gemini feature planned for Google Home is an improved Google Assistant for Nest speakers and displays, with a more “smooth” conversation style.

With Gemini’s help, the revamped Google Assistant on Nest speakers will sound “more natural” and “understand you better,” allowing for more “back and forth” as well as follow-up questions.

This sounds enticing, but it’s worth noting that Google hasn’t listed the revamped Google Assistant in its list of Google Home Improvements Coming Fall 2024which suggests we’ll have to wait longer for this feature to arrive. (AI video descriptions and “help me create” automations did make the cut.)