Google is upgrading its visible search app, Lens, with the power to reply near-real-time questions on your environment.
English-speaking Android and iOS customers with the Google app put in can now begin capturing a video through Lens and ask questions on objects of curiosity within the video.
Lou Wang, director of product administration for Lens, mentioned the characteristic makes use of a “customized” Gemini mannequin to make sense of the video and pertinent questions. Gemini is Google’s household of AI fashions and powers a variety of merchandise throughout the corporate’s portfolio.
“Let’s say you want to learn more about some interesting fish,” Wang mentioned in a press briefing. “[Lens will] produce an overview that explains why they’re swimming in a circle, along with more resources and helpful information.”
To entry Lens’ new video evaluation characteristic, you should join Google’s Search Labs program, in addition to decide in to the “AI Overviews and more” experimental options in Labs. Within the Google app, holding your smartphone’s shutter button prompts Lens’ video-capturing mode.
Ask a query whereas recording a video, and Lens will hyperlink out to a solution equipped by AI Overviews, the characteristic in Google Search that makes use of AI to summarize info from across the internet.
In keeping with Wang, Lens makes use of AI to find out which frames in a video are most “interesting” and salient — and above all, related to the query being requested — and makes use of these to “ground” the reply from AI Overviews.
“All this comes from an observation of how people are trying to use things like Lens right now,” Wang mentioned. “If you lower the barrier of asking these questions and helping people satisfy their curiosity, people are going to pick this up pretty naturally.”
The launch of video for Lens comes on the heels of a related characteristic Meta previewed final month for its AR glasses, Ray-Ban Meta. Meta plans to carry real-time AI video capabilities to the glasses, letting wearers ask questions on what’s round them (e.g., “What type of flower is this?”).
OpenAI has additionally teased a characteristic that lets its Superior Voice Mode software perceive movies. Ultimately, Superior Voice Mode — a premium ChatGPT characteristic — will be capable of analyze movies in actual time and take context under consideration because it solutions you.
Google has crushed each firms to the punch, it appears — minus the truth that Lens is asynchronous (you’ll be able to’t chat with it in actual time), and assuming that the video characteristic works as marketed. We weren’t proven a stay demo in the course of the press briefing, and Google has a historical past of overpromising with regards to its AI’s capabilities.
Other than video evaluation, Lens also can now search with photographs and textual content in a single go. English-speaking customers, together with these not enrolled in Labs, can launch the Google app and maintain the shutter button to take a photograph, then ask a query by talking out loud.
Lastly, Lens is getting new e-commerce-specific performance.
Beginning in the present day, when Lens on Android or iOS acknowledges a product, it’ll show details about it, together with the value and offers, model, evaluations, and inventory. Product ID works on uploaded and newly snapped pictures (however not movies), and it’s restricted to pick international locations and sure procuring classes, together with electronics, toys, and sweetness, for now.
“Let’s say you saw a backpack, and you like it,” Wang mentioned. “You can use Lens to identify that product and you’ll be able to instantly see details you might be wondering about.”
There’s an promoting element to this, too. The outcomes web page for Lens-identified merchandise can even present “relevant” procuring advertisements with choices and costs, Google says.
Why stick advertisements in Lens? As a result of roughly 4 billion Lens searches every month are associated to procuring, per Google. For a tech big whose lifeblood is promoting, it’s just too profitable a chance to move up.
