Google has introduced multimodal capabilities to its AI Mode, incorporating Lens visual search functionalities. This enhancement allows users to capture or upload images, pose questions regarding those images, and receive comprehensive, context-rich answers along with useful links for further exploration. Notably, this AI Mode is now accessible to millions of free Google Labs users in the United States. Previously, the experimental AI Mode feature was limited to Google One AI Premium subscribers.

However, it has now expanded to reach a much larger audience in the U.S. With the addition of multimodal capabilities through Google Lens, users can expect a more interactive search experience. This upgrade facilitates the ability to analyze images, extracting detailed information that integrates both visual and contextual elements. The multimodal search features are powered by Google Lens and a tailored version of Gemini. These technologies enable the AI Mode to grasp the entirety of a scene in a photo, recognizing the relationships between various elements—such as materials, colors, shapes, and arrangements.

Google’s advanced visual search skills allow the Lens to accurately identify each object present in the image. The updated AI Mode functionalities extend beyond basic visual search. They analyze not only the image itself but also the contextual relationships among objects within it. By employing a method called “query fan-out,” the AI Mode can perform multiple inquiries, providing a deeper understanding of the visual content.

One practical example showcased how the AI Mode could identify titles of books on a shelf from a photo. It then produced relevant queries to offer insights about these titles, including high-rated suggestions for additional reading. Furthermore, users can refine their searches with follow-up questions, enhancing their experience even further. Google’s AI Mode presents a competitive alternative to platforms like Perplexity and ChatGPT Search that offer chatbot-like interactions.

The company aims to refine the search experience and expand AI Mode features in the coming months.