Google is expanding the scope of Google Lens, which can serve up information about what’s in a photo, to the Google iOS app and Google Chrome on the desktop.
“To make it easier to shop at the moment of inspiration, we’re making the products you see online instantly shoppable with Google Lens,” Matt Madrigal, VP and general manager of merchant shopping at Google, said at today’s Google SearchOn event.
“Say you’re redecorating and are browsing Apartment Therapy for ideas. Soon, you’ll see a new button in the Google app on your iPhone to make all images on the page instantly searchable,” Madrigal said. The same functionality will arrive on Chrome “in the coming months,” he said.
In a graphic shared on Twitter, we see a right-click on a web page bring up a drop-down menu with the option to Search Part of the Page with Google Lens. This allows the user to drag a selection box over a portion of an image on the web page. Google then provides search results in a sidebar that include visually similar content.
The functionality is powered by a new machine learning model that uses on-device processing to recognize products in as little as 200 milliseconds, according to Madrigal.
In a blog post, Google President of Commerce, Payments, & NBU Bill Ready explained that in Chrome, you’ll soon “be able to select images, video, and text content on a website with Lens to quickly see search results in the same tab—without leaving the page you’re on.”
(The Lens portion starts at the 24-minute mark in the video above.)
In a separate tweet, Google shared a look at its Multitask Unified Model (MUM), a special AI model. It will allow users to combine an image search with a voice command—ask “how to fix” alongside a picture of a bike derailleur, for example—streamlining the process of identifying the bike part and finding resources explaining how to fix it.
Google Lens is currently available as a standalone app for certain Android phones, and is built into Google Photos for iOS and Android. It’s also in the search bar of the Google app, if you want to snap a photo of something and have Lens identify it.
But for those browsing or shopping on the desktop, Google Lens will be a solid addition to Google Chrome. The ability to highlight a portion of an image will simplify the process of performing a reverse image search and could provide a way to quickly search images in videos. We’ll have to wait and see how many of the latest upgrades to Google Lens make their way to the browser version. Be sure to keep Chrome updated to see them when they roll out.