Wednesday, May 18, 2011

Image-Search-by-Image: Search that Uses a Picture as the Criterion

I wonder how far away we are from an image search that would allow users to search for a picture with a picture? I've been thinking about this one for some time now, and it seems like a good leap for Google, Bing, or maybe even a new innovator in the search arena. Maybe that would be their entrance into the search arena? In terms of search innovation, it doesn't seem like there has been much in the past couple years. Unless you equate search dominance with innovation. That's a joke, of course.

In case you're having trouble following, presently image search functionality like Google Images requires users to enter a string of keywords to come up with a resultant group of images. This is generally fine, but in the event that you have a picture but don't know what it is, image-search-by-image could help you to discover just what that thing is.

Here's an example. Let's say you're going through your grandparents things prior to an estate sale and you find some really cool item that might appear on Antiques Roadshow. You try to do a search for it through a regular search engine, but a search string like "old red dish" doesn't provide much in the way of useful results. On the other hand, image-search-by-image would allow you to take a photo of the item, and then use some or all of that image as the search criterion.

A system like this would need three parts, though. The first, and most important, is a database of images that are cached based on the subtleties of visual appearance and not just textual descriptors. This would require a giant network of computers and a caching algorithm that could describe what is unique about each image by storing just enough information: not too much or not too little. Just as Google and Bing continue to crawl the web and recache everything they find for weighted usage in their textual search databases, an image search database would have the same general requirements.

The second part is a utility that users would use to select the image they want to use as their search criterion. I've given a lot of thought to this one, and I like the approach of a toolbar in your web browser that you can use to click-and-drag to select a portion of the screen. This would make it easy to select just about anything as your input, but also help you to hone the selection to just the thing you are searching for. In the example of the "old red dish" a user would bring this picture up in their web browser (from somewhere on their hard drive), and then select all of the dish but as little of the background as possible. Even better, maybe they could take a quick picture with their iPad 2 and then use their fingers to hone the selection to only what is relevant.

The third part happens after the user clicks search. The selected image is sent off to Google, Bing, or whomever, and it is quickly characterized in the same way that the entire database of images was characterized to produce a database of searchable images. Now that this has happened, the image can be used as the "search string" and located in the massive image database.

I'm not sure that it is much more complicated than this. Image-search-by-image would certainly help people find out what some mysterious things are by leveraging the fact that someone, somewhere in the world who has a web page already knows. Coin collectors, stamp collectors, anything pop-culture new or old could benefit from this type of search. I'm pretty sure we have the technology for this, and I'm also pretty sure that someone, somewhere, is working on this right now. Who knows, maybe Google bought the start-up yesterday.

No comments:

Post a Comment