Breakthroughs are being made in the field of image processing every day. One such breakthrough is PhotoSketch.
The image above on the right hand side is a rough freehand sketch. Those are simple hand drawn figures annotated with texts which describe what they are. To the left and right of the sketch, two sailboats are drawn. At the top left, the scene is described as a sunset on a beach. Seagulls are also represented to be flying in the air. And at the center of the image, a rough trapezium is drawn of which the annotated text read ‘wedding kiss.’
PhotoSketch analyzes this sketch, and cross-references the text and images with the vast number of images (and their descriptions) online. Because of the sheer number of images available online, many of which do not have correct descriptions (for instance, the image of a dog could have “Meet Tom” as its description) might result in unwanted results. Working around this cross-referencing problem is where the uniqueness of PhotoSketch comes into play.
On their website, PhotoSketch’s team states:
“Although online image search generates many inappropriate results, our system is able to automatically select suitable photographs to generate a high quality composition, using a filtering scheme to exclude undesirable images. We also provide a novel image blending algorithm to allow seamless image composition. Each blending result is given a numeric score, allowing us to find an optimal combination of discovered images. Experimental results show the method is very successful; we also evaluate our system using the results from two user studies.”
When PhotoSketch runs our original sketch (the one pasted above) it comes up with a number of images which match the descriptions provided on the sketch:
The PhotoSketch program, as if having a mind of its own, combines these images in the best possible way to generate the optimal results which most closely resemble our sketch. But even then, more than one result is possible:
Both the images above give us the real-life replica of our initial freehand sketch. But what the human eye misses PhotoSketch catches; therefore the images are refined further and further until the best results are achieved:
The research paper which lead to PhotoSketch can be read here in PDF file format.
Here is how the cross-reference results of other sketches were refined:
PhotoSketch is truly an amazing breakthrough in image processing. Once the technology is perfected and available for public use, the possibilities would be endless.
