Google is introducing a new AI-driven feature in its Lookout app – Assisted Vision. Visually impaired people can thus discuss with the AI to ask it, through several questions, all the details of an image.
Describing an image is no longer a very complex challenge for a well-honed artificial intelligence. However, Google has managed to take the concept one step further to make it more practical and impressive. The American giant has just unveiled the “Image Q&A” function in its accessibility application Lookout – Assisted Vision and we were able to have a brief demonstration of it during Google I/O 2023 a few days before.
The concept is easily summarized: you ask any question to understand what is in the image and the artificial intelligence provides you with a clear answer every time. The purpose of this feature — and of Lookout in general — is to help visually impaired people know what’s in a photo or drawing. Hence the notion of Q&A which means questions and answers. The demo, she had a little something quite stunning.
Basically, you upload an image to the app and ask the AI to describe it. The latter will execute and explain to you what we see. This first description is effective and might suffice if you just wanted a quick summary. However, for those who need to go into more detail, just ask a question, in writing or out loud.
AI answers your questions to describe the image
During the demo at I/O, Lookout’s Image Q&A option was confronted with an image and here’s the conversation it then had with the human user.
AI: In this picture I can see a dog running on the beach. I can also see that he has a bullet in his mouth. In the background I can see water, mountains and the sky.
Human : What breed is the dog?
AI: The dog is a crossbreed.
Human : Does the dog look playful?
AI: Yes, the dog is playful.
Human : Describe the ocean.
AI: The ocean is calm and there is a mountain in the background.
Human : Are the waves big?
AI: No, the waves are small.
Human : Are there clouds in the sky?
AI: Yes, there is a cloud in the sky.
Human : Is the beach crowded?
AI: No, the beach is empty.
The conversation is not necessarily the most natural, but it has enough fluidity and clarity to represent an interesting solution for visually impaired people who, on the web, often have to content themselves with very brief descriptions.
There, they have a tool with a great potential to give them a precise idea of an image.
We filmed the demonstration in question. Let’s just specify that this recording is not of high quality, it was mainly intended as a support for the writing of this article.
YouTube linkSubscribe to Frandroid
Finally, remember that Lookout is an application available for a while on the Play Store. This accessibility tool already had several interesting features for the visually impaired: reading business and loyalty cards or labels, detecting the value of a banknote and helping to scan a document.
To follow us, we invite you to download our Android and iOS application. You can read our articles, files, and watch our latest YouTube videos.