Google Gemini Live Can Now Record Screen to Answer Questions in Real-Time

Google's AI assistant Gemini can now record a smartphone’s screen and answer real-time contextual questions. Announced as part of the demonstration of Google’s Project Astra last year, the new AI features make Gemini Live aware of what is on the screen, such as apps, the content inside them, and even sensitive data such as photos stored on the phone.

The feature was spotted by a user who posted a screen recording on Reddit, showing how it works on their Xiaomi phone, according to 9to5Google. When triggered, Gemini Live offers an option to record the screen, letting users ask any questions related to everything the AI assistant can read. The Verge has cited Google spokesperson Alex Joseph to confirm the rollout of Gemini Live’s screen-reading capabilities. However, he did not say whether it is an incremental rollout or if the access is currently with a handful of eligible smartphone users.

Gemini Live’s screen awareness is one of the two features Google announced at the beginning of March that would be available to the Google One AI Premium plan later in the month. It appears to be rolling out to only Gemini Advanced Subscribers currently, according to the announcement. The other feature is called Live Video, allowing users to record a video and ask Gemini Live questions related to it in real-time. In a demonstration earlier this month, Google showed a user taking Gemini Live’s help to decide on a paint colour for a piece of pottery shown in the video. This feature is yet to become available.

Google’s timely rollout of AI features is crucial in its fight against increasing competition. Although Apple is late to the party, OpenAI, Microsoft, and now Amazon are doubling down on their efforts to increase the video-centric capabilities of their AI assistants. Even Siri — Gemini Live’s biggest rival — is expected to have a similar feature, but it is indefinitely delayed, and even though Apple is putting out all the stops to deliver it as quickly as possible, it is still months, if not weeks, away. Gemini’s ability to read phone screens and live video feeds could help increase its appeal, especially on Android phones.

Read more: Realme’s New P3 Is a Waterproof Phone At Under ₹17,000