Overview
Recent reports from AssembleDebug and Android Authority indicate that Google is testing a “screen automation” feature for its Gemini AI. The beta version of the Google app reportedly includes prompts such as “place orders and book rides,” suggesting the assistant could interact directly with other apps on the device.
How Screen Automation Might Work
According to leaked code strings, Gemini would receive a user prompt, interpret the desired action, and then simulate touches or inputs within supported apps. The automation is limited to “certain apps,” and the system appears to capture screenshots during the process for quality‑control review.
Potential Use Cases
If the feature reaches production, typical scenarios could include:
- Ordering food or groceries through delivery apps (e.g., Uber Eats, DoorDash)
- Booking rides with Uber or Lyft
- Purchasing tickets or making reservations without manual navigation
Security and Privacy Concerns
Google’s own warning advises users not to enter sensitive information while Gemini is active, because screenshots taken during automation are sent to “trained reviewers” when “Keep Activity” is enabled. This raises questions about data handling, consent, and the potential exposure of personal details.
Relation to Project Astra
The leak also links the screen‑automation effort to Google’s Project Astra, a research initiative that was merged into the Gemini team earlier this year. Project Astra focuses on AI‑driven task execution, indicating that the new feature may be a broader step toward fully autonomous assistants.
Conclusion
While the rumors are promising for convenience, users should stay informed about the capabilities and limitations of Gemini’s automation, especially regarding privacy safeguards. As Google refines the feature, clearer guidelines and opt‑in controls will be essential.