Posted by Matthew McCullough, VP of Product Administration, Android Improvement
Person expectations for AI on their units are essentially shifting how they work together with their apps. As an alternative of opening apps to do duties step-by-step, they’re asking AI to do the heavy lifting for them. On this new interplay mannequin, success is shifting from getting customers to open your app, to efficiently fulfilling their duties and serving to them get extra finished quicker.
That will help you evolve your apps for this agentic future, we’re introducing early stage developer capabilities that bridge the hole between your apps and agentic apps and customized assistants, resembling Google Gemini. Whereas we’re within the early, beta levels of this journey, we’re designing these options with privateness and safety at their core as our first step in exploring this paradigm shift as an app ecosystem.
Empowering apps with AppFunctions
Android AppFunctions permits apps to show information and performance on to AI brokers and assistants. With the AppFunctions Jetpack library and platform APIs, builders can create self-describing capabilities that agentic apps can uncover and execute by way of pure language. Mirroring how backend capabilities are declared by way of MCP cloud servers, AppFunctions supplies an on-device resolution for Android apps. Very like WebMCP, it executes these capabilities domestically on the gadget somewhat than on a server.
The Samsung Gallery integration with Gemini on the Galaxy S26 sequence showcases AppFunctions in motion. As an alternative of manually scrolling by picture albums, now you can merely ask Gemini to “Present me footage of my cat from Samsung Gallery.” Gemini takes the consumer question, intelligently identifies and triggers the appropriate perform, and presents the returned pictures from Samsung Gallery instantly within the Gemini app, so customers by no means want to go away. This expertise is multimodal and might be finished by way of voice or textual content. Customers may even use the returned pictures in follow-up conversations, like sending them to mates in a textual content message.
Enabling agentic apps with clever UI automation
Whereas AppFunctions supplies a structured framework and extra management for apps to speak with AI brokers and assistants, we all know that not each interplay has a devoted integration but. We’re additionally creating a UI automation framework for AI brokers and assistants to intelligently execute generic duties on customers’ put in apps, with consumer transparency and management inbuilt. That is the platform doing the heavy lifting, so builders can get agentic attain with zero code. It’s a low-effort strategy to lengthen their attain with out a main engineering carry proper now.
To get suggestions as we refine this framework, we’re beginning with an early preview on the Galaxy S26 sequence and choose Pixel 10 units, the place customers will have the ability to delegate multi-step duties to Gemini with only a lengthy press of the facility button. Launching as a beta characteristic within the Gemini app, this may assist a curated collection of apps within the meals supply, grocery, and rideshare classes within the US and Korea to start out. Whether or not customers want to put a fancy pizza order for his or her members of the family with specific tastes, coordinate a multi-stop rideshare with co-workers, or reorder their final grocery buy, Gemini may help full duties utilizing the context already obtainable out of your apps, with none developer work wanted.
Customers are in management whereas a activity is being actioned within the background by UI automation. For any automation motion, customers have the choice to watch a activity’s progress by way of notifications or “reside view” and may change to handbook management at any level to take over the expertise. Gemini can also be designed to alert customers earlier than finishing delicate duties, resembling making a purchase order.
Trying forward
In Android 17, we’re seeking to broaden these capabilities to achieve much more customers, builders, and gadget producers.
We’re presently constructing experiences with a small set of app builders, specializing in high-quality consumer experiences because the ecosystem evolves. We plan to share extra particulars later this 12 months on how you need to use AppFunctions and UI automation to allow agentic integrations on your app. Keep tuned for updates.