Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
157971 stories
·
33 followers

Building out a course on agent skills

1 Share
I'm trying to build out a course on agent skills. For a while I've been looking for something more substantial and stable to sink my teeth into (similar to what I did with API documentation), and I think agent skills just might be it.
Read the whole story
alvinashcraft
48 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Introducing Gemini 3.6 Flash, 3.5 Flash-Lite, and 3.5 Flash Cyber

1 Share
We’re introducing new Gemini models, including Gemini 3.6 Flash, 3.5 Flash-Lite and 3.5 Flash Cyber.
Read the whole story
alvinashcraft
50 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Sizing Copilot Credits for Cowork? Let Your Users Use Cowork Investment Advisor Agent.

1 Share

Right, let's talk about a headache. If you're helping a customer allocate Copilot Credits for M365 Copilot Cowork, you have to size it all up first — and doing that by hand is a fiddly, time-consuming job. Here's the problem, and a much easier way to solve it. 

Start with the Cowork Estimator 

Here's the good news: you don't have to start from scratch. Microsoft has published the Customer Cowork Estimator — a handy tool that turns personas, prompt complexity and expected usage into an estimated credit number. It's the perfect place to begin, and it does the core maths for you. 

To get the most out of it, you just need to feed it good inputs — and that's where a little groundwork comes in. For each part of the business, you'll want to: 

  • Work out who's who. There are four types of user — corporate knowledge workers, management and senior leaders, customer-facing folks, and technical staff. 
  • Count how many of each. Quick for a small team, a bit more involved for a big one. 
  • Pin down what they'd genuinely use Cowork for. The real multi-step workflows that hop across apps and actually do things — not just a quick chat or a summary. 
  • Judge how heavy each workflow is. Light, Medium or Heavy — since each level uses a different number of credits per run. 
  • Estimate how often it runs. Daily adds up to a lot of runs a month; weekly is far fewer. 

The estimator handles the sums beautifully once those inputs are in. Gathering the inputs themselves — persona by persona, workflow by workflow, across a few thousand people — is simply the part that takes time. And that's exactly where this agent lends a hand: it builds on the estimator by automating the groundwork that feeds it. 

How this agent helps 

 

TheCoworkInvestment Advisor agent — one place for users to size their ownCoworkneeds, with built-in starter prompts.

 This agent takes on that groundwork for you. Instead of sizing everyone from the outside, the admin simply switches the agent on for every user. Each person then sizes their own needs — and the agent does the clever bits for them, ready to drop into the estimator. Here's what it does behind the scenes: 

  • Spots the right persona. It works out which of the four personas each user fits. 
  • Finds the real workflows. It looks at the top Cowork scenarios the user would actually run. 
  • Keeps it honest. It checks each one is a genuine Cowork job — several steps, more than one app, real actions and a bit of decision-making — not something a Scheduled Prompt or plain Copilot Chat could do just as well. 
  • Grounds it in evidence. It reads the signals from recent work — emails, meetings, documents and Teams chats — so the estimate is based on what people actually do, not thin air. 
  • Does the sums. It maps each workflow's complexity to credits per run and totals it up — giving you numbers that line up neatly with the estimator. 
  • Shows the value. It gives a view of pay-as-you-go versus a pre-purchase plan, and a sense of the return on the spend. 
  • Stays transparent. It states its assumptions, flags how confident it is, and sticks to permitted data and the usual privacy and compliance rules. 

And here's the kind of report it hands back — persona, the top workflows, complexity, credits, cost and a clear recommendation, all in one place: 

 

TheCoworkInvestment Advisor agent — one place for users to size their ownCoworkneeds, with built-in starter prompts.

Why this approach works better 

So why hand it to the users? A few good reasons: 

  • More accurate. The numbers come from each person's real workload, not a top-down guess. 
  • Consistent by design. Everyone follows the same method — same personas, same complexity bands, same maths — so the results line up and roll into one clean figure. 
  • No over-buying. Because it weeds out the workflows that don't really need Cowork, your customer only pays for credits they'll genuinely use. 
  • A proper business case. You get cost clarity and a feel for the return, so you're handing over more than just a number. 
  • It scales. Ten users or ten thousand — the effort on your side stays much the same. 
  • It saves you hours. You swap manual sizing for gathering and rolling up. Your time goes on advising, not tallying. 

Not ready to roll it out? Other ways in 

If an admin isn't quite ready to switch the agent on for everyone, that's OK— there's no need to. This agent is an M365 Copilot agent, and it has free access to Work IQ — the same engine that will eventually power Cowork. Because that access is free, the agent can read how people really work and take the guesswork right out of sizing. And there's more than one way to get at it: 

  1. Build it with Agent Builder. Ready-made instructions, a description and a starter prompt are all sitting in the GitHub repo. Anyone with access to M365 Copilot can pop them into Agent Builder and stand the agent up in minutes. 
  1. Prefer not to build an agent? Just use the prompt. There's a comprehensive prompt you can drop straight into the M365 Copilot Chat experience. Same sizing, same evidence — no agent to create. 

Ready to give it a go? 

Whichever route suits you best, getting started is dead simple. If you're rolling the agent out to your users: 

  1. Have a quick word with your customer's admin about switching the agent on for their users. 
  1. Let each person size their own Cowork needs. 
  1. Gather it all up and roll it into a single estimate. 

Prefer to keep it in your own hands? Build the agent from the GitHub repo with Agent Builder, or drop the comprehensive prompt into M365 Copilot Chat — you'll get to the same place with even less setup. 

 

THE BOTTOM LINE 

That's the heavy lifting done for you. You'll save yourself hours, spare yourself the guesswork, and hand your customer a credit plan they can genuinely trust — with the numbers and the business case sitting right behind it. Give it a spin on your next engagement and see how much quicker it gets you there. 

Resources 

Everything you need is in one place: 

Read the whole story
alvinashcraft
52 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Build intelligent Android apps: Integrate into Android's intelligence system using AppFunctions

1 Share

Posted by Ben Weiss, Senior Developer Relations Engineer, Android Developer Relations



Welcome back to the blog post series "Build intelligent Android apps" where we take a basic Android app and transform it into a personalized, intelligent, and agentic experience. In our previous post, we explored how to leverage Firebase AI Logic to build cloud-hosted and hybrid AI features.

Traditional mobile UIs excel at focused, hands-on tasks, and the Android intelligence system is introducing complementary features to make complex, multi-step actions even easier. By supplementing traditional user interfaces, AppFunctions provide a powerful new entry point: A privileged agent on the device can access app features in the background. This can be particularly helpful when users are driving, walking or otherwise multitasking.

In this article, we'll show you how we designed and integrated these capabilities into our travel planning app, JetPacker, using Android AppFunctions. We'll explore the rationale behind our feature choices, discuss the specialized tooling we used to accelerate development, and dive into the code that makes it all work.

Designing AI-ready features: making choices that matter for your users

To select which features to provide to the intelligence system, we looked for tasks where a voice or text command is objectively faster than tapping through screens. In this side-by-side screen recording you can see this contrast perfectly: on the left, a user tapping through multiple screens to log an expense; on the right, the same task completed instantly in the background via a privileged agent.



Our first choice was expense tracking. Logging a coffee expense during a trip usually takes quite a few taps—unlocking the phone, opening the app, finding the active trip, navigating to the expenses tab, tapping the add button, taking a picture of the receipt, and checking the result. By providing the addExpense and getExpenses features as AppFunctions, the system agent handles the heavy lifting. When the user says, "Add a five-dollar coffee expense to my Paris trip," the agent automatically searches for the correct trip ID in the background and inserts the expense, skipping the manual UI flow entirely.

We also prioritized itinerary management. Finding what activity is next on a busy trip itinerary usually requires scrolling through a dense timeline view. By providing getItinerary and addItineraryEvent to the system, the user can simply ask, "What am I doing next in Paris?" and get an immediate answer.



Finally, we focused on hands-free note capturing. Typing out reminders or notes while walking down a busy street is difficult and unsafe. Exposing a voice note capability allows the user to say, "The flight was amazing, I saw a beautiful sunset and managed to sleep well," and the privileged agent automatically transcribes and saves it directly into the travel database  using the addVoiceNote AppFunction.

Android MCP powered by AppFunctions

This entire experience is built on Android MCP. Under this design, the app acts as a local MCP server. Rather than remote APIs, you provide your app features directly to the on-device intelligence system.

Android AppFunctions is the API that brings this concept to life. It reads annotated Kotlin functions and compiles them into type-safe, sandboxed tool definitions that the privileged agent can discover and invoke locally on the device.







Diagram highlighting our apps, the android platform, and system agents coordinate AppFunctions.

Under the Android MCP model, your app acts as a local MCP server that exposes structured tools, while the Android platform serves as the central tool registry. On the MCP client side, agent apps are registered with the intelligence system after being granted system-privileged permissions to access the registry.

When a user interacts with a registered agent, its LLM determines if the request can be handled by an AppFunction, queries the platform's metadata, and executes the appropriate registered functions in the background. This local MCP client-server design gives you full control: you choose exactly which features are accessible to the agent, keeping the rest of your app's data private.

How we accelerated development with Android skills

To streamline the integration process, we leveraged the AppFunctions development skill. The AppFunctions development skill is a complete development companion. It guided us through the entire lifecycle: mapping Kotlin data classes to serialize parameters, generating the necessary Service entry points, refining our KDoc documentation to ensure the LLM understands parameter boundaries, and setting up automated testing using ADB.

Providing app features to the intelligence system

Enough with the theory, let's dive into the implementation.

Configuration and dependency setup

We begin by adding the AppFunctions dependencies. One for the API and one for the Kotlin Symbol Processing compiler.

implementation("androidx.appfunctions:appfunctions:1.0.0-alpha10")
ksp("androidx.appfunctions:appfunctions-compiler:1.0.0-alpha10")

Modeling custom data types

Any custom object exchanged with the agent must be annotated with @AppFunctionSerializable. In our TripSerializable.kt file, we define our trip data model:

@AppFunctionSerializable(isDescribedByKDoc = true)
data class TripSerializable(
    /** The trip's unique identifier. */
    val id: String,
    /** The trip's title. */
    val title: String,
    /** The trip's destination location. */
    val location: String,
    /** The trip's start date in milliseconds. */
    val startDate: Long,
    /** The trip's end date in milliseconds. */
    val endDate: Long,
    /** A list of participants. */
    val participants: List<String>,
)

Providing features using the @AppFunction annotation

Next, the skill wrote the Kotlin functions that perform the database queries and annotate them with @AppFunction. We can view this in searchTrip:

/**
 * Looks for trips based on optional filters like id, title (name), location, and dates.
 *
 * @param id The unique identifier of the trip.
 * @param title The title or name of the trip.
 * @param location The destination location.
 * @param startDate The minimum start date in milliseconds.
 * @param endDate The maximum end date in milliseconds.
 * @return A list of trips matching the filters.
 */
@AppFunction(isDescribedByKDoc = true)
suspend fun searchTrip(
    id: String? = null,
    title: String? = null,
    location: String? = null,
    startDate: Long? = null,
    endDate: Long? = null
): List<TripSerializable> {
    return withContext(Dispatchers.IO) {
    // implementation
}

Since AppFunctions run on the UI thread by default, we use withContext(Dispatchers.IO) to switch to a background dispatcher. Additionally, we refine our KDoc to use clear, imperative verbs and specify parameter constraints. This documentation compiles directly into the tool's schema, which the privileged agent uses to resolve parameters and handle runtime errors.

The service entry point and Hilt integration

To register these features with the intelligence system, we create an abstract base class that extends AppFunctionService. We annotate it with @AppFunctionServiceEntryPoint:

@RequiresApi(36)
@AndroidEntryPoint
@AppFunctionServiceEntryPoint(
    serviceName = "JetPackerAppFunctionService",
    appFunctionXmlFileName = "jetpacker_app_function_service"
)
abstract class BaseJetPackerAppFunctionService : AppFunctionService() {
    @Inject internal lateinit var tripDao: TripDao
    // DAOs and database references are injected here...
}

During compilation, KSP generates the final concrete service subclass, JetPackerAppFunctionService, as declared with the serviceName parameter. We also register app_metadata.xml in the app's manifest. This file provides global operational rules for JetPacker's declared AppFunctions.

Testing and verifying your AppFunctions

Once implemented, you should verify that your AppFunctions are registered and working correctly.

Running devices or emulators with Android 17 or newer, you can use ADB commands from your terminal to list and invoke your functions. Running adb shell cmd app_function list-app-functions displays all registered functions for your package. You can then execute a specific function and test its database integration by running adb shell cmd app_function execute-app-function while passing a raw JSON parameters string.

Instead of these ADB commands, you can also use the AppFunctions Testing Agent to inspect your configuration, list and execute AppFunctions, and even see how your AppFunctions behave in a real conversational flow.

Wrapping it up

When thinking about app features that can be contributed to the intelligence system using AppFunctions requires a slight shift in how we think about code and documentation. AppFunctions enable you to use this new interaction model for apps, which allows using an agent to access app features..

First, the AppFunctions development skill is an essential lifecycle tool, helping you discover features, implement and refine AppFunctions for your apps. Second, KDoc comments are a compiled API asset; clear parameter descriptions directly impact the execution accuracy of the system agent. Finally, Android MCP provides local-first execution allowing apps to safely collaborate with AI agents.

Contributing app features through AppFunctions makes your application ready for the intelligence system. Let us know how you are adapting your apps for the agentic era!

Learn more

Check out the other parts of this blog post series:
Part 1: Introduction of the app and a high-level overview.
Part 2: On-device intelligence. Deep-dive into ML Kit’s GenAI APIs and Gemini Nano to build privacy-first features like itinerary summarization, receipt parsing, and local audio processing.
Part 3: Hybrid and cloud reasoning. Explore how to use Firebase AI Logic to ground LLM answers in real-world data like Google Maps and web context.
Part 4 (this post!): System integration. Integrating with the Android intelligence system using AppFunctions.
Part 5 (coming soon): In-app agentic workflows. Extend the app with an end-to-end booking assistant powered by A2UI and ADK.

Interested in more on Android Development? Follow Android Developers on YouTube or LinkedIn!

All code snippets in this blog post follow the following copyright notice:

Copyright 2026 Google LLC.
SPDX-License-Identifier: Apache-2.0
Read the whole story
alvinashcraft
52 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Build intelligent Android apps: Cloud and hybrid inference

1 Share
Posted by Thomas Ezan, Jolanda Verhoef, Caren Chang, Senior Developer Relations Engineers, Android Developer Relations



Welcome back to the blog post series "Build intelligent Android apps" where we take a basic Android app and transform it into a personalized, intelligent, and agentic experience. In our previous post we explored how to build intelligent on-device features using Gemini Nano through ML Kit's Prompt API.

In this post, we will look at how you can leverage Firebase AI Logic to build cloud-hosted and hybrid AI features: 

  • Grounding answers in real-world context
  • Routing requests dynamically between cloud and local execution using hybrid inference
  • Translating content with custom routing systems


Sometimes a use case requires AI models with greater world knowledge, a much larger context window, or the ability to handle complex queries. In those scenarios, we can leverage cloud models. 

Other times, you want the best of both worlds: using hybrid inference to run on-device when available to lower costs, while falling back to the cloud to ensure compatibility for all devices.


Cloud and hybrid features in Jetpacker: Museum assistant with web grounding, hybrid restaurant review drafting, and  support chat featuring custom-routed live translation.

Let’s look at how we implemented three cloud and hybrid features in Jetpacker:

  • a museum assistant with web grounding
  • hybrid restaurant review drafting
  • hotel support chat featuring custom-routed live translation.

Use LLM grounding for up-to-date informationMuseum assistant chatbot with LLM grounding

The Museum assistant is an interactive chatbot designed to help users plan their museum visits. It provides visitors with up-to-date details regarding specific exhibits, current opening hours, ticket pricing, and more.


Museum assistant is a chatbot that answers questions, such as 
‘How can I get a ticket discount for Le Louvre?’

When building AI features, getting the model to answer with fresh, accurate, and specific real-world information is a common challenge. While cloud models possess massive amounts of world knowledge, they might not know about seasonal exhibits or the current day’s opening hours. 



Grounding data is added to the context window to enable the model
 to answer questions correctly and accurately.

To bridge this gap, we can use grounding techniques to add extra context to the model’s context window. The Firebase AI Logic SDK supports three types of grounding:

  • URL grounding: Grounding responses using content from a specific webpage (e.g. current ticket prices or museum rules).
  • Google Search grounding: Letting the model query the real-time Google search index for up-to-date details.
  • Maps grounding: Using Google Maps location data.

In Jetpacker, we dynamically construct the available tools based on enabled feature flags and initialize the generative model using the Firebase AI SDK:

// implementation("com.google.firebase:firebase-ai-logic")

private var toolList = mutableListOf<Tool>()

init {
    if (ENABLE_SEARCH_GROUNDING) {
        toolList.add(Tool.googleSearch())
    }
    if (ENABLE_URL_GROUNDING) {
        toolList.add(Tool.urlContext())
    }
}

private val generativeModel = Firebase.ai(backend = GenerativeBackend.googleAI())
    .generativeModel(
        modelName = "gemini-3-flash",
        systemInstruction = content {
            text("You are a helpful museum assistant answering questions about a museum. Use plain text.")
        },
        tools = toolList
    )

When the user queries the assistant, if URL grounding is enabled, we append the specific museum resource URLs directly into the prompt:

val groundingText = if (FeatureFlags.ENABLE_URL_GROUNDING) {
    "\n If the following message above is about the rules and terms to visit Le Louvre, " +
    "if needed answer this urls ${urlList.joinToString()}"
} else {
    ""
}

val prompt = "$text $groundingText"

var response = chat.sendMessage(prompt)

Hybrid inference: On-device review generation with Maps deep link

Not every AI task requires a cloud-based model, and not every device is online. To help developers balance latency, cost, and offline availability, we recently introduced the Firebase API for Hybrid Inference.

In Jetpacker, the restaurant review feature lets users review select topics and automatically drafts a review. To enable this for all users, we prioritize local execution with Gemini Nano, and fall back to cloud models on devices that don’t support Gemini Nano. 


The restaurant review feature uses hybrid inference to draft a review based on topics

// implementation("com.google.firebase:firebase-ai-logic")
// implementation("com.google.firebase:firebase-ai-ondevice:16.0.0-beta03")


// Initialize the model with hybrid routing configuration
val reviewModel = Firebase.ai.generativeModel(
    modelName = "gemini-3.1-flash-lite",
    onDeviceConfig = OnDeviceConfig(
        inferenceMode = InferenceMode.PREFER_ON_DEVICE
    )
)

The Hybrid Inference API supports four distinct routing modes:

  • PREFER_ON_DEVICE: Prioritizes local execution and falls back to cloud if Gemini Nano is unavailable.
  • PREFER_IN_CLOUD: Prioritizes cloud execution and falls back to on-device if the device goes offline.
  • ONLY_ON_DEVICE: Restricts execution strictly to the device.
  • ONLY_IN_CLOUD: Restricts execution strictly to the cloud.

Once the review is generated, we copy it to the clipboard and use an intent to open Google Maps directly to the restaurant's review page, providing a seamless user experience:

private fun copyAndOpenMapsReview(context: Context, reviewText: String, placeId: String) {
    val clipboard = context.getSystemService(Context.CLIPBOARD_SERVICE) as ClipboardManager
    val clip = ClipData.newPlainText("User Review", reviewText)
    clipboard.setPrimaryClip(clip)

    val uri = Uri.parse("https://search.google.com/local/writereview/mobile?placeid=$placeId")
    val intent = Intent(Intent.ACTION_VIEW, uri).apply {
        setPackage("com.google.android.apps.maps")
    }
    context.startActivity(intent)
}

Custom hybrid routing: Hotel support chat translation with simulated personas

The hotel support chat was built to let users finalize logistics and check on hotel details. This feature uses system instructions to configure a localized receptionist assistant. By passing specific information—such as the preferred language and hotel information—in the instructions, we can set up a conversational persona representing a specific hotel.

private val generativeModel = Firebase.ai(backend = GenerativeBackend.googleAI())
    .generativeModel(
        systemInstruction = content {
            text("""
              You are a helpful hotel receptionist at $hotelName only speaking $language. 
              Answer politely in $language. The bar closes at 10pm and breakfast is from 7am to 10am.
              There's someone at the desk 24/7. You can retrieve your luggage from the storage room 
              at the back of the lobby at any time.
              """)
        },
        modelName = "gemini-3-flash-preview"
    )

Because receptionist responses are in the hotel's local language (for example, French for Hotel Le Meurice in Paris), we need to translate messages to the user’s preferred language. 


Hotel support chat messages are automatically translated to the user’s preferred language 

While hybrid models can configure simple routing preferences, complex scenarios require custom routing logic. In Jetpacker, we implement a custom routing stack that takes into account:

  • Language identification: Using the on-device ML Kit Language Identification API, we can detect the incoming message language.
  • On-device translation (Gemini Nano): ML Kit’s Prompt API lets us translate common language pairs directly on the device, saving bandwidth and cloud cost.
  • Cloud translation (Gemini 3 Flash): For more complex languages, we use Gemini Flash 3 to get a higher quality translation.
// implementation("com.google.android.gms:play-services-mlkit-language-id:17.0.0") 

// ML Kit for Language Identification (powered by Google Play Services)
private val languageIdentifier = LanguageIdentification.getClient()

// On-device translator model (prefer Gemini Nano) for translating common language pairs
private val hybridTranslationModel = Firebase.ai(backend = GenerativeBackend.googleAI())
    .generativeModel(
        modelName = "gemini-3-flash",
        onDeviceConfig = OnDeviceConfig(mode = InferenceMode.PREFER_ON_DEVICE)
    )

// Cloud translator model for more complex language pairs
private val cloudTranslationModel = Firebase.ai(backend = GenerativeBackend.googleAI())
    .generativeModel(
        modelName = "gemini-3-flash"
    )

When a message needs to be translated, we identify the source language and apply our custom routing logic, executing either on-device or cloud translation:

fun translateMessage(message: SupportChatMessage) {
    viewModelScope.launch {
        // 1. Detect language using ML Kit Language Identification
        val sourceLang = try {
            Tasks.await(languageIdentifier.identifyLanguage(message.text))
        } catch (e: Exception) {
            "Undefined"
        }

        // 2. Custom routing: we've verified the translation quality for English and Korean with Gemini Nano, and will translate message on-device for those two languages
        val routeToCloud = sourceLang != "en" && sourceLang != "kr"

        val prompt = "Translate the following text to $selectedLanguage. Just return the translated sentence: ${message.text}."

        val (translatedText, routePrefix) = if (routeToCloud) {
            val result = cloudTranslationModel.generateContent(prompt)
            result.text to "[Cloud]"
        } else {
            val result = hybridTranslationModel.generateContent(prompt)
            result.text to "[On-Device]"
        }

        if (translatedText != null) {
            _translations.update { current ->
                current + (message.id to "$routePrefix: $translatedText")
            }
        }
    }
}

In this example, the custom routing logic only takes into consideration the translation’s source and target language. However, based on your app’s use case, you can expand the routing logic to include other factors such as the on-device model version, network connectivity, battery status, and more.

Securing the AI Pipelines: Firebase App Check

Lastly, using AI in the cloud opens up possibilities of API key abuse or unauthorized billing. To secure API calls, we integrated Firebase App Check using both Play Integrity (production) and the local Debug Provider (for local development or emulators).

In the JetPackerApplication.kt file, we install the debug provider at startup and trigger anonymous authentication to establish a secure user session:

//  implementation("com.google.firebase:firebase-appcheck-playintegrity") 
//  implementation("com.google.firebase:firebase-appcheck-debug")  
//  implementation("com.google.firebase:firebase-auth") 

override fun onCreate() {
    super.onCreate()
    Firebase.initialize(context = this)
    Firebase.appCheck.installAppCheckProviderFactory(
        DebugAppCheckProviderFactory.getInstance()
    )
    Firebase.auth.signInAnonymously()
}

When building locally on an emulator, App Check prints a local token secret to logcat:

Enter this debug secret into the allow list in the Firebase Console: a8c2dd4c-xxxx-xxxx-xxxx-ef6c114ba27e

Once registered in the Firebase console, local requests are fully verified and authenticated by App Check, protecting our backend while letting us test the app locally.

Conclusion

By combining cloud model capabilities (grounding, system instructions) with on-device capabilities (hybrid routing, translation, security app checks), we created a travel app that is smart, secure, and available offline.

Check out the full source code for Jetpacker on GitHub, and explore the Firebase documentation to get started:

Firebase AI Logic Documentation
Firebase Hybrid Inference API

Learn more

Check out the other parts of this blog post series:

Part 1: Introduction of the app and a high-level overview.
Part 2: On-device intelligence. Deep-dive into ML Kit’s GenAI APIs and Gemini Nano to build privacy-first features like itinerary summarization, receipt parsing, and local audio processing.
Part 3 (this post!): Hybrid and cloud reasoning. Explore how to use Firebase AI Logic to ground LLM answers in real-world data like Google Maps and web context.
Part 4: System integration. Integrating with the Android intelligence system using AppFunctions. 
Part 5 (coming soon): In-app agentic workflows. Extend the app with an end-to-end booking assistant powered by A2UI and ADK.

Interested in more on Android Development? Follow Android Developers on YouTube or LinkedIn!

All code snippets in this blog post follow the following copyright notice:

Copyright 2026 Google LLC.
SPDX-License-Identifier: Apache-2.0
Read the whole story
alvinashcraft
52 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

Build intelligent Android apps: On-device inference

1 Share
Posted by Caren Chang, Developer Relations Engineer, Android Developer Relations



Welcome back to the blog post series "Build intelligent Android apps" where we take a basic Android app and transform it into a personalized, intelligent, and agentic experience. In our previous post we introduced Jetpacker, the demo app we'll use throughout this series.

In this blog post, we will share how you can use Gemini Nano through ML Kit’s Prompt API to build intelligent on-device features.

Building intelligent on-device features refers to the ability to process prompts and data directly on a device without sending data to a server. This offers a few advantages:

  • User data can be processed locally on the device, preserving user privacy
  • Functionality of the model is reliable even with spotty or no internet connection
  • No additional cloud inference cost, since everything runs on the user’s hardware

With the benefits of on-device in mind, we identified three features to add in Jetpacker that can improve the user experience: summarizing trip itineraries, managing expenses, and capturing voice notes.

On-device features in Jetpacker: Summarizing trip itineraries, managing expenses, and voice notes

High quality tailored summarization of short texts

The itinerary screen gives users a quick overview of all activities for a given trip. Since this screen contains a lot of information, it can quickly become overwhelming. To help users prepare without feeling overwhelmed, we can add a ‘Get ready for your trip’ section at the top.

The romantic Paris trip is summarized as a classic Parisian adventure blending art, sights, and delicious food. A tip and some useful phrases are also added.

By inputting a trip itinerary and asking an LLM to summarize it, we can generate a quick summary of the trip along with packing tips and useful local phrases. This is a great use case for an on-device model for several reasons:

  • Performance and quality: Both the input and output text are relatively short. With that, we can expect the performance and quality of an on-device solution to be on par with more powerful cloud models.
  • Scalability: Shifting inference on-device allows us to scale this feature from a few users to millions without worrying about managing increasing cloud inference costs.
  • Low latency and reliability: On-device inference guarantees low latency, providing a reliable experience even when users are offline.

To build with on-device, we use Gemini Nano, Google’s most efficient model optimized for mobile devices. Gemini Nano was first introduced a few years ago, and is now running on over 140 million devices. The latest version of the model, Gemini Nano 4, is built on the architecture foundation of the recently released Gemma 4 model, and is further optimized for maximum battery and performance efficiency.

Using ML Kit’s Prompt API, we can take advantage of Gemini Nano 4’s new model capabilities to prototype our on-device features. We’ll create a prompt that includes the itinerary of a trip and ask the model to generate a summary along with any preparation tips.

// implementation("com.google.mlkit:genai-prompt:1.0.0-beta3") 

// Define the configuration for Gemini Nano 4 E2B preview model
val previewFastConfig = generationConfig {
    modelConfig = modelConfig {
        releaseStage = ModelReleaseStage.PREVIEW
        preference = ModelPreference.FAST
    }
}

val geminiNano2BPreviewModel = Generation.getClient(previewFastConfig)

val tripItinerary = ...

val getReadyForYourTripSummary = geminiNano2BPreviewModel
 .generateContent("Given this trip itinerary: $tripItinerary, 
     generate the following: overall vibe, tips on how to prepare for this
     trip, and common short phrases to learn for the trip.")

Finding the optimal prompt usually requires some iteration, and the AICore app is perfect for this step in the process. After opting into the developer preview option for AICore, we can download preview models such as Gemini Nano 4 to test prompts and see the model’s expected outputs. With a few iterations on the prompt, we were able to improve the speed of the response from 13 seconds to under 2 seconds! Check out the final code implementation and prompt here.

The first iteration of our prompt generated way too many tokens, and optimizing it helped keep responses quick and to the point.

Local processing for sensitive user input

Next, to help users enjoy their trip even more, we’ll build a simple expense manager that takes the manual work out of sorting through receipts and calculating budgets.


Taking a photo of a restaurant bill, data is parsed and shown in the expense overview screen of the app.

Since receipts might contain sensitive information like credit card number and addresses, this is another great use case for an on-device solution. With on-device, users can be confident that private information will be processed locally on the device without any of their data being sent to the cloud.

In addition, Gemini Nano 4 has improved model capabilities for multimodality, especially for image understanding tasks like OCR and visual data extraction, making it a great solution for tasks like extracting information from receipts.

For this use case, the prompt will analyze an image of the receipt, and output information such as: a generated title, amount spent and category of the expense. To ensure the model outputs the information in the preferred format, we can use ML Kit’s Structured Output API to seamlessly output a Kotlin data object that we define.

// implementation("com.google.mlkit:genai-prompt:1.0.0-beta3")
// ksp("com.google.mlkit:genai-schema-compiler:1.0.0-alpha1")

@Generable("Information extracted from an expense receipt")
data class ParsedReceipt(
  @Guide("Generated title for the expense less than 6 words. Based on restaurant or activity name.")
  val title: String,
  @Guide("Total amount of the expense. Look for values at the bottom and words like total or balance due.")
  val amount: Double,
  @Guide("Type of expense", enumValues = ["travel", "food", "shopping", "entertainment", "other"])
  val category: String,
)

val prompt = "Determine if the image is a receipt or expense. 
    If it is NOT a receipt or expense, output the text 'NOT_A_RECEIPT'.
    Otherwise, parse the receipt information."

val request = generateContentRequest(ImagePart(bitmap), TextPart(prompt)) {}
val requestWithStructuredOutput = generateTypedContentRequest(request, ParsedReceipt::class)

// Define the configuration for Gemini Nano 4 E4B preview model  
// When selecting models, you can specify which performance charactertists are most important
//  for your use case. Use ModelPreference.FULL when you want to prioritize reasoning power over speed. 
//  Use ModelPreference.FAST when complex logic is not required and latency is a priority.
val previewFullConfig = generationConfig {
    modelConfig = modelConfig {
        releaseStage = ModelReleaseStage.PREVIEW
        preference = ModelPreference.FULL
    }
}

val geminiNano4BPreviewModel = Generation.getClient(previewFullConfig)
val response = geminiNano4BPreviewModel.generateContent(requestWithStructuredOutput)
val parsedReceipt: ParsedReceipt? = response.candidates.firstOrNull()?.response

Multimodal input

Lastly, to help users record audio memos during the trip, let’s build a fully on-device voice notes feature. Using ML Kit’s Speech Recognition API, we’ll enable users to record short voice notes that are automatically transcribed to text. With the transcribed text, we’ll use ML Kit’s Prompt API to identify which trip activity is associated with the recorded voice note, letting users easily recap their trip as they scroll through the trip’s itinerary.

The Roman holiday itinerary shows voice note extracts.

The ML Kit GenAI Speech Recognition API allows you to transcribe audio content to text fully on-device using two distinct modes. Basic mode uses a traditional on-device speech recognition model and is available on most Android devices with API level 31 and higher. Advanced mode uses Gemini Nano to offer broader language coverage and better quality, and is currently supported on Pixel 10 devices.

For our feature we combine the Speech Recognition API with the ML Kit GenAI Prompt API:

// implementation("com.google.mlkit:genai-prompt:1.0.0-beta3")
// implementation("com.google.mlkit:genai-speech-recognition:1.0.0-alpha1")

val tripEvents = ... 

// Set up speech recognition
val speechRecognizerOptions =
    speechRecognizerOptions {
        locale = Locale.US
        preferredMode = SpeechRecognizerOptions.Mode.MODE_ADVANCED
    }
val speechRecognizer: SpeechRecognizer = SpeechRecognition.getClient(speechRecognizerOptions)

suspend fun transcribeVoiceNote(recognizer: SpeechRecognizer) {
    // Display partial text as the user is recording audio
    var partialTextResponse = ""

    // Display the full text once user is finished recording audio
    var transcription = ""

    val request: SpeechRecognizerRequest
        = speechRecognizerRequest { audioSource = AudioSource.fromMic() }
    recognizer.startRecognition(request).collect { response ->
        when (response) {
            is SpeechRecognizerResponse.PartialTextResponse -> {
                partialTextResponse = response.text
            }
            is SpeechRecognizerResponse.FinalTextResponse -> {
                transcription = response.text
                processAndCategorizeVoiceNote(transcription, tripEvents)
            }
        }
    }
}

fun processAndCategorizeVoiceNote(transcribedVoiceNote: String, events: List) {
    val prompt = "Given the voice note $transcribedVoiceNote
     and the following events for this trip: $events, rewrite this transcription
     to remove filler words. Then, identify which events from the
     list this rewritten transcription matches to."

     // Utilize ML Kit's Prompt API to process voice note and tag it with the relevant trip activities
     Generation.getClient().generateContent(prompt)
}

Conclusion

Using ML Kit’s GenAI APIs, we were able to take advantage of Gemini Nano to develop fully on-device intelligent features for the JetPacker app, and provide an improved user experience without any additional cloud costs.

Check out the full source code for Jetpacker on Github, and watch the video Build Intelligent Android apps with Google’s AI to learn more about how to integrate intelligent features directly into your app using on-device models, cloud-powered reasoning, and the latest agentic frameworks.

Learn more

Check out the other parts of this blog post series:

Part 1: Introduction of the app and a high-level overview.
Part 2 (this post!): On-device intelligence. Deep-dive into ML Kit’s GenAI APIs and Gemini Nano to build privacy-first features like itinerary summarization, receipt parsing, and local audio processing.
Part 3: Hybrid and cloud reasoning. Explore how to use Firebase AI Logic to ground LLM answers in real-world data like Google Maps and web context.
Part 4: System integration. Integrating with the Android intelligence system using AppFunctions.
Part 5 (coming soon): In-app agentic workflows. Extend the app with an end-to-end booking assistant powered by A2UI and ADK.

Interested in more on Android Development? Follow Android Developers on YouTube or LinkedIn!

All code snippets in this blog post follow the following copyright notice:

Copyright 2026 Google LLC.
SPDX-License-Identifier: Apache-2.0

Read the whole story
alvinashcraft
52 minutes ago
reply
Pennsylvania, USA
Share this story
Delete
Next Page of Stories