Earlier this year, Google announced Gemini Nano 4 is available as a Preview release via the AI Core Beta. This means developers can start building for Nano 4 to provide fully on-device AI experiences for their Android apps.

As Gemini Nano advances and starts to add more and more powerful models, I’ve been asking myself how I could harness this on-device advantage to deliver unique experiences.

The DM’s Assistant

I love playing tabletop roleplaying games. Mostly D&D 5e. Ever since I got into the hobby, I’ve mostly been a dungeon master.¹ The DM is the one charged with guiding the story and adjudicating the rules of the game.

As a DM, one of the hardest things to manage is having the right rules on-hand at the right time. Players are always going to try to grapple the Giant Spider at the weirdest moments. Usually this means either quickly googling rules or flipping through a rulebook.

I’ve always wished I could have an assistant sitting next to me at our game listening to the flow of play and making sure the right information was always at my fingertips. The major challenge being that D&D happens in person or through a huge array of digital tools, and few DMs have the same setup. You could probably integrate with Roll20 to follow the flow of the game, but then you’d have to do something different for campaigns on D&D Beyond or when your group all meets in person.

If your phone could listen to you (and, optionally, your players) talk through the flow of the game, there may be a way to reason about what rules you need to see when. On-device ML/AI would make listening easier, and summarization could make the information glanceable so that you’d never have to take your focus off the roleplaying.

So I decided to see how I could apply these (now more powerful) Nano-based APIs to this fun problem.

Nano APIs

First I got a bit more familiar with what was possible with Nano 4. For Android developers, access to this model is available via the ML Kit Gen AI APIs…I’ll refer to these as the “Nano APIs” moving forward.

The key features of these APIs are text-based processing like Summarization / Proofreading / Rewriting, Image Descriptions and Speech Recognition. There is also a more general ‘Prompt’ API for generating text content from text-based multimodal prompting, which has some powerful implications for prompt engineering.

Based on this list, I decided using a combination of Speech Recognition and Summarization might enable me to build something that has been on my wish list for a while now.

Revelations & Limitations

After some initial experimentation, I quickly learned a few things that would shape this project.

1. All of the Nano APIs use the same underlying model running inside AICore on the Android OS. This means they are competing for requests, and the API enforces limits on your app. If you want to use Nano 4-based voice recognition while also using it to summarize, you’re quickly going to be throttled.
2. The Summarization API is very strict about its output. It summarizes text as a bulleted list. This restriction, plus some restrictions on how you send summarization requests, made me look to the more flexible solution - prompting.

The impact on this demo was straightforward. Drop the Nano-based speech recognition pipeline and use the more general Prompt API to generate glanceable rules.

ML Kit / Nano Combined Pipeline

Here are the core pieces of the app’s processing pipeline, from the DM to relevant rules summaries, with my specific caveats called out.

1. Speech Recognition
2. Rules Lookup
3. Custom Summarization

Speech Recognition

The Gen AI Speech Recognition API supports both Basic and Advanced modes.

• Advanced mode provides similar functionality to Basic, but its use of Gemini Nano gives broader language coverage and natural language reasoning.
• Advanced mode only runs on Pixel 10 devices (with more in development), while Basic runs on most Android devices with API level 31 and higher.
• Because it uses AICore, Advanced mode is subject to the model bottleneck issue when put in contention with my summary requests.

With these callouts in mind, I decided Basic would be good enough for my needs in this case. We run speech recognition in a loop, capturing every utterance to be used as keywords in our rules search.

Rules Lookup

As mentioned before, a lot of the complexity of this app comes from the actual lookup of search terms in the D&D rules database.

As voice input is converted into text, this text is then converted into search terms used to query this D&D database of rules. This logic mostly lives in my TopicExtractor class, which processes each utterance - “the goblin ducks behind cover” - and cleans it into search terms to query the database with.

The actual DB search uses Full Text Search 5 with the built in bm25 function to properly search & rank SRD entry hits. Lists of hits are de-duped before rendering them to the screen, so the user isn’t spammed with duplicate rules entries as they talk through the game.

I mention this section specifically because just the combination of Basic Speech Recognition and a powerful SRD database search delivers a ton of value. The final piece, which actually utilizes AICore and Nano 4, is like the last 10% polish on the idea.

Custom Summarization

As mentioned earlier, the Summarization API is very specific about its input and output. Pass in raw text content, and get out 2-3 bullet points in summary.

For the case of my summary text, I wanted to have a 1-2 sentence summary in prose as the output. Maybe not a huge difference by itself.

However, I also am dealing with a slightly different type of content as input. Depending on what type of SRD rules entry I get back, I may want to add that as context to generate the summary text. This added flexibility is something that doesn’t exist within the Summarization API, but is offered with the Prompt API.

Here’s a real example from the demo app, where typeHint is something like "spell", "monster stat block", or "condition".

val prompt = 
  """  
  You are a Dungeon Master's assistant. 
  Summarize the following D&D 5e SRD $typeHint in ONE sentence (max 30 words). 
  Lead with the single most important fact a DM would want at a glance. 
  Do not preface with phrases like "This $typeHint…"; start with the substance.  
  
  Title: ${entry.name}  
  Body: $body  
  """

This helps tailor our output to exactly what we want to see for each summary.

Once we define our prompt, we use the ML Kit Gen AI API to get a GenerativeModel of our choosing. Here we are specifying that we want to use the PREVIEW version of the model (to get us access to Nano 4 on our Pixel 10) and that we want to use the FULL version (as opposed to FAST).

private val model: GenerativeModel = Generation.getClient(  
    generationConfig {  
        modelConfig = ModelConfig.builder()  
            .apply {  
                releaseStage = ModelReleaseStage.PREVIEW  
                preference = ModelPreference.FULL  
            }  
            .build()  
    },  
)

Then we can use this GenerativeModel directly to generate a text response based on our prompt.

val response = model.generateContent(prompt)

The actual response is a list of candidate text content. Each response type is a piece of output text and a ‘finish reason’ - like STOP or MAX_TOKENS. In our case we’ll use the first result and only inspect the finish reason if there is no valid text.

val text = response.candidates.firstOrNull()?.text?.trim().orEmpty()

This is really all there is to prompting at its core - input text via Prompt and get text content back. The magic happens inside the LLM, and there’s a ton you can do with good prompts.

Here’s the full code for my implementation of the summarize function from my GenerativeBackend. You can see here that there is some logic to handle serializing calls to generateContent, as my app may have many calls to summarize text happening in quick succession.

// Serializes every inference: the single AICore runtime rejects overlapping  
// generateContent calls with RESPONSE_PROCESSING_ERROR.
private val inferenceMutex = Mutex()  
  
override suspend fun summarize(entry: SrdEntry): String = withContext(Dispatchers.IO) {
    prepare() // Check model availability status & call `model.warmup()`
    val prompt = Prompt.summarize(entry)
    val response = inferenceMutex.withLock { model.generateContent(prompt) }  
    val text = response.candidates.firstOrNull()?.text?.trim().orEmpty()  
    if (text.isEmpty()) {  
        val reason = response.candidates.firstOrNull()?.finishReason  
        throw IOException("Prompt returned no text for ${entry.name} (finishReason=$reason)")  
    }
    text
}

The View Model managing my app’s timeline UI manages calling my GenerativeBackend for each new card to summarize its rule content as they populate. The result is a constantly updated list of rules ready to glance at as we need them.

Demo App

Just to give you a working idea of the semi-finished product, here’s a demo of the app handling the following prompt.

“Suddenly, a goblin jumps out and attempts to grapple you. You spot another goblin behind cover in the bushes, holding a hand crossbow. What do you do?”

“I cast fireball!”

Conclusion

I’m really happy with how this app turned out! I’ve been playing with it for awhile and I feel like it does a good job, particularly with ‘Rules Glossary’ type lookups that are most often the ‘edge-cases’ in a game.

Hopefully this gives you some inspiration to build your own Android experiences with Nano 4 on-device AI. The ability to send open-ended prompts gives you a ton of flexibility to tailor your LLM use to your app’s specific needs.

Thoughts on my approach? Questions? Suggestions for what to build next? Let me know on Mastodon or Bluesky.