Ranjan Roy from Margins is back for our weekly discussion of the latest tech news. We cover: 1) The rise of Muse 2) Is Meta back? 3) OpenAI's consumer blind spot 4) Meta's butthole marketing 5) What Muse means for ecommerce 6) Meta's Muse Charm, aka; The Muse Buddy or Muse Tamagotchi 7) Meta introduces camera-free smart glasses 8) What Meta Muse says about Frontier AI's weakness 9) Standard models are on the rise vs. frontier 10) The new Dopamine Site phenomenon is... good?

---

Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice.

Want a discount for Big Technology on Substack + Discord? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b

Learn more about your ad choices. Visit megaphone.fm/adchoices

Jeff stopped answering his phone and started declining same-day meetings.

Five months later he was a director of engineering.

Everything you can see from where you’re standing says that’s backwards. You look at your boss, then at your skip, and you see the 10pm Slack. The weekend work. The calendar with no white space left in it. Then you draw a conclusion, and the conclusion sends you one of two directions.

Either you decide that’s what it takes and start paying a price nobody ever asked you for. Or you decide that’s what it takes and quietly stop wanting the job.

Same misread, opposite casualties. One of them overpays with her evenings, her health, sometimes her marriage. The other walks away from a role he’d have been great in — and usually never notices he decided anything. He just stops reaching.

Here’s the problem with the evidence you’re working from.

I stepped outside at an Airbnb one morning and the driveway was soaked. Must have rained. I said so to a friend over coffee. It hadn’t. The sprinklers ran at 5am. My observation was real, but the “why” was wrong.

That’s the move you’re making on the org chart. The evidence is always real. It’s the conclusion that’s invented.

When Jeff changed his view to one from above, he landed Director of Engineering in just five months. He created new boundaries, delegated more work, and said “no” to more requests.

Jeff’s new boundaries cleared room for a multi-million-dollar proposal he’d been too busy to work on, while a $20,000 project ate his calendar. He went to his boss and told him that’s what he was doing. His boss said: focus on what’s most important.

In this episode I draw the two columns on the whiteboard — what you can see from below, and what actually gets seen from above — and walk through how to sort the behaviors you’ve absorbed into the ones that earned the job and the ones that were only ever baggage of bad leadership along the way.

You don’t have to choose between the scope you want and the life you want.

The Happy Engineer Podcast

LISTEN NOW

Back to ALL EPISODES

The post You Don’t Have to Live at Work to Become a Director of Engineering appeared first on OACO.

Build Flutter mobile apps with cutting-edge local LLM features

One codebase, multiple platforms, and all using local LLMs — can this work without compromises? When thinking about integrating local LLM features into your app, you will also start thinking about performance soon. The answer to performance is often “build native”. It makes perfect sense considering with LLMs you’ll want GPU access and NPU acceleration. All that is platform-specific.
And yet, Flutter Gemma v1.x takes a modular approach to resolve this conflict. In this article we’ll show you exactly how this works using a real demo app for Android and iOS with six feature areas: setup, text generation, analysis, embeddings, RAG capabilities, and chat.

Why Cross-platform for on-device AI?

For classical business use cases, the argument for having one codebase is well-known: Write logic once and deploy on multiple platforms like iOS and Android. For features including local models, this becomes more important because model handling and prompting strategies come on top of classical UI frontend development. Things you really don’t want to maintain in more than one place.

The inference itself stays platform-specific. Native runtimes, GPU, and NPU access differ fundamentally between Android and iOS. Android relies on open acceleration APIs such as NNAPI and GPU delegates via LiteRT, used to run Gemini Nano on modern Android devices. Apple devices use Core ML and the Neural Engine to run the Foundation Model inside a closed, vendor-specific stack. A good cross-platform layer bridges this gap without hiding it. The developer is still in control of which engine runs on which platform. For real-time applications where performance is critical, native development remains the way to go. However, for most product features related to on-device local LLMs, Flutter is becoming an increasingly viable choice.

The Ecosystem: Cross-platform options for local LLMs

Before diving deep into Flutter Gemma, we will look at the solution space: Generic approaches rely on Dart FFI and bind directly to llama.cpp/ggml or custom platform channel bridges for each platform. For example, llamadart uses llama.cpp for GGUF models and LiteRT for .litertlm models. But there are also pure Dart approaches even without an ffi dependency like ollama_dart. In addition, there are other smaller Flutter packages with similar concepts at various stages of maturity and varying levels of maintenance and active development.

Outside of Flutter, it’s worth taking a look at the purely native solution LLM.swift for iOS. It is a platform-exclusive approach that clearly illustrates the native control you give up when you opt for cross-platform development but also how much platform-specific work you save yourself in the process.

The common thread among most approaches is that they solve the question “can the model run at all?” rather than “how do I build a product feature interface around it?” That’s exactly where Flutter Gemma comes in.

Why Flutter Gemma stands out

Since release v1.0.0 Flutter Gemma follows a modular architecture: a core package and optional engine packages, instead of one big monolithic dependency. Right now, developers can choose between the following:

• flutter_gemma_mediapipe: .task and .bin models (Android, iOS, Web)
• flutter_gemma_litertlm: .litertlm models (Android, iOS, Desktop)
• flutter_gemma_embeddings: text embeddings
• flutter_gemma_rag_sqlite: sqlite vector store enabling RAG capabilities

How to choose the correct model for Flutter Gemma?

Answering this question isn’t too easy and includes more pitfalls than you might think at first.

Format first

Flutter Gemma does not just load any GGUF or Safetensor file you give it to. It explicitly expects .task/ .bin files (MediaPipe) or .litertlm files (LiteRT-LM). Both formats are already converted and ready for the engine. Practically this means: You look for a fitting model on HuggingFace, e.g., the Gemma family from the Google community or some LiteRT-LM models from the LiteRT community, instead of converting a model yourself.

Quantization is no trivial matter

Most pre-converted models are quantized, very often as INT4 (sometimes as INT8). This drastically reduces download size and RAM requirements — exactly what makes on-device inference on a smartphone even possible in the first place. The trade-off is a certain loss of quality compared to full precision. For most everyday use cases, like rewriting, classification, and summarization, INT4 is a good compromise. If you think the output is becoming too imprecise, you should look for a higher-resolution variant of the same model, rather than simply choosing a larger model across the board.

RAM is the real limiting factor, not storage space: The download size provides only a rough estimate for the required RAM. During inference, the model requires additional RAM depending on the context length and backend (LiteRT-LM or MediaPipe), which can result in significantly larger memory usage than the file size. A model that runs smoothly on your own test device may already reach its limits on an older device or one with less RAM and crash the app when trying to load the model or run inference.

Additional Considerations for Selection

• Gated vs public
  Some models like Gemma3, Gemma4 or EmbeddingGemma require a license agreement and a HuggingFace API token, while Qwen3 or Gecko are freely available. This is relevant for user onboarding as well as CI/CD and not just for local development. We enjoy using Gemma models because they offer pretty good quality in the context of mobile on-device inference, but Qwen3 and Gecko are great alternatives that don’t require auth.
• Backend support
  Not every quantization level performs equally on GPU and CPU delegates. Some combinations only perform well on one of the two. This requires good testing with a variety of devices.
• Context length
  Smaller models often come with smaller maximum context windows. For long RAG contexts or detailed chat histories, this can quickly become a limiting factor. In our experience, this is the biggest drawback — but for all mobile local LLMs.

The Flutter Gemma Demo App

For this article, we built a small Flutter app that serves as a central theme. It uses flutter_gemma and all its submodules for model setup and inference as well as flutter_bloc for state management. We’re deliberately leaving out the navigation and similar peripheral topics. The app is divided into five tabs (Generation, Analysis, Embedding, RAG and Chat) and a setup screen. Each feature is built as its own Bloc + Widget pair with model access handled centrally using the FlutterGemma API.

The demo app uses Qwen3 0.6B for text inference and Gecko 256 for embeddings. Both models are publicly available and do not require a HuggingFace API token.

The central initialization runs once at app start and includes registering the engine.

Future<void> main() async {
  // ...
  await FlutterGemma.initialize(
    inferenceEngines: const [
      LiteRtLmEngine(), // .litertlm — Android, iOS, Desktop
      MediaPipeEngine(), // .task / .bin — Android, iOS, Web
    ],
    embeddingBackends: const [
      LiteRtEmbeddingBackend(),
    ],
    vectorStore: kIsWeb ? WebSqliteVectorStore() : SqliteVectorStore(),
    huggingFaceToken: dotenv.env['HUGGING_FACE_API_KEY'],
  );
}

Model Setup: Download, Installation, Availability

The entry point for the app is the Setup screen. First, it checks if there is any model installed without loading it.

/// [hasActiveModel] is a synchronous and cheap check
if (FlutterGemma.hasActiveModel()) { 
  emit(state.copyWith(status: SetupStatus.ready));
} else {
  emit(state.copyWith(status: SetupStatus.notInstalled));
}

If there is no model available, the app automatically initiates a download from HuggingFace with a progress bar visible in the UI.

await FlutterGemma.installModel(
      modelType: ModelType.qwen3,
      fileType: ModelFileType.litertlm,
    )
    .fromNetwork(
      ModelConstants.inferenceModelUrl,
    )
    .withProgress((progress) => add(_ProgressUpdated(progress)))
    .install();

Public models like Qwen3 or Gecko do not need API tokens. If your model requires a token, you can use a .env file in combination with the dotenv package to securely provide the API token. If the download fails, e.g., a network or storage problem occurs, the package throws a dedicated DownloadException instead of a generic Exception or Error:

try {
  await FlutterGemma.installModel(
    modelType: ModelType.qwen3,
    fileType: ModelFileType.litertlm,
  )
  .fromNetwork(
    ModelConstants.inferenceModelUrl,
    // dotenv.env['HUGGING_FACE_API_KEY'], // pass token here if required
  )
  .withProgress((progress) => add(_ProgressUpdated(progress)))
  .install();
} on DownloadException catch (e) {
  emit(
    state.copyWith(
      status: SetupStatus.error,
      errorMessage: e.error.toUserMessage(),
    ),
  );
}

In short: Setup is the only place where network or storage problems could realistically occur. The rest of the app can rely on the already installed models.

Feature: Text generation (Rewrite & Summarize)

The Generation Tab rewrites text in a specific mood or simply summarizes the text. The prompt gives the model clear instructions, including the instruction to preserve critical data like numbers or names.

enum AiMood {
  professional,
  casual,
  concise,
  analytical;

  String get label {
    return switch (this) {
      AiMood.professional => 'Professional',
      AiMood.casual => 'Casual',
      AiMood.concise => 'Concise',
      AiMood.analytical => 'Analytical',
    };
  }
}

final prompt = '''
You are a professional writing assistant. Rewrite the following text 
to sound highly ${event.mood.label.toLowerCase()}. Maintain all 
critical data facts, numbers, dates, names, and contact details 
exactly as written. Do not omit them.\n\n
Text to rewrite:\n${state.inputText};'''

The technical flow behind this is a pattern, that will be used in several places in the app:

1. Get the active model
2. Create a chat session
3. Get the response
4. Close the model

final model = await FlutterGemma.getActiveModel(
  maxTokens: ModelConstants.maxTokens,
);
try {
  final chat = await model.createChat();
  await chat.addQueryChunk(Message.text(text: prompt, isUser: true));
  final response = await chat.generateChatResponse();
  final text = response is TextResponse ? response.token : '';
  emit(state.copyWith(status: GenerationStatus.success, output: text));
} finally {
  await model.close();
}

This chat/session pattern of Flutter Gemma is perfectly fine and a good reusable pattern for every text feature, but not for classical chats, as we will see later in the chapter Feature: Chat with History. The quality of responses varies a lot between models used. For Qwen3 0.6B, the results for summarization and rewriting are quite good, although sometimes it just repeats the original text.

Feature: Structured Analysis (Classification and Entity Extraction)

The Analysis tab includes two scenarios: Classifying a support mail and extracting entities from log entries. The key to getting structured output here is prompting and telling the model (Qwen3) to generate plain JSON without Markdown or explanations.

const prompt = '''
Classify this content and respond with valid JSON only — 
no markdown, no explanation:\n
$_kScenario3Text\n\n
JSON format:\n
{"category":"...","priority":"...","labels":["...","..."]};'''

Even the bigger mobile local LLMs don’t always strictly follow these rules. That’s why post-processing matters even more. The code must be robust against format deviations. An easy fallback is returning the raw response.

String _formatClassification(String raw) {
  try {
    final json = jsonDecode(_extractJson(raw)) as Map<String, dynamic>;
    final category = json['category'] ?? '—';
    final priority = json['priority'] ?? '—';
    final labels = (json['labels'] as List?)?.join(', ') ?? '—';
    return '📋 CLASSIFICATION REPORT\n'
        'Category: $category\n'
        'Priority: $priority\n'
        'Tags: $labels';
  } catch (_) {
    return raw.trim();
  }
}

String _extractJson(String raw) {
  final start = raw.indexOf('{');
  final end = raw.lastIndexOf('}');
  if (start == -1 || end == -1) return raw;
  return raw.substring(start, end + 1);
}

While classification with Qwen3 0.6B seems to work reliably, entity extraction almost never succeeds in extracting all the information. In most cases it struggles to detect the “Impacted Users” (which is “14,000” in the example, but “0” got extracted). Repeating the entity extraction multiple times results in different outcomes for each execution.

Feature: Embeddings and Semantic Similarity

The Embedding tab uses its own model (Gecko 256), along with its own runtime ( flutter_gemma_embeddings), separate from the text generation model (Qwen3). Embeddings are a different class of models with their own workflow and use case: similarity and search rather than text generation. They need a completely different mental model consisting of vectors instead of words and similarity instead of responses. When choosing a model, there’s a classic trade-off. For example, Gecko 256 is small and publicly available, while EmbeddingGemma 512 is more accurate but gated.

The technical flow is rather simple:

1. Embed reference text
2. Embed user input
3. Calculate cosine similarity between the two vectors

final embedder = await FlutterGemma.getActiveEmbedder();
try {
  final inputVec = await embedder.generateEmbedding(
    state.similarityInput,
  );
  final score = _cosineSimilarity(state.referenceVector!, inputVec);
  emit(
    state.copyWith(
      status: EmbeddingStatus.referenceReady,
      similarityScore: score,
    ),
  );
} finally {
  await embedder.close();
}

Calculating the cosine similarity is pure Dart code and not part of the FlutterGemma API.

double _cosineSimilarity(List<double> a, List<double> b) {
  double dot = 0, magA = 0, magB = 0;
  for (var i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }
  final denom = math.sqrt(magA) * math.sqrt(magB);
  return denom == 0 ? 0 : dot / denom;
}

In the demo app, we embedded a small text about planets. Comparing our two inputs, “Planets are big rocks in the universe” (A) and “Bees are very important insects because they are the global pollinators” (B), we can see that input A has a higher similarity score than input B because it is semantically closer to the topic in the embedded text. At least with English text, Gecko 256 enables reliable similarity matching on mobile devices.

Feature: RAG with SQLite-VEC and filters

The RAG tab comes with its own small knowledge base consisting of seven short documents on science, cooking, and history. The knowledge base is indexed and searchable fully on-device, without a server or an external vector store.

Notably, we don’t need to install a second embedding model to use Flutter Gemma RAG capabilities. Instead, we use the same active Gecko 256 embedder as the Embedding tab.

Storage is handled by flutter_gemma_rag_sqlite. A SQLite-VEC store is created directly in the app’s Documents directory, and indexing is performed via addDocument() using text and JSON metadata.

await FlutterGemma.rag.initialize('${dir.path}/$_kRagDbFileName');

for (final doc in _kCorpus) {
  await FlutterGemma.rag.addDocument(
    id: doc.id,
    content: doc.content,
    metadata: doc.metadataJson, // {"topic": "...", "year": ...}
  );
}

The real “out-of-the-box” experience is delivered by searchSimilar(). Embedding the user request and performing a cosine-similarity search run in one single call — no manual vector handling required like in the Embedding chapter.

final results = await FlutterGemma.rag.searchSimilar(
  query: state.searchQuery,
  topK: 5,
  filter: _buildFilter(),
);

New in Flutter Gemma v1.x are type-safe metadata filters FilterSchema and FilterField, declared once in FlutterGemma.initialize().

await FlutterGemma.initialize(
    inferenceEngines: const [
      LiteRtLmEngine(),
      MediaPipeEngine(),
    ],
    embeddingBackends: const [
      LiteRtEmbeddingBackend(),
    ],
    vectorStore: kIsWeb ? WebSqliteVectorStore() : SqliteVectorStore(),
    filterSchema: const FilterSchema(
      fields: [
        FilterField(name: 'topic', type: FilterFieldType.string),
        FilterField(name: 'year', type: FilterFieldType.number),
      ],
    ),
    huggingFaceToken: dotenv.env['HUGGING_FACE_API_KEY'],
  );

As a result, the filters for topic and year run as true vec0 column conditions directly in sqlite-vec, rather than post-filtering in Dart. Every column in a virtual table follows strict rules for typing, formatting, and structure requirements. This means robustness, better performance, and less development overhead. In the demo, topic chips and a “from 2020” filter combine to form exactly such a typed condition.

Filter? _buildFilter() {
  final must = <Condition>[];
  if (state.topicFilter != kAllTopics) {
    must.add(FieldEquals(key: 'topic', value: state.topicFilter));
  }
  if (state.recentOnly) {
    must.add(FieldRange(key: 'year', gte: 2020));
  }
  return must.isEmpty ? null : Filter(must: must);
}

Each match also provides a similarity score, which makes it easy to distinguish strong matches from weak ones. While the Embedding chapter shows the basic concept of similarity between two texts, the RAG tab shows a practical application as a searchable knowledge base.

In short, with sqlite-vec and typed filters, Flutter Gemma delivers RAG as a ready-made component rather than a patchwork of individual parts (or packages in Flutter terms).

To get a good mixture of knowledge in the demo app, we provided documents from different centuries about science, history, and cooking with rather complex information to really challenge the on-device LLM capabilities. Again, the Gecko 256 Embedder does a great job, and RAG features of Flutter Gemma seem to work like a charm. Asking, “How is sourdough bread made?

Feature: Chat with History

The Chat tab differs from the previous features in a crucial way: the Generation and Analysis tabs create a new chat session for each request, fetch a single response, and close the model again. For a real chat, this is insufficient because the model needs to remember the whole conversation. In Flutter Gemma this is not a special case but a matter of lifecycle management: InferenceChat maintains its own history internally if you reuse the same instance across multiple turns, rather than creating a new one for each message.

Future<void> _onStarted(ChatStarted event, Emitter<ChatState> emit) async {
  final model = await FlutterGemma.getActiveModel(
    maxTokens: ModelConstants.maxTokens,
  );
  final chat = await model.createChat(
    modelType: ModelType.qwen3, 
  );
}

The parameter modelType is not a minor detail here. For Qwen3 Flutter Gemma automatically appends /no_think to every user message to suppress the built-in thinking mode of the model. The <think> tags are also filtered from the response stream. Without this parameter, Qwen3 behaves significantly more unpredictably.

Every new message is passed to the chat instance via addQueryChunk(). The method generateChatResponseAsync() streams the response token by token. An emit.forEach loop is used to transfer those tokens into UI state.

Future<void> _onSubmitted(ChatMessageSubmitted event, Emitter<ChatState> emit,) async {
  await chat.addQueryChunk(Message.text(text: text, isUser: true));

  final buffer = StringBuffer();
  await emit.forEach<ModelResponse>(
    chat.generateChatResponseAsync(),
    onData: (response) {
      if (response is TextResponse) buffer.write(response.token);
      return state.copyWith(messages: _withStreamedReply(buffer.toString()));
    },
  );
}

History is being persisted in session memory as well as internally by InferenceChat from Flutter Gemma. Like this, the next call of generateChatResponseAsync() knows the whole conversation and not only the last message. This means we don’t have to glue something together or maintain the history ourselves.

Resetting a conversation with clearHistory() does not only clear the message list but also closes the native session and reopens it. Otherwise, the model would continue to respond based on a transcript that is no longer visible in the UI.

await chat.clearHistory();

During this chapter, we learned that multi-turn chat is not an extra feature of Flutter Gemma but rather a direct result of the InferenceChat lifecycle. Chat history is being persisted by keeping the chat instance alive and not because you rebuild it yourself.

Qwen3 0.6B has met exactly the expectations we had for an LLM running on smartphone hardware. The quality of the chat is “okay”, but not really much more than that. Qwen3 0.6B has a mediocre understanding of context: It can stick to a topic quite well, but most information is hallucinated entirely. Sometimes the model strays from the topic after one or two requests. In one case, the model responded with an empty text after three questions, and it did not respond to requests anymore. After clearing the chat, it worked again. Usually, we can write about 8–15 messages before the end of the context window is reached. What I’ve noted very positively is that mixing languages like English and German seems to work better than expected. The model can translate a sentence and correctly switch back to the original language of the conversation. The only problem that occurs when using a language other than English: there are garbled characters at the end of the messages. I don’t know why but it always happens as soon as I start chatting in German. Of course, you must take the model size into account here, and for just 0.6B parameters, it’s astonishing that it produces such a quality at all. Newer models like Gemma4, with more parameters and better quantization, are likely to perform even significantly better in this regard. However, such a comparison would go beyond the scope of this article.

Limitations, Pitfalls and Practical Lessons

On-device models have real storage and memory budgets. Download size and RAM requirements during inference on mobile devices are not an implementation detail — they are a hard limiting factor. Recurring patterns in the demo app: Models need to be closed (e.g., model.close() and embedder.close()) after finishing the task, or otherwise the resources will not be freed.

Despite a unified Dart API, platform differences persist under the hood. The engines of MediaPipe and LiteRT, as well as the systems iOS and Android themselves, do not always behave identically. There are cases where native approaches remain the better choice: Performance-critical scenarios or highly platform-specific hardware acceleration (as with LLM.swift mentioned earlier).

Conclusion

Cross-platform and local AI are not mutually exclusive. This article’s demo app demonstrates how the modular approach of Flutter Gemma resolves an apparent conflict of objectives. For teams that maintain a shared codebase for Android and iOS and don’t have extremely performance-critical real-time requirements, this is a very practical path to on-device LLM features.

Flutter Gemma really shines for small tasks and is a great way to implement local AI features for older smartphones that don’t have native support for Google’s Gemini Nano or Apple’s Foundation Model. Where native solutions continue to lead remains a matter of requirements.

This is evident across all five feature areas: Flutter Gemma doesn’t just answer the question, “Does the model even run?”. Instead, it provides the concepts needed for real product features, ranging from the download pipeline and typed RAG filters to the session lifecycle in chat. The actual work then lies in prompt engineering and product logic, not in the infrastructure.

The quality of the output from on-device smartphone LLMs remains a matter of debate. In our view, it’s already suitable for some use cases, but not so much for others. For example, summarizing and classifying shorter texts, as well as embedding texts, works surprisingly well with these small models. However, when working with complex, longer texts or performing entity extraction, there are still too many errors, so these tasks should continue to be handled by a powerful external model.

You can find the complete code in the our GitHub repository:

GitHub - EXXETA/flutter_local_llm: A small cross-platform mobile app to demonstrate on-device LLMs with Flutter Gemma. It features text generation, classification, entity extraction, RAG capabilities and a chat.

Feel free to try out the app right away with your own model!
(by Jonas Klock)

Flutter Gemma: The hidden gem of cross-platform local AI was originally published in Flutter Community on Medium, where people are continuing the conversation by highlighting and responding to this story.