Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
159400 stories
·
33 followers

Meta Patents AI Glasses to Use Facial Recognition to Identify People, Make Highlight Reels of Your Dinner Party

1 Share
Meta has patented a smart-glasses system that could use facial recognition to identify people and automatically create personalized highlight reels of events such as dinner parties. The patent doesn't guarantee the feature will ship, but it offers a detailed look at how Meta is exploring facial recognition and AI-powered memory capture for its wearable devices. 404 Media reports: "I've generated some highlights of tonight's dinner party. Would you like to see them?" a prompt from the system says, alongside various thumbnails of what look like people laughing, according to one illustration in the patent. One section says the system may personalize highlight files using "user relationship data." The illustrations clearly show a person wearing a pair of glasses, looking at a group of people, then the glasses focusing on one or more people in particular. Patentlyze, an organization that tracks patents, first alerted 404 Media to the patent on Friday. The patent is dense with how such a system would work, but in sum, the system with one or more cameras receives an input from the user, then uses machine-learning and "sensory data" to figure out points of interest in the camera's field of view. That can include detecting people in the shot "based on one or more facial recognition algorithms," identifying those specific people, detecting their facial expressions, using "eye gaze data of the user captured by the client system," and figuring out other points of interest "based on scene and semantic understanding." Although the patent is for "particular camera-based tasks by particular systems in a particular manner" -- in this case, the company's smart glasses -- Meta writes it "contemplates assisting users in any suitable camera-based task by any suitable system in any suitable manner." Meaning that although this technology is focused on the glasses, maybe the company will use it for other purposes in the future.

Read more of this story at Slashdot.

Read the whole story
alvinashcraft
8 hours ago
reply
Pennsylvania, USA
Share this story
Delete

The Evolution of Modern C-Sharp: Practical Best Practices for Cloud-Native and Enterprise Apps

1 Share

The C# programming language has come a remarkably long way since its early days, continuously evolving into one of the most expressive, performant, and versatile languages in modern computing. Today, with modern .NET innovations, writing high-performance enterprise applications has become smoother and more intuitive than ever.

 

Whether you are architecting large-scale microservices, crafting responsive web APIs, or building AI-driven solutions, keeping pace with language enhancements is essential for every developer. In this comprehensive guide, we explore the newest language capabilities, memory efficiencies, and practical patterns that will elevate your daily engineering workflow.

 

Modern C# and .NET Language Evolution
Exploring modern C# language features, clean syntax enhancements, and cloud-native runtime optimizations.

 

 

  • Explore how modern releases build upon the historical milestones of the language to deliver zero-allocation idioms.
  • Understand the power of expanded params collections supporting ReadOnlySpan and generic collections directly.
  • Learn how the dedicated System.Threading.Lock type brings cleaner code semantics and enhanced concurrency throughput.
  • Discover advancements in ref struct lifetimes, allow ref struct generics, and new escape character formatting.
  • Master real-world enterprise architectures leveraging Native AOT compilation for microservices and cloud workloads.

 

1. The Continued Evolution of the C# Language

If you have been developing software on the Microsoft platform for a decade or more, you will recall how transformative each milestone has been. In our classic retrospective on the evolution of C# from 1.0 to 5.0, we witnessed fundamental leaps like generics, LINQ, and the async-await pattern that completely reshaped developer productivity.

 

Over subsequent releases, from the introduction of null-conditional operators in C# 6.0 and concise expression-bodied method syntax to modern pattern matching, the C# language team has maintained a clear focus: making code more expressive while dramatically reducing boilerplate.

 

Modern C# is engineered around high performance, zero-allocation memory paradigms, and cloud-native scalability. Today's language features do not simply offer syntactic sugar; they actively empower developers to write cleaner, safer code that executes with bare-metal speed across Linux containers, macOS, and Windows.

 

Core Themes Driving Modern Language Design

  • Reducing memory allocations across high-throughput server loops by leveraging Span and ReadOnlySpan semantics natively in language constructs.
  • Improving type safety and compile-time verification to catch edge cases, null references, and threading issues long before code reaches production.
  • Streamlining cloud-native development by enhancing Native Ahead-of-Time (AOT) compilation compatibility and minimizing container startup latency.

 

 

2. Expanded Params Collections and Clean Syntax

One of the most welcomed enhancements in recent C# versions is the modernization of the classic params modifier. Historically, the params keyword was strictly limited to single-dimensional arrays, meaning every method invocation with multiple arguments inevitably allocated a temporary array on the managed heap.

 

With expanded params collections, developers can now use params with any recognized collection type, including ReadOnlySpan<T>, Span<T>, IEnumerable<T>, List<T>, and immutable collections. This allows for clean, variadic method calls without incurring unnecessary garbage collection overhead.

 

Using ReadOnlySpan with params enables zero-allocation variadic methods on the stack. The compiler automatically maps arguments into a stack-allocated buffer when available, providing immediate throughput improvements for high-traffic Web API endpoints and logging utilities.

 

Key Advantages of Modern Params Collections

  • Zero-Allocation Calling: Defining methods taking params ReadOnlySpan<T> avoids heap array creation completely, reducing GC pressure in high-frequency trading and telemetry pipelines.
  • Direct Collection Support: Methods can now seamlessly accept strongly typed List<T> or custom collection types without writing separate overload wrappers for arrays.
  • Backward Compatibility: Existing callers requiring array arguments continue to function smoothly while newer code paths immediately reap memory and performance benefits.

 

 

3. Dedicated Concurrency Primitives: The New System.Threading.Lock

For over two decades, C# developers synchronized concurrent threads using the lock statement alongside arbitrary reference objects (typically new object()). While familiar, this approach relied on internal synchronization blocks in the runtime header of every object, which lacked explicit synchronization intent.

 

Modern .NET introduces the enhanced lock object via the dedicated System.Threading.Lock type. When the C# compiler encounters a lock statement targeting a Lock instance, it generates optimized code utilizing the EnterScope() pattern instead of legacy Monitor methods.

 

This dedicated primitive provides clearer semantic meaning in your architecture, enables modern ref struct scoping for synchronization guards, and delivers measurable performance gains in heavily multi-threaded workloads.

 

Benefits of System.Threading.Lock

  • Optimized Runtime Execution: The JIT compiler optimizes the lock acquisition and release path, reducing CPU cycles compared to generic monitor lookups on arbitrary heap objects.
  • Scope-Based RAII Semantics: Invoking myLock.EnterScope() returns a lightweight ref struct that releases the lock deterministically upon exiting the using block.
  • Prevention of Common Locking Pitfalls: Discourages locking on public string literals or externally accessible instances, eliminating subtle deadlocks in complex systems.

 

 

4. Memory Safety, Ref Structs, and String Enhancements

Ensuring memory safety while maintaining maximum throughput is a foundational philosophy of modern C#. Recent language versions have expanded generic constraints to support allows ref struct, enabling high-performance types like Span<T> to be used within generic abstractions for the very first time.

 

In addition to advanced memory mechanics, developers also enjoy subtle yet delightful daily syntax refinements. For instance, the new \e escape sequence provides a clean, standard shorthand for the ASCII escape character (0x1B), eliminating cumbersome octal or Unicode workarounds when formatting terminal outputs and ANSI color streams.

 

The expansion of ref struct capabilities allows developers to build high-speed parsers without sacrificing type safety. Libraries handling JSON deserialization, binary protocols, and stream parsing can now write generic algorithms that operate directly on stack memory.

 

Advancements in Memory and Type Safety

  • Allow Ref Struct Generics: The allows ref struct anti-constraint permits generic interfaces and classes to accept stack-only ref structs, unlocking unprecedented performance in serialization libraries.
  • Enhanced Method Group Natural Types: The compiler now determines unambiguous natural types for overloaded method groups more accurately, simplifying delegate construction and LINQ expressions.
  • Clean Terminal Escape Codes: The new \e escape sequence standardizes terminal formatting in CLI tools, console dashboards, and ANSI colorized loggers across platforms.

 

 

5. Cloud-Native Performance and Ahead-of-Time (AOT) Compilation

As enterprise architectures shift toward Kubernetes and serverless microservices, cold start times and memory footprints have become crucial economic factors. In cloud-native .NET applications, Native Ahead-of-Time (AOT) compilation compiles C# code directly into architecture-specific machine code without requiring a heavy JIT runtime.

 

Recent runtime updates have expanded Native AOT support across ASP.NET Core minimal APIs, gRPC services, and background workers. Microservices compiled with Native AOT launch in single-digit milliseconds and consume a fraction of the baseline RAM required by traditional JIT runtimes.

 

Moreover, developers are combining these high-speed runtimes with intelligent workflows. As shown in our tutorial on building an agentic AI workflow in C# with Microsoft AutoGen, the ecosystem provides first-class tooling for running AI orchestration and LLM integrations directly inside performant .NET services.

 

Through systematic performance optimization in memory allocators, vectorized SIMD instructions, and tiered compilation, .NET continues to lead industry benchmarks for web request throughput and raw computing efficiency.

 

Why Cloud-Native .NET Excels in Enterprise Deployments

  • Sub-Millisecond Startup: Native AOT executables eliminate dynamic JIT warmup latency, making them ideal for instant auto-scaling in serverless cloud environments.
  • Drastically Reduced Memory Footprint: Stripping unused metadata and JIT compilation infrastructure allows dozens of container instances to run on smaller virtual machines.
  • End-to-End Enterprise Tooling: Built-in OpenTelemetry metrics, health checks, rate-limiting middleware, and structured logging ready for distributed cloud architectures.

 

 

6. Developing in Modern Visual Studio and Cloud Environments

Writing cutting-edge C# code is greatly enhanced by the rich developer tooling available today. In our guide on using Visual Studio for building cross-platform apps, we examined how unified IDE workflows enable building for mobile, cloud, desktop, and web from a single workstation.

 

Modern editions of Visual Studio and Visual Studio Code offer AI-assisted IntelliCode completions, automated refactorings for new language syntax, and integrated profiling tools that highlight memory allocations directly within your code editor.

 

Adopting modern language idioms not only makes your codebase more elegant and readable, but also ensures that your solutions take full advantage of runtime optimizations engineered by the Microsoft compiler teams.

 

 

7. Frequently Asked Questions (FAQs)

1. What is the biggest advantage of modern C# for everyday developers?

The primary advantage is the combination of enhanced developer productivity and built-in performance. Features like pattern matching, record types, and expanded params allow developers to express complex business logic cleanly while minimizing heap allocations and runtime overhead.

 

2. How do expanded params collections differ from classic params arrays?

Classic params required declaring a single-dimensional array, which always resulted in a heap allocation when passed multiple arguments. Expanded params support ReadOnlySpan, Span, List, and IEnumerable, allowing zero-allocation stack buffers and direct collection passing.

 

3. Why should I use System.Threading.Lock instead of object for locking?

System.Threading.Lock provides dedicated synchronization semantics that the compiler and runtime optimize specifically for locking. It avoids allocating synchronization blocks in general object headers and enables clean, scope-based locking with EnterScope().

 

4. What does the "allows ref struct" constraint do?

The allows ref struct anti-constraint enables generic types and methods to work with ref struct types such as Span and ReadOnlySpan. Previously, ref structs could not be used in generic parameters, which limited their reusability in high-performance generic algorithms.

 

5. What is Native AOT in .NET, and when should I use it?

Native Ahead-of-Time (AOT) compilation compiles your C# application directly into native machine code during publishing. It is ideal for cloud-native microservices, serverless functions, and containerized workloads where instant startup time and minimal memory consumption are critical.

 

6. Can I use modern C# features with older .NET Framework applications?

Many syntax-level features (like pattern matching and record structs) can work with older frameworks if configured in the project file, but runtime-dependent features (such as System.Threading.Lock, Native AOT, and Span optimizations) require modern .NET runtimes.

 

7. What is the new \e escape sequence in C#?

The \e escape sequence represents the ASCII escape character (hex 0x1B, decimal 27). It provides a standard, convenient way to write ANSI escape codes for coloring and formatting text in terminal and console applications.

 

8. How does modern C# help reduce Garbage Collection (GC) pressure?

By providing memory-safe primitives like Span, ReadOnlySpan, ref structs, and stackalloc alongside params collections, C# enables data manipulation directly in contiguous stack memory, drastically reducing the number of objects created on the garbage-collected heap.

 

9. How can I migrate my existing C# codebase to the latest version?

You can upgrade your project's Target Framework Moniker (TFM) to the latest .NET release in the .csproj file. Visual Studio and the .NET Upgrade Assistant provide automated tooling to refactor deprecated code paths into modern idioms.

 

10. Is C# suitable for AI and machine learning development?

Yes. With frameworks like Microsoft Semantic Kernel, AutoGen.NET, ML.NET, and ONNX Runtime bindings, C# has become a premier enterprise language for orchestrating generative AI workflows, agentic systems, and local model inference.

 

 

 

8. Concluding Thoughts and Next Steps

As we wrap up this technical overview, it is truly inspiring to see how C# continues to balance rapid modernization with robust backward compatibility. Each new language iteration provides tangible ways to write cleaner, more expressive code that simultaneously improves throughput in production environments.

 

I encourage you to test these new features in your day-to-day experiments and side projects. Refactor a few legacy utility classes to use params collections, try out the new Lock primitive in your background workers, and explore the benefits of Native AOT in your next microservice deployment.

 

What are your favorite new features in modern C#? Which language enhancements have made the biggest difference in your daily development workflow? Please share your thoughts, questions, and insights in the comments section below so we can keep the conversation going!

 

Thank you for reading, and happy coding!

 

Read the whole story
alvinashcraft
8 hours ago
reply
Pennsylvania, USA
Share this story
Delete

Dogfooding BlocSignal on the Web: Building a 100K Ops/sec Reactive App with Jaspr and Dart 3.13

1 Share

Building Pure Dart Web Apps Without Compromise

When developers evaluate Dart for the web, they typically face a stark tradeoff:

  1. Flutter Web: Exceptional for canvas-driven applications, design systems, and cross-platform desktop/mobile parity—but heavy for content-first landing pages, docs, and fast-loading SEO sites.
  2. Jaspr Web: A lightweight, component-driven framework that compiles pure Dart to HTML and CSS with instant first paint and full search engine indexing.

When we built the official documentation and showcase site for BlocSignal, we knew Jaspr was the perfect foundation. But like many engineers diving into a new UI paradigm, our initial implementation took a shortcut: we used raw StatefulComponent lifecycles and manual .subscribe() callbacks to wire up our state machines.

It worked—but it wasn't idiomatic.

In this behind-the-scenes case study, we walk through the process of dogfooding bloc_signals_jaspr across blocsignal.dev, replacing manual subscription glue with declarative consumer components, achieving 100,000 operations/sec in compiled JavaScript, and exploring the sheer developer ergonomics of Dart 3.13 primary constructors.

The "Manual Subscription Trap": Why Raw .subscribe() Fails at Scale

In classic Flutter or Jaspr development, when you create a state machine without framework-level consumer widgets, you might be tempted to subscribe inside initState():

// ❌ THE ANTI-PATTERN: Manual subscription glue in StatefulComponent
class LiveVisualizerState extends State<LiveVisualizer> {
  late final LiveCounterBloc _bloc;

  @override
  void initState() {
    super.initState();
    _bloc = LiveCounterBloc();
    // ⚠️ Flaw 1: Every state change triggers a full component setState
    _bloc.state.subscribe((_) {
      if (mounted) setState(() {});
    });
  }

  @override
  void dispose() {
    // ⚠️ Flaw 2: Manual dispose tracking
    _bloc.close();
    super.dispose();
  }
}

While this appears harmless in a simple counter demo, it introduces three severe architectural flaws:

  1. The Double Re-Render Penalty: When a user clicks a button that calls both _bloc.add(Event()) and a local setState(), the component executes two back-to-back render passes in the exact same frame.
  2. Batch UI Thrashing: If you execute a high-frequency benchmark (e.g. dispatching 1,000 events in a loop), manual subscriptions attempt to invoke setState() 1,000 times during the loop, creating massive JS event-loop thrashing.
  3. Coarse-Grained Rebuilds: The entire component tree rebuilds on every change, even if only a single badge or numeric label changed value.

To solve this, we brought the full power of bloc_signals_flutter's declarative consumer components over to Jaspr in bloc_signals_jaspr.

1. Declarative Routing with a NavigationCubit

Rather than relying on ad-hoc URL parsing scattered across components, we modeled the entire site navigation as a pure Dart state machine:

// lib/src/cubits/navigation_cubit.dart
import 'dart:js_interop';
import 'package:bloc_signals_jaspr/bloc_signals_jaspr.dart';
import 'package:web/web.dart' as web;

@JS('trackGaPageView')
external void _trackGaPageView(JSString path);

class NavigationCubit() extends CubitSignal<String> {
  this : super(initialState: _resolveCurrentPath()) {
    // Listen to browser history navigation
    web.window.addEventListener('popstate', ((web.Event _) => _sync()).toJS);
    web.window.addEventListener('hashchange', ((web.Event _) => _sync()).toJS);
    _trackPageView(stateValue);
  }

  static String _resolveCurrentPath() {
    final path = web.window.location.pathname;
    final hash = web.window.location.hash.toLowerCase();
    if (path.startsWith('/showcase') || hash.contains('showcase')) return '/showcase';
    if (path.startsWith('/minesweeper') || hash.contains('minesweeper')) return '/minesweeper';
    if (path.startsWith('/publications') || hash.contains('publications')) return '/publications';
    return '/';
  }

  void _sync() {
    final next = _resolveCurrentPath();
    if (next != stateValue) {
      emit(next);
      _trackPageView(next);
    }
  }

  void _trackPageView(String route) {
    try {
      _trackGaPageView(route.toJS);
    } catch (_) {}
  }
}

At the root of the application, we inject the cubit using BlocSignalProvider and build the active page using BlocSignalBuilder:

// lib/src/app.dart
class const App({super.key}) extends StatelessComponent {
  @override
  Component build(BuildContext context) {
    return BlocSignalProvider<NavigationCubit>(
      create: (_) => NavigationCubit(),
      child: const _AppRouter(),
    );
  }
}

class const _AppRouter() extends StatelessComponent {
  @override
  Component build(BuildContext context) {
    return BlocSignalBuilder<NavigationCubit, String>(
      builder: (context, currentPath) => switch (currentPath) {
        '/showcase' => const ShowcasePage(),
        '/minesweeper' => const MinesweeperPage(),
        '/publications' => const PublicationsPage(),
        _ => const HomePage(),
      },
    );
  }
}

Now, anywhere in the component tree—such as our sticky navigation header—we can reactively highlight active links with zero prop-drilling using context.select():

// lib/src/components/navbar.dart
final activePath = context.select<NavigationCubit, String>((c) => c.stateValue);

a(
  href: '/showcase',
  classes: activePath == '/showcase' ? 'nav-active' : '',
  [Component.text('Showcase')],
)

2. Fine-Grained DOM Updates with BlocSignalSelector

On the blocsignal.dev homepage, the Interactive Live Visualizer demonstrates real-time state updates across multiple metrics:

  1. Primary State: The raw integer count.
  2. Computed State (2x): state * 2.
  3. Parity & Status: EVEN / ODD and POSITIVE / NEGATIVE / ZERO.

Instead of rebuilding the entire visualizer card on every tick, each card uses BlocSignalSelector to isolate its DOM mutations:

// 1. Primary Count Selector
BlocSignalSelector<LiveCounterBloc, int, int>(
  selector: (state) => state,
  builder: (context, count) => span(classes: 'metric-value', [
    Component.text('$count'),
  ]),
),

// 2. Computed 2x Doubled Selector
BlocSignalSelector<LiveCounterBloc, int, int>(
  selector: (state) => state * 2,
  builder: (context, doubled) => span(classes: 'metric-value', [
    Component.text('$doubled'),
  ]),
),

// 3. Record-based Multi-Value Selector
BlocSignalSelector<LiveCounterBloc, int, ({String parity, String status})>(
  selector: (state) => (
    parity: state % 2 == 0 ? 'EVEN' : 'ODD',
    status: state > 0 ? 'POSITIVE' : (state < 0 ? 'NEGATIVE' : 'ZERO'),
  ),
  builder: (context, derived) => div(classes: 'status-row', [
    span(classes: 'chip', [Component.text(derived.parity)]),
    span(classes: 'chip', [Component.text(derived.status)]),
  ]),
)

Buttons dispatch events directly using context.read<LiveCounterBloc>():

button(
  classes: 'btn-increment',
  onClick: () => context.read<LiveCounterBloc>().add(IncrementEvent()),
  [Component.text('+ 1 Increment')],
)

And background telemetry logging is captured cleanly with BlocSignalListener:

BlocSignalListener<LiveCounterBloc, int>(
  listener: (context, state) {
    _appendLog('⚡ TRANSITION -> State: $state [0ms Synchronous]');
  },
  child: visualizerMarkup,
)

3. Side-by-Side: Traditional Dart 3.5 vs. Modern Dart 3.13

Because our website is an application rather than a published library package, we can take full advantage of Dart 3.13 primary constructors and constructor shorthands.

Look at the difference in boilerplate when defining a reactive Jaspr card:

Traditional Dart 3.5 Syntax

class MetricCard extends StatelessComponent {
  const MetricCard({
    required this.title,
    required this.value,
    super.key,
  });

  final String title;
  final String value;

  @override
  Component build(BuildContext context) {
    return div(classes: 'metric-card', [
      span([Component.text(title)]),
      h3([Component.text(value)]),
    ]);
  }
}

Modern Dart 3.13 Syntax

class const MetricCard(final String title, final String value, {super.key}) 
    extends StatelessComponent {
  @override
  Component build(BuildContext context) {
    return div(classes: 'metric-card', [
      span([Component.text(title)]),
      h3([Component.text(value)]),
    ]);
  }
}

By placing fields directly in the primary constructor parameter list, 5 lines of boilerplate collapse into a clean, single-line class header with zero repetition.

4. Web Performance Reality: 100K Operations/Sec in Browser JavaScript

One of the biggest surprises for developers testing the live visualizer on blocsignal.dev is the built-in stress test:

Benchmark: Dispatches 1,000 synchronous transitions in a tight loop.

In traditional stream-based architectures (like classic BLoC or Rx on the web), dispatching 1,000 events allocates 1,000 StreamController events and queues 1,000 microtask hops through Dart's async runtime.

In BlocSignal, state changes propagate through a synchronous dependency graph:

  • 0 Microtask Queue Hops: State transitions resolve in the exact same call stack.
  • 0 Intermediate Frame Tearing: The DOM settles cleanly without intermediate stutter.
  • Throughput: Even running in compiled JS inside a standard browser tab, it clocks over 100,000 operations per second (~10ms for 1,000 full event-state cycles).

Summary & Live Demo

Dogfooding bloc_signals_jaspr on our own production website proved that building web applications in pure Dart doesn't require choosing between developer discipline and raw performance:

Feature Classic Web BLoC / Rx bloc_signals_jaspr
Reactivity Latency Microtask Queue Delay (Async) 0ms Synchronous Call Stack
Component Wiring Manual subscribe / dispose Declarative BlocSignalBuilder
DOM Rebuild Scoping Coarse Component Rebuilds Fine-Grained BlocSignalSelector
JS Web Throughput ~2,000 – 10,000 ops/sec ~100,000+ ops/sec
Syntax Overhead Verbose Field & Constructor Maps Dart 3.13 Primary Constructors

You can try the interactive visualizer and play the live Minesweeper case study right now at blocsignal.dev!

All the source code is open source and visible directly in our GitHub monorepo at RandalSchwartz/BlocSignal. ⭐️

Read the whole story
alvinashcraft
16 hours ago
reply
Pennsylvania, USA
Share this story
Delete

Serving Gemma4 with Rust on vLLM 🦀

1 Share

This tutorial walks through installing and setting up the Rust toolchain for vLLM on an
AWS EC2 G5g instance — Graviton2 (aarch64) with an NVIDIA T4G GPU — and getting vLLM's
Rust frontend (vllm-rs) built, running, and verified.

This paper is a follow-on to the original G5g Gemma 4 build.

Everything below was run on the box. 🦀

Wait, vLLM has Rust in it?

You betcha. Since PR #40848 (merged
2026-05-21), vLLM vendors a 14-crate Rust workspace:

bench  chat  cmd  engine-core-client  llm  managed-engine  metrics
mock-engine  parser  parser/python  server  text  tokenizer  tracing

Edition 2024, resolver 3. Straight from the vendored rust/Cargo.toml:

Crate Version Job
axum 0.8.8 the HTTP server
tokio 1.47.1 async runtime
zeromq 0.6.0 talks to the Python engine
rmp-serde / rmpv 1.3.1 msgpack on the wire
minijinja 2.22 chat templates
tonic / prost 0.14.6 / 0.14.3 gRPC — remember this one

It's a drop-in replacement for the Python FastAPI server. Two artifacts get built:

  • 🦀 vllm-rs — the axum frontend binary
  • 🐍 vllm._rust_tool_parser — a PyO3 extension module

Rust is a build requirement now

That's the headline, and it's reason enough on its own: you cannot build vLLM from source at
v0.27.2rc0 without Rust in the picture.
setup.py imports it at module scope, line 21,
unguarded:

from setuptools_rust.build import build_rust

No try, no feature flag, no opt-out. Metadata generation doesn't happen without it.

And this isn't a quirk of one release. vLLM's Rust surface is 14 crates covering the HTTP
frontend, the tool parser, the tokenizer and the benchmark client, and it has been growing
since it landed. If you build inference infrastructure from source, a Rust toolchain is
becoming table stakes — so it's worth knowing how to drive it properly rather than working
around it.

Three things do get conflated, though, and they have different scopes:

Component Needed to build vLLM? Needed to serve?
setuptools_rust (Python pkg) yes, always no
cargo / rustc toolchain for working Rust artifacts no
protoc for vllm-rs specifically no

Then why doesn't pip install vllm need this?

Because normally pip installs it for you. pyproject.toml declares it:

[build-system]
requires = [
    "cmake>=3.26.1", "ninja", "packaging>=24.2",
    "setuptools>=77.0.3,<81.0.0", "setuptools-scm>=8.0",
    "setuptools-rust>=1.9.0",          # <- pip grabs this automatically
    "torch == 2.13.0",                 # <- ...and this. Which is the problem.
    "wheel", "jinja2",
]

Under normal build isolation, pip creates a clean env, installs that list, and builds.
You never see setuptools_rust because you never had to think about it.

But look at the torch pin. Building in isolation means pip installs torch 2.13.0 from
PyPI
— and the PyPI aarch64 wheels are built for sm_80 and up. No sm_75. Which
destroys the entire reason for building from source on a T4G.

So on this box you must build against the DLAMI's own torch, and that means:

python use_existing_torch.py
pip install -e . --no-build-isolation

--no-build-isolation turns off the automatic install of everything in that requires
list.
From that moment on, every build dependency is yours to supply by hand — including
setuptools_rust, which is why it turns up as a bare ModuleNotFoundError minutes into a
build that has nothing visibly to do with Rust.

So the toolchain was always required; isolation was just hiding it. Building this way means
you own the dependency list, which is the rest of this walk-through. ⚡

What the DLAMI gives you, and what it doesn't

The AWS Deep Learning ARM64 AMI ships a runtime, not a build environment. On a fresh box:

Thing Present?
PyTorch 2.12 with sm_75
NVIDIA driver
nvcc / CUDA toolkit
Rust toolchain
setuptools_rust
protoc

Four of those six are on you. Let's install them.

Step 1 — Rust itself

Standard rustup, nothing aarch64-specific about it:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"
stable-aarch64-unknown-linux-gnu installed - rustc 1.97.1 (8bab26f4f 2026-07-14)

Rust is installed now. Great!

Note the triple: stable-aarch64-unknown-linux-gnu. Rust's aarch64 support is a complete
non-event
, which is a lovely change of pace on this hardware. ⚡

Step 2 — setuptools-rust

python3 -m pip install setuptools_rust

Per the section above: --no-build-isolation means pip won't do this for you. Do it early
— the failure lands during metadata generation, minutes into a build, as a bare
ModuleNotFoundError: No module named 'setuptools_rust' nowhere near anything that looks
like Rust.

⚠️ Install it into the same interpreter you'll build with. On the DLAMI that's
/opt/pytorch/bin/python3, not the system python3 — they're different, and the one that
matters is whichever owns the torch you're building against.

Step 3 — protoc 🔎

This is the one nobody documents:

apt-get install -y protobuf-compiler
protoc --version
libprotoc 3.21.12

Why: vllm-rs depends on the vllm-server crate, vllm-server builds gRPC stubs with
tonic/prost, and prost-build shells out to protoc. Skip it and the frontend binary
does not get built — see the summary at the end for how loudly that doesn't fail.

The tool parser has no protobuf dependency, which is why it builds either way.

Step 4 — the CUDA toolkit, while you're here

Not Rust, but the same class of problem, and you need it for vLLM's kernels:

# NVIDIA's **sbsa** repo — not the x86 one, easy reflex to get wrong on Arm
apt-get install -y cuda-toolkit-13-2

Step 5 — build the Rust artifacts

cd /opt/vllm-src
python tools/build_rust.py --release

⚠️ Do not omit --release. setuptools-rust builds inplace targets in debug by default,
and pip install -e . is an inplace build. The difference is not subtle:

Artifact Debug Release
_rust_tool_parser.abi3.so 100,913,216 B 1,009,080 B

100x. The debug artifact is four times the size of every CUDA kernel in vLLM combined.

Timing on a g5g.xlarge (4 vCPU), cold:

real    9m1.746s
user    25m9.199s
sys     1m35.023s

501 crates. Zero warnings. Exit 0. 🟢

Rust's aarch64 support does not put up a fight here — which is a pleasant contrast with the
CUDA side of this box, where SM 7.5 on Graviton needs a custom arch list and a patched
kernel.

Step 6 — check what you got

ls -la vllm/vllm-rs vllm/_rust_tool_parser.abi3.so
-rwxr-xr-x 1 root root 50039024 vllm/vllm-rs
-rwxr-xr-x 1 root root  1009080 vllm/_rust_tool_parser.abi3.so
file vllm/vllm-rs
ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV),
dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, not stripped
vllm/vllm-rs --help
Rust frontend and managed-engine CLI for vLLM.

Commands:
  frontend  Run the Rust OpenAI frontend as a Python-supervised worker
  serve     Launch a managed Python headless engine, then run the Rust OpenAI frontend
  bench     Run vLLM benchmarks
  render    Run engine-free request rendering and preprocessing

If vllm/vllm-rs isn't there, go back to Step 3.

Step 7 — run it, and mind the entrypoint ⚠️

VLLM_USE_RUST_FRONTEND=1 vllm serve google/gemma-4-E2B-it \
  --dtype float16 \
  --kv-cache-dtype auto \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 8 \
  --tensor-parallel-size 1 \
  --host 0.0.0.0 --port 8000

It must be vllm serve. If you launch the module directly —

# ❌ VLLM_USE_RUST_FRONTEND is IGNORED here
python -m vllm.entrypoints.openai.api_server --model--host 0.0.0.0 --port 8000

— the variable does nothing. No warning, no Unknown vLLM environment variable line. The
server comes up healthy and serves happily on the Python frontend, and a benchmark run
against it looks entirely normal.

The flag is read in exactly two places:

vllm/entrypoints/cli/serve.py:62        envs.VLLM_RUST_FRONTEND_PATH if envs.VLLM_USE_RUST_FRONTEND else None
vllm/entrypoints/openai/dp_supervisor.py:261   if envs.VLLM_USE_RUST_FRONTEND and envs.VLLM_RUST_FRONTEND_PATH:

api_server.py never mentions it.

How do I know it's actually Rust? 🔎

Three checks. Do all three the first time.

1. The server: header:

curl -si localhost:8000/health | grep -i '^server:'
Frontend Response
🐍 Python server: uvicorn
🦀 Rust (no server: header at all)

2. The process:

pgrep -af vllm-rs
26588 /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17
  --input-address  ipc:///tmp/f60f3962-d45b-4bcd-9026-c0dc32736028
  --output-address ipc:///tmp/5f75411d-2787-43bb-b4fc-14bf504a1cce
  --engine-start-index 0 --engine-count 1 --data-parallel-size 1

3. The log prefix(RustFrontend pid=…) instead of (APIServer pid=…):

INFO [utils.py:392] Launching Rust frontend: /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17 …

So where does Rust actually sit?

In two places, and they're quite different. One is a separate process; the other is a
shared object loaded inside the Python process. Here's the whole VM:

┌─ EC2 g5g.4xlarge ── Graviton2, aarch64 ─────────────────────────────────────┐
│                                                                             │
│  Deep Learning ARM64 AMI · Ubuntu 24.04 · NVIDIA driver 595.71.05           │
│  you add > cuda-toolkit-13-2 (sbsa) · rustup 1.97.1 · protobuf-compiler     │
│                                                                             │
│      HTTP :8000                                                             │
│          |                                                                  │
│          v                                                                  │
│  ┌───────────────────────────────┐                                          │
│  │ [RUST] vllm-rs                │  50 MB aarch64 ELF, its OWN process      │
│  │        axum 0.8.8 · tokio     │  built from the vendored rust/ workspace │
│  │        minijinja · fastokens  │  <- Step 5                               │
│  └────────┬─────────────▲────────┘                                          │
│           |             |                                                   │
│  ipc://   | ROUTER      | PULL     msgpack (rmp-serde / rmpv)               │
│           v             |                                                   │
│  ┌────────┴─────────────┴────────┐                                          │
│  │ [PY]   vLLM supervisor        │  `vllm serve` opens the socket, then     │
│  │                               │  hands listen-fd 17 down to vllm-rs      │
│  └────────┬──────────────────────┘                                          │
│           | spawns                                                          │
│           v                                                                 │
│  ┌───────────────────────────────┐                                          │
│  │ [PY]   EngineCore             │  torch 2.12.0+cu132, arch list has sm_75 │
│  │  ┌─────────────────────────┐  │                                          │
│  │  │ [RUST] _rust_tool_parser│  │  PyO3 .so LOADED INTO the Python         │
│  │  │        1.0 MB release   │  │  process — not a process of its own      │
│  │  └─────────────────────────┘  │                                          │
│  └────────┬──────────────────────┘                                          │
│           | CUDA                                                            │
│           v                                                                 │
│  ┌───────────────────────────────┐                                          │
│  │ NVIDIA T4G · SM 7.5           │  15,360 MiB GDDR6 · 277 GB/s measured    │
│  │ TRITON_ATTN kernels           │  weights 9.94 GiB · KV 2.95 GiB          │
│  └───────────────────────────────┘                                          │
└─────────────────────────────────────────────────────────────────────────────┘

Two things worth pulling out of that picture:

  • vllm-rs is not a sidecar you point at a port. The Python side opens the listening socket and passes the file descriptor down. It's a worker the supervisor forks and feeds.
  • _rust_tool_parser is Rust living inside Python. It's the one that always builds (no protoc needed), which is why a broken install still leaves Rust on the box — just not the Rust you wanted.

And note where the GPU sits relative to all of this: at the bottom, behind everything. That's
the reason the benchmark below comes out the way it does.

Bonus: there's a Rust benchmark client too

VLLM_USE_RUST_BENCH=1 vllm bench serve …

Same binary, bench subcommand. Requires VLLM_RUST_FRONTEND_PATH to resolve, so it needs
the same Step 3 → Step 5 you just did.

Two warnings you should not scroll past 🔴

The server came up healthy. These went by in the startup log anyway.

Gemma 4 defeats the fast tokenizer:

INFO    [hf.rs:200] loading tokenizer with fastokens
WARNING [hf.rs:221] failed to load tokenizer with fastokens; falling back to
        HuggingFace tokenizers
        error=tokenizer error: normalizer error: unsupported normalizer type: Replace

fastokens 0.2.1 doesn't implement the Replace normalizer that Gemma 4's tokenizer.json
uses, so it falls back to the same HuggingFace tokenizers the Python path uses. Note the
fallback is graceful and correct — you just don't get the fast path on this model yet. It's
a coverage gap in a young crate, and one normalizer away from closing.

Multimodal isn't wired up for this model:

WARNING [multimodal.rs:446] multimodal model spec is not registered; disabling
        image/video support   model_id="google/gemma-4-E2B-it" model_type="gemma4"

Gemma 4 E2B is a vision model, and gemma4 isn't in the Rust multimodal spec table yet.
Text requests behave identically and the endpoint is healthy, so nothing in a normal check
reveals it. Also a registration gap rather than a design problem — but check it for your model
before you switch, because a healthy endpoint won't tell you.

Is it faster?

On a T4G, no. Output token throughput, same engine config, client on the box against
localhost:

Concurrency 🐍 Python 🦀 Rust
1 28.65 29.30
4 97.48 97.26
8 168.33 169.39
16 169.96 170.19
32 170.99 170.34

Median TTFT tracks just as tightly — 14305 ms against 14311 ms at concurrency 32.

That's the expected result, and worth saying plainly: decode on this card is
bandwidth-bound at a measured 277 GB/s, and the engine saturates at --max-num-seqs 8.
A frontend rewrite targets CPU-side per-request overhead. Here that overhead hides behind the
GPU, so swapping it can't move a bottleneck-limited number. If you want the Rust frontend
to buy you tokens per second on a small GPU, it won't.

One signal does appear, in median inter-token latency at high concurrency:

Concurrency 🐍 Python 🦀 Rust Δ
16 38.55 36.18 −6.4%
32 38.41 36.23 −5.9%

Mean TPOT barely moves, so this is the middle of the distribution tightening rather than
everything speeding up — the shape you'd expect from a frontend scheduling streaming work
more evenly once many streams are in flight. Worth knowing if you serve at concurrency; not
worth switching for on its own. 📊

If a plain pip install -e . already ran

A from-source vLLM install done without the steps above succeeds, exits 0, and leaves you
with a 96 MB debug tool parser and no frontend binary. Four defaults stack up to make that
silent:

Symptom Cause Fix
No vllm/vllm-rs after a clean build protoc absent ⇒ vllm-server fails with code 101 Step 3
pip install exits 0 anyway optional=not should_require_rust_frontend() — setuptools-rust swallows it VLLM_REQUIRE_RUST_FRONTEND=1
_rust_tool_parser.abi3.so is ~96 MB editable ⇒ inplace ⇒ debug profile --release
FileNotFoundError: … vllm-rs was not found the above, discovered at import time Steps 3 + 5
Healthy server, but server: uvicorn flag set on the api_server module, which never reads it vllm serve

VLLM_REQUIRE_RUST_FRONTEND=1 turns the second row into a hard build failure, which is what
you want on any machine you plan to serve from.

Cheat sheet

# toolchain — you supply these by hand because the sm_75 requirement
# forces --no-build-isolation, which disables pip's automatic build deps
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. "$HOME/.cargo/env"
/opt/pytorch/bin/python3 -m pip install setuptools_rust   # the BUILD interpreter
apt-get install -y protobuf-compiler cuda-toolkit-13-2

# build against the DLAMI's torch, not a PyPI one (PyPI aarch64 has no sm_75)
cd /path/to/vllm
python use_existing_torch.py
TORCH_CUDA_ARCH_LIST=7.5 VLLM_REQUIRE_RUST_FRONTEND=1 \
  pip install -e . --no-build-isolation

# Rust artifacts, release profile (editable installs default to debug: 100x bigger)
VLLM_REQUIRE_RUST_FRONTEND=1 python tools/build_rust.py --release

# confirm
ls -la vllm/vllm-rs && vllm/vllm-rs --help

# run — `vllm serve`, NOT the api_server module
VLLM_USE_RUST_FRONTEND=1 vllm serve <model> --host 0.0.0.0 --port 8000

# verify it's really Rust
curl -si localhost:8000/health | grep -i '^server:'   # Rust sends none
pgrep -af vllm-rs

Run on EC2 g5g.xlarge and g5g.4xlarge, us-east-1a, NVIDIA T4G (SM 7.5). vLLM
0.27.2rc1.dev0+g7f7a32cfe, rustc 1.97.1, setuptools-rust 1.13.0, libprotoc 3.21.12,
torch 2.12.0+cu132. Benchmarks are one run per cell for Rust and two for Python; treat the
TPOT delta as suggestive.

Read the whole story
alvinashcraft
16 hours ago
reply
Pennsylvania, USA
Share this story
Delete

RNR 370 - CopilotKit with Mike Ryan

1 Share

Robin and Mazen talk with Mike Ryan of CopilotKit about bringing AI agents to React Native apps. They break down AG-UI, generative UI, shared state, and the guardrails developers need to build useful, trustworthy mobile experiences.

 

Show Notes

  1. CopilotKit: Bring Users and AI Agents together inside real apps

 

Connect With Us!

 

Sponsored by Infinite Red

Infinite Red is a premier mobile app consultancy, especially focused on Expo and React Native, located fully remote in the US. We’re a team of 30 with highly experienced mobile app developers and have been doing this for over a decade. We are also one of the first development teams to adopt agentic coding in a way that keeps high quality standards and aren’t afraid to do things the old school way if we need to. If you’re looking for mobile app or React Native or Expo expertise for your next project, hit us up at infinite.red/radio.





Download audio: https://cdn.simplecast.com/media/audio/transcoded/1208ee61-9c16-43c1-bc4c-ca790717f4a8/2de31959-5831-476e-8c89-02a2a32885ef/episodes/audio/group/220a20e4-2625-473e-8b16-6e06389afbe6/group-item/5abcc824-fe72-452b-8ccd-605efc8f7832/128_default_tc.mp3?aid=rss_feed&feed=hEI_f9Dx
Read the whole story
alvinashcraft
17 hours ago
reply
Pennsylvania, USA
Share this story
Delete

PPP 519 | Overcoming the Inner Propaganda That Sabotages Projects and Teams, with Owen Fitzpatrick

1 Share

Summary

In this episode, Andy sits down with Owen Fitzpatrick, a psychologist, speaker, and author of Inner Propaganda: Leading Hearts and Minds through Turbulent Times. Owen has spent close to 30 years studying how beliefs form and change, interviewing people everywhere from North Korea to Rwanda to Afghanistan. His thesis is unsettling: our brains do not simply take in facts and reach objective conclusions. They build a story we then experience as reality.

Owen and Andy work through what that means on real projects. You'll hear how a warning from a colleague can quietly harden into a conviction about a teammate, and how Bayesian reasoning gives you a way out. You'll learn Owen's five types of truth, how to tell courageous conviction from dangerous denial, and what leaders can actually make stable when they cannot promise a stable outcome. Owen also explains why pushing harder for buy-in is often the very reason people resist, and how an antifragile identity helps teams face uncertainty like AI without denial or panic.

If you're looking for a fresh way to think about belief, influence, and leading through turbulent times, this episode is for you!

Sound Bites

  • "We all live in that world where we think we're the one person that isn't the victim of propaganda, and my point or my thesis is we're all victims of our own inner propaganda."
  • "We're not necessarily just convinced by others. We convince ourselves."
  • "Because I think when we say, 'I'm no good at something,' we lock ourselves into it."
  • "Well, if you're not great at communicating with people, get great."
  • "Most of the time our beliefs just create a sort of a reality for us, and that reality can help us or harm us."
  • "So our brains are prediction machines."
  • "Whenever we talk about belief, believing in your ability to succeed in the future is critical if you want to succeed in the future, but that doesn't mean you deny the present."
  • "It, it's not that we think negatively, it's we believe negatively, and we see the world through those lenses."
  • "So I think we want to be able to challenge our beliefs and build a bit more and get more comfortable with uncertainty away from the table, but when we're at the table with our team, that's when we bring the certainty."
  • "We like the idea that we are making this decision of our own free will."
  • "And when you look at what a belief is, a belief is an idea we feel certain about, and that word feel is the most important word of that sentence."
  • "It's okay to believe less in certain things. It's okay to believe more in certain things. It's okay to believe better."

Chapters

  • 00:00 Introduction
  • 02:29 Start of Interview
  • 02:40 The Belief About Himself Owen Took Too Long to Update
  • 05:56 A Sweaty Debate Speech and What Came After
  • 08:28 Not Good at Something Is a Skill Gap, Not an Identity
  • 09:45 The Belief Growth Mindset
  • 12:55 The Sandra Story: When a Warning Becomes a Conviction
  • 14:07 Why the Brain Craves Cognitive Closure
  • 17:15 Using Bayesian Reasoning to Loosen a Belief
  • 21:00 Melanie Perkins and Alan Mulally: Conviction or Denial?
  • 21:36 The Five Types of Truth
  • 26:24 Just Because It Is Your Truth Does Not Make It True
  • 28:41 Where Objective Truth Actually Belongs
  • 31:47 What "Leadership Is Propaganda" Does Not Mean
  • 34:45 Why Change Management and AI Adoption Stall
  • 36:30 Why Owen Chose Propaganda Over Self-Talk
  • 37:57 Do I Believe It Because It Makes Me Look Good?
  • 38:49 Answering the AI Question Without Denial or Hype
  • 40:15 Building an Antifragile Identity
  • 44:27 Certainty Is Contagious, and When to Change Course
  • 45:20 What Leaders Can Make Stable in a Volatile Period
  • 47:45 Control What You Can, Influence What You Can, Accept the Rest
  • 51:01 When Pushing for Buy-In Creates the Resistance
  • 52:45 You're Asking the Wrong Question About Persuasion
  • 54:30 Reading the Person Before You Make the Ask
  • 57:30 Helping Kids Hold Strong Beliefs Without Contempt
  • 1:01:46 End of Interview
  • 1:02:15 Andy Comments After the Interview
  • 1:05:09 Outtakes

Learn More

You can learn more about Owen and his work at InnerPropaganda.com.

For more learning on this topic, check out:

  • Episode 370 with Chantel Prat. One of the smartest, clearest, and funniest books on the brain, and why all of it matters for how you lead.
  • Episodes 59 and 60 with Cathy Davidson. She explains how the brain science of attention changes everything.
  • Episode 32 with Brad Kolar. A look at the direct implications of neuroscience for leadership.
  • Owen's TED Talk, available here on YouTube.

Chat with PMeLa

You can chat directly with PMeLa, the podcast's AI persona, to get episode recommendations and answers to your project management and leadership questions. Visit PeopleAndProjectsPodcast.com/PMeLa to chat with her.

Join Us for LEAD52

I know you want to be a more confident leader–that's why you listen to this podcast. LEAD52 is a global community of people like you who are committed to transforming their ability to lead and deliver. It's 52 weeks of leadership learning, delivered right to your inbox, taking less than 5 minutes a week. And it's all for free. Learn more and sign up at GetLEAD52.com. Thanks!

Thank you for joining me for this episode of The People and Projects Podcast!

Talent Triangle: Power Skills

Topics: Leadership, Belief, Persuasion, Influence, Uncertainty, Change Management, Growth Mindset, Psychological Reactance, Buy-In, Artificial Intelligence, Resilience, Project Management

The following music was used for this episode:

Music: The Fantastical Ferret by Tim Kulig
License (CC BY 4.0): https://filmmusic.io/standard-license

Music: Funny by Frank Schroeter
License (CC BY 4.0): https://filmmusic.io/standard-license





Download audio: https://traffic.libsyn.com/secure/peopleandprojectspodcast/519-OwenFitzpatrick.mp3?dest-id=107017
Read the whole story
alvinashcraft
17 hours ago
reply
Pennsylvania, USA
Share this story
Delete
Next Page of Stories