Pod Certificate / Cluster Trust Bundles Blog Post

Kubernetes brings a wealth of features that make it easy to run your production workloads securely and reliably. While aspects like scheduling, health checks and resource limits are probably at the front of your mind, one other important feature of Kubernetes is production identity — how your workload can authenticate to other systems in order to do its job.

Up until now, the primary production identity mechanism built into Kubernetes has been service account JWTs (JSON Web Tokens). These are cryptographically-signed tokens, issued by the control plane of your cluster, that let anyone in the world understand who is calling when your workload uses them.

In Kubernetes 1.37, the foundations of a new built-in production identity technology have gone GA. Pod Certificates (and the closely-associated Cluster Trust Bundles) build X.509 certificate issuance for TLS and mTLS directly into core Kubernetes.

Why?

Service account JWTs have a lot going for them:

• They are built directly into Kubelet, and work pretty magically. They are written to your workload container’s filesystem before your workload starts up, and automatically kept up to date.
• The issuance system follows least-privilege principles; the node restriction admission plugin ensures that tokens can only be requested by the Kubelet that is actually currently running your pod.
• They can be federated, allowing you to use them to authenticate to other systems outside of Kubernetes. Service account JWTs underpin the pod-to-cloud authentication store for all of the largest cloud providers, and have widespread support across many additional services and software packages. If it can understand JWTs, you can authenticate to it with a service account token.

However, service account JWTs have one big downside — they are bearer tokens. With bearer tokens, if you have the token, then you are the identity asserted by the token. And since you necessarily have to hand copies of the JWT to all your peers in order to authenticate to them, they can be you, too.

There are partial mitigations for this, and service account tokens make use of them (time-, object-, and audience-binding), but none are complete defences.

A solution to this problem lies in proof-of-possession credentials, where you don’t send your entire credential to your peer, but only a proof that you possess the credential. In practice, these schemes are always built on asymmetric cryptographic signatures (RSA, ECDSA, and friends).

There are few different standard approaches, such as request signing (AWS SigV4, JWT DPoP, RFC 9421), but the most widely-deployed and understood solution is X.509 certificates, as used in TLS. In TLS, your credential is split into two pieces

• A private key, which for maximum security should be generated within your workload (or within a hardware security module), and never leave.
• A certificate, which is a description of your identity and public key, signed by a Certificate Authority.

The goal of Pod Certificates is to make using X.509 certificates from your Kubernetes workload just as easy as using service account JWTs, while maintaining Kubernetes’ high security bar. I think we’ve hit this target.

As I’ll cover in the architecture and example sections below, there are many similarities between the design of service account JWT issuance and Pod Certificates. One significant place they diverge, however, is that Pod Certificates is a much more flexible mechanism. Kubernetes only offers one flavor of service account JWTs, with standardized claims.

The X.509 ecosystem is significantly more varied than the JWT ecosystem, and X.509 certificates used for different purposes contain different extensions and information. For this reason, Pod Certificates has common machinery built into Kubelet, but offers a pluggable interface so that many different types of certificates can be issued within a single cluster, at the same time.

In the fullness of time, I expect Kubernetes to offer at least two built-in certificate providers:

• One that issues server TLS certificates for the DNS names used by Kubernetes services.
• One that offers SPIFFE client certificates, filling the same role that service account JWTs fill today.

In the remainder of this article, I’ll take you through the overall architecture of a Kubernetes workload using Pod Certificates, as well as give you an example of installing and using a real (toy) Pod Certificates signer controller.

Architecture

When you use Pod Certificates and Cluster Trust Bundles, there are the following major components:

• Your application, which requests certificates in its pod spec, and reads the keys, certificates and trust bundles from the container filesystem to use for (m)TLS.
• Kubelet, which issues PodCertificateRequest objects and reads ClusterTrustBundle objects on behalf of your workload.
• The signer controller, which answers PodCertificateRequests and publishes ClusterTrustBundles.

The best way to get a sense of what these components each do is to follow the issuance process chronologically:

1. Once your application pod is scheduled to a node, Kubelet identifies all of the podCertificate and clusterTrustBundle projected volumes sources in its spec.
2. For each podCertificate source:
   1. Kubelet generates a new private key according to the keyType field.
   2. Kubelet creates a PodCertificateRequest addressed to the signer named in the source.
   3. The signer controller sees the PodCertificateRequest and decides whether or not to issue the certificate.
   4. The signer controller issues the certificate by filling out the status.certificateChain field.
   5. The signer controller also fills out the status.beginRefreshAt field to instruct Kubelet when it should begin trying to refresh the certificate.
      certificate to the container filesystem.
3. For each clusterTrustBundle source:
   6) Kubelet retrieves the issued certificate, and writes the private key and
   1. Kubelet collects all the ClusterTrustBundles that match the signer name
   2. Kubelet unifies all of the certificates from all matching ClusterTrustBundles, and (stably) reorders them (to prevent applications from accidentally depending on a particular ordering).
      and label selectors in the source.
   3. Kubelet writes the certificates to the file path named in the source.
      and trust anchors from the filesystem.
4. Your application pod starts up, and the application reads keys, certificates,
5. Kubelet periodically updates the files from clusterTrustBundle sources as the contents of the selected ClusterTrustBundles changes. The application must pick up the changes using inotify or polling.
6. As each certificate’s beginRefreshAt time passes, Kubelet repeats the process in step 2 to refresh the certificates, and write the update private keys and certificate chains to the filesystem. As in step 5, the application must pick up changes using inotify or polling.

Some key takeaways:

• Automatic rotation is built in. Applications must properly handle it. Any signers eventually shipped in core Kubernetes will issue certificates with a max lifetime of 24 hours. The maximum lifetime allowed for other signers is 91 days.
• To make automatic rotation support as simple as possible, Kubelet supports writing the private key and certificate chain to a single file (a credential bundle) This allows the application to simply subscribe to inotify events for (or poll) the single file, read the contents, and use them. Kubelet does support writing the private key and certificate chain to separate files, but then the application needs to carefully manage the potential race conditions of reading the files mid-rotation.
• Wherever possible, security checks are built into kube-apiserver, rather than burdening signer or application developers. As an example, the built-in node restriction admission plugin enforces node isolation, ensuring that one compromised node cannot spread access by requesting certificates for pods that aren’t scheduled to it.

Try it out

Because the Kubernetes project does not yet ship any Pod Certificate signers in core, in order to try these features out, you will need to install a third-party signer into your cluster. To make this easier, I have written Tinycert, which you can install into your cluster (or a Kind cluster).

Tinycert is not a full production solution, but it’s a good starting point for experimenting with Pod Certificates, as well as a base for creating your own signers.

Tinycert provides:

• The ahmedtd.github.io/tinycert-service signer, which issues certificates with DNS SANs for all of the Kubernetes Services your Pod is part of.
• The ahmedtd.github.io/tinycert-spiffe signer, which issues SPIFFE-compatible certificates that identify the namespace and service account of your Pod. These can be used as both client and (with effort) server certificates.
• A Go library, github.com/ahmedtd/tinycert/lib/spiffefsd to help your applications load SPIFFE certificates and trust bundles from a SPIFFE Filesystem Delivery (Draft Standard) folder, as well as configure the Go TLS library for proper client and server authentication.
• An example of a SPIFFE client and server application communicating using mutual TLS and SPIFFE certificates.

What next?

• Take a look at the documentation for Pod Certificates and Cluster Trust Bundles.
• Review and offer feedback on the SPIFFE Filesystem Delivery draft standard, which aims to make it as easy as possible to use SPIFFE certificates directly on native Kubernetes.
• Participate in Kubernetes SIG Auth to help shape the future of signers that are built directly in to core Kubernetes.
• Try building your own signer based on Tinycert.

Happy hacking!

The 2002 release of the original .NET Framework shipped ADO.NET and System.Data.SqlClient, the driver for SQL Server that gave .NET applications a standard connection approach. Over the years, updates, enhancements, and patches solidified these BCL classes as an integral dependency for millions of line-of-business applications that fueled innovation, business, and the world economy.

Then in 2019, 17 years later, Microsoft introduced Microsoft.Data.SqlClient, a new .NET driver for SQL Server that was better in almost every way. Side-stepping the calcification all software inherits over time, but maintaining Microsoft’s critical commitment to backward compatibility, it replaced System.Data.SqlClient with an elegant new code base offering new features, security, and capabilities for the modern data-driven application.

Based on telemetry, a startling number of applications haven’t read the memo and updated. So, let me take a moment to talk about an incredible feature of SqlClient we previewed in 2021: Configurable Retry Logic for SqlConnection and SqlCommand. This native implementation of resiliency side-steps the need for third-party libraries like Polly, providing incredible handling of transient availability right out of the box.

Configurable Retry Logic

Since reaching General Availability in SqlClient 4.0, configurable retry logic in Microsoft.Data.SqlClient has provided a practical answer to a simple reality: sometimes, stuff happens. The best hardware, the best software, and the best networks still experience hiccups. A brief outage, a dropped connection, or even someone tripping over a power cable can make the best-designed architecture fail for just a second.

Retry logic is often the best answer for transient failures. It’s in the same category as closing and reopening the app, hitting refresh, or just turning something off and back on again. A simple retry is often all it takes to handle the intermittency of the real world. Its simplicity is its beauty; it doesn’t require additional bandwidth, redundancy, or a dead-letter queue. A well-defined retry policy can cover a myriad of troubles with just the right finesse.

Top features

Of course, every developer would expect to see the basics: fixed, incremental, and exponential retry intervals; configurable retry counts and delays; and support for both SqlConnection and SqlCommand. But SqlClient goes further with SQL-aware transient error detection, customizable transient error lists, command filtering, retry event notifications, and policies that can be configured in code or configuration. Because the retry happens inside the driver, it understands SQL Server in ways a general-purpose retry library simply cannot.

In its most basic usage, the syntax is simple:

var options = new SqlRetryLogicOption
{
    NumberOfTries = 5,
    DeltaTime = TimeSpan.FromSeconds(1),
    MaxTimeInterval = TimeSpan.FromSeconds(20)
};

var retryProvider =
    SqlConfigurableRetryFactory.CreateExponentialRetryProvider(options);

using var connection = new SqlConnection(connectionString)
{
    RetryLogicProvider = retryProvider
};

await connection.OpenAsync();

That’s it. If OpenAsync() encounters one of SqlClient’s recognized transient errors, it retries automatically using exponential backoff with built-in jitter. NumberOfTries = 5 means one initial attempt plus up to four retries. Also,  BaselineTransientErrors now exposes the built-in transient-error list, making it easier to extend SQL-aware retry behavior.

Better than Polly?

Fundamentally, configurable retry in SqlClient solves the same retry problem as Polly, but there are key areas where being inside the SQL driver gives it an advantage. SqlClient already understands SQL Server transient errors, knows whether it is retrying a connection or command, avoids retrying commands inside active transactions, and exposes SQL-specific retry events and configuration.

What’s more, SqlClient’s retry provider is reusable. Define the policy once, then assign it directly to SqlConnection or SqlCommand without wrapping every database call in a separate resilience pipeline. The retry stays close to the failure, where the driver has the most context to decide what should happen next.

Getting Started

Start with an upgrade. If you are still using System.Data.SqlClient then its time to upgrade to Microsoft.Data.SqlClient and take advantage of decades of improvements and scores of enhancements like configurable retry. Now in version 7.x, Microsoft.Data.SqlClient is typically a drop-in replacement. In some cases, it can take a little well-deserved refactoring to get going, especially if new features are your motivation.

The post Try the new SqlClient and Retry connections natively appeared first on Azure SQL Dev Corner.

What is visual storytelling? Discover 10 essential visual storytelling techniques writers can use to create emotion, tension, and unforgettable characters in fiction.

Each is supported by examples that will help you understand the technique. It will inspire you to build radiant images, unforgettable characters, and breath-taking tension.

What Is Visual Storytelling?

See, Feel, Write

In modern novels and short stories, there is a lot of emphasis on the ‘visual’ in modern storytelling – visual fiction that holds the same kinaesthetic quality of cinema.

One could theorise that readers ‘see’ stories first – striking, moving pictures in the imagination that come alive in in the mind’s eye.

Or we could argue that we live in a world where people relate more to visual stimuli. Do readers expect the same visual experience when they pick up a novel or short story?

We could possibly debate the question, but the truth is that images bring stories and characters to life. I would go so far as to say that all stories are primarily visual. In fact, writers have used figurative language for centuries.

Visual storytelling helps you show and not tell.

If you can show the world of your characters, the story becomes relatable and you create empathy in the reader.

The story world that you have created becomes realistic and believable to the reader. Similes, metaphors, and other imagery can make the story less complex and more fascinating.

Let us look at 10 techniques.

10 Essential Visual Storytelling Techniques Every Writer Should Know

1 | Think Like A Screenwriter

Pick up just about any novel from today’s bestseller charts and you’ll find that most writers seem to be using their novels as calling cards for Hollywood, or prose auditions for a new Netflix series.

In fact, some novels even bring in scriptwriting techniques. In JP Delaney’s psychological thriller Believe Me (2018), the author sporadically uses a script format to show how the protagonist, an unbalanced actress, sees the scene from detached viewpoint:

INT.DELTON HOTEL BAR, W. 44 TH ST. NEW YORK – NIGHT
Already I’m getting to my feet, pulling my bag onto my shoulder. Defusing the drama.
ME
Sorry – I hadn’t realised. I’ll find somewhere else.

Of course, this kind of device works only if it suits the character and the story. Otherwise, it really is just a gimmick.

2 | See Like A Poet

For this technique, we turn to the poet for inspiration. Let us look at the imagery in Ezra Pound’s short poem, ‘In A Station of the Metro’ (1913):

The apparition of these faces in the crowd:
Petals on a wet, black bough.

We can almost see the pale, indistinct faces (‘apparition’) of the passengers in the busy Paris underground train station. The image of a ‘bough’ as a long branch creates in our imagination the line of the platform or the interior of the train in our imagination.

We also see this sense of poetic imagery in Alice Hoffman’s novel, Here On Earth (1997): The sky is already purple; the first few stars have appeared, suddenly, as if someone had thrown a handful of silver across the edge of the world.

We pick up the sense of movement in the words ‘suddenly’ and ‘thrown’ and we see the colour contrast of the silver stars in the purple twilight. Don’t you love the way ‘across the edge’ creates a visual tilting or slanting sensation to the paragraph?

From the two short examples, we can see how writing can be richer and more visceral when we play up the visual elements. Readers are given both a mental and an emotional picture of the scene.

3 | Paint With Words

When you see a beautiful painting, something that is so vivid and evocative, your instinct may be to touch it to know if it’s real. Well, it’s the same with good writing – your readers shouldn’t believe that it isn’t real.

As storytellers, we are painting stories with words. We want readers to see a story and we want them to feel a story and the two are powerfully interconnected.

We do this through giving the reader visual cues, yes, but also in the way we structure our sentences and themes to create the desired effect.

Visual writing should include the other senses. Think about what your characters can taste, touch, hear, and smell. Consider how you can create the sensation of movement.

Let us look at some examples.

1. His insistent warm fingers delved beneath slippery cool silk.
2. The roar of jet engines broke the silence.
3. Fizzy cola burned her throat with sweetness; it made her eyes water.
4. The room smelled of dead roses and fresh potpourri polish.
5. Her running shoes smacked rhythmically against the pavement, arms efficient pistons at her side, while beads of sweat broke from her wet face and flew back into the chill air.

Top Tip: Buy our Visual Storytelling Workbook

4 | Create A Cinematic Tone

Cinematic writing is close-knit and connected writing. In short, it is writing that isn’t simply visually interesting, but that replicates the experience of watching a movie.

The imagery carefully combines many elements and techniques to create a singular and unified experience for the reader.

Cinematographers understand the importance of lighting in a production. The way you light a film creates a certain tone. If we can understand the way light and shadows works in composing a story, we can use this in our writing to great effect.

Here are extracts from a short story, ‘41’, I wrote in 2014: White light in clear lines cut the hard wood floor under his broad, naked feet.

And later: The smeared bright colours of the day mocked him after the erotic darkness of the club.

Once I’d created the contrast between light and dark, I went a step further and played out this tonal composition in the character’s state of mind: Sometimes he thought someone was following him. Other times the world around him seemed to retreat and he was left in isolated silence, in an abandoned city and on desolate beaches ringed by bright, blue seas.

5 | Cluster Images Together

As we’ve seen, visual metaphors and similes can build images that the reader can relate to.

A multiplicity of images, image clusters or word chains, are groupings that speak to each other or create a clever juxtaposition. Used together, the central images will elicit a certain emotion or mood in certain scenes or an entire short story or novel. More importantly, the images will keep themes and characters linked throughout the story.

The repetition of the same or similar images in a scene or story will trigger the same sensation in the reader and help you, as a writer, to emphasise certain themes.

The images could include animals or birds (for example, use dogs to show loyalty or companionship), symbols (the loss of a wedding ring), art (a rare painting), or even other images themselves (old photographs, home movies, etc.).

6 | Visualise A Unique Point Of View

As a writer, you could limit your character’s point of view to create a visual ‘edge’.

For example, imagine you are writing a scene where a teenager is recovering from a hangover on a sofa while his father lectures him on the dangers of alcohol.

He is too tired to move much, so from his limited point of view he can only see things at eye-level. His father’s belly pressing against his polo shirt, the hairs on his father’s knuckles, the Spaniel curled up at his bare feet.

Perhaps, when his head is very sore, he places a washcloth over his eyes, and he can only hear what his father is saying from the cool darkness.

Similarly, you can use this visual technique to highlight sound in a story. Imagine a scene where a young woman is enduring the endless gossip of group of older women in the stifling summer room of a grand home.

Slowly, she becomes aware of a bee trapped in the curtains, beating against the window.  The soft drone of the bee becomes a focal and auditory point for this character as the chatter of the women fades into the background.

7 |Deliver Tension Visually

As writers, we can also use visual storytelling to bring a key scene to life and to release a build-up of tension.

In the novel The Face of Trespass (1974), suspense author Ruth Rendell manages this superbly: He began to walk towards her. Before he was halfway down the path, before he could fetch a word from his dry throat, the thicket of bracken split open. It burst with a crack like tearing sacking and the big golden dog leapt upon him, the violence of her embrace softened by the wet warmth of her tongue and the rapture in her kind eyes.

The appearance of the dog is a pivotal and powerful moment in the book and sets about a major reversal for the main character.

When we look at the scene, it is focused on movement. The focus is dramatic: it has sound, colour, tension, and a strong sense of emotion.

We see this sense of violent movement in the first line of Jack of Spades (2015), a short novel by Joyce Carol Oates: Out of the air, the axe.

When we read this line, we almost want to physically duck out of the way of the weapon. The line seems to come out of nowhere! With just six words, the author leaves us tense and fearful. She has created this feeling through visceral, visual storytelling.

And later, she adds in detail about the brutal attack: A fleeting glimpse of the assailant’s stubby fingers and dead-white ropey-muscled arms inside the flimsy sleeves of nightwear.

We can note, from the visual details she provides, the swiftness of the scene (‘fleeting glimpse’) and the cadaverous power of the assailant (‘dead-white ropey-muscled arms’).

8 | Follow The Main Character’s Eye

When we are deeply attached to a singular character, we tend to extract more from visual techniques.

If we go inside the character and see other characters from his eyes, our stories become stronger and more reliable. In essence, we filter the story through the lens of the primary character’s experiences and emotions.

Here is an extract from Forbidden Colours (1951) by Yukio Mishima.

The young man turned once again and glanced at the old man. Perhaps it was the effect of the summer sun shining across his eyelashes, but his eyes were quite dark.
Shunsuké wondered why the youth, who had shone so resplendently earlier in his nakedness, had lost his air of happiness, if nothing more. The youth took another path. It was going to be difficult to keep up with him.

The viewpoint character here is an old writer, almost at the end of his life. We sense his obsession with youth and beauty through the way he looks at the handsome young man who has stolen the writer’s young mistress.

Note, too, how the ‘letterbox’ focus draws our attention to the young man’s eyes.

9 | Bring Setting To Life In Pictures

Setting is integral to good storytelling. It creates atmosphere and paints a picture of the landscape in which the characters found themselves.

Let us look at a scene from the novel, Life Sentences (2005) by Alice Blanchard.

He dropped her off at a small, ugly motel in the middle of West Los Angeles. A low-grade fear was making her ill. The sky was deep cobalt, and the closer you looked, the more stars you could see.
She paid the driver, who tipped his hat and sped off. Then she dragged her luggage across the asphalt toward the manager’s office. The middle-aged manager had a face like a tight ball. His mouth was slightly open, and he stared at the colour TV on his desk. A ball game was playing.
‘Daisy Hubbard,’ she said. ‘I made a reservation.’

The scene not only shows us the shabbiness of the motel, but it mirrors the emotional exhaustion of the character.

Except for a glimpse of the stars, the descriptions are bland, urban and one-dimensional. Throughout the descriptions, we sense that Daisy is not excited to be here.

Top Tip: Buy the Visual Storytelling Workbook

10| Frame A Scene Like A Camera

A movie camera follows its subjects as a silent, technical observer and, as such, creates a detached point of view. The camera is merely a tool to be manipulated. It can stay in the background, or it can track in for a close-up and, sometimes, even for an extreme close-up – but it doesn’t provide judgement.

In his book, Characters & Viewpoint (1988), Orson Scott Card says cinematic narration is cool and distant in that it ‘gives no attitude, except as it is revealed by facial expressions, gestures, pauses, words.’

I used this technique in an experimental short story I wrote, ‘The Fischers’ (2020):

The chair reclines, a white replica Le Corbusier. A long body follows the curve of the chair: toffee-coloured corduroy trousers, black roll-neck sweater. Dr. Dominic Fischer lies back and touches the edges of the black VR goggles that cover his eyes. He waves his arm as if he is holding an invisible conductor’s baton. The symphony plays through the headset. He moves his head like a blind man. The light comes in between the bristles of his beard, the grey stands out like tiny iron filings.

I wanted to capture the emotional isolation of a family in lockdown, the ‘detached’ camera point of view helped me achieve this effect.

The Last Word

Visual storytelling helps writers create scenes readers can see, feel, and remember. By using these techniques, you can add emotion, atmosphere, and depth to every story you write.

Top Tip: If you want to learn how to write a screenplay, sign up for our online course: The Script

by Anthony Ehlers

More Posts From Anthony:

1. 7 Ways ‘What If?’ Helps Writers Create Better Stories
2. 5 Secret Tricks To Strengthen Your Writing
3. 5 Famous Writers On Writing A Villain
4. 7 Deadly Rules For Creating A Villain
5. How To Write The Tragic Love Story – A 10-Step Formula
6. 6 Fascinating Fictional Character Types
7. 7 Spine-Chilling Tips For Writing An Unforgettable Horror Story
8. The Moment Of Change In Fiction
9. What’s Your Story’s Tone?
10. What Is Freewriting & How Do I Use It?

Top Tip: Sign up for our free daily writing links.

The post What Is Visual Storytelling? 10 Essential Techniques Every Writer Should Know appeared first on Writers Write.

Happy Friday. I’m at inbox zero, and hoping for a fun weekend with the family. And might write a blog post if the stars align.

[blog] Teamwork: When AI Becomes a Research Partner. This is outstanding. I liked the deep dive into the patterns this tool applies to agent teams. And the results are impressive.

[blog] Don’t Waste Your Career Getting Comfortable. Tons of good advice in here, much of which I follow diligently myself. Get comfortable being uncomfortable.

[blog] Dynamic capacity management for AI infrastructure. Now isn’t the time to commit to a ton of fixed infrastructure. Who knows what you’ll need in six months, let alone two years. Here’s a good post on embracing a more fluid foundation.

[article] AI Transformation Requires Redesigning Work, Not Cutting Roles. There’s good advice in here. Especially the examples of where companies go down the wrong path.

[blog] Simplify your resilience testing strategy with Fault Injection Testing. How you feeling about your disaster recovery plans? Tested them in a while? What about when you’ve got many workloads in a public cloud? Here’s a new way to simulate meaningful failures to see how your automated resilience handles it.

[blog] Friday Forward – Expert Silence. We’re all wrong—yes, even “experts”—all the time. It’s completely normal. But be smart enough to acknowledge new facts when more information comes in.

[blog] Designing Websites for People and AI Agents with WebMCP. I’d encourage you to be aware of WebMCP, even if you don’t want to go deep on it.

[blog] Unlock Antigravity 2.0: Logins, Plans, Cost, and Quota. Getting started with new tools can be intimidating, especially when you have a handful of initial choices to make. Alexis breaks this down in a very understandable way.

[blog] Introducing Agent Native Design: An Open-Source Figma Alternative. Build interactive prototypes that you can export anywhere.

Want to get this update sent to you every day? Subscribe to my RSS feed or subscribe via email below: