53 images that users uploaded into OpenAI models were included in training data — and then AI agents in an OpenAI research environment posted those 53 images on public image hosting sites. While posted as links that weren't publicly listed, "the images could still be discovered even if the links were not publicly listed," reports TechCrunch: OpenAI said it was working with the hosting providers to remove this content, though some of it is apparently still online. OpenAI said it could not notify the affected users because "our technical approach and privacy policy" prevent it from "reassociating" the images with the original providers, but declined to say how the lab determined whether the images were provided by users. The news came in a post collecting public statements from the lab's ongoing review of incidents in which its models escaped the company's scrutiny, accessed the open internet, and misbehaved in various ways. OpenAI said it would continue disclosing anonymized accounts of incidents like these, and said it had contacted dozens of victims, including governments, universities, public agencies, to notify them of the agents' activities. Friday night news also broke that OpenAI's agents also tried unsuccessfully to infiltrate the U.S. Department of Education's site this summer "without the company's knowledge," reports Politico. And OpenAI's models also accessed the website of the U.S. Commerce Department using credentials found in online code repositories, according to the article. OpenAI confirmed the incident Friday, "saying its technology did not manage to access information that was not already public or change government data and systems." The article adds that OpenAI's models also accessed the web site for America's Securities and Exchange Commission: One senior federal IT official said the government still did not have a clear understanding of what happened across the three agencies. "We still don't know what public data was accessed and how it was accessed, because OpenAI has not shared specific technical details with us yet," said the official, who was granted anonymity because they were not authorized to speak publicly about it. OpenAI discovered the Commerce and SEC incidents as part of its ongoing review of incidents where its technology has acted in unintended or "misaligned" ways. About the models posting user-uploaded images, TechCrunch's article notes that OpenAI stressed "that its enterprise users are automatically opted out of having their interactions used to train future models; however, consumer users are opted in unless they affirmatively choose not to share their data." (As OpenAI's announcement describes it, some of their agents' training data "contains content from, or derived from, training-eligible user interactions.") Posting the images is "not an appropriate use of this data," OpenAI acknowledged, adding that it happened before new safeguards added after the Hugging Face incident. This latest incident appears as an update on a new OpenAI page that "brings together our reports and updates on the Hugging Face incident, related research and public presentations, additional activity we have identified, what we have learned about the role of model misalignment, and measures we're taking to strengthen our systems." (It also notes that there's now a name for models posting on third party sites — "agent spam" — which they consider distinct from cybersecurity, though "we need to address both.") "As part of our response to our ongoing investigation, we have improved our training and evaluation processes, including building safety cases, securing and red-teaming our systems to prevent the model from exfiltrating data, and implemented additional monitoring. We are continuing to review agent activity in research and evaluation runs, working backward month by month starting from the Hugging Face incident."

Read more of this story at Slashdot.

1. Where CoreDNS fits in AKS

A pod can resolve another Kubernetes Service but fail to resolve a database name outside the cluster. CoreDNS can be running normally in both cases.

That distinction matters because several problems can look like the same DNS outage:

• A cluster Service name fails. The namespace, Service record, or path from the pod to cluster DNS may be wrong.
• A name outside the cluster fails. CoreDNS may be reachable, but the upstream server or its network path may not be.
• A lookup eventually succeeds, but the application fails. The application may have stopped waiting before DNS returned an answer.

CoreDNS is the default cluster DNS service in AKS. It runs as pods in kube-system, rather than as code inside your application or as a server hosted in the AKS control plane.

It has two important jobs:

1. Kubernetes service discovery: Answer names associated with Kubernetes resources, so workloads do not have to track changing addresses.
2. Upstream forwarding: Send queries that require another DNS server to the configured upstream resolvers.

Microsoft's AKS DNS concepts guide explains this architecture. The key operational lesson is simple: CoreDNS is part of the resolution path, but it is not the owner of every DNS answer.

Who this is for: Platform engineers, application developers, and operations teams who want a practical understanding of CoreDNS in AKS.

2. How a pod gets a DNS answer

For a typical pod using dnsPolicy: ClusterFirst, without a node-local DNS layer, the path looks like this:

Application in a pod -> cluster DNS Service: kube-dns -> CoreDNS pod -> Kubernetes Service information: answer a cluster Service name -> upstream DNS server: forward a name handled outside the cluster -> DNS answer returns to the application

The Service is still named kube-dns even when CoreDNS handles its traffic. That name does not mean the cluster is running the older kube-dns implementation.

Kubernetes configures the pod's resolver settings, including its nameserver and search domains. For Linux containers, these are visible in /etc/resolv.conf.

Cluster Service names

Consider a regular ClusterIP Service named orders in the checkout namespace. With the cluster domain cluster.local, its full name is:

orders.checkout.svc.cluster.local

CoreDNS's kubernetes plugin uses Kubernetes resource information to answer that name with the Service's cluster IP. It does not need to ask a corporate or public DNS server for that Service record.

The namespace matters. A pod in checkout may use the short name orders; a pod in another namespace normally needs orders.checkout or the full name. See Kubernetes DNS for Services and Pods for the naming rules.

Names handled by another DNS server

For a name outside its configured cluster zones, CoreDNS can use the forward plugin to contact another DNS server. That may be an Azure-provided resolver or a custom resolver, depending on the cluster's network and DNS configuration.

Do not assume that every AKS cluster forwards to the same address or that two configured addresses always mean primary and backup. Inspect the effective CoreDNS configuration and the upstream resolver settings it references.

Caching can also change the path. A valid cached answer may avoid an upstream query entirely.

LocalDNS can change the first hop

If AKS LocalDNS is enabled, the pod can contact a DNS proxy and cache on its node first. Cluster-domain queries are forwarded to CoreDNS, while other queries can go through CoreDNS or directly to an upstream, depending on the DNS policy and LocalDNS configuration.

The diagram above is therefore a starting point, not a universal packet trace. Check whether LocalDNS is active before deciding which resolver to inspect. The AKS DNS concepts guide describes these alternate paths.

3. What AKS manages and what you can customize

AKS manages the CoreDNS deployment and its main configuration. The Corefile, stored in the coredns ConfigMap, tells CoreDNS which plugins and forwarding rules to use.

Three plugin responsibilities are useful to recognize:

An upstream failure affects queries that need that upstream. It does not automatically mean that normal Kubernetes Service records have become unavailable.

For supported customization, AKS provides the coredns-custom ConfigMap. Microsoft documents custom entries with names ending in .server or .override, including domain-specific forwarding examples.

Do not edit the managed main Corefile as a shortcut. A CoreDNS option being valid does not mean that every way of applying it is supported in AKS.

For example, forwarding one private domain to designated resolvers is different from replacing the managed root forwarding behavior for the entire cluster. Follow the AKS CoreDNS customization guidance for the intended change.

Also check the deployed image. The tests discussed here used CoreDNS 1.13.1. A setting shown in documentation for a newer CoreDNS release may not exist in that image.

4. How CoreDNS selects and checks upstream servers

Once a query reaches the forward plugin, three separate decisions matter: which server to try, when to try another, and whether a server should be skipped on later queries.

Selection policy chooses the starting server

For CoreDNS 1.13.1:

• random chooses among eligible upstreams. It is the default when no policy is specified.
• round_robin rotates the initial choice.
• sequential tries upstreams in their configured order, preferring the first one not marked unhealthy.

Here is an illustrative isolated-lab fragment, not a change to apply to AKS-managed CoreDNS. The addresses are placeholders:

forward . <primary-dns-ip> <backup-dns-ip> { policy sequential failover SERVFAIL }

policy sequential controls the order. failover SERVFAIL permits another upstream attempt when the response says the server could not complete the lookup. Those are different settings for different decisions.

A missing reply and an error reply are different

If packets are silently dropped, CoreDNS receives no answer and must wait for a timeout. If the server returns SERVFAIL, CoreDNS has received a DNS error response.

By default, the forward plugin returns that SERVFAIL to the caller instead of trying another server. An explicit failover rule can change the handling of selected response codes.

Protocol matters too. User Datagram Protocol (UDP) does not open a connection before sending the query. Transmission Control Protocol (TCP) does.

In this release, the read timeout is two seconds. Opening a TCP connection has a separate timeout that starts at 30 seconds and can decrease toward one second based on recent connection times. There is no single timeout that describes every failure.

These behaviors are documented in the CoreDNS 1.13.1 forward reference.

Health checks change later selections

The forward plugin starts upstream health checking after a network error. It does not continuously poll every untouched healthy upstream before any failure occurs.

The documented default check interval is 0.5 seconds. That is not a promise that failover finishes in half a second: the triggering query may already be waiting for a timeout.

max_fails controls how many failed health checks mark an upstream unhealthy. Its default is 2; the application-impact lab used 1 to make the change easier to observe.

A DNS error response can still prove network reachability. Health checking therefore does not prove that an upstream can successfully resolve every name your application needs.

Each CoreDNS process keeps its own upstream health state. Testing one replica does not establish the state of every replica, and restarting a resolver resets that state. The health-check guide and evidence distinguish source-backed behavior from measured, partial, and inconclusive cases.

5. What failure tests on AKS showed

The run used Kubernetes v1.35.7 and image mcr.microsoft.com/oss/v2/kubernetes/coredns:v1.13.1-20.

Tests ran against separate CoreDNS resolvers and controlled upstream servers in an isolated namespace. They did not inject faults into AKS-managed CoreDNS or validate every production DNS path, including LocalDNS.

The test resolver used sequential selection and max_fails 1. Query times below are in milliseconds (ms).

Five additional independent UDP failure cycles reached the backup in 2,000-2,004 ms on the first query. Their later queries took 0-4 ms.

The result summary and raw query output retain the details.

These are observations for one image and controlled failure conditions, not guaranteed AKS timings. A displayed 0 ms reflects the tool's timing precision, not zero latency.

The practical lesson: an available CoreDNS pod, a reachable backup, and a successful application request are three different things.

6. Troubleshooting the right part of the DNS path

Start by separating cluster Service resolution from upstream resolution. Then inspect the configuration before changing it.

Prerequisites: PowerShell 7+, installed and authenticated kubectl, and permission to read CoreDNS Deployments, pods, Services, and ConfigMaps in the intended cluster. Check the context first and stop if it is not the correct cluster. The following commands are read-only.

kubectl config current-context kubectl get deployment coredns -n kube-system ` -o jsonpath='{.spec.template.spec.containers[0].image}' kubectl get pods -n kube-system -l k8s-app=kube-dns kubectl get service kube-dns -n kube-system kubectl get configmap coredns -n kube-system -o yaml

Use the findings to narrow the investigation:

• Only a cluster Service name fails: Check the Service name, namespace, and record type. Confirm that the Service exists and that the pod is using the expected cluster DNS path.
• Only names requiring an upstream fail: Check the matching forwarding rule, upstream reachability, and the upstream's ability to resolve that name.
• Only some pods fail: Compare their DNS policies, resolver settings, node-local DNS path, and network access. A successful query from another pod is useful, but not conclusive.
• The first query is slow and later queries are fast: Check upstream health state and caching. Do not assume the first result was an unrelated glitch.
• CoreDNS returns SERVFAIL: Identify whether it came from an upstream response or another failure. Do not assume it means packets were lost.

Measure DNS latency with dig's Query time. The elapsed time of kubectl exec also includes Kubernetes API communication and process startup.

Finally, look at the application. A reused connection may avoid DNS, a cached answer may hide an outage, and retries may increase load. A client can time out even if CoreDNS eventually finds an answer.

The latest suite recorded 13 passing DNS and environment checks and five blocked application-specific cases. No representative application was supplied, so caching, connection reuse, retries, application telemetry, and service-level objective (SLO) impact were not established. A DNS test alone cannot prove the user's experience.

7. A repeatable way to validate CoreDNS behavior

The CoreDNS repository folder organizes these questions into five detailed guides:

Each guide maps a question to named test cases, expected results, pass/fail criteria, and an Established evidence section. That section should tell you what happened, when it happened, and what the result does not prove.

Prerequisites for fault testing: An authorized test cluster, PowerShell 7+, authenticated kubectl, permission to create and remove namespace-local test resources, and a network implementation that enforces the test NetworkPolicies. Review the selected guide's full setup and cleanup instructions before execution.

Use an isolated test resolver. Do not change managed kube-system resources to reproduce an upstream failure. If another test namespace already exists and ownership is unclear, choose a unique namespace instead of deleting it.

The published examples include the original lab context and paths beginning with 07-CoreDNS. A GitHub clone uses coredns. Adjust these values for your environment before running a suite.

Cleanup is part of the test. In the current run, the unique test namespace was removed, all six base-lab Deployments remained Available, both managed CoreDNS replicas remained Available, and the managed Deployment resource version was unchanged.

8. Conclusion and next steps

CoreDNS connects AKS workloads to both Kubernetes service discovery and the wider DNS environment. Understanding where a query is answered is the first step toward understanding why it failed.

Start with the pod's DNS path. Inspect the CoreDNS version and configuration. Separate Service records from upstream queries. Then test selection, timeouts, health checks, and application behavior without changing managed cluster DNS.

Understand the path before changing the policy. Validate the behavior before promising the outcome.

Try it now

Prerequisites: Git, PowerShell, network access to GitHub, and a working directory without an existing aks-stuff folder. No Azure permissions are needed to download the guides. These commands do not deploy resources or run failure tests.

git clone https://github.com/jvargh/aks-stuff.git Set-Location .\aks-stuff Get-Content .\coredns\README.md

Learn more

• CoreDNS guidance and test suites
• DNS concepts in AKS
• Supported CoreDNS customization in AKS
• Latest validation summary and recorded query output

Connect and contribute

Share a reproducible finding through the GitHub repository. Include the CoreDNS version, DNS path, test case, failure condition, client timeout, measured result, and cleanup outcome. Remove credentials, private addresses, and customer identifiers before sharing logs.

Start here: github.com/jvargh/aks-stuff/tree/main/coredns

On September 23, 2026, Microsoft announced guidance that will help Azure IoT Central customers migrate their solutions to use new features of Azure IoT Hub and Microsoft Fabric. The new approach leverages IoT Hub capabilities such as certificate management, along with Microsoft Fabric capabilities such as the operations agent, to modernize connected solutions. Existing users can continue using their Azure IoT Central solution as-is until September 20, 2029 if they don't want to take advantage of these new features right away. This is a natural step in our ongoing evolution of Azure IoT: we continue to invest deeply in the platform services that power Azure connected solutions, building on core Azure constructs such as RBAC and ARM resource models, while integrating AI capabilities across Azure and Fabric.  If you build on IoT Central today, keep reading to understand what is changing and how to transition to a modern Azure IoT architecture.

A platform that keeps getting better

Azure IoT Central pioneered a managed application experience that made easy to get started with IoT.  Since then, Azure IoT has grown significantly. Azure IoT Hub and Device Provisioning Service (DPS) provide the foundation for secure, high-scale device connectivity and provisioning, while new investments add richer device lifecycle capabilities such as certificate management and deeper integration with Azure Device Registry. Azure Device Registry brings devices into the Azure management plane as ARM resources, enabling more consistent governance across connected environments. 

These investments also align with Microsoft's broader vision for connected operations, where operational technology, enterprise data, analytics and AI come together on a unified platform.  As discussed in our recent work on connected operations and industrial AI, Azure IoT, Azure Device Registry, and Microsoft Fabric Real-Time Intelligence provide the foundation for connecting device telemetry with operational and business context. This enables organizations to move beyond monitoring toward real-time intelligence, automation, and increasingly AI-powered operational experiences.

By concentrating our investment on a unified, Azure-native platform rather than a fixed application experience, we can innovate faster and give customers a more flexible and scalable foundation to build connected solutions tailored to their business needs.

What's changing

Azure IoT Central is evolving toward an architecture built on Azure IoT Hub and Microsoft Fabric. Your existing applications continue to function and can be managed as-is until September 20, 2029.

Key dates

The key milestones are:

• New IoT Central application creation is unavailable starting September 23, 2026.
• Existing applications continue to work as-is through September 20, 2029.
• After September 20, 2029, IoT Central applications are no longer available.

The modern Azure IoT platform

The recommended path is Azure IoT Hub with DPS and Microsoft Fabric, with Azure Device Registry where applicable. This preserves the core device connectivity and provisioning you rely on today and upgrades your analytics and dashboards to Microsoft Fabric Real-Time Intelligence.  You can continue to use your existing devices and data while benefiting from greater scalability, stronger lifecycle controls, and a modern real-time analytics experience.

The following table maps common IoT Central capabilities to their modern Azure-native targets:

Get started faster with the Fabric solution accelerator

To move your analytics forward quickly, start with the Azure IoT Solution Accelerator Workload for Microsoft Fabric Real-Time Intelligence (https://github.com/Azure-Samples/azure-iot-accelerator-workload-for-fabric-rti). Instead of rebuilding dashboards and pipelines from scratch, the accelerator deploys a working, end-to-end Fabric Real-Time Intelligence workload on top of your IoT Hub telemetry - Eventstream ingestion, an Eventhouse/KQL database, and ready-to-use real-time dashboards - so you can see your device data flowing in minutes and can then tailor it to your solution

How to get started

You can start planning your transition today:

1. Review your current IoT Central deployment, including device templates, device groups, jobs, rules, exports, dashboards, and users.
2. Evaluate the Azure IoT Hub and DPS target architecture, and Azure Device Registry where applicable.
3. Plan your migration in phases, keeping IoT Central available during the transition.
4. Follow the customer migration playbook at https://aka.ms/AzureIoTCentralMigrationPlaybook, which maps common IoT Central scenarios to Azure IoT-native services.

Migration partners

If you would like additional help, experienced Microsoft partners can support discovery, migration planning, and execution. Both partners below build on Azure IoT Hub, DPS, Azure Device Registry, and Microsoft Fabric:

• Mesh Systems builds on Azure IoT Hub and Azure IoT Operations to accelerate connected-product development and migration. Read their story in the following link: How Mesh Systems Builds on Azure IoT Hub.
• Helin delivers an Intelligent Edge Application Platform for industrial AI at the edge, built on Azure IoT Hub, DPS, Azure Device Registry, and Microsoft Fabric. Read their story in the following link: Scaling Industrial AI at the Edge with Helin and Azure IoT.

Our commitment

We are committed to helping you transition smoothly and to continuing our deep investment in Azure IoT. Thank you for building on Azure IoT - we look forward to supporting your move to a modern, Azure-native foundation for connected operations.

Help and support

• Get answers from community experts in Microsoft Q&A Microsoft Q & A for IoT Central
• If you have a support plan and need technical help, create a support request in the Azure portal.
• Review the full guidance and recommended migration partners at https://aka.ms/AzureIoTCentralEvolution

---

In general, you should be suspicious of engineers who are trying to give you advice. Even during ordinary times, this industry is so wide and changes so quickly that nobody really knows anything for sure. And we are not in ordinary times. The advent of LLMs and AI agents is the largest change to software engineering in my professional lifetime, and possibly the largest change ever. That said, here’s my advice:

• Don’t trust senior engineers who are telling you to pick political fights
• Don’t play games. Keep your head down and be helpful
• Be conscientious and try hard to actually understand what you’re working on
• Don’t panic about AI, and don’t delegate your judgement to it
• Don’t avoid AI — keep thinking!
• Don’t lose hope

Don’t trust ZIRP-era advice

Most experienced software engineers today have spent the bulk of their career in the ZIRP era. This was a time when investment money flooded the industry, driving up engineer bargaining power. Big tech companies spent a lot of money and effort trying to make their engineers happy and comfortable. If you were an engineer at one of those companies, you could expect to have a decent say in what kind of work you did, and even what kind of politics your company had. Unless you worked at a handful of unusual companies (e.g. Amazon) you could expect to be practically immune from layoffs, and to only be fired after many months (sometimes years) of low performance.

A lot of advice floating around is ZIRP-era advice: either it was written during that era, or it’s written by engineers whose habits of thought were formed during that era. This kind of advice typically tells you to take a stand (e.g. to unionize¹), to speak up more against “unethical technologies” like AI, and to insist on being given time to practice your craft the way you want. This was great advice in 2016, but it’s not good advice in 2026.

In fact, I think it’s unethical for senior engineers to give advice like this to junior engineers who are more likely to take it (because they’re young and naive) and more likely to be punished for taking it (because they lack leverage). If you’re a new engineer, don’t fall for it! Let your colleagues with more experience and political capital take risks like that.

Be friendly and conscientious

Instead, I recommend adapting to the demands of the current era of software engineering. Try to make yourself useful to your team and your manager. Be pragmatic about your actual bargaining power (fairly low, unless you’re useful enough to be irreplaceable). Keep your expectations of yourself under control: don’t spiral out in an attempt to do something astonishing, just try to be consistently helpful.

Don’t start fights. Being pleasant to work with (particularly when you don’t get your way) covers many sins. Once you’re further along in your career, you will be expected to start some fights, but this is not a tactic beginners should adopt: there are a lot of factors² that determine when and how to start fights, and getting it wrong can be costly. Just don’t risk it.

In general, you should stay out of the political game at all costs. Even quite senior engineers are political tools, not movers and shakers, and this goes double for junior engineers. Simply trying to be friendly and helpful will get you infinitely further, politically speaking, than any amount of Machiavellian game-playing. Keep your head down, stick with your management chain (not random people who try to assign you work), and you’ll be fine.

As an engineer, your main technical value is conscientiousness. You should be asking lots of questions, both of yourself and of the engineers around you. You should be actively trying to make sense of the systems you work with, instead of just assuming someone else has it covered. Software systems are complicated enough that a few weeks of careful attention will mean you know technical details that nobody else does, which is a really easy way to add value.

Don’t panic about AI

If you ought to be suspicious of most software engineering advice, the good news is that you should also be suspicious of doomsayers who predict the end of the industry. Before LLMs, people thought outsourcing would end software engineering in Western countries; before that, people thought high-level languages and low-code tools would end software engineering as a profession. Now people think AI means it’s all over. Maybe! But there are also reasons³ to think that software engineering will simply change. We’ll all find out together.

Many software engineers will tell you to avoid AI entirely. Even though agentic AI did not exist during the ZIRP era, this is still ZIRP era advice. Your company will expect you to use AI for the same reasons that builders are expected to use power tools. Pushing back hard on that as a beginning engineer is a great way to get laid off: you simply do not have the bargaining power to fight back against an industry trend this powerful.

That said, it’s really important that you don’t delegate your own judgement to AI. Don’t just trust the suggestions or approaches that your agents propose. Ask questions and substitute your own opinions where you disagree: even if they’re wrong, you’ll learn more that way. If you don’t understand something the AI is telling you, either drill down until you do or just ignore it. Under no circumstances should you pass the AI’s message on to your colleagues verbatim. In other words, don’t be a meat proxy.

I think most people become a meat proxy as a form of panic: they feel like it’s over for their own skills, and that the AI model is smarter than them, so they can’t add any value beyond simply deferring to Claude or GPT-6. I can understand why people panic. It’s a crazy time in the industry, after all. But panic almost never helps you make good decisions. Keep your head, try to remain confident that your skills are still relevant, and use the agents to inform your own understanding⁴ instead of to replace it.

Don’t lose hope

It was really nice to work in tech in the 2010s when everything was stable. But we’re not in that world anymore. This is an age of wonders and terrors. Things will go worse than we think in some ways, but better than we could imagine in others.

The doomsayers — the people who are saying it’s all over, and that there’s no hope — are almost certainly wrong. They can’t predict the future because nobody can. Technological change of this magnitude always has knock-on effects that are impossible to see coming, both positive and negative.

The nature of the job might change, but it will always be valuable to be smart, friendly and conscientious. Delegating your judgement to an AI model might feel like a relief from despair in the short term — at least now it’s the AI’s responsibility, not yours — but it’s a bad idea. Don’t give up!

---