In the past year, there’s been a pattern of supply chain attacks that target weaknesses in package repositories and CI/CD systems to quickly spread malware to hundreds of open source projects. This malware seeks to exfiltrate credentials both to broadly spread the attack, as well as for later exploitation.

We’ve written a few times about our plans for hardening the supply chain: Our plan for a more secure npm supply chain in September 2025, Strengthening supply chain security: Preparing for the next malware campaign in December 2025, and What’s coming to our GitHub Actions 2026 security roadmap in March 2026. In this post, we’re updating you on changes we’ve implemented that directly disrupt some of the most common and impactful supply chain attack techniques.

Anatomy of supply chain attacks

Supply chain attacks chain together several weaknesses, and there is no single security capability that can stop them. Addressing them takes a holistic approach, prioritizing the mitigations that break the most impactful links in the attack chain. Our teams have been studying these attacks to deploy several improvements that disrupt them and limit their impact. This is possible thanks to collaboration with the security research and developer communities.

The attacks vary in how they spread across the software ecosystem. However, most of these attacks follow similar techniques to gain initial access to a project, escalate privileges, and distribute across users and software. Improvements made to npm and GitHub Actions in the past few months have been focused on cutting off specific, common techniques and providing ways for customers to identify and respond to these attacks.

Initial compromise

Attacks start by compromising a single project, often by directly compromising a maintainer’s account or by targeting the project’s actions workflows.

• npm adds preventive account protection for high-impact accounts (June 2026): Frequently, attacks start with a phishing campaign targeting maintainers. With this change, high-impact npm accounts are now put into a read-only mode for 72 hours when they change their email or use a 2FA recovery code. This delay allows maintainers time to respond and recover the account before their account can be used to start an attack.
• Safer pull_request_target defaults for GitHub Actions checkout (June 2026): A common vulnerability in a project’s CI/CD pipelines are “pwn requests,” where a workflow triggers on pull requests from forks and then executes user-submitted and untrusted code from that fork. We changed the default behavior of actions/checkout to prevent the checkout of untrusted code from forks in commonly exploited triggers unless you explicitly opt-out (after reviewing your risk). This change and its backport to older versions cut off one of the most common vulnerable code patterns leading to code execution in GitHub Actions CI/CD workflows and initial project compromise.
• Control who and what triggers GitHub Actions workflows (June 2026): Maybe you’d prefer to opt-out of these risky action triggers altogether or limit who can trigger them. This new control lets you set enterprise, organization, or repository level policies on who is allowed to trigger workflows and what trigger types are allowed. These workflow execution policies provide a governable and customizable layer of least-privilege around Action workflows that reduce the attack surface of your CI/CD infrastructure.
• Read-only Actions cache for untrusted triggers (June 2026): After an attacker has achieved code execution in an Actions workflow, they then look to escalate to more privileged workflows (and therefore credentials) through poisoning the cache entries shared across workflows. With this change, we restrict the ability for less trusted workflows to modify the cache shared with other workflows. This directly closes a common path attackers have used to turn a vulnerability with limited impact into one that compromises highly privileged credentials used by release and publishing workflows.

Exfiltrate credentials

Once an attacker has access to a single package, they then focus on detecting and exfiltrating credentials to gain further access and use in later exploitation across ecosystems.

• npm trusted publishing now supports CircleCI (April 2026): The number one thing you can do to disrupt these attacks is to remove long-lived credentials from your CI/CD pipeline. Trusted publishing is a great way to authorize publishes to your package repository without a long-lived credential. By adding CircleCI as a trusted publishing provider, we’ve made it possible for more people to remove the credentials these attacks attempt to exfiltrate.
• Actions network firewall (In technical preview): This technical preview logs all outbound network traffic from your Action workflow runs so you can detect unusual behavior like pulling down malicious code or exfiltrating credentials to a new domain. Future work will enable network egress restrictions and policies to block these attacks before they lead to further escalation and exfiltration.

Propagating the attack

With the credentials harvested from the previous step, attackers attempt to use those credentials to distribute their malware and compromise more projects and maintainers as quickly as possible.

• Staged publishing for npm (May 2026): With staged publishing, it’s not enough to have credentials to publish a new package on npm; those packages are staged until additional approval and 2FA authentication is provided in the npm cli or on npmjs.com. This opt-in security control allows maintainers to ensure that any version of their package published has gone through this additional authorization. By decoupling the credentials used in CI/CD pipelines and automation from those that can publish to the registry, the attack chain from a CI/CD pipeline to malware distribution is cut off.
• Upcoming breaking changes for npm v12 (June 2026): To spread their malware as quickly as possible, attackers use npm install-time scripts to exfiltrate credentials instead of waiting for code to be executed by the package at runtime. With npm v12, we are rolling out a breaking change that disables these install scripts by default. Since install scripts have legitimate use within the package installation processes that several popular packages rely on, you can reenable them by approving specific scripts. Additional vectors for install-time code execution have also been blocked by disabling dependencies via git or remote URLs by default.
• Dependabot version updates introduce default package cooldown (July 2026): Attackers rely on speed, hoping a malicious release gets pulled into as many downstream projects as possible before anyone notices. Version updates through Dependabot now wait until a release has been available for at least three days before opening a pull request, giving detection signals time to surface before a malicious release reaches your project. This cooldown is on by default, and security updates still open immediately, so critical fixes are never delayed.

Identifying and responding to supply chain attacks

In parallel to hardening npm and GitHub Actions to disrupt and limit the impact of supply chain attacks, we have also been working on making features and tools available to users to identify and respond to supply chain incidents that have impacted their projects and accounts.

• Self-service credential revocation for incident response (June 2026): GitHub credentials remain an ongoing target of attackers in supply chain attacks. This feature provides self-service tooling to instantly revoke all credentials for a given user in an enterprise. This builds on the enterprise-wide credential management tools for incident response released in February and allows enterprise admins and members to quickly respond if credentials have been compromised in a supply chain attack.
• Expanded credential revocation API support (March 2026): Recent attacks have included exfiltrating credentials within publicly accessible content. To help the community respond to these attacks, we have expanded support of our credential revocation API (first introduced in April 2025 for personal access tokens) to support the revocation of GitHub OAuth and App tokens. This allows the quick and self-service revocation of these leaked credentials, regardless of where they are found, and limits the lifetime during which they can be abused.

What’s Next?

Making our products more secure by default is a priority across npm and GitHub and we are prioritizing this work to target and disrupt supply chain attacks across the open source ecosystem. We’re proud of the work we’ve shipped towards this goal over the past months. There’s more to come, but we wanted to provide an update on the progress we’ve made and make folks aware of the new capabilities available to them. Be sure to follow our changelog and blog posts as we continue to roll out improvements.

Open source software is an incredible public good that we all benefit from, and this is one of several ways GitHub is working to continue to support the security, sustainability, and continued success of open source communities and the enterprises that depend on them.

The post Disrupting supply chain attacks on npm and GitHub Actions appeared first on The GitHub Blog.

Part 3 of a series where we take public “token saver” add-ons for coding agents and run the same paired A/B benchmark against each of them. Part 1 was the caveman skill (advertised −65%, measured −8.5%). Part 2 was rtk (advertised −60–90%, measured +7.6%).

We ran 80 paired tasks to test the ponytail skill for Claude Code. Advertised: −54% code, -22% tokens, -20% cost, -27% time. Measured: −15% code, −10.3% cost and -11% time. Here’s what actually happened.

Real savings, although roughly a quarter to a half of what is advertised, it is the first tool in this series with a statistically solid cost-saving signal. We found no quality difference, though ~80 pairs can only rule out large ones. The catch: the code cut only shows up where there was room to over-build.

Why we ran this

Ponytail skill is designed to make AI agents write less code. Its core premise: a senior developer who has seen everything replaces your fifty lines with one. Ask for a date picker and instead of installing flatpickr and writing a wrapper component, it writes <input type="date"> and moves on.

Mechanically it is a ladder the model climbs before writing anything. Does this need to exist at all? Is it already in the codebase? Does the standard library do it? A native platform feature? An installed dependency? Can it be one line? Only then: write the minimum that works. The ladder runs after understanding the problem, not instead of it, and validation, error handling, security and accessibility are explicitly off the chopping block.

Here is what that looks like in practice, from our own run. Both agents were asked to export a three.js scene to a Blender-ready OBJ file; both produced a file the verifier accepted. Both wrote the same fiddly loop to expand instanced meshes, because three.js’s OBJExporter cannot handle them. The difference is everything around that loop. To rotate the scene into Blender’s orientation and write it out, the plain agent builds a wrapper object to hold the rotation and names every intermediate step:

// no skill — 10 statements to rotate the scene and write the file
const exportRoot = new THREE.Group();
exportRoot.name = 'blender_export_root';
exportRoot.rotation.x = -Math.PI / 2;
exportRoot.add(root);
exportRoot.updateMatrixWorld(true);

const exporter = new OBJExporter();
const objString = exporter.parse(exportRoot);

const outputPath = '/root/output/object.obj';
fs.mkdirSync(path.dirname(outputPath), { recursive: true });
fs.writeFileSync(outputPath, objString);


// ponytail — the same job, 5 statements
root.rotation.x = -Math.PI / 2;
root.updateMatrixWorld(true);

const obj = new OBJExporter().parse(root);
fs.mkdirSync('/root/output', { recursive: true });
fs.writeFileSync('/root/output/object.obj', obj);

Nothing was sacrificed there. Ponytail rotated the object it already had instead of building a parent to rotate it for it, and skipped an import while it was at it. Ten statements became five, both files exported the same geometry, and both scored 1.0. That is the effect working exactly as advertised — on one file, on one task.

The headline claim is −54% code, plus −22% tokens, −20% cost and −27% time. What made this one worth testing is that the claim is unusually well documented. The authors rebuilt their benchmark in response to a critique (issue #126) that their original numbers came from a chatty baseline, and they publish the honest version: a real headless Claude Code session editing a real FastAPI + React repo, scored on the git diff it leaves behind. They even document a contamination bug they found in their own harness, where a SessionStart hook was firing on every arm and secretly running ponytail in the baseline.

That is more methodological candour than most tools in this space manage. So the question here is whether the effect survives a benchmark the authors did not choose, on a stronger model, with verifier-scored quality.

Setup

Harness	Harbor 0.18 — Docker sandboxes, task verifiers, paired runs
Agent	Claude Code 2.1.201, headless, bypassPermissions, pinned in both arms
Model	claude-sonnet-5 at medium reasoning effort
Benchmark	SkillsBench, 80 paired tasks, auto-graded 0–1 with partial credit
Arm A	stock Claude Code
Arm B	ponytail v4.8.4: skill installed and its ruleset injected, byte-identical to the ruleset text its own SessionStart hook generates (the hook’s other first-run output is not reproduced). A close emulation of the shipped plugin’s full mode, with three documented differences (no first-run statusline nudge, no subagent re-injection, ruleset appended after the task rather than before it)
Volume	3 paired stages (10-task smoke, same 10 at k=3, full 80), plus self-activation and wiring checks — 251 billed agent trials in the complete evaluation program, USD 246.09. A few SkillsBench tasks are excluded: one that cannot run in a local sandbox, and a handful that fail identically in both arms on our hardware

One detail matters more than it looks. We generated arm B’s injected text by calling ponytail’s own hooks/ponytail-instructions.js rather than writing a summary of it, so the ruleset the model saw is the skill’s own text rather than our paraphrase of it. That covers the ruleset the hook generates, not every side-effect the hook has on a real first run. Every with-ponytail trial is audited afterwards to confirm the ruleset actually reached the model; every baseline trial is audited to confirm it did not. That check is the direct descendant of the contamination bug ponytail found in its own benchmark, and it came back clean: 100% of treatment trials, 0% of baselines.

Finding 1 — Does ponytail skill self-activate in Claude Code?

Before the paid runs we tested the obvious install path: drop the skill in and let Claude Code decide when to use it. Ponytail’s description invites exactly that, telling the model to use it on “ANY coding task: writing, adding, refactoring, fixing, reviewing, or designing code.”

Across all ten sessions it self-activated zero times. Not rarely. Never. The skill sat installed and visible and the model did not once reach for it.

This is not a bug in ponytail, and it is why the tool ships as a plugin with a SessionStart hook that injects the ruleset whether you ask or not. But it does mean the install method decides whether you get anything at all. Copy the SKILL.md into a skills folder and you will very likely measure nothing. Every number below comes from the arm where the ruleset is actually injected.

Finding 2 — the observed code cut is a third of the advertised size

Across 80 paired tasks a typical task shed 15.4% of the code the agent wrote; in total, 10,205 lines became 8,756. That is a substantial observed reduction. At p=0.088, however, it is the softest of our headline numbers. It is also nowhere near 54%.

Two things to say about the gap, both fair to the tool. First, their −54% is a mean across twelve hand-picked feature tickets; ours is a median across 80 tasks nobody chose for this purpose. Means and medians on skewed data are different animals, and their own writeup is explicit that the figure “reaches 94% where an agent over-builds and is near zero where the code is already minimal.”

Second, and this is the more interesting half: our own data points the same way.

Finding 3 — the saving concentrates where there was room to over-build

Split the tasks by how much code the baseline wrote. Ponytail cannot pick its own bucket that way, since the plain agent decides it. Worth saying plainly though: we chose these thresholds after seeing the data, and grouping by the baseline’s own output can stretch a gradient like this on its own. Read the chart as a strong hint about where the effect lives, not as a measured law.

On big builds the cut reaches −31%. On tasks where the plain agent already wrote almost nothing, the typical task moved by zero — though the totals in that group actually rose, 104 lines to 910, and that gap is where the run’s one real surprise turned up.

On seven tasks our counter recorded zero lines for the plain agent and 51 to 230 for ponytail. Reading the transcripts, that gap is mostly about where the code lived rather than how much of it there was. The plain agent piped its solution straight into a Python interpreter as a heredoc, which produced the deliverable and left no script behind. Ponytail wrote the same kind of logic to a file. Our counter treats a saved file as code and an inline heredoc as scratch, so one arm got charged for it and the other did not.

To be clear about what those files are: all seven are ordinary work scripts — edit.py, diff.py, build_model.py — not tests. So this is not ponytail’s “leave one runnable check behind” rule showing up; it simply saved its solution to disk where the plain agent piped the equivalent through an interpreter. We cannot say ponytail wrote more code on those tasks, only that more of its code was persisted.

Does that bias the headline? Slightly, and in both directions. Ponytail alone persisted code on 7 tasks (761 lines); the plain agent alone did on 4 (567 lines). Net, about 190 lines out of 10,205 land against ponytail — under 2%, and too small to lean on either way. We are not claiming the −15.4% is conservative because of it.

Finding 4 — the bill drops, and this time the signal is solid

A typical task cost 10.3% less with ponytail installed: p=0.004 across 80 pairs, cheaper on 46 tasks and dearer on 34. That is the strongest positive cost result in this series so far, and the first that is a solid saving rather than a solid penalty — rtk’s +7.6% was every bit as significant, just pointing the wrong way. Caveman also came out around 10% cheaper once we removed a single pricing-tier outlier, but that was a fragile number resting on one exclusion; this is the first time the cost difference has survived a paired test on a full run.

One honest qualifier, because we would want it applied to a vendor: the median saving is −10.3%, but the spread around it is wide enough that a bootstrap interval on the median just touches zero. The direction is well supported and the per-task test is clear. “Roughly 10% cheaper on this workload” is defensible; “ponytail saves you 10%” is not.

Worth noting what did not move cleanly: the input side. Re-reading its own history fell 8.4% and fresh tokens 3.9%, neither of them significant (p=0.138 and p=0.085). In part 2 we found that an agent’s bill is dominated by that re-reading, which is why a tool compressing command output barely dented it. Ponytail attacks the other side of the ledger, what the model writes, and on this benchmark that is the side the money moved on.

Finding 5 — no quality difference we can detect

The obvious worry about a skill whose whole personality is “write less” is that it gets there by deleting things that mattered. Ponytail claims it never touches validation, error handling, security or accessibility, and reports 100% safety in a separate adversarial tier of its own benchmark.

We cannot speak to that safety claim, and want to be explicit about why: SkillsBench verifiers score whether a task was completed. They are not a security, validation or accessibility suite. Nothing below tests whether ponytail preserves a guard, only whether the work still passes.

Nine tasks scored slightly worse, six slightly better, 65 identical — statistically indistinguishable. That is a null result, not a clean bill of health: this run was never powered to prove equivalence, and the data remain compatible with a small degradation as well as a small improvement. What we can say is that nothing here looks like the obvious failure mode, where writing less quietly stops the tests passing.

One small note on adherence. Ponytail’s ruleset asks the model to mark deliberate shortcuts with a ponytail: comment naming the ceiling and the upgrade path. Across 80 trials with the ruleset demonstrably in context, that happened once. The ladder gets followed; the paperwork does not.

Finding 6 — small samples lied to us, in both directions

Worth showing because it is the trap this whole series exists to avoid. Our ten-task smoke run said ponytail cut code by 3% and made things 9.6% more expensive, with mean task scores collapsing from 0.51 to 0.31. Had we published that, we would have written a very different and completely wrong article.

Verdict

Ponytail works. Across 80 paired tasks, it cut the typical bill by 10.3% and reduced code written by 15%, with no quality difference we could detect. It is the first tool in this series that clearly saved money. If you install it and forget about it, you should be modestly better off.

Do not expect the advertised 54% everywhere. Ponytail’s benchmark uses tasks with obvious over-building traps. Ours did not. In our runs, code fell 31% on larger builds and barely moved on tasks that were already lean. The more over-building your agent does, the more ponytail can cut.

Got a tool that claims to save tokens? Tell us which one and we will run it through the same benchmark.

Methodology notes

• Never trust k=1. Escalation ladder: a free transcript audit, then a 10-task smoke, then the same 10 at k=3, then the full 80. Finding 6 shows what the smoke would have told us.
• Paired analysis only. Per-task comparison between arms; any task that errored in either arm is dropped from both. Sign test for quality, per-task medians plus Wilcoxon for everything else, because one long-context session can bill 25× normal and wreck a mean.
• Endpoints fixed before the paid runs: reward, code written, output tokens, fresh input tokens, cost, turns, wall-clock. Total tokens was added afterwards, once we checked which metric ponytail’s own benchmarks/agentic/run.py actually advertises. It sums input, cache and output, so comparing our output-only figure against its −22% would have flattered us threefold.
• What a null result here does and does not mean. The quality comparison is a significance test, not an equivalence test. “No difference detected” is the honest reading; proving quality is genuinely unchanged would take a non-inferiority design with a pre-declared margin, and — for a 5-point shift in pass rate at 80% power around this benchmark’s ~40% baseline — on the order of several hundred paired tasks per arm rather than 80.
• Adoption instrumented. Every trial audited for whether the ruleset reached the model — 100% in the treatment arm, 0% in the baseline — so “ponytail saved nothing” can never be confused with “ponytail never ran.”
• Measuring code without a workspace diff. Ponytail’s own benchmark counts git diff added lines. Harbor keeps no post-agent workspace, so we reconstruct the equivalent from the agent’s tool calls: Write, Edit and shell heredocs redirected into a file, counted as non-blank non-comment lines exactly as ponytail’s benchmarks/loc.js does. This is cumulative lines emitted, not final implementation size: a line written and later rewritten counts each time. Heredocs piped to an interpreter are throwaway analysis and are excluded. We audited the extractor’s coverage on the run these figures come from: Write and Edit account for 95.6% of counted lines (15,632 and 2,496 of 18,961), so the metric is not an artifact of missing where the code went. Two things it cannot see, in both arms equally: files written by a script at runtime, and code written inside a subagent.
• Provenance. ponytail pinned at commit 16f2980 (v4.8.4, MIT); agent version pinned in both arms; the injected ruleset generated by ponytail’s own hook code, sha256 recorded. Seven of SkillsBench’s 87 tasks are excluded: one that cannot run in a local sandbox at all, and six that fail identically in both arms on our hardware. Exclusions are symmetric — a task is dropped from both arms or neither — and the full list is retained with the evaluation artifacts.
• What this cannot tell you. SkillsBench is data, analysis and repair work; it contains few of the front-end over-build traps that produce ponytail’s largest wins. This is a fair test of the cost, speed and quality claims and a conservative test of the code claim. It does not refute their −54% on their own task set.

Chart style borrowed from dither-kit, reimplemented here as a dependency-free inline widget.

For as long as we’ve taught programming, the student’s code has provided a window into the students’ thinking. Errors, the code structure, the awkward working solution—all of it showed how someone reasoned and where they got stuck.

It was never a clean window. Students have always copied, crammed, and borrowed, sometimes turning in work they didn’t fully understand. But the code still left clues. Generative AI has changed that: A finished program now tells us more about a student’s prompts than their ideas. And here’s the part that should unsettle us—often, the better the code looks, the less we can say about what the student actually learned.

This raises a bigger question: If AI can write code, should we still teach coding? I believe the answer is yes, at least for some students and situations. But that’s another topic. Here, I want to focus on the next step: If we continue teaching coding in a world with AI, how can we know if students are really learning?

Some schools have responded by trying to catch students. They use AI detectors, surveillance tools, locked-down browsers, stricter rules, and clearer honor codes. This has also led to more suspicion.

Some of these responses make sense. Teachers want to protect learning, and schools want to keep things fair. But using detection as the main way to assess students is weak. Stanford researchers found that popular AI detectors often falsely flagged writing by nonnative English speakers, with 61.22% of TOEFL essays in one study marked as AI-generated. OpenAI even retired its own AI Text Classifier in 2023 because it wasn’t accurate enough. If the company that created the tool can’t reliably detect AI, it’s probably not a good idea to base your honor code on it.

But detection isn’t the real issue. Even if we had a perfect detector, we’d still be asking the wrong question. Instead of asking, “How do we stop students from using AI?” we should ask, “How do we teach coding in a world with AI, making use of its benefits, while still being able to see if students are learning?”

Borrowing from the studio

We’re seeing this challenge with students at AET, the Arts and Entertainment Technologies Department at the University of Texas at Austin. Although my usual home is Computer Science, it so happens that AET is within the College of Fine Arts at UT, which offers many other ways to learn and assess: studio work, critique, rehearsal, revision, and performance.

In the arts, the final piece has never been the whole story. A painting doesn’t explain the choices behind it. A performance doesn’t reveal the rehearsals. A design board doesn’t show the discarded versions. A composition doesn’t tell you where the student struggled or what they finally learned to hear.

Art education has developed practices that focus on visible progress. Students bring in sketches and drafts, discuss influences, revisions, and failures, and rehearse, perform, and critique each other’s work while it’s still in progress.

At AET, we teach creative coding, which means programming to create art, design, games, or experiences. That doesn’t mean coding for poets. Our students—game designers, web developers, and programmers—start from scratch and learn advanced concepts in tools like Processing and p5.js. In the creative coding tradition, a program is often called a sketch, borrowing the term from the art world. It means something temporary, exploratory, and open to change—something you make, test, revise, and share.

So in creative coding, we were already leaning toward the studio model of sketches, experiments, iterations, and critique. Now we’re pushing that further as we rethink how we teach coding in an AI world. Here are three things we’re already using or actively developing.

Make the work public

We run the class like a studio. It’s not that work never happens at home, but the most important work needs to be seen in the classroom. Students show their code, including false starts, revisions, the choices they made, and the reasons behind them. Assignments are no longer just things you submit—they become projects you develop in public.

AI isn’t banned from the classroom. Instead, it’s treated as a helpful assistant to learn from. Students share prompts and techniques. They use AI, Google, Stack Overflow, classmates, or any other resources.

But you still need to take responsibility for your work. If you submit or present it, you must explain what the code does, why you made those choices, and how it works. If I need to ask your AI to understand your code, something is wrong. Getting help is fine, but hiding behind that help is not.

You can’t outsource to AI what the whole room watched you build.

A real studio needs students talking out loud together in the room every day. This also helps with another issue that isn’t about AI. Many people say students today are quieter than in the past. While this is mostly based on stories rather than long-term studies, these stories are common and consistent. Faculty on all types of campuses talk about silent classrooms and students who hesitate to speak up, especially since 2020.

Whatever the reason, this silence can be changed, and the solution is the same as for AI challenges: encourage students to participate. Communication is one of the most important skills in any career, including explaining ideas, defending choices, and persuading others in real time. Students don’t develop these skills by just submitting AI-guided work online. When they share their work publicly, it not only prevents AI misuse but also helps them build the skills they need most.

Invert the roles: AI as teacher and assessor

We know the usual pattern: A student asks, AI answers, and the student copies. We’ve tried to invert this. In our new approach, the AI works with the student on a set of topics, engages them in a conversation they must navigate, and ultimately assesses how well they understand the material, which leads to a grade.

This idea has a research background that goes back before ChatGPT. Teachable-agent systems like Betty’s Brain showed that explaining—even to a software agent—forces students to organize their knowledge, make connections clear, and find gaps. Our model uses this insight differently. The student isn’t teaching the bot. Instead, the student is having a conversation with it, learning, discussing, debating, and showing what they understand.

How did we do this? With fairly simple prompt engineering, we created an avatar chatbot of Vera Molnár (1924–2023), a pioneer of algorithmic art. The bot takes on Molnár’s role, drawing students into conversations about randomness, computation, generative art, and creative choices. Her practice sits exactly where creative coding students need to think: between rule and variation, system and choice, computation and visual judgment.

A system prompt sets the topics and types of questions to ask. The bot goes through these with the student, asks for more detail on unclear answers, and keeps following up until there is proof of understanding. At the end, it reviews the conversation against a rubric, giving us a clear record of which ideas the student covered, where they struggled, and how well they improved.

Besides the assessment, which is often accurate, the transcript becomes a different kind of proof, showing what a typical assignment might hide. What did the student notice? What did they misunderstand? Could they connect the concept to the code? Could they defend their choices? Could they revise their explanation when challenged?

When we switch the roles, something surprising appears: the one thing a finished submission can’t show.

A student thinking out loud.

Make understanding performative: Make students perform

Programming has never really had a tradition of performance. Musicians have it, painters have it, and dancers have it. Live coding is starting to change that.

Every semester at AET, students from different disciplines stage an algorave together—short for algorithmic rave. Audio sets, projection pieces, game demos, lasers, drones, experience design. The creative coding class brings live visuals into the live-coding tradition: Code is written and modified in real time, the screen is projected, and the audience watches the editor change as the visuals respond to the music other students are playing.

No prerender. No hiding the machinery.

The Live Coding manifesto, written in 2004 by TOPLAP, includes a line that fits every AI-era assessment conversation: “Obscurantism is dangerous. Show us your screens.” This is not just a performance ethic; it’s also an assessment strategy.

A student walks on stage. The projected screen is their editor. The room can read it. The music starts. And they build up a line of code on screen like:

osc(18, 0.08, 1.2)

.modulate(noise(3), 0.25)

.rotate(() => time * 0.1)

.out()

This is JavaScript building visuals in real time. FFTs, chained functions, higher-order manipulations. When you’re manipulating code like that on stage, you’d better know what you’re doing.

AI can help you prepare. Good. Let it.

But once you’re on stage, the question shifts from “Can you copy and paste code?” to “Can you control it?” You can paste code into a file, but you can’t paste your way through three minutes of public debugging while the whole projection turns into a beige rectangle. In a live build, understanding has nowhere to hide.

Can you read the code, make changes on purpose, and recover when something unexpected happens? That’s fluency: knowing what to do next while the system is still running.

It is very hard to plagiarize panic.

A note on assessment

So far, our results are based on our own observations. We haven’t conducted a controlled study or compared different groups, so what we have seen might just be early variation rather than patterns that apply more broadly. For now, these efforts are experiments, not final answers.

Assessment in studio and live performance settings is always subjective and focused on people. It relies on monitoring students’ progress, providing feedback, and observing how they handle challenges. We do not plan to change this core approach.

For the Molnár conversation assignment, students discussed Molnár using an AI system. The AI then created a summary and analysis of each student’s understanding. Teaching assistants reviewed this analysis, conducted their own assessments, and assigned grades. In our small experiments, the AI’s assessments using the rubric matched closely with the teaching assistants’ own evaluations.

We also used AI to help grade the end-of-term coding assignment. In this project, students improved an object-oriented game by adding strategies like heuristics, search algorithms, and learned behaviors. Since our teaching assistants had limited experience with object-oriented programming, we developed a detailed rubric and had an AI model use it to evaluate each submission. The AI’s analysis was given to the teaching assistants as support. It helped them see how each project was structured, spot important OOP design choices, and use the rubric with more confidence. The teaching assistants still made their own grading decisions. I was available as the OOP expert for any questions they could not answer. From what I observed, this substantially helped the teaching assistants understand and grade the students’ OOP design work.

More broadly, both approaches appear to enable substantive feedback at a scale that would otherwise be difficult given our current student-to-teaching-assistant ratios.

The process is the proof

We spent the first two years of the generative AI panic asking how to catch students using AI—or prohibit it altogether. Wrong question.

The real question is whether the assignment gives students a real way to show and develop their understanding. This view isn’t limited to educators. NVIDIA CEO Jensen Huang recently argued that students should not focus on finding an “AI-proof” subject. Instead, he suggested they consider how AI can help them learn more deeply and develop their skills and sense of purpose. He highlighted storytelling, creativity, design, and judgment as abilities that will stay important even as AI takes over more tasks. This supports a key idea in coding education: The aim is not to prove you didn’t use any tools, but to help students show how they think, make choices, revise, and take responsibility for their work.

These three practices are experiments, not universal solutions. They work especially well in creative coding, where code already has a public, visual, and performative aspect. But they suggest a broader principle: As finished work becomes easier to generate, assessment needs to focus more on process, explanation, revision, and mastery.

This matters outside of school too. A polished memo no longer proves there was real thinking behind it. A working prototype no longer proves product sense. A passing pull request no longer proves the developer made the change carefully and thoughtfully. AI makes production easier, so evaluation must focus more on how people think, choose, revise, and recover—in code review, hiring, and performance management. The artifact is no longer the proof. The process is.

Generative AI didn’t make assessment impossible. It just made a hidden weakness obvious. We were putting too much trust in finished work. The arts always knew better.

Show us your screens.

---

Acknowledgements

Thanks to Mike Loukides, Michael Baker, Mk Haley, Elisabeth Robson, and Honoria Starbuck for feedback on this article.

References

OpenAI. “New AI classifier for indicating AI-written text.” OpenAI Blog, January 31, 2023. Updated July 20, 2023, to note the classifier was no longer available due to low accuracy.

Liang, Weixin, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. “GPT detectors are biased against non-native English writers.” Stanford HAI, July 10, 2023.

Winthrop, R. (2026, May 27). Writing with A.I. weakens your creativity. The New York Times.

TOPLAP. “TOPLAP Manifesto.”

Schell, J., Ford, K., & Markman, A. B. (2025). Building responsible AI chatbot platforms in higher education: An evidence-based framework from design to implementation. Frontiers in Education, 10, Article 1604934. https://doi.org/10.3389/feduc.2025.1604934

Biswas, Gautam, Daniel Schwartz, John Bransford, and the Teachable Agents Group at Vanderbilt. “Technology support for complex problem solving: From SAD environments to AI.” In Learning to Solve Complex Scientific Problems, 2001.

Leelawong, Krittaya, and Gautam Biswas. “Designing learning by teaching agents: The Betty’s Brain system.” International Journal of Artificial Intelligence in Education, 2008.

Tan, Huileng. “Jensen Huang Says It Doesn’t Matter What Kids Study in the AI Era.” Business Insider, May 26, 2026. https://www.businessinsider.com/nvidia-jensen-huang-what-kids-should-study-ai-education-advice-2026-5

DAM Digital Art Museum. “Vera Molnár.” Artist biography and timeline.

Odysseus is interesting because it shows how a project with serious ambitions can now be built outside a traditional software company. 

After speaking with Ivan Mihić, a software engineer who spent time testing PewDiePie’s AI project, I ended up thinking that the software itself might not be the most interesting part of this story.

Odysseus may come from a YouTube creator with a massive audience, but the more important question is what it says about software development itself. 

Another AI app – or something more?

When Felix “PewDiePie” Kjellberg unveiled Odysseus, the internet reacted in the way it usually does when a major creator launches a software project. Some treated it as a celebrity side project, others focused on the privacy claims and some compared it with other AI tools such as GPT or Claude. 

For those of you who might be late to the party (same as we are with this text), Odysseus is an open source, self hosted AI workspace designed to run locally. Instead of depending only on cloud services, it combines local models with web search, document analysis, long term memory, and AI agents inside a single browser interface. If you want, you can also add cloud models to it as well.

Under the hood, the application integrates components such as ChromaDB for memory, SearXNG for search, and local model serving, while also letting users decide whether they want everything on their own hardware or connected to external APIs. 

What stood out to Ivan was not just what Odysseus does, but what its existence says about how software gets built nowadays:

The biggest thing isn’t that Odysseus is a good tool, It’s that a project of this size probably wouldn’t exist without AI.

A few years ago, building a system that combined models, search, memory, document pipelines, and agents would usually have required a much larger engineering team. Today, a determined individual or a small group can at least prototype that kind of product and keep iterating on it.

To me as an outsider, this looks like the phase in which we built AI tools mainly for internal use is slowly coming to an end. More and more of these tools are now being prepared for public release, even if they are still far from fully polished.

Ivan also admints that he was was surprised. Not because the tool is perfect, because it as you would expect isn’t but because something that started from a hobbyist, vibe-coding context grew into a project of this scale. For him, Odysseus is less a sign that AI has replaced engineers and more a sign that AI has changed the starting line. And I couldn’t agree more on this one.

Privacy alone won’t protect you 

Much of Odysseus’ appeal comes from privacy. Running AI models locally means sensitive data does not have to leave your machine, which is a real advantage. But Ivan pointed out that privacy and security are not the same thing. 

Odysseus is local-first, but that doesn’t automatically make it secure. 

The platform can access files, execute shell commands, perform web research, and interact with external services, therefore a bad configuration can quickly create security risks. That matters even more if users expose their instance to the public internet. 

Ivan’s point is simple: a tool can be private in design and still be unsafe in practice if people do not understand what they are connecting, exposing, or automating. 

When a tool presents itself through the lens of privacy, security becomes extremely important. Privacy without security doesn’t mean much.

Odysseus is not mature… yet

Architecturally, Ivan sees Odysseus as a large monolithic application coordinating several supporting services rather than a fully mature enterprise platform. 

It’s not the cleanest or most mature architecture I’ve seen. You can clearly tell it evolved through experimentation.

That experimental nature is visible throughout the product. Odysseus tries to combine chat, agents, research, memory, document analysis, and productivity tools into one personal AI workspace, with varying degrees of success.

A few features stood out to Ivan in particular. “Compare” lets users run the same prompt across multiple models at once. “Deep Research” automatically performs multi step web research and returns cited sources. “Cookbook” helps users choose, download, and serve models based on the hardware they actually have. 

Still, for someone who is just entering the world of local AI, Ivan thinks more established tools currently make more sense, just because tools like Open WebUI has had more time to mature.

This is not exactly a beginner-friendly tool

Talking about Odysseus once it’s installed is fine, but getting there as a regular AI user might prove to be a challenging task. From my experience, users need to be wary of the fact that, in order to fully use Odysseus, many setup steps are needed. They also need to know that they might not understand a single thing about what’s happening, which is also a risk in itself.

Odysseus is not an app you can install and run in a minute or two, while other tools do just that. The user experience for a beginner might be overwhelming and, at times, just simply not worth it if you’re in it just to experiment and see what the tool can offer.

Ivan Šimić, the third Ivan in this story, a senior writer at ShiftMag who kept experimenting with the tool to figure it out, added:

Trying to hop on the bandwagon of local LLM’s after reading about Odysseus might not be the greatest idea, since it requires a capable machine, a basic understanding of key aspects of making things happen on a PC or Mac, patience and time. For many interested in trying it, this might prove to be too much “tinkering” for not a lot of results.

Big ideas in software are no longer a big-team privilege

It would be easy to frame Odysseus as another celebrity tech experiment but that would miss the broader point. 

What matters here is that AI assisted development has lowered the cost of building ambitious software. A project that once would have required a sizable engineering team is now something a small group, or even one person, can assemble and refine much faster than before. 

And yes, while a fair portion of the pessimistic internet is claiming this means software engineers are disappearing, I highly doubt it. The job is changing, and the gap between an idea and a working product is smaller than it used to be.

As Ivan nicely put it, it’s like Jack Black’s quest for the magical guitar pick in The Pick of Destiny. In the film, that pick gives its owner superhuman guitar skills. In reality, even the best tool won’t make him or anybody else the world’s best developer, but it can certainly make everyday job a lot easier.

Therefore, the most interesting thing PewDiePie built may not be another AI assistant, but proof that ambitious software is no longer reserved for traditional software companies and big teams. Big Tech, are you scared yet?

The post PewDiePie’s AI Experiment Is What Software Development Looks Like Now appeared first on ShiftMag.

Apple Upgrade — a new hardware leasing program — is now available on the Apple Store online, in the Apple Store app, and at Apple Store locations in the United States.

Artificial intelligence has changed the cybersecurity landscape faster than most organizations anticipated. Over the past decade, security teams have become accustomed to protecting cloud environments, hybrid identities, remote workforces, and increasingly complex application portfolios. Each of these shifts introduced new challenges, but they largely built upon existing security models. AI is different. It is not simply another technology to secure; it is fundamentally changing how both attackers and defenders operate.

Cybercriminals are already using AI to write convincing phishing emails, accelerate malware development, automate reconnaissance, and identify vulnerabilities more efficiently than ever before. At the same time, organizations are rapidly adopting generative AI, AI-powered business applications, and intelligent agents that have access to corporate data and business processes. Security teams now face the challenge of protecting not only traditional infrastructure, but also AI systems that reason, make decisions, and interact with sensitive information.

Microsoft believes this requires a fundamental rethink of enterprise security. Rather than continuing to build increasingly sophisticated tools that generate more alerts for analysts to investigate, the company has introduced Project Perception, a new architectural vision for security in the age of AI. Instead of focusing solely on detecting attacks, Project Perception is designed to understand context, reason across multiple security signals, and help organizations identify and reduce risk before those risks become incidents.

While the announcement has generated significant attention, it is important to understand that Project Perception is not simply another security product. It represents Microsoft’s view of how cybersecurity must evolve as AI becomes embedded into every organization and every business process.

Why Enterprise Security Needs to Evolve

Modern organizations have never had more visibility into their environments. Every endpoint generates telemetry. Every identity produces authentication logs. Cloud platforms record configuration changes, applications emit diagnostic information, and network devices constantly monitor traffic. Added together, security teams often receive millions of individual events every day.

The challenge has never been collecting information. Most organizations already collect more security data than any team could reasonably review. The real challenge is determining which events actually matter, how they relate to one another, and which ones represent genuine business risk.

Traditional security tools typically analyze individual events extremely well. Endpoint detection platforms understand endpoint behavior, identity platforms understand authentication activity, vulnerability scanners identify missing patches, and cloud security platforms monitor infrastructure. The difficulty arises when analysts need to correlate information across all of these systems while simultaneously understanding the business context surrounding an event.

For example, an unusual sign-in, a newly discovered vulnerability, an AI application requesting additional permissions, and a privileged account making configuration changes may each appear harmless when viewed individually. However, when considered together, they could represent the early stages of an attack. Recognizing those relationships requires context, reasoning, and experience—qualities that have traditionally depended upon skilled security professionals.

As organizations continue adopting AI, this challenge becomes even more complex. AI applications often interact with multiple data sources, access sensitive information, invoke business workflows, and communicate with other services. Understanding whether those interactions represent normal business activity or malicious behavior requires a much broader understanding than traditional security analytics alone can provide.

This growing complexity is one of the primary reasons Microsoft believes security must evolve beyond simply generating alerts. The goal is no longer to provide security professionals with more information; it is to provide them with a better understanding.

Introducing Project Perception

Project Perception is Microsoft’s vision for what the next generation of enterprise security should look like. Rather than treating security products as isolated tools that each solve a specific problem, Project Perception introduces an intelligent security layer capable of reasoning across the entire environment.

The concept behind the project is relatively straightforward. Organizations already possess enormous amounts of security data, but much of that information remains isolated within individual products and services. Project Perception aims to connect those signals together, understand how they relate, and provide meaningful insight instead of simply presenting raw events.

The key difference is that Microsoft is building this around AI agents rather than traditional automation. An AI agent is designed to perform a specific task, maintain context over time, collaborate with other agents, and continuously evaluate changing conditions. Instead of responding only when a predefined rule is triggered, these agents are intended to reason about situations, gather additional information, and help determine what actions to take next.

This represents an important shift in thinking. For many years, automation has been based on fixed logic. If a particular event occurs, execute a predefined action. While this remains valuable, it becomes increasingly difficult to build automation for every possible security scenario. AI agents provide greater flexibility because they can evaluate context, adapt to changing conditions, and collaborate with other agents that possess different areas of expertise.

Microsoft describes Project Perception as creating an intelligent security fabric where specialized agents work together rather than independently. Instead of every security product operating in isolation, these agents continuously exchange information, build context, and develop a much richer understanding of the organization’s security posture.

Thinking Beyond Individual Security Tools

One of the most significant ideas behind Project Perception is that security should no longer be viewed as a collection of disconnected products. For years, organizations have assembled security platforms by purchasing individual solutions for endpoint protection, email security, identity management, vulnerability assessment, cloud security, and data protection. Each product performs its own analysis, produces its own alerts, and often requires its own specialist knowledge.

This approach has served the industry well, but it also places a considerable burden on security teams. Analysts frequently spend more time correlating information between different tools than they do actually responding to threats. They must determine whether multiple alerts represent the same incident, understand which vulnerabilities pose the greatest business risk, and decide which issues require immediate attention.

Project Perception attempts to reduce this complexity by shifting the focus away from individual products and toward shared intelligence. Rather than expecting people to manually connect information from numerous security platforms, AI agents continuously perform much of that correlation themselves. As they gather additional information, they refine their understanding of the situation and provide increasingly informed recommendations to security teams.

The value of this approach is not simply faster investigation. It is a better understanding of the relationships between identities, applications, infrastructure, AI workloads, and business processes. Those relationships often provide the context needed to distinguish between routine activity and genuinely suspicious behavior.

From Alerts to Understanding

Many organizations measure security success by the number of alerts they investigate or the speed with which they respond to incidents. While these metrics remain important, they do not necessarily indicate whether the organization is becoming more secure.

Project Perception focuses on moving beyond alerts toward understanding. Instead of treating every event as an isolated occurrence, the system attempts to answer broader questions.

For example:

• What changed in the environment?
• Why is this change significant?
• Does it relate to other activity already taking place?
• Could multiple seemingly unrelated events indicate a larger attack?
• What is the potential business impact?
• Which risks should be addressed first?

These questions require reasoning rather than simple pattern matching. They also require access to information across multiple systems rather than within a single product.

By continuously building context, Project Perception aims to reduce the time security professionals spend collecting information and increase the time they spend making informed decisions. Instead of asking analysts to manually assemble the bigger picture, the platform helps construct that picture automatically.

Why Businesses Should Care

It is easy to assume that Project Perception is relevant only to very large enterprises with mature security operations centers. In reality, many of the problems it addresses are common across organizations of every size.

Most businesses struggle with limited security resources. Hiring experienced cybersecurity professionals remains difficult, while the number of technologies that require protection continues to grow. Cloud services, remote work, SaaS applications, AI assistants, regulatory requirements, and third-party integrations all contribute to an increasingly complex environment.

Simply adding more security tools rarely solves this problem. In many cases, it actually increases operational complexity by introducing additional dashboards, alerts, and management interfaces. What organizations often need is not more data, but better prioritization and better understanding.

Project Perception addresses this challenge by helping organizations make more effective use of the information they already possess. Instead of expecting analysts to review thousands of low-priority events, AI agents can help identify the issues most likely to affect the business, explain why they matter, and recommend appropriate next steps.

For businesses, this could provide several important benefits:

• Reduced time spent investigating low-value alerts.
• Better prioritization of security risks.
• Improved visibility across hybrid and cloud environments.
• Greater confidence when adopting AI-powered business applications.
• More effective use of existing security teams.
• Faster identification of emerging threats before they develop into major incidents.

Perhaps the most important benefit, however, is that organizations can begin shifting their focus from reactive security toward proactive risk reduction.

AI Defending an AI-Driven Organization

Another reason Project Perception is significant is that it recognizes how quickly enterprise environments are changing. AI is no longer limited to experimental chatbots or isolated productivity tools. Organizations are increasingly deploying AI agents that interact with customer information, generate business content, automate workflows, analyze documents, and make recommendations that influence real business decisions.

Every one of these AI systems introduces new considerations for security teams. Questions surrounding data access, identity permissions, prompt injection, information leakage, governance, and regulatory compliance become increasingly important as AI becomes embedded within everyday business operations.

Microsoft’s broader security strategy reflects this shift. Rather than viewing AI as another application that requires traditional protection, the company increasingly sees AI as both something that must be secured and something capable of improving security itself.

Project Perception sits at the intersection of these two ideas. It is designed to help organizations safely adopt AI while simultaneously using AI to improve visibility, reasoning, and decision-making across their existing security operations.

Humans Remain at the Center

Whenever AI becomes part of a security discussion, one of the first questions raised is whether it will eventually replace security professionals. Microsoft’s vision suggests something quite different.

Project Perception is intended to augment human expertise rather than eliminate it. Security remains fundamentally a business discipline that requires judgment, governance, communication, and strategic decision-making. AI can analyze information, identify relationships, and recommend actions, but organizations still need experienced professionals to determine acceptable levels of risk, approve remediation plans, communicate with leadership, and ensure compliance with legal and regulatory requirements.

In practice, this means AI becomes another member of the security team rather than a replacement for it. Analysts spend less time collecting evidence and more time evaluating recommendations. Engineers spend less time identifying routine issues and more time implementing meaningful improvements. Security leaders receive greater context for making strategic decisions instead of simply reviewing large numbers of alerts.

This balance between human expertise and AI assistance is likely to become one of the defining characteristics of enterprise cybersecurity over the next decade.

Looking Ahead

Project Perception is best viewed as Microsoft’s long-term direction for enterprise security rather than a standalone feature or product announcement. It reflects a recognition that cybersecurity must evolve alongside AI and that traditional approaches alone are unlikely to meet the demands of increasingly intelligent attacks and increasingly complex enterprise environments.

The announcement also reinforces a broader trend that is becoming evident across the technology industry. Future security platforms will be defined less by individual products and more by their ability to understand relationships, maintain context, reason across multiple domains, and help organizations make better decisions. AI agents, shared intelligence, and continuous reasoning are likely to become foundational capabilities rather than optional enhancements.

For organizations already invested in Microsoft Security, Project Perception offers an early glimpse into how Microsoft’s security portfolio may continue to evolve. For everyone else, it provides valuable insight into where enterprise cybersecurity is heading regardless of which technology platform they ultimately adopt.

The security industry has always adapted to major technological shifts, from virtualization and cloud computing to mobile devices and remote work. Artificial intelligence represents the next major transformation, and arguably the most significant yet. Successfully navigating that transformation will require more than faster tools or larger security teams. It will require security platforms capable of understanding context, reasoning across enormous volumes of information, and helping people make better decisions.

Project Perception is Microsoft’s vision for achieving that goal. Whether every aspect of that vision develops exactly as planned remains to be seen, but the direction is becoming increasingly clear. As AI becomes a core part of modern business, cybersecurity must evolve from simply detecting threats to continuously understanding risk. That shift, more than any individual feature, is what makes Project Perception one of Microsoft’s most important security announcements in recent years.

Official Microsoft information

https://www.microsoft.com/en-us/security/business/ai-powered-cybersecurity/project-perception-agentic-system

https://blogs.microsoft.com/blog/2026/07/27/rethinking-security-for-the-age-of-ai

https://techcommunity.microsoft.com/blog/microsoft-security-blog/how-nationwide-stays-ahead-of-attackers-with-project-perception/4540534