We’re announcing even more new capabilities in Managed Agents in Gemini API so developers can build reliable, production-ready agents.

Perplexity has expanded its agentic Personal Computer tool to Windows, allowing computers running the world's most popular OS to be used as a locally run AI system. Like the Mac version that Perplexity launched in April, Personal Computer for Windows operates like a "general-purpose digital worker" that can access local files and apps to perform actions on your behalf, such as creating documents and updating spreadsheets.

This launch builds on Personal Computer integrations that Perplexity launched for Microsoft's 365 workspace apps and Teams virtual meeting software in May. Personal Computer for Windows aims to bridge the remaining gap by …

Read the full story at The Verge.

In the past year, there’s been a pattern of supply chain attacks that target weaknesses in package repositories and CI/CD systems to quickly spread malware to hundreds of open source projects. This malware seeks to exfiltrate credentials both to broadly spread the attack, as well as for later exploitation.

We’ve written a few times about our plans for hardening the supply chain: Our plan for a more secure npm supply chain in September 2025, Strengthening supply chain security: Preparing for the next malware campaign in December 2025, and What’s coming to our GitHub Actions 2026 security roadmap in March 2026. In this post, we’re updating you on changes we’ve implemented that directly disrupt some of the most common and impactful supply chain attack techniques.

Anatomy of supply chain attacks

Supply chain attacks chain together several weaknesses, and there is no single security capability that can stop them. Addressing them takes a holistic approach, prioritizing the mitigations that break the most impactful links in the attack chain. Our teams have been studying these attacks to deploy several improvements that disrupt them and limit their impact. This is possible thanks to collaboration with the security research and developer communities.

The attacks vary in how they spread across the software ecosystem. However, most of these attacks follow similar techniques to gain initial access to a project, escalate privileges, and distribute across users and software. Improvements made to npm and GitHub Actions in the past few months have been focused on cutting off specific, common techniques and providing ways for customers to identify and respond to these attacks.

Initial compromise

Attacks start by compromising a single project, often by directly compromising a maintainer’s account or by targeting the project’s actions workflows.

• npm adds preventive account protection for high-impact accounts (June 2026): Frequently, attacks start with a phishing campaign targeting maintainers. With this change, high-impact npm accounts are now put into a read-only mode for 72 hours when they change their email or use a 2FA recovery code. This delay allows maintainers time to respond and recover the account before their account can be used to start an attack.
• Safer pull_request_target defaults for GitHub Actions checkout (June 2026): A common vulnerability in a project’s CI/CD pipelines are “pwn requests,” where a workflow triggers on pull requests from forks and then executes user-submitted and untrusted code from that fork. We changed the default behavior of actions/checkout to prevent the checkout of untrusted code from forks in commonly exploited triggers unless you explicitly opt-out (after reviewing your risk). This change and its backport to older versions cut off one of the most common vulnerable code patterns leading to code execution in GitHub Actions CI/CD workflows and initial project compromise.
• Control who and what triggers GitHub Actions workflows (June 2026): Maybe you’d prefer to opt-out of these risky action triggers altogether or limit who can trigger them. This new control lets you set enterprise, organization, or repository level policies on who is allowed to trigger workflows and what trigger types are allowed. These workflow execution policies provide a governable and customizable layer of least-privilege around Action workflows that reduce the attack surface of your CI/CD infrastructure.
• Read-only Actions cache for untrusted triggers (June 2026): After an attacker has achieved code execution in an Actions workflow, they then look to escalate to more privileged workflows (and therefore credentials) through poisoning the cache entries shared across workflows. With this change, we restrict the ability for less trusted workflows to modify the cache shared with other workflows. This directly closes a common path attackers have used to turn a vulnerability with limited impact into one that compromises highly privileged credentials used by release and publishing workflows.

Exfiltrate credentials

Once an attacker has access to a single package, they then focus on detecting and exfiltrating credentials to gain further access and use in later exploitation across ecosystems.

• npm trusted publishing now supports CircleCI (April 2026): The number one thing you can do to disrupt these attacks is to remove long-lived credentials from your CI/CD pipeline. Trusted publishing is a great way to authorize publishes to your package repository without a long-lived credential. By adding CircleCI as a trusted publishing provider, we’ve made it possible for more people to remove the credentials these attacks attempt to exfiltrate.
• Actions network firewall (In technical preview): This technical preview logs all outbound network traffic from your Action workflow runs so you can detect unusual behavior like pulling down malicious code or exfiltrating credentials to a new domain. Future work will enable network egress restrictions and policies to block these attacks before they lead to further escalation and exfiltration.

Propagating the attack

With the credentials harvested from the previous step, attackers attempt to use those credentials to distribute their malware and compromise more projects and maintainers as quickly as possible.

• Staged publishing for npm (May 2026): With staged publishing, it’s not enough to have credentials to publish a new package on npm; those packages are staged until additional approval and 2FA authentication is provided in the npm cli or on npmjs.com. This opt-in security control allows maintainers to ensure that any version of their package published has gone through this additional authorization. By decoupling the credentials used in CI/CD pipelines and automation from those that can publish to the registry, the attack chain from a CI/CD pipeline to malware distribution is cut off.
• Upcoming breaking changes for npm v12 (June 2026): To spread their malware as quickly as possible, attackers use npm install-time scripts to exfiltrate credentials instead of waiting for code to be executed by the package at runtime. With npm v12, we are rolling out a breaking change that disables these install scripts by default. Since install scripts have legitimate use within the package installation processes that several popular packages rely on, you can reenable them by approving specific scripts. Additional vectors for install-time code execution have also been blocked by disabling dependencies via git or remote URLs by default.
• Dependabot version updates introduce default package cooldown (July 2026): Attackers rely on speed, hoping a malicious release gets pulled into as many downstream projects as possible before anyone notices. Version updates through Dependabot now wait until a release has been available for at least three days before opening a pull request, giving detection signals time to surface before a malicious release reaches your project. This cooldown is on by default, and security updates still open immediately, so critical fixes are never delayed.

Identifying and responding to supply chain attacks

In parallel to hardening npm and GitHub Actions to disrupt and limit the impact of supply chain attacks, we have also been working on making features and tools available to users to identify and respond to supply chain incidents that have impacted their projects and accounts.

• Self-service credential revocation for incident response (June 2026): GitHub credentials remain an ongoing target of attackers in supply chain attacks. This feature provides self-service tooling to instantly revoke all credentials for a given user in an enterprise. This builds on the enterprise-wide credential management tools for incident response released in February and allows enterprise admins and members to quickly respond if credentials have been compromised in a supply chain attack.
• Expanded credential revocation API support (March 2026): Recent attacks have included exfiltrating credentials within publicly accessible content. To help the community respond to these attacks, we have expanded support of our credential revocation API (first introduced in April 2025 for personal access tokens) to support the revocation of GitHub OAuth and App tokens. This allows the quick and self-service revocation of these leaked credentials, regardless of where they are found, and limits the lifetime during which they can be abused.

What’s Next?

Making our products more secure by default is a priority across npm and GitHub and we are prioritizing this work to target and disrupt supply chain attacks across the open source ecosystem. We’re proud of the work we’ve shipped towards this goal over the past months. There’s more to come, but we wanted to provide an update on the progress we’ve made and make folks aware of the new capabilities available to them. Be sure to follow our changelog and blog posts as we continue to roll out improvements.

Open source software is an incredible public good that we all benefit from, and this is one of several ways GitHub is working to continue to support the security, sustainability, and continued success of open source communities and the enterprises that depend on them.

The post Disrupting supply chain attacks on npm and GitHub Actions appeared first on The GitHub Blog.

Part 3 of a series where we take public “token saver” add-ons for coding agents and run the same paired A/B benchmark against each of them. Part 1 was the caveman skill (advertised −65%, measured −8.5%). Part 2 was rtk (advertised −60–90%, measured +7.6%).

We ran 80 paired tasks to test the ponytail skill for Claude Code. Advertised: −54% code, -22% tokens, -20% cost, -27% time. Measured: −15% code, −10.3% cost and -11% time. Here’s what actually happened.

Real savings, although roughly a quarter to a half of what is advertised, it is the first tool in this series with a statistically solid cost-saving signal. We found no quality difference, though ~80 pairs can only rule out large ones. The catch: the code cut only shows up where there was room to over-build.

Why we ran this

Ponytail skill is designed to make AI agents write less code. Its core premise: a senior developer who has seen everything replaces your fifty lines with one. Ask for a date picker and instead of installing flatpickr and writing a wrapper component, it writes <input type="date"> and moves on.

Mechanically it is a ladder the model climbs before writing anything. Does this need to exist at all? Is it already in the codebase? Does the standard library do it? A native platform feature? An installed dependency? Can it be one line? Only then: write the minimum that works. The ladder runs after understanding the problem, not instead of it, and validation, error handling, security and accessibility are explicitly off the chopping block.

Here is what that looks like in practice, from our own run. Both agents were asked to export a three.js scene to a Blender-ready OBJ file; both produced a file the verifier accepted. Both wrote the same fiddly loop to expand instanced meshes, because three.js’s OBJExporter cannot handle them. The difference is everything around that loop. To rotate the scene into Blender’s orientation and write it out, the plain agent builds a wrapper object to hold the rotation and names every intermediate step:

// no skill — 10 statements to rotate the scene and write the file
const exportRoot = new THREE.Group();
exportRoot.name = 'blender_export_root';
exportRoot.rotation.x = -Math.PI / 2;
exportRoot.add(root);
exportRoot.updateMatrixWorld(true);

const exporter = new OBJExporter();
const objString = exporter.parse(exportRoot);

const outputPath = '/root/output/object.obj';
fs.mkdirSync(path.dirname(outputPath), { recursive: true });
fs.writeFileSync(outputPath, objString);


// ponytail — the same job, 5 statements
root.rotation.x = -Math.PI / 2;
root.updateMatrixWorld(true);

const obj = new OBJExporter().parse(root);
fs.mkdirSync('/root/output', { recursive: true });
fs.writeFileSync('/root/output/object.obj', obj);

Nothing was sacrificed there. Ponytail rotated the object it already had instead of building a parent to rotate it for it, and skipped an import while it was at it. Ten statements became five, both files exported the same geometry, and both scored 1.0. That is the effect working exactly as advertised — on one file, on one task.

The headline claim is −54% code, plus −22% tokens, −20% cost and −27% time. What made this one worth testing is that the claim is unusually well documented. The authors rebuilt their benchmark in response to a critique (issue #126) that their original numbers came from a chatty baseline, and they publish the honest version: a real headless Claude Code session editing a real FastAPI + React repo, scored on the git diff it leaves behind. They even document a contamination bug they found in their own harness, where a SessionStart hook was firing on every arm and secretly running ponytail in the baseline.

That is more methodological candour than most tools in this space manage. So the question here is whether the effect survives a benchmark the authors did not choose, on a stronger model, with verifier-scored quality.

Setup

Harness	Harbor 0.18 — Docker sandboxes, task verifiers, paired runs
Agent	Claude Code 2.1.201, headless, bypassPermissions, pinned in both arms
Model	claude-sonnet-5 at medium reasoning effort
Benchmark	SkillsBench, 80 paired tasks, auto-graded 0–1 with partial credit
Arm A	stock Claude Code
Arm B	ponytail v4.8.4: skill installed and its ruleset injected, byte-identical to the ruleset text its own SessionStart hook generates (the hook’s other first-run output is not reproduced). A close emulation of the shipped plugin’s full mode, with three documented differences (no first-run statusline nudge, no subagent re-injection, ruleset appended after the task rather than before it)
Volume	3 paired stages (10-task smoke, same 10 at k=3, full 80), plus self-activation and wiring checks — 251 billed agent trials in the complete evaluation program, USD 246.09. A few SkillsBench tasks are excluded: one that cannot run in a local sandbox, and a handful that fail identically in both arms on our hardware

One detail matters more than it looks. We generated arm B’s injected text by calling ponytail’s own hooks/ponytail-instructions.js rather than writing a summary of it, so the ruleset the model saw is the skill’s own text rather than our paraphrase of it. That covers the ruleset the hook generates, not every side-effect the hook has on a real first run. Every with-ponytail trial is audited afterwards to confirm the ruleset actually reached the model; every baseline trial is audited to confirm it did not. That check is the direct descendant of the contamination bug ponytail found in its own benchmark, and it came back clean: 100% of treatment trials, 0% of baselines.

Finding 1 — Does ponytail skill self-activate in Claude Code?

Before the paid runs we tested the obvious install path: drop the skill in and let Claude Code decide when to use it. Ponytail’s description invites exactly that, telling the model to use it on “ANY coding task: writing, adding, refactoring, fixing, reviewing, or designing code.”

Across all ten sessions it self-activated zero times. Not rarely. Never. The skill sat installed and visible and the model did not once reach for it.

This is not a bug in ponytail, and it is why the tool ships as a plugin with a SessionStart hook that injects the ruleset whether you ask or not. But it does mean the install method decides whether you get anything at all. Copy the SKILL.md into a skills folder and you will very likely measure nothing. Every number below comes from the arm where the ruleset is actually injected.

Finding 2 — the observed code cut is a third of the advertised size

Across 80 paired tasks a typical task shed 15.4% of the code the agent wrote; in total, 10,205 lines became 8,756. That is a substantial observed reduction. At p=0.088, however, it is the softest of our headline numbers. It is also nowhere near 54%.

Two things to say about the gap, both fair to the tool. First, their −54% is a mean across twelve hand-picked feature tickets; ours is a median across 80 tasks nobody chose for this purpose. Means and medians on skewed data are different animals, and their own writeup is explicit that the figure “reaches 94% where an agent over-builds and is near zero where the code is already minimal.”

Second, and this is the more interesting half: our own data points the same way.

Finding 3 — the saving concentrates where there was room to over-build

Split the tasks by how much code the baseline wrote. Ponytail cannot pick its own bucket that way, since the plain agent decides it. Worth saying plainly though: we chose these thresholds after seeing the data, and grouping by the baseline’s own output can stretch a gradient like this on its own. Read the chart as a strong hint about where the effect lives, not as a measured law.

On big builds the cut reaches −31%. On tasks where the plain agent already wrote almost nothing, the typical task moved by zero — though the totals in that group actually rose, 104 lines to 910, and that gap is where the run’s one real surprise turned up.

On seven tasks our counter recorded zero lines for the plain agent and 51 to 230 for ponytail. Reading the transcripts, that gap is mostly about where the code lived rather than how much of it there was. The plain agent piped its solution straight into a Python interpreter as a heredoc, which produced the deliverable and left no script behind. Ponytail wrote the same kind of logic to a file. Our counter treats a saved file as code and an inline heredoc as scratch, so one arm got charged for it and the other did not.

To be clear about what those files are: all seven are ordinary work scripts — edit.py, diff.py, build_model.py — not tests. So this is not ponytail’s “leave one runnable check behind” rule showing up; it simply saved its solution to disk where the plain agent piped the equivalent through an interpreter. We cannot say ponytail wrote more code on those tasks, only that more of its code was persisted.

Does that bias the headline? Slightly, and in both directions. Ponytail alone persisted code on 7 tasks (761 lines); the plain agent alone did on 4 (567 lines). Net, about 190 lines out of 10,205 land against ponytail — under 2%, and too small to lean on either way. We are not claiming the −15.4% is conservative because of it.

Finding 4 — the bill drops, and this time the signal is solid

A typical task cost 10.3% less with ponytail installed: p=0.004 across 80 pairs, cheaper on 46 tasks and dearer on 34. That is the strongest positive cost result in this series so far, and the first that is a solid saving rather than a solid penalty — rtk’s +7.6% was every bit as significant, just pointing the wrong way. Caveman also came out around 10% cheaper once we removed a single pricing-tier outlier, but that was a fragile number resting on one exclusion; this is the first time the cost difference has survived a paired test on a full run.

One honest qualifier, because we would want it applied to a vendor: the median saving is −10.3%, but the spread around it is wide enough that a bootstrap interval on the median just touches zero. The direction is well supported and the per-task test is clear. “Roughly 10% cheaper on this workload” is defensible; “ponytail saves you 10%” is not.

Worth noting what did not move cleanly: the input side. Re-reading its own history fell 8.4% and fresh tokens 3.9%, neither of them significant (p=0.138 and p=0.085). In part 2 we found that an agent’s bill is dominated by that re-reading, which is why a tool compressing command output barely dented it. Ponytail attacks the other side of the ledger, what the model writes, and on this benchmark that is the side the money moved on.

Finding 5 — no quality difference we can detect

The obvious worry about a skill whose whole personality is “write less” is that it gets there by deleting things that mattered. Ponytail claims it never touches validation, error handling, security or accessibility, and reports 100% safety in a separate adversarial tier of its own benchmark.

We cannot speak to that safety claim, and want to be explicit about why: SkillsBench verifiers score whether a task was completed. They are not a security, validation or accessibility suite. Nothing below tests whether ponytail preserves a guard, only whether the work still passes.

Nine tasks scored slightly worse, six slightly better, 65 identical — statistically indistinguishable. That is a null result, not a clean bill of health: this run was never powered to prove equivalence, and the data remain compatible with a small degradation as well as a small improvement. What we can say is that nothing here looks like the obvious failure mode, where writing less quietly stops the tests passing.

One small note on adherence. Ponytail’s ruleset asks the model to mark deliberate shortcuts with a ponytail: comment naming the ceiling and the upgrade path. Across 80 trials with the ruleset demonstrably in context, that happened once. The ladder gets followed; the paperwork does not.

Finding 6 — small samples lied to us, in both directions

Worth showing because it is the trap this whole series exists to avoid. Our ten-task smoke run said ponytail cut code by 3% and made things 9.6% more expensive, with mean task scores collapsing from 0.51 to 0.31. Had we published that, we would have written a very different and completely wrong article.

Verdict

Ponytail works. Across 80 paired tasks, it cut the typical bill by 10.3% and reduced code written by 15%, with no quality difference we could detect. It is the first tool in this series that clearly saved money. If you install it and forget about it, you should be modestly better off.

Do not expect the advertised 54% everywhere. Ponytail’s benchmark uses tasks with obvious over-building traps. Ours did not. In our runs, code fell 31% on larger builds and barely moved on tasks that were already lean. The more over-building your agent does, the more ponytail can cut.

Got a tool that claims to save tokens? Tell us which one and we will run it through the same benchmark.

Methodology notes

• Never trust k=1. Escalation ladder: a free transcript audit, then a 10-task smoke, then the same 10 at k=3, then the full 80. Finding 6 shows what the smoke would have told us.
• Paired analysis only. Per-task comparison between arms; any task that errored in either arm is dropped from both. Sign test for quality, per-task medians plus Wilcoxon for everything else, because one long-context session can bill 25× normal and wreck a mean.
• Endpoints fixed before the paid runs: reward, code written, output tokens, fresh input tokens, cost, turns, wall-clock. Total tokens was added afterwards, once we checked which metric ponytail’s own benchmarks/agentic/run.py actually advertises. It sums input, cache and output, so comparing our output-only figure against its −22% would have flattered us threefold.
• What a null result here does and does not mean. The quality comparison is a significance test, not an equivalence test. “No difference detected” is the honest reading; proving quality is genuinely unchanged would take a non-inferiority design with a pre-declared margin, and — for a 5-point shift in pass rate at 80% power around this benchmark’s ~40% baseline — on the order of several hundred paired tasks per arm rather than 80.
• Adoption instrumented. Every trial audited for whether the ruleset reached the model — 100% in the treatment arm, 0% in the baseline — so “ponytail saved nothing” can never be confused with “ponytail never ran.”
• Measuring code without a workspace diff. Ponytail’s own benchmark counts git diff added lines. Harbor keeps no post-agent workspace, so we reconstruct the equivalent from the agent’s tool calls: Write, Edit and shell heredocs redirected into a file, counted as non-blank non-comment lines exactly as ponytail’s benchmarks/loc.js does. This is cumulative lines emitted, not final implementation size: a line written and later rewritten counts each time. Heredocs piped to an interpreter are throwaway analysis and are excluded. We audited the extractor’s coverage on the run these figures come from: Write and Edit account for 95.6% of counted lines (15,632 and 2,496 of 18,961), so the metric is not an artifact of missing where the code went. Two things it cannot see, in both arms equally: files written by a script at runtime, and code written inside a subagent.
• Provenance. ponytail pinned at commit 16f2980 (v4.8.4, MIT); agent version pinned in both arms; the injected ruleset generated by ponytail’s own hook code, sha256 recorded. Seven of SkillsBench’s 87 tasks are excluded: one that cannot run in a local sandbox at all, and six that fail identically in both arms on our hardware. Exclusions are symmetric — a task is dropped from both arms or neither — and the full list is retained with the evaluation artifacts.
• What this cannot tell you. SkillsBench is data, analysis and repair work; it contains few of the front-end over-build traps that produce ponytail’s largest wins. This is a fair test of the cost, speed and quality claims and a conservative test of the code claim. It does not refute their −54% on their own task set.

Chart style borrowed from dither-kit, reimplemented here as a dependency-free inline widget.

For as long as we’ve taught programming, the student’s code has provided a window into the students’ thinking. Errors, the code structure, the awkward working solution—all of it showed how someone reasoned and where they got stuck.

It was never a clean window. Students have always copied, crammed, and borrowed, sometimes turning in work they didn’t fully understand. But the code still left clues. Generative AI has changed that: A finished program now tells us more about a student’s prompts than their ideas. And here’s the part that should unsettle us—often, the better the code looks, the less we can say about what the student actually learned.

This raises a bigger question: If AI can write code, should we still teach coding? I believe the answer is yes, at least for some students and situations. But that’s another topic. Here, I want to focus on the next step: If we continue teaching coding in a world with AI, how can we know if students are really learning?

Some schools have responded by trying to catch students. They use AI detectors, surveillance tools, locked-down browsers, stricter rules, and clearer honor codes. This has also led to more suspicion.

Some of these responses make sense. Teachers want to protect learning, and schools want to keep things fair. But using detection as the main way to assess students is weak. Stanford researchers found that popular AI detectors often falsely flagged writing by nonnative English speakers, with 61.22% of TOEFL essays in one study marked as AI-generated. OpenAI even retired its own AI Text Classifier in 2023 because it wasn’t accurate enough. If the company that created the tool can’t reliably detect AI, it’s probably not a good idea to base your honor code on it.

But detection isn’t the real issue. Even if we had a perfect detector, we’d still be asking the wrong question. Instead of asking, “How do we stop students from using AI?” we should ask, “How do we teach coding in a world with AI, making use of its benefits, while still being able to see if students are learning?”

Borrowing from the studio

We’re seeing this challenge with students at AET, the Arts and Entertainment Technologies Department at the University of Texas at Austin. Although my usual home is Computer Science, it so happens that AET is within the College of Fine Arts at UT, which offers many other ways to learn and assess: studio work, critique, rehearsal, revision, and performance.

In the arts, the final piece has never been the whole story. A painting doesn’t explain the choices behind it. A performance doesn’t reveal the rehearsals. A design board doesn’t show the discarded versions. A composition doesn’t tell you where the student struggled or what they finally learned to hear.

Art education has developed practices that focus on visible progress. Students bring in sketches and drafts, discuss influences, revisions, and failures, and rehearse, perform, and critique each other’s work while it’s still in progress.

At AET, we teach creative coding, which means programming to create art, design, games, or experiences. That doesn’t mean coding for poets. Our students—game designers, web developers, and programmers—start from scratch and learn advanced concepts in tools like Processing and p5.js. In the creative coding tradition, a program is often called a sketch, borrowing the term from the art world. It means something temporary, exploratory, and open to change—something you make, test, revise, and share.

So in creative coding, we were already leaning toward the studio model of sketches, experiments, iterations, and critique. Now we’re pushing that further as we rethink how we teach coding in an AI world. Here are three things we’re already using or actively developing.

Make the work public

We run the class like a studio. It’s not that work never happens at home, but the most important work needs to be seen in the classroom. Students show their code, including false starts, revisions, the choices they made, and the reasons behind them. Assignments are no longer just things you submit—they become projects you develop in public.

AI isn’t banned from the classroom. Instead, it’s treated as a helpful assistant to learn from. Students share prompts and techniques. They use AI, Google, Stack Overflow, classmates, or any other resources.

But you still need to take responsibility for your work. If you submit or present it, you must explain what the code does, why you made those choices, and how it works. If I need to ask your AI to understand your code, something is wrong. Getting help is fine, but hiding behind that help is not.

You can’t outsource to AI what the whole room watched you build.

A real studio needs students talking out loud together in the room every day. This also helps with another issue that isn’t about AI. Many people say students today are quieter than in the past. While this is mostly based on stories rather than long-term studies, these stories are common and consistent. Faculty on all types of campuses talk about silent classrooms and students who hesitate to speak up, especially since 2020.

Whatever the reason, this silence can be changed, and the solution is the same as for AI challenges: encourage students to participate. Communication is one of the most important skills in any career, including explaining ideas, defending choices, and persuading others in real time. Students don’t develop these skills by just submitting AI-guided work online. When they share their work publicly, it not only prevents AI misuse but also helps them build the skills they need most.

Invert the roles: AI as teacher and assessor

We know the usual pattern: A student asks, AI answers, and the student copies. We’ve tried to invert this. In our new approach, the AI works with the student on a set of topics, engages them in a conversation they must navigate, and ultimately assesses how well they understand the material, which leads to a grade.

This idea has a research background that goes back before ChatGPT. Teachable-agent systems like Betty’s Brain showed that explaining—even to a software agent—forces students to organize their knowledge, make connections clear, and find gaps. Our model uses this insight differently. The student isn’t teaching the bot. Instead, the student is having a conversation with it, learning, discussing, debating, and showing what they understand.

How did we do this? With fairly simple prompt engineering, we created an avatar chatbot of Vera Molnár (1924–2023), a pioneer of algorithmic art. The bot takes on Molnár’s role, drawing students into conversations about randomness, computation, generative art, and creative choices. Her practice sits exactly where creative coding students need to think: between rule and variation, system and choice, computation and visual judgment.

A system prompt sets the topics and types of questions to ask. The bot goes through these with the student, asks for more detail on unclear answers, and keeps following up until there is proof of understanding. At the end, it reviews the conversation against a rubric, giving us a clear record of which ideas the student covered, where they struggled, and how well they improved.

Besides the assessment, which is often accurate, the transcript becomes a different kind of proof, showing what a typical assignment might hide. What did the student notice? What did they misunderstand? Could they connect the concept to the code? Could they defend their choices? Could they revise their explanation when challenged?

When we switch the roles, something surprising appears: the one thing a finished submission can’t show.

A student thinking out loud.

Make understanding performative: Make students perform

Programming has never really had a tradition of performance. Musicians have it, painters have it, and dancers have it. Live coding is starting to change that.

Every semester at AET, students from different disciplines stage an algorave together—short for algorithmic rave. Audio sets, projection pieces, game demos, lasers, drones, experience design. The creative coding class brings live visuals into the live-coding tradition: Code is written and modified in real time, the screen is projected, and the audience watches the editor change as the visuals respond to the music other students are playing.

No prerender. No hiding the machinery.

The Live Coding manifesto, written in 2004 by TOPLAP, includes a line that fits every AI-era assessment conversation: “Obscurantism is dangerous. Show us your screens.” This is not just a performance ethic; it’s also an assessment strategy.

A student walks on stage. The projected screen is their editor. The room can read it. The music starts. And they build up a line of code on screen like:

osc(18, 0.08, 1.2)

.modulate(noise(3), 0.25)

.rotate(() => time * 0.1)

.out()

This is JavaScript building visuals in real time. FFTs, chained functions, higher-order manipulations. When you’re manipulating code like that on stage, you’d better know what you’re doing.

AI can help you prepare. Good. Let it.

But once you’re on stage, the question shifts from “Can you copy and paste code?” to “Can you control it?” You can paste code into a file, but you can’t paste your way through three minutes of public debugging while the whole projection turns into a beige rectangle. In a live build, understanding has nowhere to hide.

Can you read the code, make changes on purpose, and recover when something unexpected happens? That’s fluency: knowing what to do next while the system is still running.

It is very hard to plagiarize panic.

A note on assessment

So far, our results are based on our own observations. We haven’t conducted a controlled study or compared different groups, so what we have seen might just be early variation rather than patterns that apply more broadly. For now, these efforts are experiments, not final answers.

Assessment in studio and live performance settings is always subjective and focused on people. It relies on monitoring students’ progress, providing feedback, and observing how they handle challenges. We do not plan to change this core approach.

For the Molnár conversation assignment, students discussed Molnár using an AI system. The AI then created a summary and analysis of each student’s understanding. Teaching assistants reviewed this analysis, conducted their own assessments, and assigned grades. In our small experiments, the AI’s assessments using the rubric matched closely with the teaching assistants’ own evaluations.

We also used AI to help grade the end-of-term coding assignment. In this project, students improved an object-oriented game by adding strategies like heuristics, search algorithms, and learned behaviors. Since our teaching assistants had limited experience with object-oriented programming, we developed a detailed rubric and had an AI model use it to evaluate each submission. The AI’s analysis was given to the teaching assistants as support. It helped them see how each project was structured, spot important OOP design choices, and use the rubric with more confidence. The teaching assistants still made their own grading decisions. I was available as the OOP expert for any questions they could not answer. From what I observed, this substantially helped the teaching assistants understand and grade the students’ OOP design work.

More broadly, both approaches appear to enable substantive feedback at a scale that would otherwise be difficult given our current student-to-teaching-assistant ratios.

The process is the proof

We spent the first two years of the generative AI panic asking how to catch students using AI—or prohibit it altogether. Wrong question.

The real question is whether the assignment gives students a real way to show and develop their understanding. This view isn’t limited to educators. NVIDIA CEO Jensen Huang recently argued that students should not focus on finding an “AI-proof” subject. Instead, he suggested they consider how AI can help them learn more deeply and develop their skills and sense of purpose. He highlighted storytelling, creativity, design, and judgment as abilities that will stay important even as AI takes over more tasks. This supports a key idea in coding education: The aim is not to prove you didn’t use any tools, but to help students show how they think, make choices, revise, and take responsibility for their work.

These three practices are experiments, not universal solutions. They work especially well in creative coding, where code already has a public, visual, and performative aspect. But they suggest a broader principle: As finished work becomes easier to generate, assessment needs to focus more on process, explanation, revision, and mastery.

This matters outside of school too. A polished memo no longer proves there was real thinking behind it. A working prototype no longer proves product sense. A passing pull request no longer proves the developer made the change carefully and thoughtfully. AI makes production easier, so evaluation must focus more on how people think, choose, revise, and recover—in code review, hiring, and performance management. The artifact is no longer the proof. The process is.

Generative AI didn’t make assessment impossible. It just made a hidden weakness obvious. We were putting too much trust in finished work. The arts always knew better.

Show us your screens.

---

Acknowledgements

Thanks to Mike Loukides, Michael Baker, Mk Haley, Elisabeth Robson, and Honoria Starbuck for feedback on this article.

References

OpenAI. “New AI classifier for indicating AI-written text.” OpenAI Blog, January 31, 2023. Updated July 20, 2023, to note the classifier was no longer available due to low accuracy.

Liang, Weixin, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. “GPT detectors are biased against non-native English writers.” Stanford HAI, July 10, 2023.

Winthrop, R. (2026, May 27). Writing with A.I. weakens your creativity. The New York Times.

TOPLAP. “TOPLAP Manifesto.”

Schell, J., Ford, K., & Markman, A. B. (2025). Building responsible AI chatbot platforms in higher education: An evidence-based framework from design to implementation. Frontiers in Education, 10, Article 1604934. https://doi.org/10.3389/feduc.2025.1604934

Biswas, Gautam, Daniel Schwartz, John Bransford, and the Teachable Agents Group at Vanderbilt. “Technology support for complex problem solving: From SAD environments to AI.” In Learning to Solve Complex Scientific Problems, 2001.

Leelawong, Krittaya, and Gautam Biswas. “Designing learning by teaching agents: The Betty’s Brain system.” International Journal of Artificial Intelligence in Education, 2008.

Tan, Huileng. “Jensen Huang Says It Doesn’t Matter What Kids Study in the AI Era.” Business Insider, May 26, 2026. https://www.businessinsider.com/nvidia-jensen-huang-what-kids-should-study-ai-education-advice-2026-5

DAM Digital Art Museum. “Vera Molnár.” Artist biography and timeline.

Odysseus is interesting because it shows how a project with serious ambitions can now be built outside a traditional software company. 

After speaking with Ivan Mihić, a software engineer who spent time testing PewDiePie’s AI project, I ended up thinking that the software itself might not be the most interesting part of this story.

Odysseus may come from a YouTube creator with a massive audience, but the more important question is what it says about software development itself. 

Another AI app – or something more?

When Felix “PewDiePie” Kjellberg unveiled Odysseus, the internet reacted in the way it usually does when a major creator launches a software project. Some treated it as a celebrity side project, others focused on the privacy claims and some compared it with other AI tools such as GPT or Claude. 

For those of you who might be late to the party (same as we are with this text), Odysseus is an open source, self hosted AI workspace designed to run locally. Instead of depending only on cloud services, it combines local models with web search, document analysis, long term memory, and AI agents inside a single browser interface. If you want, you can also add cloud models to it as well.

Under the hood, the application integrates components such as ChromaDB for memory, SearXNG for search, and local model serving, while also letting users decide whether they want everything on their own hardware or connected to external APIs. 

What stood out to Ivan was not just what Odysseus does, but what its existence says about how software gets built nowadays:

The biggest thing isn’t that Odysseus is a good tool, It’s that a project of this size probably wouldn’t exist without AI.

A few years ago, building a system that combined models, search, memory, document pipelines, and agents would usually have required a much larger engineering team. Today, a determined individual or a small group can at least prototype that kind of product and keep iterating on it.

To me as an outsider, this looks like the phase in which we built AI tools mainly for internal use is slowly coming to an end. More and more of these tools are now being prepared for public release, even if they are still far from fully polished.

Ivan also admints that he was was surprised. Not because the tool is perfect, because it as you would expect isn’t but because something that started from a hobbyist, vibe-coding context grew into a project of this scale. For him, Odysseus is less a sign that AI has replaced engineers and more a sign that AI has changed the starting line. And I couldn’t agree more on this one.

Privacy alone won’t protect you 

Much of Odysseus’ appeal comes from privacy. Running AI models locally means sensitive data does not have to leave your machine, which is a real advantage. But Ivan pointed out that privacy and security are not the same thing. 

Odysseus is local-first, but that doesn’t automatically make it secure. 

The platform can access files, execute shell commands, perform web research, and interact with external services, therefore a bad configuration can quickly create security risks. That matters even more if users expose their instance to the public internet. 

Ivan’s point is simple: a tool can be private in design and still be unsafe in practice if people do not understand what they are connecting, exposing, or automating. 

When a tool presents itself through the lens of privacy, security becomes extremely important. Privacy without security doesn’t mean much.

Odysseus is not mature… yet

Architecturally, Ivan sees Odysseus as a large monolithic application coordinating several supporting services rather than a fully mature enterprise platform. 

It’s not the cleanest or most mature architecture I’ve seen. You can clearly tell it evolved through experimentation.

That experimental nature is visible throughout the product. Odysseus tries to combine chat, agents, research, memory, document analysis, and productivity tools into one personal AI workspace, with varying degrees of success.

A few features stood out to Ivan in particular. “Compare” lets users run the same prompt across multiple models at once. “Deep Research” automatically performs multi step web research and returns cited sources. “Cookbook” helps users choose, download, and serve models based on the hardware they actually have. 

Still, for someone who is just entering the world of local AI, Ivan thinks more established tools currently make more sense, just because tools like Open WebUI has had more time to mature.

This is not exactly a beginner-friendly tool

Talking about Odysseus once it’s installed is fine, but getting there as a regular AI user might prove to be a challenging task. From my experience, users need to be wary of the fact that, in order to fully use Odysseus, many setup steps are needed. They also need to know that they might not understand a single thing about what’s happening, which is also a risk in itself.

Odysseus is not an app you can install and run in a minute or two, while other tools do just that. The user experience for a beginner might be overwhelming and, at times, just simply not worth it if you’re in it just to experiment and see what the tool can offer.

Ivan Šimić, the third Ivan in this story, a senior writer at ShiftMag who kept experimenting with the tool to figure it out, added:

Trying to hop on the bandwagon of local LLM’s after reading about Odysseus might not be the greatest idea, since it requires a capable machine, a basic understanding of key aspects of making things happen on a PC or Mac, patience and time. For many interested in trying it, this might prove to be too much “tinkering” for not a lot of results.

Big ideas in software are no longer a big-team privilege

It would be easy to frame Odysseus as another celebrity tech experiment but that would miss the broader point. 

What matters here is that AI assisted development has lowered the cost of building ambitious software. A project that once would have required a sizable engineering team is now something a small group, or even one person, can assemble and refine much faster than before. 

And yes, while a fair portion of the pessimistic internet is claiming this means software engineers are disappearing, I highly doubt it. The job is changing, and the gap between an idea and a working product is smaller than it used to be.

As Ivan nicely put it, it’s like Jack Black’s quest for the magical guitar pick in The Pick of Destiny. In the film, that pick gives its owner superhuman guitar skills. In reality, even the best tool won’t make him or anybody else the world’s best developer, but it can certainly make everyday job a lot easier.

Therefore, the most interesting thing PewDiePie built may not be another AI assistant, but proof that ambitious software is no longer reserved for traditional software companies and big teams. Big Tech, are you scared yet?

The post PewDiePie’s AI Experiment Is What Software Development Looks Like Now appeared first on ShiftMag.