Agentic coding benchmarks are getting closer to real-world software development. For Kotlin teams, the most important question is how reliably AI agents can complete end-to-end Kotlin tasks, from reading an issue to producing a solution that passes validation.
We’re taking the first step in addressing that gap by releasing the Kotlin Benchmark, JetBrains’ official benchmark for evaluating AI coding agents on Kotlin software engineering tasks. Our goal is to give developers a credible, public way to assess how different agents perform on Kotlin and compare agent setups using tasks that are closer to day-to-day dev work.
Alongside the benchmark release, we’re publishing the benchmark assets on GitHub and launching the official leaderboard to track the evaluation results.
The first public iteration of the Kotlin Benchmark is based on the SWE-bench methodology and focuses on repository-level Kotlin software engineering tasks.
Kotlin already has strong model-focused evaluation assets, including Kotlin_HumanEval and Kotlin_QA, which help measure a model’s understanding of the language’s syntax and core concepts. The Kotlin Benchmark looks at a different layer: how well an AI coding agent can complete validated software engineering tasks in existing Kotlin projects.
The dataset features 105 engineering tasks sourced from active open-source repositories. Each task requires the AI agent to interpret a real issue description, navigate the project’s context, and generate a functional patch. Solutions are strictly verified in containerized environments, and a task is only marked as resolved when the generated solution passes the required test verification.
You can read more about our environment setup and data collection on the Methodology page.
First results
The first evaluations show that leading coding agents can complete a large share of the current Kotlin Benchmark tasks. These results reflect the first public iteration of the benchmark and do not yet include the most recent model releases. We are already working on the second iteration and will update the leaderboard as newer evaluations are added.
In this run, the top result came from Claude Code with Opus 4.7 xhigh, which resolved 90 of 105 tasks, an 85.71% resolution rate. JetBrains Junie with Opus 4.7 max (81.9%) and Codex with GPT 5.5 xhigh (81.9%) followed closely.
The full leaderboard is available on kotlinlang.org/benchmark, where you can compare agents and configurations in detail.
Results shown here reflect the first public iteration of the Kotlin Benchmark. The leaderboard will be updated as newer model evaluations are added.
For teams evaluating coding agents, the benchmark provides a shared frame of reference for comparing setups on Kotlin tasks instead of relying only on vendor claims. The scores are intended as a signal, not a guarantee for every codebase. Real-world results depend on your architecture, internal APIs, coding standards, tooling, and validation process.
What’s next
We value an open approach, which is why we built this benchmark on the open-source Multi-SWE-bench infrastructure and made all datasets and test harnesses publicly available.
We treat benchmarks as a continuous quality measurement pipeline. Moving forward, we plan to expand the framework in these areas:
Broader Kotlin ecosystem coverage: We want the task mix to better reflect how Kotlin is used in practice, including areas such as Android and Kotlin Multiplatform, and cover a wider range of task difficulty levels.
More evaluation metrics: Passing tests is a useful correctness signal, but it is only one part of agent evaluation. Future iterations will look at cost, performance, maintainability, and code quality.
More agents and model setups: We plan to evaluate more commercial agents, agent-model configurations, and open-weight models, so teams can compare a wider range of setups.
The benchmark is open, so you can inspect the tasks, compare results, and tell us which Kotlin scenarios we should cover next.
Why preserve a cookbook? In the second episode of our special six-part series on Vanishing Culture, host Vida Vojić speaks with Katie Livingston, a doctoral researcher at Stanford University who studies domestic culture and women’s literature. Through the lens of family cookbooks, recipe collections, and food traditions, Katie explores why everyday cultural artifacts deserve preservation and what they can teach us about history, identity, and community.
The aftermath of Apple's price hikes. Apple is taking its fight against Epic to the Supreme Court. More AI features coming to Apple's Creature Studio. And Apple is showing confidence in its upcoming iPhone Fold, expecting to sell 10 million units!
America is having MacBook sticker shock.
MacBook price hikes expected to contribute to 13.6% drop in global laptop shipments.
Apple weighs buying RAM from two blacklisted Chinese suppliers to curb rising costs.
Broadcom and Apple extend custom silicon pact to 2031.
iPhone 18 Pro leaks: Qualcomm or Apple C2 model, A20 details, camera upgrades.
Apple takes Epic fight over app store fees to the Supreme Court.
Tim Cook's government liaison position comes into focus before stepping down as Apple CEO.
Apple in Russia's crosshairs again, facing $52M fine for not installing state-required apps.
If you wanted more AI in Apple's Creator Studio, Tuesday's update gives it to you.
Safari's new MCP server lets coding agents inspect and debug websites.
Siri AI can pull info from third-party apps in the latest developer beta.
Apple to launch 5 new iPhone models to gain market share amid memory crunch.
Confident Apple increases its iPhone Fold orders to 10 million.
Apple TV teases major new sci-fi series: Neuromancer.
iPhone 17 Pro Max buried in America's 250th anniversary time capsule: to be opened in 2276.
Join Club TWiT for Ad-Free Podcasts!
Support what you love and get ad-free audio and video feeds, a members-only Discord, and exclusive content. Join today: https://twit.tv/clubtwit
How can Azure Policies help you? While at Techorama in Belgium, Richard sat down with Barbara Forbes to discuss how Azure Policies have evolved and the techniques sysadmins are using to improve security, cost controls, efficiency, and more. Barbara talks about how the default policies are designed to get folks started in Azure quickly - not necessarily optimally. And there are plenty of policy templates out there, but before you implement them, it's worthwhile to review each policy and ask the question "why?" Keeping good documentation on policies makes it easier to know intent, especially when it comes to changing them - and you'll need to change them! There are a number of ways to apply policies, but in the end, they are just more Infrastructure-as-Code, and so easily repeatable. Azure Policies are there to help you provide freedom with guardrails if you implement them carefully!
Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesday at 7am PT. Older video versions available there too.
Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to our friends of the show list, we'll never share it.
war - new archive format spec from Astral (same team as uv/ruff), v0.0.2, still no binary encoding defined yet
Header-Index-Store layout: header IDs the file, index maps names to store offsets, store holds compressed data
Index uses a finite-state transducer (FST) to dedupe common path prefixes across entry names
Supports three entry types (file, directory, link) and three compression modes (store/DEFLATE/zstd), plus an "executable" metadata flag
Unpacking is atomic - writes to a temp dir, then renames into place, so a failed extract never leaves a half-unpacked directory
Strict name-segment rules (no NUL/control chars, no leading/trailing whitespace, blocks Windows-reserved names like CON/PRN) to avoid path traversal and cross-platform footguns
Michael #3:Hermes Agent: The AI agent that grows with you
Hermes Agent is an open-source, Python-built AI agent framework from Nous Research - think ChatGPT-style assistant, but connected to your tools, files, shell, browser, calendar, memory, and messaging apps
I’m using it in Discord as a long-running agent conversation, not just a one-off chatbot session
Hermes can connect through a gateway to platforms like Discord, Telegram, Slack, WhatsApp, email, webhooks, and more - so the same assistant can follow you across surfaces
In my setup, I can send Hermes voice/text from Discord, keep project context across turns as threads, and ask it to actually do things: read GitHub repos, run commands, edit files, schedule calendar events, generate drafts, and verify results
A fun workflow: I can trigger one-shot actions from an Apple Watch shortcut - dictate a request, send it to Hermes, and have the agent execute it asynchronously
Hermes has persistent memory, so it can remember durable preferences and facts - for example, how I like my research formatted
It also has “skills,” which are reusable procedures the agent can load later, so Hermes can self-improve over time instead of rediscovering the same workflow repeatedly
It supports scheduled jobs / cron-style automations, so it can proactively watch for releases, send summaries, run checks, or remind you about things
It’s provider-agnostic: OpenRouter, Anthropic, Google, xAI, local models, Nous Portal, and others
The big idea: Hermes turns an LLM from “a chat box I visit” into “an agent I can reach from anywhere that knows my workflows and can take real actions and learns over time.”
Simon Willison built a Claude/Codex-style coding agent on top of his llm library, using an alpha of the llm package plus his python-lib-template-repo
Built almost entirely via prompted TDD - asked an agent to write a spec.md, then commit + implement with red/green tests, occasionally hitting a real OpenAI key to sanity-check
Shipped to PyPI as an alpha: uvx --prerelease=allow --with llm-coding-agent llm code
Also exposes a Python API - CodingAgent(model="gpt-5.5", root=..., approve=True).run(...) - which Simon didn't ask for but got anyway
Demo: llm code --yolo told GPT-5.5 to build a SwiftUI CLI clock; model correctly noted SwiftUI isn't really CLI-friendly and still produced an ASCII-art time display
In this article, we explore several game engines and frameworks, highlighting some of the key advantages each of them offers. After reading this article, you will have several options to choose from to make the next big hit. Note that we only focus on the 2D aspects of these platforms. Let's get started.
1. Cocos2D
Cocos2d is a free, open-source framework, compatible with both Swift and Objective-C. It has support for iOS and OS X, and it also supports Android through the SpriteBuilder Android plugin provided your code is written in Objective-C. Swift support for Android is in development.
Projects in Cocos2d are created through SpriteBuilder, a graphical design environment that you can use to quickly prototype and build games. You are not required to use SpriteBuilder to build your game, however, project creation must be done through SpriteBuilder.
Scene management is done through the CCDirector class, which can utilize numerous Transitions with the CCTransition class. It offers animations through the CCAnimation class and actions, such as move, scale, and rotate, with its CCAction class. Cocos2d has support for particle systems with the CCParticleSystem class and has support for tile maps with the CCTiledMap class.
Cocos2d also uses OpenGL as its rendering engine. It uses Chipmunk as its physics engine. If you enjoy Objective-C or Swift, and you want to develop for multiple platforms (iOS and Android), then Cocos2d is a framework worth considering.
2. Cocos2d-x
Cocos2d-x is a C++ port of Cocos2d that can compile to many other platforms. Cocos2d-x can compile to iOS, Android, Windows Phone, OS X, Windows, and Linux.
Cocos2d-x does not come with SpriteBuilder. However, it does have some interesting projects, such as Cocos Creator, which provides the framework, scene editing, debugging, game preview, and publishing to multiple platforms.
3. Unity
Unity is a mobile game engine that supports C# and UnityScript, a language designed specifically for Unity and modeled after JavaScript. It is cross-platform and can deploy to many platforms, including PlayStation and Xbox. There is a free edition available and a professional edition offering more features. More information is available on the Unity website.
There is a built-in sprite editor in which you can visually slice your images and an animator window that allows you to lay out and organize animations and add key frame animations to sprites. It is also possible to visually design particle systems within the Unity editor.
Unity uses its own proprietary physics engine. Unity has a built-in camera, which makes navigating in the game world simple. You can visually lay out scenes and Unity has a user interface system with components, such as buttons, sliders, and menus. There is also support for sprite packing within the Unity editor.
Unity is a powerhouse when it comes to game engines. If you need a cross-platform solution that has the ability to a wide range of devices and you prefer to program in either C# or UnityScript, then Unity is a good choice. One other nice thing about Unity is the asset store in which you can find or purchase assets you can use in your games.
4. SpriteKit
SpriteKit is Apple's proprietary 2D game development framework. It is only available on iOS and OS X. SpriteKit supports both Swift and Objective-C, a combination of the two.
Scene management is done with the SKView class and there is a wide variety of transitions with the SKTransition class. At the heart of SpriteKit are actions, instance of the SKAction class, which are used to move, rotate, and scale games objects. Actions can also be used to play sounds and execute custom code.
SpriteKit has a scene editor in which you can visually design the levels of your game. Also available is a particle editor in which you can visually design particle systems. A number of predesigned particle systems are available, such as fire and rain.
SpriteKit uses Box2D for its physics engine. SpriteKit provides a friendly wrapper around Box2D, which makes it very easy to use. SpriteKit also has a built-in camera through the SKCameraNode class, which makes navigating around the game world easy.
If you are tied to the Apple ecosystem, then SpriteKit is a very good choice.
5. Titanium
Titanium is a framework that uses JavaScript as its programming language. While more traditionally used to build applications instead of games, it can be a viable game development platform. It is free to use during development. If you want to publish your game, then you need a license.
Titanium supports many native controls from iOS and Android. This makes it easy to animate views and images. I have built a Blackjack game, complete with animations and a nice card flipping effect using Titanium.
One of the nice things about Titanium is the market place for downloading custom modules. There is, for example, a Box2D module that you can plug into your app. If you do not need the complexity of Box2D, then using a simple bounding box collision system can work just as fine.
While Titanium would not be ideal for games with a lot of moving parts and particle explosions, using it for simple card games, trivia games, and games with a limited number of moving objects is more than a viable option.
6. PhoneGap
PhoneGap allows you to take your existing HTML, CSS, and JavaScript skills and package them up as a mobile app. The way it works is by embedding a web view in the app and loading your assets into that web view. This means that you can use the Canvas API to create games.
PhoneGap allows access to native APIs, such as the device's camera, microphone, and, most importantly for game development, the accelerometer.
At its core, PhoneGap is pretty straightforward. It takes an HTML-driven app and packages it up as a mobile application using a web view.
We will take a look at two other JavaScript libraries that you can use to create a Canvas/WebGL game, Phaser and CreateJS.
7. Phaser
Phaser is an open-source JavaScript framework for writing 2D games. It uses Canvas or WebGL if available.
Phaser has support for animations, particles, and tweens to move, scale, and rotate sprites. It has built-in support for tile maps and uses three different physics engines, depending on your needs. It also has a built-in camera, which makes it simple to navigate your game world.
There are some plugins available for Phaser, including a particle system designer and a Box2D plugin. If JavaScript is your language of choice, then Phaser, in combination with PhoneGap, could be a good choice.
8. CreateJS
CreateJS is a suite of four different open source libraries:
EaselJS is a library based on the Canvas API and it can use WebGL if available. TweenJS is focused on tweening elements while SoundJS is aimed at audio playback. Last but not least, PreloadJS helps preload assets.
EaselJS has support for sprite sheets and filters that you can place on your elements. There is also a Shape class should you need to do vector drawing. Moving, scaling, and rotating is done by incorporating the TweenJS library into your game.
While CreateJS doesn't have many APIs that are tailored to game development, it is written in JavaScript so you can incorporate any extra JavaScript libraries you want, such as a JavaScript port of Box2D. One library I had success using in the past is a collision detection library for EaselJS.
I have written a couple of articles on CreateJS on Envato Tuts+. If you are looking for a detailed introduction to these libraries, then you may find these interesting:
Unreal Engine is a game engine that uses C++ as its programming language. It is free to use with limits. For more details, visit the Unreal Engine website. You can deploy your game to iOS and Android.
Unreal Engine includes a too, Blueprint Editor, that allows you to visually script your applications without the need for programming. The editor allows you to design game levels, add physics simulation, user interface, animations, visual effects, and more, in a visual manner.
10. Corona SDK
The Corona SDK is a software development kit that uses Lua as its scripting language. It is available on Windows and OS X. You can develop your games and apps for free with the Corona SDK. Visit their website for more information about pricing.
The Corona SDK exports to iOS, Android, Kindle, and Windows Phone 8. Windows and OS X exports are in private beta and will be coming soon.
Corona Labs offers some other attractive features, such as Corona Editor, a Sublime Text plugin that includes a debugger, code completion, and a few other nice options. Composer GUI, available on OS X, gives you a graphical environment in which you can design your game's levels and see how the objects interact with each other using Corona's physics engine.
The Corona SDK uses OpenGL as its rendering engine and Box2D for physics. Its physics engine provides a wrapper around Box2D, which makes its easier to use. The Corona SDK includes over a thousand APIs you can use to build your games.
It also has built-in scene management through its composer library and offers a great transition library that helps move, scale, and rotate your game's graphical assets.
Unfortunately, the Corona SDK doesn't have the option to build sprite atlases and using complex shapes for physics collision detection can be difficult. However, there are some great third party applications that can ease this pain, such as Texture Packer and Physics Editor.
Overall, the Corona SDK is a great SDK to explore if you are looking for a cross-platform solution and want the ease of the Lua scripting language.
11. Gideros
Gideros, like Corona, uses Lua as its scripting language. Gideros is free and open-source, and it can compile to iOS, Android, Windows Phone, OS X, Windows, and Windows RT.
While Lua is not an object-oriented programming language with a class-based system, Gideros provides a class-based API, allowing developers to write clean and reusable code.
Gideros has a scene manager with all kinds of transitions, a MovieClip class for animations, and provides tweening through its GTween library, which allows you to move, scale, rotate entities and more.
Gideros, like Corona, uses Box2D for physics, although the Box2D API of Gideros is a little more close to the original Box2D API and therefore slightly more difficult to use.
It provides some extra tools like a texture packer tool and a font creator, which can be found in the installation's directory. Gideros allows you do instant testing of your projects on your mobile device through a Wi-Fi connection.
As with Corona, this is a great platform if you are looking for cross-platform development with the ease of the Lua programming language.
12. AndEngine
AndEngine is a mobile development framework for Android. It is free and open-source. However, as the name hints at, it is only for Android. AndEngine, like many other mobile platforms, uses OpenGL as its rendering engine and it uses Box2D as its physics engine through an extension.
There is no dedicated scene management in AndEngine, but you can find plenty of example on how to build a simple scene manager. If you are looking to only release an Android app and Java is your language of choice, then this framework is a good option.
13. libGDX
libGDX is another Java game development framework. It is cross-platform and can compile to Windows, OS X, Linux, Android, iOS, Blackberry, and HTML5. libGDX also uses OpenGL as its rendering engine and it leverages Box2D for simulating physics.
libGDX has a Camera class, an Action class to move, scale, and rotate objects, and aParticleEmitter class for displaying particles. It has support for tile maps via theTiledMap class, which can be used with the Tiled editor.
If Java is your preferred language and you are looking to create a cross-platform game, then libGDX is a good framework to look into.
Conclusion
In this article, we went over several game development frameworks and engines. You should now have a better idea which solutions best fit your needs and that of your next game. Thanks for reading and I hope you found the article useful.
Log in
