|
Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
|
Back for its 9th year, Maker Faire Long Island is settling in to its new home on the SUNY Stonybrook campus. This year’s event “Maker Faire Long Island is where creativity becomes something visitors can see, touch, build and experience,” said Lisa Rodriguez, Co-Producer of Maker Faire Long Island and Director of Digital Marketing for […]
The post Maker Faire Long Island #9: Saturday, June 6th appeared first on Make: DIY Projects and Ideas for Makers.
Greg Wilson has noticed that lots of folks are using dodgy metrics to figure out if AI tools are worth their costs.
Would you measure lines of code generated, or tickets closed? Or would you send out a survey asking whether developers feel more productive? Each of those approaches is flawed in a different way;
He lists lots of common metrics, and why they are flawed. Sadly he doesn’t give any suggestions on what would be better. In my view, since we cannot measure productivity, any metrics are weak evidence at the best of times.
I do somewhat use one of his flawed measures: “Asking Developers If They Feel More Productive”. While I acknowledge the problems he gives with this measure, I find that in an environment where decent measures are hard to find, even such a dim light is the best we have. In this situation these kinds of qualitative metrics may not be conclusive, but they are useful.
❄ ❄ ❄ ❄ ❄
Benedict Evans observes that extensive automation didn’t mean the demise of professions in the past.
we spent a century automating accounting: we built calculating machines, punch cards, mainframes, data processing, databases, PCs, spreadsheets, ERPs, cloud… in fact, we built half of the tech industry around automating this. Yet the number of accountants kept going up.
He goes into the myriad of problems that exist when we’re trying to forecast the impact of a technology on jobs. There’s the much-talked-about Jevons paradox - once something becomes cheaper, people do it more, which can increase demand. Often this leads to the nature of jobs changing, even if it’s called the same thing.
Accountants today aren’t doing exactly the same work that they did in 1970 or 1980 ‘but more’ - they’re still called ‘accountants’ but the job is different. New technology often starts out being used for ‘the old thing but more’, but it rarely ends up like that.
Technologies often affect whole businesses - consider the impact of the internet on news publishing. Did anyone observing the rise of smart phones in the early 2000s realize that a consequence of this would change the economics of taxis due to the rise of ride-sharing apps? The conclusion is that it is, at the very least, almost impossible to forecast the impact of AI on our work.
❄ ❄ ❄ ❄ ❄
Stephen O’Grady looks at how closed and open models have performed on benchmarks over time.
Closed models are setting the pace of innovation, and constantly breaking new ground from a capabilities standpoint. Open models are chasing them, and the cycle times seem to be getting shorter. There are no clear capability moats, and what is frontier today is table stakes tomorrow.
It tooks 13-18 months for open models to catch up to GPT-4 on these benchmarks, but only 2-7 months to catch up to GPT-4o.
There’s a bunch of caveats to this analysis, that he lists, but it’s a worthwhile survey of how various kinds of models perform against the various measures we are trying to assess them with.
❄ ❄ ❄ ❄ ❄
One of the starkest examples of sloppy AI use is hallucinated citations - a give-away of both usage of LLMs and carelessness driving them. GPTZero is a company that makes tools to detect AI writing. I’ve no insight as to whether their tool is effective or not, but they do publish investigations of AI usage, and have published several articles highlighting hallucinated citations. One post focuses on Ernst & Young Canada’s report on cyber threats to loyalty systems and found that more than half its references were hallucinations. The post uses a lot of extremely annoying animations in how it presents its information (breaking Safari’s reader mode in the process). But the harm that these kind of AI generated reports can do goes further than just some misled humans:
Publishing a report online is essentially a form of data injection into the pool of knowledge that is the internet. When the report includes fake information (either vibed citations or false claims) it can “poison the well” by misleading future researchers, especially if the report is published by a well-known consulting firm and hosted on a high-traffic website.
❄ ❄ ❄ ❄ ❄
As LLMs get more capable in programming, we are rightly worried that people will use them attack software systems. But these models can also be used for defense, allowing teams to find bugs before attackers do. Some folks from Mozilla posted an article on how they’ve used AI model to identify and fix an unprecedented number of latent security bugs in Firefox.
Just a few months ago, AI-generated security bug reports to open source projects were mostly known for being unwanted slop. Dealing with reports that look plausibly correct but are wrong imposes an asymmetric cost on project maintainers: it’s cheap and easy to prompt an LLM to find a “problem” in code, but slow and expensive to respond to it.
It is difficult to overstate how much this dynamic changed for us over a few short months. This was due to a combination of two main factors. First, the models got a lot more capable. Second, we dramatically improved our techniques for harnessing these models — steering them, scaling them, and stacking them to generate large amounts of signal and filter out the noise.
During 2025, there were 17-31 security bugs fixed each month. In April 2026, they fixed 423.
❄ ❄ ❄ ❄ ❄
Pavel Voronin riffs on Unmesh Joshi’s post on What is Code. He observes that cruft in a codebase (technical debt) has always added friction to software development. But the consequences of this cruft are compounded when LLMs are using existing code as context for future work.
In a degraded codebase, the model does not see “technical debt” as debt. It sees examples. It sees precedent. It sees a style to continue.
LLMs multiply what’s currently happening. I hear reports that good code might take the place of much of what’s put in markdown, because LLMs will imitate what’s already in the code base. But bad code multiplies too. Inevitably he introduces another variation of rampant debt metaphors:
Cognitive debt accumulates when a team uses abstractions it no longer understands. Generative debt accumulates when a codebase contains confused concepts that models are likely to continue.
Cognitive debt is about what the team no longer understands. Generative debt is about what the model is now likely to reproduce.
❄ ❄ ❄ ❄ ❄
Jason Koebler, from the very worthwhile 404 media, has written a plaintive essay on how AI-generated slop is driving us crazy. Not just because its filling the web with this slop, but also because how it’s making us humans react to slop and the threat of slop. We review our own writing and notice: it’s not just reading AI slop that hurts us, it’s the risk that we write something that looks like AI slop. If I use phrasing that AI copied from me, does it seem like I’m copying AI? This has led to the appearance of “humanizers” - AI tools that make our writing look less like AI.
Humanizers add typos, randomly replaces words, removes “AI tells,” and sometimes inserts random characters.
It’s another step on the way to the Zombie internet:
I called it the Zombie Internet because the truth is that large parts of the internet are not just bots talking to bots or bots talking to people. It’s people talking to bots, people talking to people, people creating “AI agents” and then instructing them to interact with people. […] It’s my email inbox, in which I used to occasionally get poorly-formatted, poorly written, extremely long emails from delusional people who were positive the CIA had imprisoned them in a virtual torture chamber using undisclosed secret technology but where I now get well-formatted, passably written, extremely long emails from delusional people who are positive they have proven AI sentience and have the AI transcripts to prove it.
❄ ❄ ❄ ❄ ❄
Andy Osmani points out that spawning lots of agents is like launching a bunch of parallel processes that all rely on a single orchestrating thread - yourself.
Python has the Global Interpreter Lock (GIL). You can spawn as many threads as you want but only one executes python bytecode at a time because they must acquire the lock. You are the GIL of your AI agents. They all can run at once. But when any of their work needs genuine understanding of the architecture or resolving merge conflicts, that work has to acquire the lock. There is one lock. You hold it.
This means you must design the workflow with the agents with that GIL in mind. You shouldn’t launch more agents than you can properly review. It’s handy to separate background tasks that can be offloaded to an agent from complex tasks that require applied attention. Don’t use that precious brain for things that the machine can verify itself. [And I’d add - do get the machine to build tools that ease human verification. For example, it’s better to surface test case data in tables rather than buried in assert statements.]
Spawning agents is not the skill. Anyone can run 20.
The real skill is designing the system around the one serial resource that cannot be cloned or parallelized. That resource is your attention.
❄ ❄ ❄ ❄ ❄
Jamie Hurst is a Principal Engineer at booking.com, where he works in developer experience with a focus on AI tooling. He’s written realistically about the gains and losses of using LLMs in this work.
The cost of building has collapsed, but the cost of aligning organisationally has not. If anything, it’s gone up. When three different teams can each produce a working solution to the same problem in the time it used to take to write a proposal, the bottleneck moves from engineering to coordination.
He thinks he’s able to do more as a senior engineer, but is concerned about how sustainable it is, both for him personally and for the organization he works for. He’s able to shape directions for multiple workstreams at once, in a way that he couldn’t three years ago. But one loss is that he doesn’t have enough time for mentoring, which will exact a toll on his employer in the longer term. He also finds he doesn’t have enough time to think.
The productivity gains from AI got captured by output volume rather than output quality. The org’s expectations rose to absorb the speed-up, and the slack that used to exist between tasks, the unstructured time where strategic thinking actually happens, got eaten first because it’s invisible on a dashboard. I’m at a point in my career where thinking is supposed to be most of the job, and most of it now happens on holiday because the working week doesn’t accommodate it.
This article is from Making AI Work, MIT Technology Review’s limited-run newsletter examining how to apply LLMs across industries. To receive it in your inbox,sign up here.
From accounting to design to market research and product development, there’s a staggering breadth of skills needed to run a business. A large company can hire experts to handle these tasks, but small businesses don’t always have this luxury.
That’s where AI comes in. Today’s AI models do a decent job at these tasks. The trick for small businesses is to understand where AI is good enough and where it’s not.
One place where a “good enough” AI can already be quite valuable to small business owners is in providing secretarial skills and handling basic administrative matters. Let’s take a look at how one private tutor is using it to improve his recordkeeping and free up his time.
Sam Finnegan-Dehn works in fundraising for a charity, but he moonlights as a math and philosophy tutor for university students from his home in London. Through this part-time business, he can leverage his degrees in philosophy and share his love of the subject with clients.
But meeting with students is only a fraction of the work it takes to be a good tutor. He also plans lessons and finds fresh reading materials, creates assignments, sends invoices, and keeps up with new research—all on top of his regular job. Given these demands, Finnegan-Dehn doesn’t have as much time as he’d like to grow his tutoring roster.
So he’s turned to AI for some help in managing the day-to-day aspects of his business. He says AI has taken on a secretarial role across all of his digital notebooks, where he jots down reminders about his clients’ progress and new readings to keep himself up-to-date. He describes using AI as kind of like having a second memory that helps him connect ideas he’s written down in various places.
While he has experimented with different tools like Claude and ChatGPT, he’s now landed on Notion AI because it integrates better with his tutoring notes, which live across his notebook tabs in the Notion app. Finnegan-Dehn doesn’t use AI to create teaching materials, but he does let Notion AI record meetings with his clients (after getting their consent), and then uses its automated summaries to refine his teaching strategy. For example, if he notices from the AI’s summary that it seems like a certain technique was not helping a student, he may change how he approaches the subject next time.
Beyond this, Notion AI also helps him with goal-setting, drafting lesson notes, invoicing, and generating and syncing social media posts. For goal-setting, for example, Finnegan-Dehn says he understands his long-term goals for his business but not always the concrete steps to build to them. He uses AI to help fill in these gaps. He starts by writing down a “North Star” goal—say, to have a certain number of clients by the end of the year. Next, he asks his AI to generate the steps that he needs to take to get there, given the profile he has built up in the app. Then, he can reflect on the results and choose which tasks to tackle first.
Notion has been a big player in note-taking software for many years. Its AI add-on, released in late 2023, now has tools that enable it to interact with many other online productivity platforms. There’s an email client, calendar integrations, and a newly released agent. And while this level of access has raised privacy concerns, it can also make for a pretty powerful virtual assistant.
Many of the tasks targeted by Notion AI are less creative and more rote: syncing information across documents or searching through old scribbles, for example. This makes the tool especially appealing to small business owners, who have limited bandwidth, particularly for menial work.
Other companies are developing tools targeted at specific industries. For example, Grandma’s Quilt Shop in Yuma, Arizona, uses Rain, which has a software suite tailored to craft companies, to generate inventory descriptions and pricing for its stock of fabric designs. The owners claim this AI tool cuts the time it takes to list items by 60 to 80%.
There are drawbacks, though, as Finnegan-Dehn described some of Notion AI’s idiosyncrasies as “clunky” at times. And the AI add-on for Notion costs $20 per month. As with all new tools, small business owners should carefully assess how the potential gains and headaches measure up against the cost of just doing the job themselves.
Consider these points when thinking about whether AI might be able to help you run a business, or make any part of your work life just a little bit easier.
Sign up for Making AI Work, MIT Technology Review’s limited-run newsletter examining how to apply LLMs across healthcare, climate tech, education, and more.
We are excited to announce the General Availability (GA) of the AWS IoT Device SDK for Swift. This release gives Swift developers a production-ready SDK with stable APIs and integrated service clients to connect applications to AWS IoT Core.
The GA release now provides easy-to-configure service clients for three essential AWS IoT Core services:
The SDK supports macOS, iOS, tvOS, and Linux, with X.509 certificate-based authentication and TLS 1.3 encryption on iOS and tvOS. For a detailed overview of platform and security capabilities, see Introducing the AWS IoT Device SDK for Swift (Developer Preview).
The SDK provides service client samples (Device Shadow, Jobs, and Device provisioning) on the GitHub website. The following walkthrough demonstrates how to set up a Shadow client and retrieve a shadow state.
Before you start with the service client, set up the required AWS IoT resources: For more information, see What is AWS IoT? and Create AWS IoT resources.
After you complete these steps, you will have three items required for client configuration:
The IoT Shadow client is a product within the aws-iot-device-sdk-swift package. Add the package as a dependency and reference the Shadow client in your target’s Package.swift file:
Log in
let package = Package(
name: "MyApp",
dependencies: [
.package(
url: "https://github.com/aws/aws-iot-device-sdk-swift.git",
from: "1.0.0"
),
],
targets: [
.executableTarget(
name: "MyApp",
dependencies: [
.product(name: "IotShadowClient", package: "aws-iot-device-sdk-swift"),
]
)
]
)Before you create a Shadow client, create an MQTT 5 client. This example uses the endpoint and certificate files from the Prerequisites section.
// Create an Mqtt5ClientBuilder configured using a certificate and private key
let clientBuilder = try Mqtt5ClientBuilder.mtlsFromPath(
endpoint: self.endpoint,
certPath: self.cert,
keyPath: self.key)
// Create the MQTT5 client using the Mqtt5ClientBuilder and start a connection session
let client = try builder.build()
client.start()
For more information about the certificates, see X.509 client certificates.
After you start the MQTT 5 client, configure the client options and create the Shadow client. These options to cap the client’s subscription usage and reserve capacity for other parts of your IoT application.
Configure the following options based on your application’s subscription needs:
maxRequestResponseSubscription: Maximum number of concurrent subscriptions that request-response client uses. Each request usually uses 1-2 subscriptions until completion.maxStreamingSubscription: Maximum number of concurrent streaming operation subscriptions that the client will allow. Set based on the number of streaming operations you plan to use simultaneously.operationTimeout: Request timeout in seconds.The following example creates a Shadow client with values that work well for applications that use a single shadow with limited streaming operations. Adjust these values based on your application’s needs:
// Set up options for the MqttRequestResponseClient
let options = MqttRequestResponseClientOptions(
maxRequestResponseSubscription: 3,
maxStreamingSubscription: 2,
operationTimeout: 5)
// Create a Shadow client using the MQTT5 client and the options created above
let shadowClient = try IotShadowClient(mqttClient: client, options: options)With the Shadow client ready, you can perform operations such as retrieving, updating, or deleting a shadow state. The following example retrieves a named shadow state:
let request: GetShadowRequest = GetShadowRequest(thingName: inputThingName)
do {
let response = try await shadowClient.getShadow(request: request)
} catch {
// Log errors
}
For more information, see the Shadow sample on the GitHub website. Additional service client samples are also available on GitHub:
The SDK has the following known limitations:
For updates on these limitations, see GitHub Discussions.
In this post, we showed you how to get started with the AWS IoT Device SDK for Swift from adding the SDK dependency to performing shadow operations. The SDK also includes clients for Jobs and Device Provisioning. Use the Jobs client to manage remote device operations or the Device Provisioning client to automate certificate creation at scale.
For more information, see Getting started with AWS IoT Core tutorials and connect your first device to AWS IoT Core.
Let us know how you’re using the SDK in the comments. You can also:
