You’ll be able to securely link and interact with your go-to services directly in AI Mode.

NotebookLM is now Gemini Notebook: the same standalone product with deeper Google integration and a secure cloud computer.

The tech industry is measuring AI productivity all wrong, and Mozilla MLOps engineer and University of Chicago instructor Chelsea Troy makes a strong case for why. The real opportunity, she argues, isn’t shipping more code faster but finally having the bandwidth to run the experiments, tests, and simulations that engineering teams have always wanted to run but never had time for. Chelsea joined Ben to cover the state of entry-level hiring, why the software engineering interview has been broken for decades, what it means to teach Python in 2026, and why token efficiency should replace token consumption as the industry’s dominant productivity metric.

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts.

Transcript

This transcript was created with the help of AI and has been lightly edited for clarity.

00.31
Ben Lorica: All right. So today we have Chelsea Troy. She’s part of the machine learning operations team at Mozilla. And she’s also developing a bunch of courses for O’Reilly around agentic coding skills. Chelsea, welcome to the podcast.

00.47
Chelsea Troy: Thank you for having me.

00.49
All right. So two things that pop out there: agentic coding and skills. So first of all, agentic coding. Chelsea, so you personally, to what extent are you using any of these agentic coding tools.

01.06
Sure. So I think that. . . I have sort of a number of different jobs that I do. I work, as you mentioned, as a machine learning operations engineer at Mozilla, where I help machine learning engineering teams get their work to production. And then I also teach at the University of Chicago, and I teach a machine learning class within the set of courses that I teach, in addition to some of the stuff at O’Reilly.

So in all three of those areas, I find myself needing some expertise in agentic coding, not, like even in addition to specifically whatever I might be doing with it, because a lot of my colleagues or my students are using it, and it’s important for me to understand how it works, because I need to be able to advise on that, and I need to be able to assist with that.

01.55
So right now, for example, at Mozilla, we are exploring the extent to which agentic coding suits our values, to which, the extent to which agentic coding suits our, like, workflow, the kinds of things that we are trying to do, particularly internally. But, actually the places where I’ve seen it most in the places in which I have found myself needing to develop the most nuanced takes on agentic coding come from the work that I’m doing with my students, because I have these students, the graduate students in computer science, and they are trying to figure out how to navigate early career software engineer type of roles.

How are they going to apply to them? How are they going to be evaluated for them? How are they going to succeed at them? How are they going to be promoted out of those roles? And I think that they have a lot of questions about those things that are coming to me. They want to know the answers to these questions, and these are not questions that I naturally have experience to answer, because at this point, I’ve been a software engineer for the better part of two decades.

The last time that I applied for a role was many years ago. The last time that I applied for an entry level role, things were so drastically different than what these students are experiencing now. And so I find myself doing a lot of my research, a lot of my implementation, a lot of my experimentation towards this end of understanding how this is going to work for them, how can students expect to learn now? What are students going to be expected to know? What our entry level engineer is going to be expected to know? What are companies expecting of entry level engineers now, and what is it going to mean for them to have people advance in skills as these tools are available and with the expectation that these tools are going to be available for students. So, a lot of what I do is around figuring out how to answer those questions right now.

03.57
All right. I have lots of questions before, but before I do that, a quick shout out to the University of Chicago, where I have friends on the faculty, Mike Franklin and Bob Grossman in particular. All right. So I assume, Chelsea, that, the difference between the people graduating this year, 2026, and the people who graduated last year, 2025, as far as interesting expectations around agentic coding tools, there’s a big difference, right?

04.30
I think so, and I think that part of that is that over the past year, we’ve seen a great deal of development in these products specifically for programming uses. And I would say that my specialization within the use of these tools is pretty much exclusively their use on programming and then data visualization projects. I would say that outside of that, my expertise peters off very quickly, but I’ve spent a lot of time on the intersection of these tools and learning on these tools and completing the tasks that people are expected to complete inside of a workplace, and what that means inside of the more holistic view of what needs to get done on a team.

But I would say that in 2025, students still. . .and this is a verification and sort of their cycle of work is still very important for them to maintain a very firm handle on. But in terms of the results that they’re able to get from using an agentic tool, for example, on completion of a project they might be doing for their academic degree, they’re having a lot more success now than they were a year ago, which raises, interesting questions about what they need to be doing by hand, whether we can verify that they’re doing it by hand. But I think also more broadly and perhaps more importantly, like what do they need to be keeping in mind while using these tools? What are the values for them to take forward as they’re using these tools? And what skills are important for them to make sure that they’re developing? And to what extent can we support them in building those skills and verify that they’re building those skills?

06.02
So I am assuming the class you taught in 2025 is very different from the class you taught in 2026, which might be also very different from the class you’ll be teaching in 2027.

06.14
It’s possible for sure. And part of that is because some of the classes that I taught this past year, I taught applied data analysis, which is a machine learning and data analysis class, that we’re changing the name of to, I want to say applied statistical learning next year. But this past year was the first time that I taught it.

However, in years prior to that, I had taught intermediate Python several times. This is an accelerated version of the Python programming class, and it’s one that I have taught in the fall for a couple of years running, but I ended up completely redesigning this class the last time that I taught it, and the reason that I ended up completely redesigning it was that the previous curriculum for this class focused heavily on the syntax, what syntax people need to know, what that syntax does in Python, and how to remember what that syntax does, the difference between the different syntaxes. And the thing about programming languages in general, in Python in particular, is that they play very well with these types of agentic coding tools. And part of the reason for that is that the way that a large language model is built is by training on the patterns in text, and the patterns in programming text are remarkably strong relative to the patterns in natural language.

We have a much smaller set of tokens that are used in programming relative to natural language. We don’t really have things like pronouns and referential verbs, or referential nouns inside of programming. If you want to refer to a variable, you refer to the variable by its exact name, with the possible exception of like self or something like that. 

07.51
And so we have much stronger patterns. We have much stronger patterns as to the order in which these tokens are used. And so these tools have a lot of success from a relatively small number of patterns of programming language, but particularly Python, which has an especially small set of tokens and an especially strong pattern as to how it’s built, it can look at a relatively small number of examples and deliver valid outputs and valid output for whatever it is the problem is that you are having and to the extent that you’ve been able to describe that problem precisely, LLMs have a lot of success at generating valid Python, which begets the question, what is it important now for a Python programmer to know if they have these automated solutions available for generating Python? And so when I redesigned the class, I refocused it less on the syntax and more on the why.

Why is Python implemented the way it is? How is the Python implementation different from other programming language implementations? I think an idea that students do not have as much exposure to as I think might be useful is that different programming languages exist for a reason. They have different philosophies as to how an interpreter should work. There are choices to be made. There are trade-offs to be navigated in the design of a programming language, such that different answers exist that result in different programming languages being appropriate for different tasks. This is particularly a revolution for students who have done most or all of their programming in Python without being told necessarily why that is. And of course, part of the reason that that is, is that Python is a relatively useful. . . It generalizes fairly well to the type of problems that we’re teaching students to solve.

And it also has, because of a relatively small number of tokens, a relatively friendly learning curve for students. And so now the class focuses on why Python for which tasks, what were the trade-offs that people navigated and why. 

09.52
The other thing that the class now focuses on is what we can learn from Python about the growth and maintenance of a code base. Because there are relatively few code bases in the world that match Python’s degree of complexity and the number of users that Python has, but also the amount of openness with which it has been developed. There are reams of documentation on every code change. There is publicly available discussion on all of the code changes that have been made to the Python interpreter, as well as detailed documentation on the alternatives that were considered and passed up in favor of the way that Python works now.

And so all of that documentation makes Python a really useful case study for how you might work on such a massively impactful programming project yourself in the future, whether or not it’s in Python, because Python provides us with sort of like, a gold standard for how a complex project with a large user base might be maintained over time.

10.51
So in your work at Mozilla, I’m assuming you interview a wide-range of potential engineers, from the entry level to the more senior. So what kinds of tips are you giving your students in terms of. . . What is the change in the interview process in light of the agenda and coding tools? Because before they would give you all these little coding assignments, right?

For example, I work with startups where they even encourage some of the candidates to spend a day or two days at the company. And here, here, maybe you can try out this little project and then at the end of the day, well, we can discuss it. So what is the change, Chelsea, in terms of the interview process?

11.48
Yeah. So it’s an interesting question because I think that interview processes in programming have in some ways codified a difference between how we evaluate developers and how developers provide value to an organization for a pretty long time. Hillel Wayne has this really excellent series about the history of software engineering interviews, and the fact that many of our most common interview questions—and this is before the advent of agentic coding—many of our most common interview questions or interview questions we inherited over time from a period in which programmers had to do a lot more from scratch.

So, for example, we would ask interview candidates to implement a linked list from scratch. And if you were to ask a programmer in 2005 why we ask them to implement a linked list from scratch, the reason that we would give is that we want to evaluate their critical thinking capability and their architectural design capability and all of these things.

But that’s actually a retcon answer as to why we would ask that interview question. The reason we ask that interview question is that we inherited it over time, from an interview process that happened decades ago. And in that interview process, the reason that we asked developers to draw up a linked list from scratch is that, in fact, we did not have high-level programming languages that provided you with a linked list. And so in order to be able to do your work, you needed to be able to make a linked list. We got that question not because it’s some sort of theoretical critical thinking question but because at the time that it was developed, it was a very pragmatic question that related directly to the job that people were supposed to be doing.

13.37
And as programming languages developed, that question was no longer really pragmatic in the sense that it wasn’t a thing that developers were going to need to be able to do on the job anymore. But because we had lost touch with the reason that we asked that question, because we had lost touch with the developers of that question, because the programming industry had changed so much in the intervening period, and also because of a sort of a selection bias associated with who evaluates interview questions—anybody who’s in a position to evaluate an interview question is a person who passed that interview question because they work here—the question never changed. The why got lost. So we came up with this new why that didn’t quite fit the question.

And I think that for a long time we operated without the why. As to our interview processes in programming, famously there was this book, of course, Cracking the Coding Interview, which was theoretically about how to do how to succeed at coding interviews as a candidate, and after Cracking the Coding Interview came out, many companies started using Cracking the Coding Interview as a model of what they imagined Google did in the interview process, which therefore meant that was what they should do in the interview process, because Google was such an exciting place to work.

And so this book had these follow-on effects. I think that, to be honest, a lot of the programming industry has been kind of thrashing around on how to conduct an interview appropriately for a pretty long time. And I think that that continues as the tools that are available to our engineers evolve, while our interview process continues to be kind of this sort of decentralized thrashing as to what it is that we need to do.

15.21
And so I think the question of how the interview process is evolving, it ends up being highly variable from company to company. I think that some companies are changing relatively quickly. Some companies are changing more slowly. Some companies are embracing the use of AI in the completion of interview questions, and some companies are asking that they are able to continue to evaluate based skills and looking for ways to attempt to evaluate based skills, which of course means verifying that folks are not using this tool in the interview, if that’s the thing that they want to do.

And so from company to company, I find that it’s different, which makes it challenging to instruct students on how to address this. But I find myself thinking about this question from two angles. One of them is as a designer of interviews, I’ve designed some of the programming interviews that Mozilla uses for my team, and the other is as an advisor of students who might be taking these interviews.

Those angles are a little bit different because, on my team, currently the lowest position for which I have designed an interview has been what we call IC3. This is a senior software engineer. So I’ve designed for senior, I’ve designed for staff, and then I’ve designed for senior staff as well. So those are IC3, 4, or 5.

And in those roles, it is already supposed to be important that developers are able to evaluate trade-offs at the strategic architectural level for a codebase. And so in those interviews—we do them live; we don’t do a take home—I am working with developers to understand how they are going to navigate trade-offs in the design of a system, and we may ask them to write a line of code here or there.

We may ask them to write a function, but are largely asking them to walk us through their process. And it’s not the lines of code that are important. I have not found this interview style to need to change very much from the past, because it is so much a part of a conversation, and I think that that is still valuable and relevant to the work that we end up using.

17.22
A long, long time ago, when I was a junior engineer, I interviewed at Pivotal Labs and Pivotal Labs’ interview at the time was, I don’t know if this is still true, but at the time it was relatively famous for being the same entry-level tech, or rather the same sort of tech interview as you were entering the company for everyone. It was called the RPI, which stood for Rob’s programming interview, referring to Rob Mee, who was one of the founders of the company. And what it was was it was asking you to build. . . You could find it all over the internet. Technically, we’re not supposed to talk about what was in the interview, but if you want to go look, you can find it on the internet.

But we were asked to build a specific thing. We were asked to do it in Java. However, we were not the interview candidates writing the code. The interviewer was responsible for typing in the code and the interviewee was responsible for communicating the idea of what needed to happen sufficiently precisely, that the interviewer would then be able to implement that towards the goal that we had. And I think about that interview a lot, because I’m not going to say that interview was ahead of its time. I don’t think it was predicting that something like a. . .

18.40
Prompt engineering.

18.42
Right, but it was indeed this. Programming language aside, a part of the reason that the interviewer was the one typing the code was that we wanted to be able to interview folks coming from any language, but we were going to do the interview in Java because at Pivotal, the thing that you did was that you were working as a consultant on different projects.

It was theoretically possible for you to get staffed on a project in a language you didn’t know, and you were expected to be consulting level on it within three weeks, which meant you need to be able to learn programming languages fast, but the expertise that we’re selling people is precisely this thing your judgment: your ability to articulate what needs to happen in a system regardless of the programming language.

19.21
And I do think that that skill set remains the one that is the most important, both for companies to interview on and for interview candidates to be able to produce. You know, some companies still do this thing where they’ll put you on a video call and they’ll ask you to write down Dijkstra’s in 40 minutes. And theoretically it is a critical thinking challenge.

And where I land on this is that ultimately, that interview is a validation that you have already been taught Dijkstra’s algorithm because Dijkstra did not come up with Dijkstra’s in 40 minutes. So this is not some general critical thinking thing; it’s a memorization question effectively. For a memorization question, I don’t know that I have an opinion on like whether or not you should actually validate that people memorized it versus determined that they’re not, I don’t know, using an LLM to pretend that they memorized it or whatever, because I don’t think that this type of tech screen, asterisk is particularly useful.

20.24
Anyway, I think a much more useful tech screen is one that evaluates people’s decision-making. And I think that to the extent that LLMs have forced the interview process to move towards actually evaluating decision-making, that might be a good thing for tech interviewing overall. And I think it could be a good thing for junior developers as well, because it focuses—to the extent that junior developers are able to pick up on that—entry level developers are then developing that skill set that’s much closer to what’s actually important on the job than whether you’ve memorized Dijkstra’s, which you’re never going to have to code from scratch yourself.

21.04
Have you noticed, Chelsea, among your students who are on the job market. . . So this year in the job market, compared to on the job market last year, has it been more challenging to get this first or this entry level or first job for these students year to year?

21.29
I think that it is really challenging right now. I don’t envy students who are trying to go into industry at the moment. And I think that actually is. . . LLMs play a part in that. I think the biggest parts that LLMs play in that is that companies are experiencing a lot of turmoil figuring out, first of all, how to evaluate entry-level candidates.

And also, there’s all this consternation about whether companies need entry-level candidates. There’s this idea that, maybe if we just have senior engineers, they can delegate to agentic coding tools, and then we don’t need to hire entry level engineers. I think companies are going to be able to kind of try that for a few years. And I think then eventually it’s going to become clear that continuing to invest in talent for the industry is going to be an important thing for companies to do, regardless of the tools that are or are not available.

But I think we are still currently in this few-year phase where companies are experimenting with whether we can eliminate this entire class of employees. I think ultimately the conclusion is going to be we cannot. But because we are in that period, I think that currently there’s a lot of anxiety among students about whether there’s going to be availability of roles.

22.57
And also it has been the case for a long time that students feel like they have a hard time getting that first role. I remember 15 years ago being very, very concerned about like, oh, once I get blah level of experience, I know I’m going to have my pick of jobs, but until I get that much experience is going to be really challenging and I needed to go the extra mile a fair amount back then as well. . .and, you know, build relationships with hiring managers, build relationships with other engineers, understand what it was going to be like at various organizations.

I think a lot of students try cold-emailing like 100 companies or sending their résumé to 100 separate companies, and that doesn’t work. And then they feel like things are very hard and they are—things are really hard right now. But I would say that a lot of the challenges associated with getting hired now are similar in shape to challenges of getting hired from before that, you know, [are] much more intense right now.

24.00
Yeah. Yeah. The other thing that it seems like, Chelsea, companies are doing. . . So there’s the notion of “Maybe we should slow down hiring entry-level.” That’s one of the mistakes they’re making. The other thing that seems to be fashionable these days is, “Hey, actually, we should have all these managers code again, right?” Because basically now that there’s these coding tools, we don’t need these managers.

24.29
I think there’s. . .

24.30
Am I just imagining this? Because I’ve had these conversations with a bunch of people. It seems like it’s a real thing.

24.39
You know, it may be the case. I don’t think I’ve had as many conversations with folks in environments where managers were compelled to code. I do know that in my own personal experience, I’ve talked to a number of managers who are very excited about the way that agentic coding tools now give them the ability to write code with. . . A lot of times, it’s like a bandwidth issue. They have limited time; they have other responsibilities. Or sometimes it’s this like, “Well, I became a manager six years ago, and because the pace of technology moves very fast, that means that my skills are now obsolete. And so I no longer have the ability to actually keep my hand on the wheel as to what we’re doing. But now with agentic tools, I don’t necessarily need that same level of update, because I still have the ability to precisely communicate my requirements,” is the idea, “and if I can precisely communicate my requirements then agentic tools can do it for me.” I think a lot is still up in the air as to how useful this is going to be.

25.35
I know that a number of larger companies that pivoted towards attempting to siphon more work into LLM tools are now backing out and looking at taking a more holistic view as to how that’s going to work. So from a larger industry perspective, I think I still have a lot of questions about where that’s going to go. Is it going to be successful? Are people going to like it? What’s going to be the impact on the products themselves?

But I think that in my kind of personal sphere, I’ve talked to a number of managers who have been really excited about the possibilities that these tools provide for giving them the entree back into some level of individual contribution.

26.22
And I think that there is a lot of value for us to derive from that excitement in terms of understanding, like what managers missed about individual contribution previously and what we can learn about role development from that. I think that it’s been the case in the tech industry for a long time that we kind of make fun of the fact that you write code, you’re a good technologist, you do your things, you create value.

And to the extent that you are successful at it, you get rewarded with a promotion to a job that uses none of the skills that you just developed, and a whole bunch of skills that you now don’t have with, depending on the employer, widely differing levels of support on developing the completely new skill set that you’re now going to need as a manager.

And I wonder whether there is light to be shed by the advent of these tools. On and on and on, the possibilities for alternatives to that strategy where somebody coming from individual contribution has the ability to continue an individual contribution while also helping to grow teams. 

27.38
There is a developer who back in the Twitter days I used to follow, his name is Marco Rogers. His handle was Polotek, and he would talk about career development as a person who, if I recall correctly, started as an IC, became a manager, and then crafted a career path for himself in which he bounced back and forth between individual contribution and leadership roles and found that that worked really well for him, or posited that that could work really well, particularly juxtaposed against the sort of traditional career path that we talk about where if you become a good-enough developer, then you become a manager, and now you’re exclusively in the managerial track, despite the fact that your interest, your skill, and in a lot of cases for many of these people, your passion lay in the building of things. And now there is an argument to be made that you’re still building things, but you’re building as a team, you’re building a community, all of these things. 

But if we take that sort of like metaphor out of it for a moment, a lot of times these folks in leadership deeply miss this piece of the craft that they’ve lost access to. And this tool creates sort of a detour that allows them to express that interest in the craft again, which I think gives us license to examine whether they should have been separated from the craft in the first place, whether that was the appropriate way to develop the standard career path in software engineering.

29.02
I like that. I like that bouncing back and forth because I think that I’ve actually had a lot of friends who’ve done that as well. And if anything, I think the misunderstanding of these agentic coding tools probably is much more in the senior leadership role rather than the middle management role.

I’ve actually just tried to compile a bunch of studies. Because, on the one hand, you have these developer surveys, and obviously developers always have a tendency of overestimating things. And then there’s the actual telemetry. It turns out there’s this kind of an attenuation. So this intensity funnel where, you know, developers might be writing a lot of code now with these tools, but the number of software shipped actually hasn’t grown as much.

And then if you go all the way down to the end to the app stores—so Apple App Store, Google Play, and all these places—the actual number of. . . This usage of software hasn’t actually moved the needle. The tools haven’t moved the needle as much, just as much as the fact that, let’s say, a single developer might be writing 3x more code, right? But if you follow the trail all the way down, it hasn’t actually moved the needle.

And I think part of it is, we all probably feel productive in the sense that if it’s a one-off thing, yes, these tools can make me super productive. I’m never going to use this code again. I’m just going to use one of these tools. But if something gets more serious, then it turns out that it doesn’t move the needle as much because people obviously still have to follow all the rigorous processes. I don’t know what you think.

30.53
Yeah, I think that with regard to the way that these tools are used at the organizational level and the outcomes that we’re seeing, if I were to offer a half-baked, perhaps cancellable take on the situation, I’m a little trepidatious and saddened that a lot of the zeitgeist around the way to use these tools for productivity, theoretically, productivity gains is this idea that what we need is for developers. . . Like the proof of productivity is going to be the developers are closing more tickets; developers are shipping more code; developers are getting through things faster. I think that that focus demonstrates, possibly, a lack of vision as to what these tools could provide for us, because I’ve now been on the ground as an engineer for a while.

31.50
And the biggest problems that we run into are there are many. And of course, there’s always been that there’s not enough hours in the day. We can’t hire enough developers. But truly, that’s usually not actually the main problem that teams have had, in my experience over the last many years. Instead, the things that come up the most often are “We were evaluating trade-offs, and we selected this implementation because we only have the bandwidth for one, and we think this one is going to be the right choice. And we don’t have the opportunity to implement all of the others and experiment. And then based on real experiments, use the implementation that is working the best. So we take a guess or there will be like, you know, we would have liked to do comprehensive testing on that, but we just didn’t have the bandwidth to do the comprehensive testing on that. And so we’re making a guess.”

There’s a lot of developer estimates being baked into the systems that we’ve built because we don’t have the bandwidth to actually run all of the experiments that we might like to run. We don’t have the ability to include all of the rigor that we might like to include. And as you referenced earlier, developer estimates have the level of accuracy that they have, which is, you know, known largely in industry to be not perfect, right?

33.21
I am much less interested in what it means for a developer to ship three times as much code. I’m much less interested in that than I am in what it would mean for a developer to be able to use three times as much code to arrive at the ultimate solution, which might be approximately the same volume as the solution would have been before, or ideally, perhaps even lower volume than the solution before.

Because instead of needing to hedge against all of these possibilities and make an estimate and maybe even, maybe even overengineer preemptively based on all of these different possibilities, we have the ability to instead actually run the simulations, actually try the alternatives against each other, actually run tests, and arrive at this theoretical better solution. That we always knew we were making a guess at, that we felt forced to make a guess at because of our bandwidth limitations.

34.24
I run into this in data visualization as well. You know, we have all of these tools that have been available for a long time to theoretically help us visualize data and create dashboards, because executives want dashboards, and developers don’t have the ability to make custom dashboards all the time. So we have Looker for this, and we have Redash for this, and we have all of these various dashboarding tools that are available.

But the thing about those tools is that they have a limited number of things they can give you. They can give you a bar chart; they give you a pie chart; they give you these various other things. And you compare this to books written by folks who are professionally like artistic data visualizers, right? And they have all of these other options available.

And when we talk about the availability of AI and automation for the purpose of automating dashboards, what we talk about is making more and more customized dashboards with the same bar charts and pie charts and stuff that we’ve been writing before. And the the way that the zeitgeist focus is on the increase in volume that AI makes available I think disappoints me because the availability of this tool removes all of these bandwidth limitations that previously prevented us from being able to doggedly pursue the best quality of the thing that it is that we’re trying to ship. I think our focus on volume as a stand-in for productivity hamstrings us in our ability to actually improve our engineering product with these tools.

35.59
Yeah. I like what you said there. So it seems like then, Chelsea, companies that put themselves in a position where they can actually run these experiments and track the results. . . In other words, I don’t know what the equivalent of an experiment platform. . . You have a staging platform of some kind where you can test out all these ideas. It seems like that’s the right investment to make, right?

So in terms of a company wanting to be able to really leverage these tools, it’s being able to try out all the things that you wish you could try, applying the same rigor you used to apply to only one try. You can now try the equivalent of almost hyperparameter tuning in machine learning. So now if you put yourself in the position where you have this platform where you can try all sorts of ideas, maybe that’s the right investment.

37.05
I think so. I think that there is a lot of opportunity in having the ability to do these things. The thing that I’ve been experimenting the most with lately is data visualization. And I do this for a number of reasons. I work on data visualization, of course, in my day job, because we talk about how to provide dashboards to machine learning engineers to help them understand how their models are performing.

And we also talk a fair amount within the data science team, as you can imagine, on how to present analytics in ways that allow leaders to make business decisions based on the data that we have. So there’s that aspect of it, but there’s also this element of it associated with teaching students. And, you know, I talk to them about a lot of relatively complex concepts, how different models train and things like that. And a lot of times the way that we represent those concepts is with writing or formulae. And one of the things that I’ve been working on is how to represent these concepts for them graphically in a way that helps them understand. And the majority of my experience as a software engineer has been chiefly in backend engineering and a little bit of mobile engineering, but I have not done an enormous amount of frontend engineering.

I certainly have not done enough frontend engineering to have the kind of HTML and CSS skills that it would require for me to hand-code in an afternoon a tree ring diagram that represents the evolution of data science concepts over time, or something like that. That’s a thing that if I wanted to do it, I could do it.

38.40
But like I need to devote a fair amount of my summer to figuring out how I’m going to go about doing that. Meanwhile, HTML and CSS are both text-based mediums for generating images, which means that it is possible to use a large language model to develop at least a baseline on that. And then once I have that, figure out how to tune it using what HTML and CSS are both legible, at least legible to me, in a way that SVGs are not as much.

And so I’ve been largely using HTML and CSS for this. But what they do is there, or what the what the tool has done for me, is it is opened up this possibility for finding ways to represent information in ways that inspire my students and lead them to ask questions, as opposed to intimidating my students and leading them to retreat further back into the tools, because they are afraid that they are not going to be able to implement what they need to implement without them. Rather than pushing them in that direction, I’m trying to pull them forward into a curiosity about the internal mechanisms that I am attempting to explain to them, and I find these tools to be useful to me in providing a layer of text-to-image translation that gives me the ability, to the extent that I’m able, to precisely describe what it is that I want, to build those visualizations.

Which is not to say that it’s a quick process. It’s not a quick process at all. There’s a lot of tweaking, figuring out how the data should be organized, understanding why the data is organized, how it is recognizing all of these discrepancies that then pop up the minute you do this, that aren’t widely understood because we haven’t done this a whole bunch before. But there has been a very real increase in my ability to experiment with visualizations for teaching, because the text to visualization pipeline is streamlined for me by these tools.

40.43
All right. So in closing, I’ll have you predict, which I’m sure is going to be difficult to do given that these things change every week. So in one year’s time and in two years’ time, how does the day of a typical developer or software engineer change?

41.03
Oh, that’s an excellent question. But I think. . .

41.08
One year first and then be more speculative in the two years.

41.12
Sure. As I think about answering this question, I’m thinking back to how the experiences of engineers have changed over the period of other major technical advancements in our field. I think certainly if I were to predict over the next year, I think that engineers’ dependence on these tools will increase.

I think we saw the same thing with the advent of the search engine. Developers existed before the search engine; developers existed after the search engine. The search engine did not take away developers’ jobs by any stretch of the imagination. However, I worked at companies in 2015, where if the internet went down, we all went and played ping-pong because it was generally accepted that if we couldn’t Google stuff, we couldn’t do our jobs.

Nobody would have thought to go play ping-pong if the internet went down in 1985, because largely programmers did not have general access to the internet in 1985. And so I think that dependence on these tools will increase. We’re already seeing folks when the tools go down so they can’t get their jobs done, etc., etc. I think that kind of thing will become. . .

42.20
Or if they’re on the flight and the Wi-Fi is spotty.

42.23
Well, right. There’s this sort of like, yeah, I think that there will be adjudication around the dependence on these tools that is acceptable for developers to have and also acceptable for developers to communicate at the two-year mark. . .

42.40
You know what I will tell you at the two-year mark, here’s what I think/hope will happen—giant error bars around us. Right now, we’re using as a metric tokens consumed for developers. And I think that number of tokens consumed and leaderboards on number of tokens consumed are going to become less attractive for developers to top as subsidies within sort of the LLM industry start to end, and it becomes way more expensive to use tokens.

I am hopeful, in fact, that our focus pivots hard from token usage as a metric for productivity to token efficiency as a metric for skill at using these tools. I am hopeful that that will happen. I am also hopeful that at the two-year mark, we’re well on our way to seeing folks focus on using these tools in some of the ways that you and I have talked about earlier in this conversation, not just as a way to get through tickets faster but as a way to arrive at each ticket and an end that is much more rigorously researched and constructed.

Because the things that we used to just guess at because we didn’t have time to code them ourselves are now things we no longer have to guess at because we don’t have to code them ourselves. And so we develop and start to normalize a practice of actually having tried a few things and arrived at a best solution based on outcomes based on data, rather than making a guess. And then including that in our report as to why we arrived at the conclusion we did, and why the pull request we’ve submitted is the one that it is.

44.27
And with that, thank you, Chelsea.

---

---

Dr. Chinasa T. Okolo is the Founder and Scientific Director of Technecultura and a consultant for the United Nations and the World Bank. She talks with Scott about the yawning gap between AI's pace of development and policymakers' ability to understand it. She also discusses what it means to do AI governance research with a focus on Global Majority communities that are too often left out of the conversation.

• https://www.chinasatokolo.com

Download audio: https://r.zen.ai/r/cdn.simplecast.com/media/audio/transcoded/75c667ea-2739-4306-96be-e15097ef0853/24832310-78fe-4898-91be-6db33696c4ba/episodes/audio/group/09d8dba5-0fff-46db-86b5-1775a3a72c61/group-item/152f54e2-73a6-40c5-b37e-d320978e578c/128_default_tc.mp3?aid=rss_feed&feed=gvtxUiIf