D1: Coding in the Age of AI
What do we do now?
Friday Discussion — ~50 minutes. Come prepared having skimmed at least one of the readings. No code. No problem set. Just a real conversation about something that’s changing the nature of your profession in real time.
Before You Arrive
Core reading (10 min): Vasilescu (2026), AI raises the productivity bar. Science 391, 763–764. [DOI: 10.1126/science.aef5239]
Research article (~20 min, skim methods → results → discussion): Science 2026. Experimental evidence on the effect of AI assistance on software developer productivity. [DOI: 10.1126/science.adz9311]
Video (~20 min): How to use AI effectively — a corporate onboarding video one of my students watched at a data science internship. [YouTube]
Bonus: Anthropic’s free courses on using AI as a tool, including for coding and research. [anthropic.skilljar.com]
The Core Finding
The Science paper ran a controlled experiment with software developers of different experience levels. The headline result is counterintuitive: AI tools made junior developers produce more output — but made experienced developers more productive at meaningful tasks. For the most experienced developers, AI was a force multiplier. For the least experienced, it mostly generated code they didn’t fully understand, which then had to be reviewed and debugged by others.
The perspective piece summarizes this as a threshold model:
“The quality of human input translates into productivity gains only after a minimum amount of expertise is reached. When content can be generated cheaply, productivity depends less on producing output and more on integrating AI-generated material into complex systems. Below this threshold, it may be that more generated content simply absorbs working time — output must still be read, edited, and maintained at substantial cost. Once the threshold is crossed, however, AI could accelerate productive work by reducing execution time. In this sense, AI does not lower the productivity bar but raises it.”
— Vasilescu, Science (2026)
A separate RCT (METR, 2025) found that experienced open-source developers actually took 19% longer on tasks when AI tools were available — they spent time managing and correcting AI output rather than just writing code themselves. The productivity picture is messier than the hype suggests.
A Metaphor Worth Arguing About
Here is one way to think about it. Generative AI is like a very powerful race car. If you know how to drive — if you understand the road, the physics, the route — it gets you somewhere faster than walking. If you don’t know how to drive, the car doesn’t make you a better driver. It makes you a liability.
The key question isn’t whether AI is powerful. It’s whether you’re in a position to steer it.
That means: Can you tell when the code it writes is wrong? Can you ask the right question? Can you evaluate the statistical analysis it produces? Can you recognize a plausible-looking but fundamentally flawed model? If yes, AI is a tool. If no, AI is a crutch — and like all crutches, it prevents you from building the strength you need.
There’s a counterargument worth taking seriously: maybe the kind of expertise that matters is shifting. Maybe what you need is not “how to write a for loop from scratch” but “how to evaluate AI-generated code and integrate it into a larger system.” Maybe the threshold isn’t about depth of coding knowledge so much as judgment, domain expertise, and the ability to ask precise questions.
This is an open question. What do you think?
Discussion Questions
Work through whichever ones resonate. You don’t need to cover all of them.
1. The learning paradox
If AI can do the thing for you, why learn to do the thing yourself? What’s your answer to this? Does it depend on which thing we’re talking about?
2. AI and statistical understanding
A student uses ChatGPT to write the R code for their entire problem set. It works. The answers look right. What did they learn? What didn’t they learn? What happens in their dissertation when the dataset is messy and the standard approach doesn’t work?
3. The threshold model
The Science paper argues that AI amplifies existing expertise rather than replacing it — that the productivity gains go to people who were already good. If true, what does that imply for: - How you should spend your time in this course? - The career advice you’d give a first-year PhD student right now? - How AI will affect inequality in science?
4. What should the policy be in this class?
“No AI” is one extreme. “AI for everything” is the other. What’s actually useful? What produces real learning? How would you design this course if you were the instructor?
Consider: students who never use AI may be under-prepared for professional contexts where it’s expected. Students who over-rely on AI may never build the judgment to use it well. Where’s the right balance?
5. The career trajectory question
Suppose you’re entering the job market in 2028. What skills will you wish you had built in 2026? Does AI change that list, or just change how you build those skills?
In-Class Activity
Two options — use one or both depending on time.
Option A — Live prompting (~15 min)
Open Claude, ChatGPT, or your AI tool of choice. Use it to answer this question:
“I have a dataset of plant heights measured at two time points (pre/post treatment) in 15 field plots. I want to test whether treatment had an effect. Write R code to do this.”
Run the code it gives you. Then:
- Does the code run?
- Is the statistical test appropriate? Why or why not?
- What didn’t it ask you that it should have?
- Rewrite the prompt to get a better answer.
Then try the reverse: write the analysis yourself first, then ask AI to review and improve it. How does that feel different?
The point is not to evaluate AI’s capabilities. It’s to notice what you need to know in order to use it well.
Option B — The Code Audit (~25 min)
→ Full activity page with instructions and code
Students annotate a block of AI-generated R code line-by-line using a three-level system (✓ = I own this / ? = I recognize this / ✗ = I don’t know what this does). Includes a reflection prompt for Canvas.
This can also be used as a take-home exercise or as a standing requirement for any homework: “You may use AI — but annotate every line you submit.”
The Instructor’s Take (Food for Thought)
I’m going to be honest: I don’t have a firm answer on what the right policy is, and I think anyone who is certain hasn’t thought about it hard enough.
What I do believe:
This course is fundamentally about building judgment. The tests, the concepts, the code — these are not the point. The point is that when you’re three years into your dissertation and your collaborator sends you a dataset and says “figure out what’s going on,” you can actually do that. That requires having built something real in your head, not just having run a lot of scripts.
AI does not compress the development of judgment. You can use AI to write a correct linear regression in R. You cannot use AI to know whether a linear regression is the right question to ask. That second thing is what you’re here to learn.
Used well, AI is genuinely powerful. Writing boilerplate, debugging syntax errors, generating a first draft of a visualization, translating between languages — these are real time savings. Once you understand what you’re doing, AI can help you do it faster.
Used badly, AI is a liability. Code you don’t understand is not your code. A statistical result you can’t explain is not your result.
My suggestion: use AI as a tutor, not a ghostwriter. Ask it to explain things. Ask it to critique your code. Ask it to generate examples. Don’t ask it to do your thinking for you.
Resources
Free courses: - Anthropic Academy (free) — courses on using Claude effectively, from basics to API/coding use cases - Anthropic courses on GitHub — open-source notebooks
Research: - METR study: AI makes experienced developers 19% slower — a sobering counterpoint to the hype - Noy & Zhang 2023 (MIT) — early evidence that AI boosts productivity for writing tasks; low-skill workers benefited more in that context (contrast with coding) - The Jagged Technological Frontier (HBS) — landmark study showing AI excels at some tasks but fails quietly at others
Broader reading: - The ChatGPT advice slides from Module 1 are available here: ChatGPT Advice for Stats (PDF) — originally prepared for this course; covers practical do’s and don’ts for using AI in statistical analysis.