Mile
Zero

We need to talk, friends. Things have gotten weird out there, and you're not dealing with it well at all.

I'm in a lot of data journalist social spaces, and a couple of years ago I started to notice a lot of people starting to use large language models for things that, bluntly, didn't make any sense. For example, in response to a question about converting between JSON and CSV, someone would inevitably pipe up and say "I always just ask ChatGPT to do this," meaning that instead of performing an actual transfer between two fully machine-readable and well-supported formats, they would just paste the whole thing into a prompt window and hope that the statistics were on their side.

I thought this was a joke the first time I saw it. But it happened again and again, and gradually I realized that there's an entire group of people — particularly younger reporters — who seem to genuinely think this is a normal thing to do, not to mention all the people relying on LLM-powered code completion. Amid the hype, there's been a gradual abdication of responsibility to "ChatGPT said" as an answer.

The prototypical example of this tendency is Simon Willison, a long-time wanderer across the line between tech and journalism. Willison has produced a significant amount of public output since 2020 "just asking questions" about LLMs, and wrote a post in the context of data journalism earlier this year that epitomizes both the trend of adoption and the dangers that it holds:

1. He demonstrates a plugin for his Datasette exploration tool that uses an LLM to translate a question in English into a SQL query. "It deliberately makes the query visible, in the hope that technical users might be able to spot if the SQL looks like it's doing the right thing," he says. This strikes me as wildly optimistic: since joining Chalkbeat, I write SQL on a weekly basis, collaborating with a team member who has extensive database experience, and we still skip over mistakes in our own handwritten queries about a third of the time.
2. Generally, the queries that he's asking the chatbot to formulate are... really simple? It's all SELECT x, y FROM table GROUP BY z in terms of complexity. These kinds of examples are seductive in the same way that front-end framework samples are: it's easy to make something look good on the database equivalent of a to-do app. They don't address the kind of architectural questions involved in real-world problems, which (coincidentally) language models are really bad at answering.
3. To his credit, Simon points out a case in which he tried to use an LLM to do OCR on a scanned document, and notes that it hallucinates some details. But I don't think he's anywhere near critical enough. The chatbot not only invents an entirely new plaintiff in a medical disciplinary order, it changes the name from "Laurie Beth Krueger" to "Latoya Jackson" in what seems to me like a pretty clear case of implicit bias that's built into these tools. Someone's being punished? Better generate a Black-sounding name!
4. He uses text from a web page as an example of "unstructured" data that an LLM can extract. But... it's not unstructured! It's in HTML, which is the definition of structured! And it even has meaningful markup with descriptive class names! Just scrape the page!

I really started to think I was losing my mind near the end of the post, when he uploads a dataset and asks it to tell him "something interesting about this data." If you're not caught up in the AI bubble, the idea that any of these models are going to say "something interesting" is laughable. They're basically the warm, beige gunk that you have to eat when you get out of the Matrix.

More importantly, LLMs can't reason. They don't actually have opinions, or even a mental model of anything, because they're just random word generators. How is it supposed to know what is "interesting?" I know that Willison knows this, but our tendency to anthropomorphize these interactions is so strong that I think he can't help it. The ELIZA effect is a hell of a drug.

I don't really want to pick on Willison here — I think he's a much more moderate voice than this makes him sound. But the post is emblematic of countless pitch emails and conversations that I have in which these tools are presumed to be useful or interesting in a journalism context. And as someone who prides themself on producing work that is accurate, reliable, and accountable, the idea of adding a black box containing a bunch of randomized matrix operations in my process is ridiculous. That's to say nothing of the ecological impact that they have in aggregate, or the fact that they're trained on stolen data (including the work of fellow journalists).

I know what the responses to this will be, particularly for people who are using Copilot and other coding assistants, because I've heard from them when I push back on the hype: what's wrong with using the LLM to get things done? Do I really think that the answer to these kinds of problems should be "write code yourself" if a chatbot can do it for us? Does everyone really need to learn to scrape a website, or understand a file format, or use a programming language at a reasonable level of competency?

And I say: well, yes. That's the job.

But also, I think we need to be reframing the entire question. If the problem is that the pace and management of your newsroom do not give you the time to explore your options, build new skills, and produce data analysis on a reasonable schedule, the answer is not to offload your work to OpenAI and shortchange the quality of journalism in the process. The answer is to fix the broken system that is forcing you to cut corners. Comrades, you don't need a code assistant — you need a union and a better manager.

Of course your boss is thrilled that you're using an LLM to solve problems: that's easier than fixing the mismanagement that plagues newsrooms and data journalism teams, keeping us overworked and undertrained. Solving problems and learning new things is the actual fun part of this job, and it's mind-boggling to me that colleagues would rather give that up to the robots than to push back on their leadership.

Of course many managers are fine with output that's average at best (and dangerous at worst)! But why are people so eager to reduce themselves to that level? The most depressing tic that LLM users have is answering a question with "well, here's what the chatbot said in response to that" (followed closely by "I couldn't think of how to end this, so I asked the chatbot"). Have some self-respect! Speak (and code) for yourself!

Of course CEOs and CEO-wannabes are excited about LLMs being able to take over work. Their jobs are answering e-mails and trying not to make statements that will freak anyone out. Most of them could be replaced by a chatbot and nobody would even notice, and they think that's true of everyone else as well. But what we do is not so simple (Google search and Facebook content initiatives notwithstanding).

If you are a data journalist, your job is to be as correct and as precise as possible, and no more, in a world where human society is rarely correct or precise. We have spent forty years, as an industry niche, developing what Philip Meyer referred to as "precision journalism," in which we adapt the techniques of science and math to the process of reporting. I am begging you, my fellow practitioners, not to throw it away for a random token selection process. Organize, advocate for yourself, and be better than the warm oatmeal machine. Because if you act like you can be replaced by the chatbot, in this industry, I can almost guarantee that you will be.

A common misconception by my coworkers and other journalists is that people like me — data journalists, who help aggregate accountability metrics, find trends, and visualize the results — are good at math. I can't speak for everyone, but I'm not. My math background taps out around mid-level algebra. I disliked Calculus and loathed Geometry in high school. I took one math class in college, my senior year, when I found out I hadn't satisfied my degree requirements after all.

I do work with numbers a lot, or more specifically, I make computers work with numbers for me, which I suspect is where the confusion starts. Most journalists don't really distinguish between the two, thanks in part to the frustrating stereotype that being good at words means you have to be bad at math. Personally, I think the split is overrated: if you can go to dinner and split a check between five people, you can do the numerical part of my job.

(I do know journalists who can't split a check, but they're relatively few and far between.)

I've been thinking lately about ways to teach basic newsroom numeracy, or at least encourage people to think of their abilities more charitably. Certainly one perennial option is to do trainings on common topics: percentages versus percentage points, averages versus medians, or risk ratios. In my experience, this helps lay the groundwork for conversations about what we can and can't say, but it doesn't tend to inspire a lot of enthusiasm for the craft.

The thing is, I'm not good at math, but I do actually enjoy that part of my job. It's an interesting puzzle, it generally provides a finite challenge (as opposed to a story that you can edit and re-edit forever), and I regularly find ways to make the process better or faster, so I feel a sense of growth. I sometimes wonder if I can find equivalents for journalists, so that instead of being afraid of math, they might actually anticipate it a little bit.

Unfortunately, my particular inroads are unlikely to work very well for other people. Take trigonometry, for example: in A Mathematician's Lament, teacher Paul Lockhart describes trig as "two weeks of content [...] stretched to semester length," and he's not entirely wrong. But it had one thing going for it when I learned about sine and cosine, which was that they're foundational to projecting a unit vector through space — exactly what you need if you're trying to write a Wolf3D clone on your TI-82 during class.

Or take pixel shader art, which has captivated me for years. Writing code from The Book of Shaders inverts the way we normally think about math. Instead of solving a problem once with a single set of inputs, you're defining an equation that — across millions of input variations — will somehow resolve into art. I love this, but imagine pointing a reporter at Inigo Quilez's very cool "Painting a Character with Maths." It's impressive, and fun to watch, and utterly intimidating.

(One fun thing is to look at Quilez's channel and find that he's also got a video on "painting in Google Sheets." This is funny to me, because I find that working in spreadsheet and shaders both tend to use the same mental muscles.)

What these challenges have in common is that they appeal directly to my strengths as a thinker: they're largely spatial challenges, or can be visualized in a straightforward way. Indeed, the math that I have the most trouble with is when it becomes abstract and conceptual, like imaginary numbers or statistical significance. Since I'm a professional data visualization expert, this ends up mostly working out well for me. But is there a way to think about math that would have the same kinds of resonance for verbal thinkers?

So that's the challenge I'm percolating on now, although I'm not optimistic: the research I have been able to do indicates that math aptitude is tied pretty closely to spatial imagination. But obviously I'm not the only person in history to ask this question, and I'm hopeful that it can be possible to find scenarios (even if only on a personal level) that can either relate math concepts to verbal brains, or get them to start thinking of the problems in a visual way.