A Finished Report and No Idea How

You come back from the kitchen with a coffee, and the report on the three suppliers is finished. It runs to twelve pages, with tables comparing the suppliers, a recommendation, and footnotes with sources. There's even a section on risks you never asked for.

The AI took twenty minutes, and the report looks like a week's work.

You'd happily put your name on it, but you can't say what's in it. You don't know which websites the AI read and which it skipped, what it treated as a reliable source, or where it guessed. None of that happened in front of you.

The old kind of AI wrote a paragraph, and you checked the paragraph. This one did the whole job while you were out of the room.

Why can't I see how it got there?

Ethan Mollick, who writes a newsletter called One Useful Thing, calls this working with wizards. The phrase is his, and so is the comparison that makes the problem easier to handle.

The more an AI can do, the less of the work you can watch, so you end up judging results without ever seeing how they were made.

We already trust work we can't check. You don't watch the pilot fly the plane, and you can't inspect the surgeon's work. Instead you give a kind of trust that can be taken back. You trust them because of their track record and because you have a reason to believe they work properly. And you're ready to stop trusting them the moment something seems wrong.

That's how I'd treat an AI that works while you're away. Trust it for now, and make it earn more trust one job at a time.

Where do I start?

Start with how the AI made the report. The report will always look right, so reading it first tells you nothing. Aim your checking at the places where a convincing report usually goes wrong. Ten minutes on twelve pages is enough.

Before I read the report, tell me how you made it. Which sources did you rely on most, and why those? What did you look for and not find? Where did you have to guess, estimate or choose between sources that disagreed? What would you have done differently with more time? Be specific and don't reassure me.

The answer tells you where to look. Say the AI admits, "I couldn't find current pricing for supplier B so I used their 2025 rate card". That one admission is worth more than the whole table built on top of it.

Sometimes the AI sends back a tidy summary that admits nothing, and then you push back. "That's the version for a client. Give me the version for a colleague who's going to check it."

Can I trust what it says about its own working?

Only as a map. The AI's account of how it worked is just more AI writing, so the account can be wrong and still sound sure of itself, just like the report.

Use the account to find the weak spots, then check for yourself whether those weak spots are real.

Choose three facts yourself. Pick one number, one quote, and one claim that would change the recommendation if it were wrong. Leave the facts the AI sounds surest about for later. Check each of your three at the source, for example the document, the website or a phone call.

If all three hold up, your trust goes up a notch. If one fails, the whole report goes back to being a draft.

What has it missed?

A report built from what can be found online will miss anything that was never written down. It won't know about the supplier your colleague had a bad experience with, or the contract detail everyone in your industry checks for.

Ask yourself what you'd have checked that no website records, and then check it.

I wouldn't do all this to a colleague, would I?

No. You'd read their report, ask a question or two, and trust them. But a colleague has a track record with you, and they have something to lose if they're wrong.

The AI has neither, yet. Your trust in AI can grow, but on the first few jobs the AI has to earn that trust.

When am I finished?

You're finished when you can write the verdict in your own words. Write one paragraph, put it above the report, and say what you decided and what you're still unsure about. If you can't write that paragraph, you haven't finished, because you've only read something.

The next time a tidy summary arrives with nothing admitted, you know what to say.

"Give me the version for a colleague who's going to check it."

With thanks to Ethan Mollick. The phrase "working with wizards" is his, and so are the point that more powerful AI is harder to see inside and the pilot comparison. They come from "On Working with Wizards" on One Useful Thing (September 2025).