What happened
On September 14, 2026, 404 Media reported on “Project Lily” — an OpenAI initiative that has hired hundreds of external contractors to read and rate real ChatGPT conversations as part of training the model to behave better, including reducing excessive agreeableness and self-anthropomorphizing language. The contractors see full conversation content. OpenAI says it strips identifying information, like usernames, before conversations reach reviewers — but the company itself acknowledges that “sensitive details can still get through.”
ChatGPT has more than 900 million weekly users, and — as the report notes — many use it the way they’d use a private notebook: for legal questions, financial planning, and real business work. One contractor told the reporter plainly: “I don’t think they would imagine some contractor somewhere is analyzing the conversations.”
This isn't unique to OpenAI. Anthropic has separately confirmed it also uses human review as part of improving its models. The specifics differ by company, but the underlying practice — that a person, somewhere, may read what you typed — is closer to standard across the industry than most users assume.
Why it matters
Small business owners paste real things into AI chat windows: a draft email to a client, a contract clause you want explained, a spreadsheet of numbers you want summarized, a tricky HR conversation you're trying to word carefully, pricing you haven't published yet. It feels private because it looks like a private conversation. It isn't automatically one.
That gap between how these tools feel and what actually happens to what you type is exactly where risk accumulates quietly. Not from a dramatic data breach — from the ordinary, repeated habit of treating a chat window like a locked drawer when it's closer to a shared filing cabinet with a lock that mostly works.
What it means for a small business
This isn't a reason to stop using ChatGPT, Claude, or any other AI tool — human review for model improvement is a standard, disclosed part of how these systems get better, and most of it is genuinely routine. It's a reason to know, tool by tool, what “standard” actually means for the account you're using.
The account tier matters more than most people realize. Free and consumer-tier accounts typically have different data-handling terms than paid business or team tiers — and the difference usually isn't obvious unless you go looking for it. If you've never checked which tier you're on, or what its policy actually says about training and review, you don't currently know the answer. You're assuming one.
What to do next
Three things worth doing this week, not eventually:
- Check the account-level privacy settings for the AI tools you actually use for business. Most chat AI products now have a toggle or account tier that affects whether your conversations are used for training and review — know your current setting; don't assume it.
- Write down, in one sentence per category, what should never go into a general AI chat window at your business — client-identifying details, financial account numbers, anything under an NDA, health information, unreleased pricing or legal matters — and make sure that line is written somewhere, not just understood by you.
- If a tool's data-handling policy for your tier isn't clearly stated anywhere you can find it, treat that as an answer, not a gap — either dig until you actually know, or keep the sensitive material out of it.
None of this requires walking away from AI tools that are genuinely useful. It requires the same thing every other business record gets: knowing where it goes and who can see it.
That’s exactly the gap Scalable Studio System’s Organize Your AI Workflow ebook is built for: a self-guided framework for mapping out how AI tools actually fit into your day-to-day operations — what’s doing what, and where — so you’re working from a clear structure instead of an assumption.
The model doesn't need to leak anything for this to be a real risk. It just needs you to have never checked.