Privacy in AI agents is a judgment problem, not a classification problem

I have been designing the privacy model for Aether for weeks, and the deeper I go, the clearer one conclusion gets: privacy in AI agents is a dead end if you try to solve it technically.

This sounds like a technical problem. It isn't.

The requirement that was wrong

We started with a requirement that sounds obviously correct: the agent must be able to distinguish what is private from what is not.

That requirement is wrong. It assumes privacy is a derivable property of data. It isn't.

Take a user's name. On a business card, it's fine. Linked to the fact that the user is in therapy, it's private. The name didn't change. The context did.

And "the user is in therapy" isn't inherently private either. It's private because the user doesn't want others to know. Social judgment, embarrassment, vulnerability. That's a preference, not a property of the information.

Follow any piece of private information to its root and you arrive at the same place: someone's preference.

A financial transaction is the clearest case. Sharing it with the counterparty is expected. Sharing it with a stranger is sometimes fine and sometimes catastrophic. Sometimes you would actually want to share it, to signal status or further your interests. Same data, different answer depending on the person and the moment.

There is no universal rule that says health information is private. In some cultures it isn't. Even within one culture, the same information in the same context may be private to one person and not to another. The only variable is what they prefer.

So no classification algorithm, no tagging system, no rule engine can determine what is private for a given user in a given context. Pattern-based detection solves the easy part: card numbers, national ID numbers, API keys. The hard part is everything with no pattern, and that part is judgment.

Someone always judges

If the information itself cannot tell you, then somewhere in the system a judgment has to be made. There are only two candidates.

The user can judge, which costs autonomy. Every share, every channel, every piece of information gets a sign-off. This is the safe answer, and it is the route some vendors have taken. It is also close to useless. An agent that asks permission for everything is a glorified permission dialog, not an autonomous agent.

Or the agent can judge, which costs trust. You have to believe its judgment is good enough to let it decide, knowing the blast radius is larger when it is wrong.

You can delegate the judgment to an LLM, but notice that this is still judgment. It is judgment that has been moved, not removed.

The tension nobody escapes

Any agent that acts on your behalf sits between trust and usefulness.

Without trust, you will not share everything. Without intimate knowledge of you, the agent cannot be useful. So an agent that shares nothing is safe and useless, and an agent that is genuinely useful has to be trusted with things you would not post publicly.

This is exactly how it works with a human assistant. You do not hand someone your calendar, your finances and your private context on day one. You extend trust as they demonstrate judgment, and their usefulness grows with it.

The parallel holds on the other side too. In human organizations, you trust employees and give them autonomy, or you don't trust them and you build approval loops. More autonomy always means more trust in someone's judgment. There is no configuration where you get both without the trust.

What I decided to build

I went with autonomy, and put the work into trustability instead.

Aether learns your preferences over time through feedback and alignment, the same way you would build trust with a person. Not a fixed policy about categories of data, because that policy cannot exist. A model of what this particular user wants shared, with whom, in what context.

The second piece is structural, and it also comes from how human organizations work. Instead of one agent that knows everything, you run several agents with different information access, cooperating.

That gives you three things. An attacker attempting prompt injection has to compromise several agents at once rather than one, which is much harder. There are multiple checkpoints before anything leaves. And because each agent holds less, the damage from any single bad judgment is smaller.

To make that practical you need very cheap agent spawning and real per-agent information boundaries. Those are engineering problems, and engineering problems are the kind I would rather have.

Why this matters beyond one product

In a few years every person and every company will have at least one agent working for them, probably a swarm. Those agents will know intimate details: finances, health, conversations, plans. And they will be talking to the outside world constantly, on your behalf.

If the deciding factor is trust rather than capability, then the question of who owns the agent stops being a detail. An agent you rent from a vendor asks you to trust their judgment about your data. An agent you own and run yourself asks you to trust your own.

That is the whole reason I care about sovereign AI. The bottleneck was never model size.