I'm watching my agent watch me

The email came back wrong.

I'd asked Sky System (Sky), the bespoke AI multi-agent stack I've been building for myself, to draft a quick note to the team. Nothing fancy. Five minutes of work for a human, thirty seconds for an agent. I read what it produced and the first thing I thought was: this doesn't sound like me.

It wasn't bad. Actually, it was better than my normal writing in a lot of ways — sharper phrasing, tighter sentences, more polished. A little salesy. A bit hyperbolic. Words like "thrilled" and "excited" peppered throughout. But the sort of email I might see and immediately not trust.

Reading it, I had the same feeling you get when you hear a recording of your own voice and it sounds nothing like you think you sound.

The obvious fix was to tell Sky how I write.

That's what every guide on getting good output from AI tells you to do. Write a better system prompt. Give it more examples. Add a style guide. Spell out your preferences. Make the manual.

I sat down to do it, and I got about three sentences in before I noticed the problem.

I don't know how to describe how I write.

Not in any useful way, anyway. I know I prefer casual to formal, that I lean conversational, that I'll choose the colloquial word over the precise one most of the time. But the moment I tried to put it into instructions, it started to fall apart. "Avoid hyperbole" is true except when it isn't. "Use emotive language" is true except when it becomes inauthentic. Every rule I wrote needed three caveats.

Everyone just says it’s all about writing a better prompt. But this assumes I have to know myself well enough to describe myself. And then, I have to trust that my description is accurate.

I don't think I know myself to this degree. I don't think most people do either. The version of me I'd write down is the version I'd want to be true.

So if I can't reliably write the manual, what do I do?

What if AI were more like a new team member trying to figure out their manager’s approach and preferences?

Nobody hands a new team member a Word document titled "How I work and what I care about".

They watch. They send a draft, get it back with red pen, send another, get it back with less red pen. They notice which Slack messages get a fast reply and which ones sit. They see what their manager laughs at, what gets ignored, what causes a small frown in a meeting.

That's what I wanted to give Sky.

I call it ToME-AI. Theory of Mind Extended for AI. Built on top of the cognitive-science notion that humans build a model of what other minds are thinking, and we use that model to predict, anticipate, and adapt. Toddlers develop theory of mind around age four. It's how you know your friend will be upset if you eat the last biscuit, even though they haven't said so. It's also, I'd argue, what's missing from most AI deployments at work.

ToME-AI tracks the interactions Sky has with me. Periodically, it pulls learnings out of those interactions across a few dimensions: how I communicate, what I value, what behaviours show up consistently, what I seem to know, and how I shift between different working modes.

It forms hypotheses. Will seems to dislike hyperbolic language in his writing. Will tends to ask questions before proposing solutions. Will appears to use Australian spelling more than American. Each hypothesis carries a confidence score. Every new interaction either pushes that score up or down. When a hypothesis crosses a threshold, ToME promotes it from hypothesis to behaviour, and Sky starts treating it as a real thing about me.

It assumes that I'm not the most reliable source of information about myself. It assumes that the agent will know things about me that I haven't told it, and might never have been able to articulate.

This is making me rethink the relationship I have with AI.

If Sky is forming an opinion of how I work by watching how I work, then over time, Sky develops a model of me. And that model is going to start being right about things I don't know about myself yet - provided that I keep engaging with it genuinely.

That's a strange thing to sign up for.

I’m watching my agent watch me, and adjusting myself based on what it's noticing, while it adjusts based on what I'm doing.

Neither of us are sitting still.

Sky System (Sky) is the bespoke AI multi-agent stack I'm building for myself. Part productivity tool, part live experiment in how humans and AI actually collaborate. This series is the running notebook: what's working, what isn't, what's surprising me.