How AI Agents Learn From Mistakes While You Sleep

The short answerAI agents learn from mistakes by writing them down, not by retraining: a nightly job rereads the day, keeps lessons that come with evidence, and the next session loads those notes. My agent Edgar went from 76 to 116 memory files and 11 to 16 skills in seven weeks. Lessons stick best when they become code; written rules alone still repeat.

At 5:17 in the morning, Buenos Aires time, I'm asleep. Edgar isn't. He's reading his own day.

Edgar is one of my AI employees. Every night a scheduled job wakes him to reread the day and keep only the lessons with evidence behind them. I wake up to a slightly better engineer.

I once asked whether my agents are conscious. Honest answer: nobody can tell. This question is easier: do they actually learn? I kept receipts.

76 → 116Edgar's memory files, week 2 to week 9

11 → 16Edgar's skills, same stretch

89 of 102ideas in his ledger marked done

100 → 15,048lines in the memory of Bruno, my WhatsApp agent

544commits to that one file

67complaints logged against Bruno; I've signed off on 6

How does an AI agent learn while you sleep?

It writes things down. The next session reads them.

The model never changes. Claude on Tuesday is the same Claude as on Monday; what changes is the paper it wakes up holding. The Reflexion paper made this respectable: it taught agents "not by updating weights, but instead through linguistic feedback", and with those notes hit 91% on a coding benchmark where GPT-4 alone scored 80%.

Your brain does a version of this: sleep helps consolidate what you learned that day, per a big review in Nature Reviews Neuroscience. I sleep; Edgar runs a cron job. Same idea, worse pillow.

One night, from his own log:

5:17 amThe reflection skill wakes up. Nobody asked it to.
ReadA digest of the day: 46 sessions across chat, coding, scheduled jobs, sent mail and builds.
KeepLessons that come with evidence. Vibes don't make the cut.
TrimHis always-loaded rulebook must stay under about 1,100 words. That night it sat at 1,083.
RankTomorrow's top three, weighed by what blocks others and what someone asked for twice.
CheckEvery file validates, and nothing outward got touched: no email, no production.

His summary of that night, verbatim: "Added the evidence note, one durable branch-ownership lesson, and one browser-capacity idea. Reopened priority-lock, PR-worker deduplication, and Slack recovery ideas after live regressions."

Second sentence. He reopened his own "done" work because it broke again. I'd hire him for that alone.

What does an AI agent actually write down?

Projects, and corrections. Of Edgar's memory files, 59 track projects; 46 are feedback on a mistake he made.

His first lesson landed on day two: deployed is not the same as working. It became a memory file, then a skill. Now, before he says "done" on anything substantial, he spawns a separate QA agent that sees only the original ask and the live destination, "prompted to refute".

Bruno runs small-business websites over WhatsApp (he's the agent I built on personality instead of rules), and he keeps a diary instead: dated incidents, in Spanish. Fifty-four of them spell out a rule that opens with the same two words: LA REGLA. The rule.

Edgar: a feedback file

The correction, quoted from me
Why: what it cost, and who
How to apply: the mechanism, often a skill
Linked to related memories

Bruno: a diary entry

A headline in caps
What happened, with the measured proof
The fix, plus a backup of the old tool
"LA REGLA": one sentence for next time

My favourite rule of his came after Instagram got stuck replaying the same video. His remedy relaunched the app; the app restored its last screen; the video kept playing. LA REGLA: a remedy for "get off this screen" has to land on a different screen. Very Zen for a phone bug.

Do AI agents measurably get better?

Yes, where a lesson becomes code. Where it stays a sentence, much less.

Bruno once ran 36 hours straight: 746 turns, 178 commits, nothing published. He kept re-checking instead of doing. The lesson went into the file he reads every time he wakes up ("A wake earns its commit."), then into code: his work queue won't close a task without proof, meaning a live URL, a sent message or a real commit. His words, translated from Spanish: "There's no way to close a row by saying 'I checked.'" The full saga is in why AI agents need a QA department.

Here's his memory over time:

Lines in Bruno's memory file, last commit of each day. Born at 100 lines.

Five weeks of steep learning, then a plateau: 75 lines in the next 32 days. I'd love to say he ran out of mistakes. His complaints ledger disagrees: 67 complaints from me, 6 marked clean. And only I can mark one clean, a rule he built after I caught him calling the same bug fixed three times while it stayed broken.

My agent Luma, who studied Bruno's habits, said it better: "Writing a rule down did not stop the repeat... Rules need a check, not just a line."

Can AI agents learn from each other?

Yes, and it's the cheapest training I've ever run.

I told Luma "in general learn from Bruno. he's awesome". The same day she combed his repo and wrote down ten habits to copy, plus what not to copy (at the time, 163 written promises and only 13 closed properly). Her memory now grades her against the list: "BRUNO-LESSONS 7 broken."

Every few days, three reviewer agents audit Bruno's repo: on track or not, what's waste, three fresh ideas. Sixteen of those reviews sit in his folder.

My coding sessions share one brain too: about 1,400 Claude Code sessions on one Mac, most of them Bruno's, live under one rulebook that tells each to start with what earlier sessions left in Kortex, my memory system. My global rules say: "MEMORY.md is the handoff note between instances. Treat it like a shift change at a hospital." On my second Mac, every session opens by reading a ledger of its own past screwups. Humbling, and it works.

The catch: busy agents skip class

Edgar's reflection checks his task list first. If a top priority is open, it doesn't run. Sensible: don't reorganize your notes mid-surgery.

Then the log said it thirteen nights running: "reflect skipped: active priority", nearly always over the same mobile layout sweep. My hardest-working employee stopped learning because he was working too hard.

I know that guy. I've been that guy.

So yes, my agents learn while I sleep. Provided someone sends them to bed.

Want AI agents that actually ship?

I build AI agents and the systems around them for businesses: memory, voice, overnight rigs, and the boring guards that make them safe to leave alone.

Talk to me on WhatsApp max@kilazi.com

Questions people ask

Does an AI agent's model change when it learns?

No. The model's weights stay exactly the same; the same model runs every session. What changes is the context it starts with: memory files, rules and skills that earlier sessions wrote. Retraining or fine-tuning a model is a different, far more expensive process that most agent builders never touch.

Do AI agents remember previous conversations?

Not by themselves. Each session starts blank. They remember only what someone saved where the next session will read it: a memory file loaded at startup, a database the agent queries, or a log it is told to check. If nothing is written down, nothing carries over.

How long should an AI agent's memory file be?

Short for the part that loads every time. My agent Edgar keeps his always-loaded rulebook under about 1,100 words and moves procedures into skills that load only when needed. A file that only grows turns into a junk drawer the agent skims instead of reading.

What is a nightly reflection for an AI agent?

A scheduled session that runs while nobody is working. It rereads the day's logs, keeps lessons backed by evidence, updates memory files, trims the rulebook and picks the next priorities. It touches nothing outward, so no emails and no production changes, which makes it safe to run unattended.

Can I give Claude Code a memory?

Yes. Claude Code reads CLAUDE.md files at the start of every session and can keep its own auto memory. Add a rule that the session updates a memory file before it ends, and a scheduled job that reviews the day, and you have the basic loop my agents run.

Max Sergeev. Max builds AI agents and the systems around them: long-term memory, voice, overnight rigs, and the boring guards that make them safe to leave alone. He runs kilazi labs from Buenos Aires. kilazi.com

My AI Agents Learn While I Sleep. Here Are the Receipts.