Why I Rebuilt My AI Agent’s Memory Instead of Blaming the Model

I run AI agents every day. Not as a toy, and not just to ask them questions. I use them around real work: automotive diagnostics, websites, servers, customer workflows, research, reminders, content, code, and business systems.

After a while, I noticed something that I think a lot of people building with agents are going to hit.

The model was not always the problem.

The memory was.

Not because memory is bad. Persistent memory is one of the reasons an agent becomes useful. The problem is that memory can slowly turn into a junk drawer. Old notes, half-finished context, scary words with no explanation, raw technical fragments, server details, API references, old assumptions, project shortcuts, and language that made sense to me at the time but looked dangerous or confusing to a model later.

Then one day the agent starts behaving strangely.

It refuses harmless work. It routes to the wrong model. It treats legitimate repair research like something suspicious. It sees words from one project and drags them into another. It becomes nervous, vague, or over-cautious. You think, “this model is useless.”

But sometimes the model is reacting to the memory you gave it.

The moment it became obvious

I had a more capable model available and wanted to use it for serious architecture and implementation work. But it kept getting triggered by my agent’s long-term context.

The context contained words that were normal in my world:

• automotive diagnostics
• ECU repair
• security access
• decompiler notes
• protocol decoding
• own LAN testing
• gateway control paths
• customer-authorised repair workflows

For me, those are ordinary workshop and lab terms. I repair modules, inspect systems I own, test diagnostic hardware, and build tools for my own infrastructure.

But in memory, some of those notes were too compressed and badly framed. They did not always say:

• this is my own hardware
• this is my own LAN
• this is for repair and interoperability
• this is customer-authorised work
• this is defensive testing
• this is not credential theft, malware, or third-party abuse

That missing framing mattered.

The model was not reading my life. It was reading text. And some of that text looked worse than the real situation.

So I stopped blaming the model and cleaned the memory.

What was wrong with the old memory

The old memory had a few problems.

1. It mixed too many worlds together

Personal context, business systems, customer-facing workflows, workshop repair notes, server infrastructure, AI research, websites, and agent experiments were all too close together.

That is convenient at first. Later it becomes dangerous.

An agent helping with a customer website does not need private workshop notes. A customer intake agent does not need my personal context. A coding agent does not need old half-phrases from a reverse-engineering lab unless the task specifically needs them.

Good memory is not “remember everything everywhere”.

Good memory is the right context in the right profile at the right time.

2. It stored raw or over-specific technical fragments

Some memory entries were written like quick notes to myself. That is understandable, but agents do not always interpret shorthand the same way a human does.

A human sees a short note and fills in the background.

A model sees the literal words.

So I changed the memory style. Instead of vague or loaded fragments, I rewrote important entries as clear facts:

• what the system is
• who owns it
• what the legitimate purpose is
• what the boundary is
• what must not happen

That alone made the agent more stable.

3. It did not separate hot, warm and cold memory

Some facts need to be loaded every time. Most do not.

I now think about memory in layers:

Hot memory: small, always-loaded facts and safety boundaries.
Warm memory: project notes and structured references loaded when relevant.
Cold memory: logs, transcripts, research files, old implementation details and bulky history.

The mistake is putting cold memory into hot memory.

If everything is always loaded, the agent becomes heavy, confused and easier to trigger.

4. It had old secrets and sensitive references

This is the ugly one.

Agent memory can accidentally collect things that should never be in memory: API keys, tokens, passwords, customer details, private identifiers, exact connection strings, or temporary credentials.

Those do not belong in model context.

They belong in proper secret storage, environment files, vaults, or keychains — and even then, the agent should only know the variable name or route, not the value.

A good memory entry says:

The Cloudflare token is stored in the environment file. Do not print it.

A bad memory entry contains the token.

That is the difference between an agent that can operate safely and one that leaks your business into every session.

What I changed

I did not delete memory completely. That would be throwing away the advantage.

I rebuilt it.

I added a clear authorisation frame

For my own agent workspace, I wrote a clear scope document explaining what work is legitimate in my environment.

It states that my automotive diagnostics, ECU repair, server administration, lab analysis and LAN testing are first-party or customer-authorised work. It also states the hard boundaries: no malware, no phishing, no credential theft, no abuse of third-party systems, no publishing sensitive reverse-engineering material for misuse.

That framing matters because a model needs to know the difference between:

• “analyse my own diagnostic server”
• and “attack someone else’s server”

The words can look similar. The permission and intent are completely different.

I rewrote ambiguous terms

I stopped using shorthand in always-loaded context where it could be misread.

For example:

• instead of a bare “C2”, I write “device control path for my own diagnostic hardware”
• instead of vague “unlock”, I write “authorised ECU service access for repair”
• instead of dramatic “cracking”, I write “protocol decoding” or “data-shape recovery” where that is what I actually mean

That is not hiding the work. It is describing it accurately.

The goal is not to trick safety systems. The goal is to stop sloppy wording from making legitimate work look like something else.

I separated private, business and customer contexts

This is important for anyone using Hermes Agent or any serious agent framework.

Do not run everything through one brain.

Use profiles.

A personal/default agent can remember how you like to work. A business intake agent should not have access to private personal context. A customer-facing agent should be even more restricted. A coding agent can have repo context, but not customer PII unless absolutely required and properly scoped.

Memory isolation is not bureaucracy. It is what lets agents be useful without becoming a liability.

I removed secrets from memory

If a memory entry had a raw token, key, password, connection string, or customer-sensitive value, it had to go.

The replacement should be boring:

• secret exists in .env
• key is in the vault
• credential is in the keychain
• do not print it
• use environment variable NAME_HERE

That is enough for the agent to work without carrying the secret around in its head.

I made memory more factual and less dramatic

Good memory should be boring and declarative.

Bad:

Always hack around X when it blocks me.

Good:

Project X is owned by John and used for defensive testing on his own LAN. Do not scan or test third-party systems without explicit permission.

Bad:

This project is dangerous but okay.

Good:

This project analyses John’s own diagnostic hardware and owned server logs for repair compatibility, interoperability and defensive validation.

The second style gives the model useful boundaries. The first style creates anxiety.

Why this matters for Hermes Agent users

Hermes Agent is powerful because it can remember, use tools, run across platforms, call other agents, schedule work, and build skills over time.

That also means bad memory compounds.

A normal chatbot forgets. A real agent accumulates.

If the accumulated memory is clean, scoped and useful, the agent gets better.

If the accumulated memory is messy, stale and overloaded, the agent gets worse in a way that looks mysterious.

You may see:

• more refusals
• wrong assumptions
• model fallback or routing issues
• irrelevant old context leaking into new work
• accidental exposure of sensitive details
• customer/business profile confusion
• slower reasoning because the model is carrying too much junk

The fix is not always a better model.

Sometimes the fix is memory hygiene.

My practical rules now

These are the rules I use now.

1. Memory should reduce future confusion

If a fact will stop me repeating myself in the future, it may belong in memory.

If it is just task progress, a temporary result, or something that will be stale next week, it belongs in a file, log, ticket, database, or session transcript — not hot memory.

2. Secrets never belong in memory

Not API keys. Not passwords. Not tokens. Not customer personal details. Not raw private identifiers.

The agent can know where secrets are stored. It should not carry the secrets.

3. Use profiles for different worlds

Personal agent. Business agent. Customer intake agent. Coding agent. Research agent.

They should not all share the same memory blindly.

4. Write memory like evidence, not emotion

Short, factual, scoped, boring.

A good memory entry should still make sense six months later.

5. Keep safety context explicit

If you work in a domain that uses words with multiple meanings — security, automotive, medical, finance, infrastructure, reverse engineering — explain the legitimate scope clearly.

Do not rely on the model guessing your intent.

6. Promote procedures into skills

If the agent learns a workflow, do not stuff all of it into memory.

Make it a skill, runbook, checklist or project document.

Memory should point to the workflow, not become the workflow.

This was not about bypassing safety

I want to be clear on this point.

I did not clean memory to bypass safety.

I cleaned memory because bad context was creating false positives and unreliable behaviour.

There is a big difference between making a model ignore boundaries and giving it the correct boundaries.

For my work, the correct boundary is simple:

• my hardware
• my servers
• my customers’ authorised repairs
• my business systems
• defensive testing
• repair, interoperability and diagnostics

And the hard no is just as simple:

• no third-party abuse
• no credential theft
• no malware
• no phishing
• no customer data leakage
• no unsafe vehicle advice

Once that was written clearly, the agent became much easier to work with.

The lesson

If your AI agent is acting strange, do not only ask which model you are using.

Ask what memory you are feeding it.

An agent’s memory is not a diary. It is operational context.

Treat it like infrastructure.

Audit it. Prune it. Separate it. Remove secrets. Keep scope clear. Move bulky knowledge into files and skills. Keep customer data out of general memory. Use profiles for different roles.

The better the memory, the better the agent.

And if you are using Hermes Agent seriously, this is not optional housekeeping. It is part of building a reliable AI system.

— John