How to build a memory system for an AI agent?

In my earlier work on autonomous GIS, I often wanted to create a memory system for AI agents that could build on previous interactions instead of starting over with every task. I did not have the opportunity or time to fully explore that memory system then.

Recently, I have returned to this question from another direction: designing memory for a companion AI assistant. The app provides a concrete setting for exploring ideas I also want to bring into autonomous GIS.

Imagine discussing a job interview with an assistant for several days. Then you tell it the interview was canceled. The next morning, it encourages you to keep preparing.

It remembered something, but missed the change that mattered.

That experience captures the challenge: useful memory requires preserving a story while updating our understanding of it. Here are three principles shaping our design.

Keep the conversation, not just the summary

Summaries help, but they can lose details or carry an earlier misunderstanding forward.

We preserve the original user messages and assistant responses, including who said what and when. Summaries and profiles link back to those records, so an interpretation can be checked and revised.

The distinction between speakers matters. An assistant’s suggestion is not something the user has done. A question about moving does not mean someone actually moved. Original records establish what was said, not whether every statement is true.

Build a profile that leaves room for change

A useful profile includes background, preferences, important relationships, and goals. It also separates lasting preferences from temporary situations.

“I prefer short answers” may remain useful for months. “I’m overwhelmed this week” should not become a permanent personality label.

Likewise, “I’m considering changing jobs” and “I accepted an offer yesterday” describe different stages. New information should update the current view while keeping earlier statements understandable in their original context.

I want the assistant to understand people better without trapping them in an old description of themselves.

Give important details another way to be found

People rarely repeat exactly the same wording. Searching by meaning helps connect “that interview” with an earlier conversation about a company.

But exact words also matter: names, places, unusual phrases, and details that never reached a summary. Our design therefore combines meaning-based search with keyword checks against original conversations, including support for Chinese text.

I think of this as a “don’t skip the check” principle. Important keywords deserve a search path even when other records rank higher.

This does not promise perfect recall. When information is missing or a check cannot be completed, the assistant should acknowledge that and ask rather than invent an answer.

Let the user shape what remembering means

These mechanisms need clear boundaries. Users should be able to correct information, restrict its use, ask the assistant not to mention something, and delete it. Those choices must survive system updates.

A detail shared in conversation is not automatic permission for an unsolicited reminder. Preserving complete conversations must also respect retention and deletion choices.

Inside the memory system: preserve the conversation, update the understanding, and check relevant evidence before responding.

Test a continuing life, not just isolated answers

We created 300 synthetic user profiles and more than 330,000 questions as test material. This does not mean every question has already been answered or evaluated.

The useful test is continuity: follow a person through changing plans, recurring concerns, and corrections. Hide parts of their profile and check whether earlier conversations provide enough evidence to recover them.

This work is still evolving, but the connection to autonomous GIS is clear to me. A GIS agent could remember the research objective, chosen datasets, analytical decisions, and methods already tried. If the study area or objective changes, it should update that understanding rather than blindly repeat the previous workflow. Just as a companion should distinguish a suggestion from a real event, a GIS agent should distinguish a proposed analysis from an executed and verified result.

That is the kind of memory I want to build: useful, checkable, and flexible enough to grow with its user.

What would make an AI’s memory feel helpful to you, and what would make it feel intrusive?

You do not have to build every component from scratch. Existing solutions such as Mem0 and Hindsight can provide a foundation for storing and retrieving memories. The work is in adapting that foundation to your application, whether companion AI or autonomous GIS: preserving original records, updating context, and respecting the user’s choices.

#AI #AIAgents #AICompanions #Personalization #AutonomousGIS