That sounds like a simple request. For an AI application with memory, it contains a routing decision. Should the system answer from the current conversation, use a summary, or search older records for dietary preferences?

Now add one detail: the user mentioned a food restriction a few messages ago. Searching long-term memory may be unnecessary. Put the same detail outside the current context, and skipping retrieval may lose the information the answer needs.

The wording of the request has not changed. The available context has.

This is the problem behind one part of our memory evaluation at Sipplay, where we build AI character chat and interactive stories. I am part of the team. The examples above are illustrations; the measurements below come from our internal development tests.

The decision before vector search

It is tempting to begin memory engineering by choosing an embedding model and a vector database. There is a decision upstream of both: whether to search at all.

Search on every turn and the system spends time retrieving things it already knows. It may also introduce an older record into a conversation with a newer answer. Search too conservatively and a perfectly capable retriever never gets called.

We tested this decision separately from embedding quality. The model receives the current message, recent conversation, and a medium-term summary. Its job is to produce a retrieval plan.

That plan needs more detail than a yes/no flag. Which person is involved? Does “last time” mean the most recent event? Is the user asking for a preference, a relationship event, or a current plan?

Here is an illustrative output shape, translated and simplified from the evaluation design:

{
  "need_retrieval": true,
  "memory_types": ["relationship_event"],
  "entities": ["user", "Lin"],
  "time_scope": "most_recent",
  "search_query": "reason and outcome of the user's most recent argument with Lin",
  "confidence": 0.93
}

Valid JSON is only the first check. A confidently structured query can still name the wrong person or silently drop a time condition. The confidence value above is an example, not a calibrated probability.

Give the model reasons not to search

Our English intent test contained ten scenarios. Each supplied 20 recent messages, a medium-term summary, and six independent probe messages. That produced 60 probes: 30 requiring long-term retrieval and 30 that should not trigger it.

The negative examples were important. Some were ordinary conversation. Others had an answer already available in the recent messages or summary.

Without those cases, an eager model can look good simply by searching for everything.

The four candidates received the same English system prompt and inputs, and each completed two runs. We scored both sides of the decision rather than relying only on overall accuracy.

In that development set, Claude Haiku 4.5 triggered retrieval on 96.7% of the positive probes and correctly avoided it on all negative probes in both runs. That corresponds to 29 of 30 positive cases and 30 of 30 negative cases.

Gemini 3.5 Flash Lite also triggered on 29 of 30 positive probes, but it searched unnecessarily on three of the 30 negative probes in each run. Two of those false triggers concerned answers already present in the medium-term summary.

The differences were small in absolute numbers. They still revealed a useful failure pattern: the model recognized a memory-related question without consistently recognizing that the relevant memory was already in front of it.

“Never searches unnecessarily” can hide a different problem

GPT-5.4 mini correctly avoided retrieval on all negative probes, but missed two or three of the 30 positive probes, depending on the run. GPT-5 nano also avoided false triggers, while missing roughly 11 or 12 positive probes.

Those results describe this prompt and this small dataset. They do not establish a general ranking of the models.

They do show why a single cost or accuracy number is a poor basis for routing decisions. A system can look restrained because it has become reluctant to act.

The report's proposed gate requires at least 95% on both positive triggering and correct non-triggering. Only Haiku passed both in this round. An independent English holdout set is still required before treating that as a release decision.

Keep three forms of context distinct

The evaluation design uses recent messages, a medium-term summary, and longer-lived memory cards. Each has a different job.

Recent messages preserve the immediate exchange. The summary carries forward the broader situation. Long-term cards hold selected facts, preferences, events, and plans that may be useful after the current window is gone.

That separation creates a practical policy: check what is already available before asking the retrieval system for more.

It also creates a maintenance problem. A plan can appear in a summary but disappear when that summary is replaced. In a separate memory-extraction test, the report notes that ongoing travel and flight plans were sometimes preserved in summaries but omitted from long-term cards.

The proposed fix is specific: unfinished plans that remain useful across windows should become cards, with completion or expiration handled later. That change needs another test; writing the instruction is not proof that the failure is gone.

Test the handoff between components

Once the intent model produces a plan, feed that actual plan into retrieval. A human-written reference query is useful for isolating the embedding model, but it cannot reveal the mistakes introduced by the query-writing step.

For each failure, ask:

1. Did the system trigger retrieval when needed?
2. Did it preserve the correct person and time range?
3. Did the search query preserve the user's meaning?
4. Did the correct card enter the result set?
5. Did an outdated or wrong-person card outrank it?

In this English round, the auxiliary retrieval check used an existing Chinese card corpus. That is a useful cross-language observation, but it cannot validate the whole English memory experience. The next dataset needs to match the intended language and conversational conditions.

What this means for AI character chat

For a player, all these decisions collapse into one impression: does this character seem to remember what matters?

A forgotten preference can make a conversation feel generic. An unnecessary reference to an old event can make it feel out of step. Searching more often does not automatically improve either experience.

For our work on Sipplay, the useful lesson is to make the retrieval decision visible and testable. Give the model context, make it state what it needs, and inspect both the searches it starts and the searches it skips.

Before buying a faster vector database, it is worth checking whether the right question is reaching it.

Method note: all named model results are reported from Sipplay's internal evaluation, not independently rerun for this article. The 60 probes form a development set, with two runs per model. This is an account of an evaluation process, not a production performance guarantee. The raw prompts and fixtures are not published with this article.