Memory Pipeline

Every memory passes through a deterministic, multi-stage pipeline before reaching persistent storage. No stage can be bypassed.

Pipeline Overview

When an agent calls store(), the content traverses five sequential stages. Each stage produces an audit payload that is returned to the caller, making the pipeline fully observable.

1
Sanitize
2
Validate
3
Embed
4
Noise (beta)
5
Persist

The pipeline is fail-closed: if any stage encounters an error, the entire operation is rejected. No partial writes occur.

Stage 1: Content Sanitization

The Semantic Privacy Guard™ scans raw content for PII using 12+ regex pattern categories (emails, phone numbers, API keys, credit cards, IP addresses, and more). Detected items are replaced with typed placeholders like [EMAIL_REDACTED].

Each redaction is logged with a SHA-256 forensic hash, enabling audit without retaining the original sensitive value.

Note

Sanitization is enabled by default. It can be disabled via sanitize_enabled=False, but this is not recommended for production use.

Stage 2: Intent Validation

The Intelligent Intent Validation™ v2.1 classifies sanitized content into one of six intent categories: preference,fact, procedural,bio, ephemeral, or critical.

Classification uses a two-step process: keyword-based heuristic scoring followed by a confidence gate. Memories containing critical keywords (safety, emergency, allergy-related terms) are automatically promoted to the criticalcategory regardless of heuristic score.

This stage is always active and cannot be disabled. It determines how memories are prioritized during recall.

Stage 3: Embedding Generation

Sanitized content is converted into a 1536-dimensional vector embedding. In the Forge production environment, this uses OpenAI’stext-embedding-3-smallmodel. The Python SDK uses a deterministic hash-based pseudo-embedding for local development and testing.

These embeddings power the semantic recall system, enabling similarity-based memory retrieval via cosine distance.

Stage 4: Embedding Noise (Experimental)

A Gaussian-noise experiment is applied to a derived embedding used for sizing and audit. It does not yet provide a formal differential-privacy guarantee for the persisted semantic embedding.

The experimental parameter (ε) controls the noise-utility tradeoff in this experiment:

  • Lower ε (e.g., 0.1) → more noise, lower recall precision
  • Higher ε (e.g., 5.0) → less noise, higher recall precision
  • Default: ε = 0.5 (balanced)

After noise injection, the experimental vector is L2-normalized. The audit payload includes sigma (σ) and signal-to-noise ratio (SNR in dB) as experimental metadata.

Stage 5: Persist & Audit

The processed memory is written to persistent storage with:

  • A unique memory_id (SHA-256 hash of sanitized content)
  • The persisted semantic embedding (1536 dimensions; generated asynchronously)
  • A Trust Quotient score (computed from validation confidence)
  • Intent classification and criticality flag
  • AES-256-GCM encryption at rest (Forge production)

The StoreResultreturned to the caller contains the full audit payload: sanitization stats, privacy parameters, validation details, and Trust Quotient.

Recall Flow

Recall operates through the RecallRouter, which supports five modes:

ModeStrategy
semanticCosine similarity against stored embeddings (pgvector HNSW index)
temporalMost recent memories first
priorityOrdered by Trust Quotient score (highest first)
hybridWeighted combination of semantic + temporal + priority
autoSystem selects the optimal mode based on query characteristics

Guarantees & Boundaries

What the pipeline guarantees:

  • No raw PII reaches storage (when sanitization is enabled)
  • Every memory has an intent classification
  • The experimental path can add noise to a derived embedding (when enabled; not a formal DP guarantee)
  • Every operation produces an auditable result
  • No partial writes — all-or-nothing persistence

What the pipeline does NOT guarantee:

  • 100% PII detection (regex-based — context-dependent PII may be missed)
  • Perfect semantic recall (experimental noise introduces a precision tradeoff)
  • Cross-tenant data sharing (strict isolation enforced)