Memory Pipeline
Every memory passes through a deterministic, multi-stage pipeline before reaching persistent storage. No stage can be bypassed.
On this page
Pipeline Overview
When an agent calls store(), the content traverses five sequential stages. Each stage produces an audit payload that is returned to the caller, making the pipeline fully observable.
The pipeline is fail-closed: if any stage encounters an error, the entire operation is rejected. No partial writes occur.
Stage 1: Content Sanitization
The Semantic Privacy Guard™ scans raw content for PII using 12+ regex pattern categories (emails, phone numbers, API keys, credit cards, IP addresses, and more). Detected items are replaced with typed placeholders like [EMAIL_REDACTED].
Each redaction is logged with a SHA-256 forensic hash, enabling audit without retaining the original sensitive value.
Note
sanitize_enabled=False, but this is not recommended for production use.Stage 2: Intent Validation
The Intelligent Intent Validation™ v2.1 classifies sanitized content into one of six intent categories: preference,fact, procedural,bio, ephemeral, or critical.
Classification uses a two-step process: keyword-based heuristic scoring followed by a confidence gate. Memories containing critical keywords (safety, emergency, allergy-related terms) are automatically promoted to the criticalcategory regardless of heuristic score.
This stage is always active and cannot be disabled. It determines how memories are prioritized during recall.
Stage 3: Embedding Generation
Sanitized content is converted into a 1536-dimensional vector embedding. In the Forge production environment, this uses OpenAI’stext-embedding-3-smallmodel. The Python SDK uses a deterministic hash-based pseudo-embedding for local development and testing.
These embeddings power the semantic recall system, enabling similarity-based memory retrieval via cosine distance.
Stage 4: Embedding Noise (Experimental)
A Gaussian-noise experiment is applied to a derived embedding used for sizing and audit. It does not yet provide a formal differential-privacy guarantee for the persisted semantic embedding.
The experimental parameter (ε) controls the noise-utility tradeoff in this experiment:
- Lower ε (e.g., 0.1) → more noise, lower recall precision
- Higher ε (e.g., 5.0) → less noise, higher recall precision
- Default: ε = 0.5 (balanced)
After noise injection, the experimental vector is L2-normalized. The audit payload includes sigma (σ) and signal-to-noise ratio (SNR in dB) as experimental metadata.
Stage 5: Persist & Audit
The processed memory is written to persistent storage with:
- A unique
memory_id(SHA-256 hash of sanitized content) - The persisted semantic embedding (1536 dimensions; generated asynchronously)
- A Trust Quotient score (computed from validation confidence)
- Intent classification and criticality flag
- AES-256-GCM encryption at rest (Forge production)
The StoreResultreturned to the caller contains the full audit payload: sanitization stats, privacy parameters, validation details, and Trust Quotient.
Recall Flow
Recall operates through the RecallRouter, which supports five modes:
| Mode | Strategy |
|---|---|
| semantic | Cosine similarity against stored embeddings (pgvector HNSW index) |
| temporal | Most recent memories first |
| priority | Ordered by Trust Quotient score (highest first) |
| hybrid | Weighted combination of semantic + temporal + priority |
| auto | System selects the optimal mode based on query characteristics |
Guarantees & Boundaries
What the pipeline guarantees:
- No raw PII reaches storage (when sanitization is enabled)
- Every memory has an intent classification
- The experimental path can add noise to a derived embedding (when enabled; not a formal DP guarantee)
- Every operation produces an auditable result
- No partial writes — all-or-nothing persistence
What the pipeline does NOT guarantee:
- 100% PII detection (regex-based — context-dependent PII may be missed)
- Perfect semantic recall (experimental noise introduces a precision tradeoff)
- Cross-tenant data sharing (strict isolation enforced)