Memory Pipeline
Every memory passes through a deterministic, multi-stage pipeline before reaching persistent storage. No stage can be bypassed.
On this page
Pipeline Overview
When an agent calls store(), the content traverses five sequential stages. Each stage produces an audit payload that is returned to the caller, making the pipeline fully observable.
The pipeline is fail-closed: if any stage encounters an error, the entire operation is rejected. No partial writes occur.
Stage 1: Content Sanitization
The Semantic Privacy Guard™ scans raw content for PII using 12+ regex pattern categories (emails, phone numbers, API keys, credit cards, IP addresses, and more). Detected items are replaced with typed placeholders like [EMAIL_REDACTED].
Each redaction is logged with a SHA-256 forensic hash, enabling audit without retaining the original sensitive value.
Note
sanitize_enabled=False, but this is not recommended for production use.Stage 2: Intent Validation
The Intelligent Intent Validation™ v2.1 classifies sanitized content into one of six intent categories: preference,fact, procedural,bio, ephemeral, or critical.
Classification uses a two-step process: keyword-based heuristic scoring followed by a confidence gate. Memories containing critical keywords (safety, emergency, allergy-related terms) are automatically promoted to the criticalcategory regardless of heuristic score.
This stage is always active and cannot be disabled. It determines how memories are prioritized during recall.
Stage 3: Embedding Generation
Sanitized content is converted into a 1536-dimensional vector embedding. In the Forge production environment, this uses OpenAI’stext-embedding-3-smallmodel. The Python SDK uses a deterministic hash-based pseudo-embedding for local development and testing.
These embeddings power the semantic recall system, enabling similarity-based memory retrieval via cosine distance.
Stage 4: Differential Privacy
Before persistence, calibrated Gaussian noise is injected into the embedding vector. This implements ε-differential privacy, ensuring that individual memories cannot be reconstructed from the stored embedding alone.
The privacy budget (ε) controls the noise-utility tradeoff:
- Lower ε (e.g., 0.1) → stronger privacy, more noise, lower recall precision
- Higher ε (e.g., 5.0) → weaker privacy, less noise, higher recall precision
- Default: ε = 0.5 (balanced)
After noise injection, vectors are L2-normalized. The audit payload includes the noise sigma (σ) and signal-to-noise ratio (SNR in dB).
Stage 5: Persist & Audit
The processed memory is written to persistent storage with:
- A unique
memory_id(SHA-256 hash of sanitized content) - The noisy embedding vector (1536 dimensions)
- A Trust Quotient score (computed from validation confidence)
- Intent classification and criticality flag
- AES-256-GCM encryption at rest (Forge production)
The StoreResultreturned to the caller contains the full audit payload: sanitization stats, privacy parameters, validation details, and Trust Quotient.
Recall Flow
Recall operates through the RecallRouter, which supports five modes:
| Mode | Strategy |
|---|---|
| semantic | Cosine similarity against stored embeddings (pgvector HNSW index) |
| temporal | Most recent memories first |
| priority | Ordered by Trust Quotient score (highest first) |
| hybrid | Weighted combination of semantic + temporal + priority |
| auto | System selects the optimal mode based on query characteristics |
Guarantees & Boundaries
What the pipeline guarantees:
- No raw PII reaches storage (when sanitization is enabled)
- Every memory has an intent classification
- Every embedding has calibrated DP noise (when privacy is enabled)
- Every operation produces an auditable result
- No partial writes — all-or-nothing persistence
What the pipeline does NOT guarantee:
- 100% PII detection (regex-based — context-dependent PII may be missed)
- Perfect semantic recall (DP noise introduces a precision tradeoff)
- Cross-tenant data sharing (strict isolation enforced)