Custom AI automation
2027 forecast — What Canadian buyers should require to test auditable RAG lineage and vector‑DB deletion (Scalix research brief)
· 6 min read · Scalix, Calgary
Published by Scalix. For questions or corrections, contact us.
Quick answer
Buyers increasingly question vendor claims about ‘deletion’, ‘revocation’ and ‘audit logs’ for RAG pipelines that use vector databases. This brief translates technical risks (e.g., recoverable 'ghost vectors' in some HNSW indexes) and Canadian regulatory expectations into a compact, runnable acceptance suite: six technical tests, mapped contractual clauses, a one‑page procurement checklist, and a short pseudo test harness. The brief describes how Scalix can run these tests in a Scalix free audit and what artifacts buyers should request from vendors.
Executive problem statement
Scalix prepared this brief to help Canadian buyers verify vendor claims that embeddings are deleted, RAG answers are traceable to exact source versions, and revocations are provably issued. Recent experiments show that soft‑deletes in some HNSW-based indexes can leave recoverable 'ghost vectors', creating a tangible verification gap for procurement and privacy reviews.
This brief converts that technical risk into a small set of acceptance tests and contractual artifacts procurement and technical teams can demand, and maps those tests to next steps buyers can take with a Scalix free audit.
- Buyer risk: soft‑delete markers may leave recoverable embeddings (ghost vectors) in some index/storage layouts.
- Buyer need: per‑retrieval lineage linking retrieval event → matched vector → source document version.
- Buyer ask: signed, exportable evidence (deletion receipts, audit packs) within a short SLA for sampled windows.
Short normative requirements buyers should demand
Translate procurement requirements into three minimal, normative acceptance items that map directly to tests and contract language. These are the baseline items a procurement or technical reviewer can include in an RFP schedule or vendor questionnaire.
- Deletion propagation: vendor must prove deletion requests propagate to all index and storage layers and supply a signed deletion receipt with vector ids and file manifests.
- Retrieval lineage: every retrieval produces a per‑retrieval provenance record (retrieval_id, timestamp, matched_vector_id, source_doc_id, source_version_hash).
- Signed revocation receipts and exportable audit packs: vendor must be able to export a zipped audit pack with raw source versions, SIEM‑compatible logs and a signed manifest for a sampled window.
Technical acceptance tests (six focused tests and expected artifacts)
Below are six runnable acceptance tests buyers can ask vendors to execute in a witnessed sandbox or during a Scalix technical audit. Each test pairs with the evidence artifact procurement should require in contracts.
- Deletion propagation test — Expected artifacts: deletion token, signed deletion receipt listing vector ids, and index/storage file manifest showing removed ids and file hash changes.
- Embedding non‑reconstructability test — Expected artifacts: recorded inversion attempt results (failure) or key‑rotation evidence and KMS/event timeline that demonstrates cryptographic erasure.
- Retrieval lineage test — Expected artifacts: per‑retrieval record (retrieval_id, timestamp, matched_vector_id, source_doc_id, source_version, source_hash, similarity) exported as NDJSON.
- Exportable audit pack test — Expected artifacts: zipped export containing raw source versions, SIEM‑compatible logs (JSON/NDJSON), a signed manifest (X.509 or equivalent) and checksum files.
- Revocation receipt test — Expected artifacts: time‑stamped, signed revocation token (revocation_id) verifiable against a published vendor public key and recorded in logs.
- RAG reproducibility sample — Expected artifacts: same retrieval_id + seed + provenance bundle reproduces the same answer in a vendor sandbox and the provenance metadata matches.
Contractual acceptance items mapped to tests
Map each technical test to concise contractual clauses. Use short SLAs for evidence export and require a signature scheme for deletion and revocation receipts. The clauses below are starting points for legal and procurement teams.
- Evidence export SLA — vendor must deliver an exportable audit pack for a sampled window (e.g., N=100 retrievals) within 48 hours of request; format: NDJSON + zipped raw sources + signed manifest (X.509 or equivalent).
- Deletion and revocation clause — deletion request triggers a deletion token; vendor must provide a signed deletion receipt listing vector ids and storage/file manifests; lack of valid proof triggers remediation rights.
- Third‑party responsibility clause — vendor must identify third‑party vector DBs and each provider's deletion semantics; vendor remains contractually responsible for delivering the evidence export even when using managed services.
Sample test harness (pseudo) and hypothetical worked example
The pseudo harness below is intentionally concise. It should run in a vendor sandbox with buyer witnesses or during a Scalix free audit. Labeled fields indicate the expected artifacts to capture.
Hypothetical worked example (explicitly hypothetical): a Canadian mid‑market firm asks for a Scalix‑observed sandbox run for 25 retrievals. Assumptions (hypothetical) include sandbox access and vendor willingness to sign receipts. Numerical assumptions are illustrative only and not Scalix guarantees.
- Pseudo‑harness steps: 1) Upload N documents → record source_doc_id/version/hash; 2) Index embeddings; 3) Execute retrievals with known queries → record retrieval_id and matched_vector_id; 4) Issue deletion for selected vector_ids → record deletion_token.
- Pseudo‑code (short, illustrative): connect(sandbox); upload_docs(docs); index(); for q in queries: r = retrieve(q); log(r.retrieval_id, r.vector_id, r.source_version_hash); delete(vector_id); request_export(); verify_signature(manifest.sig, vendor_pubkey).
- Sample audit pack contents: raw_source_versions/, retrievals.ndjson (retrieval_id, timestamp, vector_id, similarity, source_doc_id, source_version_hash), deletion_receipts.json (signed), storage_manifests.json (file hashes), signed_manifest.sig
Three‑item buyer checklist, limitations and Scalix next steps
Use this three‑item procurement quick‑filter, then attach the full acceptance tests into RFP schedules or request a Scalix free audit that uses the same test harness. Assumptions and limitations dated 2026‑11‑21: vendor features, vector DB internals and regulatory guidance can change; this brief does not perform vendor reverse‑engineering and assumes sandbox access when required.
- Checklist item 1 — Evidence export: require a sampled audit pack (N retrievals) with raw source versions, retrieval NDJSON and a signed manifest delivered within 48 hours.
- Checklist item 2 — Deletion proof: require deletion token → signed deletion receipt → storage/index file manifest showing removed vector ids and file hash changes; include an inversion test or epoch key rotation evidence.
- Checklist item 3 — Per‑retrieval lineage: require retrieval_id → matched_vector_id → source_doc_id → source_version_hash → timestamp exported in NDJSON and retained per agreed retention policy; require SIEM‑compatible logs and retention statements.
- Scalix next steps (transparent affiliation): Scalix can convert this brief into a publishable evaluation post, produce the one‑page buyer acceptance checklist, build the minimal test harness, and run the sandbox acceptance tests as part of a Scalix free audit.
Common questions
- Why is a vendor 'deletion' claim insufficient for buyers?
- Academic tests show some vector stores use soft‑delete markers that can leave embeddings recoverable from index or storage files; buyers should require exportable manifests, cryptographic receipts and sandboxed inversion tests rather than accepting statements alone. See Ghost Vectors (arXiv).
- What is a practical first acceptance test procurement teams can request?
- Require a sampled audit pack for N retrievals that includes raw matched source versions, per‑retrieval lineage NDJSON and a signed manifest. Ask the vendor to reproduce a retrieval in a witnessed sandbox and provide the exported audit pack within an SLA (e.g., 48 hours).
- How does Canadian guidance affect procurement requirements for RAG systems?
- Federal guidance (Directive on Automated Decision‑Making) and OPC principles emphasise traceability, record‑keeping and transparency. When procurement touches automated decision systems, buyers should demand provenance records, logs and exportable evidence as part of vendor evaluation. See TBS and OPC links.
Find your biggest time-saver
Book a free AI audit, or call our 24-hour line at (825) 782-4956.