MatrixArk benchmark: game-changing LLM context without a hot-path VectorDB.

Latest local run

This run used the local deterministic benchmark path so it is repeatable in development and CI without external model downloads. The same benchmark can run with OSS embedding and extraction providers when local model downloads are enabled.

Storage objects created

The benchmark is useful because it exercises the same compact TemporalStore records that the product uses. It does not only score an offline retrieval function.

Six benchmark tasks

The tasks are intentionally operational. They test whether MatrixArk returns the right context for a current decision, not whether it can retrieve a vaguely similar paragraph from a giant corpus.

Architecture comparison: before and after MatrixArk

This comparison is not a universal claim that every VectorDB, graph memory product, or filesystem-style memory layer is bad. It is a scoped-serving claim: for vertical AI, enterprise copilots, and Cursor-style workflows, the request usually arrives with tenant, user, session, project, ticket, account, object, or workflow scope. MatrixArk exploits that structure first instead of treating every prompt as a broad memory search. The before/after is architectural: fewer systems in the request path, no mandatory VectorDB, and no custom ingestion pipeline per product surface.

The practical before/after

Why MatrixArk should be faster and easier to operate

The core performance difference is not a magic model. It is a smaller request-time plan. MatrixArk lets the customer-facing graph/tree stay readable, then compiles it into storage primitives that are cheap to serve.

MatrixArk's graph/tree is filesystem-like for navigation, but better for serving: it is typed, indexed, time-aware, compressed, replayable, permission-aware, and token-budgeted.

VikingMem paper score reference

VikingMem is the best published reference for this category, so the benchmark now carries the paper's LOCOMO and LongMemEval scores beside our local MatrixArk product gate. These are not the same benchmark: VikingMem reports public long-memory benchmark LLM-judge scores, while MatrixArk's current local run tests our scoped TemporalStore serving loop. The point is to compare against the right target and make the next benchmark step explicit.

Source: VikingMem arXiv 2605.29640v2, Table 1. MatrixArk should use these numbers as the external score target, then run the same LOCOMO/LongMemEval harness now that the dataset adapter exists; the remaining step is official dataset execution plus judge-model automation.

LOCOMO and LongMemEval support added

The codebase now has a dataset-facing benchmark path for the same public benchmark families used by VikingMem. LOCOMO stresses long conversational memory across roughly 1,000+ messages per conversation. LongMemEval_s is much larger, with long timestamped histories and questions about multi-session reasoning and memory updates. MatrixArk now has the adapter layer needed to run those exports instead of only our six-task product gate.

PYTHONPATH=matrixark python3 matrixark/examples/memory_dataset_benchmark.py /path/to/locomo.json --name locomo --max-prompt-tokens 1800 --recall-mode tree_plus_embedding
PYTHONPATH=matrixark python3 matrixark/examples/memory_dataset_benchmark.py /path/to/longmemeval_s.jsonl --name longmemeval_s --max-prompt-tokens 1800 --recall-mode tree_plus_embedding

We should not publish MatrixArk LOCOMO or LongMemEval leaderboard claims until the official dataset files and judge protocol are run. The important implementation step is done: the runtime can now ingest multi-session benchmark histories, traverse them through TemporalStore-only tree-plus-embedding recall, and emit the measurements needed for an apples-to-apples VikingMem comparison.

Official LOCOMO run: completed, with honest baseline numbers

We have now run the official LOCOMO locomo10.json file through MatrixArk's TemporalStore-only benchmark path. This is not yet the final LOCOMO LLM-judge QA score; it is a retrieval/context-pack benchmark plus answer-substring check. The value is that the full official dataset now runs end to end and exposes the next quality gaps clearly.

The benchmark pass also improved the runtime: native LOCOMO and LongMemEval parsers, wider benchmark candidate budgets, event metadata preservation for replay, semantic event reranking after TemporalStore tree/time/filter narrowing, auxiliary LOCOMO extraction from session summaries and observations, and official-style hypothesis JSONL output. The new gap-closure run reached 16.57% answer-containing context at a 1,200-token budget. A follow-up balanced source-packing run raised answer-containing context to 17.22%. Evidence-linked packing then raised full LOCOMO retrieval/context hit to 46.27% and evidence-session recall to 40.11% by pinning raw dialogue evidence next to selected extracted observations. The next benchmark gap is answer generation and official judge integration, not basic ingestion/retrieval plumbing.

Extraction and storage efficiency from the paper

The paper also shows why our ingestion pipeline should keep moving toward one-pass extraction, temporal compression, and compact serving records. Those are infrastructure wins, not only retrieval-score wins.

Why no hot-path VectorDB wins here

The benchmark shows why the VectorDB should not be mandatory for MatrixArk's core path. The system does not brute-force every record. It starts at the scoped root, lists children, applies cheap filters and secondary indexes, scores summary embeddings exactly among bounded siblings, and only reads selected leaf timelines or resource chunks. Temporal compression keeps old history available without keeping every old raw event hot.

This is the quality argument: MatrixArk is not trying to retrieve the most chunks. It is trying to retrieve the most informative, current, permissioned context under a fixed token budget. Exact scoring over a small candidate set is both simpler and more accurate than approximate search over a broad pile.

Benchmark workflow

Mock tenant/team/project signals
-> ingest approvals, budget, risk, resources, feedback, and confirmation
-> extract compact MatrixArk internal schema
-> write ContextNode, ContextChildRef, ContextEvent, ContextIndex
-> generate summaries and embeddings in TemporalStore
-> compress old windows
-> retrieve with scoped tree traversal
-> expand strongly filtered refs through TemporalStore secondary indexes
-> apply temporal decay, stale blocking, and token budget
-> write ContextPackAudit for replay

The important detail is that benchmark failures map to product behavior. If resource chunks do not appear, resource ingestion or pack construction is wrong. If stale facts leak into a current-state answer, temporal decay or stale blocking is wrong. If feedback does not affect the next query, feedback memory is wrong.

Run it locally

The deterministic path is the default because it is fast and repeatable. Use the OSS model path when you want local embedding and extraction model downloads.

cd /root/src/github-services/matrixarkai
PYTHONPATH=matrixark python3 matrixark/examples/context_benchmark.py --max-prompt-tokens 240 --recall-mode tree_only

MATRIXARK_ALLOW_MODEL_DOWNLOAD=1 \
MATRIXARK_EMBEDDING_PROVIDER=sentence-transformers \
MATRIXARK_EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2 \
MATRIXARK_EXTRACTION_PROVIDER=oss-llm \
MATRIXARK_EXTRACTION_MODEL=google/flan-t5-small \
PYTHONPATH=matrixark python3 matrixark/examples/context_benchmark.py --max-prompt-tokens 240 --recall-mode tree_only

What this does not claim

A VectorDB can still be useful for broad discovery, cross-tenant offline exploration, or generic document search where there is no useful scope and no natural hierarchy. MatrixArk's claim is sharper: once the product is serving scoped, temporal, permissioned LLM context, TemporalStore is the better default because the serving problem is bounded before embedding similarity is applied.