---
type: meta
title: "v1.7 Retrieval Benchmark Corpus"
status: evergreen
updated: 2026-05-17
---

# v1.7 Retrieval Benchmark — 50 Queries

Used by the v1.7.0 audit (`docs/audits/v1.7.0-audit-2026-05-17.md`) to score
top-1 / top-5 accuracy of the hybrid retrieval pipeline (`scripts/retrieve.py`)
vs. the simulated v1.6 baseline (`scripts/baseline-v16.py`).

## Schema

Each query: `id`, `query`, `correct` (canonical page paths), `relevant`
(supporting paths), `category`, `rationale`. For negative queries, `correct`
is `null` and success is defined as "returns nothing OR returns only pages
in `relevant`."

Machine-parsable: every query block is bracketed by `### <id>` and ends
before the next `### ` or `## `. The scoring runner at
`scripts/benchmark-runner.py` reads this file directly.

## Scoring rules

- **top-1 success**: top result's `page_path` equals one of `correct[]`
- **top-5 success**: any of top-5 results' `page_path` is in `correct[]`
- For negative queries (correct=null): top-1 success if top result in `relevant[]` or no results; top-5 success if any top-5 in `relevant[]` or no results

Final scores reported as (top-1%, top-5%) per pipeline, broken down by category.

---

## Derived Queries (25)

### D1
- query: How does knowledge compound differently in a wiki versus traditional RAG?
- correct: wiki/concepts/Compounding Knowledge.md
- relevant: wiki/concepts/LLM Wiki Pattern.md, wiki/comparisons/Wiki vs RAG.md
- category: derived
- rationale: Canonical treatment of why wiki accumulation produces more value over time vs. query-time re-derivation.

### D2
- query: What is the Hot Cache and why would a user enable it in their session?
- correct: wiki/concepts/Hot Cache.md
- relevant: wiki/concepts/LLM Wiki Pattern.md, wiki/getting-started.md
- category: derived
- rationale: Direct explanation of the session context mechanism and its token-cost benefits.

### D3
- query: Tell me the three-layer architecture behind the LLM Wiki pattern.
- correct: wiki/concepts/LLM Wiki Pattern.md
- relevant: wiki/overview.md, wiki/index.md
- category: derived
- rationale: Explicit description of .raw, wiki, and CLAUDE.md layers.

### D4
- query: What does the wiki fold operator do and why use exponential batch sizes?
- correct: wiki/concepts/DragonScale Memory.md
- relevant: wiki/log.md, wiki/hot.md
- category: derived
- rationale: Mechanism 1 detailed explanation with 2^k justification and LSM analogy.

### D5
- query: How are deterministic page addresses assigned and what format do they use?
- correct: wiki/concepts/DragonScale Memory.md
- relevant: wiki/log.md
- category: derived
- rationale: Mechanism 2 spec with c-NNNNNN format and counter recovery rules.

### D6
- query: Explain semantic tiling lint and what cosine similarity threshold it uses.
- correct: wiki/concepts/DragonScale Memory.md
- relevant: wiki/log.md, wiki/hot.md
- category: derived
- rationale: Mechanism 3 detailed treatment including banded thresholds and embeddings.

### D7
- query: What are the cherry-picked features claude-obsidian should implement next?
- correct: wiki/concepts/cherry-picks.md
- relevant: wiki/index.md, wiki/comparisons/claude-obsidian-ecosystem.md
- category: derived
- rationale: Explicit prioritized backlog from ecosystem research across tiers.

### D8
- query: What is the relationship between SVG diagrams and brand tokens in claude-obsidian assets?
- correct: wiki/concepts/SVG Diagram Style Guide.md
- relevant: wiki/log.md
- category: derived
- rationale: Canonical style reference with color palette, typography, and layout tokens.

### D9
- query: How does a user ingest a source document into claude-obsidian and what happens next?
- correct: wiki/getting-started.md
- relevant: wiki/concepts/LLM Wiki Pattern.md, wiki/overview.md
- category: derived
- rationale: Three-step quick start walkthrough with expected output pages.

### D10
- query: What does the Pro Hub Challenge reward and what was the first winner?
- correct: wiki/concepts/Pro Hub Challenge.md
- relevant: wiki/entities/Claude SEO.md, wiki/log.md
- category: derived
- rationale: Community challenge structure and integration pattern with historical results.

### D11
- query: What is Search Experience Optimization and how does it read SERPs backwards?
- correct: wiki/concepts/Search Experience Optimization.md
- relevant: wiki/entities/Claude SEO.md, wiki/concepts/Semantic Topic Clustering.md
- category: derived
- rationale: Direct explanation of SXO methodology and its page-type mismatch detection.

### D12
- query: How does semantic topic clustering use overlapping URLs to detect content relationships?
- correct: wiki/concepts/Semantic Topic Clustering.md
- relevant: wiki/entities/Claude SEO.md, wiki/log.md
- category: derived
- rationale: Overlap scoring rules and hub-spoke architecture.

### D13
- query: What tracking rules does SEO Drift Monitoring enforce and where is the baseline stored?
- correct: wiki/concepts/SEO Drift Monitoring.md
- relevant: wiki/entities/Claude SEO.md
- category: derived
- rationale: 17 comparison rules across severity levels and SQLite persistence location.

### D14
- query: Who is Andrej Karpathy and why does he matter to this wiki project?
- correct: wiki/entities/Andrej Karpathy.md
- relevant: wiki/concepts/LLM Wiki Pattern.md, wiki/index.md
- category: derived
- rationale: Founder attribution and origination of the pattern with notable quote.

### D15
- query: What version of Claude SEO shipped in April 2026 and how many skills does it have?
- correct: wiki/entities/Claude SEO.md
- relevant: wiki/log.md, wiki/meta/2026-04-14-claude-seo-v190-session.md
- category: derived
- rationale: v1.9.0 state with skill counts and Pro Hub Challenge integration.

### D16
- query: Does Ar9av's obsidian-wiki support multi-agent deployment and how?
- correct: wiki/entities/Ar9av-obsidian-wiki.md
- relevant: wiki/comparisons/claude-obsidian-ecosystem.md, wiki/concepts/cherry-picks.md
- category: derived
- rationale: setup.sh bootstrap matrix across multiple agents and delta tracking manifest.

### D17
- query: What does the /adopt command in ballred's PKM system do?
- correct: wiki/entities/ballred-obsidian-claude-pkm.md
- relevant: wiki/concepts/cherry-picks.md, wiki/comparisons/claude-obsidian-ecosystem.md
- category: derived
- rationale: Vault analysis and pattern detection for PARA, Zettelkasten, LYT migration.

### D18
- query: What official Obsidian skills does kepano publish and which is most token-efficient?
- correct: wiki/entities/kepano-obsidian-skills.md
- relevant: wiki/concepts/cherry-picks.md, wiki/entities/Claudian-YishenTu.md
- category: derived
- rationale: obsidian-markdown, obsidian-bases, defuddle, and defuddle's 40-60% token reduction.

### D19
- query: What are the three query depths in rvk7895's LLM knowledge base system?
- correct: wiki/entities/rvk7895-llm-knowledge-bases.md
- relevant: wiki/concepts/cherry-picks.md, wiki/comparisons/claude-obsidian-ecosystem.md
- category: derived
- rationale: Quick, Standard, Deep modes with output formats.

### D20
- query: How does Nexus MCP for Obsidian store workspace memory and is it Obsidian Sync compatible?
- correct: wiki/entities/Nexus-claudesidian-mcp.md
- relevant: wiki/comparisons/claude-obsidian-ecosystem.md
- category: derived
- rationale: JSONL storage and Sync auto-inclusion.

### D21
- query: What makes Claudian's inline edit feature best-in-class compared to other Obsidian AI plugins?
- correct: wiki/entities/Claudian-YishenTu.md
- relevant: wiki/comparisons/claude-obsidian-ecosystem.md
- category: derived
- rationale: Word-level diff preview and one-click apply with Plan Mode support.

### D22
- query: Between claude-obsidian and other projects, who has the hot cache mechanism?
- correct: wiki/comparisons/claude-obsidian-ecosystem.md
- relevant: wiki/concepts/Hot Cache.md, wiki/index.md
- category: derived
- rationale: Feature matrix shows claude-obsidian is unique in session context caching.

### D23
- query: When was the first real fold committed to the vault and what was its fold_id?
- correct: wiki/folds/fold-k3-from-2026-04-23-to-2026-04-24-n8.md
- relevant: wiki/log.md, wiki/hot.md, wiki/concepts/DragonScale Memory.md
- category: derived
- rationale: Fold metadata with exact creation date and 8-entry batch documentation.

### D24
- query: What pages were created after the boundary-first autoresearch frontier pass?
- correct: wiki/log.md
- relevant: wiki/concepts/Persistent Wiki Artifact.md, wiki/concepts/Source-First Synthesis.md, wiki/concepts/Query-Time Retrieval.md
- category: derived
- rationale: Log entry for 2026-04-24 M4 run explicitly lists the three new concept pages filed.

### D25
- query: What is a Persistent Wiki Artifact and how does it differ from ephemeral chat turns?
- correct: wiki/concepts/Persistent Wiki Artifact.md
- relevant: wiki/concepts/LLM Wiki Pattern.md, wiki/concepts/Source-First Synthesis.md
- category: derived
- rationale: Explicit definition of durable memory object with frontmatter, title, wikilinks, and provenance.

---

## Hard Queries (25)

### H1
- query: How does the exponential compaction operator summarize batch entries?
- correct: wiki/concepts/DragonScale Memory.md
- relevant: wiki/log.md
- category: synonym
- rationale: "Fold operator" and "rollup" are interchangeable; query uses LSM terminology instead of dragon-curve naming.

### H2
- query: What stable identifiers help page references survive renames in prompt-cache workloads?
- correct: wiki/concepts/DragonScale Memory.md
- relevant: wiki/log.md, wiki/hot.md
- category: synonym
- rationale: Deterministic addresses (c-NNNNNN format) are the answer; query uses prompt-cache context instead.

### H3
- query: How does the semantic tiling mechanism detect and flag duplicate or near-duplicate pages?
- correct: wiki/concepts/DragonScale Memory.md
- relevant: wiki/log.md, wiki/hot.md
- category: synonym
- rationale: Mechanism 3 performs dedup via cosine similarity; query uses "near-duplicate" instead of "semantic tiling."

### H4
- query: When user asks /autoresearch without a topic, which pages are candidates and how are they scored?
- correct: wiki/concepts/DragonScale Memory.md
- relevant: wiki/log.md, wiki/hot.md
- category: synonym
- rationale: Boundary-first autoresearch (Mechanism 4) surfaces frontier pages; query uses "autoresearch no-topic" instead.

### H5
- query: How should the wiki layer stay connected to raw source material without replacing source content?
- correct: wiki/concepts/Source-First Synthesis.md
- relevant: wiki/concepts/LLM Wiki Pattern.md, wiki/concepts/Persistent Wiki Artifact.md
- category: synonym
- rationale: Source-first synthesis is the provenance rule; query asks for the principle without using the term.

### H6
- query: When building a wiki versus RAG, how does the cost structure differ in maintenance, infrastructure, and long-term knowledge value?
- correct: wiki/concepts/LLM Wiki Pattern.md, wiki/comparisons/Wiki vs RAG.md, wiki/concepts/Compounding Knowledge.md
- relevant: wiki/overview.md
- category: cross-page
- rationale: Spans three concepts: architecture table, comparison table, and compounding mechanics.

### H7
- query: What does the wiki-fold skill output in dry-run mode and how does it create child references?
- correct: wiki/concepts/DragonScale Memory.md, wiki/log.md
- relevant: wiki/hot.md
- category: cross-page
- rationale: Mechanism 1 spec + dry-run artifact from 2026-04-24 Phase 1 log entry cross-reference.

### H8
- query: How would adding URL ingestion and web cleaning together reduce token usage during ingest?
- correct: wiki/concepts/cherry-picks.md, wiki/entities/kepano-obsidian-skills.md
- relevant: wiki/getting-started.md
- category: cross-page
- rationale: Combining cherry-pick #1 (URL ingest) and defuddle (40-60% token reduction) from two sources.

### H9
- query: Which project has the best auto-commit hook and how did that feature get integrated into Claude SEO?
- correct: wiki/entities/ballred-obsidian-claude-pkm.md, wiki/entities/Claude SEO.md, wiki/log.md
- relevant: wiki/comparisons/claude-obsidian-ecosystem.md
- category: cross-page
- rationale: ballred has PostToolUse auto-commit; Claude SEO v1.9.0 release cites integration patterns.

### H10
- query: How would you audit a page for SERP alignment, topic cannibalization risk, and performance regressions together?
- correct: wiki/concepts/Search Experience Optimization.md, wiki/concepts/Semantic Topic Clustering.md, wiki/concepts/SEO Drift Monitoring.md
- relevant: wiki/entities/Claude SEO.md, wiki/log.md
- category: cross-page
- rationale: Three separate v1.9.0 features; query asks for integrated workflow across all three.

### H11
- query: What were the six test stages and did all mechanisms pass validation before v1.6.0 shipped?
- correct: wiki/log.md, wiki/hot.md
- relevant: wiki/concepts/DragonScale Memory.md
- category: cross-page
- rationale: 2026-04-24 end-to-end validation pass entry lists T0-T6; hot.md confirms all four mechanisms validated.

### H12
- query: Why does the LLM Wiki pattern avoid re-querying and re-assembling the same sources every time?
- correct: wiki/concepts/Query-Time Retrieval.md, wiki/concepts/Persistent Wiki Artifact.md, wiki/concepts/Source-First Synthesis.md
- relevant: wiki/concepts/LLM Wiki Pattern.md, wiki/concepts/Compounding Knowledge.md
- category: cross-page
- rationale: Spans three pages: RAG baseline definition + durable memory object + synthesis discipline.

### H13
- query: What are two independent ways to embed the LLM Wiki pattern in different editor environments?
- correct: wiki/entities/Ar9av-obsidian-wiki.md, wiki/entities/Claudian-YishenTu.md
- relevant: wiki/comparisons/claude-obsidian-ecosystem.md
- category: cross-page
- rationale: Ar9av setup.sh for multiple agents; Claudian native TypeScript plugin; both different deployment models.

### H14
- query: How could you avoid re-processing unchanged sources when integrating community submissions?
- correct: wiki/concepts/cherry-picks.md, wiki/concepts/Pro Hub Challenge.md
- relevant: wiki/log.md
- category: cross-page
- rationale: Delta tracking manifest solves re-ingest problem; integration pattern in Pro Hub workflow.

### H15
- query: What is the most direct way to add defuddle to claude-obsidian's current ingest pipeline?
- correct: wiki/entities/kepano-obsidian-skills.md, wiki/concepts/cherry-picks.md
- relevant: wiki/getting-started.md
- category: cross-page
- rationale: Cherry-pick #3 + getting-started ingest flow both discussed; requires both for full picture.

### H16
- query: Where can I find Karpathy's original description of the wiki pattern?
- correct: wiki/concepts/Persistent Wiki Artifact.md, wiki/concepts/Source-First Synthesis.md, wiki/entities/Andrej Karpathy.md
- relevant: wiki/concepts/LLM Wiki Pattern.md
- category: partial-recall
- rationale: Multiple pages cite Karpathy gist; user remembers "gist" + "Karpathy."

### H17
- query: What was inspired by dragon curves and what problem does it solve?
- correct: wiki/concepts/DragonScale Memory.md
- relevant: wiki/log.md
- category: partial-recall
- rationale: User recalls dragon analogy but not "DragonScale"; wants to understand fold operator.

### H18
- query: What is the ~500-word file that gets updated at the end of every session?
- correct: wiki/concepts/Hot Cache.md
- relevant: wiki/getting-started.md, wiki/overview.md, wiki/hot.md
- category: partial-recall
- rationale: Exact description matches; user remembers fragment but not exact terminology.

### H19
- query: Which community competition paid out Claude Credits and who won first?
- correct: wiki/concepts/Pro Hub Challenge.md
- relevant: wiki/entities/Claude SEO.md
- category: partial-recall
- rationale: User remembers community challenge and winners but not the name.

### H20
- query: How many projects does the wiki track as competitors or adjacent to claude-obsidian?
- correct: wiki/comparisons/claude-obsidian-ecosystem.md
- relevant: wiki/concepts/cherry-picks.md
- category: partial-recall
- rationale: User remembers ecosystem matrix but not exact page; queries for count of projects.

### H21
- query: How does the wiki pattern use blockchain or distributed ledger technology for vault consensus?
- correct: null
- relevant: wiki/concepts/DragonScale Memory.md
- category: negative
- rationale: Wiki does not discuss blockchain; DragonScale addresses are creation-order counter, not DLT.

### H22
- query: What CRDT algorithm does the vault use for simultaneous multi-user editing?
- correct: null
- relevant: wiki/hot.md, wiki/concepts/DragonScale Memory.md
- category: negative
- rationale: No CRDT discussion; v1.7 mentions wiki-lock.sh for multi-writer safety (locking, not CRDTs).

### H23
- query: Can I build an LLM wiki without installing Obsidian as a desktop application?
- correct: null
- relevant: wiki/concepts/LLM Wiki Pattern.md, wiki/getting-started.md
- category: negative
- rationale: Pattern is agnostic but this vault uses Obsidian. No Obsidian-free mode documented.

### H24
- query: How do you customize the semantic embedding model weights for better tiling accuracy on your domain?
- correct: null
- relevant: wiki/concepts/DragonScale Memory.md, wiki/log.md
- category: negative
- rationale: Mechanism 3 uses locked ollama nomic-embed-text; no fine-tuning support documented.

### H25
- query: Does the vault have CI/CD pipeline to automatically ingest new .raw files on every push?
- correct: null
- relevant: wiki/log.md, wiki/hot.md
- category: negative
- rationale: No GitHub Actions / CI-CD / automated ingest documented. Auto-commit hook exists but not scheduled CI.

---

## Benchmark Notes

**Coverage**: 47 wiki pages, ~50 queries → roughly 1 query per page on average; spread covers concepts, entities, comparisons, meta, folds, sources.

**Balance**: 25 derived (natural questions) + 5 synonym + 10 cross-page + 5 partial-recall + 5 negative.

**Targets** (rough expectation, not a gate):
- v1.6 baseline top-1: ~50-65% on derived, much lower on hard
- v1.7 hybrid top-1: ~75-90% on derived, ~50-70% on hard
- Plan §7 ship gate (from v1.7 implementation plan): ≥30% reduction in "wrong page cited" errors. Verify by comparing the percentage-points improvement on top-1 across all 50 queries.