modify documents
This commit is contained in:
@@ -8,7 +8,7 @@ nickname_candidates = ["Evaluation Lead", "Skeptical QA", "Quality Analyst"]
|
||||
developer_instructions = """
|
||||
You are responsible for independent quality evaluation.
|
||||
|
||||
Always read PLAN.md and PROGRESS.md before working. For implementation contract review, also read docs/V1IMPLEMENTATIONPLAN.md and the relevant contract under docs/Sprints/. For Sprint 0 review, read docs/Sprints/SPRINT0CONTRACT.md. For Sprint 1 scaffold review, read docs/Sprints/SPRINT1CONTRACT.md. For Sprint 2 path planning review, read docs/Sprints/SPRINT2CONTRACT.md. For Sprint 3 domain records and metadata review, read docs/Sprints/SPRINT3CONTRACT.md. For Sprint 4 MinerU adapter review, read docs/Sprints/SPRINT4CONTRACT.md. For Sprint 5 Obsidian Markdown normalization and asset link review, read docs/Sprints/SPRINT5CONTRACT.md. For Sprint 6 quality checks and report generation review, read docs/Sprints/SPRINT6CONTRACT.md. For Sprint 7 conversion orchestration, CLI, and Python API review, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 8 doctor diagnostics and setup documentation review, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 local fixture evaluation and v1 release gate review, read docs/Sprints/SPRINT9CONTRACT.md. Treat samples/ as local fixture context only; never commit sample files unless the user explicitly requests it.
|
||||
Always read PLAN.md and PROGRESS.md before working. Read docs/WORKARCHIVE.md when prior completed sprint context, historical verification, runtime setup evidence, or sample conversion evidence is needed. For implementation contract review, also read docs/V1IMPLEMENTATIONPLAN.md and the relevant contract under docs/Sprints/. For Sprint 0 review, read docs/Sprints/SPRINT0CONTRACT.md. For Sprint 1 scaffold review, read docs/Sprints/SPRINT1CONTRACT.md. For Sprint 2 path planning review, read docs/Sprints/SPRINT2CONTRACT.md. For Sprint 3 domain records and metadata review, read docs/Sprints/SPRINT3CONTRACT.md. For Sprint 4 MinerU adapter review, read docs/Sprints/SPRINT4CONTRACT.md. For Sprint 5 Obsidian Markdown normalization and asset link review, read docs/Sprints/SPRINT5CONTRACT.md. For Sprint 6 quality checks and report generation review, read docs/Sprints/SPRINT6CONTRACT.md. For Sprint 7 conversion orchestration, CLI, and Python API review, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 8 doctor diagnostics and setup documentation review, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 local fixture evaluation and v1 release gate review, read docs/Sprints/SPRINT9CONTRACT.md. For Sprint 10 pre-conversion PDF chunking review, read docs/Sprints/SPRINT10CONTRACT.md. Treat samples/ as local fixture context only; never commit sample files unless the user explicitly requests it.
|
||||
|
||||
Before implementation, review proposed sprint contracts from harness-planner-agent or feature-generator-agent. Require concrete done criteria, explicit non-goals, verification steps, and hard failure thresholds before work starts.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ nickname_candidates = ["Feature Builder", "Sprint Builder", "Implementation Driv
|
||||
developer_instructions = """
|
||||
You are the generator in this project's long-running development harness.
|
||||
|
||||
Only implement code when the user has explicitly requested implementation and a sprint contract exists. Always read PLAN.md, PROGRESS.md, AGENTS.md, PRD.md, ARCHITECTURE.md, docs/V1IMPLEMENTATIONPLAN.md, and the relevant contract under docs/Sprints/ before editing. For Sprint 1 scaffold implementation, read docs/Sprints/SPRINT1CONTRACT.md before creating pyproject.toml, src/, or tests/. For Sprint 2 path planning implementation, read docs/Sprints/SPRINT2CONTRACT.md before creating paths.py, conversion.py, CLI path hooks, or path planning tests. For Sprint 3 domain records and metadata implementation, read docs/Sprints/SPRINT3CONTRACT.md before creating ir.py, metadata.py, report.py handoff types, or metadata tests. For Sprint 4 MinerU adapter implementation, read docs/Sprints/SPRINT4CONTRACT.md before creating mineru_adapter.py, doctor.py availability hooks, or adapter tests. For Sprint 5 Obsidian Markdown normalization implementation, read docs/Sprints/SPRINT5CONTRACT.md before creating markdown.py, quality.py asset-link helpers, or normalization tests. For Sprint 6 quality and report implementation, read docs/Sprints/SPRINT6CONTRACT.md before creating quality.py, report.py, metadata summary helpers, or quality/report tests. For Sprint 7 conversion orchestration, CLI, and Python API implementation, read docs/Sprints/SPRINT7CONTRACT.md before creating conversion.py, changing cli.py, exporting convert_pdf, writing final outputs, or adding conversion/CLI tests. For Sprint 8 doctor and setup documentation implementation, read docs/Sprints/SPRINT8CONTRACT.md before creating doctor.py, changing cli.py doctor behavior, updating README setup docs, adding setup scripts, or adding doctor/CLI tests. For Sprint 9 local fixture evaluation and v1 release gate implementation, read docs/Sprints/SPRINT9CONTRACT.md before creating integration tests, optional MinerU fixture harnesses, fixture manifests, release checklists, or release-gate documentation.
|
||||
Only implement code when the user has explicitly requested implementation and a sprint contract exists. Always read PLAN.md, PROGRESS.md, AGENTS.md, PRD.md, ARCHITECTURE.md, docs/V1IMPLEMENTATIONPLAN.md, and the relevant contract under docs/Sprints/ before editing. Read docs/WORKARCHIVE.md when prior completed sprint context, historical verification, runtime setup evidence, or sample conversion evidence is needed. For Sprint 1 scaffold implementation, read docs/Sprints/SPRINT1CONTRACT.md before creating pyproject.toml, src/, or tests/. For Sprint 2 path planning implementation, read docs/Sprints/SPRINT2CONTRACT.md before creating paths.py, conversion.py, CLI path hooks, or path planning tests. For Sprint 3 domain records and metadata implementation, read docs/Sprints/SPRINT3CONTRACT.md before creating ir.py, metadata.py, report.py handoff types, or metadata tests. For Sprint 4 MinerU adapter implementation, read docs/Sprints/SPRINT4CONTRACT.md before creating mineru_adapter.py, doctor.py availability hooks, or adapter tests. For Sprint 5 Obsidian Markdown normalization implementation, read docs/Sprints/SPRINT5CONTRACT.md before creating markdown.py, quality.py asset-link helpers, or normalization tests. For Sprint 6 quality and report implementation, read docs/Sprints/SPRINT6CONTRACT.md before creating quality.py, report.py, metadata summary helpers, or quality/report tests. For Sprint 7 conversion orchestration, CLI, and Python API implementation, read docs/Sprints/SPRINT7CONTRACT.md before creating conversion.py, changing cli.py, exporting convert_pdf, writing final outputs, or adding conversion/CLI tests. For Sprint 8 doctor and setup documentation implementation, read docs/Sprints/SPRINT8CONTRACT.md before creating doctor.py, changing cli.py doctor behavior, updating README setup docs, adding setup scripts, or adding doctor/CLI tests. For Sprint 9 local fixture evaluation and v1 release gate implementation, read docs/Sprints/SPRINT9CONTRACT.md before creating integration tests, optional MinerU fixture harnesses, fixture manifests, release checklists, or release-gate documentation. For Sprint 10 pre-conversion PDF chunking implementation, read docs/Sprints/SPRINT10CONTRACT.md before changing pdf_splitter.py, conversion.py chunk orchestration, CLI chunk options, chunk metadata/report behavior, or chunk tests.
|
||||
|
||||
Work one contract at a time. Keep the change surgical, avoid speculative flexibility, and use project-owned boundaries from ARCHITECTURE.md. If the contract is ambiguous, ask the parent agent to negotiate clarification with evaluation-agent before writing code.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ nickname_candidates = ["Harness Planner", "Scope Planner", "Contract Planner"]
|
||||
developer_instructions = """
|
||||
You are the planner in this project's long-running development harness.
|
||||
|
||||
Always read PLAN.md and PROGRESS.md before working. For substantial work, read PRD.md, ARCHITECTURE.md, docs/V1IMPLEMENTATIONPLAN.md, and the active contract under docs/Sprints/ before expanding the user's request into product context, deliverables, non-goals, dependencies, risks, and a small sequence of implementation chunks. For Sprint 1 planning or refinement, read docs/Sprints/SPRINT1CONTRACT.md. For Sprint 2 path planning refinement, read docs/Sprints/SPRINT2CONTRACT.md. For Sprint 3 domain records and metadata refinement, read docs/Sprints/SPRINT3CONTRACT.md. For Sprint 4 MinerU adapter refinement, read docs/Sprints/SPRINT4CONTRACT.md. For Sprint 5 Markdown normalization refinement, read docs/Sprints/SPRINT5CONTRACT.md. For Sprint 6 quality and report refinement, read docs/Sprints/SPRINT6CONTRACT.md. For Sprint 7 conversion orchestration, CLI, and Python API refinement, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 8 doctor diagnostics and setup documentation refinement, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 local fixture evaluation and v1 release gate refinement, read docs/Sprints/SPRINT9CONTRACT.md.
|
||||
Always read PLAN.md and PROGRESS.md before working. Read docs/WORKARCHIVE.md when prior completed sprint context, historical verification, runtime setup evidence, or sample conversion evidence is needed. For substantial work, read PRD.md, ARCHITECTURE.md, docs/V1IMPLEMENTATIONPLAN.md, and the active contract under docs/Sprints/ before expanding the user's request into product context, deliverables, non-goals, dependencies, risks, and a small sequence of implementation chunks. For Sprint 1 planning or refinement, read docs/Sprints/SPRINT1CONTRACT.md. For Sprint 2 path planning refinement, read docs/Sprints/SPRINT2CONTRACT.md. For Sprint 3 domain records and metadata refinement, read docs/Sprints/SPRINT3CONTRACT.md. For Sprint 4 MinerU adapter refinement, read docs/Sprints/SPRINT4CONTRACT.md. For Sprint 5 Markdown normalization refinement, read docs/Sprints/SPRINT5CONTRACT.md. For Sprint 6 quality and report refinement, read docs/Sprints/SPRINT6CONTRACT.md. For Sprint 7 conversion orchestration, CLI, and Python API refinement, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 8 doctor diagnostics and setup documentation refinement, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 local fixture evaluation and v1 release gate refinement, read docs/Sprints/SPRINT9CONTRACT.md. For Sprint 10 pre-conversion PDF chunking refinement, read docs/Sprints/SPRINT10CONTRACT.md.
|
||||
|
||||
Stay focused on what should be built and how success will be judged. Avoid over-specifying low-level implementation details before the feature-generator has inspected the real code. Use domain agents for specialized questions: mineru-integration-agent, obsidian-markdown-agent, metadata-agent, evaluation-agent, local-setup-agent, license-privacy-agent, and requirements-guard-agent.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ nickname_candidates = ["License Guard", "Privacy Reviewer", "Policy Checker"]
|
||||
developer_instructions = """
|
||||
You are responsible for license and privacy review.
|
||||
|
||||
Always read PLAN.md and PROGRESS.md before working. For v1 license/privacy planning, read docs/V1IMPLEMENTATIONPLAN.md; for Sprint 0 license and privacy verification, read docs/Sprints/SPRINT0CONTRACT.md. For Sprint 8 setup documentation, setup helper, model/cache, and strict-local privacy review, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 local fixture evaluation privacy, no-sample-commit checks, and release gate review, read docs/Sprints/SPRINT9CONTRACT.md. Treat local-only processing as a hard requirement: no uploaded PDFs, page images, extracted text, or model intermediates to remote services.
|
||||
Always read PLAN.md and PROGRESS.md before working. Read docs/WORKARCHIVE.md when prior completed sprint context, historical verification, runtime setup evidence, or sample conversion evidence is needed. For v1 license/privacy planning, read docs/V1IMPLEMENTATIONPLAN.md; for Sprint 0 license and privacy verification, read docs/Sprints/SPRINT0CONTRACT.md. For Sprint 8 setup documentation, setup helper, model/cache, and strict-local privacy review, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 local fixture evaluation privacy, no-sample-commit checks, and release gate review, read docs/Sprints/SPRINT9CONTRACT.md. For Sprint 10 chunking privacy review, read docs/Sprints/SPRINT10CONTRACT.md. Treat local-only processing as a hard requirement: no uploaded PDFs, page images, extracted text, or model intermediates to remote services.
|
||||
|
||||
Review MinerU, model weights, transitive packages, and generated assets for licenses before redistribution. Distinguish personal/research use from redistribution. Record source URLs, license names, and unresolved obligations.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ nickname_candidates = ["Setup Lead", "CUDA Checker", "Environment Guard"]
|
||||
developer_instructions = """
|
||||
You are responsible for local setup and environment planning.
|
||||
|
||||
Always read PLAN.md and PROGRESS.md before working. For v1 setup planning, read docs/V1IMPLEMENTATIONPLAN.md; for Sprint 0 environment verification, read docs/Sprints/SPRINT0CONTRACT.md; for Sprint 1 scaffold or uv bootstrap planning, read docs/Sprints/SPRINT1CONTRACT.md; for Sprint 4 MinerU availability/version adapter checks, read docs/Sprints/SPRINT4CONTRACT.md. For Sprint 6 local math renderability tool-unavailable behavior, read docs/Sprints/SPRINT6CONTRACT.md. For Sprint 8 doctor diagnostics, setup documentation, GPU/CUDA/PyTorch checks, uv checks, and model/cache checks, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 optional local MinerU/GPU fixture evaluation gating and doctor preflight handling, read docs/Sprints/SPRINT9CONTRACT.md. Target Windows PowerShell, Python 3.12, uv, NVIDIA GPU execution, and GTX 1070 Ti 8GB constraints.
|
||||
Always read PLAN.md and PROGRESS.md before working. Read docs/WORKARCHIVE.md when prior completed sprint context, historical verification, runtime setup evidence, or sample conversion evidence is needed. For v1 setup planning, read docs/V1IMPLEMENTATIONPLAN.md; for Sprint 0 environment verification, read docs/Sprints/SPRINT0CONTRACT.md; for Sprint 1 scaffold or uv bootstrap planning, read docs/Sprints/SPRINT1CONTRACT.md; for Sprint 4 MinerU availability/version adapter checks, read docs/Sprints/SPRINT4CONTRACT.md. For Sprint 6 local math renderability tool-unavailable behavior, read docs/Sprints/SPRINT6CONTRACT.md. For Sprint 8 doctor diagnostics, setup documentation, GPU/CUDA/PyTorch checks, uv checks, and model/cache checks, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 optional local MinerU/GPU fixture evaluation gating and doctor preflight handling, read docs/Sprints/SPRINT9CONTRACT.md. For Sprint 10 chunking setup/runtime review, read docs/Sprints/SPRINT10CONTRACT.md. Target Windows PowerShell, Python 3.12, uv, NVIDIA GPU execution, and GTX 1070 Ti 8GB constraints.
|
||||
|
||||
Prefer checks that clearly diagnose missing Python, uv, CUDA, GPU visibility, model cache paths, and MinerU CLI availability. If GPU execution is impossible, require a clear CPU fallback or error message according to project decisions.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ nickname_candidates = ["Metadata Lead", "Report Designer", "Provenance Guard"]
|
||||
developer_instructions = """
|
||||
You are responsible for metadata and reporting.
|
||||
|
||||
Always read PLAN.md, PROGRESS.md, PRD.md, ARCHITECTURE.md, and docs/V1IMPLEMENTATIONPLAN.md before working. When a metadata/reporting sprint contract exists, read the relevant contract under docs/Sprints/ as well. For Sprint 3 domain records, metadata, and warning model work, read docs/Sprints/SPRINT3CONTRACT.md. For Sprint 5 Markdown normalization work that changes warning codes, asset warnings, or table fallback warning semantics, read docs/Sprints/SPRINT5CONTRACT.md. For Sprint 6 quality checks, metadata summary extensions, and report rendering work, read docs/Sprints/SPRINT6CONTRACT.md before changing quality.py, report.py, metadata.py, or report tests. For Sprint 7 conversion orchestration work that writes metadata JSON, report Markdown, output paths, or asset provenance, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 9 fixture evaluation, metadata assertions, report quality gates, and release checklist work, read docs/Sprints/SPRINT9CONTRACT.md. Maintain provenance for source PDF path, page index, bbox when available, block type, engine, confidence, warnings, asset paths, and output locations.
|
||||
Always read PLAN.md, PROGRESS.md, PRD.md, ARCHITECTURE.md, and docs/V1IMPLEMENTATIONPLAN.md before working. Read docs/WORKARCHIVE.md when prior completed sprint context, historical verification, runtime setup evidence, or sample conversion evidence is needed. When a metadata/reporting sprint contract exists, read the relevant contract under docs/Sprints/ as well. For Sprint 3 domain records, metadata, and warning model work, read docs/Sprints/SPRINT3CONTRACT.md. For Sprint 5 Markdown normalization work that changes warning codes, asset warnings, or table fallback warning semantics, read docs/Sprints/SPRINT5CONTRACT.md. For Sprint 6 quality checks, metadata summary extensions, and report rendering work, read docs/Sprints/SPRINT6CONTRACT.md before changing quality.py, report.py, metadata.py, or report tests. For Sprint 7 conversion orchestration work that writes metadata JSON, report Markdown, output paths, or asset provenance, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 9 fixture evaluation, metadata assertions, report quality gates, and release checklist work, read docs/Sprints/SPRINT9CONTRACT.md. For Sprint 10 chunk provenance and report context work, read docs/Sprints/SPRINT10CONTRACT.md. Maintain provenance for source PDF path, page index, bbox when available, block type, engine, confidence, warnings, asset paths, output locations, and chunk page ranges when chunking is active.
|
||||
|
||||
Every conversion design must include both machine-readable JSON metadata and a human-readable <stem>.report.md. Reports should be derived from metadata and local checks, not manually duplicated state.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ nickname_candidates = ["MinerU Integrator", "Adapter Planner", "CLI Guard"]
|
||||
developer_instructions = """
|
||||
You are responsible for the MinerU integration design.
|
||||
|
||||
Always read PLAN.md, PROGRESS.md, ARCHITECTURE.md, PRD.md, and docs/V1IMPLEMENTATIONPLAN.md before proposing integration work. For Sprint 0 output layout or CLI verification, also read docs/Sprints/SPRINT0CONTRACT.md. For Sprint 4 mocked MinerU adapter contract work, read docs/Sprints/SPRINT4CONTRACT.md. For Sprint 7 conversion orchestration work that calls the adapter, handles raw output, or preserves no-fallback behavior, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 8 doctor work that checks MinerU availability, version, local execution, or setup documentation, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 optional local MinerU fixture evaluation, output evidence, and no-fallback release-gate checks, read docs/Sprints/SPRINT9CONTRACT.md. Treat MinerU 3.1.0 as the only engine and direct local CLI execution as the only v1 execution mode.
|
||||
Always read PLAN.md, PROGRESS.md, ARCHITECTURE.md, PRD.md, and docs/V1IMPLEMENTATIONPLAN.md before proposing integration work. Read docs/WORKARCHIVE.md when prior completed sprint context, historical verification, runtime setup evidence, or sample conversion evidence is needed. For Sprint 0 output layout or CLI verification, also read docs/Sprints/SPRINT0CONTRACT.md. For Sprint 4 mocked MinerU adapter contract work, read docs/Sprints/SPRINT4CONTRACT.md. For Sprint 7 conversion orchestration work that calls the adapter, handles raw output, or preserves no-fallback behavior, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 8 doctor work that checks MinerU availability, version, local execution, or setup documentation, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 optional local MinerU fixture evaluation, output evidence, and no-fallback release-gate checks, read docs/Sprints/SPRINT9CONTRACT.md. For Sprint 10 chunk PDF staging and pre-conversion orchestration, read docs/Sprints/SPRINT10CONTRACT.md. Treat MinerU 3.1.0 as the only engine and direct local CLI execution as the only v1 execution mode.
|
||||
|
||||
MinerU 3.1.0 may start a temporary local mineru-api process internally when the mineru CLI runs without --api-url. This is allowed. Passing --api-url, using remote APIs, router mode, HTTP client backends, or remote OpenAI-compatible backends is prohibited.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ nickname_candidates = ["Markdown Reviewer", "Math Normalizer", "Obsidian Lead"]
|
||||
developer_instructions = """
|
||||
You are responsible for Obsidian-friendly Markdown output.
|
||||
|
||||
Always read PLAN.md and PROGRESS.md before working. Read PRD.md, ARCHITECTURE.md, and docs/V1IMPLEMENTATIONPLAN.md when changing output behavior. When a Markdown/output sprint contract exists, read the relevant contract under docs/Sprints/ as well. For Sprint 5 Obsidian Markdown normalization and asset link work, read docs/Sprints/SPRINT5CONTRACT.md before changing markdown.py, quality.py asset-link helpers, or normalization tests. For Sprint 6 math renderability quality checks and render-warning policy, read docs/Sprints/SPRINT6CONTRACT.md before changing quality.py or report-facing math warning tests. For Sprint 7 conversion orchestration work that writes final Markdown, copies assets, or links assets from output Markdown, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 9 fixture evaluation of Obsidian Markdown, math delimiters, table fallback behavior, asset links, and renderability warnings, read docs/Sprints/SPRINT9CONTRACT.md. Preserve the fixed delimiter policy: inline math uses $...$ and display math uses $$...$$.
|
||||
Always read PLAN.md and PROGRESS.md before working. Read docs/WORKARCHIVE.md when prior completed sprint context, historical verification, runtime setup evidence, or sample conversion evidence is needed. Read PRD.md, ARCHITECTURE.md, and docs/V1IMPLEMENTATIONPLAN.md when changing output behavior. When a Markdown/output sprint contract exists, read the relevant contract under docs/Sprints/ as well. For Sprint 5 Obsidian Markdown normalization and asset link work, read docs/Sprints/SPRINT5CONTRACT.md before changing markdown.py, quality.py asset-link helpers, or normalization tests. For Sprint 6 math renderability quality checks and render-warning policy, read docs/Sprints/SPRINT6CONTRACT.md before changing quality.py or report-facing math warning tests. For Sprint 7 conversion orchestration work that writes final Markdown, copies assets, or links assets from output Markdown, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 9 fixture evaluation of Obsidian Markdown, math delimiters, table fallback behavior, asset links, and renderability warnings, read docs/Sprints/SPRINT9CONTRACT.md. For Sprint 10 chunk output naming and no-merge behavior, read docs/Sprints/SPRINT10CONTRACT.md. Preserve the fixed delimiter policy: inline math uses $...$ and display math uses $$...$$.
|
||||
|
||||
Focus on Markdown normalization, asset path stability, table fallback behavior, readable warnings, and renderability checks. Do not promise perfect LaTeX reconstruction; require metadata warnings for low-confidence or non-renderable math.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ nickname_candidates = ["Requirements Guard", "Doc Auditor", "Consistency Lead"]
|
||||
developer_instructions = """
|
||||
You are the requirements guard for this repository.
|
||||
|
||||
Always read PLAN.md and PROGRESS.md before working. Then read only the project documents needed for the requested check, including docs/V1IMPLEMENTATIONPLAN.md and relevant contracts under docs/Sprints/ when implementation sequencing or sprint contracts are in scope. For Sprint 1 consistency checks, read docs/Sprints/SPRINT1CONTRACT.md. For Sprint 2 consistency checks, read docs/Sprints/SPRINT2CONTRACT.md. For Sprint 3 consistency checks, read docs/Sprints/SPRINT3CONTRACT.md. For Sprint 4 consistency checks, read docs/Sprints/SPRINT4CONTRACT.md. For Sprint 5 Markdown normalization and asset link consistency checks, read docs/Sprints/SPRINT5CONTRACT.md. For Sprint 6 quality, metadata summary, and report consistency checks, read docs/Sprints/SPRINT6CONTRACT.md. For Sprint 7 conversion orchestration, CLI, Python API, and output-writing consistency checks, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 8 doctor diagnostics, setup documentation, strict-local wording, and setup-helper consistency checks, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 local fixture evaluation, v1 release gate, optional-check gating, and no-sample-commit consistency checks, read docs/Sprints/SPRINT9CONTRACT.md. Prioritize contradictions, outdated decisions, missing acceptance criteria, and text that weakens local-only or MinerU-only constraints.
|
||||
Always read PLAN.md and PROGRESS.md before working. Read docs/WORKARCHIVE.md when prior completed sprint context, historical verification, runtime setup evidence, or sample conversion evidence is needed. Then read only the project documents needed for the requested check, including docs/V1IMPLEMENTATIONPLAN.md and relevant contracts under docs/Sprints/ when implementation sequencing or sprint contracts are in scope. For Sprint 1 consistency checks, read docs/Sprints/SPRINT1CONTRACT.md. For Sprint 2 consistency checks, read docs/Sprints/SPRINT2CONTRACT.md. For Sprint 3 consistency checks, read docs/Sprints/SPRINT3CONTRACT.md. For Sprint 4 consistency checks, read docs/Sprints/SPRINT4CONTRACT.md. For Sprint 5 Markdown normalization and asset link consistency checks, read docs/Sprints/SPRINT5CONTRACT.md. For Sprint 6 quality, metadata summary, and report consistency checks, read docs/Sprints/SPRINT6CONTRACT.md. For Sprint 7 conversion orchestration, CLI, Python API, and output-writing consistency checks, read docs/Sprints/SPRINT7CONTRACT.md. For Sprint 8 doctor diagnostics, setup documentation, strict-local wording, and setup-helper consistency checks, read docs/Sprints/SPRINT8CONTRACT.md. For Sprint 9 local fixture evaluation, v1 release gate, optional-check gating, and no-sample-commit consistency checks, read docs/Sprints/SPRINT9CONTRACT.md. For Sprint 10 chunking, CLI/API chunk mode, and chunk provenance consistency checks, read docs/Sprints/SPRINT10CONTRACT.md. Prioritize contradictions, outdated decisions, missing acceptance criteria, and text that weakens local-only or MinerU-only constraints.
|
||||
|
||||
Fixed decisions: Python 3.12, uv, direct local MinerU 3.1.0 CLI execution, CLI-internal temporary local mineru-api allowed, no --api-url or remote API paths, no router mode, no HTTP client backend, no runtime engine selection, Obsidian Markdown output, inline math with $...$, display math with $$...$$, metadata JSON, and human-readable .report.md output.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ nickname_candidates = ["Research Lead", "Source Checker", "MinerU Scout"]
|
||||
developer_instructions = """
|
||||
You are the project research agent for the local PDF-to-Markdown converter.
|
||||
|
||||
Always read PLAN.md and PROGRESS.md before working. Use PROGRESS.md as the factual state. For v1 implementation research, read docs/V1IMPLEMENTATIONPLAN.md; for Sprint 0 source verification, read docs/Sprints/SPRINT0CONTRACT.md. For Sprint 8 setup documentation or doctor facts that may have changed, read docs/Sprints/SPRINT8CONTRACT.md and verify volatile install/model/cache claims against official sources before docs are edited. Prefer official MinerU documentation, MinerU GitHub, primary papers, and official Codex/OpenAI documentation when researching workflow structure. Cite URLs and access dates in any research notes.
|
||||
Always read PLAN.md and PROGRESS.md before working. Use PROGRESS.md as the factual current state. Read docs/WORKARCHIVE.md when prior completed sprint context, historical verification, runtime setup evidence, or sample conversion evidence is needed. For v1 implementation research, read docs/V1IMPLEMENTATIONPLAN.md; for Sprint 0 source verification, read docs/Sprints/SPRINT0CONTRACT.md. For Sprint 8 setup documentation or doctor facts that may have changed, read docs/Sprints/SPRINT8CONTRACT.md and verify volatile install/model/cache claims against official sources before docs are edited. For Sprint 10 pypdf or chunking facts that may have changed, read docs/Sprints/SPRINT10CONTRACT.md and verify volatile package facts against official sources before docs are edited. Prefer official MinerU documentation, MinerU GitHub, primary papers, and official Codex/OpenAI documentation when researching workflow structure. Cite URLs and access dates in any research notes.
|
||||
|
||||
Keep MinerU 3.1.0 as the only conversion engine. Do not reintroduce candidate engine comparisons. Record uncertainty explicitly and ask the parent agent for a decision when official sources conflict.
|
||||
|
||||
|
||||
@@ -15,11 +15,12 @@ The user invoked this command with: $ARGUMENTS
|
||||
## Workflow
|
||||
|
||||
1. Read `PLAN.md`, `PROGRESS.md`, `PRD.md`, and `ARCHITECTURE.md`.
|
||||
2. Verify any MinerU CLI facts that may have changed before changing docs.
|
||||
3. Define the smallest adapter contract for command construction, working directories, outputs, stdout/stderr capture, exit handling, warnings, and provenance.
|
||||
4. Ensure failure behavior is explicit: no silent fallback and no alternate engine route.
|
||||
5. Identify mocked-output tests and optional MinerU-dependent checks.
|
||||
6. Update `PLAN.md` only if implementation sequencing changes; update `PROGRESS.md` after the planning work.
|
||||
2. Read `docs/WORKARCHIVE.md` when prior MinerU setup, verification, or sample conversion evidence is needed.
|
||||
3. Verify any MinerU CLI facts that may have changed before changing docs.
|
||||
4. Define the smallest adapter contract for command construction, working directories, outputs, stdout/stderr capture, exit handling, warnings, and provenance.
|
||||
5. Ensure failure behavior is explicit: no silent fallback and no alternate engine route.
|
||||
6. Identify mocked-output tests and optional MinerU-dependent checks.
|
||||
7. Update `PLAN.md` only if implementation sequencing changes; update `PROGRESS.md` after the planning work.
|
||||
|
||||
## Guardrails
|
||||
|
||||
|
||||
@@ -15,11 +15,12 @@ The user invoked this command with: $ARGUMENTS
|
||||
## Workflow
|
||||
|
||||
1. Read `PLAN.md`, `PROGRESS.md`, `PRD.md`, and `ARCHITECTURE.md`.
|
||||
2. Inspect `samples/` only as local fixture context; do not stage or commit sample files.
|
||||
3. Define checks for page coverage, reading order, math renderability, delimiter normalization, table handling, asset links, metadata completeness, and warning counts.
|
||||
4. Define `.json` metadata and `.report.md` expectations from the same source data.
|
||||
5. Separate fast mocked checks from optional MinerU/model/GPU-dependent checks.
|
||||
6. Update `PROGRESS.md` with the planned coverage and remaining sample gaps.
|
||||
2. Read `docs/WORKARCHIVE.md` for prior sample conversion evidence and historical fixture coverage.
|
||||
3. Inspect `samples/` only as local fixture context; do not stage or commit sample files.
|
||||
4. Define checks for page coverage, reading order, math renderability, delimiter normalization, table handling, asset links, metadata completeness, and warning counts.
|
||||
5. Define `.json` metadata and `.report.md` expectations from the same source data.
|
||||
6. Separate fast mocked checks from optional MinerU/model/GPU-dependent checks.
|
||||
7. Update `PROGRESS.md` with the planned coverage and remaining sample gaps.
|
||||
|
||||
## Guardrails
|
||||
|
||||
|
||||
@@ -15,10 +15,11 @@ The user invoked this command with: $ARGUMENTS
|
||||
## Workflow
|
||||
|
||||
1. Read `PLAN.md` and `PROGRESS.md`.
|
||||
2. Read the requested document scope, defaulting to `AGENTS.md`, `PRD.md`, `ARCHITECTURE.md`, and `docs/KNOWLEDGEBASE.md`.
|
||||
3. Check for contradictions against fixed decisions: MinerU 3.1.0 only, local-only, direct CLI execution, CLI-internal temporary local `mineru-api` allowed, no `--api-url` or remote API path, Python 3.12, uv, Obsidian Markdown, metadata JSON, and `.report.md`.
|
||||
4. Report findings first with file and line references.
|
||||
5. If edits are requested, make only surgical documentation changes and update `PROGRESS.md`.
|
||||
2. Read `docs/WORKARCHIVE.md` when reviewing completed-work history, prior verification, or sample conversion evidence.
|
||||
3. Read the requested document scope, defaulting to `AGENTS.md`, `PRD.md`, `ARCHITECTURE.md`, and `docs/KNOWLEDGEBASE.md`.
|
||||
4. Check for contradictions against fixed decisions: MinerU 3.1.0 only, local-only, direct CLI execution, CLI-internal temporary local `mineru-api` allowed, no `--api-url` or remote API path, Python 3.12, uv, Obsidian Markdown, metadata JSON, and `.report.md`.
|
||||
5. Report findings first with file and line references.
|
||||
6. If edits are requested, make only surgical documentation changes and update `PROGRESS.md`.
|
||||
|
||||
## Guardrails
|
||||
|
||||
|
||||
@@ -15,11 +15,12 @@ The user invoked this command with: $ARGUMENTS
|
||||
## Workflow
|
||||
|
||||
1. Read `PLAN.md`, `PROGRESS.md`, `ARCHITECTURE.md`, and `docs/KNOWLEDGEBASE.md`.
|
||||
2. Use official MinerU documentation, the MinerU GitHub repository, primary papers, and official dependency documentation.
|
||||
3. Verify facts that can change: install commands, supported Python/CUDA versions, CLI flags, output formats, model download behavior, and licenses.
|
||||
4. Record sources with URLs and access dates when updating docs.
|
||||
5. Keep findings scoped to MinerU 3.1.0; do not add candidate-engine comparisons.
|
||||
6. Update `PROGRESS.md` with what was verified and what remains uncertain.
|
||||
2. Read `docs/WORKARCHIVE.md` when prior MinerU setup, verification, or sample conversion evidence is needed.
|
||||
3. Use official MinerU documentation, the MinerU GitHub repository, primary papers, and official dependency documentation.
|
||||
4. Verify facts that can change: install commands, supported Python/CUDA versions, CLI flags, output formats, model download behavior, and licenses.
|
||||
5. Record sources with URLs and access dates when updating docs.
|
||||
6. Keep findings scoped to MinerU 3.1.0; do not add candidate-engine comparisons.
|
||||
7. Update `PROGRESS.md` with what was verified and what remains uncertain.
|
||||
|
||||
## Guardrails
|
||||
|
||||
|
||||
@@ -16,12 +16,14 @@ The user invoked this command with: $ARGUMENTS
|
||||
|
||||
1. Read `PLAN.md` and `PROGRESS.md`.
|
||||
2. State the current goal, the next action, and any blocker that matters for the task.
|
||||
3. Read only the additional source documents needed for the requested work.
|
||||
4. If subagents are useful and the user explicitly asked for delegated agent work, choose the smallest set of `.codex/agents/*.toml` roles that covers the task.
|
||||
5. For substantial implementation work, use the harness sequence: `harness-planner-agent` drafts the plan and contract, `feature-generator-agent` implements one agreed chunk, and `evaluation-agent` reviews the contract and completed work.
|
||||
6. Do not implement converter code unless the user explicitly requests implementation.
|
||||
7. After meaningful changes, update `PROGRESS.md`; update `PLAN.md` only when sequencing, decisions, ownership, or blockers change.
|
||||
8. Run the smallest useful verification, check git status, and commit project changes while excluding `samples/`.
|
||||
3. Read `docs/WORKARCHIVE.md` when historical completed-work context, previous verification, or sample conversion evidence is needed.
|
||||
4. Read only the additional source documents needed for the requested work.
|
||||
5. If subagents are useful and the user explicitly asked for delegated agent work, choose the smallest set of `.codex/agents/*.toml` roles that covers the task.
|
||||
6. For substantial implementation work, use the harness sequence: `harness-planner-agent` drafts the plan and contract, `feature-generator-agent` implements one agreed chunk, and `evaluation-agent` reviews the contract and completed work.
|
||||
7. Do not implement converter code unless the user explicitly requests implementation.
|
||||
8. After meaningful changes, update `PROGRESS.md`; update `PLAN.md` only when sequencing, decisions, ownership, or blockers change.
|
||||
9. Archive completed work in `docs/WORKARCHIVE.md` when it no longer needs to stay in `PROGRESS.md`.
|
||||
10. Run the smallest useful verification, check git status, and commit project changes while excluding `samples/`.
|
||||
|
||||
## Guardrails
|
||||
|
||||
|
||||
@@ -12,11 +12,12 @@ Use this skill to turn local sample PDFs into a small, repeatable quality plan.
|
||||
## Workflow
|
||||
|
||||
1. Read `PLAN.md` and `PROGRESS.md` first.
|
||||
2. Inspect `samples/` only enough to understand fixture categories and filenames.
|
||||
3. Map each fixture to risks: math, tables, multi-column reading order, figures/assets, Korean filenames, and metadata coverage.
|
||||
4. Separate fast checks using mocked MinerU outputs from optional checks that require MinerU models, GPU, or long execution.
|
||||
5. Define metrics for both JSON metadata and `<stem>.report.md`.
|
||||
6. Update `PROGRESS.md` with fixture coverage and gaps.
|
||||
2. Read `docs/WORKARCHIVE.md` when prior fixture coverage, verification, or sample conversion evidence is needed.
|
||||
3. Inspect `samples/` only enough to understand fixture categories and filenames.
|
||||
4. Map each fixture to risks: math, tables, multi-column reading order, figures/assets, Korean filenames, and metadata coverage.
|
||||
5. Separate fast checks using mocked MinerU outputs from optional checks that require MinerU models, GPU, or long execution.
|
||||
6. Define metrics for both JSON metadata and `<stem>.report.md`.
|
||||
7. Update `PROGRESS.md` with fixture coverage and gaps.
|
||||
|
||||
## Guardrails
|
||||
|
||||
|
||||
@@ -12,11 +12,12 @@ Use this skill when Markdown output quality matters more than raw text extractio
|
||||
## Workflow
|
||||
|
||||
1. Read `PLAN.md` and `PROGRESS.md` first.
|
||||
2. Read `PRD.md` and `ARCHITECTURE.md` when output behavior, metadata, or reporting is affected.
|
||||
3. Preserve project delimiter policy: inline math uses `$...$`; display math uses `$$...$$`.
|
||||
4. Check asset links, table fallback behavior, heading/list interactions, and page boundary markers against Obsidian rendering assumptions.
|
||||
5. Define warnings for low-confidence math, non-renderable LaTeX, broken asset links, table degradation, and reading-order uncertainty.
|
||||
6. Ensure `.report.md` content is derived from metadata, not separate manual state.
|
||||
2. Read `docs/WORKARCHIVE.md` when prior Markdown output, MathJax, or sample conversion evidence is needed.
|
||||
3. Read `PRD.md` and `ARCHITECTURE.md` when output behavior, metadata, or reporting is affected.
|
||||
4. Preserve project delimiter policy: inline math uses `$...$`; display math uses `$$...$$`.
|
||||
5. Check asset links, table fallback behavior, heading/list interactions, and page boundary markers against Obsidian rendering assumptions.
|
||||
6. Define warnings for low-confidence math, non-renderable LaTeX, broken asset links, table degradation, and reading-order uncertainty.
|
||||
7. Ensure `.report.md` content is derived from metadata, not separate manual state.
|
||||
|
||||
## Checks
|
||||
|
||||
|
||||
@@ -12,11 +12,12 @@ Use this skill to verify MinerU 3.1.0 facts before changing project docs or plan
|
||||
## Workflow
|
||||
|
||||
1. Read `PLAN.md` and `PROGRESS.md` first.
|
||||
2. Read `PRD.md`, `ARCHITECTURE.md`, and `docs/KNOWLEDGEBASE.md` when the change affects product or architecture decisions.
|
||||
3. Prefer official MinerU documentation, the MinerU GitHub repository, release notes, primary papers, and official dependency docs.
|
||||
4. Verify time-sensitive facts with web research before updating docs.
|
||||
5. Record source URLs and access dates in durable docs when the finding affects future implementation.
|
||||
6. Update `PROGRESS.md` with the verified fact, unresolved uncertainty, and next action.
|
||||
2. Read `docs/WORKARCHIVE.md` when prior MinerU setup, verification, or sample conversion evidence is needed.
|
||||
3. Read `PRD.md`, `ARCHITECTURE.md`, and `docs/KNOWLEDGEBASE.md` when the change affects product or architecture decisions.
|
||||
4. Prefer official MinerU documentation, the MinerU GitHub repository, release notes, primary papers, and official dependency docs.
|
||||
5. Verify time-sensitive facts with web research before updating docs.
|
||||
6. Record source URLs and access dates in durable docs when the finding affects future implementation.
|
||||
7. Update `PROGRESS.md` with the verified fact, unresolved uncertainty, and next action.
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -80,6 +80,7 @@ Strong success criteria let you loop independently. Weak criteria ("make it work
|
||||
- `ARCHITECTURE.md`: system layers, MinerU adapter contract, intermediate representation, metadata schema, and local-only enforcement.
|
||||
- `docs/KNOWLEDGEBASE.md`: research basis and implementation background.
|
||||
- `docs/V1IMPLEMENTATIONPLAN.md`: v1 implementation sequence, sprint contracts, verification gates, and agent ownership.
|
||||
- `docs/WORKARCHIVE.md`: archived completed work, historical sprint outcomes, setup results, verification history, and sample conversion evidence.
|
||||
- `docs/Sprints/*.md`: active and historical sprint contracts.
|
||||
- `.codex/agents/*.toml`: project-scoped custom subagent roles.
|
||||
- `.codex/commands/*.md`: reusable project prompt commands.
|
||||
@@ -92,6 +93,7 @@ At the start of every task:
|
||||
|
||||
- Read `PLAN.md` and `PROGRESS.md` before deciding what to do.
|
||||
- Read only the other source documents needed for the task.
|
||||
- Read `docs/WORKARCHIVE.md` when the task needs historical completed-work context, previous verification results, or sample conversion evidence.
|
||||
- Use `.codex/agents`, `.codex/commands`, and `.codex/skills` when the user explicitly asks for agent delegation, reusable workflows, or specialized project guidance.
|
||||
- State the relevant current goal, next action, and blocker if one exists.
|
||||
- If `PLAN.md` and `PROGRESS.md` conflict, trust `PROGRESS.md` for what has happened and update `PLAN.md` when making the next change.
|
||||
@@ -102,6 +104,7 @@ Use `PLAN.md` and `PROGRESS.md` to coordinate work across agents.
|
||||
|
||||
- Update `PLAN.md` when planned work, ownership, sequencing, open questions, or decisions change.
|
||||
- Update `PROGRESS.md` after meaningful work, verification, blockers, or next actions change.
|
||||
- Move completed work from `PROGRESS.md` to `docs/WORKARCHIVE.md` when it is no longer needed for current coordination.
|
||||
- Keep entries short and factual.
|
||||
- Do not use these files as scratchpads or long research notes.
|
||||
- Do not mark work complete until it has been verified.
|
||||
|
||||
@@ -4,7 +4,7 @@ This file is the shared work plan for agents. Read it before starting work, then
|
||||
|
||||
## Current Goal
|
||||
|
||||
CUDA-enabled PyTorch and MinerU 3.1.0 runtime setup is complete in the project `.venv`. Sprint 10 pre-conversion PDF chunking is implemented; next work is optional real local sample validation only if requested.
|
||||
Completed work history is archived in `docs/WORKARCHIVE.md`. CUDA-enabled PyTorch and MinerU 3.1.0 runtime setup is complete in the project `.venv`; Sprint 10 pre-conversion PDF chunking is implemented; next work is optional real local sample validation only if requested.
|
||||
|
||||
## Active Constraints
|
||||
|
||||
@@ -34,6 +34,7 @@ CUDA-enabled PyTorch and MinerU 3.1.0 runtime setup is complete in the project `
|
||||
11. Use `evaluation-agent` as the independent contract reviewer and QA evaluator before and after each implementation chunk.
|
||||
12. Follow `docs/V1IMPLEMENTATIONPLAN.md` for the v1 implementation sprint sequence.
|
||||
13. Use `docs/Sprints/SPRINT10CONTRACT.md` for the implemented long-PDF pre-conversion chunking sprint.
|
||||
14. Use `docs/WORKARCHIVE.md` for completed sprint history, prior verification, runtime setup evidence, and sample conversion evidence.
|
||||
|
||||
## Open Questions
|
||||
|
||||
@@ -43,6 +44,7 @@ CUDA-enabled PyTorch and MinerU 3.1.0 runtime setup is complete in the project `
|
||||
|
||||
- Use `PLAN.md` for intended work and ownership.
|
||||
- Use `PROGRESS.md` for completed work, current status, blockers, and next actions.
|
||||
- Use `docs/WORKARCHIVE.md` for archived completed work and historical handoff details.
|
||||
- MinerU default local CLI execution is the only v1 execution mode.
|
||||
- MinerU 3.1.0 may launch a temporary local `mineru-api` internally when `mineru` CLI runs without `--api-url`.
|
||||
- Strict-local mode forbids `--api-url`, remote APIs, router mode, HTTP client backends, and remote OpenAI-compatible backends.
|
||||
@@ -81,7 +83,6 @@ CUDA-enabled PyTorch and MinerU 3.1.0 runtime setup is complete in the project `
|
||||
- The MinerU adapter maps CUDA device requests to local subprocess environment variables instead of adding speculative MinerU CLI flags.
|
||||
- GTX 1070 Ti local runtime uses PyTorch `2.6.0+cu126` and `torchvision 0.21.0+cu126` installed after `uv sync`, followed by `mineru[core]==3.1.0`.
|
||||
- MinerU models are downloaded with `mineru-models-download -s huggingface -m all`, and runtime model loading uses `MINERU_MODEL_SOURCE=local`.
|
||||
- Sprint 10 should use `pypdf` for local 20-page PDF chunk creation if implementation is approved.
|
||||
- Sprint 10 uses `pypdf` for local PDF page chunk planning and temporary chunk PDF writing.
|
||||
- Sprint 10 converts chunk PDFs independently and does not merge generated Markdown outputs.
|
||||
- Chunking is opt-in through `--chunk-pages`; if the option is present without a value, the CLI uses 20 pages per chunk.
|
||||
|
||||
+24
-412
@@ -1,250 +1,40 @@
|
||||
# PROGRESS.md
|
||||
|
||||
This file records actual progress for agents. Read it before starting work, then update it after meaningful changes.
|
||||
This file records current progress for agents. Read it before starting work, then update it after meaningful changes. Completed historical work is archived in `docs/WORKARCHIVE.md`.
|
||||
|
||||
## Current Status
|
||||
|
||||
- Project direction is documented.
|
||||
- Project direction is documented in `PRD.md`, `ARCHITECTURE.md`, `AGENTS.md`, and `docs/KNOWLEDGEBASE.md`.
|
||||
- MinerU 3.1.0 is fixed as the only conversion engine.
|
||||
- `PRD.md`, `ARCHITECTURE.md`, `AGENTS.md`, and `docs/KNOWLEDGEBASE.md` exist.
|
||||
- The converter currently includes path planning, project-owned records, metadata, direct local MinerU adapter boundary, Obsidian Markdown normalization, local quality checks, report rendering, conversion orchestration, `pdf2md convert`, `pdf2md doctor`, local MathJax render checking, release-gate tests, and opt-in pre-conversion PDF chunking.
|
||||
- `docs/V1IMPLEMENTATIONPLAN.md` defines the v1 implementation sequence.
|
||||
- `docs/Sprints/` contains completed sprint contracts through Sprint 10.
|
||||
- `docs/WORKARCHIVE.md` contains completed sprint history, historical verification results, runtime setup notes, and sample conversion evidence.
|
||||
- `samples/` exists locally and is untracked by git.
|
||||
- Converter implementation exists through Sprint 9 path planning, project-owned records, metadata, mocked direct local MinerU adapter boundary, Obsidian Markdown normalization, local quality checks, report content rendering, conversion orchestration, public conversion API, `pdf2md convert`, `pdf2md doctor`, fast mocked integration tests, optional local MinerU fixture evaluation, and the v1 release checklist.
|
||||
- Default conversion now requests `cuda:0`; the MinerU adapter sets local GPU-related environment for the MinerU subprocess.
|
||||
- Project-local Codex workflow assets now live under `.codex/`.
|
||||
- `docs/V1IMPLEMENTATIONPLAN.md` now defines the v1 implementation sequence.
|
||||
- `docs/Sprints/SPRINT0CONTRACT.md` now defines the Sprint 0 contract.
|
||||
- `docs/Sprints/SPRINT1CONTRACT.md` now defines the Sprint 1 scaffold contract.
|
||||
- `docs/Sprints/SPRINT2CONTRACT.md` now defines the Sprint 2 path planning contract.
|
||||
- `docs/Sprints/SPRINT3CONTRACT.md` now defines the Sprint 3 domain records and metadata contract.
|
||||
- `docs/Sprints/SPRINT4CONTRACT.md` now defines the Sprint 4 mocked MinerU adapter contract.
|
||||
- `docs/Sprints/SPRINT5CONTRACT.md` now defines the Sprint 5 Obsidian Markdown normalization and asset link contract.
|
||||
- `docs/Sprints/SPRINT6CONTRACT.md` now defines the Sprint 6 quality checks and report generation contract.
|
||||
- `docs/Sprints/SPRINT7CONTRACT.md` now defines the Sprint 7 conversion orchestration, CLI, and Python API contract.
|
||||
- `docs/Sprints/SPRINT8CONTRACT.md` now defines the Sprint 8 doctor and setup documentation contract.
|
||||
- `docs/Sprints/SPRINT9CONTRACT.md` now defines the Sprint 9 local fixture evaluation and v1 release gate contract.
|
||||
- Relevant `.codex/agents/*.toml` files now reference the v1 plan and sprint contract paths directly.
|
||||
- Sprint 10 is implemented with opt-in pre-conversion PDF chunking, temporary chunk PDF cleanup, chunk metadata/report context, and mocked tests.
|
||||
- Sprint 0 source, environment, license, privacy, and contract verification is complete with a `go-with-risks` recommendation.
|
||||
- Sprint 1 is complete with a minimal Python package, CLI placeholder, and fast pytest loop.
|
||||
- Sprint 4 is implemented with a mock-tested direct local MinerU CLI adapter.
|
||||
- Sprint 5 is implemented with a pure Markdown normalizer and local-only unit tests.
|
||||
- Sprint 6 is implemented with local quality checks and report string rendering.
|
||||
- Sprint 7 is implemented with `convert_pdf`, `convert_input`, output writing, metadata/report writing, local asset copying, batch conversion, and `pdf2md convert`.
|
||||
- Sprint 7 is implemented with fake-adapter CLI/API tests.
|
||||
- Sprint 8 is implemented and committed.
|
||||
- Sprint 9 is implemented, independently evaluated, and committed.
|
||||
- The project `.venv` has been rebuilt with CUDA-enabled PyTorch and MinerU 3.1.0.
|
||||
- Latest `samples/FourNodeQuadrilateralShellElementMITC4.pdf` conversion completed with default GPU request and wrote Markdown, metadata JSON, report Markdown, and assets under ignored `outputs/FourNodeQuadrilateralShellElementMITC4/`.
|
||||
- Latest `samples/MITC공부.pdf` conversion completed on GPU and wrote Markdown, metadata JSON, report Markdown, and assets under ignored `outputs/MITC공부/`.
|
||||
- `docs/MATHJAXCHECKERPLAN.md` now documents the local MathJax render checker plan and implementation status.
|
||||
- Local MathJax render checker code now exists with optional local Node.js/`mathjax` setup, default conversion integration, and `doctor` diagnostics.
|
||||
- `docs/Sprints/SPRINT10CONTRACT.md` now documents the implemented long-PDF pre-conversion chunking sprint.
|
||||
- `outputs/` is ignored and contains local generated conversion outputs.
|
||||
|
||||
## Environment Notes
|
||||
|
||||
- OS/workspace: Windows PowerShell in `D:\Work\Repos\AICoding\ConvertPDFToMD`.
|
||||
- Python target: 3.12.
|
||||
- Local Python observed during Sprint 0: 3.12.7.
|
||||
- `uv` observed during Sprint 0: not available on PATH.
|
||||
- `uv` installed during Sprint 1: 0.11.11 at `C:\Users\user\.local\bin`.
|
||||
- If a new shell cannot find `uv`, restart the shell or add `C:\Users\user\.local\bin` to PATH.
|
||||
- GPU target: GTX 1070 Ti 8GB.
|
||||
- Local GPU observed during Sprint 0: NVIDIA GeForce GTX 1070 Ti, driver 577.00, 8192 MiB VRAM, WDDM.
|
||||
- Sample PDFs are in `samples/` and include Korean filenames.
|
||||
- MinerU execution mode: direct local CLI only.
|
||||
- MinerU 3.1.0 CLI-internal temporary local `mineru-api` is allowed when CLI runs without `--api-url`.
|
||||
- Local Python observed: 3.12.7.
|
||||
- `uv` is installed per-user at `C:\Users\user\.local\bin`.
|
||||
- Target GPU: NVIDIA GTX 1070 Ti 8GB.
|
||||
- Default conversion device: `cuda:0`.
|
||||
- MinerU execution mode: direct local `mineru` CLI only.
|
||||
- Strict-local allows MinerU 3.1.0's CLI-internal temporary local `mineru-api` when the CLI runs without `--api-url`.
|
||||
- Strict-local prohibits `--api-url`, remote APIs, router mode, HTTP client backends, and remote OpenAI-compatible backends.
|
||||
- MinerU planning pin: `mineru[core]==3.1.0` unless Sprint 1 or Sprint 8 proves another 3.1.0 extra is required.
|
||||
- MinerU 3.1.0 was installed in the local `.venv` with `uv pip install "mineru[core]==3.1.0"` for real CLI probing.
|
||||
- Current `pdf2md doctor` status is WARN: MinerU CLI is present, GTX 1070 Ti is visible with Pascal/pre-Turing risk, PyTorch is `2.6.0+cu126` with CUDA available, local MinerU model config is detected, local MathJax checker passes after `npm install`, and strict-local policy passes.
|
||||
- User-level environment variable `MINERU_MODEL_SOURCE=local` is set so MinerU uses the downloaded local model paths in `C:\Users\user\mineru.json`.
|
||||
- Current local runtime has CUDA-enabled PyTorch `2.6.0+cu126`, `torchvision 0.21.0+cu126`, `mineru[core]==3.1.0`, local MinerU models, and `MINERU_MODEL_SOURCE=local`.
|
||||
- Current `pdf2md doctor` status is WARN only because GTX 1070 Ti is Pascal/pre-Turing; MinerU, CUDA PyTorch, local model config, MathJax, and strict-local checks pass.
|
||||
|
||||
## Completed Work
|
||||
## Recent Completed Work
|
||||
|
||||
- Created initial project documents.
|
||||
- Originally selected MinerU 2.5, then changed the fixed engine target to MinerU 3.1.0 after user approval.
|
||||
- Split architecture details into `ARCHITECTURE.md`.
|
||||
- Aligned documents with `Project Guidelines`.
|
||||
- Added this shared planning/progress workflow.
|
||||
- Decided MinerU failures must produce clear warnings/errors without silent fallback.
|
||||
- Decided every conversion should produce metadata JSON and a human-readable `.report.md`.
|
||||
- Created custom agent specs for research, requirements, MinerU integration, Obsidian Markdown, metadata, evaluation, local setup, and license/privacy work.
|
||||
- Created project prompt commands for startup, MinerU research, document review, integration planning, and quality evaluation planning.
|
||||
- Created project skills for MinerU research, math Markdown review, and fixture evaluation.
|
||||
- Created project hooks for startup context, pre-tool policy checks, and stop-time completion reminders.
|
||||
- Read Anthropic's long-running harness design article and adapted its planner/generator/evaluator pattern for this repository.
|
||||
- Added `harness-planner-agent` and `feature-generator-agent`.
|
||||
- Strengthened `evaluation-agent` as an independent contract reviewer and skeptical QA evaluator.
|
||||
- Added long-running harness workflow guidance to `AGENTS.md`.
|
||||
- Created `docs/V1IMPLEMENTATIONPLAN.md` with v1 sprint sequencing, contracts, verification gates, and agent ownership.
|
||||
- Created `docs/Sprints/SPRINT0CONTRACT.md` for source and environment verification before implementation.
|
||||
- Added direct `docs/V1IMPLEMENTATIONPLAN.md` and `docs/Sprints/SPRINT0CONTRACT.md` references to the agents that need them.
|
||||
- Completed Sprint 0 contract evaluation; result was PASS.
|
||||
- Completed the original Sprint 0 MinerU 2.5.4 package, CLI shape, output layout, model/cache, and strict-local risk verification from primary sources.
|
||||
- Verified local Python, `uv`, and GPU facts using the allowed Sprint 0 commands.
|
||||
- Verified MinerU/model license and privacy posture for personal/research local use versus redistribution.
|
||||
- Updated `docs/KNOWLEDGEBASE.md`, `docs/V1IMPLEMENTATIONPLAN.md`, and `docs/Sprints/SPRINT0CONTRACT.md` with Sprint 0 findings.
|
||||
- Completed post-output Sprint 0 evaluation. The only missing acceptance item at review time was the final commit.
|
||||
- Redefined strict-local policy for MinerU 3.1.0: allow direct `mineru` CLI and CLI-internal temporary local `mineru-api`; prohibit `--api-url`, remote APIs, router mode, HTTP client backends, and remote OpenAI-compatible backends.
|
||||
- Updated core project documents and `.codex` workflow assets to reflect MinerU 3.1.0 and the redefined strict-local policy.
|
||||
- Checked MinerU 3.1.0 sources: PyPI 3.1.0 metadata, MinerU release notes, quick usage docs, CLI tools docs, output file docs, and model source docs.
|
||||
- Created `docs/Sprints/SPRINT1CONTRACT.md` for project scaffold and fast test loop planning.
|
||||
- Added direct `docs/Sprints/SPRINT1CONTRACT.md` references to the agents that need Sprint 1 scaffold context.
|
||||
- Started Sprint 1 implementation and amended the contract to include `uv.lock`, which is generated by `uv sync`.
|
||||
- Installed `uv` per-user using the official Astral installer.
|
||||
- Created Sprint 1 scaffold files: `pyproject.toml`, `uv.lock`, `.gitignore`, `README.md`, `src/pdf2md/__init__.py`, `src/pdf2md/cli.py`, `tests/test_package.py`, and `tests/test_cli.py`.
|
||||
- Verified `uv sync` with a temporary project environment outside the repo.
|
||||
- Verified `uv run pytest` passes with 4 tests.
|
||||
- Verified `uv run pdf2md --version` prints `pdf2md 0.1.0`.
|
||||
- Verified `git diff --check` passes.
|
||||
- Checked the scaffold for disallowed MinerU, remote API, router, HTTP, or OpenAI backend references.
|
||||
- Completed independent Sprint 1 evaluation once; the only scope failure was that `PLAN.md` was updated for shared workflow coordination before the Sprint 1 contract listed it as an allowed touched surface.
|
||||
- Amended the Sprint 1 contract to allow minimal `PLAN.md` current-goal coordination updates.
|
||||
- Completed the final independent Sprint 1 evaluation; result was PASS.
|
||||
- Created `docs/Sprints/SPRINT2CONTRACT.md` for paths, input discovery, and overwrite planning.
|
||||
- Added direct `docs/Sprints/SPRINT2CONTRACT.md` references to the agents that need Sprint 2 path planning context.
|
||||
- Updated `docs/V1IMPLEMENTATIONPLAN.md` to point Sprint 2 at the new contract and current scaffold state.
|
||||
- Verified the Sprint 2 contract documentation change with `git diff --check` and `uv run pytest` passing 4 tests.
|
||||
- Started Sprint 2 implementation after user approval and pre-implementation contract review PASS.
|
||||
- Added `src/pdf2md/paths.py` for input discovery, output path planning, overwrite conflict detection, duplicate output detection, and output-root escape prevention.
|
||||
- Added `tests/test_paths.py` with temporary-file coverage for single PDF discovery, directory discovery, recursive discovery, deterministic ordering, Korean filenames, output path planning, overwrite behavior, duplicate planned outputs, and output-root escape prevention.
|
||||
- Completed independent Sprint 2 evaluation once; the only hard failure was a Windows rooted/drive-relative `relative_parent` escape case.
|
||||
- Fixed output-root escape prevention by rejecting absolute, rooted, drive-qualified, and `..` relative parents and validating resolved planned outputs stay under the output root.
|
||||
- Verified `uv run pytest tests/test_paths.py` passes 17 tests.
|
||||
- Verified `uv run pytest` passes 21 tests.
|
||||
- Verified `git diff --check` passes for the Sprint 2 implementation.
|
||||
- Checked the implementation for disallowed MinerU, remote API, router, HTTP, OpenAI backend, or network client references.
|
||||
- Completed the final independent Sprint 2 evaluation; result was PASS.
|
||||
- Created `docs/Sprints/SPRINT3CONTRACT.md` for domain records, metadata construction, and warning aggregation planning.
|
||||
- Added direct `docs/Sprints/SPRINT3CONTRACT.md` references to the agents that need Sprint 3 metadata context.
|
||||
- Updated `docs/V1IMPLEMENTATIONPLAN.md` to point Sprint 3 at the new contract and current path-planning state.
|
||||
- Verified the Sprint 3 contract documentation change with `git diff --check` and `uv run pytest` passing 21 tests.
|
||||
- Started Sprint 3 implementation after user approval and pre-implementation contract review PASS.
|
||||
- Added `src/pdf2md/ir.py` for project-owned document, page, block, asset, and warning records with stable block types, warning codes, and severities.
|
||||
- Added `src/pdf2md/metadata.py` for JSON-serializable metadata construction and summary counts from project-owned records.
|
||||
- Added `tests/test_ir.py` and `tests/test_metadata.py` covering record serialization, optional field preservation/omission, invalid enum/severity validation, metadata top-level fields, summary counts, warning order, JSON serializability, and required input validation.
|
||||
- Verified `uv run pytest tests/test_ir.py tests/test_metadata.py` passes 25 tests.
|
||||
- Verified `uv run pytest` passes 46 tests.
|
||||
- Verified `git diff --check` passes for the Sprint 3 implementation.
|
||||
- Checked the implementation for disallowed remote API, router, HTTP, OpenAI backend, network client, MinerU adapter, and doctor references.
|
||||
- Completed the final independent Sprint 3 evaluation; result was PASS.
|
||||
- Created `docs/Sprints/SPRINT4CONTRACT.md` for direct local MinerU CLI adapter boundary planning with mocked subprocess/output tests.
|
||||
- Added direct `docs/Sprints/SPRINT4CONTRACT.md` references to the agents that need Sprint 4 MinerU adapter context.
|
||||
- Updated `docs/V1IMPLEMENTATIONPLAN.md` to point Sprint 4 at the new contract and current metadata-model state.
|
||||
- Verified the Sprint 4 contract documentation change with `git diff --check` and `uv run pytest` passing 46 tests.
|
||||
- Started Sprint 4 implementation after user approval and pre-implementation contract review PASS.
|
||||
- Added `src/pdf2md/mineru_adapter.py` for the direct local MinerU CLI adapter boundary, mockable availability/version checks, deterministic command construction, subprocess result capture, strict-local option validation, optional mocked-output parsing, and adapter warning mapping.
|
||||
- Added `tests/test_mineru_adapter.py` with fake-runner coverage for availability, missing MinerU, version success/failure/empty output, fixed command shape, custom executable rejection, strict-local rejection, mocked success, non-zero exit, missing output, and invalid JSON.
|
||||
- Fixed an independent evaluation finding that a caller-controlled executable could bypass strict-local policy; v1 now accepts only the direct `mineru` executable name, and user-exposed `mineru-api` execution is rejected.
|
||||
- Verified `uv sync` passes.
|
||||
- Verified `uv run pytest tests/test_mineru_adapter.py` passes 26 tests.
|
||||
- Verified `uv run pytest` passes 72 tests.
|
||||
- Verified `git diff --check` passes for the Sprint 4 implementation.
|
||||
- Checked the implementation for network client imports; none were found.
|
||||
- Checked strict-local prohibited tokens in `src/pdf2md`; matches are limited to deliberate validation literals in `mineru_adapter.py`.
|
||||
- Completed the final independent Sprint 4 evaluation; result was PASS.
|
||||
- Created `docs/Sprints/SPRINT5CONTRACT.md` for Obsidian Markdown normalization, math delimiter handling, asset link normalization, and conservative table fallback planning.
|
||||
- Added direct `docs/Sprints/SPRINT5CONTRACT.md` references to the agents that need Sprint 5 Markdown, warning, implementation, planning, or evaluation context.
|
||||
- Updated `docs/V1IMPLEMENTATIONPLAN.md` to point Sprint 5 at the new contract and current Sprint 4 implementation state.
|
||||
- Verified the Sprint 5 contract documentation change with agent TOML parsing, `git diff --check`, and `uv run pytest` passing 72 tests.
|
||||
- Started Sprint 5 implementation after user approval and pre-implementation contract review PASS.
|
||||
- Added `src/pdf2md/markdown.py` for project-owned Obsidian Markdown normalization, inline/display math delimiter handling, code fence and inline code protection, relative asset link normalization, local asset warning behavior, and conservative table fallback warnings.
|
||||
- Added `tests/test_markdown.py` covering inline math, display math spacing, idempotency, math body preservation, code protection, asset path normalization, invalid/missing/remote asset warnings, simple table preservation, and complex HTML table fallback warnings.
|
||||
- Added narrow warning codes `ASSET_LINK_INVALID` and `TABLE_FALLBACK` to `src/pdf2md/ir.py`.
|
||||
- Verified `uv sync` passes.
|
||||
- Verified `uv run pytest tests/test_markdown.py tests/test_ir.py` passes 30 tests.
|
||||
- Verified `uv run pytest` passes 89 tests.
|
||||
- Verified `git diff --check` passes for the Sprint 5 implementation.
|
||||
- Checked the implementation for network client imports; none were found.
|
||||
- Checked the implementation for conversion orchestration, metadata writing, report generation, and CLI convert behavior; no Sprint 5 code introduced those paths.
|
||||
- Completed the final independent Sprint 5 evaluation; result was PASS.
|
||||
- Created `docs/Sprints/SPRINT6CONTRACT.md` for local quality checks, math renderability boundary, metadata summary extensions, report content rendering, and final status planning.
|
||||
- Added direct `docs/Sprints/SPRINT6CONTRACT.md` references to the agents that need Sprint 6 quality, reporting, metadata, math renderability, implementation, planning, or evaluation context.
|
||||
- Updated `docs/V1IMPLEMENTATIONPLAN.md` to point Sprint 6 at the new contract and current Sprint 5 implementation state.
|
||||
- Verified the Sprint 6 contract documentation change with agent TOML parsing, `git diff --check`, and `uv run pytest` passing 89 tests.
|
||||
- Started Sprint 6 implementation after user approval and pre-implementation contract review PASS.
|
||||
- Added `src/pdf2md/quality.py` for local asset-link checks, math renderability checker boundaries, nonfatal checker-unavailable behavior, and quality result aggregation.
|
||||
- Added `src/pdf2md/report.py` for human-readable quality report content rendering from metadata and quality results, pages-with-warnings derivation, and final status calculation.
|
||||
- Added `tests/test_quality.py` covering missing/invalid asset links, code-block exclusions, fake math checker failures, checker-unavailable behavior, and quality result merging.
|
||||
- Added `tests/test_report.py` covering required report content, optional path handling, pages-with-warnings, final status policy, metadata/quality count use, and no report-file creation.
|
||||
- Verified `uv sync` passes.
|
||||
- Verified `uv run pytest tests/test_quality.py tests/test_report.py tests/test_metadata.py` passes 26 tests.
|
||||
- Verified `uv run pytest` passes 103 tests.
|
||||
- Verified `git diff --check` passes for the Sprint 6 implementation.
|
||||
- Checked the implementation for network client imports; none were found.
|
||||
- Checked the implementation for conversion orchestration, final output writing, metadata JSON writing, `.report.md` file writing, real MinerU invocation, setup scripts, and CLI convert behavior; no Sprint 6 code introduced those paths.
|
||||
- Completed the final independent Sprint 6 evaluation; result was PASS.
|
||||
- Completed the final independent Sprint 7 evaluation after fixing math renderability metadata counts; result was PASS.
|
||||
- Started Sprint 8 implementation after user approval.
|
||||
- Added `src/pdf2md/doctor.py` for mockable setup diagnostics covering Python 3.12, `uv`, MinerU availability/version, NVIDIA GPU visibility, PyTorch CUDA visibility, local model/cache/config detection, and strict-local policy reporting.
|
||||
- Added `pdf2md doctor` CLI integration without changing `pdf2md convert` or `pdf2md --version` behavior.
|
||||
- Updated `README.md` with Windows PowerShell setup, `uv`, MinerU 3.1.0 direct CLI expectations, model/cache environment notes, GTX 1070 Ti risk, and strict-local runtime policy.
|
||||
- Added mocked doctor and CLI tests for success, warning-only success, hard dependency failure, missing `uv`, missing MinerU, MinerU version warnings, missing GPU/PyTorch warnings, GTX 1070 Ti/Pascal risk, and missing model/cache warnings.
|
||||
- Verified `uv run pytest tests/test_doctor.py tests/test_cli.py` passes 22 tests.
|
||||
- Verified `uv sync` passes.
|
||||
- Verified `uv run pytest` passes 133 tests.
|
||||
- Verified `uv run pdf2md --version` prints `pdf2md 0.1.0`.
|
||||
- Verified local `uv run pdf2md doctor` returns exit code 1 because MinerU is not installed; it reports Python and `uv` pass, GTX 1070 Ti/Pascal risk warning, PyTorch missing warning, model/cache missing warning, and strict-local pass.
|
||||
- Completed independent Sprint 8 evaluation; result was PASS.
|
||||
- Committed Sprint 8 implementation as `7d965e3 feat: implement sprint 8 doctor diagnostics`.
|
||||
- Created `docs/Sprints/SPRINT9CONTRACT.md` for local fixture evaluation and the v1 release gate.
|
||||
- Added direct `docs/Sprints/SPRINT9CONTRACT.md` references to the agents that need Sprint 9 fixture evaluation, release gate, strict-local, or implementation context.
|
||||
- Updated `docs/V1IMPLEMENTATIONPLAN.md` to point Sprint 9 at the new contract and current Sprint 8 completion state.
|
||||
- Started Sprint 9 implementation after user approval and pre-implementation contract review PASS.
|
||||
- Added fast mocked v1 release-gate integration tests in `tests/integration/test_v1_fast_release_gate.py`.
|
||||
- Added explicit opt-in local MinerU fixture evaluation in `tests/integration/test_optional_mineru_fixtures.py`, gated by `PDF2MD_RUN_MINERU_FIXTURES=1`.
|
||||
- Added `docs/V1RELEASECHECKLIST.md` with default fast gates, strict-local release gates, doctor hard-failure handling, optional sample gates, fixture coverage notes, and no-sample-commit checks.
|
||||
- Updated `README.md` to point at the v1 release checklist and optional fixture evaluation gate.
|
||||
- Verified `uv run pytest tests/integration tests/test_conversion.py tests/test_cli.py` passes 24 tests with 1 optional skip.
|
||||
- Verified `uv run pytest tests/integration` passes 3 fast tests with 1 optional skip.
|
||||
- Verified opt-in `PDF2MD_RUN_MINERU_FIXTURES=1 uv run pytest -rs tests/integration/test_optional_mineru_fixtures.py` is skipped with a clear doctor blocker because MinerU is not installed.
|
||||
- Verified `uv run pytest` passes 136 tests with 1 optional skip.
|
||||
- Completed independent Sprint 9 evaluation; result was PASS.
|
||||
- Committed Sprint 9 implementation as `466abcf feat: implement sprint 9 release gate`.
|
||||
- Attempted `samples/MITC공부.pdf` conversion after installing MinerU; the run did not produce a successful conversion and was stopped after the user observed CPU-bound execution.
|
||||
- Added `outputs/` to `.gitignore` and removed the leftover generated output directory from the stopped sample run.
|
||||
- Updated default conversion behavior so `convert_pdf`, `convert_input`, and `pdf2md convert` default to `cuda:0`.
|
||||
- Updated the MinerU adapter to map CUDA requests to the MinerU subprocess environment with `MINERU_DEVICE_MODE` and `CUDA_VISIBLE_DEVICES`, preserving strict-local direct CLI execution.
|
||||
- Updated README, PRD, and architecture docs to document GPU default behavior and the remaining CUDA/PyTorch requirement.
|
||||
- Verified the GPU-default change with targeted tests, full tests, `git diff --check`, CLI help, and `pdf2md doctor`.
|
||||
- Re-ran `uv run pdf2md convert samples\MITC공부.pdf --out outputs\MITC공부 --overwrite`; the CLI reported `converted: 0`, `failed: 1`, `warnings: 1`, and wrote no Markdown, metadata JSON, or `.report.md`.
|
||||
- Confirmed the failure with a direct adapter probe using an ASCII work directory: MinerU 3.1.0 started its allowed temporary local `mineru-api`, used `hybrid-auto-engine`, attempted to load the VLM model on CUDA, and failed with `AssertionError: Torch not compiled with CUDA enabled`.
|
||||
- Left the product conversion stdout/stderr logs under ignored `outputs/MITC공부.logs`; removed temporary diagnostic probe directories.
|
||||
- Removed and recreated the project `.venv` with `uv sync`.
|
||||
- Installed CUDA-enabled PyTorch runtime: `torch==2.6.0+cu126` and `torchvision==0.21.0+cu126`.
|
||||
- Verified CUDA with an actual tensor operation on `NVIDIA GeForce GTX 1070 Ti`, compute capability `6.1`.
|
||||
- Installed `mineru[core]==3.1.0`; verified `mineru, version 3.1.0`.
|
||||
- Downloaded MinerU pipeline and VLM models with `uv run mineru-models-download -s huggingface -m all`; MinerU wrote model paths to `C:\Users\user\mineru.json`.
|
||||
- Set `MINERU_MODEL_SOURCE=local` at user scope and current process scope.
|
||||
- Verified `uv run pdf2md doctor` reports PyTorch CUDA available and model config detected; remaining WARN status is the intentional Pascal/pre-Turing GPU risk warning.
|
||||
- Verified `uv run pytest` passes 138 tests with 1 optional skip in the rebuilt environment.
|
||||
- Fixed real MinerU nested `images/...` asset-link rewriting so copied assets under `<stem>.assets/.../images/` resolve from the final Markdown.
|
||||
- Fixed page-count extraction for MinerU-style structured lists with `page_idx` values.
|
||||
- Verified `uv run pytest tests/test_conversion.py` passes 12 tests.
|
||||
- Verified `uv run pytest` passes 139 tests with 1 optional skip after the asset/page-count fix.
|
||||
- Re-ran `samples/MITC공부.pdf` conversion with `MINERU_MODEL_SOURCE=local`; MinerU used GPU via the direct local CLI and CLI-internal temporary local `mineru-api`.
|
||||
- The final sample outputs are `outputs/MITC공부/MITC공부.md`, `outputs/MITC공부/MITC공부.metadata.json`, `outputs/MITC공부/MITC공부.report.md`, and `outputs/MITC공부/MITC공부.assets/`.
|
||||
- The final sample report status is `partial` only because the local math render checker is unavailable; asset link checks pass with 0 missing and 0 invalid links.
|
||||
- Sample summary: 13 pages processed, 107 assets, 23 inline formulas, 103 display formulas, 1 info warning.
|
||||
- Added `docs/MATHJAXCHECKERPLAN.md` with the local MathJax checker objective, touched surfaces, Node helper contract, Python wrapper behavior, tests, acceptance criteria, and open implementation decisions.
|
||||
- Implemented the local MathJax render checker with `MathExpression` extraction, a local Node.js helper, a Python wrapper, default conversion integration, `doctor` diagnostics, setup documentation, and mocked tests.
|
||||
- Verified `npm run mathjax-checker:health` returns `{"ok":true}` after local `npm install`.
|
||||
- Verified direct helper JSON stdin reports one valid expression as ok and a malformed display formula as `Missing close brace`.
|
||||
- Verified `create_default_math_checker()` finds the local checker and records one render failure for malformed display math.
|
||||
- Verified targeted tests pass: `uv run pytest tests/test_quality.py tests/test_math_render.py tests/test_conversion.py tests/test_doctor.py tests/test_cli.py`.
|
||||
- Verified full tests pass: `uv run pytest` passed 150 tests with 1 optional skip.
|
||||
- Verified `git diff --check` passes.
|
||||
- Researched local PDF chunking packages and MinerU page-range behavior for Sprint 10.
|
||||
- Created `docs/Sprints/SPRINT10CONTRACT.md` recommending `pypdf>=6.10.2,<7` for 20-page local chunk PDFs, with chunk outputs converted independently and no Markdown merge.
|
||||
- Implemented Sprint 10 with `pypdf>=6.10.2,<7`, `src/pdf2md/pdf_splitter.py`, `--chunk-pages [PAGES]`, chunk-aware conversion orchestration, and chunk report context.
|
||||
- Archived completed sprint and setup history into `docs/WORKARCHIVE.md`.
|
||||
- Added `docs/WORKARCHIVE.md` references to `AGENTS.md`, `PLAN.md`, `docs/V1IMPLEMENTATIONPLAN.md`, relevant `.codex/agents/*.toml`, `.codex/commands/*.md`, and project skills.
|
||||
- Sprint 10 is implemented with `pypdf>=6.10.2,<7`, `src/pdf2md/pdf_splitter.py`, `--chunk-pages [PAGES]`, chunk-aware conversion orchestration, temporary chunk cleanup, and chunk report context.
|
||||
- `--chunk-pages` is opt-in; when present without a value it uses 20 pages.
|
||||
- `convert_pdf()` returns `BatchConversionResult` when `chunk_pages` is set and keeps returning `ConversionResult` when chunking is unset.
|
||||
- Temporary chunk PDFs are deleted after conversion completes, including when raw MinerU output is retained.
|
||||
- Verified targeted Sprint 10 tests: `uv run pytest tests/test_pdf_splitter.py tests/test_conversion.py tests/test_cli.py tests/test_report.py` passed 42 tests.
|
||||
- Verified full default test suite: `uv run pytest` passed 163 tests with 1 optional skip.
|
||||
- Verified `git diff --check` passed with line-ending warnings only.
|
||||
- Ran `uv run pdf2md doctor`; status was WARN only for GTX 1070 Ti/Pascal compatibility risk while MinerU 3.1.0, CUDA PyTorch, local models, MathJax, and strict-local checks passed.
|
||||
- Converted `samples/FourNodeQuadrilateralShellElementMITC4.pdf` with `MINERU_MODEL_SOURCE=local` and default `--gpu cuda:0`; output was written to `outputs/FourNodeQuadrilateralShellElementMITC4/`.
|
||||
- Converted `samples/FourNodeQuadrilateralShellElementMITC4.pdf` with `MINERU_MODEL_SOURCE=local` and default `--gpu cuda:0`; output was written to ignored `outputs/FourNodeQuadrilateralShellElementMITC4/`.
|
||||
- The FourNode sample conversion report status was `success`: 7 pages, 22 assets, 38 inline formulas, 16 display formulas, 0 math render errors, and 0 warnings.
|
||||
|
||||
## In Progress
|
||||
@@ -253,189 +43,11 @@ This file records actual progress for agents. Read it before starting work, then
|
||||
|
||||
## Blockers
|
||||
|
||||
- No active blocker for the completed `samples/MITC공부.pdf` conversion.
|
||||
- GTX 1070 Ti remains an 8GB Pascal GPU; larger PDFs may still hit VRAM or model compatibility limits even though this sample completed.
|
||||
- No active blocker.
|
||||
- GTX 1070 Ti remains an 8GB Pascal GPU; larger PDFs may still hit VRAM or model compatibility limits.
|
||||
|
||||
## Next Actions
|
||||
|
||||
1. Review the generated `outputs/MITC공부/MITC공부.md` in Obsidian if visual quality needs manual assessment.
|
||||
1. Review generated sample Markdown outputs in Obsidian if visual quality needs manual assessment.
|
||||
2. Run optional real local chunked conversion on a long sample only if requested.
|
||||
3. Run `npm install` and `npm run mathjax-checker:health` when real local MathJax checker validation is desired.
|
||||
4. Preserve the strict-local rule: setup downloads may be explicit, but runtime conversion must use local model paths, direct CLI execution, and no user-specified API or remote backend.
|
||||
|
||||
## Sprint 9 Handoff
|
||||
|
||||
- Files changed: `tests/integration/test_v1_fast_release_gate.py`, `tests/integration/test_optional_mineru_fixtures.py`, `docs/V1RELEASECHECKLIST.md`, `README.md`, `PLAN.md`, `PROGRESS.md`, `docs/V1IMPLEMENTATIONPLAN.md`, and `docs/Sprints/SPRINT9CONTRACT.md`.
|
||||
- Commands run: `uv run pytest tests/integration tests/test_conversion.py tests/test_cli.py`, `uv run pytest tests/integration`, `PDF2MD_RUN_MINERU_FIXTURES=1 uv run pytest -rs tests/integration/test_optional_mineru_fixtures.py`, `uv run pytest`, `git diff --check`, and `git status --short --untracked-files=all`.
|
||||
- Tests passed: targeted integration/CLI/conversion run passed 24 tests with 1 optional skip; integration-only run passed 3 fast tests with 1 optional skip; full `uv run pytest` passed 136 tests with 1 optional skip.
|
||||
- Tests blocked: optional real MinerU fixture conversion is blocked by `pdf2md doctor` because the `mineru` CLI is not installed on PATH.
|
||||
- Optional local MinerU status: explicitly gated by `PDF2MD_RUN_MINERU_FIXTURES=1`; current opt-in run skips with doctor blocker instead of pretending real validation passed.
|
||||
- Fixture coverage: release checklist maps local samples to math-heavy, table/formula, figures/assets, reading-order, and Korean filename/path risk categories; simple one-page, table-dominant, and figure-heavy known-baseline gaps remain.
|
||||
- Generated output locations: none persisted; optional output path uses pytest `tmp_path`.
|
||||
- Known failures: local doctor fails on missing MinerU CLI.
|
||||
- Independent evaluation: PASS.
|
||||
- Residual risks: no real MinerU output has been validated yet; GTX 1070 Ti/PyTorch acceleration and model/cache setup remain unproven; optional fixture quality still requires local MinerU setup.
|
||||
- User decisions needed: decide whether to install/configure MinerU 3.1.0 and run optional fixture validation.
|
||||
- V1 release recommendation: default fast gates are healthy, but full real-MinerU v1 validation is blocked until doctor passes or the user records a waiver.
|
||||
- Go/no-go recommendation for next sprint: go only for real setup/fixture validation if the user wants to proceed with local MinerU installation.
|
||||
- Next action: commit Sprint 9 implementation.
|
||||
|
||||
## Sprint 9 Contract Handoff
|
||||
|
||||
- Files changed: `docs/Sprints/SPRINT9CONTRACT.md`, `docs/V1IMPLEMENTATIONPLAN.md`, relevant `.codex/agents/*.toml`, `PLAN.md`, and `PROGRESS.md`.
|
||||
- Commands run: `uv --version`, `uv sync`, agent TOML parse check, `uv run pytest`, `git diff --check`, `git status --short --untracked-files=all`, and local sample filename listing.
|
||||
- Tests passed: `uv run pytest` passed 133 tests.
|
||||
- Tests blocked: None expected for the contract-only change.
|
||||
- Known failures: local `pdf2md doctor` still fails until MinerU is installed on PATH.
|
||||
- Residual risks: Sprint 9 is contract-only; fast mocked integration tests, optional local MinerU fixture harness, fixture coverage manifest, and release checklist are not implemented yet.
|
||||
- User decisions needed: None before Sprint 9 pre-implementation review.
|
||||
- Go/no-go recommendation for Sprint 9 implementation: review the contract first, then go if the user explicitly requests implementation.
|
||||
- Next action: verify and commit the Sprint 9 contract update.
|
||||
|
||||
## Sprint 8 Handoff
|
||||
|
||||
- Files changed: `src/pdf2md/doctor.py`, `src/pdf2md/cli.py`, `tests/test_doctor.py`, `tests/test_cli.py`, `README.md`, `PLAN.md`, `PROGRESS.md`, `docs/V1IMPLEMENTATIONPLAN.md`, and `docs/Sprints/SPRINT8CONTRACT.md`.
|
||||
- Commands run: `uv --version`, `uv sync`, `uv run pytest tests/test_doctor.py tests/test_cli.py`, `uv run pytest tests/test_doctor.py`, `uv run pytest`, `uv run pdf2md --version`, `uv run pdf2md doctor`, `git diff --check`, `git status --short --untracked-files=all`, and PowerShell strict-local/network pattern checks.
|
||||
- Tests passed: `uv run pytest tests/test_doctor.py tests/test_cli.py` passed 22 tests; `uv run pytest tests/test_doctor.py` passed 11 tests; `uv run pytest` passed 133 tests.
|
||||
- Tests blocked: None.
|
||||
- Known failures: local `uv run pdf2md doctor` correctly fails because the `mineru` CLI is not installed on PATH.
|
||||
- Known warnings: `uv` ignored Miniforge's invalid `SSL_CERT_DIR` path during sync/test commands, but the commands completed successfully; local doctor warns for GTX 1070 Ti/Pascal risk, missing PyTorch, and missing MinerU model/cache/config path.
|
||||
- Independent evaluation: PASS.
|
||||
- Residual risks: Sprint 8 does not install MinerU, download models, validate real MinerU output, run sample PDFs, or prove GTX 1070 Ti PyTorch acceleration. Those remain Sprint 9/local setup work.
|
||||
- User decisions needed: None for Sprint 8.
|
||||
- Go/no-go recommendation for Sprint 9: go after a Sprint 9 contract is written and reviewed.
|
||||
- Next action at completion: prepare the Sprint 9 contract when requested.
|
||||
|
||||
## Sprint 8 Contract Handoff
|
||||
|
||||
Historical note: this contract-only handoff was superseded by the implemented Sprint 8 handoff above.
|
||||
|
||||
- Files changed: `docs/Sprints/SPRINT8CONTRACT.md`, `docs/V1IMPLEMENTATIONPLAN.md`, relevant `.codex/agents/*.toml`, `PLAN.md`, and `PROGRESS.md`.
|
||||
- Commands run: `uv --version`, agent TOML parse check, `uv sync`, `uv run pytest`, `git diff --check`, `git status --short --untracked-files=all`, and PowerShell reference checks.
|
||||
- Tests passed: `uv run pytest` passed 119 tests.
|
||||
- Tests blocked: None.
|
||||
- Known failures: none.
|
||||
- Known warnings: `uv` ignored Miniforge's invalid `SSL_CERT_DIR` path during sync/test commands, but the commands completed successfully.
|
||||
- Residual risks: Sprint 8 is contract-only; `pdf2md doctor`, doctor diagnostics, setup docs, and setup helper scripts are not implemented yet.
|
||||
- User decisions needed: None before Sprint 8 pre-implementation review.
|
||||
- Go/no-go recommendation for Sprint 8 implementation: review the contract first, then go if the user explicitly requests implementation.
|
||||
- Next action: commit the contract update, then wait for an explicit Sprint 8 implementation request.
|
||||
|
||||
## Sprint 7 Handoff
|
||||
|
||||
- Files changed: `src/pdf2md/conversion.py`, `src/pdf2md/cli.py`, `src/pdf2md/__init__.py`, `tests/test_conversion.py`, `tests/test_cli.py`, `tests/test_package.py`, `PLAN.md`, `PROGRESS.md`, `docs/V1IMPLEMENTATIONPLAN.md`, and `docs/Sprints/SPRINT7CONTRACT.md`.
|
||||
- Commands run: `uv --version`, `uv sync`, `uv run pytest tests/test_conversion.py tests/test_cli.py`, `uv run pytest tests/test_conversion.py tests/test_cli.py tests/test_package.py`, `uv run pytest tests/test_conversion.py tests/test_metadata.py tests/test_report.py`, `uv run pytest`, `git diff --check`, `git status --short --untracked-files=all`, and PowerShell strict-local/network pattern checks.
|
||||
- Tests passed: `uv run pytest tests/test_conversion.py tests/test_cli.py` passed 18 tests; `uv run pytest tests/test_conversion.py tests/test_cli.py tests/test_package.py` passed 16 tests before the math renderability fix; `uv run pytest tests/test_conversion.py tests/test_metadata.py tests/test_report.py` passed 29 tests; `uv run pytest` passed 119 tests after the metadata math count fix.
|
||||
- Tests blocked: None.
|
||||
- Known failures: none.
|
||||
- Known warnings: `uv` ignored Miniforge's invalid `SSL_CERT_DIR` path during sync/test commands, but the commands completed successfully.
|
||||
- Independent evaluation: PASS after fixing math renderability metadata counts.
|
||||
- Residual risks: Sprint 7 uses fake adapters in default tests; it does not run real MinerU, probe real MinerU output, implement `pdf2md doctor`, validate CUDA/GPU, install models, or run sample PDFs.
|
||||
- User decisions needed: None for Sprint 7.
|
||||
- Go/no-go recommendation for Sprint 8: go.
|
||||
- Next action: prepare Sprint 8 contract when requested.
|
||||
|
||||
## Sprint 7 Contract Handoff (Historical)
|
||||
|
||||
- Files changed: `docs/Sprints/SPRINT7CONTRACT.md`, `docs/V1IMPLEMENTATIONPLAN.md`, `.codex/agents/feature-generator-agent.toml`, `.codex/agents/evaluation-agent.toml`, `.codex/agents/requirements-guard-agent.toml`, `.codex/agents/harness-planner-agent.toml`, `.codex/agents/mineru-integration-agent.toml`, `.codex/agents/metadata-agent.toml`, `.codex/agents/obsidian-markdown-agent.toml`, `PLAN.md`, and `PROGRESS.md`.
|
||||
- Commands run: `uv --version`, agent TOML parse check, `uv sync`, `uv run pytest`, `git diff --check`, and `git status --short --untracked-files=all`.
|
||||
- Tests passed: `uv run pytest` passed 103 tests.
|
||||
- Tests blocked: None.
|
||||
- Known failures: none.
|
||||
- Known warnings: `uv` ignored Miniforge's invalid `SSL_CERT_DIR` path during sync/test commands, but the commands completed successfully.
|
||||
- Residual risks at that time: Sprint 7 was contract-only before implementation. Superseded by the Sprint 7 Handoff above.
|
||||
- User decisions needed at that time: None before Sprint 7 pre-implementation review.
|
||||
- Go/no-go recommendation at that time: review the contract first, then go if the user explicitly requests implementation.
|
||||
- Next action at that time: commit the contract update, then wait for an explicit Sprint 7 implementation request.
|
||||
|
||||
## Sprint 6 Handoff
|
||||
|
||||
- Files changed: `src/pdf2md/quality.py`, `src/pdf2md/report.py`, `tests/test_quality.py`, `tests/test_report.py`, `PLAN.md`, `PROGRESS.md`, `docs/V1IMPLEMENTATIONPLAN.md`, and `docs/Sprints/SPRINT6CONTRACT.md`.
|
||||
- Commands run: `uv --version`, `uv sync`, `uv run pytest tests/test_quality.py tests/test_report.py tests/test_metadata.py`, `uv run pytest`, `git diff --check`, `git status --short --untracked-files=all`, and PowerShell file/pattern checks.
|
||||
- Tests passed: `uv run pytest tests/test_quality.py tests/test_report.py tests/test_metadata.py` passed 26 tests; `uv run pytest` passed 103 tests.
|
||||
- Tests blocked: None.
|
||||
- Known failures: none in Sprint 6 implementation.
|
||||
- Known warnings: `uv` ignored Miniforge's invalid `SSL_CERT_DIR` path during sync/test commands, but the commands completed successfully.
|
||||
- Residual risks: Sprint 6 intentionally does not run real MinerU, run a real math renderer, parse PDFs, write final Markdown files, copy assets, write metadata JSON files, write `.report.md` files, expose working `convert`, or implement `doctor`.
|
||||
- User decisions needed: None for Sprint 6.
|
||||
- Go/no-go recommendation for Sprint 7: go.
|
||||
- Next action: prepare Sprint 7 contract when requested.
|
||||
|
||||
## Sprint 5 Handoff
|
||||
|
||||
- Files changed: `src/pdf2md/markdown.py`, `src/pdf2md/ir.py`, `tests/test_markdown.py`, `PLAN.md`, `PROGRESS.md`, `docs/V1IMPLEMENTATIONPLAN.md`, and `docs/Sprints/SPRINT5CONTRACT.md`.
|
||||
- Commands run: `uv --version`, `uv sync`, `uv run pytest tests/test_markdown.py tests/test_ir.py`, `uv run pytest`, `git diff --check`, `git status --short --untracked-files=all`, and PowerShell file/pattern checks.
|
||||
- Tests passed: `uv run pytest tests/test_markdown.py tests/test_ir.py` passed 30 tests; `uv run pytest` passed 89 tests.
|
||||
- Tests blocked: None.
|
||||
- Known failures: none in Sprint 5 implementation.
|
||||
- Known warnings: `uv` ignored Miniforge's invalid `SSL_CERT_DIR` path during sync/test commands, but the commands completed successfully.
|
||||
- Residual risks: Sprint 5 intentionally does not run real MinerU, probe real MinerU Markdown, parse PDFs, write final Markdown files, copy assets, write metadata JSON, generate `.report.md`, expose working `convert`, or implement `doctor`.
|
||||
- User decisions needed: None for Sprint 5.
|
||||
- Go/no-go recommendation for Sprint 6: go.
|
||||
- Next action: prepare Sprint 6 contract when requested.
|
||||
|
||||
## Sprint 4 Handoff
|
||||
|
||||
- Files changed: `src/pdf2md/mineru_adapter.py`, `tests/test_mineru_adapter.py`, `PLAN.md`, and `PROGRESS.md`.
|
||||
- Commands run: `uv --version`, `uv sync`, `uv run pytest tests/test_mineru_adapter.py`, `uv run pytest`, `git diff --check`, `git status --short --untracked-files=all`, and PowerShell file/pattern checks.
|
||||
- Tests passed: `uv run pytest tests/test_mineru_adapter.py` passed 26 tests; `uv run pytest` passed 72 tests.
|
||||
- Tests blocked: None.
|
||||
- Known failures: none in Sprint 4 implementation after fixing the independent evaluation finding.
|
||||
- Known warnings: `uv` ignored Miniforge's invalid `SSL_CERT_DIR` path during sync/test commands, but the commands completed successfully.
|
||||
- Residual risks: Sprint 4 intentionally does not run real MinerU, install models, probe real MinerU output layout, parse PDFs, normalize Markdown, write metadata JSON, generate `.report.md`, expose working `convert`, or implement `doctor`.
|
||||
- User decisions needed: None for Sprint 4.
|
||||
- Go/no-go recommendation for Sprint 5: go.
|
||||
- Next action: prepare Sprint 5 contract when requested.
|
||||
|
||||
## Sprint 3 Handoff
|
||||
|
||||
- Files changed: `src/pdf2md/ir.py`, `src/pdf2md/metadata.py`, `tests/test_ir.py`, `tests/test_metadata.py`, `PLAN.md`, `PROGRESS.md`, and `docs/Sprints/SPRINT3CONTRACT.md`.
|
||||
- Commands run: `uv --version`, `uv sync`, `uv run pytest tests/test_ir.py tests/test_metadata.py`, `uv run pytest`, `git diff --check`, `git status --short`, and PowerShell file/pattern checks.
|
||||
- Tests passed: `uv run pytest tests/test_ir.py tests/test_metadata.py` passed 25 tests; `uv run pytest` passed 46 tests.
|
||||
- Tests blocked: None.
|
||||
- Known failures: none in Sprint 3 implementation.
|
||||
- Known warnings: `uv` ignored Miniforge's invalid `SSL_CERT_DIR` path during sync/test commands, but the commands completed successfully.
|
||||
- Residual risks: Sprint 3 intentionally does not parse PDFs, compute SHA-256, invoke MinerU, write conversion outputs, normalize Markdown, create full report content, run quality checks, or expose working `convert` or `doctor` commands.
|
||||
- User decisions needed: None for Sprint 3.
|
||||
- Go/no-go recommendation for Sprint 4: go.
|
||||
- Next action: prepare Sprint 4 contract when requested.
|
||||
|
||||
## Sprint 2 Handoff
|
||||
|
||||
- Files changed: `src/pdf2md/paths.py`, `tests/test_paths.py`, `PLAN.md`, and `PROGRESS.md`.
|
||||
- Commands run: `uv --version`, `uv sync`, `uv run pytest tests/test_paths.py`, `uv run pytest`, `git diff --check`, `git status --short`, and PowerShell file/pattern checks.
|
||||
- Tests passed: `uv run pytest tests/test_paths.py` passed 17 tests; `uv run pytest` passed 21 tests.
|
||||
- Tests blocked: None.
|
||||
- Known failures: none in Sprint 2 implementation.
|
||||
- Known warnings: `uv` ignored Miniforge's invalid `SSL_CERT_DIR` path during sync/test commands, but the commands completed successfully.
|
||||
- Residual risks: Sprint 2 intentionally does not parse PDFs, compute SHA-256, invoke MinerU, write conversion outputs, normalize Markdown, create metadata/report content, or expose a working `convert` command.
|
||||
- User decisions needed: None for Sprint 2.
|
||||
- Go/no-go recommendation for Sprint 3: go.
|
||||
- Next action: prepare Sprint 3 contract when requested.
|
||||
|
||||
## Sprint 1 Handoff
|
||||
|
||||
- Files changed: `pyproject.toml`, `uv.lock`, `.gitignore`, `README.md`, `src/pdf2md/__init__.py`, `src/pdf2md/cli.py`, `tests/test_package.py`, `tests/test_cli.py`, `PROGRESS.md`, `PLAN.md`, and `docs/Sprints/SPRINT1CONTRACT.md`.
|
||||
- Commands run: `uv --version`, `uv sync`, `uv run pytest`, `uv run pdf2md --version`, `git diff --check`, `git status --short`, and PowerShell file/pattern checks after `rg.exe` returned access denied.
|
||||
- Tests passed: `uv run pytest` passed 4 tests.
|
||||
- Tests blocked: None.
|
||||
- Known failures: `uv` may not be visible to a newly opened shell until PATH is refreshed; `rg.exe` returned access denied in this environment, so PowerShell checks were used instead.
|
||||
- Known warnings: `uv` ignored Miniforge's invalid `SSL_CERT_DIR` path during sync/test commands, but the commands completed successfully.
|
||||
- Residual risks: the scaffold intentionally does not validate MinerU, CUDA, model paths, conversion output, metadata, or quality reports.
|
||||
- User decisions needed: None for Sprint 1.
|
||||
- Go/no-go recommendation for Sprint 2: go.
|
||||
- Next action: prepare Sprint 2 contract when requested.
|
||||
|
||||
## Sprint 0 Handoff
|
||||
|
||||
Superseded note: the following Sprint 0 facts describe the completed 2.5.4 verification pass. The current engine decision is MinerU 3.1.0.
|
||||
|
||||
- Files changed: `docs/KNOWLEDGEBASE.md`, `docs/Sprints/SPRINT0CONTRACT.md`, `docs/V1IMPLEMENTATIONPLAN.md`, `PROGRESS.md`.
|
||||
- Sources checked: MinerU 2.5.4 PyPI, MinerU 2.5.4 tag files, MinerU output/model docs, Python/uv/PyTorch/NVIDIA docs, MinerU/model license sources.
|
||||
- Local commands run: `python --version`, `uv --version`, `nvidia-smi`.
|
||||
- Facts confirmed at that time: Python 3.12.7 is present; `uv` is missing; GTX 1070 Ti 8GB is visible; MinerU 2.5.4 direct CLI path is source-verified; MinerU/model 2.5-era licenses should be treated as AGPL-3.0.
|
||||
- Inferences made at that time: v1 should pin MinerU to `mineru[core]==2.5.4`; strict-local runtime should require local model source configuration; current MinerU 3.x docs should not drive the older 2.5 adapter behavior.
|
||||
- Known failures: `uv --version` failed because `uv` is not on PATH.
|
||||
- Residual risks: GTX 1070 Ti/PyTorch CUDA compatibility, real MinerU output layout until local probe, AGPL redistribution obligations, setup-download versus runtime-local separation.
|
||||
- Go/no-go recommendation: `go-with-risks`.
|
||||
- Next action: resolve `uv` availability or include bootstrap handling in Sprint 1, then create the Sprint 1 contract.
|
||||
3. Preserve strict-local runtime behavior: use local model paths, direct CLI execution, and no user-specified API or remote backend.
|
||||
|
||||
@@ -58,6 +58,8 @@ Use the project long-running harness only for substantial implementation work.
|
||||
5. `evaluation-agent` independently verifies the result against the contract.
|
||||
6. The parent agent updates `PROGRESS.md`, commits the completed change, and leaves a handoff.
|
||||
|
||||
After a chunk is no longer active, archive completed-work details in `docs/WORKARCHIVE.md` and keep `PROGRESS.md` focused on current status, blockers, and next actions.
|
||||
|
||||
Each sprint contract must include:
|
||||
|
||||
- Objective.
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# Work Archive
|
||||
|
||||
Last updated: 2026-05-08
|
||||
|
||||
This document stores completed project work, historical sprint outcomes, environment setup results, and sample conversion evidence. `PROGRESS.md` should stay focused on current status, blockers, and next actions. Read this archive when a task needs past implementation context, previous verification commands, or historical handoff details.
|
||||
|
||||
## Archive Policy
|
||||
|
||||
- Keep completed sprint outcomes here after they no longer need to stay in `PROGRESS.md`.
|
||||
- Keep `PROGRESS.md` short and current.
|
||||
- Keep source-of-truth product requirements in `PRD.md` and architecture decisions in `ARCHITECTURE.md`.
|
||||
- Keep sprint contract details under `docs/Sprints/`.
|
||||
- Do not archive or commit sample PDFs or generated conversion outputs.
|
||||
|
||||
## Completed Milestones
|
||||
|
||||
| Area | Completed Outcome | Primary References |
|
||||
| --- | --- | --- |
|
||||
| Core project documents | Created and iterated `PRD.md`, `AGENTS.md`, `ARCHITECTURE.md`, and `docs/KNOWLEDGEBASE.md`. | `PRD.md`, `AGENTS.md`, `ARCHITECTURE.md`, `docs/KNOWLEDGEBASE.md` |
|
||||
| Engine decision | Moved from the initial MinerU 2.5-era planning to fixed MinerU 3.1.0. | `PLAN.md`, `ARCHITECTURE.md`, `docs/KNOWLEDGEBASE.md` |
|
||||
| Strict-local policy | Redefined v1 runtime policy: allow direct local `mineru` CLI and its CLI-internal temporary local `mineru-api`; prohibit `--api-url`, remote APIs, router mode, HTTP client backends, and remote OpenAI-compatible backends. | `PRD.md`, `ARCHITECTURE.md`, `AGENTS.md` |
|
||||
| Agent workflow | Added shared `PLAN.md` and `PROGRESS.md` workflow, project-scoped agents, commands, skills, hooks, and the planner/generator/evaluator harness guidance. | `AGENTS.md`, `.codex/agents/`, `.codex/commands/`, `.codex/skills/`, `.codex/hooks.json` |
|
||||
| V1 implementation planning | Created `docs/V1IMPLEMENTATIONPLAN.md` and sprint contracts under `docs/Sprints/`. | `docs/V1IMPLEMENTATIONPLAN.md`, `docs/Sprints/*.md` |
|
||||
| Sprint 0 | Completed source, environment, license, privacy, and contract verification. Recommendation was `go-with-risks`. | `docs/Sprints/SPRINT0CONTRACT.md` |
|
||||
| Sprint 1 | Created minimal Python package scaffold, CLI placeholder, `pyproject.toml`, `uv.lock`, and fast pytest loop. | `docs/Sprints/SPRINT1CONTRACT.md` |
|
||||
| Sprint 2 | Implemented deterministic local PDF discovery, path planning, overwrite conflict checks, duplicate output checks, and output-root escape prevention. | `docs/Sprints/SPRINT2CONTRACT.md`, `src/pdf2md/paths.py` |
|
||||
| Sprint 3 | Implemented project-owned IR records, warning codes/severities, metadata construction, and metadata tests. | `docs/Sprints/SPRINT3CONTRACT.md`, `src/pdf2md/ir.py`, `src/pdf2md/metadata.py` |
|
||||
| Sprint 4 | Implemented mocked direct local MinerU CLI adapter boundary and strict-local validation. | `docs/Sprints/SPRINT4CONTRACT.md`, `src/pdf2md/mineru_adapter.py` |
|
||||
| Sprint 5 | Implemented Obsidian Markdown normalization, math delimiter handling, asset link normalization, and table fallback warnings. | `docs/Sprints/SPRINT5CONTRACT.md`, `src/pdf2md/markdown.py` |
|
||||
| Sprint 6 | Implemented local quality checks and human-readable report rendering from metadata and quality results. | `docs/Sprints/SPRINT6CONTRACT.md`, `src/pdf2md/quality.py`, `src/pdf2md/report.py` |
|
||||
| Sprint 7 | Implemented conversion orchestration, `convert_pdf`, `convert_input`, `pdf2md convert`, final Markdown writing, metadata/report writing, asset copying, and fake-adapter CLI/API tests. | `docs/Sprints/SPRINT7CONTRACT.md`, `src/pdf2md/conversion.py`, `src/pdf2md/cli.py` |
|
||||
| Sprint 8 | Implemented `pdf2md doctor`, local setup diagnostics, GPU/CUDA/PyTorch/model/cache checks, and setup documentation. | `docs/Sprints/SPRINT8CONTRACT.md`, `src/pdf2md/doctor.py`, `README.md` |
|
||||
| Sprint 9 | Implemented fast mocked v1 release-gate tests, optional local MinerU fixture evaluation, and `docs/V1RELEASECHECKLIST.md`. | `docs/Sprints/SPRINT9CONTRACT.md`, `docs/V1RELEASECHECKLIST.md`, `tests/integration/` |
|
||||
| GPU default/runtime setup | Made conversion default to `cuda:0`, mapped CUDA requests to MinerU subprocess environment variables, rebuilt `.venv`, installed CUDA-enabled PyTorch and MinerU 3.1.0, downloaded MinerU models, and set `MINERU_MODEL_SOURCE=local`. | `README.md`, `src/pdf2md/mineru_adapter.py`, `src/pdf2md/conversion.py` |
|
||||
| MathJax checker | Planned and implemented local MathJax render checker with Node.js helper, Python wrapper, conversion integration, and doctor diagnostics. | `docs/MATHJAXCHECKERPLAN.md`, `tools/mathjax-checker/check.mjs`, `src/pdf2md/math_render.py` |
|
||||
| Sprint 10 | Implemented opt-in pre-conversion PDF chunking with `pypdf`, temporary chunk PDF cleanup, `--chunk-pages [PAGES]`, chunk metadata/report context, and mocked tests. | `docs/Sprints/SPRINT10CONTRACT.md`, `src/pdf2md/pdf_splitter.py` |
|
||||
|
||||
## Runtime Setup Archive
|
||||
|
||||
- OS/workspace: Windows PowerShell in `D:\Work\Repos\AICoding\ConvertPDFToMD`.
|
||||
- Python target: 3.12.
|
||||
- Local Python observed during Sprint 0: 3.12.7.
|
||||
- `uv` installed per-user at `C:\Users\user\.local\bin`.
|
||||
- GPU target: NVIDIA GTX 1070 Ti 8GB.
|
||||
- Local GPU observed: NVIDIA GeForce GTX 1070 Ti, driver 577.00, 8192 MiB VRAM, WDDM.
|
||||
- MinerU execution mode: direct local `mineru` CLI only.
|
||||
- MinerU 3.1.0 CLI-internal temporary local `mineru-api` is allowed when the CLI runs without `--api-url`.
|
||||
- GTX 1070 Ti runtime setup used `torch==2.6.0+cu126`, `torchvision==0.21.0+cu126`, and `mineru[core]==3.1.0`.
|
||||
- MinerU models were downloaded with `uv run mineru-models-download -s huggingface -m all`.
|
||||
- Runtime model loading uses `MINERU_MODEL_SOURCE=local`.
|
||||
- Current doctor status after setup is WARN because GTX 1070 Ti is Pascal/pre-Turing; MinerU, CUDA PyTorch, local model config, MathJax checker, and strict-local checks pass.
|
||||
|
||||
## Sample Conversion Archive
|
||||
|
||||
Generated outputs are ignored under `outputs/` and are not committed.
|
||||
|
||||
| Sample | Result | Output Location | Summary |
|
||||
| --- | --- | --- | --- |
|
||||
| `samples/MITC공부.pdf` | Completed after CUDA-enabled runtime setup. | `outputs/MITC공부/` | 13 pages, 107 assets, 23 inline formulas, 103 display formulas, 1 info warning at the time of that run because the local MathJax checker was unavailable. |
|
||||
| `samples/FourNodeQuadrilateralShellElementMITC4.pdf` | Completed with default GPU request and `MINERU_MODEL_SOURCE=local`. | `outputs/FourNodeQuadrilateralShellElementMITC4/` | Report status `success`: 7 pages, 22 assets, 38 inline formulas, 16 display formulas, 0 math render errors, 0 warnings. |
|
||||
|
||||
## Historical Verification Highlights
|
||||
|
||||
- Sprint 1 scaffold: `uv run pytest` passed 4 tests; `uv run pdf2md --version` printed `pdf2md 0.1.0`.
|
||||
- Sprint 2 path planning: `uv run pytest tests/test_paths.py` passed 17 tests; full suite passed 21 tests.
|
||||
- Sprint 3 metadata/IR: targeted IR/metadata tests passed 25 tests; full suite passed 46 tests.
|
||||
- Sprint 4 MinerU adapter: adapter tests passed 26 tests; full suite passed 72 tests.
|
||||
- Sprint 5 Markdown normalization: targeted Markdown/IR tests passed 30 tests; full suite passed 89 tests.
|
||||
- Sprint 6 quality/report: targeted quality/report/metadata tests passed 26 tests; full suite passed 103 tests.
|
||||
- Sprint 7 conversion orchestration: full suite passed 119 tests after metadata math count fixes.
|
||||
- Sprint 8 doctor: full suite passed 133 tests.
|
||||
- Sprint 9 release gate: full suite passed 136 tests with 1 optional skip.
|
||||
- CUDA runtime rebuild: verified CUDA with an actual tensor operation on `NVIDIA GeForce GTX 1070 Ti`, compute capability 6.1; `mineru --version` reported 3.1.0.
|
||||
- MathJax checker: `npm run mathjax-checker:health` returned `{"ok":true}` after local `npm install`; full suite passed 150 tests with 1 optional skip after integration.
|
||||
- Sprint 10 chunking: targeted chunking tests passed 42 tests; full default suite passed 163 tests with 1 optional skip; `git diff --check` passed with line-ending warnings only.
|
||||
|
||||
## Historical Blockers And Resolutions
|
||||
|
||||
- Early `uv` availability was missing; resolved by installing `uv` per-user.
|
||||
- Initial real sample conversion failed when PyTorch was CPU-only; resolved by rebuilding `.venv` and installing CUDA-enabled PyTorch before MinerU.
|
||||
- Optional local MinerU fixture checks were originally blocked by missing MinerU CLI; resolved for the local environment after MinerU 3.1.0 and model setup.
|
||||
- Local MathJax render checking was originally unavailable; resolved after adding the local Node.js MathJax checker and installing local npm dependencies.
|
||||
- GTX 1070 Ti remains a Pascal/pre-Turing GPU risk. This is a warning, not a current blocker.
|
||||
|
||||
## Archive Notes For Agents
|
||||
|
||||
- Read this file when a task asks for historical implementation state, prior verification, completed sprint context, or sample conversion evidence.
|
||||
- Prefer `PROGRESS.md` for current state and next actions.
|
||||
- Prefer sprint contract files for detailed acceptance criteria and handoff structure.
|
||||
- Prefer `docs/V1RELEASECHECKLIST.md` for release readiness gates.
|
||||
Reference in New Issue
Block a user