baram2584/PDFToMD
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
dev
PDFToMD/docs/WORKARCHIVE.md
T
김경종 dc11880140 modify pdftomd
2026-05-14 10:16:59 +09:00

21 KiB
Raw Permalink Blame History

Work Archive

Last updated: 2026-05-13

This document stores completed project work, historical sprint outcomes, environment setup results, and sample conversion evidence. PROGRESS.md should stay focused on current status, blockers, and next actions. Read this archive when a task needs past implementation context, previous verification commands, or historical handoff details.

Archive Policy

  • Keep completed sprint outcomes here after they no longer need to stay in PROGRESS.md.
  • Keep PROGRESS.md short and current.
  • Keep source-of-truth product requirements in PRD.md and architecture decisions in ARCHITECTURE.md.
  • Keep sprint contract details under docs/Sprints/.
  • Do not archive or commit sample PDFs or generated conversion outputs.

Completed Milestones

Area Completed Outcome Primary References
Core project documents Created and iterated PRD.md, AGENTS.md, ARCHITECTURE.md, and docs/KNOWLEDGEBASE.md. PRD.md, AGENTS.md, ARCHITECTURE.md, docs/KNOWLEDGEBASE.md
Engine decision Moved from the initial MinerU 2.5-era planning to fixed MinerU 3.1.0. PLAN.md, ARCHITECTURE.md, docs/KNOWLEDGEBASE.md
Strict-local policy Redefined v1 runtime policy: allow direct local mineru CLI and its CLI-internal temporary local mineru-api; prohibit --api-url, remote APIs, router mode, HTTP client backends, and remote OpenAI-compatible backends. PRD.md, ARCHITECTURE.md, AGENTS.md
Agent workflow Added shared PLAN.md and PROGRESS.md workflow, project-scoped agents, commands, skills, hooks, and the planner/generator/evaluator harness guidance. AGENTS.md, .codex/agents/, .codex/commands/, .codex/skills/, .codex/hooks.json
V1 implementation planning Created docs/V1IMPLEMENTATIONPLAN.md and sprint contracts under docs/Sprints/. docs/V1IMPLEMENTATIONPLAN.md, docs/Sprints/*.md
Sprint 0 Completed source, environment, license, privacy, and contract verification. Recommendation was go-with-risks. docs/Sprints/SPRINT0CONTRACT.md
Sprint 1 Created minimal Python package scaffold, CLI placeholder, pyproject.toml, uv.lock, and fast pytest loop. docs/Sprints/SPRINT1CONTRACT.md
Sprint 2 Implemented deterministic local PDF discovery, path planning, overwrite conflict checks, duplicate output checks, and output-root escape prevention. docs/Sprints/SPRINT2CONTRACT.md, src/pdf2md/paths.py
Sprint 3 Implemented project-owned IR records, warning codes/severities, metadata construction, and metadata tests. docs/Sprints/SPRINT3CONTRACT.md, src/pdf2md/ir.py, src/pdf2md/metadata.py
Sprint 4 Implemented mocked direct local MinerU CLI adapter boundary and strict-local validation. docs/Sprints/SPRINT4CONTRACT.md, src/pdf2md/mineru_adapter.py
Sprint 5 Implemented Obsidian Markdown normalization, math delimiter handling, asset link normalization, and table fallback warnings. docs/Sprints/SPRINT5CONTRACT.md, src/pdf2md/markdown.py
Sprint 6 Implemented local quality checks and human-readable report rendering from metadata and quality results. docs/Sprints/SPRINT6CONTRACT.md, src/pdf2md/quality.py, src/pdf2md/report.py
Sprint 7 Implemented conversion orchestration, convert_pdf, convert_input, pdf2md convert, final Markdown writing, metadata/report writing, asset copying, and fake-adapter CLI/API tests. docs/Sprints/SPRINT7CONTRACT.md, src/pdf2md/conversion.py, src/pdf2md/cli.py
Sprint 8 Implemented pdf2md doctor, local setup diagnostics, GPU/CUDA/PyTorch/model/cache checks, and setup documentation. docs/Sprints/SPRINT8CONTRACT.md, src/pdf2md/doctor.py, README.md
Sprint 9 Implemented fast mocked v1 release-gate tests, optional local MinerU fixture evaluation, and docs/V1RELEASECHECKLIST.md. docs/Sprints/SPRINT9CONTRACT.md, docs/V1RELEASECHECKLIST.md, tests/integration/
GPU default/runtime setup Made conversion default to cuda:0, mapped CUDA requests to MinerU subprocess environment variables, rebuilt .venv, installed CUDA-enabled PyTorch and MinerU 3.1.0, downloaded MinerU models, and set MINERU_MODEL_SOURCE=local. README.md, src/pdf2md/mineru_adapter.py, src/pdf2md/conversion.py
MathJax checker Planned and implemented local MathJax render checker with Node.js helper, Python wrapper, conversion integration, and doctor diagnostics. docs/MATHJAXCHECKERPLAN.md, tools/mathjax-checker/check.mjs, src/pdf2md/math_render.py
Sprint 10 Implemented opt-in pre-conversion PDF chunking with pypdf, temporary chunk PDF cleanup, --chunk-pages [PAGES], chunk metadata/report context, and mocked tests. docs/Sprints/SPRINT10CONTRACT.md, src/pdf2md/pdf_splitter.py
Sprint 11 Implemented conservative MathJax warning mitigation with failed-expression details, src/pdf2md/math_repair.py, shared convert/recheck repair integration, and MATH_RENDER_REPAIRED info warnings. docs/Sprints/SPRINT11CONTRACT.md, src/pdf2md/math_repair.py, src/pdf2md/quality.py, src/pdf2md/conversion.py
UI research and Sprint 12 planning Researched minimal Windows UI launcher options and planned a thin tkinter/ttk launcher over the existing CLI with PyInstaller build output at dist/pdf2md-ui.exe. docs/UI_RESEARCH.md, docs/Sprints/SPRINT12CONTRACT.md, PLAN.md
Sprint 12 Implemented a minimal tkinter/ttk Windows UI launcher over pdf2md or uv run pdf2md, with fixed argument-list subprocess calls, worker-thread logging, cancellation, Recheck support, and PyInstaller build output at dist/pdf2md-ui.exe. docs/Sprints/SPRINT12CONTRACT.md, src/pdf2md_ui/, tests/test_ui_runner.py
Sprint 13 Implemented local pypdf text layer fidelity diagnostics, including Hangul count deltas, unexpected CJK counts, text similarity, Hangul spacing anomaly ratios, replacement-candidate markers, metadata/report integration, and recheck support without automatic body-text replacement. docs/Sprints/SPRINT13CONTRACT.md, src/pdf2md/text_fidelity.py, src/pdf2md/conversion.py, src/pdf2md/metadata.py, src/pdf2md/report.py
Sprint 14 Changed chunk mode so MinerU receives one source page per run while final Markdown, metadata, report, and assets are grouped by chunk_pages. Failed page conversions are nonfatal within partially successful groups and are recorded in metadata/report output. docs/Sprints/SPRINT14CONTRACT.md, src/pdf2md/conversion.py, src/pdf2md/report.py, tests/test_conversion.py
Sprint 15 Implemented NVIDIA GPU inventory parsing, optional --gpu auto, default --mineru-profile auto, conservative MinerU environment tuning, profile provenance in metadata/report output, and doctor GPU/profile recommendations. docs/Sprints/SPRINT15CONTRACT.md, src/pdf2md/gpu.py, src/pdf2md/mineru_profile.py, src/pdf2md/conversion.py, src/pdf2md/doctor.py
Sprint 16 Simplified public conversion outputs to one PDF-stem folder, numbered Markdown parts, shared images/, one _report.md, no persisted metadata JSON, compatibility-no-op --metadata, and legacy-only recheck. docs/Sprints/SPRINT16CONTRACT.md, src/pdf2md/paths.py, src/pdf2md/conversion.py, src/pdf2md/report.py, src/pdf2md/cli.py
UI direct-folder batch conversion Added a minimal UI workflow that selects one folder, discovers direct-child PDFs only, and sequentially runs the existing pdf2md convert command for each file with the selected options. docs/superpowers/specs/2026-05-13-ui-folder-batch-conversion-design.md, docs/superpowers/plans/2026-05-13-ui-folder-batch-conversion.md, src/pdf2md_ui/runner.py, src/pdf2md_ui/app.py
Sprint 17 planning Planned a large offline Windows installer, then abandoned the sprint at the user's request before implementation began. docs/Sprints/SPRINT17CONTRACT.md, docs/superpowers/plans/2026-05-12-offline-installer.md
Documentation archive cleanup Moved completed implementation details out of PLAN.md, PROGRESS.md, and docs/V1IMPLEMENTATIONPLAN.md, then removed Sprint 17 from active planned work after it was abandoned. PLAN.md, PROGRESS.md, docs/V1IMPLEMENTATIONPLAN.md, docs/WORKARCHIVE.md

Runtime Setup Archive

  • OS/workspace: Windows PowerShell in D:\Work\Repos\AICoding\ConvertPDFToMD.
  • Python target: 3.12.
  • Local Python observed during Sprint 0: 3.12.7.
  • uv installed per-user at C:\Users\user\.local\bin.
  • GPU target: NVIDIA GTX 1070 Ti 8GB.
  • Local GPU observed: NVIDIA GeForce GTX 1070 Ti, driver 577.00, 8192 MiB VRAM, WDDM.
  • Default conversion device/profile: --gpu cuda:0 and --mineru-profile auto.
  • MinerU execution mode: direct local mineru CLI only.
  • MinerU 3.1.0 CLI-internal temporary local mineru-api is allowed when the CLI runs without --api-url.
  • GTX 1070 Ti runtime setup used torch==2.6.0+cu126, torchvision==0.21.0+cu126, and mineru[core]==3.1.0.
  • MinerU models were downloaded with uv run mineru-models-download -s huggingface -m all.
  • Runtime model loading uses MINERU_MODEL_SOURCE=local.
  • Current doctor status after setup is WARN because GTX 1070 Ti is Pascal/pre-Turing; MinerU, CUDA PyTorch, local model config, MathJax checker, and strict-local checks pass. Sprint 15 doctor output selects cuda:0 for --gpu auto on this machine and recommends MinerU profile safe.

Sample Conversion Archive

Generated outputs are ignored under outputs/ and are not committed.

Sample Result Output Location Summary
samples/MITC공부.pdf Completed after CUDA-enabled runtime setup. outputs/MITC공부/ 13 pages, 107 assets, 23 inline formulas, 103 display formulas, 1 info warning at the time of that run because the local MathJax checker was unavailable.
samples/FourNodeQuadrilateralShellElementMITC4.pdf Completed with default GPU request and MINERU_MODEL_SOURCE=local. outputs/FourNodeQuadrilateralShellElementMITC4/ Report status success: 7 pages, 22 assets, 38 inline formulas, 16 display formulas, 0 math render errors, 0 warnings.
samples/FourNodeQuadrilateralShellElementMITC4.pdf Sprint 14 sample smoke stalled and was terminated. No final output directory. On 2026-05-12, --chunk-pages entered the one-page conversion path and used cuda:0 with GPU utilization near 100%. Source page 1 completed, but source page 2 stayed active for more than 15 minutes total runtime with no final grouped output, so the process tree was terminated and the temporary pdf2md.pages.* directory was removed.
samples/MITC공부.pdf Reconverted after Sprint 11 mitigation. outputs/MITC공부/ and outputs/sprint11-MITC공부/ Report status partial from 2 MATH_RENDER_REPAIRED info warnings: 13 pages, 107 assets, 23 inline formulas, 103 display formulas, 0 MathJax render errors, and 0 missing or invalid asset links.
samples/2007쉘구조물의유한요소해석에대하여.pdf Completed after Sprint 13 validation with 1-page chunking. outputs/2007쉘구조물의유한요소해석에대하여_pages1/ A fresh --chunk-pages 5 attempt stayed on part 001 for over 40 minutes with GPU near full utilization and no output, so it was terminated. The clean --chunk-pages 1 run completed 13/13 chunks with 0 failures, 44 warnings, 0 MathJax render errors, 13 low text-fidelity pages, 15 unexpected CJK characters, 13 diagnostic replacement-candidate pages, and 0 uncertain page mappings.
samples/SolidElement.pdf Completed after Sprint 15 GPU/profile implementation with --gpu auto --mineru-profile auto --chunk-pages. outputs/SolidElement_sprint15_auto_20260512/ Completed in about 11 minutes 51 seconds on GTX 1070 Ti. Report status partial: 6 pages, 0 failed pages, safe profile applied, 71 assets, 3 inline formulas, 55 display formulas, 0 MathJax render errors, 0 missing/invalid asset links, 11 warnings, and 5 low text-fidelity pages.
samples/SolidElement.pdf Completed after Sprint 16 simplified output layout with --gpu auto --mineru-profile auto --chunk-pages. outputs/SolidElement/ Completed in about 17 minutes 51 seconds on GTX 1070 Ti. Produced SolidElement_001.md, SolidElement_report.md, shared images/ with 71 assets, and no persisted metadata JSON. Report status partial: 6 pages, 0 failed pages, safe profile applied, 3 inline formulas, 55 display formulas, 0 MathJax render errors, 0 missing/invalid asset links, 11 warnings, and 5 low text-fidelity pages.

Historical Verification Highlights

  • Sprint 1 scaffold: uv run pytest passed 4 tests; uv run pdf2md --version printed pdf2md 0.1.0.

  • Sprint 2 path planning: uv run pytest tests/test_paths.py passed 17 tests; full suite passed 21 tests.

  • Sprint 3 metadata/IR: targeted IR/metadata tests passed 25 tests; full suite passed 46 tests.

  • Sprint 4 MinerU adapter: adapter tests passed 26 tests; full suite passed 72 tests.

  • Sprint 5 Markdown normalization: targeted Markdown/IR tests passed 30 tests; full suite passed 89 tests.

  • Sprint 6 quality/report: targeted quality/report/metadata tests passed 26 tests; full suite passed 103 tests.

  • Sprint 7 conversion orchestration: full suite passed 119 tests after metadata math count fixes.

  • Sprint 8 doctor: full suite passed 133 tests.

  • Sprint 9 release gate: full suite passed 136 tests with 1 optional skip.

  • CUDA runtime rebuild: verified CUDA with an actual tensor operation on NVIDIA GeForce GTX 1070 Ti, compute capability 6.1; mineru --version reported 3.1.0.

  • MathJax checker: npm run mathjax-checker:health returned {"ok":true} after local npm install; full suite passed 150 tests with 1 optional skip after integration.

  • Sprint 10 chunking: targeted chunking tests passed 42 tests; full default suite passed 163 tests with 1 optional skip; git diff --check passed with line-ending warnings only.

  • Sprint 11 MathJax warning mitigation: targeted tests passed 56 tests; full default suite passed 172 tests with 1 optional skip; requested samples/MITC공부.pdf validation produced 0 MathJax render errors and 2 traceable repair info warnings.

  • UI research and Sprint 12 planning: docs/UI_RESEARCH.md and docs/Sprints/SPRINT12CONTRACT.md were added; no implementation tests were required because this was documentation and planning only.

  • Sprint 12 UI implementation: uv run pytest tests\test_ui_runner.py passed 16 tests; uv run pytest passed 188 tests with 1 optional skip; uv run --group ui-build pyinstaller --clean --onefile --windowed --name pdf2md-ui src\pdf2md_ui\app.py produced dist\pdf2md-ui.exe; uv run pdf2md doctor returned WARN only for the documented GTX 1070 Ti/Pascal compatibility risk; launch smoke confirmed the executable process starts.

  • Sprint 12 residual smoke risk: a direct CLI conversion smoke using samples\FourNodeQuadrilateralShellElementMITC4.pdf and the same command shape used by the UI exceeded the 15-minute timeout on 2026-05-11. The spawned process tree was terminated with taskkill.

  • Sprint 13 text fidelity diagnostics: uv run pytest tests/test_text_fidelity.py tests/test_metadata.py tests/test_report.py tests/test_conversion.py passed 49 tests; uv run pytest passed 198 tests with 1 optional skip.

  • Sprint 13 sample validation on 2026-05-11: samples/2007쉘구조물의유한요소해석에대하여.pdf completed with --chunk-pages 1 under outputs/2007쉘구조물의유한요소해석에대하여_pages1/; generated 13 Markdown files, 13 metadata JSON files, and 13 report files.

  • Sprint 14 grouped page conversion: targeted red tests first failed against the Sprint 10 chunking behavior, then passed after implementation. uv run pytest tests/test_conversion.py tests/test_cli.py tests/test_report.py tests/test_pdf_splitter.py tests/test_paths.py tests/test_metadata.py tests/test_ui_runner.py passed 101 tests; full uv run pytest passed 202 tests with 1 optional skip.

  • Sprint 14 sample smoke on 2026-05-12: uv run pdf2md convert samples\FourNodeQuadrilateralShellElementMITC4.pdf --out outputs\FourNodeQuadrilateralShellElementMITC4_sprint14_20260512_112342 --chunk-pages --strict-local used cuda:0 with GPU utilization near 100%, reached source page 2, then exceeded 15 minutes total runtime without producing a final output directory. The process tree was terminated and the leftover temporary directory was removed.

  • Sprint 15 NVIDIA GPU detection/profile tuning: targeted tests uv run pytest tests/test_gpu.py tests/test_mineru_profile.py tests/test_mineru_adapter.py tests/test_conversion.py tests/test_cli.py tests/test_doctor.py passed 101 tests. Full uv run pytest passed 225 tests with 1 optional skip. uv run pdf2md doctor returned WARN on the local GTX 1070 Ti, reported GPU 0 with 8192 MiB VRAM, selected cuda:0 for --gpu auto, and recommended profile safe. Optional stronger-PC real MinerU conversion validation was not run in this workspace.

  • SolidElement sample validation on 2026-05-12: uv run pdf2md convert samples\SolidElement.pdf --out outputs\SolidElement_sprint15_auto_20260512 --overwrite --chunk-pages --gpu auto --mineru-profile auto --strict-local completed successfully with one grouped output and no failed source pages.

  • Sprint 16 simplified output layout: focused verification uv run pytest tests/test_paths.py tests/test_conversion.py tests/test_cli.py tests/test_report.py tests/test_ui_runner.py tests/integration/test_v1_fast_release_gate.py -q passed 91 tests; full uv run pytest passed 227 tests with 1 optional skip; git diff --check passed with line-ending warnings only. New conversions write <out>/<stem>/<stem>_001.md, shared <out>/<stem>/images/, and <out>/<stem>/<stem>_report.md; no public .metadata.json is written.

  • Sprint 16 SolidElement sample validation on 2026-05-12: uv run pdf2md convert samples\SolidElement.pdf --out outputs --overwrite --chunk-pages --gpu auto --mineru-profile auto --strict-local completed successfully with one simplified Markdown part, one report, shared images, no public metadata JSON, and no failed source pages.

  • UI direct-folder batch conversion on 2026-05-13: uv run pytest tests/test_ui_runner.py -q passed 19 tests; uv run python -m py_compile src\pdf2md_ui\app.py src\pdf2md_ui\runner.py passed; uv run pytest -q passed 230 tests with 1 skipped; PyInstaller rebuilt dist\pdf2md-ui.exe; a short process-start smoke confirmed the executable starts.

  • Sprint 17 planning on 2026-05-12: docs/Sprints/SPRINT17CONTRACT.md and docs/superpowers/plans/2026-05-12-offline-installer.md were added. No implementation tests were required because this was planning only.

  • Sprint 17 abandonment on 2026-05-13: offline installer planning was abandoned at the user's request before implementation began. The contract and plan remain historical records only.

Archived V1 Implementation Plan

docs/V1IMPLEMENTATIONPLAN.md now tracks current state and planned next work only. Completed Sprint 0 through Sprint 16 details are archived here and in their respective docs/Sprints/SPRINT*CONTRACT.md files.

Current completed v1 capability summary:

  • Python 3.12 package and pdf2md CLI.
  • Direct local MinerU 3.1.0 CLI adapter with strict-local enforcement.
  • Obsidian Markdown normalization, local quality checks, internal provenance, and one human-readable report.
  • pdf2md doctor, local MathJax checking, conservative MathJax warning mitigation, and pypdf text fidelity diagnostics.
  • Opt-in grouped page conversion where MinerU receives one source page per run.
  • NVIDIA GPU detection, --gpu auto, and --mineru-profile auto|safe|performance.
  • Simplified public output layout with no public metadata JSON for new conversions.
  • Minimal Windows UI launcher with direct-folder batch conversion through sequential existing CLI calls.

Current planned next work:

  • No active implementation sprint. Future substantial work should start from a new user-approved requirement and sprint contract.

Historical Blockers And Resolutions

  • Early uv availability was missing; resolved by installing uv per-user.
  • Initial real sample conversion failed when PyTorch was CPU-only; resolved by rebuilding .venv and installing CUDA-enabled PyTorch before MinerU.
  • Optional local MinerU fixture checks were originally blocked by missing MinerU CLI; resolved for the local environment after MinerU 3.1.0 and model setup.
  • Local MathJax render checking was originally unavailable; resolved after adding the local Node.js MathJax checker and installing local npm dependencies.
  • GTX 1070 Ti remains a Pascal/pre-Turing GPU risk. This is a warning, not a current blocker.

Archive Notes For Agents

  • Read this file when a task asks for historical implementation state, prior verification, completed sprint context, or sample conversion evidence.
  • Prefer PROGRESS.md for current state and next actions.
  • Prefer sprint contract files for detailed acceptance criteria and handoff structure.
  • Prefer docs/V1RELEASECHECKLIST.md for release readiness gates.
Reference in New Issue View Git Blame Copy Permalink