modify harness framework
This commit is contained in:
KOKO\Mimi
@@ -10,7 +10,7 @@ Mission:
|
||||
- Run build and test validation only after Implementation Agent work.
|
||||
- Execute independent C++/MSVC/CMake/CTest validation and summarize failures for handoff.
|
||||
- Record command, exit code, duration, stdout/stderr summary, failed test names, and failure classification.
|
||||
- Keep the output aligned with AGENTS.md, docs/SOLVER_AGENT_DESIGN.md, scripts/validate_workspace.py, and the implementation plan/report.
|
||||
- Keep the output aligned with AGENTS.md, docs/HARNESS_WORKFLOW.md, docs/SOLVER_AGENT_DESIGN.md, `.harness/config.json` when present, and the implementation plan/report.
|
||||
|
||||
Skill references:
|
||||
- Use $fesa-cpp-msvc-tdd when running C++/MSVC/CMake/CTest validation, recording validation evidence, classifying build/test failures, or preparing build/test handoffs.
|
||||
@@ -32,23 +32,26 @@ Input priorities:
|
||||
2. Implementation Agent report.
|
||||
3. docs/implementation-plans/<feature-id>-implementation-plan.md.
|
||||
4. AGENTS.md and docs/SOLVER_AGENT_DESIGN.md.
|
||||
5. scripts/validate_workspace.py.
|
||||
6. CMakePresets.json, CMakeLists.txt, CMake files, and CTest metadata when present.
|
||||
5. `.harness/config.json` when present.
|
||||
6. CMakePresets.json, CMakeLists.txt, CMake files, Visual Studio solution/project files, and CTest metadata when present.
|
||||
7. Related docs/reference-models/<feature-id>-reference-models.md when present.
|
||||
8. Stored reference artifacts when present, read-only.
|
||||
|
||||
Execution contract:
|
||||
- Default validation is python scripts/validate_workspace.py.
|
||||
- If the implementation plan requires harness self-test, run python -m unittest discover -s scripts -p "test_*.py" first.
|
||||
- If the implementation plan lists feature-specific CTest commands, run those before full workspace validation.
|
||||
- Run full workspace validation with python scripts/validate_workspace.py last.
|
||||
- scripts/validate_workspace.py resolves HARNESS_VALIDATION_COMMANDS, CMakePresets.json msvc-debug, or CMake/MSVC x64 Debug commands.
|
||||
- The default CMake/MSVC x64 Debug commands are:
|
||||
1. cmake -S . -B build/msvc-debug -G "Visual Studio 17 2022" -A x64
|
||||
2. cmake --build build/msvc-debug --config Debug
|
||||
3. ctest --test-dir build/msvc-debug --output-on-failure -C Debug
|
||||
- Resolve the validation path from `.harness/config.json` first, then Harness project auto detection.
|
||||
- If Harness Python, Hook, or agent-config behavior changed, run `uv run --with pytest python -m pytest -v -rs` first.
|
||||
- Configure and build before running feature-specific and full tests.
|
||||
- If the implementation plan lists feature-specific CTest commands, run them after build and before the full test run.
|
||||
- For a non-preset CMake project, run:
|
||||
1. cmake -S . -B .harness/build -A x64
|
||||
2. cmake --build .harness/build --config Debug
|
||||
3. ctest --test-dir .harness/build -C Debug -R <feature-or-label> --output-on-failure when specified
|
||||
4. ctest --test-dir .harness/build -C Debug --show-only=json-v1
|
||||
5. ctest --test-dir .harness/build -C Debug --output-on-failure
|
||||
- If `.harness/config.json` selects CMake presets, use its configure/build/test presets and binary directory.
|
||||
- If it selects direct MSBuild, use its solution, configuration, platform, and required `msbuild.testCommand`.
|
||||
- Preserve command order, exit code, duration, and stdout/stderr tail for every executed command.
|
||||
- For no-CMake workspaces, record the scripts/validate_workspace.py informational success path instead of treating it as a failure.
|
||||
- Record a no-project pass only when no C/C++ files and no build metadata exist. C/C++ files without build metadata are an error.
|
||||
- Stop after the first decisive failure unless the implementation plan explicitly asks for additional diagnostic commands.
|
||||
|
||||
Failure classification:
|
||||
@@ -57,13 +60,13 @@ Failure classification:
|
||||
- link: link step failed.
|
||||
- test: CTest or unit/integration tests failed.
|
||||
- reference-comparison: reference comparison test ran and reported comparison failure.
|
||||
- harness: Python harness self-test or validation script failed.
|
||||
- harness: Python Harness test, PreToolUse/Stop Hook, config loading, discovery, or adapter validation failed.
|
||||
- environment: generator, compiler, Python, path, permission, or local machine dependency is missing.
|
||||
- upstream-contract: implementation plan, requirements, formulation, I/O definition, reference artifacts, or tolerance policy is inconsistent or incomplete.
|
||||
|
||||
Required Build/Test Report sections:
|
||||
1. Metadata: feature_id, source implementation report, status, owner_agent, date.
|
||||
2. Execution Environment: OS, generator, platform, config, build dir, and active override env vars.
|
||||
2. Execution Environment: OS, generator, platform, config, build dir, Harness config presence, and project selection path.
|
||||
3. Command Log Summary: command, exit code, duration, stdout/stderr tail.
|
||||
4. Validation Results: harness self-test, configure, build, CTest, and feature-specific tests.
|
||||
5. Failure Classification: configure | compile | link | test | reference-comparison | harness | environment | upstream-contract.
|
||||
|
||||
@@ -48,8 +48,9 @@ Execution contract:
|
||||
- MINIMAL FIX: modify only implementation-owned source, header, test, or CMake files needed to fix the classified failure.
|
||||
- MINIMAL FIX: keep changes surgical and traceable to the failure report or implementation plan acceptance criterion.
|
||||
- VERIFY: rerun the targeted command that reproduced the failure first.
|
||||
- VERIFY: run python scripts/validate_workspace.py after the targeted command.
|
||||
- VERIFY: run python -m unittest discover -s scripts -p "test_*.py" when harness, hook, or agent config behavior is involved.
|
||||
- VERIFY: run the full MSVC build/test commands resolved from `.harness/config.json` or Harness auto detection after the targeted command.
|
||||
- VERIFY: run `uv run --with pytest python -m pytest -v -rs` when Harness Python, Hook, or agent config behavior is involved.
|
||||
- VERIFY: allow Stop to rerun whole-project MSVC build/test before the correction Step ends.
|
||||
- If the same classification repeats after two focused correction attempts, stop and hand off to Coordinator Agent or the relevant upstream agent.
|
||||
- If a fix requires changing requirements, formulations, I/O contracts, reference artifacts, tolerance policies, or reference provenance, stop with needs-upstream-decision.
|
||||
- If the failure is environment-owned, do not work around it with code changes; classify it as needs-environment-fix.
|
||||
@@ -61,7 +62,7 @@ Failure classification:
|
||||
- link: linker, symbol resolution, library registration, or target dependency failed.
|
||||
- test: CTest, unit, integration, parser/I/O, or ordinary regression test failed.
|
||||
- reference-comparison: deterministic reference comparison test failed against stored artifacts.
|
||||
- harness: Python harness self-test, TDD guard, hook, or validation script failed.
|
||||
- harness: Python Harness test, PreToolUse/Stop Hook, config loading, discovery, or adapter validation failed.
|
||||
- environment: MSVC, CMake, Python, path, permission, generator, or local dependency issue.
|
||||
- upstream-contract: requirements, formulation, I/O, reference artifact, tolerance, or implementation plan is incomplete or inconsistent.
|
||||
|
||||
@@ -70,7 +71,7 @@ Required Correction Report sections:
|
||||
2. Failure Triage: classification, first failed command, failed target or test, and evidence tail.
|
||||
3. Root Cause Summary: implementation defect, test defect, CMake registration issue, environment issue, or upstream-contract issue.
|
||||
4. Correction Scope: changed source, header, test, and CMake files plus excluded upstream contract files.
|
||||
5. Verification Evidence: targeted command, python scripts/validate_workspace.py, and Python harness self-test when relevant.
|
||||
5. Verification Evidence: targeted command, config-resolved full MSVC build/test, Stop result, and Harness Python pytest when relevant.
|
||||
6. Traceability: requirement id, task id, test id, failing command, corrected file, and acceptance criterion.
|
||||
7. Handoff Recommendation: Implementation Agent, Build/Test Executor Agent, Reference Verification Agent, Physics Evaluation Agent, upstream agent, or Coordinator Agent.
|
||||
8. Stop Condition: repeated failure, upstream ambiguity, reference artifact gap, or environment blocker.
|
||||
|
||||
@@ -43,7 +43,9 @@ Execution contract:
|
||||
- RED: write the planned C++ unit, integration, parser/I/O, or reference-comparison test first.
|
||||
- RED: run the targeted test and verify failure before production implementation.
|
||||
- GREEN: implement the minimum code needed for the planned task and acceptance criterion.
|
||||
- VERIFY: run the targeted CTest command, then the workspace validation commands.
|
||||
- VERIFY: run the targeted CTest command, then the full MSVC build/test commands resolved from `.harness/config.json` or the Harness defaults.
|
||||
- VERIFY: record RED and GREEN evidence explicitly; PreToolUse only checks that a related test file exists.
|
||||
- VERIFY: allow Stop to rerun whole-project MSVC build/test before the Step ends.
|
||||
- If a C++ production file changes, a related C++ test file must be present in the same patch or already exist.
|
||||
- CMake/CTest changes must stay compatible with MSVC x64 Debug validation.
|
||||
- Abaqus reference CSV files are read-only verification inputs.
|
||||
@@ -69,15 +71,19 @@ Required Implementation Report sections:
|
||||
2. Implemented Scope: completed task ids, skipped task ids, and reason.
|
||||
3. Test Evidence: tests written first, observed RED failure, GREEN pass, and commands.
|
||||
4. Code Changes: source, header, test, and CMake/CTest change summary.
|
||||
5. Validation Evidence: ctest -C Debug, python scripts/validate_workspace.py, and python -m unittest discover -s scripts -p "test_*.py" when relevant.
|
||||
5. Validation Evidence: targeted CTest, config-resolved full MSVC build/test, Stop result, and `uv run --with pytest python -m pytest -v -rs` when Harness Python behavior is relevant.
|
||||
6. Traceability: requirement id, task id, test id, and acceptance criterion.
|
||||
7. Blockers: upstream document mismatch, reference artifact gaps, formulation ambiguity, I/O ambiguity, or repeated failure.
|
||||
8. Downstream Handoff: Build/Test Executor Agent, Correction Agent, and Reference Verification Agent.
|
||||
|
||||
Validation commands:
|
||||
- python -m unittest discover -s scripts -p "test_*.py"
|
||||
- python scripts/validate_workspace.py
|
||||
- ctest -C Debug -R <feature-or-label>
|
||||
- cmake -S . -B .harness/build -A x64
|
||||
- cmake --build .harness/build --config Debug
|
||||
- ctest --test-dir .harness/build -C Debug -R <feature-or-label> --output-on-failure
|
||||
- ctest --test-dir .harness/build -C Debug --show-only=json-v1
|
||||
- ctest --test-dir .harness/build -C Debug --output-on-failure
|
||||
- Use configured CMake presets or direct MSBuild commands instead when `.harness/config.json` selects them.
|
||||
- Run `uv run --with pytest python -m pytest -v -rs` when Harness Python, Hook, or agent-config behavior changes.
|
||||
|
||||
Status rules:
|
||||
- in-progress: implementation is underway.
|
||||
|
||||
@@ -55,11 +55,11 @@ Required Implementation Plan sections:
|
||||
3. Implementation Scope: included behavior, excluded behavior, and non-goals.
|
||||
4. Work Breakdown: small ordered implementation tasks with task ids and dependencies.
|
||||
5. TDD Test Plan: unit, integration, parser/I/O, and reference-comparison tests ordered by RED/GREEN cycle.
|
||||
6. CMake/CTest Plan: target candidates, add_test needs, labels, and ctest -C Debug execution expectations.
|
||||
6. CMake/CTest Plan: target candidates, add_test needs, labels, and `.harness/config.json` or default `.harness/build` execution expectations.
|
||||
7. Candidate Files and Ownership: candidate source/header/test/CMake files and responsibility boundary; never final API.
|
||||
8. Data Flow Contract: Abaqus .inp input, internal model, solver results.h5, Abaqus reference CSV files under reference/<model-id>/, and FESA HDF5-to-reference-CSV comparison flow.
|
||||
9. Acceptance Traceability Matrix: requirement id, task id, test id, reference model id, and acceptance criterion.
|
||||
10. Validation Commands: python -m unittest discover -s scripts -p \"test_*.py\", python scripts/validate_workspace.py, and feature-specific CTest commands.
|
||||
10. Validation Commands: config-resolved full MSVC build/test commands, feature-specific CTest commands, and `uv run --with pytest python -m pytest -v -rs` when Harness Python behavior is in scope.
|
||||
11. Risks and Downstream Handoff: Implementation Agent, Build/Test Executor Agent, Correction Agent, and Reference Verification Agent.
|
||||
12. Open Issues: requirements, formulation, I/O, reference artifacts, tolerance, or architecture gaps that prevent ready-for-implementation.
|
||||
|
||||
|
||||
@@ -62,7 +62,7 @@ Required Release Report sections:
|
||||
2. Release Scope: included functionality, excluded functionality, supported analysis type, elements, materials, I/O subset, and artifact scope.
|
||||
3. Gate Evidence Inventory: requirements, formulation, numerical review, I/O definition, reference model, implementation, build/test, reference verification, and physics evaluation status.
|
||||
4. Acceptance Traceability: requirement id, acceptance criterion, test id, reference model id, verification report, and release disposition.
|
||||
5. Validation Evidence: python scripts/validate_workspace.py, CMake/MSVC/CTest evidence, reference verification status, and physics evaluation status.
|
||||
5. Validation Evidence: Build/Test report's config-resolved CMake/MSVC/CTest commands, Harness Python pytest when applicable, reference verification status, and physics evaluation status.
|
||||
6. Known Limitations: unsupported Abaqus keywords, element/material/analysis constraints, deferred issues, accepted risks, and open items.
|
||||
7. Release Notes Draft: user-facing feature summary, verification scope, main limitations, artifact paths, and usage notes.
|
||||
8. Release Verdict: ready-for-release | needs-correction | needs-reference-verification | needs-physics-evaluation | needs-documentation | needs-upstream-decision | blocked.
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
#:schema https://developers.openai.com/codex/config-schema.json
|
||||
|
||||
[features]
|
||||
codex_hooks = true
|
||||
hooks = true
|
||||
|
||||
+13
-9
@@ -1,25 +1,29 @@
|
||||
{
|
||||
"description": "Harness TDD, command safety, and MSVC C/C++ validation hooks.",
|
||||
"hooks": {
|
||||
"PreToolUse": [
|
||||
{
|
||||
"matcher": "^Bash$",
|
||||
"matcher": "Bash|shell_command|PowerShell|apply_patch|Edit|MultiEdit|Write",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "python -c \"import pathlib, runpy, subprocess; root = pathlib.Path(subprocess.check_output(['git', 'rev-parse', '--show-toplevel'], text=True).strip()); runpy.run_path(str(root / '.codex' / 'hooks' / 'pre_commit_checks.py'), run_name='__main__')\"",
|
||||
"timeout": 600,
|
||||
"statusMessage": "Running pre-commit checks"
|
||||
"command": "python3 -X utf8 \"$(git rev-parse --show-toplevel)/scripts/hooks/pre_tool_use.py\"",
|
||||
"commandWindows": "python -X utf8 \"$(git rev-parse --show-toplevel)/scripts/hooks/pre_tool_use.py\"",
|
||||
"timeout": 30,
|
||||
"statusMessage": "Checking Harness policies"
|
||||
}
|
||||
]
|
||||
},
|
||||
}
|
||||
],
|
||||
"Stop": [
|
||||
{
|
||||
"matcher": "^(apply_patch|Edit|Write)$",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "python -c \"import pathlib, runpy, subprocess; root = pathlib.Path(subprocess.check_output(['git', 'rev-parse', '--show-toplevel'], text=True).strip()); runpy.run_path(str(root / '.codex' / 'hooks' / 'tdd-guard.py'), run_name='__main__')\"",
|
||||
"timeout": 30,
|
||||
"statusMessage": "Checking TDD guard"
|
||||
"command": "python3 -X utf8 \"$(git rev-parse --show-toplevel)/scripts/hooks/stop_validation.py\"",
|
||||
"commandWindows": "python -X utf8 \"$(git rev-parse --show-toplevel)/scripts/hooks/stop_validation.py\"",
|
||||
"timeout": 1800,
|
||||
"statusMessage": "Running MSVC build and tests"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,89 +0,0 @@
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _repo_root(cwd: Path) -> Path:
|
||||
try:
|
||||
root = subprocess.check_output(
|
||||
["git", "rev-parse", "--show-toplevel"],
|
||||
cwd=cwd,
|
||||
text=True,
|
||||
stderr=subprocess.DEVNULL,
|
||||
).strip()
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
return cwd
|
||||
return Path(root)
|
||||
|
||||
|
||||
def _is_git_commit(command: str) -> bool:
|
||||
return re.search(
|
||||
r"^\s*git(?:\s+(?:-[A-Za-z]\s+\S+|--[A-Za-z0-9-]+(?:=\S+)?))*\s+commit\b",
|
||||
command,
|
||||
) is not None
|
||||
|
||||
|
||||
def _deny(reason: str) -> None:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"hookSpecificOutput": {
|
||||
"hookEventName": "PreToolUse",
|
||||
"permissionDecision": "deny",
|
||||
"permissionDecisionReason": reason,
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _tail(text: str, limit: int = 1200) -> str:
|
||||
text = text.strip()
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return text[-limit:]
|
||||
|
||||
|
||||
def _build_pre_commit_commands(root: Path) -> list[list[str]]:
|
||||
return [
|
||||
[sys.executable, "-m", "unittest", "discover", "-s", "scripts", "-p", "test_*.py"],
|
||||
[sys.executable, "scripts/validate_workspace.py"],
|
||||
]
|
||||
|
||||
|
||||
def _run_checks(root: Path) -> str | None:
|
||||
for command in _build_pre_commit_commands(root):
|
||||
result = subprocess.run(command, cwd=root, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
details = _tail(result.stdout + "\n" + result.stderr)
|
||||
label = " ".join(command)
|
||||
if details:
|
||||
return f"{label} failed:\n{details}"
|
||||
return f"{label} failed with exit code {result.returncode}."
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
payload = json.load(sys.stdin)
|
||||
except json.JSONDecodeError:
|
||||
return 0
|
||||
|
||||
command = payload.get("tool_input", {}).get("command", "")
|
||||
if not isinstance(command, str) or not _is_git_commit(command):
|
||||
return 0
|
||||
|
||||
cwd = Path(payload.get("cwd") or Path.cwd())
|
||||
root = _repo_root(cwd)
|
||||
failure = _run_checks(root)
|
||||
if failure:
|
||||
_deny(f"PRE-COMMIT CHECKS: {failure}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,205 +0,0 @@
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SOURCE_SUFFIXES = {".h", ".hpp", ".hh", ".hxx", ".c", ".cc", ".cpp", ".cxx", ".ixx"}
|
||||
TEST_SUFFIXES = {".h", ".hpp", ".hh", ".hxx", ".c", ".cc", ".cpp", ".cxx", ".ixx"}
|
||||
CONFIG_SUFFIXES = {".json", ".md", ".yml", ".yaml", ".txt", ".cmake"}
|
||||
|
||||
|
||||
def _repo_root(cwd: Path) -> Path:
|
||||
try:
|
||||
root = subprocess.check_output(
|
||||
["git", "rev-parse", "--show-toplevel"],
|
||||
cwd=cwd,
|
||||
text=True,
|
||||
stderr=subprocess.DEVNULL,
|
||||
).strip()
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
return cwd
|
||||
return Path(root)
|
||||
|
||||
|
||||
def _extract_patch_paths(command: str) -> list[str]:
|
||||
prefixes = (
|
||||
"*** Add File: ",
|
||||
"*** Update File: ",
|
||||
"*** Delete File: ",
|
||||
"*** Move to: ",
|
||||
)
|
||||
paths: list[str] = []
|
||||
for raw_line in command.splitlines():
|
||||
line = raw_line.strip()
|
||||
for prefix in prefixes:
|
||||
if line.startswith(prefix):
|
||||
paths.append(line[len(prefix) :].strip())
|
||||
break
|
||||
return paths
|
||||
|
||||
|
||||
def _touched_paths(payload: dict) -> list[str]:
|
||||
tool_input = payload.get("tool_input", {})
|
||||
if not isinstance(tool_input, dict):
|
||||
return []
|
||||
|
||||
file_path = tool_input.get("file_path")
|
||||
if isinstance(file_path, str) and file_path:
|
||||
return [file_path]
|
||||
|
||||
command = tool_input.get("command")
|
||||
if isinstance(command, str):
|
||||
return _extract_patch_paths(command)
|
||||
|
||||
return []
|
||||
|
||||
|
||||
def _normalize(path_text: str) -> str:
|
||||
return path_text.replace("\\", "/").lower()
|
||||
|
||||
|
||||
def _is_test_path(path_text: str) -> bool:
|
||||
normalized = _normalize(path_text)
|
||||
name = normalized.rsplit("/", 1)[-1]
|
||||
path = Path(path_text)
|
||||
return (
|
||||
"/tests/" in f"/{normalized}"
|
||||
or "/test/" in f"/{normalized}"
|
||||
or name.endswith("_test.cpp")
|
||||
or name.startswith("test_")
|
||||
or ".test." in name
|
||||
or ".spec." in name
|
||||
) and path.suffix.lower() in TEST_SUFFIXES
|
||||
|
||||
|
||||
def _token(text: str) -> str:
|
||||
return "".join(ch for ch in text.lower() if ch.isalnum())
|
||||
|
||||
|
||||
def _module_token(path: Path) -> str:
|
||||
parts = [part.lower() for part in path.parts]
|
||||
for marker in ("include", "src"):
|
||||
if marker not in parts:
|
||||
continue
|
||||
idx = parts.index(marker)
|
||||
if marker == "include" and idx + 2 < len(parts) and parts[idx + 1] == "fesa":
|
||||
return _token(parts[idx + 2])
|
||||
if marker == "src" and idx + 1 < len(parts):
|
||||
return _token(parts[idx + 1])
|
||||
return ""
|
||||
|
||||
|
||||
def _related_tokens(path: Path) -> set[str]:
|
||||
tokens = {_token(_base_name(path))}
|
||||
module = _module_token(path)
|
||||
if module:
|
||||
tokens.add(module)
|
||||
return {token for token in tokens if token}
|
||||
|
||||
|
||||
def _candidate_test_paths(paths: list[str], cwd: Path, root: Path) -> list[Path]:
|
||||
candidates: list[Path] = []
|
||||
for path_text in paths:
|
||||
resolved = _resolve_path(path_text, cwd)
|
||||
if _is_test_path(str(resolved)):
|
||||
candidates.append(resolved)
|
||||
|
||||
for test_root_name in ("tests", "test"):
|
||||
test_root = root / test_root_name
|
||||
if not test_root.is_dir():
|
||||
continue
|
||||
for suffix in TEST_SUFFIXES:
|
||||
candidates.extend(test_root.rglob(f"*{suffix}"))
|
||||
|
||||
return candidates
|
||||
|
||||
|
||||
def _has_related_test(path: Path, candidate_tests: list[Path]) -> bool:
|
||||
tokens = _related_tokens(path)
|
||||
for test_path in candidate_tests:
|
||||
test_token = _token(test_path.stem)
|
||||
if any(token and token in test_token for token in tokens):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _is_exempt(path_text: str) -> bool:
|
||||
normalized = _normalize(path_text)
|
||||
path = Path(path_text)
|
||||
name = path.name.lower()
|
||||
|
||||
if name == "cmakelists.txt":
|
||||
return True
|
||||
if _is_test_path(path_text):
|
||||
return True
|
||||
if path.suffix.lower() in CONFIG_SUFFIXES:
|
||||
return True
|
||||
if "/cmake/" in normalized:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def _resolve_path(path_text: str, cwd: Path) -> Path:
|
||||
path = Path(path_text)
|
||||
if path.is_absolute():
|
||||
return path
|
||||
return (cwd / path).resolve()
|
||||
|
||||
|
||||
def _base_name(path: Path) -> str:
|
||||
for suffix in sorted(SOURCE_SUFFIXES, key=len, reverse=True):
|
||||
if path.name.lower().endswith(suffix):
|
||||
return path.name[: -len(suffix)]
|
||||
return path.stem
|
||||
|
||||
|
||||
def _guarded_paths(paths: list[str], cwd: Path, root: Path) -> list[str]:
|
||||
missing_tests: list[str] = []
|
||||
candidate_tests = _candidate_test_paths(paths, cwd, root)
|
||||
for path_text in paths:
|
||||
if _is_exempt(path_text):
|
||||
continue
|
||||
|
||||
path = _resolve_path(path_text, cwd)
|
||||
if path.suffix.lower() not in SOURCE_SUFFIXES:
|
||||
continue
|
||||
if not _has_related_test(path, candidate_tests):
|
||||
missing_tests.append(_base_name(path))
|
||||
|
||||
return missing_tests
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
payload = json.load(sys.stdin)
|
||||
except json.JSONDecodeError:
|
||||
return 0
|
||||
|
||||
cwd = Path(payload.get("cwd") or Path.cwd())
|
||||
root = _repo_root(cwd)
|
||||
missing_tests = _guarded_paths(_touched_paths(payload), cwd, root)
|
||||
if not missing_tests:
|
||||
return 0
|
||||
|
||||
names = ", ".join(sorted(set(missing_tests)))
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"hookSpecificOutput": {
|
||||
"hookEventName": "PreToolUse",
|
||||
"permissionDecision": "deny",
|
||||
"permissionDecisionReason": (
|
||||
"TDD GUARD: missing test file for "
|
||||
f"{names}. Write or add the test first."
|
||||
),
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -26,10 +26,12 @@ Read these first:
|
||||
3. RED: write the planned unit, integration, parser/I/O, or reference-comparison test first.
|
||||
4. RED: run the targeted test and verify the expected failure before production code.
|
||||
5. GREEN: implement the minimum C++17/MSVC-compatible code needed for the task.
|
||||
6. VERIFY: run the targeted command, then `python scripts/validate_workspace.py`.
|
||||
6. VERIFY: run the targeted command, then the full MSVC build/test commands resolved from `.harness/config.json` or the Harness defaults.
|
||||
7. For C++ production changes, require a related C++ test file in the same patch or already present.
|
||||
8. For failure triage, classify as `configure | compile | link | test | reference-comparison | harness | environment | upstream-contract`.
|
||||
9. Fix implementation-owned failures only and keep changes traceable to the implementation plan.
|
||||
8. Treat PreToolUse as a test-file-existence guardrail, not proof that RED was observed. Record the RED and GREEN commands and results in the implementation report.
|
||||
9. Let Stop perform the final whole-project MSVC build/test before the Step ends.
|
||||
10. For failure triage, classify as `configure | compile | link | test | reference-comparison | harness | environment | upstream-contract`.
|
||||
11. Fix implementation-owned failures only and keep changes traceable to the implementation plan.
|
||||
|
||||
## Output Contract
|
||||
|
||||
@@ -43,17 +45,19 @@ Produce one of these, depending on role:
|
||||
Required validation commands:
|
||||
|
||||
```powershell
|
||||
python -m unittest discover -s scripts -p "test_*.py"
|
||||
python scripts/validate_workspace.py
|
||||
ctest -C Debug -R <feature-or-label>
|
||||
cmake -S . -B .harness/build -A x64
|
||||
cmake --build .harness/build --config Debug
|
||||
ctest --test-dir .harness/build -C Debug -R <feature-or-label> --output-on-failure
|
||||
ctest --test-dir .harness/build -C Debug --show-only=json-v1
|
||||
ctest --test-dir .harness/build -C Debug --output-on-failure
|
||||
```
|
||||
|
||||
Default MSVC path:
|
||||
Use configured CMake presets or direct MSBuild commands instead when
|
||||
`.harness/config.json` selects them. For Harness Python, Hook, or agent-config
|
||||
changes, also run:
|
||||
|
||||
```powershell
|
||||
cmake -S . -B build/msvc-debug -G "Visual Studio 17 2022" -A x64
|
||||
cmake --build build/msvc-debug --config Debug
|
||||
ctest --test-dir build/msvc-debug --output-on-failure -C Debug
|
||||
uv run --with pytest python -m pytest -v -rs
|
||||
```
|
||||
|
||||
## Boundaries
|
||||
@@ -73,6 +77,7 @@ ctest --test-dir build/msvc-debug --output-on-failure -C Debug
|
||||
- Every `must` requirement maps to at least one task and one test.
|
||||
- Each test has a clear RED condition, GREEN condition, linked task, and command.
|
||||
- CMake/CTest plans remain compatible with MSVC x64 Debug validation.
|
||||
- Stop validation is green for the whole discovered C/C++ project; a no-project pass is valid only when no C/C++ files and no build metadata exist.
|
||||
- Build/test reports record command, exit code, duration, stdout/stderr tail, and failure classification.
|
||||
- Correction attempts stop when repeated failure indicates upstream contract ambiguity.
|
||||
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
---
|
||||
name: harness-review
|
||||
description: Use when reviewing this C++/MSVC Harness repository: local changes, generated phase files, step outputs, implementation diffs, missing tests, MSVC build readiness, or compliance with AGENTS.md, docs/ARCHITECTURE.md, docs/ADR.md, and Harness acceptance criteria.
|
||||
---
|
||||
|
||||
# Harness Review
|
||||
|
||||
## Overview
|
||||
|
||||
Use this skill to review Harness work against the repository's persistent rules, architecture docs, C++/MSVC constraints, TDD guard policy, and executable verification requirements. Prioritize bugs, regressions, missing tests, and rule violations.
|
||||
|
||||
## Review Process
|
||||
|
||||
1. Read `/AGENTS.md`, `/docs/ARCHITECTURE.md`, and `/docs/ADR.md`.
|
||||
2. Inspect the changed files with `git status --short` and `git diff`.
|
||||
3. Check architecture, stack choices, C++ test coverage, critical rules, and MSVC/CMake readiness.
|
||||
4. Run relevant verification commands when feasible. If a command cannot be run, report that as residual risk.
|
||||
5. Lead with actionable findings. Keep summaries secondary.
|
||||
|
||||
## Checklist
|
||||
|
||||
| Item | Question |
|
||||
| --- | --- |
|
||||
| Architecture | Does the change follow `docs/ARCHITECTURE.md` ownership boundaries? |
|
||||
| Stack | Does the change stay within C++/MSVC/CMake decisions documented in `docs/ADR.md`? |
|
||||
| Tests | Are new or changed behaviors covered by Python Harness tests or C++ tests? |
|
||||
| TDD Guard | Would C++ production edits be blocked without related tests? |
|
||||
| Critical Rules | Does the change violate any `AGENTS.md` CRITICAL rule? |
|
||||
| Build | Do `python -m unittest discover -s scripts -p "test_*.py"` and `python scripts/validate_workspace.py` pass or provide an expected no-CMake message? |
|
||||
|
||||
## Output Format
|
||||
|
||||
If there are findings, list them first in severity order with file and line references when possible. Then include this table:
|
||||
|
||||
| 항목 | 결과 | 비고 |
|
||||
| --- | --- | --- |
|
||||
| 아키텍처 준수 | PASS/FAIL | {상세} |
|
||||
| 기술 스택 준수 | PASS/FAIL | {상세} |
|
||||
| 테스트 존재 | PASS/FAIL | {상세} |
|
||||
| TDD Guard | PASS/FAIL | {상세} |
|
||||
| CRITICAL 규칙 | PASS/FAIL | {상세} |
|
||||
| 빌드/검증 가능 | PASS/FAIL | {상세} |
|
||||
|
||||
When there are no findings, say that clearly, then mention any commands not run or remaining risk.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Harness Review"
|
||||
short_description: "Review Harness changes safely"
|
||||
default_prompt: "Use $harness-review to review Harness repository changes."
|
||||
@@ -1,130 +0,0 @@
|
||||
---
|
||||
name: harness-workflow
|
||||
description: Use when planning or running this C++/MSVC Harness framework: reading AGENTS.md and docs/*.md, discussing implementation scope, creating or updating phases/index.json, phases/{task}/index.json, phases/{task}/stepN.md, or invoking scripts/execute.py for staged Codex execution.
|
||||
---
|
||||
|
||||
# Harness Workflow
|
||||
|
||||
## Overview
|
||||
|
||||
Use this skill to turn a user-approved task into small, self-contained Harness steps that another Codex session can execute reliably. Keep every step grounded in repository docs, C++/MSVC constraints, TDD, and executable acceptance criteria.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Read `AGENTS.md` and relevant files under `docs/`, especially `docs/PRD.md`, `docs/ARCHITECTURE.md`, and `docs/ADR.md`.
|
||||
2. Discuss unresolved product or technical decisions with the user before writing phase files.
|
||||
3. When the user asks for an implementation plan, draft steps and get approval before creating files.
|
||||
4. Create or update `phases/index.json`, `phases/{task-name}/index.json`, and one `phases/{task-name}/stepN.md` per step.
|
||||
5. Run the phase with `python scripts/execute.py {task-name}` when asked to execute it. Use `--push` only when the user asks to push.
|
||||
|
||||
## Step Design Rules
|
||||
|
||||
- Scope each step to one layer or module. Split steps when multiple modules would otherwise change together.
|
||||
- Make every step self-contained. Do not rely on prior conversation; include all required context and file paths.
|
||||
- Force context gathering. Each step must tell Codex which docs and previous outputs to read before editing.
|
||||
- Specify interfaces and signatures, not full implementations, unless exact code is required for a constraint.
|
||||
- Put core invariants directly in the step: idempotency, numerical conventions, data integrity, API contracts, or other non-negotiables.
|
||||
- Use executable acceptance criteria such as `python scripts/validate_workspace.py`, not abstract statements.
|
||||
- For C++ behavior changes, require tests first and name the expected test file or test executable.
|
||||
- Name steps with kebab-case slugs such as `project-setup`, `core-types`, or `solver-validation`.
|
||||
|
||||
## Phase Files
|
||||
|
||||
Create or update `phases/index.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"phases": [
|
||||
{
|
||||
"dir": "0-mvp",
|
||||
"status": "pending"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Create `phases/{task-name}/index.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"project": "FESA Harness",
|
||||
"phase": "<task-name>",
|
||||
"steps": [
|
||||
{ "step": 0, "name": "project-setup", "status": "pending", "allowed_paths": ["CMakeLists.txt", "tests/"] },
|
||||
{ "step": 1, "name": "core-types", "status": "pending", "allowed_paths": ["src/fesa/core/", "tests/unit/"] },
|
||||
{ "step": 2, "name": "validation-path", "status": "pending", "allowed_paths": ["scripts/", "docs/"] }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `project` comes from `AGENTS.md`.
|
||||
- `phase` matches the task directory name.
|
||||
- `steps[].step` starts at `0`.
|
||||
- Initial status is always `pending`.
|
||||
- Each step must declare non-empty `allowed_paths` using repository-relative paths, directory prefixes, or glob patterns.
|
||||
- Do not add timestamps when creating files. `scripts/execute.py` records `created_at`, `started_at`, `completed_at`, `failed_at`, and `blocked_at`.
|
||||
|
||||
## Step Template
|
||||
|
||||
```markdown
|
||||
# Step {N}: {name}
|
||||
|
||||
## 읽어야 할 파일
|
||||
|
||||
먼저 아래 파일들을 읽고 프로젝트의 아키텍처와 설계 의도를 파악하라:
|
||||
|
||||
- `/AGENTS.md`
|
||||
- `/docs/ARCHITECTURE.md`
|
||||
- `/docs/ADR.md`
|
||||
- {previously created or modified files}
|
||||
|
||||
이전 step에서 만들어진 코드를 꼼꼼히 읽고, 설계 의도를 이해한 뒤 작업하라.
|
||||
|
||||
## 작업
|
||||
|
||||
{Concrete instructions with file paths, interfaces, signatures, and rules.}
|
||||
|
||||
## Tests To Write First
|
||||
|
||||
- {Exact C++ or Python test file and behavior to add before implementation.}
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
```bash
|
||||
python -m unittest discover -s scripts -p "test_*.py"
|
||||
python scripts/validate_workspace.py
|
||||
```
|
||||
|
||||
## 검증 절차
|
||||
|
||||
1. 위 AC 커맨드를 실행한다.
|
||||
2. 아키텍처 체크리스트를 확인한다:
|
||||
- ARCHITECTURE.md 디렉토리 구조를 따르는가?
|
||||
- ADR 기술 스택을 벗어나지 않았는가?
|
||||
- AGENTS.md CRITICAL 규칙을 위반하지 않았는가?
|
||||
- C++ 변경에는 관련 테스트가 존재하는가?
|
||||
3. 결과에 따라 `phases/{task-name}/index.json`의 해당 step을 업데이트한다:
|
||||
- 성공: `"status": "completed"`, `"summary": "산출물 한 줄 요약"`
|
||||
- 3회 수정 시도 후 실패: `"status": "error"`, `"error_message": "구체적 에러 내용"`
|
||||
- 사용자 개입 필요: `"status": "blocked"`, `"blocked_reason": "구체적 사유"` 후 중단
|
||||
|
||||
## 금지사항
|
||||
|
||||
- JavaScript/TypeScript/npm fallback을 추가하지 마라. Reason: 이 Harness는 C++/MSVC 전용이다.
|
||||
- 기존 테스트를 깨뜨리지 마라.
|
||||
```
|
||||
|
||||
## Execution And Recovery
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python scripts/execute.py {task-name}
|
||||
python scripts/execute.py {task-name} --push
|
||||
```
|
||||
|
||||
`scripts/execute.py` creates or checks out `codex/{task-name}`, refuses dirty worktrees, requires per-step `allowed_paths`, stages only explicit allowed paths and runner housekeeping files, validates before every runner-created commit, injects `AGENTS.md` and `docs/*.md` into each prompt, carries completed step summaries forward, retries failed steps up to three times, and records timestamps.
|
||||
|
||||
If a step is `error`, set it back to `pending` and remove `error_message` after fixing the cause. If a step is `blocked`, resolve `blocked_reason`, set it back to `pending`, remove `blocked_reason`, and rerun.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Harness Workflow"
|
||||
short_description: "Plan staged Harness workflow steps"
|
||||
default_prompt: "Use $harness-workflow to plan Harness phases and step files."
|
||||
Reference in New Issue
Block a user