add harness framework
This commit is contained in:
KOKO\Mimi
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: auditing-agents-md
|
||||
description: Use when a user asks to audit, check, review, improve, update, or fix AGENTS.md or AGENTS.override.md files, or asks whether Codex repository guidance is accurate and current.
|
||||
---
|
||||
|
||||
# Audit AGENTS.md
|
||||
|
||||
## Overview
|
||||
|
||||
Audit Codex guidance against the repository that it governs. Treat instruction
|
||||
precedence, factual accuracy, scope, and conciseness as observable properties:
|
||||
report evidence before proposing a change, and propose before editing.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Resolve scope.** Identify the project root, current working directory, and
|
||||
requested audit scope. Resolve Codex home from `CODEX_HOME`, falling back to
|
||||
`~/.codex`; inspect global guidance only when it is in scope.
|
||||
2. **Discover guidance.** Find `AGENTS.override.md` and `AGENTS.md` while
|
||||
excluding VCS metadata, dependencies, caches, and generated output. At each
|
||||
directory, an override is selected before the standard file. Map the active
|
||||
chain from project root to the working directory and label files that govern
|
||||
only other subtrees.
|
||||
3. **Inspect evidence.** Read each selected file completely. Check commands,
|
||||
paths, architecture claims, and conventions against manifests, source,
|
||||
tests, config, and version control. Run only safe, relevant verification;
|
||||
do not deploy, mutate external state, or expose credentials to score a file.
|
||||
4. **Score every file.** Read
|
||||
[quality-criteria.md](references/quality-criteria.md). Use its six weighted
|
||||
criteria, 100-point total, and grade bands. Check conflicts, duplication,
|
||||
misplaced scope, override behavior, and the combined 32 KiB default
|
||||
instruction budget.
|
||||
5. **Report before changing.** Emit the output contract below even when the
|
||||
guidance is already strong.
|
||||
6. **Propose targeted diffs.** Read
|
||||
[update-guidelines.md](references/update-guidelines.md). Include only claims
|
||||
supported by repository evidence. Use
|
||||
[templates.md](references/templates.md) only when a missing structure is
|
||||
genuinely useful.
|
||||
7. **Request approval.** Stop after the report and proposed diffs. Ask which
|
||||
changes the user wants applied.
|
||||
8. **Apply and verify.** Edit only approved files, preserve their organization,
|
||||
and re-check commands, paths, conflicts, active-chain order, and byte budget.
|
||||
|
||||
## Required Report
|
||||
|
||||
```markdown
|
||||
## AGENTS.md Quality Report
|
||||
|
||||
### Summary
|
||||
- Files found: N
|
||||
- Active chain: path -> path
|
||||
- Average score: N/100
|
||||
- Files needing update: N
|
||||
- Combined active-chain size: N bytes (default limit: 32 KiB)
|
||||
|
||||
### File-by-File Assessment
|
||||
#### 1. path (scope; active/inactive)
|
||||
**Score: N/100 (Grade: A-F)**
|
||||
|
||||
| Criterion | Score | Evidence |
|
||||
| --- | ---: | --- |
|
||||
| Commands and workflows | N/20 | ... |
|
||||
| Repository map and instruction scope | N/20 | ... |
|
||||
| Non-obvious constraints and gotchas | N/15 | ... |
|
||||
| Conciseness and duplication | N/15 | ... |
|
||||
| Currency against the codebase | N/15 | ... |
|
||||
| Actionability and verifiability | N/15 | ... |
|
||||
|
||||
**Issues**
|
||||
- ...
|
||||
|
||||
**Recommended changes**
|
||||
- ...
|
||||
```
|
||||
|
||||
After all assessments, show each proposed diff under `### Update: <path>` with
|
||||
a one-line `Why:`. End with an explicit approval question. Do not edit in the
|
||||
same turn unless the user already approved the exact diff.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Question | Evidence |
|
||||
| --- | --- |
|
||||
| Which file applies? | Global selection, then one selected file per directory from root to working directory |
|
||||
| Is a command current? | Manifest plus a safe command run when practical |
|
||||
| Is a path accurate? | Filesystem and repository search |
|
||||
| Is guidance misplaced? | Narrowest subtree where the rule is true |
|
||||
| Is the chain too large? | Combined selected-file bytes versus configured/default limit |
|
||||
| Should content be added? | Project-specific, durable, non-obvious, and actionable |
|
||||
|
||||
## Example
|
||||
|
||||
If `AGENTS.md` requires `npm test` but `package.json` contains only
|
||||
`test:unit`, score currency and actionability down, cite `package.json`, and
|
||||
propose the verified command. Do not invent an aggregate script or silently
|
||||
edit the file.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
- Scoring the repository once instead of scoring every guidance file.
|
||||
- Treating a nested override as deleting unrelated root guidance.
|
||||
- Claiming a file is an entry point because its name looks plausible.
|
||||
- Recommending generic best practices or facts obvious from code.
|
||||
- Running destructive, deployment, or credential-sensitive commands to audit.
|
||||
- Applying recommendations before the user approves the displayed diff.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "Audit AGENTS.md"
|
||||
short_description: "Audit Codex repository guidance for accuracy"
|
||||
default_prompt: "Use $auditing-agents-md to audit this repository guidance."
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
# AGENTS.md Quality Criteria
|
||||
|
||||
## Assessment Order
|
||||
|
||||
1. Read each selected instruction file completely.
|
||||
2. Map its scope and place in the active chain.
|
||||
3. Cross-reference every factual claim with repository evidence.
|
||||
4. Score all six criteria independently.
|
||||
5. Add the scores, assign a grade, and list evidence-backed issues.
|
||||
|
||||
## Scoring Rubric
|
||||
|
||||
### Commands and Workflows - 20 points
|
||||
|
||||
- **20:** Essential setup, build, test, lint, and handoff commands are current,
|
||||
scoped, and copy-paste ready.
|
||||
- **15:** Most essential commands are accurate; minor context is missing.
|
||||
- **10:** Only basic commands are present or important workflow context is
|
||||
absent.
|
||||
- **5:** Several commands are missing or stale.
|
||||
- **0:** No usable commands are documented.
|
||||
|
||||
### Repository Map and Instruction Scope - 20 points
|
||||
|
||||
- **20:** Important directories and relationships are accurate; root, nested,
|
||||
and override rules are placed at the narrowest correct scope.
|
||||
- **15:** The map and scope are useful with minor omissions.
|
||||
- **10:** A directory list exists but relationships or scope are unclear.
|
||||
- **5:** Paths or precedence are substantially misleading.
|
||||
- **0:** No usable map exists or the file governs the wrong content.
|
||||
|
||||
### Non-Obvious Constraints and Gotchas - 15 points
|
||||
|
||||
- **15:** Recurring quirks, prerequisites, ordering constraints, and reliable
|
||||
workarounds are concise and evidenced.
|
||||
- **10:** Some important non-obvious constraints are documented.
|
||||
- **5:** Only minimal or weakly supported constraints are present.
|
||||
- **0:** Important recurring gotchas are absent.
|
||||
|
||||
### Conciseness and Duplication - 15 points
|
||||
|
||||
- **15:** Every line earns prompt space; no filler, obvious facts, or duplicate
|
||||
rules exist across the active chain.
|
||||
- **10:** Mostly concise with limited repetition.
|
||||
- **5:** Noticeable verbosity, duplication, or misplaced detail.
|
||||
- **0:** Mostly filler or repeated content.
|
||||
|
||||
Also report the combined selected-file byte count. The default
|
||||
`project_doc_max_bytes` is 32 KiB; use the configured value when repository or
|
||||
user config proves a different limit.
|
||||
|
||||
### Currency Against the Codebase - 15 points
|
||||
|
||||
- **15:** Commands, paths, versions, architecture, and tool names match current
|
||||
repository evidence.
|
||||
- **10:** Mostly current with minor staleness.
|
||||
- **5:** Several claims are outdated or unverified.
|
||||
- **0:** Guidance is severely stale or misleading.
|
||||
|
||||
### Actionability and Verifiability - 15 points
|
||||
|
||||
- **15:** Instructions are concrete, safely verifiable, and include clear
|
||||
completion expectations.
|
||||
- **10:** Mostly actionable with small ambiguities.
|
||||
- **5:** Several vague or untestable rules remain.
|
||||
- **0:** Guidance is theoretical or unusable.
|
||||
|
||||
## Grades
|
||||
|
||||
| Score | Grade | Meaning |
|
||||
| ---: | :---: | --- |
|
||||
| 90-100 | A | Comprehensive, current, concise, and actionable |
|
||||
| 70-89 | B | Good guidance with minor gaps |
|
||||
| 50-69 | C | Useful basics with important omissions |
|
||||
| 30-49 | D | Sparse, stale, conflicting, or poorly scoped |
|
||||
| 0-29 | F | Missing or severely misleading |
|
||||
|
||||
## Red Flags
|
||||
|
||||
- A command missing from the relevant manifest or task runner.
|
||||
- A reference to a deleted or moved path.
|
||||
- Root and nested rules that conflict without an intentional override.
|
||||
- The same rule repeated across the active chain.
|
||||
- A broad file containing rules true only for one subtree.
|
||||
- An override used when additive nested guidance was intended.
|
||||
- Combined guidance near or beyond the configured byte limit.
|
||||
- Templates copied without real paths, commands, or repository details.
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
# AGENTS.md Patterns
|
||||
|
||||
Use only sections supported by repository evidence. Prefer a short accurate
|
||||
file over a comprehensive-looking template.
|
||||
|
||||
## Repository Root
|
||||
|
||||
```markdown
|
||||
# Repository guidance
|
||||
|
||||
## Commands
|
||||
|
||||
- `npm ci` - Install the locked dependency set.
|
||||
- `npm run test:unit` - Run unit tests.
|
||||
- `npm run lint` - Check formatting and static rules.
|
||||
|
||||
## Architecture
|
||||
|
||||
- `src/server.ts` - HTTP entry point.
|
||||
- `packages/payments/` - Payment domain package.
|
||||
|
||||
## Verification
|
||||
|
||||
- Run the smallest relevant test during iteration.
|
||||
- Before handoff, run `npm run lint` and `npm run test:unit`.
|
||||
```
|
||||
|
||||
## Nested Additive Guidance
|
||||
|
||||
Use nested `AGENTS.md` when local rules add to broader guidance.
|
||||
|
||||
```markdown
|
||||
# Payments package guidance
|
||||
|
||||
## Commands
|
||||
|
||||
- `npm run test:payments` - Run payment-domain tests from the repository root.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Payment fixtures use UTC timestamps; do not compare local-time strings.
|
||||
```
|
||||
|
||||
## Nested Override
|
||||
|
||||
Use `AGENTS.override.md` only when same-directory standard guidance must be
|
||||
replaced. Broader parent guidance still remains earlier in the chain.
|
||||
|
||||
```markdown
|
||||
# Generated client override
|
||||
|
||||
## Editing
|
||||
|
||||
- Do not hand-edit files under `src/generated/`.
|
||||
- Regenerate them with `npm run generate:client`.
|
||||
|
||||
## Verification
|
||||
|
||||
- Run `npm run check:generated`.
|
||||
```
|
||||
|
||||
## Scope Rules
|
||||
|
||||
- Put team-wide repository rules at the root.
|
||||
- Put subtree-only commands and constraints in the closest nested directory.
|
||||
- Put personal cross-repository preferences in Codex-home guidance only when
|
||||
the user explicitly requests global scope.
|
||||
- Avoid overrides for ordinary additive instructions.
|
||||
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
# AGENTS.md Update Guidelines
|
||||
|
||||
## Evidence Contract
|
||||
|
||||
Every proposed line must be:
|
||||
|
||||
- specific to the repository or explicitly requested global scope;
|
||||
- supported by a file, manifest, safe command result, or user-confirmed fact;
|
||||
- durable enough to help future sessions;
|
||||
- placed at the narrowest scope where it is true;
|
||||
- expressed in the shortest actionable form that preserves meaning.
|
||||
|
||||
Never add secrets, credential values, personal data, unverified claims,
|
||||
session narrative, generic advice, facts obvious from names, or one-off fixes.
|
||||
|
||||
## Safe Verification
|
||||
|
||||
Prefer read-only inspection. Run a command only when it is relevant, safe, and
|
||||
unlikely to change external state. Do not deploy, publish, migrate data, write
|
||||
credentials, or invoke destructive cleanup merely to validate guidance.
|
||||
|
||||
If a command cannot safely be run, label it unverified rather than treating it
|
||||
as current.
|
||||
|
||||
## Precedence and Size
|
||||
|
||||
- Select `AGENTS.override.md` before `AGENTS.md` at each directory.
|
||||
- Build the active project chain from root to current working directory.
|
||||
- Later, closer guidance wins on conflict.
|
||||
- Parent guidance remains active unless a later instruction contradicts it.
|
||||
- Flag duplicate or conflicting rules across selected files.
|
||||
- Compare combined selected-file bytes with configured
|
||||
`project_doc_max_bytes`, using 32 KiB when no override is evidenced.
|
||||
|
||||
## Diff Contract
|
||||
|
||||
For every proposed change:
|
||||
|
||||
```markdown
|
||||
### Update: path/to/AGENTS.md
|
||||
|
||||
**Why:** `package.json` defines `test:unit`, while the documented `npm test`
|
||||
command does not exist.
|
||||
|
||||
```diff
|
||||
- Run `npm test` before handoff.
|
||||
+ Run `npm run test:unit` before handoff.
|
||||
```
|
||||
```
|
||||
|
||||
Group diffs by destination. After all diffs, ask which changes the user wants
|
||||
applied. Do not edit before that approval.
|
||||
|
||||
## Post-Edit Validation
|
||||
|
||||
- Re-read every changed file.
|
||||
- Rebuild the active chain for the affected subtree.
|
||||
- Check for conflicts and duplication.
|
||||
- Verify changed commands and paths.
|
||||
- Recalculate the combined byte count.
|
||||
- Report exactly what was changed and what could not be verified.
|
||||
|
||||
Reference in New Issue
Block a user