add harness framework
This commit is contained in:
KOKO\Mimi
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: auditing-agents-md
|
||||
description: Use when a user asks to audit, check, review, improve, update, or fix AGENTS.md or AGENTS.override.md files, or asks whether Codex repository guidance is accurate and current.
|
||||
---
|
||||
|
||||
# Audit AGENTS.md
|
||||
|
||||
## Overview
|
||||
|
||||
Audit Codex guidance against the repository that it governs. Treat instruction
|
||||
precedence, factual accuracy, scope, and conciseness as observable properties:
|
||||
report evidence before proposing a change, and propose before editing.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Resolve scope.** Identify the project root, current working directory, and
|
||||
requested audit scope. Resolve Codex home from `CODEX_HOME`, falling back to
|
||||
`~/.codex`; inspect global guidance only when it is in scope.
|
||||
2. **Discover guidance.** Find `AGENTS.override.md` and `AGENTS.md` while
|
||||
excluding VCS metadata, dependencies, caches, and generated output. At each
|
||||
directory, an override is selected before the standard file. Map the active
|
||||
chain from project root to the working directory and label files that govern
|
||||
only other subtrees.
|
||||
3. **Inspect evidence.** Read each selected file completely. Check commands,
|
||||
paths, architecture claims, and conventions against manifests, source,
|
||||
tests, config, and version control. Run only safe, relevant verification;
|
||||
do not deploy, mutate external state, or expose credentials to score a file.
|
||||
4. **Score every file.** Read
|
||||
[quality-criteria.md](references/quality-criteria.md). Use its six weighted
|
||||
criteria, 100-point total, and grade bands. Check conflicts, duplication,
|
||||
misplaced scope, override behavior, and the combined 32 KiB default
|
||||
instruction budget.
|
||||
5. **Report before changing.** Emit the output contract below even when the
|
||||
guidance is already strong.
|
||||
6. **Propose targeted diffs.** Read
|
||||
[update-guidelines.md](references/update-guidelines.md). Include only claims
|
||||
supported by repository evidence. Use
|
||||
[templates.md](references/templates.md) only when a missing structure is
|
||||
genuinely useful.
|
||||
7. **Request approval.** Stop after the report and proposed diffs. Ask which
|
||||
changes the user wants applied.
|
||||
8. **Apply and verify.** Edit only approved files, preserve their organization,
|
||||
and re-check commands, paths, conflicts, active-chain order, and byte budget.
|
||||
|
||||
## Required Report
|
||||
|
||||
```markdown
|
||||
## AGENTS.md Quality Report
|
||||
|
||||
### Summary
|
||||
- Files found: N
|
||||
- Active chain: path -> path
|
||||
- Average score: N/100
|
||||
- Files needing update: N
|
||||
- Combined active-chain size: N bytes (default limit: 32 KiB)
|
||||
|
||||
### File-by-File Assessment
|
||||
#### 1. path (scope; active/inactive)
|
||||
**Score: N/100 (Grade: A-F)**
|
||||
|
||||
| Criterion | Score | Evidence |
|
||||
| --- | ---: | --- |
|
||||
| Commands and workflows | N/20 | ... |
|
||||
| Repository map and instruction scope | N/20 | ... |
|
||||
| Non-obvious constraints and gotchas | N/15 | ... |
|
||||
| Conciseness and duplication | N/15 | ... |
|
||||
| Currency against the codebase | N/15 | ... |
|
||||
| Actionability and verifiability | N/15 | ... |
|
||||
|
||||
**Issues**
|
||||
- ...
|
||||
|
||||
**Recommended changes**
|
||||
- ...
|
||||
```
|
||||
|
||||
After all assessments, show each proposed diff under `### Update: <path>` with
|
||||
a one-line `Why:`. End with an explicit approval question. Do not edit in the
|
||||
same turn unless the user already approved the exact diff.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Question | Evidence |
|
||||
| --- | --- |
|
||||
| Which file applies? | Global selection, then one selected file per directory from root to working directory |
|
||||
| Is a command current? | Manifest plus a safe command run when practical |
|
||||
| Is a path accurate? | Filesystem and repository search |
|
||||
| Is guidance misplaced? | Narrowest subtree where the rule is true |
|
||||
| Is the chain too large? | Combined selected-file bytes versus configured/default limit |
|
||||
| Should content be added? | Project-specific, durable, non-obvious, and actionable |
|
||||
|
||||
## Example
|
||||
|
||||
If `AGENTS.md` requires `npm test` but `package.json` contains only
|
||||
`test:unit`, score currency and actionability down, cite `package.json`, and
|
||||
propose the verified command. Do not invent an aggregate script or silently
|
||||
edit the file.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
- Scoring the repository once instead of scoring every guidance file.
|
||||
- Treating a nested override as deleting unrelated root guidance.
|
||||
- Claiming a file is an entry point because its name looks plausible.
|
||||
- Recommending generic best practices or facts obvious from code.
|
||||
- Running destructive, deployment, or credential-sensitive commands to audit.
|
||||
- Applying recommendations before the user approves the displayed diff.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "Audit AGENTS.md"
|
||||
short_description: "Audit Codex repository guidance for accuracy"
|
||||
default_prompt: "Use $auditing-agents-md to audit this repository guidance."
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
# AGENTS.md Quality Criteria
|
||||
|
||||
## Assessment Order
|
||||
|
||||
1. Read each selected instruction file completely.
|
||||
2. Map its scope and place in the active chain.
|
||||
3. Cross-reference every factual claim with repository evidence.
|
||||
4. Score all six criteria independently.
|
||||
5. Add the scores, assign a grade, and list evidence-backed issues.
|
||||
|
||||
## Scoring Rubric
|
||||
|
||||
### Commands and Workflows - 20 points
|
||||
|
||||
- **20:** Essential setup, build, test, lint, and handoff commands are current,
|
||||
scoped, and copy-paste ready.
|
||||
- **15:** Most essential commands are accurate; minor context is missing.
|
||||
- **10:** Only basic commands are present or important workflow context is
|
||||
absent.
|
||||
- **5:** Several commands are missing or stale.
|
||||
- **0:** No usable commands are documented.
|
||||
|
||||
### Repository Map and Instruction Scope - 20 points
|
||||
|
||||
- **20:** Important directories and relationships are accurate; root, nested,
|
||||
and override rules are placed at the narrowest correct scope.
|
||||
- **15:** The map and scope are useful with minor omissions.
|
||||
- **10:** A directory list exists but relationships or scope are unclear.
|
||||
- **5:** Paths or precedence are substantially misleading.
|
||||
- **0:** No usable map exists or the file governs the wrong content.
|
||||
|
||||
### Non-Obvious Constraints and Gotchas - 15 points
|
||||
|
||||
- **15:** Recurring quirks, prerequisites, ordering constraints, and reliable
|
||||
workarounds are concise and evidenced.
|
||||
- **10:** Some important non-obvious constraints are documented.
|
||||
- **5:** Only minimal or weakly supported constraints are present.
|
||||
- **0:** Important recurring gotchas are absent.
|
||||
|
||||
### Conciseness and Duplication - 15 points
|
||||
|
||||
- **15:** Every line earns prompt space; no filler, obvious facts, or duplicate
|
||||
rules exist across the active chain.
|
||||
- **10:** Mostly concise with limited repetition.
|
||||
- **5:** Noticeable verbosity, duplication, or misplaced detail.
|
||||
- **0:** Mostly filler or repeated content.
|
||||
|
||||
Also report the combined selected-file byte count. The default
|
||||
`project_doc_max_bytes` is 32 KiB; use the configured value when repository or
|
||||
user config proves a different limit.
|
||||
|
||||
### Currency Against the Codebase - 15 points
|
||||
|
||||
- **15:** Commands, paths, versions, architecture, and tool names match current
|
||||
repository evidence.
|
||||
- **10:** Mostly current with minor staleness.
|
||||
- **5:** Several claims are outdated or unverified.
|
||||
- **0:** Guidance is severely stale or misleading.
|
||||
|
||||
### Actionability and Verifiability - 15 points
|
||||
|
||||
- **15:** Instructions are concrete, safely verifiable, and include clear
|
||||
completion expectations.
|
||||
- **10:** Mostly actionable with small ambiguities.
|
||||
- **5:** Several vague or untestable rules remain.
|
||||
- **0:** Guidance is theoretical or unusable.
|
||||
|
||||
## Grades
|
||||
|
||||
| Score | Grade | Meaning |
|
||||
| ---: | :---: | --- |
|
||||
| 90-100 | A | Comprehensive, current, concise, and actionable |
|
||||
| 70-89 | B | Good guidance with minor gaps |
|
||||
| 50-69 | C | Useful basics with important omissions |
|
||||
| 30-49 | D | Sparse, stale, conflicting, or poorly scoped |
|
||||
| 0-29 | F | Missing or severely misleading |
|
||||
|
||||
## Red Flags
|
||||
|
||||
- A command missing from the relevant manifest or task runner.
|
||||
- A reference to a deleted or moved path.
|
||||
- Root and nested rules that conflict without an intentional override.
|
||||
- The same rule repeated across the active chain.
|
||||
- A broad file containing rules true only for one subtree.
|
||||
- An override used when additive nested guidance was intended.
|
||||
- Combined guidance near or beyond the configured byte limit.
|
||||
- Templates copied without real paths, commands, or repository details.
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
# AGENTS.md Patterns
|
||||
|
||||
Use only sections supported by repository evidence. Prefer a short accurate
|
||||
file over a comprehensive-looking template.
|
||||
|
||||
## Repository Root
|
||||
|
||||
```markdown
|
||||
# Repository guidance
|
||||
|
||||
## Commands
|
||||
|
||||
- `npm ci` - Install the locked dependency set.
|
||||
- `npm run test:unit` - Run unit tests.
|
||||
- `npm run lint` - Check formatting and static rules.
|
||||
|
||||
## Architecture
|
||||
|
||||
- `src/server.ts` - HTTP entry point.
|
||||
- `packages/payments/` - Payment domain package.
|
||||
|
||||
## Verification
|
||||
|
||||
- Run the smallest relevant test during iteration.
|
||||
- Before handoff, run `npm run lint` and `npm run test:unit`.
|
||||
```
|
||||
|
||||
## Nested Additive Guidance
|
||||
|
||||
Use nested `AGENTS.md` when local rules add to broader guidance.
|
||||
|
||||
```markdown
|
||||
# Payments package guidance
|
||||
|
||||
## Commands
|
||||
|
||||
- `npm run test:payments` - Run payment-domain tests from the repository root.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Payment fixtures use UTC timestamps; do not compare local-time strings.
|
||||
```
|
||||
|
||||
## Nested Override
|
||||
|
||||
Use `AGENTS.override.md` only when same-directory standard guidance must be
|
||||
replaced. Broader parent guidance still remains earlier in the chain.
|
||||
|
||||
```markdown
|
||||
# Generated client override
|
||||
|
||||
## Editing
|
||||
|
||||
- Do not hand-edit files under `src/generated/`.
|
||||
- Regenerate them with `npm run generate:client`.
|
||||
|
||||
## Verification
|
||||
|
||||
- Run `npm run check:generated`.
|
||||
```
|
||||
|
||||
## Scope Rules
|
||||
|
||||
- Put team-wide repository rules at the root.
|
||||
- Put subtree-only commands and constraints in the closest nested directory.
|
||||
- Put personal cross-repository preferences in Codex-home guidance only when
|
||||
the user explicitly requests global scope.
|
||||
- Avoid overrides for ordinary additive instructions.
|
||||
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
# AGENTS.md Update Guidelines
|
||||
|
||||
## Evidence Contract
|
||||
|
||||
Every proposed line must be:
|
||||
|
||||
- specific to the repository or explicitly requested global scope;
|
||||
- supported by a file, manifest, safe command result, or user-confirmed fact;
|
||||
- durable enough to help future sessions;
|
||||
- placed at the narrowest scope where it is true;
|
||||
- expressed in the shortest actionable form that preserves meaning.
|
||||
|
||||
Never add secrets, credential values, personal data, unverified claims,
|
||||
session narrative, generic advice, facts obvious from names, or one-off fixes.
|
||||
|
||||
## Safe Verification
|
||||
|
||||
Prefer read-only inspection. Run a command only when it is relevant, safe, and
|
||||
unlikely to change external state. Do not deploy, publish, migrate data, write
|
||||
credentials, or invoke destructive cleanup merely to validate guidance.
|
||||
|
||||
If a command cannot safely be run, label it unverified rather than treating it
|
||||
as current.
|
||||
|
||||
## Precedence and Size
|
||||
|
||||
- Select `AGENTS.override.md` before `AGENTS.md` at each directory.
|
||||
- Build the active project chain from root to current working directory.
|
||||
- Later, closer guidance wins on conflict.
|
||||
- Parent guidance remains active unless a later instruction contradicts it.
|
||||
- Flag duplicate or conflicting rules across selected files.
|
||||
- Compare combined selected-file bytes with configured
|
||||
`project_doc_max_bytes`, using 32 KiB when no override is evidenced.
|
||||
|
||||
## Diff Contract
|
||||
|
||||
For every proposed change:
|
||||
|
||||
```markdown
|
||||
### Update: path/to/AGENTS.md
|
||||
|
||||
**Why:** `package.json` defines `test:unit`, while the documented `npm test`
|
||||
command does not exist.
|
||||
|
||||
```diff
|
||||
- Run `npm test` before handoff.
|
||||
+ Run `npm run test:unit` before handoff.
|
||||
```
|
||||
```
|
||||
|
||||
Group diffs by destination. After all diffs, ask which changes the user wants
|
||||
applied. Do not edit before that approval.
|
||||
|
||||
## Post-Edit Validation
|
||||
|
||||
- Re-read every changed file.
|
||||
- Rebuild the active chain for the affected subtree.
|
||||
- Check for conflicts and duplication.
|
||||
- Verify changed commands and paths.
|
||||
- Recalculate the combined byte count.
|
||||
- Report exactly what was changed and what could not be verified.
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
name: revising-agents-md
|
||||
description: Use when a user explicitly asks to capture durable learnings from the current session in AGENTS.md after discovering reusable commands, conventions, validation steps, architecture facts, environment constraints, or recurring gotchas.
|
||||
---
|
||||
|
||||
# Revise AGENTS.md
|
||||
|
||||
## Overview
|
||||
|
||||
Turn verified session discoveries into concise Codex guidance for future work.
|
||||
Keep only durable information, place it at the narrowest scope where it
|
||||
governs work, show the proposed diff, and obtain approval before editing.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Reflect.** Extract candidate commands, conventions, validation steps,
|
||||
architecture relationships, environment constraints, and recurring
|
||||
gotchas from the current session.
|
||||
2. **Inspect evidence.** Read the relevant repository files and existing
|
||||
`AGENTS.md` or `AGENTS.override.md` chain. Check Git tracking and ignore
|
||||
state when deciding whether guidance is shared or personal. Treat a claim
|
||||
as verified only when supported by a safe command result, repository file,
|
||||
or explicit user confirmation.
|
||||
3. **Classify every candidate.** Record `candidate`, `keep/exclude`,
|
||||
`evidence`, `destination`, and `reason`. Exclude content already documented
|
||||
accurately.
|
||||
4. **Filter aggressively.** Exclude secrets, credentials, personal data,
|
||||
unverified claims, generic advice, facts obvious from code, one-off fixes,
|
||||
temporary state, and session narrative. Never reproduce a sensitive value,
|
||||
even in an exclusion explanation.
|
||||
5. **Choose the narrowest durable scope.**
|
||||
- Codex-home `AGENTS.md`: cross-repository personal preferences, only when
|
||||
the user explicitly requests global scope.
|
||||
- Repository-root `AGENTS.md`: guidance true for work across the repository.
|
||||
- Nested `AGENTS.md`: guidance true only for one package, service, feature,
|
||||
or subtree.
|
||||
- `AGENTS.override.md`: only when the user explicitly requests replacement
|
||||
semantics at that directory.
|
||||
|
||||
Scope follows the code or work governed by the instruction, not the
|
||||
directory from which its command must run. A payment-only test command
|
||||
belongs in payments guidance even when it says to run from repository root.
|
||||
6. **Draft grouped diffs.** For each destination, show a one-line reason and
|
||||
the exact minimal diff. Prefer one line per concept and preserve the
|
||||
existing file's structure.
|
||||
7. **Request approval.** End with an explicit question asking which proposed
|
||||
changes the user wants applied. Do not create or edit a file before the
|
||||
user approves its exact destination and content.
|
||||
8. **Apply and verify.** Edit only approved destinations. Re-read the resulting
|
||||
instruction chain, check conflicts and duplication, and verify changed
|
||||
commands and paths.
|
||||
|
||||
## Required Proposal
|
||||
|
||||
```markdown
|
||||
## Session Learning Review
|
||||
|
||||
| Candidate | Decision | Evidence | Destination | Reason |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| ... | Keep/Exclude | ... | path or none | ... |
|
||||
|
||||
### Update: path/to/AGENTS.md
|
||||
**Why:** one durable, evidence-backed reason
|
||||
|
||||
```diff
|
||||
+ concise instruction
|
||||
```
|
||||
|
||||
No files were edited. Which proposed changes should I apply?
|
||||
```
|
||||
|
||||
When no candidate qualifies, say so and do not propose an empty file.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Candidate | Decision |
|
||||
| --- | --- |
|
||||
| Verified recurring command | Keep at the scope whose work uses it |
|
||||
| Package-only gotcha | Keep in that package's nested `AGENTS.md` |
|
||||
| Personal preference | Global only after explicit global request |
|
||||
| Existing accurate instruction | Exclude as duplicate |
|
||||
| One-off fix or session history | Exclude |
|
||||
| Generic best practice | Exclude |
|
||||
| Unverified claim | Exclude or verify safely first |
|
||||
| Secret or personal data | Exclude without reproducing the value |
|
||||
|
||||
## Example
|
||||
|
||||
A session verifies `npm run test:payments` and proves it must run from the
|
||||
repository root. If only payment work needs that command, propose this in
|
||||
`services/payments/AGENTS.md`:
|
||||
|
||||
```diff
|
||||
+- Run `npm run test:payments` from the repository root.
|
||||
```
|
||||
|
||||
Do not place it at repository root merely because that is the execution
|
||||
directory.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
- Saving everything because the session was difficult or time-consuming.
|
||||
- Confusing command execution location with instruction scope.
|
||||
- Creating an override for ordinary additive guidance.
|
||||
- Duplicating an instruction that is already accurate.
|
||||
- Persisting unverified deployment or environment claims.
|
||||
- Showing a diff without explicitly asking for edit approval.
|
||||
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
interface:
|
||||
display_name: "Revise AGENTS.md"
|
||||
short_description: "Capture durable Codex session learnings safely"
|
||||
default_prompt: "Use $revising-agents-md to propose durable AGENTS.md updates from this session."
|
||||
policy:
|
||||
allow_implicit_invocation: false
|
||||
Reference in New Issue
Block a user