FESADev/docs/VERIFICATION_PLAN.md

# Verification Plan

## Purpose
This document defines how FESA will be verified against small focused tests and stored reference artifacts.

The project rule is strict: solver quality is maintained by comparing against reference examples and reference results.

## Source Basis
- The original Dvorkin-Bathe four-node shell paper presents a non-flat quadrilateral shell element for thin and thick shells and includes test/demonstration problems: https://web.mit.edu/kjb/www/Publications_Prior_to_1998/A_Continuum_Mechanics_Based_Four-Node_Shell_Element_for_General_Nonlinear_Analysis.pdf
- The MITC3+/MITC4+ benchmark paper states that MITC methods address shell locking and uses widely-used shell benchmarks with convergence studies: https://web.mit.edu/kjb/www/Principal_Publications/Performance_of_the_MITC3%2B_and_MITC4%2B_shell_elements_in_widely_used_benchmark_problems.pdf
- COMSOL's Scordelis-Lo example documents the benchmark geometry, material, load, constraints, and reference displacement context: https://doc.comsol.com/5.6/doc/com.comsol.help.models.sme.scordelis_lo_roof/scordelis_lo_roof.html
- NAFEMS benchmark surveys list nonlinear shell and structural benchmark classes for future phases: https://www.nafems.org/publications/pubguide/benchmarks/Page6/

## Verification Philosophy
FESA verification uses multiple layers:

1. Unit tests for low-level math, parser, and DOF management.
2. Element tests for MITC4 stiffness, rigid body modes, patch behavior, and drilling stabilization.
3. Assembly tests for small known systems.
4. Solver tests for constrained/reduced systems and reaction recovery.
5. Input-to-result integration tests using Abaqus-style `.inp` files.
6. Reference comparisons against stored Abaqus result artifacts under `references/`.

A single large benchmark is not enough. Phase 1 should use many small models that isolate failure causes.

## Abaqus Availability
Abaqus is not available locally and must not be required by CI.

Rules:
- The user provides `.inp` files and solved reference result artifacts.
- FESA tests compare against stored artifacts only.
- Reference artifacts are treated as the numerical source of truth once accepted into `references/`.

## Accepted Reference Folder Contract
The repository uses `references/` as the durable root for stored Abaqus inputs and solved output artifacts.

Initial flat-file convention:

```text
references/
  README.md
  <case_name>.inp
  <case_name>_displacements.csv
```

Accepted stored cases:

```text
references/
  quad_01.inp
  quad_01_displacements.csv
  quad_02.inp
  quad_02_phase1.inp
  quad_02_displacements.csv
```

`quad_01_displacements.csv` contains 121 nodal rows with these columns:

```text
Node Label, U-U1, U-U2, U-U3, UR-UR1, UR-UR2, UR-UR3
```

`quad_02_displacements.csv` uses the same required columns and contains 121 nodal rows.

Future manifest-driven layout is still recommended as the case set grows:

```text
references/
  README.md
  phase1-linear-static/
    manifest.json
    single-element-membrane/
      model.inp
      displacements.csv
      notes.md
    single-element-bending/
      model.inp
      displacements.csv
      notes.md
    cantilever-strip/
      model.inp
      displacements.csv
      notes.md
    scordelis-lo-roof/
      model.inp
      displacements.csv
      notes.md
```

`manifest.json` should include:

```json
{
  "schema_version": 1,
  "cases": [
    {
      "name": "single-element-membrane",
      "input": "single-element-membrane/model.inp",
      "displacements": "single-element-membrane/displacements.csv",
      "analysis_type": "linear_static",
      "element": "MITC4",
      "source_solver": "Abaqus",
      "source_solver_version": "provided-by-user",
      "unit_system_note": "self-consistent",
      "result_format": "abaqus_displacement_csv_v1",
      "tags": ["phase1", "element", "membrane"]
    }
  ]
}
```

## Displacement CSV Artifact
Initial automated displacement comparison uses Abaqus-exported CSV files named:

```text
<case_name>_displacements.csv
```

Required columns:

| CSV Column | FESA Field | Component |
|---|---|---|
| `Node Label` | `U` | entity id |
| `U-U1` | `U` | `UX` |
| `U-U2` | `U` | `UY` |
| `U-U3` | `U` | `UZ` |
| `UR-UR1` | `U` | `RX` |
| `UR-UR2` | `U` | `RY` |
| `UR-UR3` | `U` | `RZ` |

Rules:
- Column matching is by exact normalized header text after trimming whitespace.
- `Node Label` is parsed as int64.
- All displacement and rotation values are parsed as `double`.
- The comparator must match rows by node id, not by row order alone.
- Missing or nonnumeric `Node Label` values, duplicate CSV node labels, missing columns, or nonnumeric component values are reference artifact errors.
- Missing nodes in FESA output, duplicate FESA output node ids, wrong FESA `U` component labels, or wrong nodal/global field metadata are comparison errors.
- CSV displacement comparison maps to `/results/steps/<step>/frames/<frame>/fieldOutputs/U`.
- If the `.inp` has multiple steps or frames, the case metadata must state which step/frame the CSV represents.

Structured JSON or HDF5 comparison artifacts may be added later, but `*_displacements.csv` is the accepted first automated reference format.

## Other Result Artifacts
Additional Abaqus result exports may be added as the solver grows. Recommended naming:

| File Pattern | Purpose | Status |
|---|---|---|
| `*_displacements.csv` | Nodal `U` displacement/rotation comparison | Accepted initial format |
| `*_reactions.csv` | Nodal `RF` force/moment comparison | Recommended future format |
| `*_stresses.csv` | Stress output comparison | Future, after `S` output is documented |
| `*_strains.csv` | Strain output comparison | Future, after `E` output is documented |
| `*_section_forces.csv` | Shell resultant comparison | Future, after `SF` output is documented |

Each new CSV type must define required columns, component mapping, position, basis, and tolerances before automated use.

## Stored Reference Compatibility Notes
Stored Abaqus `.inp` files are preserved as generated, even when they include features outside the current FESA parser subset. This preserves provenance but does not expand Phase 1 support.

Current initial case:

| Case | Files | Notes |
|---|---|---|
| `quad_01` | `quad_01.inp`, `quad_01_displacements.csv` | Abaqus/CAE Learning Edition 2024 input; 121 displacement rows; includes `S4R`, `Part/Assembly/Instance`, `*Density`, and `NLGEOM=YES`, which are outside the current Phase 1 parser/solver subset |
| `quad_02` | `quad_02.inp`, `quad_02_displacements.csv` | Abaqus/CAE input; 121 displacement rows; uses `TYPE=S4` and `NLGEOM=NO`, but still includes `Part/Assembly/Instance` and `*Density`, so it must be normalized or handled by an explicit parser compatibility sprint before automated Phase 1 input acceptance |
| `quad_02_phase1` | `quad_02_phase1.inp`, `quad_02_displacements.csv`, `quad_02_notes.md` | Phase 1-compatible derivative input for `quad_02`; preserves ids, connectivity, material, thickness, boundary nodes, load node, and load magnitude while removing unsupported Abaqus/CAE scaffolding |

Rules:
- Original `.inp` files under `references/` should not be modified just to fit FESA Phase 1.
- If a normalized Phase 1-compatible input is needed, add it as a separate file with a clear name and note its relationship to the original.
- Unsupported features in stored reference inputs must be reported by compatibility checks, not silently accepted by parser tests.
- A reference case may be useful for future compatibility even before it is executable by the current Phase 1 solver.
- `quad_02_phase1.inp` is the accepted normalized input path for the first `quad_02` Phase 1 S4 reference regression after the MITC4 rebuild and end-to-end workflow are complete.

## Tolerance Policy
Use absolute and relative tolerance:

```text
abs_error = abs(actual - expected)
rel_error = abs_error / max(abs(expected), reference_scale)
pass if abs_error <= abs_tol or rel_error <= rel_tol
```

Initial guidance:
- Parser and exact metadata tests: exact match.
- DOF mapping tests: exact integer match.
- Small linear algebra tests: `abs_tol = 1e-12`, `rel_tol = 1e-10`.
- Element stiffness symmetry: matrix norm tolerance around `1e-10` to `1e-9`, adjusted by stiffness scale.
- Reference displacement CSV tests: start with `rel_tol = 1e-5` and refine after baseline agreement.
- Reaction equilibrium tests: use force-scale-based tolerances.

Final benchmark tolerances must be stored with each reference case.

## Phase 1 Benchmark Matrix
| Case | Purpose | Required Output |
|---|---|---|
| Parser smoke model | Verify Phase 1 keyword subset | Domain object counts, set membership |
| Single MITC4 membrane | Constant strain behavior | `U`, element strains/stresses if available |
| Single MITC4 bending | Bending stiffness sanity | `U`, rotations |
| Rigid body mode check | Verify near-zero internal forces without constraints | eigen/nullspace or controlled diagnostic |
| Constrained static sanity | Verify constrained DOF elimination | `U`, `RF` |
| Reaction balance | Verify full-vector reaction recovery | total load vs total reaction |
| Cantilever shell strip | Bending-dominated displacement convergence | tip `U`, support `RF` |
| Simply supported square plate | Thin shell/plate locking sensitivity | center displacement |
| Scordelis-Lo roof | Curved shell benchmark | midpoint vertical displacement |
| Pinched cylinder | Curved shell bending/membrane behavior | loaded-point displacement |
| Twisted beam | Warped shell and drilling sensitivity | tip displacement/rotation |

### Step 11 Analytic Patch And Benchmark Scaffold
The first MITC4 patch-test layer is kept local and analytic so it does not depend on Abaqus execution or stored CSV artifacts.

Current in-repository coverage:
- Constant membrane patch: verifies uniform local `[eps11, eps22, gamma12]` strain at all `2 x 2 x 2` samples and positive physical energy.
- Pure bending patch: verifies membrane-free midsurface bending behavior, through-thickness antisymmetry, and positive physical bending energy.
- Pure transverse shear patch: verifies constant `gamma13` reproduction through the MITC tying path.
- Pure twist patch: verifies bilinear transverse-shear reproduction for `w = twist * x * y`, covering the locking-sensitive twist/shear interpolation path before curved benchmarks.
- Drilling sensitivity negative check: verifies membrane patch energy is unchanged by drilling scale changes when no drilling DOF participates.
- Thin one-element cantilever strip: verifies tip displacement increases materially as thickness decreases, providing an early shear-locking sensitivity smoke test rather than a final convergence benchmark.

Scordelis-Lo roof remains a named benchmark scaffold for Phase 1 planning, but it is not executable in the current Step 11 test layer because Phase 1 does not yet support pressure loads and does not yet have an accepted curved-shell Abaqus reference artifact. The first executable Scordelis-Lo sprint must define load representation, normalized input compatibility, stored displacement reference data, and scale-aware displacement tolerances before it is used as an automated acceptance case.

## Minimum Phase 1 Acceptance
Before MITC4 Phase 1 is considered credible:
- All parser smoke tests pass.
- `DofManager` free/constrained mapping tests pass.
- Reduced-system solve reconstructs full `U`.
- `RF = K_full * U_full - F_full` is tested.
- MITC4 element stiffness is symmetric within tolerance for linear elastic Phase 1.
- Rigid body modes do not create artificial membrane/bending stiffness beyond documented drilling stabilization effects.
- At least three stored Abaqus reference models pass: one single-element case, one simple multi-element plate/shell case, and one curved shell benchmark.
- At least one automated `*_displacements.csv` comparison passes against a stored Abaqus reference case.

## Reference Onboarding Checklist
Every accepted reference case should include:
- an Abaqus `.inp` file.
- at least one solved result artifact, initially `*_displacements.csv`.
- raw Abaqus output when available, such as `.dat`, `.rpt`, `.csv`, or exported table files.
- `notes.md` describing model purpose, unit system note, Abaqus version, expected dominant behavior, and known limitations.
- comparison paths matching `docs/RESULTS_SCHEMA.md`.
- absolute and relative tolerances for each compared quantity.
- tags that identify phase, feature, benchmark family, and expected failure mode if negative.
- compatibility notes for unsupported Abaqus keywords or analysis options.

Reference artifacts should not be overwritten silently. If reference values change, record why they changed in `notes.md` or the manifest.

## Singular System Verification
Required negative tests:
- Model with no boundary conditions should fail with a singular-system diagnostic.
- Model with missing property should fail before assembly.
- Model with load on missing node or set should fail during input/model validation.
- Model with an unconstrained isolated node should identify free untouched DOFs.
- Model with only drilling DOF instability should mention weak or unconstrained rotational DOFs.

## What Not To Verify In Phase 1
- Mesh quality metrics such as aspect ratio, skew, warpage, Jacobian quality, or distortion warnings.
- Pressure loads.
- RBE2/RBE3.
- Nonlinear increments.
- Time integration.
- Thermal-stress coupling.

## User Inputs Needed
- Additional small Abaqus `.inp` files and solved result CSV files under `references/`.
- Reaction output artifacts when available, preferably a documented `*_reactions.csv`.
- Abaqus version used to generate each reference when it is not evident from the `.inp`.
- Unit system notes and tolerances for each case.
- Whether future Phase 1-compatible reference files will use Abaqus `S4`, while `S4R` remains deferred.