# Verification Plan

## Purpose
This document defines how FESA will be verified against small focused tests and stored reference artifacts.

The project rule is strict: solver quality is maintained by comparing against reference examples and reference results.

## Source Basis
- The original Dvorkin-Bathe four-node shell paper presents a non-flat quadrilateral shell element for thin and thick shells and includes test/demonstration problems: https://web.mit.edu/kjb/www/Publications_Prior_to_1998/A_Continuum_Mechanics_Based_Four-Node_Shell_Element_for_General_Nonlinear_Analysis.pdf
- The MITC3+/MITC4+ benchmark paper states that MITC methods address shell locking and uses widely-used shell benchmarks with convergence studies: https://web.mit.edu/kjb/www/Principal_Publications/Performance_of_the_MITC3%2B_and_MITC4%2B_shell_elements_in_widely_used_benchmark_problems.pdf
- COMSOL's Scordelis-Lo example documents the benchmark geometry, material, load, constraints, and reference displacement context: https://doc.comsol.com/5.6/doc/com.comsol.help.models.sme.scordelis_lo_roof/scordelis_lo_roof.html
- NAFEMS benchmark surveys list nonlinear shell and structural benchmark classes for future phases: https://www.nafems.org/publications/pubguide/benchmarks/Page6/

## Verification Philosophy
FESA verification uses multiple layers:

1. Unit tests for low-level math, parser, and DOF management.
2. Element tests for MITC4 stiffness, rigid body modes, patch behavior, and drilling stabilization.
3. Assembly tests for small known systems.
4. Solver tests for constrained/reduced systems and reaction recovery.
5. Input-to-result integration tests using Abaqus-style `.inp` files.
6. Reference comparisons against stored Abaqus result artifacts under `reference/`.

A single large benchmark is not enough. Phase 1 should use many small models that isolate failure causes.

## Abaqus Availability
Abaqus is not available locally and must not be required by CI.

Rules:
- The user provides `.inp` files and solved reference result artifacts.
- FESA tests compare against stored artifacts only.
- Reference artifacts are treated as the numerical source of truth once accepted into `reference/`.

## Proposed Reference Folder Contract
Recommended layout:

```text
reference/
  README.md
  phase1-linear-static/
    manifest.json
    single-element-membrane/
      model.inp
      expected.json
      notes.md
    single-element-bending/
      model.inp
      expected.json
      notes.md
    cantilever-strip/
      model.inp
      expected.json
      notes.md
    scordelis-lo-roof/
      model.inp
      expected.json
      notes.md
```

`manifest.json` should include:

```json
{
  "schema_version": 1,
  "cases": [
    {
      "name": "single-element-membrane",
      "input": "single-element-membrane/model.inp",
      "expected": "single-element-membrane/expected.json",
      "analysis_type": "linear_static",
      "element": "MITC4",
      "source_solver": "Abaqus",
      "source_solver_version": "provided-by-user",
      "unit_system_note": "self-consistent",
      "tags": ["phase1", "element", "membrane"]
    }
  ]
}
```

## Expected Result Artifact
Preferred `expected.json` structure:

```json
{
  "schema_version": 1,
  "case_name": "single-element-membrane",
  "source": {
    "solver": "Abaqus",
    "version": "provided-by-user",
    "created_by": "provided-by-user"
  },
  "units": {
    "note": "self-consistent"
  },
  "comparisons": [
    {
      "path": "/results/steps/Step-1/frames/0/fieldOutputs/U",
      "entity": {"type": "node", "id": 4},
      "component": "UX",
      "expected": 0.0,
      "abs_tol": 1e-9,
      "rel_tol": 1e-7
    }
  ]
}
```

CSV or `.rpt` files may be stored as raw source artifacts, but structured JSON or HDF5 comparison artifacts should be used by automated tests.

## Tolerance Policy
Use absolute and relative tolerance:

```text
abs_error = abs(actual - expected)
rel_error = abs_error / max(abs(expected), reference_scale)
pass if abs_error <= abs_tol or rel_error <= rel_tol
```

Initial guidance:
- Parser and exact metadata tests: exact match.
- DOF mapping tests: exact integer match.
- Small linear algebra tests: `abs_tol = 1e-12`, `rel_tol = 1e-10`.
- Element stiffness symmetry: matrix norm tolerance around `1e-10` to `1e-9`, adjusted by stiffness scale.
- Reference displacement tests: start with `rel_tol = 1e-5` and refine after baseline agreement.
- Reaction equilibrium tests: use force-scale-based tolerances.

Final benchmark tolerances must be stored with each reference case.

## Phase 1 Benchmark Matrix
| Case | Purpose | Required Output |
|---|---|---|
| Parser smoke model | Verify Phase 1 keyword subset | Domain object counts, set membership |
| Single MITC4 membrane | Constant strain behavior | `U`, element strains/stresses if available |
| Single MITC4 bending | Bending stiffness sanity | `U`, rotations |
| Rigid body mode check | Verify near-zero internal forces without constraints | eigen/nullspace or controlled diagnostic |
| Constrained static sanity | Verify constrained DOF elimination | `U`, `RF` |
| Reaction balance | Verify full-vector reaction recovery | total load vs total reaction |
| Cantilever shell strip | Bending-dominated displacement convergence | tip `U`, support `RF` |
| Simply supported square plate | Thin shell/plate locking sensitivity | center displacement |
| Scordelis-Lo roof | Curved shell benchmark | midpoint vertical displacement |
| Pinched cylinder | Curved shell bending/membrane behavior | loaded-point displacement |
| Twisted beam | Warped shell and drilling sensitivity | tip displacement/rotation |

## Minimum Phase 1 Acceptance
Before MITC4 Phase 1 is considered credible:
- All parser smoke tests pass.
- `DofManager` free/constrained mapping tests pass.
- Reduced-system solve reconstructs full `U`.
- `RF = K_full * U_full - F_full` is tested.
- MITC4 element stiffness is symmetric within tolerance for linear elastic Phase 1.
- Rigid body modes do not create artificial membrane/bending stiffness beyond documented drilling stabilization effects.
- At least three stored Abaqus reference models pass: one single-element case, one simple multi-element plate/shell case, and one curved shell benchmark.

## Singular System Verification
Required negative tests:
- Model with no boundary conditions should fail with a singular-system diagnostic.
- Model with missing property should fail before assembly.
- Model with load on missing node or set should fail during input/model validation.
- Model with an unconstrained isolated node should identify free untouched DOFs.
- Model with only drilling DOF instability should mention weak or unconstrained rotational DOFs.

## What Not To Verify In Phase 1
- Mesh quality metrics such as aspect ratio, skew, warpage, Jacobian quality, or distortion warnings.
- Pressure loads.
- RBE2/RBE3.
- Nonlinear increments.
- Time integration.
- Thermal-stress coupling.

## User Inputs Needed
- First accepted `reference/` folder layout.
- At least one Abaqus `.inp` with solved displacement and reaction output.
- Preferred raw reference format: `.dat`, `.rpt`, `.csv`, `.json`, or `.h5`.
- Abaqus version used to generate the reference.
- Whether reference files are generated from Abaqus `S4` only for Phase 1.