FESADev/docs/VERIFICATION_PLAN.md

Verification Plan

Purpose

This document defines how FESA will be verified against small focused tests and stored reference artifacts.

The project rule is strict: solver quality is maintained by comparing against reference examples and reference results.

Source Basis

• The original Dvorkin-Bathe four-node shell paper presents a non-flat quadrilateral shell element for thin and thick shells and includes test/demonstration problems: https://web.mit.edu/kjb/www/Publications_Prior_to_1998/A_Continuum_Mechanics_Based_Four-Node_Shell_Element_for_General_Nonlinear_Analysis.pdf
• The MITC3+/MITC4+ benchmark paper states that MITC methods address shell locking and uses widely-used shell benchmarks with convergence studies: https://web.mit.edu/kjb/www/Principal_Publications/Performance_of_the_MITC3%2B_and_MITC4%2B_shell_elements_in_widely_used_benchmark_problems.pdf
• COMSOL's Scordelis-Lo example documents the benchmark geometry, material, load, constraints, and reference displacement context: https://doc.comsol.com/5.6/doc/com.comsol.help.models.sme.scordelis_lo_roof/scordelis_lo_roof.html
• NAFEMS benchmark surveys list nonlinear shell and structural benchmark classes for future phases: https://www.nafems.org/publications/pubguide/benchmarks/Page6/

Verification Philosophy

FESA verification uses multiple layers:

1. Unit tests for low-level math, parser, and DOF management.
2. Element tests for MITC4 stiffness, rigid body modes, patch behavior, and drilling stabilization.
3. Assembly tests for small known systems.
4. Solver tests for constrained/reduced systems and reaction recovery.
5. Input-to-result integration tests using Abaqus-style .inp files.
6. Reference comparisons against stored Abaqus result artifacts under references/.

A single large benchmark is not enough. Phase 1 should use many small models that isolate failure causes.

Abaqus Availability

Abaqus is not available locally and must not be required by CI.

Rules:

• The user provides .inp files and solved reference result artifacts.
• FESA tests compare against stored artifacts only.
• Reference artifacts are treated as the numerical source of truth once accepted into references/.

Accepted Reference Folder Contract

The repository uses references/ as the durable root for stored Abaqus inputs and solved output artifacts.

Initial flat-file convention:

references/
  README.md
  <case_name>.inp
  <case_name>_displacements.csv

Accepted stored cases:

references/
  quad_01.inp
  quad_01_displacements.csv
  quad_02.inp
  quad_02_phase1.inp
  quad_02_displacements.csv

quad_01_displacements.csv contains 121 nodal rows with these columns:

Node Label, U-U1, U-U2, U-U3, UR-UR1, UR-UR2, UR-UR3

quad_02_displacements.csv uses the same required columns and contains 121 nodal rows.

Future manifest-driven layout is still recommended as the case set grows:

references/
  README.md
  phase1-linear-static/
    manifest.json
    single-element-membrane/
      model.inp
      displacements.csv
      notes.md
    single-element-bending/
      model.inp
      displacements.csv
      notes.md
    cantilever-strip/
      model.inp
      displacements.csv
      notes.md
    scordelis-lo-roof/
      model.inp
      displacements.csv
      notes.md

manifest.json should include:

{
  "schema_version": 1,
  "cases": [
    {
      "name": "single-element-membrane",
      "input": "single-element-membrane/model.inp",
      "displacements": "single-element-membrane/displacements.csv",
      "analysis_type": "linear_static",
      "element": "MITC4",
      "source_solver": "Abaqus",
      "source_solver_version": "provided-by-user",
      "unit_system_note": "self-consistent",
      "result_format": "abaqus_displacement_csv_v1",
      "tags": ["phase1", "element", "membrane"]
    }
  ]
}

Displacement CSV Artifact

Initial automated displacement comparison uses Abaqus-exported CSV files named:

<case_name>_displacements.csv

Required columns:

CSV Column	FESA Field	Component
Node Label	U	entity id
U-U1	U	UX
U-U2	U	UY
U-U3	U	UZ
UR-UR1	U	RX
UR-UR2	U	RY
UR-UR3	U	RZ

Rules:

• Column matching is by exact normalized header text after trimming whitespace.
• Node Label is parsed as int64.
• All displacement and rotation values are parsed as double.
• The comparator must match rows by node id, not by row order alone.
• Missing or nonnumeric Node Label values, duplicate CSV node labels, missing columns, or nonnumeric component values are reference artifact errors.
• Missing nodes in FESA output, duplicate FESA output node ids, wrong FESA U component labels, or wrong nodal/global field metadata are comparison errors.
• CSV displacement comparison maps to /results/steps/<step>/frames/<frame>/fieldOutputs/U.
• If the .inp has multiple steps or frames, the case metadata must state which step/frame the CSV represents.

Structured JSON or HDF5 comparison artifacts may be added later, but *_displacements.csv is the accepted first automated reference format.

Other Result Artifacts

Additional Abaqus result exports may be added as the solver grows. Recommended naming:

File Pattern	Purpose	Status
*_displacements.csv	Nodal U displacement/rotation comparison	Accepted initial format
*_reactions.csv	Nodal RF force/moment comparison	Recommended future format
*_stresses.csv	Stress output comparison	Future, after S output is documented
*_strains.csv	Strain output comparison	Future, after E output is documented
*_section_forces.csv	Shell resultant comparison	Future, after SF output is documented

Each new CSV type must define required columns, component mapping, position, basis, and tolerances before automated use.

Stored Reference Compatibility Notes

Stored Abaqus .inp files are preserved as generated, even when they include features outside the current FESA parser subset. This preserves provenance but does not expand Phase 1 support.

Current initial case:

Case	Files	Notes
quad_01	quad_01.inp, quad_01_displacements.csv	Abaqus/CAE Learning Edition 2024 input; 121 displacement rows; includes S4R, Part/Assembly/Instance, *Density, and NLGEOM=YES, which are outside the current Phase 1 parser/solver subset
quad_02	quad_02.inp, quad_02_displacements.csv	Abaqus/CAE input; 121 displacement rows; uses TYPE=S4 and NLGEOM=NO, but still includes Part/Assembly/Instance and *Density, so it must be normalized or handled by an explicit parser compatibility sprint before automated Phase 1 input acceptance
quad_02_phase1	quad_02_phase1.inp, quad_02_displacements.csv, quad_02_notes.md	Phase 1-compatible derivative input for quad_02; preserves ids, connectivity, material, thickness, boundary nodes, load node, and load magnitude while removing unsupported Abaqus/CAE scaffolding

Rules:

• Original .inp files under references/ should not be modified just to fit FESA Phase 1.
• If a normalized Phase 1-compatible input is needed, add it as a separate file with a clear name and note its relationship to the original.
• Unsupported features in stored reference inputs must be reported by compatibility checks, not silently accepted by parser tests.
• A reference case may be useful for future compatibility even before it is executable by the current Phase 1 solver.
• quad_02_phase1.inp is the accepted normalized input path for the first quad_02 Phase 1 S4 reference regression after the MITC4 rebuild and end-to-end workflow are complete.

Tolerance Policy

Use absolute and relative tolerance:

abs_error = abs(actual - expected)
rel_error = abs_error / max(abs(expected), reference_scale)
pass if abs_error <= abs_tol or rel_error <= rel_tol

Initial guidance:

• Parser and exact metadata tests: exact match.
• DOF mapping tests: exact integer match.
• Small linear algebra tests: abs_tol = 1e-12, rel_tol = 1e-10.
• Element stiffness symmetry: matrix norm tolerance around 1e-10 to 1e-9, adjusted by stiffness scale.
• Reference displacement CSV tests: start with rel_tol = 1e-5 and refine after baseline agreement.
• Reaction equilibrium tests: use force-scale-based tolerances.

Final benchmark tolerances must be stored with each reference case.

Step 14 Stored Reference Regression Status

The first automated stored-reference displacement regression is active for:

input: references/quad_02_phase1.inp
expected U: references/quad_02_displacements.csv

Comparison rules:

• The original references/quad_02.inp remains unsupported provenance and must still be rejected by parser compatibility tests because it contains Abaqus/CAE scaffolding.
• The normalized quad_02_phase1.inp is the executable Phase 1 input for this reference pair.
• The FESA U field is compared node-id-by-node-id against Abaqus CSV columns U-U1, U-U2, U-U3, UR-UR1, UR-UR2, and UR-UR3.
• The active tolerance is abs_tol = 1.0e-12, rel_tol = 1.0e-5, reference_scale = 1.0.
• Abaqus reaction CSV is still unavailable, so RF remains verified by full-vector equilibrium tests until a *_reactions.csv artifact is provided.

Phase 1 Benchmark Matrix

Case	Purpose	Required Output
Parser smoke model	Verify Phase 1 keyword subset	Domain object counts, set membership
Single MITC4 membrane	Constant strain behavior	U, element strains/stresses if available
Single MITC4 bending	Bending stiffness sanity	U, rotations
Rigid body mode check	Verify near-zero internal forces without constraints	eigen/nullspace or controlled diagnostic
Constrained static sanity	Verify constrained DOF elimination	U, RF
Reaction balance	Verify full-vector reaction recovery	total load vs total reaction
Cantilever shell strip	Bending-dominated displacement convergence	tip U, support RF
Simply supported square plate	Thin shell/plate locking sensitivity	center displacement
Scordelis-Lo roof	Curved shell benchmark	midpoint vertical displacement
Pinched cylinder	Curved shell bending/membrane behavior	loaded-point displacement
Twisted beam	Warped shell and drilling sensitivity	tip displacement/rotation

Step 11 Analytic Patch And Benchmark Scaffold

The first MITC4 patch-test layer is kept local and analytic so it does not depend on Abaqus execution or stored CSV artifacts.

Current in-repository coverage:

• Constant membrane patch: verifies uniform local [eps11, eps22, gamma12] strain at all 2 x 2 x 2 samples and positive physical energy.
• Pure bending patch: verifies membrane-free midsurface bending behavior, through-thickness antisymmetry, and positive physical bending energy.
• Pure transverse shear patch: verifies constant gamma13 reproduction through the MITC tying path.
• Pure twist patch: verifies bilinear transverse-shear reproduction for w = twist * x * y, covering the locking-sensitive twist/shear interpolation path before curved benchmarks.
• Drilling sensitivity negative check: verifies membrane patch energy is unchanged by drilling scale changes when no drilling DOF participates.
• Thin one-element cantilever strip: verifies tip displacement increases materially as thickness decreases, providing an early shear-locking sensitivity smoke test rather than a final convergence benchmark.

Scordelis-Lo roof remains a named benchmark scaffold for Phase 1 planning, but it is not executable in the current Step 11 test layer because Phase 1 does not yet support pressure loads and does not yet have an accepted curved-shell Abaqus reference artifact. The first executable Scordelis-Lo sprint must define load representation, normalized input compatibility, stored displacement reference data, and scale-aware displacement tolerances before it is used as an automated acceptance case.

Minimum Phase 1 Acceptance

Before MITC4 Phase 1 is considered credible:

• All parser smoke tests pass.
• DofManager free/constrained mapping tests pass.
• Reduced-system solve reconstructs full U.
• RF = K_full * U_full - F_full is tested.
• MITC4 element stiffness is symmetric within tolerance for linear elastic Phase 1.
• Rigid body modes do not create artificial membrane/bending stiffness beyond documented drilling stabilization effects.
• At least three stored Abaqus reference models pass: one single-element case, one simple multi-element plate/shell case, and one curved shell benchmark.
• At least one automated *_displacements.csv comparison passes against a stored Abaqus reference case.

Reference Onboarding Checklist

Every accepted reference case should include:

• an Abaqus .inp file.
• at least one solved result artifact, initially *_displacements.csv.
• raw Abaqus output when available, such as .dat, .rpt, .csv, or exported table files.
• notes.md describing model purpose, unit system note, Abaqus version, expected dominant behavior, and known limitations.
• comparison paths matching docs/RESULTS_SCHEMA.md.
• absolute and relative tolerances for each compared quantity.
• tags that identify phase, feature, benchmark family, and expected failure mode if negative.
• compatibility notes for unsupported Abaqus keywords or analysis options.

Reference artifacts should not be overwritten silently. If reference values change, record why they changed in notes.md or the manifest.

Singular System Verification

Required negative tests:

• Model with no boundary conditions should fail with a singular-system diagnostic.
• Model with missing property should fail before assembly.
• Model with load on missing node or set should fail during input/model validation.
• Model with an unconstrained isolated node should identify free untouched DOFs.
• Model with only drilling DOF instability should mention weak or unconstrained rotational DOFs.

What Not To Verify In Phase 1

• Mesh quality metrics such as aspect ratio, skew, warpage, Jacobian quality, or distortion warnings.
• Pressure loads.
• RBE2/RBE3.
• Nonlinear increments.
• Time integration.
• Thermal-stress coupling.

User Inputs Needed

• Additional small Abaqus .inp files and solved result CSV files under references/.
• Reaction output artifacts when available, preferably a documented *_reactions.csv.
• Abaqus version used to generate each reference when it is not evident from the .inp.
• Unit system notes and tolerances for each case.
• Whether future Phase 1-compatible reference files will use Abaqus S4, while S4R remains deferred.