FESADev/docs/VERIFICATION_PLAN.md

Verification Plan

Purpose

This document defines how FESA will be verified against small focused tests and stored reference artifacts.

The project rule is strict: solver quality is maintained by comparing against reference examples and reference results.

Source Basis

• The original Dvorkin-Bathe four-node shell paper presents a non-flat quadrilateral shell element for thin and thick shells and includes test/demonstration problems: https://web.mit.edu/kjb/www/Publications_Prior_to_1998/A_Continuum_Mechanics_Based_Four-Node_Shell_Element_for_General_Nonlinear_Analysis.pdf
• The MITC3+/MITC4+ benchmark paper states that MITC methods address shell locking and uses widely-used shell benchmarks with convergence studies: https://web.mit.edu/kjb/www/Principal_Publications/Performance_of_the_MITC3%2B_and_MITC4%2B_shell_elements_in_widely_used_benchmark_problems.pdf
• COMSOL's Scordelis-Lo example documents the benchmark geometry, material, load, constraints, and reference displacement context: https://doc.comsol.com/5.6/doc/com.comsol.help.models.sme.scordelis_lo_roof/scordelis_lo_roof.html
• NAFEMS benchmark surveys list nonlinear shell and structural benchmark classes for future phases: https://www.nafems.org/publications/pubguide/benchmarks/Page6/

Verification Philosophy

FESA verification uses multiple layers:

1. Unit tests for low-level math, parser, and DOF management.
2. Element tests for MITC4 stiffness, rigid body modes, patch behavior, and drilling stabilization.
3. Assembly tests for small known systems.
4. Solver tests for constrained/reduced systems and reaction recovery.
5. Input-to-result integration tests using Abaqus-style .inp files.
6. Reference comparisons against stored Abaqus result artifacts under references/.

A single large benchmark is not enough. Phase 1 should use many small models that isolate failure causes.

Abaqus Availability

Abaqus is not available locally and must not be required by CI.

Rules:

• The user provides .inp files and solved reference result artifacts.
• FESA tests compare against stored artifacts only.
• Reference artifacts are treated as the numerical source of truth once accepted into references/.

Accepted Reference Folder Contract

The repository uses references/ as the durable root for stored Abaqus inputs and solved output artifacts.

Initial flat-file convention:

references/
  README.md
  <case_name>.inp
  <case_name>_displacements.csv

Accepted initial case:

references/
  quad_01.inp
  quad_01_displacements.csv

quad_01_displacements.csv contains 121 nodal rows with these columns:

Node Label, U-U1, U-U2, U-U3, UR-UR1, UR-UR2, UR-UR3

Future manifest-driven layout is still recommended as the case set grows:

references/
  README.md
  phase1-linear-static/
    manifest.json
    single-element-membrane/
      model.inp
      displacements.csv
      notes.md
    single-element-bending/
      model.inp
      displacements.csv
      notes.md
    cantilever-strip/
      model.inp
      displacements.csv
      notes.md
    scordelis-lo-roof/
      model.inp
      displacements.csv
      notes.md

manifest.json should include:

{
  "schema_version": 1,
  "cases": [
    {
      "name": "single-element-membrane",
      "input": "single-element-membrane/model.inp",
      "displacements": "single-element-membrane/displacements.csv",
      "analysis_type": "linear_static",
      "element": "MITC4",
      "source_solver": "Abaqus",
      "source_solver_version": "provided-by-user",
      "unit_system_note": "self-consistent",
      "result_format": "abaqus_displacement_csv_v1",
      "tags": ["phase1", "element", "membrane"]
    }
  ]
}

Displacement CSV Artifact

Initial automated displacement comparison uses Abaqus-exported CSV files named:

<case_name>_displacements.csv

Required columns:

CSV Column	FESA Field	Component
Node Label	U	entity id
U-U1	U	UX
U-U2	U	UY
U-U3	U	UZ
UR-UR1	U	RX
UR-UR2	U	RY
UR-UR3	U	RZ

Rules:

• Column matching is by exact normalized header text after trimming whitespace.
• Node Label is parsed as int64.
• All displacement and rotation values are parsed as double.
• The comparator must match rows by node id, not by row order alone.
• Missing nodes in FESA output, duplicate CSV node labels, missing columns, or nonnumeric values are reference artifact errors.
• CSV displacement comparison maps to /results/steps/<step>/frames/<frame>/fieldOutputs/U.
• If the .inp has multiple steps or frames, the case metadata must state which step/frame the CSV represents.

Structured JSON or HDF5 comparison artifacts may be added later, but *_displacements.csv is the accepted first automated reference format.

Other Result Artifacts

Additional Abaqus result exports may be added as the solver grows. Recommended naming:

File Pattern	Purpose	Status
*_displacements.csv	Nodal U displacement/rotation comparison	Accepted initial format
*_reactions.csv	Nodal RF force/moment comparison	Recommended future format
*_stresses.csv	Stress output comparison	Future, after S output is documented
*_strains.csv	Strain output comparison	Future, after E output is documented
*_section_forces.csv	Shell resultant comparison	Future, after SF output is documented

Each new CSV type must define required columns, component mapping, position, basis, and tolerances before automated use.

Stored Reference Compatibility Notes

Stored Abaqus .inp files are preserved as generated, even when they include features outside the current FESA parser subset. This preserves provenance but does not expand Phase 1 support.

Current initial case:

Case	Files	Notes
quad_01	quad_01.inp, quad_01_displacements.csv	Abaqus/CAE Learning Edition 2024 input; 121 displacement rows; includes S4R, Part/Assembly/Instance, *Density, and NLGEOM=YES, which are outside the current Phase 1 parser/solver subset

Rules:

• Original .inp files under references/ should not be modified just to fit FESA Phase 1.
• If a normalized Phase 1-compatible input is needed, add it as a separate file with a clear name and note its relationship to the original.
• Unsupported features in stored reference inputs must be reported by compatibility checks, not silently accepted by parser tests.
• A reference case may be useful for future compatibility even before it is executable by the current Phase 1 solver.

Tolerance Policy

Use absolute and relative tolerance:

abs_error = abs(actual - expected)
rel_error = abs_error / max(abs(expected), reference_scale)
pass if abs_error <= abs_tol or rel_error <= rel_tol

Initial guidance:

• Parser and exact metadata tests: exact match.
• DOF mapping tests: exact integer match.
• Small linear algebra tests: abs_tol = 1e-12, rel_tol = 1e-10.
• Element stiffness symmetry: matrix norm tolerance around 1e-10 to 1e-9, adjusted by stiffness scale.
• Reference displacement CSV tests: start with rel_tol = 1e-5 and refine after baseline agreement.
• Reaction equilibrium tests: use force-scale-based tolerances.

Final benchmark tolerances must be stored with each reference case.

Phase 1 Benchmark Matrix

Case	Purpose	Required Output
Parser smoke model	Verify Phase 1 keyword subset	Domain object counts, set membership
Single MITC4 membrane	Constant strain behavior	U, element strains/stresses if available
Single MITC4 bending	Bending stiffness sanity	U, rotations
Rigid body mode check	Verify near-zero internal forces without constraints	eigen/nullspace or controlled diagnostic
Constrained static sanity	Verify constrained DOF elimination	U, RF
Reaction balance	Verify full-vector reaction recovery	total load vs total reaction
Cantilever shell strip	Bending-dominated displacement convergence	tip U, support RF
Simply supported square plate	Thin shell/plate locking sensitivity	center displacement
Scordelis-Lo roof	Curved shell benchmark	midpoint vertical displacement
Pinched cylinder	Curved shell bending/membrane behavior	loaded-point displacement
Twisted beam	Warped shell and drilling sensitivity	tip displacement/rotation

Minimum Phase 1 Acceptance

Before MITC4 Phase 1 is considered credible:

• All parser smoke tests pass.
• DofManager free/constrained mapping tests pass.
• Reduced-system solve reconstructs full U.
• RF = K_full * U_full - F_full is tested.
• MITC4 element stiffness is symmetric within tolerance for linear elastic Phase 1.
• Rigid body modes do not create artificial membrane/bending stiffness beyond documented drilling stabilization effects.
• At least three stored Abaqus reference models pass: one single-element case, one simple multi-element plate/shell case, and one curved shell benchmark.
• At least one automated *_displacements.csv comparison passes against a stored Abaqus reference case.

Reference Onboarding Checklist

Every accepted reference case should include:

• an Abaqus .inp file.
• at least one solved result artifact, initially *_displacements.csv.
• raw Abaqus output when available, such as .dat, .rpt, .csv, or exported table files.
• notes.md describing model purpose, unit system note, Abaqus version, expected dominant behavior, and known limitations.
• comparison paths matching docs/RESULTS_SCHEMA.md.
• absolute and relative tolerances for each compared quantity.
• tags that identify phase, feature, benchmark family, and expected failure mode if negative.
• compatibility notes for unsupported Abaqus keywords or analysis options.

Reference artifacts should not be overwritten silently. If reference values change, record why they changed in notes.md or the manifest.

Singular System Verification

Required negative tests:

• Model with no boundary conditions should fail with a singular-system diagnostic.
• Model with missing property should fail before assembly.
• Model with load on missing node or set should fail during input/model validation.
• Model with an unconstrained isolated node should identify free untouched DOFs.
• Model with only drilling DOF instability should mention weak or unconstrained rotational DOFs.

What Not To Verify In Phase 1

• Mesh quality metrics such as aspect ratio, skew, warpage, Jacobian quality, or distortion warnings.
• Pressure loads.
• RBE2/RBE3.
• Nonlinear increments.
• Time integration.
• Thermal-stress coupling.

User Inputs Needed

• Additional small Abaqus .inp files and solved result CSV files under references/.
• Reaction output artifacts when available, preferably a documented *_reactions.csv.
• Abaqus version used to generate each reference when it is not evident from the .inp.
• Unit system notes and tolerances for each case.
• Whether future Phase 1-compatible reference files will use Abaqus S4, while S4R remains deferred.