FESADev/docs/VERIFICATION_PLAN.md

Verification Plan

Purpose

This document defines how FESA will be verified against small focused tests and stored reference artifacts.

The project rule is strict: solver quality is maintained by comparing against reference examples and reference results.

Source Basis

• The original Dvorkin-Bathe four-node shell paper presents a non-flat quadrilateral shell element for thin and thick shells and includes test/demonstration problems: https://web.mit.edu/kjb/www/Publications_Prior_to_1998/A_Continuum_Mechanics_Based_Four-Node_Shell_Element_for_General_Nonlinear_Analysis.pdf
• The MITC3+/MITC4+ benchmark paper states that MITC methods address shell locking and uses widely-used shell benchmarks with convergence studies: https://web.mit.edu/kjb/www/Principal_Publications/Performance_of_the_MITC3%2B_and_MITC4%2B_shell_elements_in_widely_used_benchmark_problems.pdf
• COMSOL's Scordelis-Lo example documents the benchmark geometry, material, load, constraints, and reference displacement context: https://doc.comsol.com/5.6/doc/com.comsol.help.models.sme.scordelis_lo_roof/scordelis_lo_roof.html
• NAFEMS benchmark surveys list nonlinear shell and structural benchmark classes for future phases: https://www.nafems.org/publications/pubguide/benchmarks/Page6/

Verification Philosophy

FESA verification uses multiple layers:

1. Unit tests for low-level math, parser, and DOF management.
2. Element tests for MITC4 stiffness, rigid body modes, patch behavior, and drilling stabilization.
3. Assembly tests for small known systems.
4. Solver tests for constrained/reduced systems and reaction recovery.
5. Input-to-result integration tests using Abaqus-style .inp files.
6. Reference comparisons against stored Abaqus result artifacts under reference/.

A single large benchmark is not enough. Phase 1 should use many small models that isolate failure causes.

Abaqus Availability

Abaqus is not available locally and must not be required by CI.

Rules:

• The user provides .inp files and solved reference result artifacts.
• FESA tests compare against stored artifacts only.
• Reference artifacts are treated as the numerical source of truth once accepted into reference/.

Proposed Reference Folder Contract

Recommended layout:

reference/
  README.md
  phase1-linear-static/
    manifest.json
    single-element-membrane/
      model.inp
      expected.json
      notes.md
    single-element-bending/
      model.inp
      expected.json
      notes.md
    cantilever-strip/
      model.inp
      expected.json
      notes.md
    scordelis-lo-roof/
      model.inp
      expected.json
      notes.md

manifest.json should include:

{
  "schema_version": 1,
  "cases": [
    {
      "name": "single-element-membrane",
      "input": "single-element-membrane/model.inp",
      "expected": "single-element-membrane/expected.json",
      "analysis_type": "linear_static",
      "element": "MITC4",
      "source_solver": "Abaqus",
      "source_solver_version": "provided-by-user",
      "unit_system_note": "self-consistent",
      "tags": ["phase1", "element", "membrane"]
    }
  ]
}

Expected Result Artifact

Preferred expected.json structure:

{
  "schema_version": 1,
  "case_name": "single-element-membrane",
  "source": {
    "solver": "Abaqus",
    "version": "provided-by-user",
    "created_by": "provided-by-user"
  },
  "units": {
    "note": "self-consistent"
  },
  "comparisons": [
    {
      "path": "/results/steps/Step-1/frames/0/fieldOutputs/U",
      "entity": {"type": "node", "id": 4},
      "component": "UX",
      "expected": 0.0,
      "abs_tol": 1e-9,
      "rel_tol": 1e-7
    }
  ]
}

CSV or .rpt files may be stored as raw source artifacts, but structured JSON or HDF5 comparison artifacts should be used by automated tests.

Tolerance Policy

Use absolute and relative tolerance:

abs_error = abs(actual - expected)
rel_error = abs_error / max(abs(expected), reference_scale)
pass if abs_error <= abs_tol or rel_error <= rel_tol

Initial guidance:

• Parser and exact metadata tests: exact match.
• DOF mapping tests: exact integer match.
• Small linear algebra tests: abs_tol = 1e-12, rel_tol = 1e-10.
• Element stiffness symmetry: matrix norm tolerance around 1e-10 to 1e-9, adjusted by stiffness scale.
• Reference displacement tests: start with rel_tol = 1e-5 and refine after baseline agreement.
• Reaction equilibrium tests: use force-scale-based tolerances.

Final benchmark tolerances must be stored with each reference case.

Phase 1 Benchmark Matrix

Case	Purpose	Required Output
Parser smoke model	Verify Phase 1 keyword subset	Domain object counts, set membership
Single MITC4 membrane	Constant strain behavior	U, element strains/stresses if available
Single MITC4 bending	Bending stiffness sanity	U, rotations
Rigid body mode check	Verify near-zero internal forces without constraints	eigen/nullspace or controlled diagnostic
Constrained static sanity	Verify constrained DOF elimination	U, RF
Reaction balance	Verify full-vector reaction recovery	total load vs total reaction
Cantilever shell strip	Bending-dominated displacement convergence	tip U, support RF
Simply supported square plate	Thin shell/plate locking sensitivity	center displacement
Scordelis-Lo roof	Curved shell benchmark	midpoint vertical displacement
Pinched cylinder	Curved shell bending/membrane behavior	loaded-point displacement
Twisted beam	Warped shell and drilling sensitivity	tip displacement/rotation

Minimum Phase 1 Acceptance

Before MITC4 Phase 1 is considered credible:

• All parser smoke tests pass.
• DofManager free/constrained mapping tests pass.
• Reduced-system solve reconstructs full U.
• RF = K_full * U_full - F_full is tested.
• MITC4 element stiffness is symmetric within tolerance for linear elastic Phase 1.
• Rigid body modes do not create artificial membrane/bending stiffness beyond documented drilling stabilization effects.
• At least three stored Abaqus reference models pass: one single-element case, one simple multi-element plate/shell case, and one curved shell benchmark.

Singular System Verification

Required negative tests:

• Model with no boundary conditions should fail with a singular-system diagnostic.
• Model with missing property should fail before assembly.
• Model with load on missing node or set should fail during input/model validation.
• Model with an unconstrained isolated node should identify free untouched DOFs.
• Model with only drilling DOF instability should mention weak or unconstrained rotational DOFs.

What Not To Verify In Phase 1

• Mesh quality metrics such as aspect ratio, skew, warpage, Jacobian quality, or distortion warnings.
• Pressure loads.
• RBE2/RBE3.
• Nonlinear increments.
• Time integration.
• Thermal-stress coupling.

User Inputs Needed

• First accepted reference/ folder layout.
• At least one Abaqus .inp with solved displacement and reaction output.
• Preferred raw reference format: .dat, .rpt, .csv, .json, or .h5.
• Abaqus version used to generate the reference.
• Whether reference files are generated from Abaqus S4 only for Phase 1.