MultiPhysicsVault/.raw/AbaqusUserSubroutineManual/AbaqusUserSubroutineManual_066.md

2.1.21 OBTAINING THE MPI COMMUNICATOR IN AN Abaqus/CFD ANALYSIS

Product: Abaqus/CFD

References

• “SMACfdUserPressureBC,” Section 1.3.1
• “SMACfdUserVelocityBC,” Section 1.3.2
• “System customization parameters,” Section 4.1.5 of the Abaqus Installation and Licensing Guide

Overview

Utility routine SMACfdUserSubroutineGetMpiComm can be called from within a user subroutine to obtain the MPI communicator used in a parallel analysis job.

Interface

#include <mpi.h>
#include <SMACfdUserSubroutines.h>
MPI_Comm comm = SMACfdUserSubroutineGetMpiComm(); 

Variable returned from the utility routine

comm

MPI communicator.

Compile and link commands for utility usage

Utility subroutine SMACfdUserSubroutineGetMpiComm requires the user to modify the compile and link commands for user subroutines to point to the MPI include files and libraries. These MPI files are not included in the release, but they are typically installed on the computer where MPI development is undertaken. The modification to the compile and link commands is done using the compile_cpp and link_sl options. The include directory for the mpi.h file must be added to the compile_cpp variable, and the location of the MPI libraries must be added to the link_sl variable. The syntax for the changes to the commands is compiler and linker dependent.

2.1.22 ENSURING THREAD SAFETY

Products: Abaqus/Standard Abaqus/Explicit

References

• “Parallel execution: overview,” Section 3.5.1 of the Abaqus Analysis User’s Guide
• “Obtaining parallel processes information,” Section 2.1.4

Overview

A number of primitives are provided to help code user subroutines for thread-parallel execution.

In thread-parallel execution mutexes can be used to protect a common block or a common file from being updated by multiple threads at the same time. Abaqus provides 100 predefined mutexes for use in user subroutines. They are referenced simply by number (1–100).

Mutexes need to be initialized before they can be used. For example, MutexInit(1) initializes mutex 1. It is best to initialize mutexes at the very beginning of the analysis in user subroutines, such as user subroutines UEXTERNALDB and VEXTERNALDB.

Once initialized, mutexes can safeguard sensitive sections of the code against concurrent access. For example, MutexLock(1) and MutexUnlock(1) will respectively lock and unlock mutex 1.

Each mutex can protect a shared resource or a logical group of resources that are always accessed together. Different mutexes are provided so that users can protect a variety of shared resources or objects. For example, one mutex can protect a file and another mutex can protect common block variables. Thus, accessing a file can happen simultaneously with updating common block variables; but no two threads can write to the file at the same time, and no two threads can update the variables at the same time.

To make data transfer and accumulation easier and safer between user subroutines in a multi-threaded environment, Abaqus provides utility functions to create dynamic storage in the form of thread-local arrays, which are private to each thread, and global arrays, which are shared. Any number of arrays of any size can be created at run time. Global arrays are accessible from all user subroutines and all threads. Thread-local arrays are private and exist only within the scope of each thread. They are accessible to all user subroutines running in that thread but not across threads. Since they are not visible to neighboring threads, they do not need to be protected from concurrent access. They are designed as a thread-agnostic replacement for the COMMON BLOCKS and SAVE variables (see “Allocatable arrays,” Section 2.1.23, for more information).

All other techniques commonly used in parallel programming can also be employed; for example, restricting all file operations only to thread 0. Oftentimes, these alternatives are preferable to using mutexes because they may provide better performance.

Interface

Fortran:
#include <SMAAspUserSubroutines.hdr>

! Initialization in UEXTERNALDB/VEXTERNALDB
call MutexInit(1) ! initialize Mutex #1
! Use in all other user subs after being initialized
call MutexLock(1) ! lock Mutex #1
< critical section : update shared variables > 

call MutexUnlock(1) ! unlock Mutex #1
C++:
#include <SMAAspUserSubroutines.h>
// Initialization in UEXTERNALDB/VEXTERNALD
MutexInit(1); // initialize Mutex #1
// Use in all other user subs after being initialized
MutexLock(1); // lock Mutex #1
< critical section : update shared variables >
MutexUnlock(1); // unlock Mutex #1 

NOTE: IDs are arbitrary chosen by the user, from the pool of 1-100. Other threads, when encountering a locked mutex, will sleep. Once the first entering thread unlocks the mutex and leaves, other threads will be able to come in and execute the critical section (one at a time).

2.1.23 ALLOCATABLE ARRAYS

Products: Abaqus/Standard Abaqus/Explicit Abaqus/CFD

Reference

• “Ensuring thread safety,” Section 2.1.22

Overview

To facilitate data accumulation and transfer between user subroutines, you can use utility functions to create your own dynamic storage in the form of allocatable arrays. Thread-local and global arrays are supported. In addition to basic types, you can also vary the precision of real arrays according to the precision of Abaqus/Explicit and define arrays of user-defined data types.

SMALocalIntArrayCreate, SMALocalFloatArrayCreate

You can create any number of thread-local or global arrays. You give each array an identifier (an arbitrary positive integer) at the time of its creation. You create an array in one user subroutine and reference it in another simply by its identifier. The arrays persist in memory until you explicitly delete them or until the analysis terminates.

Thread-local arrays

A thread-local array is a mechanism to allocate storage that is local to a thread and does not need any locking for access. In a multi-threaded environment the thread safety of these arrays stems from their design and usage: they are deliberately not shared and, thus, do not need to be protected from competing threads. In fact, one thread cannot reference a local array of another thread. They can be accessed concurrently without any locking and, thus, are faster than global arrays.

Thread-local arrays are unique in each thread. They are nonintersecting and nonoverlapping in memory, with each thread starting out with its own private copy of an array. For example, Thread 0 can have a local array with ID 1 and Thread 4 can have a local array with ID 1. Those two arrays are different and separate from each other. Similarly, it is possible to have an integer array with ID 1 and a float array with ID 1. Again, they are two different arrays. It is not possible to cross-reference these arrays across different threads. However, all user subroutines running in one thread can access all arrays of that thread. In a thread-agnostic way, these arrays are shared between user subroutines but not among threads. These routines are meant as a thread-safe replacement for COMMON BLOCKs and SAVE variables.

The following utility subroutines are available to operate on thread-local arrays:

• SMALocalIntArrayCreate, SMALocalFloatArrayCreate: to create or resize a local array.
• SMALocalIntArrayAccess, SMALocalFloatArrayAccess: to locate an existing local array.

• SMALocalIntArrayDelete, SMALocalFloatArrayDelete: to delete a local array.
• SMALocalIntArraySize, SMALocalFloatArraySize: to get the size of the array.

These utility routines are accessible from both Fortran and C/C++. The details of their interfaces are described below.

Global arrays

Global arrays are visible and accessible from all threads in an executable. To prevent race conditions, protect the creation and the write access to these arrays with mutexes (mutual exclusion locks). You can have each thread execute global array creation under a mutex protection. However, only the first thread to arrive will create the global array; the later threads will simply connect to the array already created. In addition, using mutexes on every write access will incur a performance penalty. In some situations it is possible to avoid unnecessary locking by restricting all threads to operate on nonintersecting ranges of a global array. Another alternative is to use thread-local arrays.

The following utility routines are available to operate on global arrays:

• SMAIntArrayCreate, SMAFloatArrayCreate: to create or resize a global array.
• SMAIntArrayAccess, SMAFloatArrayAccess: to locate an existing global array.
• SMAIntArrayDelete, SMAFloatArrayDelete: to delete a global array.
• SMAIntArraySize, SMAFloatArraySize: to get the size of the global array.

These arrays are global and accessible from all threads within a process but not across different MPI processes. To share data between separate MPI processes, MPI facilities must be used. Abaqus supports the full use of MPI within user subroutines.

Interface

Fortran:

INTEGER*8 SMALocalIntArrayCreate(ID, SIZE, INITVAL)
INTEGER*8 SMALocalFloatArrayCreate(ID, SIZE, INITVAL) 

Example:

#include <SMAAspUserSubroutines.hdr> 

integer a(100)
pointer(ptra, a) 

real*8 b(*)
pointer(ptrb, b) 

! create a local array with ID=1 and SIZE=100
ptra = SMALocalIntArrayCreate(1,100) 

a(1) = 11 ! use as a native Fortran array
a(2) = 22 ! use as a native Fortran array
! create a local float array with ID=1 and SIZE=100, and
! initial value = -1.0
ptrb = SMALocalFloatArrayCreate(1,100,-1.0) 

C++:

#include <SMAAspUserSubroutines.h>

// Create a local integer array of with ID=1 and size=100
int* a = SMALocalIntArrayCreate(1,100);

// Create a local float array of with ID=1, size=20, and
// initial value = -1.0
real* b = SMALocalFloatArrayCreate(1,100,-1.0); 

NOTE: Float Arrays can store both SINGLE PRECISION and DOUBLE PRECISION numbers. Internally, memory is allocated in units of 64 bits (double/real*8).

NOTE: To resize an array, simply call Create() with the same ID, but give it a new SIZE parameter. If the new size is larger, the old data are copied over to the new array. No data are lost during resizing.
For example:

! resize array with ID=1 to 300 integers
ptra = SMALocalIntArrayCreate(1,300) 

NOTE: In Create() functions, there is an optional third argument -- initial value. If not supplied, all Int arrays are initialized with INT_MAX ( 2,147,483,647 ). All Float Arrays are initialized with Signaling NANs. The values of INT_MAX and signaling NANs are accessible via the 'SMAAspNumericLimits.h' and 'SMAAspNumericLimit.hdr' header files.

Variables to be provided to the utility routine

ID

ID of the array (an integer), chosen by the user at the time of creation. Using this ID, an array can be opened in any other user subroutine.

SIZE

Size of the array as the number of ints or doubles. The maximum size for thread-local arrays is INT_MAX (2,147,483,647).

INITVAL

Initial value for each item in the array. If this argument is not supplied, in the case of an integer a large value is used; in the case of a float NAN is used.

Variable returned from the utility routine

INTEGER*8 ( address )

Returns a pointer to the array created. This pointer can be associated with a native Fortran array or native C/C++ array. Each thread will receive a different pointer. Each thread will create and hold its own array. For example, Array(1) in Thread 0 is separate from Array(1) in Thread 4. These arrays are nonoverlapping and nonintersecting in any way.

SMALocalIntArrayAccess, SMALocalFloatArrayAccess

Interface

Fortran interface:
    INTEGER*8 SMALocalIntArrayAccess(ID)
    INTEGER*8 SMALocalFloatArrayAccess(ID)

Example:
#include <SMAAspUserSubroutines.hdr>
    integer a(100)
    pointer(ptra, a)

C Locate local Array(1) and associate a native array pointer with it
    ptra = SMALocalIntArrayAccess(1)