Programming model of parallel computing - Message Passing Interface (MPI)

MPI is a standard and portable communication interface. MPI provides rich functional interfaces for parallel communication and parallel file I/O access.
MPI is a programming model that supports multi presentation many data (MPMD). It is a library based system implemented by C and Fortran.

MPI Core:
	1, MPI The core of is interprocess communication, which follows the interprocess sequential communication model CSP，Each process runs in its own space.
	2, MPI Declared variables, for example int b[10]Is the private variable of each process b They are independent of each other in different processes.
	3, MPI The two main communication modes are
		Point to point communication, that is, communication between two processes
		Aggregate communication, that is, a group of inter process communication

each MPI A process has a process number in each group of processes. Starting from zero, the combined process numbers of a group of four processes are{index| index in 0, 1, 2, 3}
be-all MPI All communication takes place in the communication domain.
Communication domain contains
	A set of processes
	A hidden communication text (It is used to ensure the consistency of messages and libraries)
The communication domain object is a hidden type called a handle.
	stay C In language, the communication domain handle is MPI_Comm type
	stay Fortran In language, the communication domain handle is Type(MPI_Comm) Type (2008) or INTEGER Type (early)
stay MPI In the program, there are two predefined communication domains
	MPI_COMM_WORLD	Include all MPI process
	MPI_COMM_SELF	Contains only one process running an instance program
MPI Provide various communication domain functions
	MPI_Comm_rank	Used to get the process number			MPI_Comm_rank(*a, *b);
	MPI_Comm_size	Number of processes used to get the communication domain	MPI_Comm_size(*a, *b);
	And a function for creating a new communication domain
	MPI_Init		For initialization MPI program			MPI_Init(*a, *b);
	MPI_Finalize	Used to end MPI program			MPI_Finalize(void)

Except for a few functions, most MPI Function needs to be in MPI_Init(or MPI_Init_thread)After function and MPI_Finalize Previous call
 In parallel programs, if MPI_Init Function arguments cannot be retrieved from main To get the command line parameters from the function, you need to pass MPI Specify

/* get the MPI process id and the number of process */
#include "mpi.h"
#include <stdio.h>

int main(int argc, char *argc[]){
	int wrank, srank, wsize;
	MPI_Init(&argc, &argv);
	MPI_Comm_rank(MPI_COMM_WORLD, &wrank);
	MPI_Comm_size(MPI_COMM_WORLD, &wsize);
	MPI_Comm_rank(MPI_COMM_SELF, &srank);
	printf("World rank %d, world size %d, self rank %d\n", wrank, wsize, srank);
	MPI_Finalize();
	return 0;
}

/* end class */
MPI The execution between processes is independent of each other. In the above code, printf()Statements are executed in random order, and there is no guarantee that one line will be output in turn.

Point to point communication
The sending process needs to specify the sending data, the process number of receiving data and the communication domain. For some old message delivery systems, each message also needs a message number, and the data type is a single precision non negative integer
The receiving process needs to specify the data receiving address, data source process number, communication domain and message number. In addition, it may also need to provide a message status parameter to save the status information of data reception

For early messaging systems and most files I/O Library that specifies the data buffer with a binary array containing the address and the number of bytes
MPI Expand the binary array into a ternary array, which contains address, data type and number of data.

use C The language specifies a buffer of 10 certificate type data, MPI Statement is(address, 10, MPI_INT)
	This ensures that the compiler does not need to know the specific number of bytes of no data type
	It also ensures that the method makes MPI The program can run on mixed hardware platform

/* MPI Allows you to specify discontinuous data buffers in memory */
int msg(MAX_MSG_SIZE);
...
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
if (rank == 0){
	for (i=1; i<size; i++)
		MPI_Send(msg, msgsize, MPI_INT, i, 0, MPI_COMM_WORLD);
} else {
	MPI_Recv(msg, MAX_MSG_SIZE, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
	doWork(msg);
}
/* end function */

stay MPI_Send In the function parameter, the process number in the communication field specifies that the data is sent to the specific process and provides the communication text
 The message number is a non negative integer type, and the maximum value depends on MPI But at least 32767
MPI_Recv Receiving function and MPI_Send The parameters of the function are similar, except that the receive function allows you to specify that the size of the received data buffer is larger than the size of the actually sent data.
The message number and data source process number can be specified by specific numbers, or wildcard type numeric values can be used
	MPI_ANY_TAG			Allows users to send data of an additional certificate type
	MPI_ANY_SOURCE		The process number is determined by non deterministic algorithm
	MPI_STATUS_IGNORE	If status information is not required, it is used
 The above code is from process 0 MPI_COMM_WORLD An example of a program in which all other processes in the communication domain send the same data.
	The program is MPI Basic or standard transmission mode, MPI Other data transmission modes are also provided, such as:
		0, Synchronous transmission mode
		1, Ready to send mode
		2, Send buffer mode
		3, Non blocking communication mode

data type
One of the characteristics of MPI is that all communication functions have a data type parameter, which is used to describe the type of data sent and received, instead of describing it only in bytes in the old system
For example, if a set of integer data is transmitted, the data type of MPI function should be set to MPI in C language_ Int, MPI in Fortran_ INTEGER
The purpose of using data type parameters is to:
0. In heterogeneous environment, such as heterogeneous computing and systems with different byte storage order or different basic data type length, ensure smooth communication
When the data type is specified, MPI can be internally converted to the corresponding number of bytes.
1. It enables users to describe the data stored discontinuously in memory and realize data transmission through a separate MPI function call
MPI provides a variety of data type constructors, which can be used to describe the data stored discontinuously in memory. The data stored discontinuously in memory can be described and transmitted through the data type constructor
If the high-quality MPI implementation method is adopted, the performance will be better than the communication mode of data packaging and unpacking in transmission, or the mode of converting discontinuous data into continuous data and then transmitting
The data type of MPI can describe various data in memory. The predefined data types in MPI are similar to the basic data types in programming languages, such as MPI_FLOAT,MPI_COMPLEX
The new data type can be constructed from the old data type through different layouts
MPI_Type_vector(count, blocklength, stride, oldtype, newtype)
There are count data blocks in the new data type, and there are blocklength consecutive oldtype data in each data block. The length of each data block is stripe.
MPI also provides other more general data constructors:
MPI_Type_indexed than MPI_Type_vector has a wider scope of application and can be used to define data with different numbers of oldtype types and offsets in each data block
MPI_Type_create_struct is the most widely used, and can also define data of different data types in each data block.
MPI provides functions to define data subarrays and distributed arrays, which are most widely used in parallel file I/O.
The data constructor provided by MPI can be called recursively and can describe any data type

/* Send a column of data in a 5 * 7 two-dimensional array */
...
MPI_Type_vector(5,1,7, MPI_DOUBLE,&newtype);
MPI_Type_commit(&newtype);
MPI_Send(&buffer[2], 1, newtype, dest, tag, comm);
MPI_Type_free(&newtype);
...

Non blocking communication
Non blocking communication is one of the important characteristics of MPI. After the initialization of non blocking communication, there is no need to wait for the completion of communication before operation. This feature has two advantages
1. Non blocking communication realizes the asynchronous execution of communication and other computing operations
2. For complex communication modes, strict communication sequence and memory space management are not required.
/*An example of a complex communication pattern, where partner is the process number of other processes*/
MPI_Send(buf, 100000000, MPI_INT, partner, 0, MPI_COMM_WORLD);
MPI_Recv(rbuf, 100000000, MPI_INT, partner, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

MPI_Send Need to buf Send 100 million integers in to partner Progress.
	If the sending operation fails to apply for memory, the program will be in a waiting state until the memory application is successful
	call MPI_Recv After accepting the operation, you also need to wait for the memory application to succeed
 In the above program example, both processes call first MPI_Send Execute the sending operation, so the program may always be in the waiting state. The program is unsafe to run. Can the program run
 The correct execution depends on the memory application and the reasonable matching of sending and receiving operations

To solve the deadlock problem, there is a classic solution, which eliminates inter process waiting by changing the sending and receiving order. This method is only applicable to simple communication mode or clear communication mode during program compilation
 The complex communication mode sometimes determines the communication mode when the program is running, so this method is not used.
if (rank & 0x1){
	/* Odd rank, do send first */
	MPI_Send(...);
	MPI_Recv(...);
} else {
	/* Even rank, do receive first */
	MPI_Recv(...);
	MPI_Send(...);
}
For complex communication modes, the solution is before the end of communication MPI The communication operation can be returned without waiting for the end of the communication process.
stay MPI In, this communication mode is called non blocking communication,
	MPI_Send The non blocking send function of is MPI_Isend
	The parameter setting of non blocking communication mode is similar to that of blocking communication mode, but an output parameter is added MPI_Request
	MPI_Request Is a handle used to query the operation status and wait for the operation to complete.
/* Program example of calculating transmission data through communication and calculation overlap */
int msg[MAX_MSG_SIZE];
MPI_Request *r;
...
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
if (rank == 0){
	r = (MPI_Request *) malloc((size-1),* sizeof(MPI_Request));
	if (!r)	...error
	for (i=1; i<siz; i++)
		MPI_Isend(msg, msgsize, MPI_INT, i, 0, MPI_COMM_WORLD, &r[i-1]);
	... perform some work
	MPI_Waitall(size-1, r, MPI_STATUSES_IGNORE);
	free(r);
} else {
	MPI_Request rr;
	MPI_Irecv(msg, msgsize, MPI_INT, 0, 0, MPI_COMM_WORLD, &rr);
	... perform some work
	MPI_Wait(&rr, MPI_STATUS_IGNORE);
	doWork(msg);
}
In the above program, when the user needs to call a test or wait for the function to complete the communication operation, MPI_Wait Function blocks program execution until the communication operation is completed
MPI_Test The function does not block the execution of the program. The function returns immediately after execution to test whether the communication operation is completed
 At the same time, MPI Group test and group wait functions are provided to test and wait for multiple communication operations at the same time, such as MPI_Waitall

In many applications, the same communication mode needs to be repeated. For repeated communication requirements, MPI Provide repeated non blocking communication, for example, by using
MPI_Send_init and MPI_Recv_init Function to realize repeated communication initialization. In the repeated communication mode, message communication is not started after communication operation initialization,
Message communication by MPI_Start Function is triggered. If the message communication ends, call MPI_Wait The returned result can be obtained by calling MPI_Start Start message communication again,
Called when message communication is not required MPI_Request_free Release communication object

Aggregate communication
In addition to peer-to-peer message communication between processes, MPI also provides a large number of functions for communication between a group of processes. Communication based on a group of processes is called aggregate communication,
The process of aggregation communication needs to call the same aggregation communication function. Through a large number of aggregation communication algorithms, aggregation communication adopts the aggregation communication implementation algorithm with high performance,
Therefore, this communication mode is widely used in parallel programming

There are three types of aggregated communication:
	1, synchronization
		MPI_Barrier It is used to communicate with all functions in the process MPI_Barrier The process of the function must wait until all processes are completed
	   	Execute to MPI_Barrier Function, all processes will enter the next program operation.
	2, data migration 
		MPI Provide a large number of common data migration functions, for example, MPI_Bcast Function sends data from the root process to other processes in the communication domain. MPI_Scatter function
		Distribute the data in the data buffer to other processes in the communication domain. MPI_Gather Function and MPI_Scatter On the contrary, this function takes data from other processes
		Collected on the root process. MPI_ALLgather Function and MPI_Gather The function is similar, except that other processes in the communication domain also collect the same data except the root process,
		MPI_Alltoall Function to realize the full exchange of process data in the group. Each process can send different data to other processes or receive the data sent by other processes.
		P0{A, , , }								{A, , , }
		P1{ , , , }								{A, , , }
						Broadcast(MPI_Bcast) ->		
		P2{ , , , }								{A, , , }
		P3{ , , , }								{A, , , }

		P0{A,B,C,D}								{A, , , }
		P1{ , , , }								{B, , , }
						Scatter ->
						<- Gather
		P2{ , , , }								{C, , , }
		P3{ , , , }								{D, , , }

		P0{A, , , }								{A,B,C,D}
		P1{B, , , }								{A,B,C,D}
						Allgather ->
		P2{C, , , }								{A,B,C,D}
		P3{D, , , }								{A,B,C,D}

		P0{A0,A1,A2,A3}							{A0,B0,C0,D0}
		P1{B0,B1,B2,B3}							{A1,B1,C1,D1}
						Alltoall ->	
		P2{C0,C1,C2,C3}							{A2,B2,C2,D2}
		P3{D0,D1,D2,D3}							{A3,B3,C3,D3}

	3, Aggregation operation
		MPI Provide reduction and scanning operations to realize data arithmetic operations, such as solving minimum and maximum values, summation, logic and operation, and other user-defined operations
		Calculation algorithm, MPI_Reduce The function provides reduction operation, and the results are saved in the root process, MPI_Allreduce Save the results in all processes,
		MPI_Scan Function for each process i，For process 0, ..., i The data on is reduced, and the result exists i In the process, MPI_Exscan Wrong function
		Reduce the data on its own process, MPI_Reduce_scatter Function contains both reduction and distribution functions

		/* More efficient data transmission communication mode, MPI_ The reduce function is used to sum the data on all processes and save the results on the 0 process */
		int result[MAX_MSG_SIZE];
		...
		MPI_Bcast(msg, msgsize, MPI_INT, 0, MPI_COMM_WORLD);
		if (rank != 0)
			doWork(msg, result);
		MPI_Reduce(MPI_IN_PLACE, result, msgsize, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);

All aggregate communication functions provide two communication modes: blocking and non blocking. The above aggregate communication functions are blocking communication. The aggregate communication function returns only after the communication is completed,
Non blocking communication functions are added to the name“ I"， for example MPI_Ibcast, MPI_Ireduce，The non blocking aggregate communication function returns immediately after initializing the communication without waiting
 After the communication operation is completed, the user obtains the completion of the communication operation or waits for the completion of the communication operation by calling the test or waiting function, MPI Provide aggregation with a large amount of computing and communication overlap
 Communication functions, e.g MPI_Ibarrier function

One-Side Communications
In point-to-point communication and aggregation communication, the sending process and receiving process participate in the communication process together. MPI provides another communication mode, that is, unilateral communication mode, which can be realized by a single process
It is mainly used for sending and receiving data in single side communication such as UPC / MEM (also known as single side communication and remote communication such as shrma / MEM)

The unilateral communication model is mainly divided into three parts:
	1, establish MPI_Win Window object, which is used for the memory area that can be accessed by other processes.
	2, Inter process data movement mode, including data movement modes such as reading, writing and updating from remote processes.
	3, It can wait or check the completion of unilateral communication. Each part of the unilateral communication model has a unique MPI Format.
MPI There are four ways to create a memory access window to meet different application requirements:
	1, MPI_Win_create
	2, MPI_Win_allocate
	3, MPI_Win_shared
 The three functions are aggregation methods, which are used to specify the memory area read and written by other processes, MPI_Win_create_dynamic Function allows a separate process to request and free its memory space

The type of unilateral communication function is relatively simple, which can be divided into read, write and update. The simplest unilateral communication function is through MPI_Put Function to write data to the remote process through MPI_Get Function from remote
 Process reads data, and MPI_Accumulate Function utilization MPI The predefined reduction operation updates the data in the remote process.
because MPI Provide non blocking unilateral communication mode, so it is necessary to provide additional function check or wait for non blocking communication operation, MPI Three methods are provided to complete non blocking unilateral communication:
	1, MPI_Win_fence Function, which belongs to the group call mode, in a pair of MPI_Win_fence Call the function to complete the inter process data communication operation when MPI_Win_fence After the function is called, the	 Some unilateral communication operations have been completed. For example, the data in the data buffer has been changed, and the memory data in the local process has completed the defined data operation, which can be freely accessed or changed by the local process
	2, MPI The second unilateral communication completion method provided is relative to MPI_Win_fence The synchronization method is more general, which is called scalable synchronization method, such as MPI_Win_post,MPI_Win_start,
	   MPI_Win_complete,MPI_Win_wait Functions that specify MPI Data window, the source process and target process of unilateral communication, because these functions do not need to communicate with all processes in the domain
	   Blocking synchronization mode is used on, so the parallel scale is scalable, whether it is MPI_Win_fence Synchronization mode, or MPI_Win_post And other scalable synchronization methods need to be in the source process
	   And target processes, because both methods are automatic target synchronization.

	   /* Program example of using MPI unilateral communication to realize data calculation (reduction) operation on a process */
	 	MPI_Win_allocate((rank==0)?result:NULL, (rank==0)?msgsize:0, sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &result, &win);
	   	MPI_Win_fence(0, win);
	   	...
	   		doWork(msg, myresult);
	   		MPI_Accumulate(myresult, msgsize, MPI_INT, 0, 0, msgsize, MPI_INT, MPI_SUM, win);
	   	...
	   	MPI_Win_fence(0, win);
	   	...	process 0 has data in result
	   	MPI_Win_free(&win);

	3, MPI The third one-sided communication completion method provided only needs to make function calls through the source process, which is called passive target synchronization, in which process 0 is in while No execution in the loop MPI Function,
	   By calling MPI_Win_lock and MPI_Win_unlock Function to ensure that the data on the source process and the target process are completed MPI RMA Operation. MPI_Win_lockall and MPI_Win_unlockall
	   Function allows a process to execute data on all other processes in the memory window RMA Operation. At the same time, MPI_Win_flush Function can also be used for passive target synchronization to realize the synchronization of the target process RMA
	   Operation. MPI Return is also provided MPI_Request There are three types of operation functions: read, write and update of objects. User available MPI_Test and MPI_Wait The completion of local operation of function production reduction is not required
	   Wait for the next RMA Synchronous call operation.

	   	/* A program example of computing reduction operation using passive target synchronization in MPI unilateral communication on a single process */
	   	MPI_Win_allocate((rank==0)?result:NULL, (rank==0)?msgsize:0, sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &result, &win);
	   	while (not done) {
	   		...
	   		doWork(msg, myresult);
	   		MPI_Win_lock(MPI_LOCK_SHARED, 0, 0, win);
	   		MPI_Accumulate(myresult, msgsize, MPI_INT, 0, 0, msgsize, MPI_INT, MPI_SUM, win);
	   		MPI_Win_unlock(0, win);
	   		...
	   	}
	   	MPI_Barrier(MPI_COMM_WORLD);	// ensure all processes done
	   	if (rank == 0){
	   		MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 0, 0, win);
	   		printf("Result is %d\n", result[0]);
	   		...
	   		MPI_Win_unlock(0, win);
	   	}
	   	MPI_Win_free(&win);

	   In the above code, it should be noted that process 0 is accessing MPI Call before local data in the window MPI_Win_lock Function, by calling this function, ensure that all
	   All pending memory operations on the target process are executed, which is a detail that needs special attention in the shared and remote memory access programming model and is often misunderstood by compilers. MPI
	   The defined memory model ensures that users get continuous and correct data, and can work normally even on computer systems that cannot achieve cache consistency.
	   It is worth mentioning that, MPI In addition to the basic functions that meet the minimum memory accuracy requirements of unilateral communication, more stringent functions are also provided to ensure MPI The program can
	   Running on most computer systems, MPI-3 Based on the existing memory access model, the "unified memory model" is added, which is now called "distributed memory model". Users can
	   adopt MPI_Win_get_attr Function to know whether the computer system supports the model. In addition, MPI-3 Support atomic read, write and modify operations commonly used in many parallel algorithms,
	   for example MPI_Fetch_and_op Function to read and add functions, MPI_Compare_and_swap Realize the function of comparison and exchange. At the same time, MPI-3 Memory sharing windows can be created,
	   stay RMA Based on the operation, the function of direct memory access is added.

Parallel I/O
Many parallel science applications need to read and write a large amount of data from files, such as reading input grid, program pause, saving data / restarting pause program, data analysis and virtualization. If file I/O efficiency
If it is not high, the program performance will encounter a bottleneck. MPI provides application developers with a concise parallel I/O function interface, which can significantly improve the file I/O access performance.
The I/O interface provided by MPI retains the I/O operations commonly used in POSIX files, such as open, close, find, read and write operations. At the same time, MPI provides many advanced features, such as discontinuity
Access data in memory and files, aggregate I/O, and pass performance tips to MPI implementation.

	 _________________________________________________
	|_______|_______|______|___________________|______|
		P0		P1		P2			...			P(n-1)

The above figure is a simple program example in which each process reads data in parallel from different locations of a shared file. MPI Provides a variety of ways to read data.

/* Read data using stand-alone file I/O and file pointers */
MPI_File fh;
...
rc = MPI_File_open(MPI_COMM_WORLD, "myfile.dat", MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);
rc = MPI_File_seek(fh, rank*bufsize*sizeof(int), MPI_SEEK_SEt);
rc = MPI_File_read(fh, msg, msgsize, MPI_INT, MPI_STATUS_IGNORE);
MPI_File_close(&fh);

In the above program example, each process passes MPI_File_open Function opens a file in which the first parameter of the function MPI_COMM_WORLD Represents the communication domain, and the second parameter represents
 The path and name of the file to be opened. The third parameter represents the file opening method, and the fourth parameter passes through the MPI_INFO Attach a pair of key values to an object I/O Transfer information in implementation,
The fifth parameter is I/O The handle returned by the operation, which can be used for subsequent operations I/O Operation, in the fourth parameter MPI_INFO In, information such as file fragmentation and internal buffer size can be transferred for optimization.
Each process calls MPI_File_seek Function to move the file pointer to the corresponding position according to the reading displacement, and then change the program instance. The file pointers of each process are independent of each other.
MPI Pointers to shared files are also provided I/O Operation, that is, multiple processes share a file pointer through MPI_File_read Function that will msgsize Read integer type data from file to memory
 In the buffer, finally by calling MPI_File_close Function to close the file and end I/O Operation.

MPI The second way to read the file content is to avoid using the file pointer and read the data directly according to the offset. By calling MPI_File_read_at Function to display the offset and
 The line file reading function only needs to specify the offset in the function parameter, and there is no need to call MPI_File_seek Function. Since this function does not need a file pointer, it is easy to implement
I/O Operations are thread safe.

MPI_File_read and MPI_File_read_at It is called independently I/O Function instead of aggregate calling function. Each process can call these two functions independently, without all processes having to call them
 The same function. therefore MPI How many processes will call these two functions and cannot implement these two functions I/O Operation for performance optimization.

MPI Aggregation is also provided I/O Read and write mode, and in aggregation I/O Add to the name in read-write mode"_all"Keywords, in aggregation I/O In reading and writing mode, MPI Performance can be optimized, usually recommended
 Developers use aggregate I/O Reading and writing methods,

/* Read file data in parallel, in which each process reads different data in the file respectively */
MPI_File fh;
...
rc = MPI_File_open(MPI_COMM_WORLD, "myfile.dat", MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);
rc = MPI_File_set_view(fh, rank*bufsize, MPI_INT, MPI_INT, "native", MPI_INFO_NULL);
rc = MPI_File_read_all(fh, msg, msgsize, MPI_INT, MPI_STATUS_IGNORE);
MPI_File_close(&fh);

The program example adopts the third method to read the file data shown in the previous group of codes, that is, the "file viewport" method, MPI_File_set_view Function to specify a file I/O Accessing viewports,
That is, specify the data that the file needs to read and write or skip access. File viewports are represented by the offset, base type, and file type ternary parameters:
	1, The offset from the start position of the file header in the viewport, for example, is specified
	2, Basic type description file I/O Basic data types accessed
	3, File types are composed of basic types
 In the above code, the offset specified on each process is rank*msgsize，The basic type is MPI_INT，The file type is also MPI_INT，The parameter after the file type is used to specify the data
 Expression."native"Indicates that the storage mode of file data is consistent with that of memory. MPI_File_set_view The last parameter in the function is used to pass runtime information. It can be used when reading text
 Single or aggregate. Each process reads from the file viewport msgsize An integer is added to the memory buffer. Because the offset in the file viewport is different, each process is from the offset in the file
 Read file data from different locations.

Other features
MPI provides a rich function library for large-scale parallel software development and application. MPI provides an innovative feature, that is, it provides other execution methods for each MPI function, so as to realize in the original
Execute special code in MPI function. If PMPI interface is adopted, there are other execution methods of MPI function, namely MPI analysis interface. Such as PMPI_Bcast function is MPI_ Another method of BCAST function
In terms of execution mode, the functions provided by PMPI are consistent with the parameters and functions of MPI functions. The difference is that users can define their own MPI functions through PMPI

/* Through the analysis interface provided by MPI, get all processes to execute MPI_ Wait time before BCAST function */
#include "mpi.h"
#include <stdio.h>

static double syncTime = 0.0;

int MPI_Bcast(void *buf, int len, MPI_Datatype dtype, int root, MPI_Comm comm){
	double t1;
	t1 = MPI_Wtime();
	PMPI_Barrier(comm);
	syncTime += MPI_Wtime() - t1;
	return PMPI_Bcast(buf, len, dtype, root, comm);
}

int MPI_Finalize(void){
	printf("Synchronization time in MPI_Bcast was%.2e seconds\n", syncTime);
	fflush(stdout);
	return PMPI_Finalize();
}

In the above functions, when calling MPI_Bcast Function, the user-defined MPI_Bcast Function, that is, count and print the execution of all processes MPI_Bcast Wait time before function.

MPI support MPI Mixed programming and thread level parallel programs need to be clearly specified by the user MPI Interaction between processes and threads. MPI Four thread interaction security levels are provided:
	1, MPI_THREAD_SINGLE，		A process can only have one thread
	2, MPI_THREAD_FUNNELED, 	A process can have multiple threads, but only MPI The initialized thread can call MPI function
	3, MPI_THREAD_SERIALIZED, 	A process can have multiple threads, but only one thread can call at the same time MPI function
	4, MPI_THREAD_MULTIPLE, 	A process can have multiple threads, which can be called at the same time MPI function 

User must call MPI_Init_thread Function to specify the thread safety level, MPI The implementation returns the supported thread safety level. Users need to be responsible for managing thread safety levels, MPI
 The thread safety level of the provided function does not necessarily support higher than MPI_THREAD_SINGLE Level, but MPI Thread safe function support marked in MPI_THREAD_MULTIPLE Level,
When writing thread level parallel programs, users should not exceed MPI The thread safety level of the function to avoid potential program running problems.

MPI Provide dynamic creation of processes and independent inter process communication functions, such as through MPI_Comm_spawn perhaps MPI_Comm_spawn_multiple Function to generate child processes through
MPI_Comm_connect and MPI_Comm_accept perhaps MPI_Join Realize independent inter process communication.

MPI Provide neighbor aggregation communication functions, such as MPI_Neighbor_allgather and MPI_Neighbor_alltoall Function, etc., in Cartesian topology or image virtual processing topology
 Realize the process aggregation communication between neighbors. The neighbor aggregation communication is mainly used in the programs that need the communication between adjacent processes and the large-scale sparse communication mode of Dalian process.