OpenMP, MPI and pthreads

OpenMP and MPI are complementary.  OpenMP is used to optimize performance of a program running on a single machine with multiple CPUs, i.e an SMP.  MPI uses a message passing mechanism to divide a big problem into smaller problems. Each small problem runs on a separate machine, in parallel with the other.  It then uses a MapReduce paradigm to collect the result and solve the big problem. The advantages of the MapReduce paradigm will be explained separately in another post.

To combine them to solve a big problem, first use MPI to divide the problem into parts which can be solved on a single machine. Now to solve this single-machine problem optimally, use OpenMP to efficiently allocate the work among the multiple CPUs on that machine. This will result in a multithreaded program.

Why should we use OpenMP over pthreads? Both of them use threads to divide the work, right? Well, OpenMP enables you to specify a program at a higher abstraction than pthreads. You need not explicitly write the synchronization part of the code which is the trickiest part in multi-threaded computing.

All three, OpenMP, MPI, and pthreads are portable while internally taking advantage of the underlying OS and hardware architecture.

Of course, there are cons here. MPI is extremely difficult to learn and code and involves many low level implementation details. You may not be able to entirely substitute pthreads with MPI.  MPI parallelizes only loops; any producer-consumer kind of situation has to be explicitly dealt with using pthreads.