A mutex can be one of four types, the default usually—but not always (it depends upon the implementation)—being the normal mutex. The type of mutex used affects the behavior of the lock and unlock. The types are: PTHREAD_MUTEX_NORMAL, PTHREAD_MUTEX_ERRORCHECK, PTHREAD_MUTEX_RECURSIVE, and PTHREAD_MUTEX_DEFAULT.

The system man page on pthread_mutex_lock(3) describes the behavior depending on the mutex type with a table; for the reader's convenience, we have reproduced the same here.

If a thread attempts to relock a mutex that it has already locked, pthread_mutex_lock(3) shall behave as described in the relock column of the following table. If a thread attempts to unlock a mutex that it has not locked or a mutex which is unlocked, pthread_mutex_unlock(3) shall behave as described in the Unlock When Not Owner column of the following table:

If the mutex type is PTHREAD_MUTEX_DEFAULT, the behavior of pthread_mutex_lock(3) may correspond to one of the three other standard mutex types, as described in the preceding table. If it does not correspond to one of those three, the behavior is undefined for the cases marked †.

The relock column directly corresponds to what we described earlier in this chapter as the self-deadlock scenario, such as, what effect attempting to re-lock an already-locked lock (poetic wording, perhaps?) will have. Clearly, except for the recursive and error check mutex case, the end result is either undefined (which means that anything can happen!) or a deadlock indeed.

Similarly, attempting to unlock a mutex by any thread except the owner either results in an undefined behavior or an error. 

One might wonder: why does the locking API behave differently—in terms of error return or failures—depending on the type of the mutex? Why not just have one standard behavior for all types and thus simplify the situation? Well, it's the usual trade-off between simplicity and performance: the way it's implemented allows, for example, a well-written, programmatically proven correct real-time embedded application to forgo extra error checking and thus gain speed (which is especially important on critical code paths). On the other hand, in a development or debug environment, the developer might choose to allow extra checking to catch defects before shipping. (The man page on pthread_mutex_destroy(3) has a section entitled Tradeoff Between Error Checks and Performance Supported that describes this aspect in some detail.)

The pair of APIs to get and set a mutex's type attribute (the first column in the preceding table) are quite straightforward:

include <pthread.h>
int pthread_mutexattr_gettype(const pthread_mutexattr_t *restrict attr,     int *restrict type);
int pthread_mutexattr_settype(pthread_mutexattr_t *attr, int type);