In all the allocation functions seen so far, there is no limit imposed by the software on the amount of memory reserved for the heap. While overflowing the stack is often hard to prevent, and extremely difficult to recover from, running out of available heap memory can more often be gracefully handled by the application, for example, by canceling or postponing the operation that required the allocation. In more complex multi-threaded systems, the operating system could actively react to the memory shortage by terminating non-vital processes to free up memory for new allocations. Some advanced systems using page-swapping mechanisms, like Linux, may implement overcommit on the available memory. This mechanism guarantees that memory allocations never fail, and malloc will never return NULL to indicate a failure.

Memory-consuming processes in the system may be instead terminated at any time by a kernel thread, the out-of-memory killer, to make space for new allocations from other less resource-consuming processes. On an embedded system, especially if there is no multi-threading, the best choice is to have the allocator return NULL when there is no physical space left on the heap, so that the system can keep running and the application can possibly recover by recognizing the out-of-memory episode. The section in memory dedicated to the heap can be limited by exporting the address for its upper boundary in the linker script, for example:

_heap_end = ORIGIN(RAM) + LENGTH(RAM);

The backend for the newlib malloc implementation can account for the newly introduced upper bound in the _sbrk() function:

void * _sbrk(unsigned int incr) {
    static unsigned char *heap = NULL;
        void *old_heap = heap;
    if (((incr & 0x03) != incr)
        incr = ((incr >> 2) + 1) << 2;
    if (old_heap == NULL)
        old_heap = heap = (unsigned char *)&_start_heap;
    if ((heap + incr) >= &_end_heap)
        return (void *)(-1);
    else
        heap += incr;
    return old_heap;
}

The special value (void *)(-1) that is returned by sbrk in case of memory shortage for heap allocation indicates to the calling malloc that there is not enough space to perform the requested allocation. malloc will then return NULL to the caller.

It is very important in this case that the caller always checks the return value at each invocation of malloc(), and that the application logic is able to correctly detect that the system is out of memory, and react in the attempt to recover from it.