Compaction is a method to reduce heap fragmentation. Through compaction, objects are moved to form contiguous "live regions" of heap space, so that the case where there are plenty of small "holes" of free space disappears. Full heap compaction is hard to do without stopping the execution of a concurrently running Java program.
See also Fragmentation.