One option to optimize memory usage is to encode the images using some sort of compression (that is, PNG) on this case the TFRecord will be smaller, but you still need to uncompress the data before use, which may take some time. What is done in practice is to use another CPU core to alleviate computing.