By default, generators provide lazy evaluation: they don't perform a process action until explicitly called. This is a valuable trait when working with large datasets, such as processing millions of calculations. If you attempted to store all the results in memory at one time, that is, via a normal function call, you could run out of space.

Another option is when you don't know if you actually need to use all the values returned. There is no need to perform a calculation if you won't use it, so you can reduce the memory footprint and improve performance.

Still another option is when you want to call another generator or access some other resource, but you want to control when that access occurs. If you don't need an immediate response, for example, you don't want to store the result in a temporary variable, then being able to run the generator at the desired time can help the design process.

One great place for generators is in replacing callback functions. Callback functions are called by something else, do their processing, and occasionally send a status report back to the caller. This has the inherent problems of full-processing, that is, everything is processed at one time and stored in memory for access.

If a generator is used instead, the same processing occurs but there is no status report to the caller. The generator function simply yields when it wants to report. The caller gets the generator's result and deals with the reporting work as a simple for loop that wraps the generator call. If, for some reason, you still want to have the generator provide everything at once, you can simply wrap a generator call in list.

Python uses both of these cases for different versions. In Python 2, os.path.walk() uses a callback function, whereas Python 3 has os.walk(), which uses a filesystem-walking generator.

Finally, there is one last trick that can help with Python performance. Normally, list comprehensions are used to quickly iterate through a list, as in the following example:

l = [x for x in foo if x % 2 == 0]

A simple generator can be created in a similar fashion. Basically, you just replace the square brackets with parentheses:

g = (x for x in foo if x % 2 == 0)

Once you have that, you can use the generator instance within a simple for loop:

for i in g:

The benefit of using a generator instead of a list comprehension is that intermediate memory storage is not required. The values are created on demand, as it were, so the entire list is not dumped to memory at one time. This can achieve significant speed increases and reduce memory usage, depending on the program.