What for actually does
An iterable is anything you can call iter() on: a list, a string, a dict, a file. An iterator is what you get back — something with __next__ that raises StopIteration when it is finished.
The distinction matters because an iterator is consumed. A list can be looped over ten times; the iterator from zip, map or a generator is empty after the first pass, and the second loop silently does nothing.
for item in things:
use(item)
# is exactly this:
it = iter(things) # things.__iter__()
while True:
try:
item = next(it) # it.__next__()
except StopIteration:
break
use(item)yield
A function containing yield does not run when you call it. It returns a generator, and the body runs only as values are pulled out — resuming each time exactly where it stopped, with every local variable still in place.
That suspension is the whole feature: it turns a function into something that produces a sequence lazily, with no list in the middle.
def countdown(n):
while n > 0:
yield n # hand a value out, and pause right here
n -= 1
# falling off the end raises StopIteration
c = countdown(3)
next(c) # 3
next(c) # 2
list(c) # [1] — the two already taken are goneWhy it matters
Generators compose into a pipeline: each stage pulls one item from the one before it, so a chain of five transformations over a ten-gigabyte file still holds one line at a time. This is the single most useful thing in this lesson.
# reads the whole file into memory
lines = open("huge.log").readlines()
errors = [l for l in lines if "ERROR" in l]
# constant memory, whatever the file size
def errors(path):
with open(path, encoding="utf-8") as f:
for line in f:
if "ERROR" in line:
yield line.rstrip("\n")
for line in errors("huge.log"):
...yield from, and returning
yield from hands control to another iterable and passes its values straight through — which makes recursive generators read as naturally as recursive functions.
def flatten(nested):
for item in nested:
if isinstance(item, list):
yield from flatten(item) # delegate to another generator
else:
yield item
list(flatten([1, [2, [3, [4]]]])) # [1, 2, 3, 4]
def counted(items):
n = 0
for item in items:
n += 1
yield item
return n # ends up on StopIteration.valueitertools
groupby groups consecutive equal keys, not all equal keys; unsorted input gives you several groups with the same key and a bug that only shows up on real data. Sort by the same key first, or use a defaultdict.
from itertools import islice, chain, groupby, count, cycle, accumulate, pairwise
islice(gen, 10) # the first ten, without building a list
chain(a, b, c) # one sequence after another
accumulate([1, 2, 3]) # 1, 3, 6 — running totals
pairwise([1, 2, 3]) # (1, 2), (2, 3)
count(1) # 1, 2, 3, … forever
groupby(sorted(rows, key=k), key=k) # runs of equal keys — SORT FIRST