Python Concurrency

Overview

Python offers three concurrency models: threads (threading), processes (multiprocessing), and cooperative coroutines (asyncio). The Global Interpreter Lock (GIL) means threads do not run Python bytecode in parallel, so threads help I/O-bound work while processes are needed for CPU-bound parallelism. Choosing the right model is the single most important concurrency decision.

Syntax / Usage

concurrent.futures gives one unified API over both thread and process pools.

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def io_task(url: str) -> int:
    # Simulated network fetch (I/O-bound) -> use threads
    return len(url)

def cpu_task(n: int) -> int:
    # Heavy computation (CPU-bound) -> use processes
    return sum(i * i for i in range(n))

with ThreadPoolExecutor(max_workers=4) as pool:
    print(list(pool.map(io_task, ["a", "bb", "ccc"])))

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        print(list(pool.map(cpu_task, [10_000, 20_000])))

Process pools sidestep the GIL by running separate interpreters, at the cost of pickling arguments and results.

Examples

Guard shared mutable state with a Lock to avoid race conditions between threads:

import threading

counter = 0
lock = threading.Lock()

def increment() -> None:
    global counter
    for _ in range(100_000):
        with lock:            # atomic critical section
            counter += 1

threads = [threading.Thread(target=increment) for _ in range(4)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print(counter)  # 400000 — correct because of the lock

Submit work and collect results as they finish with as_completed:

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

def work(n: int) -> int:
    time.sleep(0.1)
    return n * 2

with ThreadPoolExecutor(max_workers=3) as pool:
    futures = [pool.submit(work, i) for i in range(5)]
    for fut in as_completed(futures):
        print(fut.result())

Common Mistakes

• Using threads for CPU-bound work and seeing no speedup because of the GIL
• Forgetting the if __name__ == "__main__": guard, which breaks multiprocessing on Windows/macOS spawn
• Sharing mutable state across threads without a lock, causing intermittent race conditions
• Assuming multiprocessing shares memory; arguments and results are pickled and copied
• Creating far more threads than needed instead of bounding a pool, exhausting resources

See Also

python-async-await python-iterators-protocol python-context-managers