Rust for Pythonistas

Lesson 25 of 26

Rust inside Python with PyO3

This is where many Python developers first meet Rust: a slow part of a Python program is rewritten in Rust, and Python keeps calling it like any other module. pydantic-core, polars, orjson and tokenizers all work this way.

Two tools do it together:

  • PyO3 is a Rust crate. Its attributes, like #[pyfunction] and #[pyclass], turn Rust functions and structs into Python functions and classes, and convert values in both directions.
  • maturin is a build tool. It runs Cargo and packs the result into a normal wheel that pip and uv can install.

Lesson 18 showed the smallest possible module. This lesson goes further. In every example, the left side is a pure-Python module and the right side is the Rust version of it. The calls at the bottom of the Python file work with either one, and the checks run them both ways to make sure they print the same thing.

The Rust Playground can't build Python modules, so the examples here have no Playground link. The exercises at the end are plain Rust.

The project: two config files

A PyO3 project is a Rust crate and a Python package at the same time, so it has both config files side by side:

stats/
├── Cargo.toml        the Rust side
├── pyproject.toml    the Python side
└── src/lib.rs        the code

The commands you need:

  • maturin develop builds the module and installs it into the active virtual environment. Rust is compiled, so run it again after every change to the Rust code, then restart Python.
  • maturin build --release builds wheels to publish, in target/wheels/.
  • pip install . and uv build work too, because pyproject.toml names maturin as the build backend.
pyproject.toml
[build-system]
requires = ["maturin>=1.9,<2"]
build-backend = "maturin"

[project]
name = "stats"
version = "0.1.0"
requires-python = ">=3.9"
Cargo.toml
[package]
name = "stats"
version = "0.1.0"
edition = "2024"

[lib]
crate-type = ["cdylib"]

[dependencies]
pyo3 = "0.30"

Rust functions Python can call

Mark a function with #[pyfunction] and register it in the #[pymodule], and Python can call it. PyO3 converts the arguments and the return value for you, using the types in the signature:

  • int ↔ i64, u8, usize… and float ↔ f64
  • str ↔ &str or String
  • list ↔ Vec<T> and dict ↔ HashMap<K, V>
  • None or a value ↔ Option<T>

Each value is converted when it crosses over: a Python list becomes a new Rust Vec, item by item.

Python · stats.py
from collections import Counter


def mean(values: list[float]) -> float:
    return sum(values) / len(values)


def word_counts(text: str) -> dict[str, int]:
    return dict(Counter(text.split()))


def greet(name: str | None = None) -> str:
    return f"Hello, {name if name is not None else 'stranger'}!"


# Use it: the same calls work with the Rust module
print(mean([1.0, 2.0, 6.0]))  # 3.0
print(sorted(word_counts("to be or not to be").items()))  # [('be', 2), ('not', 1), ('or', 1), ('to', 2)]
print(greet())  # Hello, stranger!
print(greet("Ada"))  # Hello, Ada!
Rust · src/lib.rs
use pyo3::prelude::*;
use std::collections::HashMap;

#[pyfunction]
fn mean(values: Vec<f64>) -> f64 {
    values.iter().sum::<f64>() / values.len() as f64
}

#[pyfunction]
fn word_counts(text: &str) -> HashMap<String, usize> {
    let mut counts = HashMap::new();
    for word in text.split_whitespace() {
        *counts.entry(word.to_string()).or_insert(0) += 1;
    }
    counts
}

#[pyfunction]
#[pyo3(signature = (name=None))]
fn greet(name: Option<&str>) -> String {
    format!("Hello, {}!", name.unwrap_or("stranger"))
}

#[pymodule]
fn stats(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(mean, m)?)?;
    m.add_function(wrap_pyfunction!(word_counts, m)?)?;
    m.add_function(wrap_pyfunction!(greet, m)?)?;
    Ok(())
}

Errors become exceptions

A #[pyfunction] that can fail returns a PyResult<T>, a Result whose error is a Python exception. Ok(value) returns normally. Err(PyValueError::new_err("...")) raises ValueError in Python, so the caller's try / except catches it as usual. Every built-in exception has a type like this: PyTypeError, PyKeyError and so on.

Python · ages.py
def parse_age(text: str) -> int:
    try:
        age = int(text)
    except ValueError:
        raise ValueError(f"not a number: {text}")
    if age > 150:
        raise ValueError(f"out of range: {age}")
    return age


# Use it: the same calls work with the Rust module
def show(text: str) -> str:
    try:
        return str(parse_age(text))
    except ValueError as e:
        return f"ValueError: {e}"


print(show("42"))  # 42
print(show("abc"))  # ValueError: not a number: abc
print(show("200"))  # ValueError: out of range: 200
Rust · src/lib.rs
use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;

#[pyfunction]
fn parse_age(text: &str) -> PyResult<u8> {
    let age: u8 = text
        .parse()
        .map_err(|_| PyValueError::new_err(format!("not a number: {text}")))?;
    if age > 150 {
        return Err(PyValueError::new_err(format!("out of range: {age}")));
    }
    Ok(age)
}

#[pymodule]
fn ages(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(parse_age, m)?)?;
    Ok(())
}

A Rust struct as a Python class

#[pyclass] turns a struct into a Python class, and #[pymethods] turns its impl block into methods. #[new] marks the constructor, which plays the part of __init__. Fields are hidden from Python unless you expose them; #[pyo3(get)] exposes one read-only, like a @property without a setter. Dunder methods such as __repr__ are written under their Python names.

Python · counters.py
class Counter:
    def __init__(self, step: int = 1):
        self._count = 0
        self._step = step

    @property
    def count(self) -> int:
        return self._count

    def bump(self) -> int:
        self._count += self._step
        return self._count

    def __repr__(self) -> str:
        return f"Counter(count={self._count}, step={self._step})"


# Use it: the same calls work with the Rust module
c = Counter(step=2)
c.bump()
print(c.bump())  # 4
print(c.count)  # 4
print(c)  # Counter(count=4, step=2)
Rust · src/lib.rs
use pyo3::prelude::*;

#[pyclass]
struct Counter {
    #[pyo3(get)]
    count: i64,
    step: i64,
}

#[pymethods]
impl Counter {
    #[new]
    #[pyo3(signature = (step=1))]
    fn new(step: i64) -> Self {
        Counter { count: 0, step }
    }

    fn bump(&mut self) -> i64 {
        self.count += self.step;
        self.count
    }

    fn __repr__(&self) -> String {
        format!("Counter(count={}, step={})", self.count, self.step)
    }
}

#[pymodule]
fn counters(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_class::<Counter>()?;
    Ok(())
}

Letting other threads run: releasing the GIL

Rust code called from Python normally holds the GIL (lesson 22), so other Python threads wait while it runs. A function that does a lot of pure-Rust work can let go of the GIL with py.detach(...). Inside the closure it can't touch Python objects, but other threads can run Python meanwhile, and several threads can run the Rust code truly in parallel. That is how libraries like polars use every core. (In PyO3 versions before 0.26 this was called allow_threads.)

Python · heavy.py
def checksum(n: int) -> int:
    return sum(x * x % 7 for x in range(n))


# Use it: the same calls work with the Rust module
from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor() as pool:
    print(list(pool.map(checksum, [10, 1000, 100000])))  # [19, 2001, 199999]
Rust · src/lib.rs
use pyo3::prelude::*;

#[pyfunction]
fn checksum(py: Python<'_>, n: u64) -> u64 {
    py.detach(|| (0..n).map(|x| x * x % 7).sum())
}

#[pymodule]
fn heavy(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(checksum, m)?)?;
    Ok(())
}

Where the Python analogy breaks

Cheat card

PythonRust
# a pure-Python module stats.pysrc/lib.rs with #[pymodule] fn stats(...)
def mean(values: list[float]) -> float:#[pyfunction] fn mean(values: Vec<f64>) -> f64 {
def greet(name=None):#[pyfunction] #[pyo3(signature = (name=None))] fn greet(name: Option<&str>) -> String {
raise ValueError("no values")return Err(PyValueError::new_err("no values"));
def f() -> int: # may raisefn f() -> PyResult<i64> {
class Counter:#[pyclass] struct Counter { ... }
def __init__(self, step=1):#[new] #[pyo3(signature = (step=1))] fn new(step: i64) -> Self {
@property def count(self): ...#[pyo3(get)] count: i64,
def __repr__(self): ...fn __repr__(&self) -> String {
list / dict / NoneVec<T> / HashMap<K, V> / Option<T>
pip install -e .maturin develop // again after every Rust change
python -m buildmaturin build --release

← async/await vs asyncio Reading real projects: uv, ruff and pydantic-core →