PyO3 Asyncio Rust bindings for Python's Asyncio Library. This crate facilitates interactions between Rust Futures and Python Coroutines and…
PyO3 Asyncio Rust bindings for Python's Asyncio Library. This crate facilitates interactions between Rust Futures and Python Coroutines and…
Rust bindings for Python's Asyncio Library. This crate facilitates interactions between Rust Futures and Python Coroutines and manages the lifecycle of their corresponding event loops.
Contributing Notes: github
PyO3 Asyncio is a brand new part of the broader PyO3 ecosystem. Feel free to open any issues for feature requests or bugfixes for this crate.
If you're a new-comer, the best way to get started is to read through the primer below! For v0.13 and v0.14 users I highly recommend reading through the migration section to get a general idea of what's changed in v0.14 and v0.15.
Like PyO3, PyO3 Asyncio supports the following software versions:
If you are working with a Python library that makes use of async functions or wish to provide
Python bindings for an async Rust library, pyo3-asyncio
likely has the tools you need. It provides conversions between async functions in both Python and
Rust and was designed with first-class support for popular Rust runtimes such as
tokio and async-std. In addition, all async Python
code runs on the default asyncio event loop, so pyo3-asyncio should work just fine with existing
Python libraries.
In the following sections, we'll give a general overview of pyo3-asyncio explaining how to call
async Python functions with PyO3, how to call async Rust functions from Python, and how to configure
your codebase to manage the runtimes of both.
Here are some examples to get you started right away! A more detailed breakdown of the concepts in these examples can be found in the following sections.
Here we initialize the runtime, import Python's asyncio library and run the given future to completion using Python's default EventLoop and async-std. Inside the future, we convert asyncio sleep into a Rust future and await it.
# Cargo.toml dependencies
[dependencies]
pyo3 = { version = "0.20" }
pyo3-asyncio = { version = "0.20", features = ["attributes", "async-std-runtime"] }
async-std = "1.9"//! main.rs
use pyo3::prelude::*;
#[pyo3_asyncio::async_std::main]
async fn main() -> PyResult {
let fut = Python::with_gil(|py| {
let asyncio = py.import("asyncio")?;
// convert asyncio.sleep into a Rust Future
pyo3_asyncio::async_std::into_future(asyncio.call_method1("sleep", (1.into_py(py),))?)
})?;
fut.await?;
Ok(())
}The same application can be written to use tokio instead using the #[pyo3_asyncio::tokio::main]
attribute.
# Cargo.toml dependencies
[dependencies]
pyo3 = { version = "0.20" }
pyo3-asyncio = { version = "0.20", features = ["attributes", "tokio-runtime"] }
tokio = "1.9"//! main.rs
use pyo3::prelude::*;
#[pyo3_asyncio::tokio::main]
async fn main() -> PyResult {
let fut = Python::with_gil(|py| {
let asyncio = py.import("asyncio")?;
// convert asyncio.sleep into a Rust Future
pyo3_asyncio::tokio::into_future(asyncio.call_method1("sleep", (1.into_py(py),))?)
})?;
fut.await?;
Ok(())
}More details on the usage of this library can be found in the API docs and the primer below.
PyO3 Asyncio can also be used to write native modules with async functions.
Add the [lib] section to Cargo.toml to make your library a cdylib that Python can import.
[lib]
name = "my_async_module"
crate-type = ["cdylib"]Make your project depend on pyo3 with the extension-module feature enabled and select your
pyo3-asyncio runtime:
For async-std:
[dependencies]
pyo3 = { version = "0.20", features = ["extension-module"] }
pyo3-asyncio = { version = "0.20", features = ["async-std-runtime"] }
async-std = "1.9"For tokio:
[dependencies]
pyo3 = { version = "0.20", features = ["extension-module"] }
pyo3-asyncio = { version = "0.20", features = ["tokio-runtime"] }
tokio = "1.9"Export an async function that makes use of async-std:
//! lib.rs
use pyo3::{prelude::*, wrap_pyfunction};
#[pyfunction]
fn rust_sleep(py: Python) -> PyResult {
pyo3_asyncio::async_std::future_into_py(py, async {
async_std::task::sleep(std::time::Duration::from_secs(1)).await;
Ok(())
})
}
#[pymodule]
fn my_async_module(py: Python, m: &PyModule) -> PyResult {
m.add_function(wrap_pyfunction!(rust_sleep, m)?)?;
Ok(())
}
If you want to use tokio instead, here's what your module should look like:
//! lib.rs
use pyo3::{prelude::*, wrap_pyfunction};
#[pyfunction]
fn rust_sleep(py: Python) -> PyResult {
pyo3_asyncio::tokio::future_into_py(py, async {
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
Ok(())
})
}
#[pymodule]
fn my_async_module(py: Python, m: &PyModule) -> PyResult {
m.add_function(wrap_pyfunction!(rust_sleep, m)?)?;
Ok(())
}You can build your module with maturin (see the Using Rust in Python section in the PyO3 guide for setup instructions). After that you should be able to run the Python REPL to try it out.
maturin develop && python3
Found pyo3 bindings
Found CPython 3.8 at python3
Finished dev [unoptimized + debuginfo] target(s) in 0.04s
Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>>
>>> from my_async_module import rust_sleep
>>>
>>> async def main():
>>> await rust_sleep()
>>>
>>> # should sleep for 1s
>>> asyncio.run(main())
>>>Let's take a look at a dead simple async Python function:
# Sleep for 1 second
async def py_sleep():
await asyncio.sleep(1)Async functions in Python are simply functions that return a coroutine object. For our purposes,
we really don't need to know much about these coroutine objects. The key factor here is that calling
an async function is just like calling a regular function, the only difference is that we have
to do something special with the object that it returns.
Normally in Python, that something special is the await keyword, but in order to await this
coroutine in Rust, we first need to convert it into Rust's version of a coroutine: a Future.
That's where pyo3-asyncio comes in.
pyo3_asyncio::into_future
performs this conversion for us:
…If you're interested in learning more about
coroutinesandawaitablesin general, check out the Python 3asynciodocs for more information.
Here we have the same async function as before written in Rust using the
async-std runtime:
/// Sleep for 1 second
async fn rust_sleep() {
async_std::task::sleep(std::time::Duration::from_secs(1)).await;
}Similar to Python, Rust's async functions also return a special object called a
Future:
let future = rust_sleep();We can convert this Future object into Python to make it awaitable. This tells Python that you
can use the await keyword with it. In order to do this, we'll call
pyo3_asyncio::async_std::future_into_py:
use pyo3::prelude::*;
async fn rust_sleep() {
async_std::task::sleep(std::time::Duration::from_secs(1)).await;
}
#[pyfunction]
fn call_rust_sleep(py: Python) -> PyResult {
pyo3_asyncio::async_std::future_into_py(py, async move {
rust_sleep().await;
Ok(())
})
}In Python, we can call this pyo3 function just like any other async function:
from example import call_rust_sleep
async def rust_sleep():
await call_rust_sleep()Python's event loop requires some special treatment, especially regarding the main thread. Some of
Python's asyncio features, like proper signal handling, require control over the main thread, which
doesn't always play well with Rust.
Luckily, Rust's event loops are pretty flexible and don't need control over the main thread, so in
pyo3-asyncio, we decided the best way to handle Rust/Python interop was to just surrender the main
thread to Python and run Rust's event loops in the background. Unfortunately, since most event loop
implementations prefer control over the main thread, this can still make some things awkward.
Because Python needs to control the main thread, we can't use the convenient proc macros from Rust
runtimes to handle the main function or #[test] functions. Instead, the initialization for PyO3 has to be done from the main function and the main
thread must block on pyo3_asyncio::run_forever or pyo3_asyncio::async_std::run_until_complete.
Because we have to block on one of those functions, we can't use #[async_std::main] or #[tokio::main]
since it's not a good idea to make long blocking calls during an async function.
Internally, these
#[main]proc macros are expanded to something like this:fn main() { // your async main fn async fn _main_impl() { /* ... */ } Runtime::new().block_on(_main_impl()); }Making a long blocking call inside the
Futurethat's being driven byblock_onprevents that thread from doing anything else and can spell trouble for some runtimes (also this will actually deadlock a single-threaded runtime!). Many runtimes have some sort ofspawn_blockingmechanism that can avoid this problem, but again that's not something we can use here since we need it to block on the main thread.
For this reason, pyo3-asyncio provides its own set of proc macros to provide you with this
initialization. These macros are intended to mirror the initialization of async-std and tokio
while also satisfying the Python runtime's needs.
Here's a full example of PyO3 initialization with the async-std runtime:
use pyo3::prelude::*;
#[pyo3_asyncio::async_std::main]
async fn main() -> PyResult {
// PyO3 is initialized - Ready to go
let fut = Python::with_gil(|py| -> PyResult {
let asyncio = py.import("asyncio")?;
// convert asyncio.sleep into a Rust Future
pyo3_asyncio::async_std::into_future(
asyncio.call_method1("sleep", (1.into_py(py),))?
)
})?;
fut.await?;
Ok(())
}asyncio.runIn Python 3.7+, the recommended way to run a top-level coroutine with asyncio
is with asyncio.run. In v0.13 we recommended against using this function due to initialization issues, but in v0.14 it's perfectly valid
No open issues yet, or sync has not completed.