web3.py Patterns: Intro to Async
Asynchronous support has long been one of the most requested features in web3.py. The
AsyncHTTPProvider was first introduced in v5.20.0, but has received polish in most releases since. There are a couple edge cases to go for full support, but the freshly released v6 is as good a time as any to give the async provider a try!
If you're satisfied with the performance of your web3.py dapp or script today, you're under no obligation to update it. However, dapps very commonly spend much of their runtime waiting on requests to make the round trip to and from a remote node to query block data. Asynchronous programming patterns can make much more efficient use of your computing resources, dispatching additional requests while still waiting for earlier ones to return, for example. If your application performs a lot of data queries, you may have a good candidate for a provider refactor.
A code sample
Let's skip straight to the good stuff. Here's a simple, contrived example that makes 50 requests – one for each of the first 50 blocks of the Ethereum blockchain – and logs out the block number as each request returns.
import asyncio from web3 import AsyncWeb3, AsyncHTTPProvider w3 = AsyncWeb3(AsyncHTTPProvider('https://...')) async def fetch_blocks(n): for result in asyncio.as_completed( [w3.eth.get_block(num) for num in range(n)] ): b = await result print(b.number) asyncio.run(fetch_blocks(50)) # 30 # 8 # 24 # 13 # ...
Here we define an asynchronous function and make use of
asyncio to execute it via the
asyncio.run method. Note that web3.py utilizes
aiohttp under the hood;
asyncio is the only supported async library for the moment.
asyncio has a several usage patterns, but we leverage the
as_completed method to handle each request as they return. For each coroutine (request for a block), we await the result, then print out that block number. As a result, and subject to whatever conditions exist in the network, block numbers are logged out in a fairly random order.
If you need a random number generator, there are certainly more efficient ways to go about that. Hopefully though, this is a helpful illustration of asynchronous behavior. Depending on your use case, you may need to reorder the data before you make use of it. The
wait, or callback methods may help with that; a deep rabbit hole of learning awaits those who need more sophisticated patterns.
For the uninitiated, an ENS module exists within web3.py. It too has been given asynchronous superpowers via the
from web3 import AsyncWeb3, AsyncHTTPProvider from ens import AsyncENS w3 = AsyncWeb3(AsyncHTTPProvider('https://...')) ns = AsyncENS.from_web3(w3) names = ['shaq.eth', 'vitalik.eth', 'parishilton.eth'] * 10 async def fetch_addresses(): for result in asyncio.as_completed( [ns.address(name) for name in names] ): print(await result) asyncio.run(fetch_addresses())
asyncio.as_completed pattern should look familiar; it's utilized here to print out the resolved address as soon as the response is received from the remote node.
Your optimization gains will vary greatly depending on your use case. Again, you will benefit most in cases where you spend much of your runtime waiting on responses from remote nodes. To give you a sense of the scale, here are some benchmarks while using a free remote node service.
Local benchmarks for
AsyncENS produced even greater gains:
Can't use the
If your use case can benefit from a little concurrency, but for whatever reason
asyncio is not an option for you, you may need to reach for threads. An introduction to multithreaded usage of web3.py can be found here.
If your preference or requirement is to use the Websockets or IPC providers, async support for those providers is on the roadmap, but without a firm timeline.
While continuing to receive polish, the
AsyncHTTPProvider is ready for use today. You can expect your query-intensive applications to gain some serious performance improvements by adopting these asynchronous programming patterns. Looking for support or to share what you're building? Join the Ethereum Python Community Discord.