web3.py Patterns: Intro to Async

Asynchronous support has long been one of the most requested features in web3.py. The AsyncHTTPProvider was first introduced in v5.20.0, but has received polish in most releases since. There are a couple edge cases to go for full support, but the freshly released v6 is as good a time as any to give the async provider a try!

Why async?

If you're satisfied with the performance of your web3.py dapp or script today, you're under no obligation to update it. However, dapps very commonly spend much of their runtime waiting on requests to make the round trip to and from a remote node to query block data. Asynchronous programming patterns can make much more efficient use of your computing resources, dispatching additional requests while still waiting for earlier ones to return, for example. If your application performs a lot of data queries, you may have a good candidate for a provider refactor.

A code sample

Let's skip straight to the good stuff. Here's a simple, contrived example that makes 50 requests – one for each of the first 50 blocks of the Ethereum blockchain – and logs out the block number as each request returns.

import asyncio
from web3 import AsyncWeb3, AsyncHTTPProvider


w3 = AsyncWeb3(AsyncHTTPProvider('https://...'))

async def fetch_blocks(n):
    for result in asyncio.as_completed(
        [w3.eth.get_block(num) for num in range(n)]
    ):
        b = await result
        print(b.number)

asyncio.run(fetch_blocks(50))
# 30
# 8
# 24
# 13
# ...

Here we define an asynchronous function and make use of asyncio to execute it via the asyncio.run method. Note that web3.py utilizes aiohttp under the hood; asyncio is the only supported async library for the moment.

asyncio has a several usage patterns, but we leverage the as_completed method to handle each request as they return. For each coroutine (request for a block), we await the result, then print out that block number. As a result, and subject to whatever conditions exist in the network, block numbers are logged out in a fairly random order.

If you need a random number generator, there are certainly more efficient ways to go about that. Hopefully though, this is a helpful illustration of asynchronous behavior. Depending on your use case, you may need to reorder the data before you make use of it. The gather, wait, or callback methods may help with that; a deep rabbit hole of learning awaits those who need more sophisticated patterns.

AsyncENS

For the uninitiated, an ENS module exists within web3.py. It too has been given asynchronous superpowers via the AsyncENS class:

from web3 import AsyncWeb3, AsyncHTTPProvider
from ens import AsyncENS


w3 = AsyncWeb3(AsyncHTTPProvider('https://...'))
ns = AsyncENS.from_web3(w3)

names = ['shaq.eth', 'vitalik.eth', 'parishilton.eth'] * 10

async def fetch_addresses():
    for result in asyncio.as_completed(
        [ns.address(name) for name in names]
    ):
        print(await result)

asyncio.run(fetch_addresses())

The asyncio.as_completed pattern should look familiar; it's utilized here to print out the resolved address as soon as the response is received from the remote node.

Performance

Your optimization gains will vary greatly depending on your use case. Again, you will benefit most in cases where you spend much of your runtime waiting on responses from remote nodes. To give you a sense of the scale, here are some benchmarks while using a free remote node service.

 Method (50 calls)  |   HTTPProvider  |  AsyncHTTProvider
----------------------------------------------------------
    eth_gasPrice    |    5.95 secs    |    0.52 secs   
  eth_blockNumber   |    6.93 secs    |    0.44 secs     
    eth_getBlock    |    7.75 secs    |    0.73 secs

Local benchmarks for AsyncENS produced even greater gains:

    Method (30 calls)  |       ENS      |    AsyncENS
-------------------------------------------------------
       ns.address      |    40.13 secs  |    2.78 secs
       ns.name         |    57.13 secs  |    2.96 secs

Can't use the AsyncHTTPProvider?

If your use case can benefit from a little concurrency, but for whatever reason asyncio is not an option for you, you may need to reach for threads. An introduction to multithreaded usage of web3.py can be found here.

If your preference or requirement is to use the Websockets or IPC providers, async support for those providers is on the roadmap, but without a firm timeline.

Wrapping up

While continuing to receive polish, the AsyncHTTPProvider is ready for use today. You can expect your query-intensive applications to gain some serious performance improvements by adopting these asynchronous programming patterns. Looking for support or to share what you're building? Join the Ethereum Python Community Discord.