Web3.py Patterns: Customization

[Last update: 6/25/24]

If you're looking to make Web3.py do something outside of its native functionality, you've got at least a few options: middleware, custom methods, external modules, and custom providers. This post will walk through what each of those are, when you might reach for them, and how to get started.

1. Middleware

What

Middleware allows you to add some behavior to existing methods prior to making a request or after a result is received.

When

Reach for middleware when you want something to happen every time a certain RPC call or set of calls are performed, e.g., logging, data visualization, data conversion, etc.

How

💡

Note: web3.py's middleware architecture got a rewrite in v7, rendering this section obsolete. See this post for the context and latest patterns.

A set of default middleware comes standard in Web3.py, along with more optional middleware to pull in. However, If you need to write some custom middleware, you have a couple of syntax options: functions or classes. The function syntax is more typical for straightforward use cases.

def example_middleware(make_request, w3):
    # do one-time setup operations here

    def middleware(method, params):
        # do pre-processing here

        # perform the RPC request, getting the response
        response = make_request(method, params)

        # do post-processing here

        # finally return the response
        return response
    return middleware

A middleware template utilizing the function syntax

Middleware is executed in a particular order, so the API allows you to add your new middleware to the end of the list, inject, replace or remove a layer, or clear the whole middleware stack.

2. Custom Methods

What

Arbitrary RPC methods may be added to existing modules.

When

Registering custom methods can be handy if you're working with a client with nonstandard RPC commands or are testing some custom functionality within a forked client.

Custom methods can also be used to overwrite existing methods, if you want to apply your own request or result formatter.

How

The attach_methods function is available on every module and accepts a dictionary with method names and corresponding Method:

from web3.method import Method

w3.eth.attach_methods({"create_access_list": Method("eth_createAccessList")})

w3.eth.create_access_list(...)

You may optionally include custom handlers for input munging, request, and result formatters.

from web3.method import Method

w3.eth.attach_methods({
	"example": Method(
		"eth_example",
		mungers=[...],
		request_formatters=[...],
		result_formatters=[...],
		is_property=False,
	),
})

w3.eth.example()

Adding a custom eth_example method to the Eth module

If you prefer, you can add a property instead of a method by setting is_property to True.

3. External Modules

What

External modules offer still more flexibility by allowing you to import groups of APIs under one banner. Think: plugins.

When

External modules may be useful for introducing an entire L2 API or several nonstandard RPC methods supported by one client, e.g., Erigon-specific methods like erigon_getHeaderByHash, erigon_getHeaderByNumber, and so on.

How

Modules need only be classes and can reference the parent Web3 instance. Configure your external modules at the time of Web3 instantiation using the external_modules keyword argument, or at any point via the attach_modules method:

# add modules at instantiation:
w3 = Web3(
   HTTPProvider(...),
   external_modules={"example": ExampleModule}
)

# add modules after instantiation:
w3.attach_modules({"example": ExampleModule})

# invoking external modules:
w3.example.example_method()

More context, including a nested module example, available here.

4. Custom Providers

What

At its core, a provider defines how requests are performed.

When

Building a custom provider is only relevant for the rare occasions that you're plugging into a custom testing framework, or something in that vein. If you're just looking to connect to another EVM blockchain, sidechain, or rollup, typically you can just configure one of the existing options: HTTPProvider, IPCProvider, or WebsocketProvider.

How

Providers only require two methods, make_request and isConnected, and a definition of middlewares.

from web3.providers.base import BaseProvider

class CustomProvider(BaseProvider):
    middlewares = ()

    def make_request(self, method, params):
        return {"result": {"welp": "lol"}}

    def is_connected(self):
        print(True)

w3 = Web3(CustomProvider())

w3.eth.get_block("latest")
# AttributeDict({"welp": "lol"})

Wrapping up

The options above are ordered roughly from least to most flexibility they offer. In practice, I expect middleware and external modules to get the most mileage, particularly once trusted external modules become commonplace.

Deliberately not included in this post is monkey patching. If you've gone down that path... is everything okay? Seriously though, open an issue if you have another vector you'd like to customize that requires monkey patching today.