Web3.py Patterns: Off-chain Lookups

EIP-3668 introduced a standard for secure off-chain data lookups in Ethereum. This creates a broadly accepted pattern for a smart contract to return a location where off-chain data can be referenced. This post will give a quick tour of how that works, what a use case looks like, and how to utilize the feature in Web3.py (as of v6.0.0-beta.4).

Use case

In order to extend its functionality into additional scaling layers and blockchains, ENS enables its resolvers to return addresses stored somewhere other than Mainnet Ethereum via ENSIP-10 (Wildcard Resolution) and EIP-3668 (CCIP Reads). With these two protocol upgrades, foo.whatever.eth can resolve an Ethereum (or any other) address, without the foo subdomain having been deployed to Ethereum Mainnet. An example implementation of this functionality can be found here.

How it works

EIP-3668 defines a custom exception, OffchainLookup, whose payload is used to communicate where off-chain data can be retrieved. The Solidity syntax for the custom error is as follows:

error OffchainLookup(address sender, string[] urls, bytes callData, bytes4 callbackFunction, bytes extraData)

When a user initiates an off-chain lookup by calling a function on a contract, the contract is expected to revert using that OffchainLookup exception. Within the exception payload is the URL(s) where the off-chain data can be found, data to include in the request, and the callback function to execute after completing the off-chain lookup.

Note that this feature is intentionally flexible. The security model for verifying the legitimacy of the off-chain data is up to the client and gateway to agree on and will vary based on use case. Nick Johnson, author of EIP-3668, describes the usage pattern of an ENS resolver verifying an address via an L2 Optimism gateway in this PEEP an EIP episode.

Web3.py example

A library like Web3.py can enable off-chain lookups without a user needing to understand how the feature works or that it even occurred. Let's run through a quick example.

The ENS team deployed their off-chain lookup proof of concept utilizing the domain offchainexample.eth. Using Web3.py's ens module, let's try to fetch the address of test.offchainexample.eth:

>>> from web3 import Web3, HTTPProvider

# connect to mainnet
>>> w3 = Web3(HTTPProvider('https://provider-link-here...'))

>>> w3.ens.address('test.offchainexample.eth')
'0x41563129cDbbD0c5D3e1c86cf9563926b243834d'

We can see that the subdomain test.offchainexample.eth resolved to the address 0x41563129cDbbD0c5D3e1c86cf9563926b243834d. What happens when we try to resolve the address for another random subdomain, say web3py.offchainexample.eth?

>>> w3.ens.address('web3py.offchainexample.eth')
'0x41563129cDbbD0c5D3e1c86cf9563926b243834d'

The web3py subdomain isn't registered, but the same address is returned. And just like that, you've used Web3.py to resolve an off-chain address.

Let's peek under the hood and verify that an off-chain lookup was actually performed. We'll disable CCIP Read functionality, rerun the query, and examine the resolver contract response. CCIP Read may be turned off globally via a flag on the provider: global_ccip_read_enabled. We'll then capture the OffchainLookup error and review the payload.

from web3.exceptions import OffchainLookup

# turn off global CCIP Read support on the provider
w3.provider.global_ccip_read_enabled = False

# capture the OffchainLookup revert
>>> try:
...     w3.ens.address('test.offchainexample.eth')
... except OffchainLookup as e:
...     offchain_lookup = e

# examine the payload
>>> offchain_lookup.payload
{
    'sender': '0xc1735677a60884abbcf72295e88d47764beda282',
    'urls': ('https://offchain-resolver-example.uc.r.appspot.com/{sender}/{data}.json',),
    'callData': b'\x90a\xb9#\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x1a\x04test\x0foffchainexample\x03eth\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00$;;W\xde\xc0-\xba\rS\x12\xaa&\xcaX\xb1\xed\xdf\xa1*\x16\xbc\x1c\t\xdb\xe7\xb3\x0f\xa2\xb19\x83b\x15\xc0\xe0\xc0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
    'callbackFunction': b'\xf4\xd4\xd2\xf8',
    'extraData': b'\x90a\xb9#\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x1a\x04test\x0foffchainexample\x03eth\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00$;;W\xde\xc0-\xba\rS\x12\xaa&\xcaX\xb1\xed\xdf\xa1*\x16\xbc\x1c\t\xdb\xe7\xb3\x0f\xa2\xb19\x83b\x15\xc0\xe0\xc0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
 }

The OffchainLookup payload contains the information for making off-chain requests and subsequently assembling a new data payload to be made to the callbackFunction within the same contract.

The OffchainLookup payload contains:

• urls: A list of URLs where the off-chain data can be found. Multiple URLs allow for better reliability as fallback requests for the same information. This list should be sorted by the contract in order of URL importance.
• sender: The contract address. This value is used to replace any string matching {sender} in the urls field provided by the revert.
• callData: The data to be used in the off-chain requests. If the URL contains a string matching {data}, this string is replaced with the callData value and a GET request is made to the URL. If no string matching {data} is present in the URL, the callData value is instead used as the data payload in a POST request to the URL.
• callbackFunction: The 4-byte function selector for the function in the contract to be called with the returned off-chain data plus any extraData - in this ENS example, it is the first 4 bytes of the keccak hash of "resolveWithProof(bytes,bytes)".
• extraData: Data to be used in the call to the callback function in order to retain a link to the original call that triggered the revert.

For Web3.py's part, when CCIP Read functionality is enabled, the library will catch any OffchainLookup exceptions and perform a fetch using the URL(s) provided, populating the {data} and {sender} fields as appropriate, then initiate the follow-up callbackFunction in the same contract.

Again, the callback function in the contract is responsible for verifying that the off-chain data is valid before it returns the resolved address. In the ENS example, the resolveWithProof function verifies a valid signed message before returning an address.

Wrapping  Up

Through cheeky use of a custom error, EIP-3668 introduces a new primitive that makes cross-chain interaction measurably easier. Building something with off-chain data lookups? Tell us about your use case in the Ethereum Python Discord!