Logo notonlyowner

How to test a smart contract function a million times

Weeks ago I spent a few days reviewing the first release candidate of v4.8.0 of the OpenZeppelin Contracts library. That meant attempting to uncover bugs in the full set of changes.

In most PRs the review was 100% manual. Which led me to spot a thing or two.

In other parts, I relied on the OP fuzzing skills of Foundry to automate testing. Let me tell you how.

Choosing the target

I noticed the Clones library had been changed. Apparently to make it more efficient, by tweaking its assembly code. If you are not familiar with clones (a.k.a. minimal proxies), read this article.

I had at least 3 options to review this new code:

  1. The auditor-like white-box approach. In practice, a thorough but slow line-by-line review. Also error-prone, because of the low-level assembly and raw EVM bytecode.
  2. Black-box testing it with Hardhat. Feasible, though the library is already quite tested. I didn't want to do more of the same.
  3. Swallow the Foundry pill, load the fuzzing machine guns, and hope for the best.

The pill tasted ok.

As soon as it hit: foundryup, forge init, forge install @openzeppelin/openzeppelin-contracts@v4.8.0-rc.0. Then created the usual Foundry testing file under the test folder.

My goal was to check whether the function Clones::predictDeterministicAddress worked correctly. Given a contract address, a salt and a deployer address, it should return the same address as the actual CREATE2 deployment of a clone of the contract address by the same deployer address and salt.

This shouldn't only be true for 1 deployer, 1 salt and 1 contract address. Nor 2. Nor 10. It must hold for whatever valid combination of those parameters. Good luck manually writing a test for each of them.

Thus fuzzing seemed the right tool for the job.

Preparing the ground

I like starting this kind of tests by preparing the ground.

That means first coding the simplest case I can come up with. Just to be sure all pieces are wired correctly before getting any smarter.

Only after running a few simple tests do I move to more complex setups.

So, the skeleton looked liked this:

pragma solidity ^0.8.0;

import "forge-std/Test.sol";
import "forge-std/console.sol";

import "@openzeppelin/contracts/proxy/Clones.sol";

// The contract I'll deploy clones for with CREATE2
contract SimpleContract { }

contract ClonesTest is Test {
    function test_predictDeterministicAddress() external {

    }
}

I first added code for the implementation contract to be cloned. I wanted its code to be somewhat "dynamic". In other words, that it would somehow change in each test. That way I could have stronger guarantess that the library was able to handle contracts regardless of their specific code.

However, I soon realized I didn't have an easy way to produce pseudo-random, valid and realistic EVM bytecode. And I didn't want to get the setup too complicated. Remember, I still had lots of other PRs to review in my backlog.

So I simply put an immutable state variable for a uint256 that gets set in the constructor. I know, I know. It's not nearly enough to what I originally wanted, but it felt better than nothing.

For the sake of testing, I also included a simple function to retrieve the number.

contract SimpleContract {
    uint256 private immutable _number;

    constructor(uint256 number) {
        _number = number;
    }

    function getNumber() external view returns (uint256) {
        return _number;
    }
}

Coding the initial test case

Now, my first test case. You can see below how I deploy an instance of a SimpleContract, and create a clone. Then I check the number is set correctly. Finally I test whether the actual deployment address of the clone (named cloneAddress) matches the one predicted by predictDeterministicAddress. The resulting code:

contract ClonesTest is Test {
    function test_predictDeterministicAddress() external {
        address deployer = address(0xddd);
        bytes32 salt = keccak256(abi.encodePacked("the salt"));
        uint256 number = 77;

        vm.startPrank(deployer);

        SimpleContract implementation = new SimpleContract(number);

        address cloneAddress = Clones.cloneDeterministic(address(implementation), salt);
        assertEq(implementation.getNumber(), number);
        assertEq(SimpleContract(cloneAddress).getNumber(), number);

        address predictedAddress = Clones.predictDeterministicAddress(address(implementation), salt, deployer);

        assertEq(predictedAddress, cloneAddress);
    }
}

To wrap it up, a quick forge test to double check I was on the right track:

forge test --mp test/Clones.t.sol
[⠔] Compiling...
[⠒] Compiling 1 files with 0.8.16
[⠑] Solc 0.8.16 finished in 1.01s
Compiler run successful

Running 1 test for test/Clones.t.sol:ClonesTest
[PASS] test_predictDeterministicAddress() (gas: 108350)
Test result: ok. 1 passed; 0 failed; finished in 12.65ms

All was set. I was ready to bring in the badass fuzzer.

Preparing the fuzzer

Transforming a simple unit test into a fuzzing test is absurdly simple. I did it by refactoring the test above in three steps.

First, removing the hardcoded deployer address. And instead adding the address deployer parameter to the test function.

contract ClonesTest is Test {
    function test_fuzz_predictDeterministicAddress(address deployer) external {
        bytes32 salt = keccak256(abi.encodePacked("the salt"));
        uint256 number = 77;

        vm.startPrank(deployer);

        SimpleContract implementation = new SimpleContract(number);

        address cloneAddress = Clones.cloneDeterministic(address(implementation), salt);
        assertEq(implementation.getNumber(), number);
        assertEq(SimpleContract(cloneAddress).getNumber(), number);

        address predictedAddress = Clones.predictDeterministicAddress(address(implementation), salt, deployer);

        assertEq(predictedAddress, cloneAddress);
    }
}

Then the salt, adding the corresponding bytes32 parameter:

contract ClonesTest is Test {
    function test_fuzz_predictDeterministicAddress(address deployer, bytes32 salt) external {
        uint256 number = 77;

        vm.startPrank(deployer);

        SimpleContract implementation = new SimpleContract(number);

        address cloneAddress = Clones.cloneDeterministic(address(implementation), salt);
        assertEq(implementation.getNumber(), number);
        assertEq(SimpleContract(cloneAddress).getNumber(), number);

        address predictedAddress = Clones.predictDeterministicAddress(address(implementation), salt, deployer);

        assertEq(predictedAddress, cloneAddress);
    }
}

Third, the number. Adding the corresponding uint256 parameter:

contract ClonesTest is Test {
    function test_fuzz_predictDeterministicAddress(address deployer, bytes32 salt, uint256 number) external {
        vm.startPrank(deployer);

        SimpleContract implementation = new SimpleContract(number);

        address cloneAddress = Clones.cloneDeterministic(address(implementation), salt);
        assertEq(implementation.getNumber(), number);
        assertEq(SimpleContract(cloneAddress).getNumber(), number);

        address predictedAddress = Clones.predictDeterministicAddress(address(implementation), salt, deployer);

        assertEq(predictedAddress, cloneAddress);
    }
}

I could also "guide" the fuzzer, in this case saying which values I wanted it to discard. Just to test what Foundry is capable of. For instance, stating that the zero address cannot be a deployer. I included this restriction with the cheatcode vm.assume.

contract ClonesTest is Test {
    function test_fuzz_predictDeterministicAddress(address deployer, bytes32 salt, uint256 number) external {
        vm.assume(deployer != address(0));
        vm.startPrank(deployer);

        SimpleContract implementation = new SimpleContract(number);

        address cloneAddress = Clones.cloneDeterministic(address(implementation), salt);
        assertEq(implementation.getNumber(), number);
        assertEq(SimpleContract(cloneAddress).getNumber(), number);

        address predictedAddress = Clones.predictDeterministicAddress(address(implementation), salt, deployer);

        assertEq(predictedAddress, cloneAddress);
    }
}

Needless to say, there was room to get smarter in how I guided the fuzzer to create more realistic scenarios.

Running the fuzzer

I could run this test a hundred times, or a hundred thousand times, or a million times. The number can be set in the foundry.toml file as follows:

[fuzz]
runs = 1000000

In seconds Foundy generates lots of different inputs for the three parameters, and ensure the assertions are always correct. If they are not, it will show the exact input that makes the assertion fail.

Here's my run for a million times:

forge test --mp test/Clones.t.sol

Running 1 test for test/Clones.t.sol:ClonesTest
[PASS] test_fuzz_predictDeterministicAddress(address,bytes32,uint256) (runs: 1000000, μ: 108725, ~: 108725)
Test result: ok. 1 passed; 0 failed; finished in 95.70s

That's how I was able to run a million tests to find a bug. And completely fail.

But at least it was fast!

Some may argue the test I coded is nearly not enough to ensure the tested functions always always always work as intended. Agreed. But you cannot deny that with fuzzing, I can get stronger guarantees. The last test is much stronger than the simple unit test I started with.

Might be the pill talking, but I'm starting to enjoy fuzzing :)