Effective Smart Contract Testing: Property-Based Testing
Brownie and Hypothesis: spend less time, write better tests
Behold, a simple Brownie test case for an ERC20 smart contract:
def test_transfer_adjusts_balance(token, accounts):
from_balance = token.balanceOf(accounts[0])
to_balance = token.balanceOf(accounts[1])
token.transfer(accounts[1], 1000, {'from': accounts[0]}) assert token.balanceOf(accounts[0]) == from_balance - 1000
assert token.balanceOf(accounts[1]) == to_balance + 1000
If you’ve ever tested a token contract, you’ve probably written a test similar to this one. It’s a verification that the transfer function is adjusting balances as intended. For many developers, this is enough. Ship it and on to test the next behavior!
On it’s own, however, this test isn’t very strong. Imagine if the contract being tested always modified the balance by exactly 1000, regardless of the amount that was supposed to be transferred. Our initial test would not catch this behavior. For this reason it is a good practice to test each behavior against a range of data, rather than only a single value. This can be accomplished with liberal use of copy and paste, but the smarter approach is to use a technique known as parametrization.
Test Parametrization
Parameterization is an essential tool for any test suite. It allows you to replace a single test value with a range of data, effectively creating multiple tests without having to duplicate code.
In Brownie we achieve this using the @pytest.mark.parametrize decorator, as shown in the example below:
import pytest@pytest.mark.parametrize("amount", [0, 42, 1000, 31337])
def test_transfer_adjusts_balance(token, accounts, amount):
from_balance = token.balanceOf(accounts[0])
to_balance = token.balanceOf(accounts[1])
token.transfer(accounts[1], amount, {'from': accounts[0]}) assert token.balanceOf(accounts[0]) == from_balance - amount
assert token.balanceOf(accounts[1]) == to_balance + amount
This test verifies the same behavior as our first example, but it executes four times with different amounts transferred each time. This is useful for two reasons:
- We have significantly strengthened our testing of this behavior. If there is a flaw in how the balances are adjusted, our test suite is more likely to discover it.
- In the future, if we want to expand or change how this behavior is tested, we only have to modify one test case instead of four.
Pretty good, right? With just one more line of code, we’ve taken our original test and made it so much stronger! Ship it and move on!
…but can we do better?
This is all well and good, but there’s a problem we still aren’t addressing: ultimately, we are only testing our own expectations of how a contract should work. Even with parametrized tests we are producing the data sets that our tests execute against. And where do edge cases happen? Exactly where we aren’t looking!
So how do we get past this paradox? How can we write tests for the things we don’t expect? The answer: property-based testing.
Property-based Testing
Property-based testing is a powerful tool for locating edge cases and discovering faulty assumptions within your code.
The key idea behind property-based testing is that rather than writing a test for a single scenario, you write tests that describe a range of scenarios. Then you let your computer explore the possibilities for you, rather than having to hand-write each case yourself.
This is effective because it automates one of the most time consuming parts of writing tests — coming up with the examples. And it means your contract is more likely to be hit with cases you didn’t consider, which is exactly where the bugs like to hide.
Brownie achieves this by integrating the hypothesis library. Here is our same test, written to take advantage of property-based testing:
from brownie.test import given, strategy@given(amount=strategy('uint256', max_value=1000000))
def test_transfer_adjusts_balance(token, accounts, amount):
from_balance = token.balanceOf(accounts[0])
to_balance = token.balanceOf(accounts[1])
token.transfer(accounts[1], amount, {'from': accounts[0]}) assert token.balanceOf(accounts[0]) == from_balance - amount
assert token.balanceOf(accounts[1]) == to_balance + amount
When this test executes, the logic is run fifty times. During each run, amount is a different value somewhere between 0 and 1000000. The chosen values are stored in a database, and in subsequent test runs different values are be used. If a value is found that causes the test to fail, this value will always be included in subsequent tests.
That last line is important! While you wouldn’t be wrong to describe this process as randomized testing, it is also repeatable randomized testing. While a bug may take some time to be found, it will never simply go away once it has been discovered.
Writing Property-Based Tests
Writing property-based tests is a fairly straightforward process. The magic is coming from two methods:
@givenis a decorator that converts a test function into a randomized test. It specifies which arguments in the function are to be parametrized, and how that process should be handled.strategyis a method for creating test strategies based on smart contract ABI types. A test strategy is a recipe for describing the sort of data you want to generate. It typically defines the data type, lower and upper bounds, and values that should be excluded.
By combining these two methods, we can easily generate property-based tests that will test a behavior of your contract’s specification much more thoroughly than with regular parametrized tests.
Here is another example where we add a second strategy to, to parametrize the receiver of the tokens. This new strategy uses the exclude keyword argument to avoid generating a test where the sender and receiver are the same account.
from brownie import accounts
from brownie.test import given, strategy
@given(
to=strategy('address', exclude=accounts[0]),
amount=strategy('uint256', max_value=1000000),
)
def test_transfer_adjusts_balance(token, to, amount):
from_balance = token.balanceOf(accounts[0])
to_balance = token.balanceOf(to)
token.transfer(to, amount, {'from': accounts[0]}) assert token.balanceOf(accounts[0]) == from_balance - amount
assert token.balanceOf(to) == to_balance + amount
Note that when using exclude, you should also include a separate test for the excluded value(s).
Where to Start
To learn more about property-based testing for smart contracts, check out the Brownie Property-Based Testing documentation. It explores of the above functionality in greater depth, and shows how to integrate Hypothesis into your tests to utilize this powerful testing technique.
If you’re unsure where to begin adding property-based tests within your test suite, here are some ideas:
- Look for simple tests that verify “when X, then Y”, where there is a direct relationship between X and Y. Balances, approval amounts, user permissions — smart contracts are full of situations where this sort of testing can come in handy.
- Look for code duplication in your test suite. Are there cases where you have multiple tests verifying the same behavior? Can you refactor these tests into a single, property-based case?
