An In-Depth Guide to Testing Ethereum Smart Contracts

Part Six: Parametrization and Property-Based Testing

This article is part of a series. If you haven’t yet, check out the previous articles:

Part One: Why we Test
Part Two: Core Concepts of Testing
Part Three: Writing Basic Tests
Part Four: Running Your Tests
Part Five: Tools and Techniques for Effective Testing
Part Six: Parametrization and Property-Based Testing
Part Seven: Stateful Testing

Let’s revisit one of our earlier ERC20 test cases:

Here we verify that the transfer function adjusts the sender’s balance as intended. Simple enough, right? Ship it and on to test the next behavior!

On it’s own, however, this test isn’t as strong as it could be. Imagine if the contract being tested always modified the balance by exactly 1⁰¹⁸, regardless of the amount that was supposed to be transferred. Our initial test would not catch this behavior. For this reason it is a good practice to test each behavior against a range of data, rather than only a single value. This can be accomplished with liberal use of copy and paste, but the smarter approach is to use a technique known as parametrization.

Test Parametrization

Parameterization is an essential tool for any test suite. It allows you to replace a single test value with a range of data, effectively creating multiple tests without having to duplicate code.

In Brownie we achieve this using the @pytest.mark.parametrize decorator. Here is the same test as above, rewritten to use parametrization:

This test verifies the same behavior as our first example, but it executes three times with different amounts transferred each time. This is useful for two reasons:

1. We have significantly strengthened our testing of this behavior. If there is a flaw in how the balances are adjusted, our test suite is more likely to discover it.
2. In the future, if we want to expand or change how this behavior is tested, we only have to modify one test case instead of four.

Pretty good, right? With just one more line of code, we’ve taken our original test and made it significantly more effective.

…but can we do better?

There are many testing situations where parametrization the correct tool for the job. But there’s still a problem that is doesn’t adequately address: ultimately, we are only testing our own expectations of how a contract should function. Even with parametrized tests we are producing the data sets that our tests execute against. And where do edge cases happen? Exactly where we aren’t looking!

So how do we get past this paradox? How can we write tests for the things we don’t expect? The answer: property-based testing.

Property-based Testing

Property-based testing is a powerful tool for locating edge cases and discovering faulty assumptions within your code.

The key idea behind property-based testing is that rather than writing a test for a single scenario, you write tests that describe a range of scenarios. Then you let your computer explore the possibilities for you, rather than having to hand-write each case yourself.

This is effective because it automates one of the most time consuming parts of writing tests — coming up with the examples. And it means your contract is more likely to be hit with cases you didn’t consider, which is exactly where the bugs like to hide.

Brownie achieves this by integrating the hypothesis library. Here is our same test, written to take advantage of property-based testing:

When this test executes, the logic is run fifty times. During each run, amount is a different value somewhere between 0 and 10¹⁸. The chosen values are stored in a database, and in subsequent test runs different values are be used. If a value is found that causes the test to fail, this value will always be included in subsequent tests.

That last line is important! While you wouldn’t be wrong to describe this process as randomized testing, it is also repeatable randomized testing. While a bug may take some time to be found, it will never simply go away once it has been discovered.

Writing Property-Based Tests

Writing property-based tests is a fairly straightforward process. The magic is coming from two methods:

• @given is a decorator that converts a test function into a randomized test. It specifies which arguments in the function are to be parametrized, and how that process should be handled.
• strategy is a method for creating test strategies based on smart contract ABI types. A test strategy is a recipe for describing the sort of data you want to generate. It typically defines the data type, lower and upper bounds, and values that should be excluded.

By combining these two methods, we can easily generate property-based tests that will test a behavior of your contract’s specification much more thoroughly than with regular parametrized tests.

Here is another example where we add a second strategy receiver, to parametrize the receiver of the tokens. This new strategy uses the exclude keyword argument to avoid generating a test where the sender and receiver are the same account.

Note that when using exclude, you should also include a separate test for the excluded value(s).

Where to Use Property-Based Tests

If you’re unsure where to begin adding property-based tests within your test suite, here are some ideas:

1. Look for simple tests that verify “when X, then Y”, where there is a direct relationship between X and Y. Balances, approval amounts, user permissions — smart contracts are full of situations where this sort of testing can come in handy.
2. Look for code duplication in your test suite. Are there cases where you have multiple tests verifying the same behavior? Can you refactor these tests into a single, property-based case?

Learning More

To learn more about property-based testing for smart contracts, check out the Brownie Property-Based Testing documentation. It explores of the above functionality in greater depth, and shows how to integrate Hypothesis into your tests to utilize this powerful testing technique.

What’s Next

We’re almost done! In “Part Seven: Stateful Testing”, we delve into stateful testing, an advanced form of property-based testing that can be used to write effective integration tests for complex systems.

You can also follow the Brownie Twitter account, read my other Medium articles, and join us on Gitter.