cargo-fuzz and customize the input to fuzz tests with the arbitrary crate. It also demonstrates how to adapt fuzz tests into reusable property tests with the proptest and proptest-arbitrary-interop crates. It builds on the timelock example.
Run the Example
First go through the setup process to get your development environment configured, then clone thev23.0.0 tag of soroban-examples repository:
cargo-fuzz tool, and to run cargo-fuzz you will need a nightly Rust toolchain:
fuzzing directory and run the cargo fuzz subcommand with the nightly toolchain:
If you’re developing on MacOS you may need to add the
--sanitizer=thread flag in order to fix some known linking errors.Background: Fuzz Testing and Rust
Fuzzing is a kind of testing where new inputs are repeatedly fed into a program in hopes of finding unexpected bugs. This style of testing is commonly employed to increase confidence in the correctness of security-sensitive software. In Rust, fuzzing is most often performed with thecargo-fuzz tool, which drives LLVM’s libfuzzer, though other fuzzing tools are available.
Soroban has built-in support for fuzzing Soroban contracts with cargo-fuzz.
cargo-fuzz is a mutation-based fuzzer: it runs a test program, passing it generated input; while the program is executing, the fuzzer monitors which branches the program takes, and which functions it executes; after execution the fuzzer uses this information to make decisions about how to mutate the previously-used input to create new input that might discover more branches and functions; it then runs the test again with new input, repeating this process for potentially millions of iterations. In this way cargo-fuzz is able to automatically explore execution paths through the program that may never be seen by other types of tests.
If a fuzz tests panics or hard-crashes, cargo-fuzz considers it a failure and provides instructions for repeating the test with the failing inputs.
Fuzz testing is typically an exploratory and interactive process, with the programmer devising schemes for producing input that will stress the program in interesting ways, observing the behavior of the fuzz test, and iterating on the test itself.
Resolving a fuzz testing failure typically involves capturing the problematic input in a unit test. The fuzz test itself may or may not be kept, depending on determinations about the cost of maintaining the fuzzer vs the likelihood of it continuing to find bugs in the future.
While fuzzing non-memory-safe software tends to be more lucrative than fuzzing Rust software, it is still relatively common to find panics and other logic errors in Rust through fuzzing.
In Rust, multiple fuzzers are maintained by the rust-fuzz GitHub organization, which also maintains a “trophy case” of Rust bugs found through fuzzing.
About the Example
The example used for this tutorial is based on thetimelock example program, with some changes to demonstrate fuzzing.
The contract, ClaimableBalanceContract, allows one party to deposit an arbitrary quantity of a token to the contract, specifying additionally: the claimants, addresses that may withdraw from the contract; and the time_bound, a specification of when those claimants may withdraw from the account.
The TimeBound type looks like
ClaimableBalanceContract has two methods, deposit and claim:
deposit may only be successfully called once, after which claim may be called multiple times until the balance is completely drained, at which point the contract becomes dormant and may no longer be used.
Fuzz Testing Setup
For these examples, the fuzz tests have been created for you, but normally you would use thecargo fuzz init command to create a fuzzing project as a subdirectory of the contract under test. To do that you would navigate to the contract directory, in this case, soroban-examples/fuzzing, and execute
cargo-fuzz project is its own crate, which lives in the fuzz subdirectory of the crate being tested. This crate has its own Cargo.toml and Cargo.lock, and another subdirectory, fuzz_targets, which contains Rust programs, each its own fuzz test.
Our soroban-examples/fuzzing directory looks like
Cargo.toml- this is the contract’s manifestCargo.locksrclib.rs- this is the contract code
fuzz- this is the fuzzing crateCargo.toml- this is fuzzing crate’s manifestCargo.lockfuzz_targetsfuzz_target_1.rs- this is a single fuzz testfuzz_target_2.rs
fuzzing/Cargo.toml
Note that cargo has a feature/bug that inhibits LTO of cdylibs when a crate is both a “cdylib” and “rlib”. This can be worked around by building the contract with either
soroban contract build or cargo rustc --crate-type cdylib instead of the typical cargo build.contracttype macro emits additional code needed for running fuzz tests.
Within the fuzzing crate’s manifest one must turn on the “testutils” features in both the contract crate and the soroban-sdk crate:
fuzzing/fuzz/Cargo.toml
A Simple Fuzz Test
First let’s look atfuzz_target_1.rs. This fuzz test does two things: it first deposits an arbitrary amount, then it claims an arbitrary amount.
Again, you can run this fuzzer from the soroban-examples/fuzzing directory with the following command:
fuzzing/fuzz/fuzz_targets/fuzz_target_1.rs
main function, cargo-fuzz uses a special entry point defined by the fuzz_target! macro. This macro accepts a Rust closure that accepts input, any Rust type that implements the Arbitrary trait. Here we have defined a struct, Input, that derives Arbitrary.
cargo-fuzz will be responsible for generating input and repeatedly calling this closure.
To test a Soroban contract, we must set up an Env. Note that we have disabled the CPU and memory budget: this will allow us to fuzz arbitrarily complex code paths without worrying about running out of budget; we can assume that running out of budget during a transaction always correctly fails, canceling the transaction; it is not something we need to fuzz.
Refer to the fuzz_target_1.rs source code for additional setup for this contract.
This fuzzer performs two steps: deposit, then claim:
fuzzing/fuzz/fuzz_targets/fuzz_target_1.rs
try_ client function to invoke the contract. Each contract function can be invoked with a try_ variant, which captures any errors, including panics and crashes, and returns the value on success or an error otherwise. Without using the try_ variant, a panic from within a contract will immediately cause the fuzz test to fail, but in most cases a panic within a contract does not indicate a bug - it is simply how a Soroban contract cancels a transaction. try_ returns a Result, but here we discard it.
Finally, the assert_invariants function is where we make any assertions we can about the state of the contract:
fuzzing/fuzz/fuzz_targets/fuzz_target_1.rs
Interpreting cargo-fuzz Output
If you run cargo-fuzz with fuzz_target_1, from inside the soroban-examples/fuzzing directory, you will see output similar to:
cargo to use the nightly toolchain with the +nightly flag, something that cargo-fuzz doesn’t print in its version of the command.
Another thing to notice is that by default, cargo-fuzz / libfuzzer does not print names of functions in its output, as in the stack trace:
libfuzzer needs the llvm-symbolizer program. On Ubuntu-based systems this can be installed with the llvm-dev package:
libfuzzer will print demangled function names instead of addresses:
Ctrl-C.
Next, let’s look at a single line of fuzzer output:
cov. This is a cumulative measure of branches covered by the fuzzer. When this number stops increasing the fuzzer has probably explored as much of the program as it can. The other columns are described in the libfuzzer documentation.
Finally, lets look at this warning:
libfuzzer only generates input up to 4096 bytes. In a lot of cases, this is probably reasonable, but cargo-fuzz can increase the max_len by appending the argument after --:
libfuzzer documentation for more.
Accepting Soroban Types as Input with the SorobanArbitrary Trait
Inputs to the fuzz_target! macro must implement the Arbitrary trait, which accepts bytes from the fuzzer driver and converts them to Rust values. Soroban types though are managed by the host environment, and so must be created from an Env value, which is not available to the fuzzer driver. The SorobanArbitrary trait, implemented for all Soroban contract types, exists to bridge this gap: it defines a prototype pattern whereby the fuzz_target macro creates prototype values that the fuzz program can convert to contract values with the standard soroban conversion traits, FromVal or IntoVal.
The types of prototypes are identified by the associated type, SorobanArbitrary::Prototype:
SorobanArbitrary include:
i32,u32,i64,u64,i128,u128,I256,U256,(), andbool,Error,Bytes,BytesN,Vec,Map,Address,Symbol,Val,
contracttype attribute, automatically derive SorobanArbitrary. Note that SorobanArbitrary is only derived when the “testutils” Cargo feature is active. This implies that, in general, to make a Soroban contract fuzzable, the contract crate must define a “testutils” Cargo feature, that feature should turn on the “soroban-sdk/testutils” feature, and the fuzz test, which is its own crate, must turn that feature on.
A More Complex Fuzz Test
Thefuzz_target_2.rs example, demonstrates the use of SorobanArbitrary, the advancement of time, and more advanced fuzzing techniques.
This fuzz test takes a much more complex input, where some of the values are user-defined types exported from the contract under test. This test is structured as a simple interpreter, where the fuzzing harness provides arbitrarily-generated “steps”, where each step is either a deposit command or a claim command. The test then treats each of these steps as a separate transaction: it maintains a snapshot of the blockchain state, and for each step creates a fresh environment in which to execute the contract call, simulating the advancement of time between each step. As in the previous example, assertions are made after each step.
The input to the fuzzer looks, in part, like:
fuzzing/fuzz/fuzz_targets/fuzz_target_2.rs
SorobanArbitrary::Prototype associated type to define inputs to the fuzzer. A Soroban Address can only be created with an Env, so cannot be generated directly by the Arbitrary trait. Instead we use the fully-qualified name of the Address prototype, <Address as SorobanArbitrary>::Prototype, to ask for Address’s prototype instead. Then when our fuzzer needs the Address we instantiate it with the FromVal trait:
fuzzing/fuzz/fuzz_targets/fuzz_target_2.rs
The contract we are fuzzing is a timelock contract, where calculation of time is crucial for correctness. So our testing must account for the advancement of time. The contract defines a
TimeBound type and accepts it in the deposit method:
fuzzing/src/lib.rs
DepositCommand:
fuzzing/fuzz/fuzz_targets/fuzz_target_2.rs
SorobanArbitrary::Prototype associated type to accept a TimeBound as input.
To advance time we maintain a LedgerSnapshot, defined in the soroban-ledger-snapshot crate. For each step we call Env::from_snapshot to create a fresh environment to execute the step, then Env::to_snapshot to create a new snapshot to use in the following step.
Here is a simplified outline of how this works. See the full source code for details.
fuzzing/fuzz/fuzz_targets/fuzz_target_2.rs
Converting a Fuzz Test to a Property Test
In addition to fuzz testing, Soroban supports property testing in the style of quickcheck, by using theproptest and proptest-arbitrary-interop crates in conjunction with the SorobanArbitrary trait.
Property tests are similar to fuzz tests in that they generate randomized input. Property tests though do not instrument their test cases or mutate their input based on feedback from previous tests. Thus they are a weaker form of test.
The great benefit of property tests though is that they can be included in standard Rust test suites and require no extra tooling to execute. One might take advantage of this by interactively fuzzing to discover deep bugs, then convert fuzz tests to property tests to help prevent regressions.
The proptest.rs file is a translation of fuzz_target_1.rs to a property test.