What is mutation testing?
Security is the most important aspect of smart contracts. In SmartPy we have set up a complete testing system because we believe that testing is the best way to understand how code works and to spot errors. How can we be sure that the tests cover all the important cases? With mutation testing.
Conveyor belt metaphor
Imagine a factory that produces toy robots. At the end of its conveyor belt we have added machines for quality assurance: the first machine checks if the robot has a head, the second one the legs... But one day a robot without arms passes all the quality tests. This means that a test machine for the arms was missing.
To be sure not to forget any test, the manager adds mutation testing. It's a special machine that is placed before the quality test. It takes a robot on the conveyor belt and modifies it: by removing a component, by exchanging its arms and legs... If the quality tests do not notice any problem, then this special machine will report that a quality assurance test is missing.
Mutation testing works in three steps:
- We mutate a smart contract.
- We observe how the modification affects the test results.
- If the tests still pass: there is a problem.
- We remove a command.
- We reverse a boolean.
- We remove an error message from fail (
- We remove a branch from a
- We add a
sp.else: sp.failwith(sp.unit)after a
Each mutation is applied to each suitable sub-command and sub-expression of a contract.
For example: the mutation "remove a command" is applied to the command number 1 of the code, the 3 steps of mutation testing are applied and then it starts again by removing the command number 2 on the original contract. And this until all the commands are passed by this mutation. This whole process is done for each mutation.
Currently mutation testing is supported from the SmartPy CLI only.
To add a mutation test, we add a new test containing a
test_scenario and a
mutation_test. In this
mutation_test we add the names of the tests that should be checked.
@sp.add_test(name="Mutation1") def test(): s = sp.test_scenario() with s.mutation_test() as mt: mt.add_scenario("<my test>", contract_id=0) # Replace <my test> with the name of the classic test we've already written. # We make as many `add_scenario` as there are tests.
By default, the system considers the first contract originated in the tested scenario. To test another contract you have to indicate a different
Let's imagine a contract that increments or decrements an integer.
# example.py class MyContract(sp.Contract): def __init__(self): self.init(0) @sp.entrypoint def increment(self): self.data += 1 @sp.entrypoint def decrement(self): self.data -= 1 @sp.add_test(name="Test1") def test(): c1 = MyContract() sc = sp.test_scenario() sc += c1 c1.increment() sc.verify(c1.data == 1) @sp.add_test(name="Mutation1") def test(): s = sp.test_scenario() with s.mutation_test() as mt: mt.add_scenario("Test1")
The attentive reader will have noticed that this test is not complete. Let's see if mutation testing notices it.
SmartPy.sh test example.py example tell us:
# Error displayed when running SmartPy test [error] A mutated contract passed all tests. (example.py, line 26)
Mutation testing has found a gap in the tests for the entrypoint
decrement. Let's see the rest of the error:
Mutated path: entrypoint "decrement" > . Mutated contracted: import smartpy as sp class Contract(sp.Contract): def __init__(self): self.init_type(sp.TInt) @sp.entrypoint def decrement(self): pass @sp.entrypoint def increment(self): self.data += 1
Here we see the mutated contract that passes all the tests while being different from the expected contract.
We can see that the entrypoint "decrement" has been transformed into
pass. So we need to add a check that fails on the mutated contract.
In order to address the gap uncovered by mutation testing let's modify the
# example.py @sp.add_test(name="Test1") def test(): c1 = MyContract() sc = sp.test_scenario() sc += c1 c1.increment() sc.verify(c1.data == 1) c1.decrement() sc.verify(c1.data == 0)
SmartPy.sh test example.py example again.
No errors. This means that mutation testing hasn't uncovered any gaps in our test coverage.
Alternative: add a new test
Instead of modifying the
"Test1" test, we could have added a second scenario.
# example.py # We rename `"Test1"` into `"Increment"` @sp.add_test(name="Increment") def test(): c1 = MyContract() sc = sp.test_scenario() sc += c1 c1.increment() sc.verify(c1.data == 1) # We add a scenario `"Decrement"`. @sp.add_test(name="Decrement") def test(): c1 = MyContract() sc = sp.test_scenario() sc += c1 c1.decrement() sc.verify(c1.data == -1) # We add the two scenarios in the `mutation_test`. @sp.add_test(name="Mutation1") def test(): s = sp.test_scenario() with s.mutation_test() as mt: mt.add_scenario("Increment") mt.add_scenario("Decrement")
SmartPy.sh test example.py example again.
No errors are raised. This means we have written all the necessary tests.