Debugging Ethereum Reference Tests

There’s an exceptionally valuable set of ethereum reference tests that are run by most or all of the different Ethereum clients to ensure they actually implement the specifications in a compatible way. They’re one of the most valuable resources for anyone developing an Ethereum client.

The Aleth project maintains the official test client called testeth but it’s a little cryptic to work out how to actually run things with it and then use that to debug failures happening in the client you’re actually developing. So this is what I’ve found useful:

Firstly, you need to checkout Aleth and compile it following their instructions. The one catch here is that you need to enable EVM tracing by adding the -D VMTRACE=1 flag when doing the initial cmake .. Once the whole build is finished you should find testeth in aleth/build/test/testeth Copy that to somewhere on your PATH.

Then you need to check out the actual reference tests and set ETHEREUM_TEST_PATH to point to where you checked it out.

Now you should be able to start running tests. Running a single test with loads of debug looks like:

./testeth -t GeneralStateTests/stCreate2 -- --singletest create2collisionStorage --verbosity 3 --vmtrace --statediff | less -R

The param after -t needs to point to a directory containing the group of tests you want to run. It could be as high level as GenerateStateTests or BlockchainTests or point to one of the directories under that. The –singletest option lets you run just one file underneath that directory. The name is the filename without the .json extension.

–verbosity 3 –vmtrace –statediff turns on pretty comprehensive debugging information so you get a full EVM trace, plus a record of the changes to state that were made (it appears to show each change as well as the final state though I’m not sure I’ve fully understood the format yet).

And then finally piping to less -R allows you to scroll up and down and search through the output easily. The -R tells less to render terminal control characters so you still get the pretty colours.

Now you have a reference client running the test and providing a ton of information about what the test expects to happen, go run the test in the client your developing, compare outputs, beat your head against a wall for a while and eventually find and fix your bug.

And a massive thanks to everyone who’s been involved in building up these reference tests. If you ever have to pay for your own beer at DevCon the community is doing something very wrong.