Wiki page on troubleshooting intermittent or hard to reproduce test failures

Erwan Rouchet
2022-04-19T11:24:18Z

How to identify if a test impacts another test (what led to https://gitlab.com/arkindex/backend/-/merge_requests/1644)

How to reproduce a test in CI conditions (check that https://wiki.vpn/en/arkindex/dev-doc/ci-test is up to date and link to it)

Using manage.py test options

Filter tests by name or pattern

Run tests in reverse

Run tests in a random order

PDB

Keep the test DB after running the tests

Stop at the first failure

Testing repeatedly

Browsing the test DB with manage.py testserver

Getting more debug information

pprint

rich.inspect

rich.traceback

Logging SQL queries

HTML test coverage report

Handling intermittent failures before they are fixed

Skipping tests

Expected failures

Checklist (things you ought to look at if you really don't know what could cause a test to fail somewhere, but not somewhere else)

Operating system

Python version

Docker image tags and hashes

Environment variables, especially the ARKINDEX_* or PYTHONPATH

Installed Python packages

Backend YAML configuration

django.conf.settings

System load (https://gitlab.com/arkindex/ponos/-/merge_requests/313#note_705086505)