I was getting warnings with distutils.LooseVersion because tox kept on installing v0.5.0 and not 0.5.1, which fixes the issue. Turns out we had no version numbers specified on test requirements.
distutils.LooseVersion