Speedup package install or at least test job

The test job takes forever to pass, only because of the tox setup (package installation). We should see if it's possible to accelerate that step.