Execute docker tasks in RQ

TODO:

• Handle task failure
  • + recursive update (children tasks)
• Ponos auth (via env)
• Publish logs as artifact
• Read artifact of the parent task
• Publish artifact
• Fix existing tests
• Fix exception mentioned in !2227 (comment 224950)
• Add tests for tasks scheduler

Follow-ups:

• Support downloading and extracting ZST extra files (download_extra_files method) #1695 (closed)
• Retry single task #1696 (closed)
• Add tests for RQ agent (docker mocks) #1697 (closed)

arkindex/ponos/tests/test_api.py

@@ -64,7 +64,8 @@ class TestAPI(FixtureAPITestCase):
         )
         cls.rev = Revision.objects.first()
         cls.process = Process.objects.get(mode=ProcessMode.Workers)
-        cls.process.run()
+        with patch("arkindex.project.triggers.schedule_tasks", lambda *x: None):
+            cls.process.run()
         cls.task1, cls.task2, cls.task3 = cls.process.tasks.all()
 
         # Brand new user and corpus with no preexisting rights