Execute docker tasks in RQ

TODO:

• Handle task failure
  • + recursive update (children tasks)
• Ponos auth (via env)
• Publish logs as artifact
• Read artifact of the parent task
• Publish artifact
• Fix existing tests
• Fix exception mentioned in !2227 (comment 224950)
• Add tests for tasks scheduler

Follow-ups:

• Support downloading and extracting ZST extra files (download_extra_files method) #1695 (closed)
• Retry single task #1696 (closed)
• Add tests for RQ agent (docker mocks) #1697 (closed)

arkindex/ponos/tasks.py

@@ -115,9 +115,7 @@ def run_docker_task(client, task, temp_dir):
     if task.command is not None:
         kwargs["command"] = task.command
     container = client.containers.run(task.image, **kwargs)
-    # Wait for
     while container.status == "created":
-        sleep(TASK_DOCKER_POLLING)
         container.reload()
     if container.status != "running":
         raise Exception("Container was not updated to state running")