Execute docker tasks in RQ

TODO:

• Handle task failure
  • + recursive update (children tasks)
• Ponos auth (via env)
• Publish logs as artifact
• Read artifact of the parent task
• Publish artifact
• Fix existing tests
• Fix exception mentioned in !2227 (comment 224950)
• Add tests for tasks scheduler

Follow-ups:

• Support downloading and extracting ZST extra files (download_extra_files method) #1695 (closed)
• Retry single task #1696 (closed)
• Add tests for RQ agent (docker mocks) #1697 (closed)

arkindex/ponos/tasks.py

@@ -100,6 +100,11 @@ def run_docker_task(client, task, artifacts_dir):
         "network": "host",
         "volumes": {artifacts_dir: {"bind": "/data", "mode": "rw"}},
     }
+    if task.requires_gpu:
+        # Assign all GPUs to that container
+        # TODO: Make sure this works
+        kwargs["environment"] = {"NVIDIA_VISIBLE_DEVICES": "all"}
+        logger.info("Starting container with GPU support")
     if task.command is not None:
         kwargs["command"] = task.command
     container = client.containers.run(task.image, **kwargs)