Execute docker tasks in RQ

TODO:

• Handle task failure
  • + recursive update (children tasks)
• Ponos auth (via env)
• Publish logs as artifact
• Read artifact of the parent task
• Publish artifact
• Fix existing tests
• Fix exception mentioned in !2227 (comment 224950)
• Add tests for tasks scheduler

Follow-ups:

• Support downloading and extracting ZST extra files (download_extra_files method) #1695 (closed)
• Retry single task #1696 (closed)
• Add tests for RQ agent (docker mocks) #1697 (closed)

arkindex/ponos/tasks.py

@@ -114,10 +114,18 @@ def run_docker_task(client, task, temp_dir):
     task.state = State.Running
     task.save()
 
-    # 4. Read logs (see agent.setup_logging)
+    # 4. Read logs
     logger.debug("Reading logs from the docker container")
     data = b""
     for line in container.logs(stream=True):
+        # Stop a task scheduled to be stopped
+        task.refresh_from_db()
+        if task.state == State.Stopping:
+            container.stop()
+            task.state = State.Stopped
+            task.save()
+            return
+
         data += line
         try:
             task.logs.s3_object.upload_fileobj(
@@ -128,6 +136,7 @@ def run_docker_task(client, task, temp_dir):
             logger.warning(f"Failed uploading logs for task {task}: {e}")
 
     # 5. Retrieve the state of the container
+    # Reload container to update `attrs` attribute
     container.reload()
     exit_code = container.attrs["State"]["ExitCode"]
     if exit_code != 0: