Execute docker tasks in RQ

TODO:

• Handle task failure
  • + recursive update (children tasks)
• Ponos auth (via env)
• Publish logs as artifact
• Read artifact of the parent task
• Publish artifact
• Fix existing tests
• Fix exception mentioned in !2227 (comment 224950)
• Add tests for tasks scheduler

Follow-ups:

• Support downloading and extracting ZST extra files (download_extra_files method) #1695 (closed)
• Retry single task #1696 (closed)
• Add tests for RQ agent (docker mocks) #1697 (closed)

arkindex/ponos/tasks.py

 import logging
 from urllib.parse import urljoin
 
+import requests
 from django.conf import settings
 from django.core.mail import send_mail
 from django.db.models import Count, F, Q
@@ -127,9 +128,17 @@ def run_task_rq(task: Task):
 
         # 3. Read logs (see agent.setup_logging)
         logger.debug("Reading logs from the docker container.")
+        data = b""
         for line in container.logs(stream=True):
-            # TODO: continuously publish the logs as artifact
-            logger.info(line)
+            data += line
+            try:
+                requests.put(
+                    task.logs.s3_put_url,
+                    data,
+                    headers={"Content-Type": "text/plain; charset=utf-8"},
+                )
+            except Exception as e:
+                logger.error(f"An error occurred uploading logs: {e}")
 
         # 4. Retrieve the state of the container
         container.reload()
@@ -137,7 +146,7 @@ def run_task_rq(task: Task):
         if exit_code == 0:
             task.state = State.Completed
         else:
-            logger.error("Task failed.")
+            logger.info("Task failed.")
             task.state = State.Failed
         task.save()
     except Exception as e: