Execute docker tasks in RQ

TODO:

• Handle task failure
  • + recursive update (children tasks)
• Ponos auth (via env)
• Publish logs as artifact
• Read artifact of the parent task
• Publish artifact
• Fix existing tests
• Fix exception mentioned in !2227 (comment 224950)
• Add tests for tasks scheduler

Follow-ups:

• Support downloading and extracting ZST extra files (download_extra_files method) #1695 (closed)
• Retry single task #1696 (closed)
• Add tests for RQ agent (docker mocks) #1697 (closed)

arkindex/users/api.py

@@ -53,6 +53,12 @@ from arkindex.users.utils import RightContent, get_max_level
 
 logger = logging.getLogger(__name__)
 
+# Process tasks running in RQ are hidden from user jobs
+VISIBLE_QUEUES = [
+    q for q in QUEUES.keys()
+    if q != "tasks"
+]
+
 
 @extend_schema(tags=["users"])
 @extend_schema_view(
@@ -340,7 +346,7 @@ class JobRetrieve(RetrieveDestroyAPIView):
     serializer_class = JobSerializer
 
     def get_object(self):
-        for queue_name in QUEUES.keys():
+        for queue_name in VISIBLE_QUEUES:
             job = get_queue(queue_name).fetch_job(str(self.kwargs["pk"]))
             if not job:
                 continue