Execute docker tasks in RQ

TODO:

• Handle task failure
  • + recursive update (children tasks)
• Ponos auth (via env)
• Publish logs as artifact
• Read artifact of the parent task
• Publish artifact
• Fix existing tests
• Fix exception mentioned in !2227 (comment 224950)
• Add tests for tasks scheduler

Follow-ups:

• Support downloading and extracting ZST extra files (download_extra_files method) #1695 (closed)
• Retry single task #1696 (closed)
• Add tests for RQ agent (docker mocks) #1697 (closed)

arkindex/process/models.py

@@ -397,8 +397,9 @@ class Process(IndexableModel):
         process_builder.build()
         # Save all tasks and their relations
         process_builder.save()
-        # Trigger tasks execution in RQ
-        schedule_tasks(self, process_builder.run)
+        if settings.PONOS_RQ_EXECUTION:
+            # Trigger tasks execution in RQ
+            schedule_tasks(self, process_builder.run)
 
         self.started = timezone.now()
         self.finished = None