Execute docker tasks in RQ

TODO:

• Handle task failure
  • + recursive update (children tasks)
• Ponos auth (via env)
• Publish logs as artifact
• Read artifact of the parent task
• Publish artifact
• Fix existing tests
• Fix exception mentioned in !2227 (comment 224950)
• Add tests for tasks scheduler

Follow-ups:

• Support downloading and extracting ZST extra files (download_extra_files method) #1695 (closed)
• Retry single task #1696 (closed)
• Add tests for RQ agent (docker mocks) #1697 (closed)

arkindex/project/config.py

@@ -86,7 +86,7 @@ def get_settings_parser(base_dir):
     parser.add_option("imports_worker_version", type=uuid.UUID, default=None)
     parser.add_option("workers_max_chunks", type=int, default=10)
     parser.add_option("robots_txt_disallow", type=str, many=True, default=[])
-    parser.add_option("public_hostname", type=public_hostname)
+    parser.add_option("public_hostname", type=public_hostname, default=None)
     parser.add_option("worker_activity_timeout", type=int, default=3600)
     parser.add_option("metrics_port", type=int, default=3000)
     parser.add_option("signup_default_group", type=uuid.UUID, default=None)