Run local corpus exports on the high queue
https://redmine.teklia.com/issues/7571
Corpus exports need to be on different queues depending on whether they run on the default
source, or any other source, to reduce issues when exports are running in parallel and running into database connection issues, and give more flexibility to sysadmins to manage the task distribution.
RQ's @job
decorator does not allow to run a task on a specific queue, so we have to split the export task into two.
Please remove the @job
from the export_corpus
function and instead add two more functions to run exports on different queues:
@job('high', ...)
def local_export(corpus_export: CorpusExport) -> None:
assert corpus_export.source == "default"
export_corpus(*args, **kwargs)
@job('export', ...)
def remote_export(corpus_export: CorpusExport) -> None:
assert corpus_export.source != "default"
export_corpus(*args, **kwargs)
arkindex.project.triggers.export_corpus
must be updated to pick the correct task to run.
Please add tests to check that local_export
only allows the default
source and remote_export
does the opposite.