Rework due to API changes
This depends on some work in progress in Base-Worker.
Arkindex 1.6.0 introduces breaking changes in its Dataset API. This worker will no longer depend on arkindex_worker.worker.DatasetWorker
but instead inherit from arkindex_worker.worker.BaseWorker
and arkindex_worker.worker.dataset.DatasetMixin
.
Unlike the new DatasetWorkers, this worker will work at Dataset level, grouping all sets in the process to handle the dataset directly.
Rewrite run.
def run(self):
# Configure
# List process sets
# Group by dataset
# Sets should already be sorted by dataset name so a simple groupby should be enough
# Make sure the dataset state is either `Open` or `Error`
# Update state to `Building`
# Call process_dataset but using the iterator on the sets
# Update state to `Complete` or `Error`
The idea is to still have error catching, just like we do in https://gitlab.teklia.com/workers/base-worker/-/blob/112551b9a853a5ae31117f46c1271d4e7617a31a/arkindex_worker/worker/init.py#L321
Edited by Yoann Schneider