Skip to content

Support modern configuration in update_system_workers

https://redmine.teklia.com/issues/12378

In order to allow migrating any of the Arkindex export workers to the new configuration, the arkindex update_system_workers command needs to detect whether a worker uses a new format, parse it, use the attributes from there, and update the WorkerConfigurationField instances for each system worker.

teklia_worker.path should be an optional string that specifies the path to a worker under the arkindex/ directory, since that directory is required by the new format. When it is set, we will use the new format, and when it is unset, we use the old format and .arkindex.yml.

The existing behavior with the old format stays the same. With the new format:

  1. the YAML file should be validated using the library;
  2. the docker.command value is extracted like in the old format;
  3. a WorkerVersion is created/updated as with the old format;
  4. when configuration exists within the YAML file:
    1. all existing WorkerConfigurationField on the WorkerVersion should be deleted;
    2. new fields should be created exactly the same way as in ImportWorkerVersion.

The code within ImportWorkerVersion that creates WorkerConfigurationField in bulk from the configuration should be moved to a function so it can be used in both that endpoint and the command.

We could theoretically handle updates more gracefully on those fields, but that would require some complex diff-ing between the existing and new fields. bulk_create(update_conflicts=True) would not be enough, since it would not detect deletions. We could use something like the PostgreSQL MERGE but that would be just as complex. We currently don't need any of this, we can destroy and re-create as much as we want since fields are not linked to any other table.

This should be testable against arkindex/workers/export!27:

pdf_export:
  teklia_worker:
    name: arkindex/workers/export
    version: migrate-modern
    slug: pdf-export
    path: arkindex/pdf-export.yml
Edited by Erwan Rouchet