Use a worker for S3 imports

Closes #1795 (closed)

• I removed ProcessBuilder.build_iiif, a leftover of #1302 (closed).
• I made ProcessBuilder._build_task assign the ARKINDEX_WORKER_RUN_ID based on Task.worker_run_id, to avoid duplicating code or forgetting to set the environment correctly.
• The system checks wiki page has been updated.

This needs to be tested with arkindex/workers/import!4

changed milestone to %Arkindex 1.6.2

requested review from @babadie

assigned to @erouchet

added 10 commits

• 33c9f0d6 - 1 commit from branch master
• ea3b710e - Add docker.ingest_image setting
• 7c5c1537 - Add WorkerVersion.objects.ingest_version
• 42fc1ce2 - Add system check
• efbebdfc - Remove unused code from IIIF imports
• b20dfe1a - Automatically set worker run ID env from Task.worker_run
• 4f27703b - Use S3 import worker when building S3 process
• 6ac5d742 - Remove unnecessary configuration hashing
• 8115b309 - Update retry test
• 396dc582 - Update S3 tests

Compare with previous version

added 10 commits

• 23067946 - 1 commit from branch master
• a745f16e - Add docker.ingest_image setting
• 9b7bb99c - Add WorkerVersion.objects.ingest_version
• 64ce4097 - Add system check
• 65a3e949 - Remove unused code from IIIF imports
• eddfac52 - Automatically set worker run ID env from Task.worker_run
• 5f860ba7 - Use S3 import worker when building S3 process
• 87446622 - Remove unnecessary configuration hashing
• deb8aa60 - Update retry test
• bdf7e486 - Update S3 tests

Compare with previous version

I was able to start an import, the docker task got the following env variable

                "ARKINDEX_API_URL=http://localhost:8000/api/v1/",
                "INGEST_S3_REGION=localdev",
                "ARKINDEX_CORPUS_ID=4cb276b6-d6c8-4072-a4d3-8e9b6fda8b90",
                "INGEST_S3_ENDPOINT=https://minio.ark.localhost",
                "ARKINDEX_PROCESS_ID=4a41ad68-34d7-4926-b254-417dbdbf4cff",
                "ARKINDEX_TASK_TOKEN=xxx",
                "INGEST_S3_ACCESS_KEY=minio1234",
                "INGEST_S3_SECRET_KEY=minio1234",
                "ARKINDEX_WORKER_RUN_ID=2b31a88f-b6d4-4f73-bc20-5a02b273e2cb",

Which look good.

The worker itself crashed with the following message

2024-07-03 09:15:37,142 INFO/arkindex_worker: Worker will use /data/current as working directory
2024-07-03 09:15:39,716 INFO/arkindex_worker: Loaded Worker Import S3 @ version 2 using configuration 'Configuration for process 32108f91-00ba-4275-ad95-ddec353fb9fb' from API
2024-07-03 09:15:39,717 INFO/arkindex_worker: Loaded user configuration from WorkerRun
2024-07-03 09:15:39,717 ERROR/worker_bucket_import.worker: 'bucket'

I got a successful run with updated import worker arkindex/workers/import!4

added 16 commits

• bdf7e486...a0f6ced5 - 7 commits from branch master
• 4ba638a7 - Add docker.ingest_image setting
• e488d1b2 - Add WorkerVersion.objects.ingest_version
• 1b521acb - Add system check
• da265a6a - Remove unused code from IIIF imports
• 9baa524b - Automatically set worker run ID env from Task.worker_run
• 1a21f99f - Use S3 import worker when building S3 process
• 1174a0d9 - Remove unnecessary configuration hashing
• ddd53b78 - Update retry test
• fdbf0038 - Update S3 tests

Compare with previous version

resolved all threads

enabled an automatic merge when the pipeline for fdbf0038 succeeds

merged