Skip to content

Migrate workers & their versions from a repo towards generic implem

Bastien Abadie requested to merge migrate-worker-versions into master

Closes #1481 (closed)

This new django command allows an instance administrator to quickly migrate all workers from a repos towards a Generic worker along with a model.

As we are publishing our models on Arkindex instances, and removing them from the workers Docker image, we should only have a generic implementation in the worker, and a lot of models (instead of lots of workers).

This command does the following (limited to specified repository):

  1. find a generic worker semi-automatically (looking by name, otherwise asking to choose one worker)
  2. check the generic worker has an available version on main/master. Otherwise crash. We do not want to migrate towards a bad image.
  3. migrate all workers (except training ones) by iterating on them and doing:
    1. removing all totally unused worker versions (there are a lot on preprod !)
    2. find a model matching the name of the worker (asking user to pick one)
    3. mark the model as compatible with the generic worker (will help for later work)
    4. use the last available model version, as we cannot match worker version & model version
    5. ask for confirmation
    6. migrate all worker version by doing:
      1. list all corpus with some ML results linked to that worker version
      2. iterate over all corpus
        1. create a new worker process for that corpus to migrate that worker
        2. create a new worker run in that process using generic worker version + model version on that local process previously found
        3. update all ML results to use that new worker run + generic worker version
        4. delete that worker version as it became useless
    7. delete the worker if it became unused (no versions remaining)

I developed that command against a local databse using preprod dump, so it works on real data (I had to update the weird U-FCN model name from preprod to Doc-UFCN as on prod/demo and cheat by enabling a few GitRef for master branches on generic worker versions).

I also had to open the constraint on WorkerRun so we can have several worker runs on a process using the same worker version (but different models). I now create a Workers process for each worker to migrate and for each corpus impacted by the change.

Edited by Bastien Abadie

Merge request reports

Loading