Slay the Workflow

Closes #1475 (closed)

Implements part of teklia/requests#98 (Process.started)

• RetrieveWorkflow, UpdateWorkflow and PartialUpdateWorkflow have been removed entirely; I simply couldn't get them to work with a process properly and it was easier to merge their behaviors into RetrieveProcess, UpdateProcess and PartialUpdateProcess.

  • The frontend uses RetrieveWorkflow to show a process' status.
  • The CLI uses RetrieveWorkflow for arkindex process report and arkindex process recover.
  • The API client's README uses RetrieveWorkflow in an example, but this example uses an endpoint that does not exist already (RetrieveTaskLog).
  • Only the frontend uses PartialUpdateWorkflow, to stop processes.
  • Nobody on code.vpn uses UpdateWorkflow.

• The workflow field has been removed from processes returned in ListProcess or RetrieveProcess. This was the URL to a workflow, not an UUID.

  • The CLI uses the workflow in arkindex process report and arkindex process recover.
  • The frontend uses the workflow for status polling, as well as to detect whether a process is "configured" or not.

• The with_workflow option of ListProcesses has been renamed to with_tasks. This only affects the frontend.

• RetrieveProcess now returns the farm and the tasks, which were previously returned by RetrieveWorkflow.

• The workflow_id on CreateTask is now process_id, which affects the Git import task and the Ponos agent

• The Ponos agent can no longer provide the PONOS_WORKFLOW environment variable, which affects the Git import task

• The task logs and artifacts were stored on S3 using URLs that contain the workflow IDs. They now use only the task ID, to reduce the complexity if we ever have to do other migrations.

  • Artifacts <workflow ID>/<task ID>/* → <task ID>/*
  • Task logs <workflow ID>/run_<run>/<task ID>.log → <task ID>.log

  ⚠ Nothing is provided here to make the migration: the renaming will have to be done on the infra side as discussed on Whereby.

• The manage.py cleanup command used to remove expired workflows, but not expired processes. It now deletes all tasks on expired processes, but does not delete expired processes since they would imply deleting WorkerRuns or WorkerActivities, which would be a mess.

Follow-ups

• Fix the frontend teklia/arkindex/frontend!1491
• Fix tasks teklia/arkindex/tasks!352
• Fix the Ponos agent teklia/arkindex/ponos-agent!20
• Update the CLI's arkindex process report and arkindex process recover to adapt to the RetrieveWorkflow removal
• Write the Ponos logs and artifacts renaming script, which can also be useful to developers for their local setups
• Restore the behavior of a process retry to what it had before the recipe removal: the tasks should be regenerated via build_workflow, not cloned. This was not possible without the recipe, but is now possible without the Workflow. This currently prevents restarting a Repository process.