Stop tasks when they reach their TTL
https://redmine.teklia.com/issues/8785
Requires #1871 (closed)
A new Ponos task state, State.Cancelled
, represents a task that has been stopped because it went over its TTL. It should be included in the FINAL_STATES
, and comes in second place after Running
in STATES_ORDERING
.
The run_docker_task
RQ task should react to a task that exceeds its TTL similarly to how it handles State.Stopping
: it should call container.stop()
, then set the task's state to State.Cancelled
(rather than Stopped
). A task exceeds its TTL when (timezone.now() - started).total_seconds() > ttl
.
It should also append an error line to the logs, like what is done in run_task_rq
, to report that the task has been cancelled due to reaching its TTL and make the cause of the failure clearer.