Skip to content
Snippets Groups Projects
Verified Commit 84260e3e authored by Yoann Schneider's avatar Yoann Schneider :tennis:
Browse files

mermaid flowcharts

parent 4cd08fdd
No related branches found
No related tags found
No related merge requests found
Pipeline #80098 passed
......@@ -30,7 +30,7 @@ repos:
- id: trailing-whitespace
- id: check-yaml
args: [--allow-multiple-documents]
exclude: "^worker-{{cookiecutter.slug}}/.arkindex.yml$"
exclude: "^worker-{{cookiecutter.slug}}/.arkindex.yml$|^mkdocs.yml$"
- id: mixed-line-ending
- id: name-tests-test
args: ['--django']
......
......@@ -8,13 +8,50 @@ The developer mode was designed to help worker developers reproduce and test how
would behave on Arkindex. This is why the configuration process in this mode mirrors the operations done on Arkindex while
replacing configuration API calls by CLI arguments.
The developer mode is enabled when at least either:
The developer mode (or `read-only` mode) is enabled when at least either:
- the `--dev` CLI argument is used,
- the `ARKINDEX_WORKER_RUN_ID` variable was not set in the environment.
None of these happen when running on Arkindex.
## Parallel between both modes
```mermaid
flowchart TB
subgraph configure[Configuration step]
argument_parsing[CLI argument parsing]
end
argument_parsing --> is_read_only{IsReadOnly?}
is_read_only -- Yes --> devMode
is_read_only -- No --> arkindexMode
subgraph arkindexMode[Arkindex mode]
direction TB
subgraph workerConfiguration[Worker configuration]
direction TB
retrieveWorkerRun["API call to RetrieveWorkerRun"] --> userconfig_defaults[Initialize user configuration with default values]
userconfig_defaults --> load_secrets_API["Load Secrets using API calls to RetrieveSecret"]
load_secrets_API --> load_user_config[Override user configuration by values set by user]
load_user_config --> load_model_config["Load model configuration"]
end
workerConfiguration --> cacheConfiguration
subgraph cacheConfiguration[Base worker cache setup]
direction TB
get_paths_from_parent_tasks["Retrieve paths of parent tasks' cache databases"] --> initialize_db[Create cache database and its tables]
initialize_db --> merge_parent_databases[Merge parents databases]
end
end
subgraph devMode[Developer mode]
direction TB
subgraph devWorkerConfiguration[Worker configuration]
direction TB
configuration_parsing[CLI config argument parsing] --> corpus_id[Read Corpus ID from environment]
corpus_id --> load_secrets[Load secret in local developer storage]
end
end
classDef pyMeth font-style:italic
```
## Arkindex mode
The details of a worker execution (what is called a **WorkerRun**) on Arkindex are stored in the backend. The first step of the configuration is to retrieve this information using the Arkindex API. The [RetrieveWorkerRun](https://demo.arkindex.org/api-docs/#tag/process/operation/RetrieveWorkerRun) endpoint gives information about:
......@@ -27,7 +64,7 @@ The details of a worker execution (what is called a **WorkerRun**) on Arkindex a
- the model version used in this worker execution,
- the configuration stored in this model version.
This step shows that there are a lot of sources for the actual configuration that the worker will use. The principal is that the user always has the last word. Any parameter the user chooses will override whatever was previously set in other configurations.
This step shows that there are a lot of sources for the actual configuration that the worker can use. Nothing is overridden by default, the worker has to do it in its overridden version of the configure method. In the end, any parameter set by the user **must** be applied over other known configurations.
The worker configuration may specify default values for some parameters (see [this section](../workers/yaml.md#setting-up-user-configurable-parameters) for more details about worker configuration). These default values are stored in the `user_configuration` dictionary attribute.
......
......@@ -10,3 +10,52 @@ This section presents
- the architecture
- additional configuration steps
- element processing
The following graph describes what happens when running the worker, either on Arkindex or locally. Words in italic font are actual method calls in the worker.
```mermaid
flowchart LR
subgraph all[Worker execution]
direction LR
subgraph id1[Worker initialization]
init
end
run -.-> configure
subgraph id2[Inference]
direction TB
configure --> list_elements
list_elements --> element_processing
subgraph id3[Loop over each element]
element_processing --> element_processing
end
element_processing -- Save ML report to disk --> reporting
end
init --> run
end
classDef pyMeth font-style:italic
class init,run,configure,list_elements pyMeth
```
More details about the `element_processing` step.
```mermaid
flowchart LR
subgraph all[Element processing]
direction LR
subgraph id1[Element details retrieval]
retrieve_element
end
retrieve_element --> update_activity_started
subgraph id2[Processing]
direction LR
update_activity_started[update_activity] --> process_element -- No errors --> update_activity_processed
update_activity_started -- to Started --> update_activity_started
update_activity_processed[update_activity] -- to Processed --> update_activity_processed
update_activity_error[update_activity] -- to Error --> update_activity_error
end
process_element -- Errors found --> update_activity_error
end
classDef pyMeth font-style:italic
class process_element,update_activity_started,update_activity_error,update_activity_processed pyMeth
```
......@@ -104,7 +104,11 @@ markdown_extensions:
- admonition # syntax coloration in code blocks
- codehilite
- pymdownx.details
- pymdownx.superfences
- pymdownx.superfences:
custom_fences:
- name: mermaid
class: mermaid
format: !!python/name:pymdownx.superfences.fence_code_format # yamllint disable-line
copyright: Copyright © Teklia
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment