Skip to content
Snippets Groups Projects
Commit 291f9d45 authored by Yoann Schneider's avatar Yoann Schneider :tennis: Committed by Manon Blanco
Browse files

New documentation structure

parent 86fee1f7
No related branches found
No related tags found
1 merge request!545New documentation structure
Pipeline #172650 passed
Showing
with 41 additions and 22 deletions
......@@ -38,7 +38,7 @@ repos:
rev: v0.39.0
hooks:
- id: markdownlint
args: ['--disable', 'MD013', 'MD046', '--']
args: ['--fix', '--disable', 'MD013', 'MD046', '--']
- repo: meta
hooks:
- id: check-useless-excludes
......
......@@ -71,9 +71,9 @@ This step shows that there are a lot of sources for the actual configuration tha
The convention is to always give the final word to the user. This means that when the user configuration is filled, its values must be the last to override the worker's `config` attribute. If a model configuration was set, its values must override this attribute before the user configuration's.
The worker configuration may specify default values for some parameters (see [this section](../workers/yaml.md#setting-up-user-configurable-parameters) for more details about worker configuration). These default values are stored in the `user_configuration` dictionary attribute.
The worker configuration may specify default values for some parameters (see [this section](yaml.md#setting-up-user-configurable-parameters) for more details about worker configuration). These default values are stored in the `user_configuration` dictionary attribute.
This is also when the secrets (see [this section](../secrets/usage.md#declaring-secrets-in-workers) to learn more about secrets) are actually downloaded. They are stored in the `secrets` dictionary attribute.
This is also when the secrets (see [this section](secrets.md#declaring-secrets-in-workers) to learn more about secrets) are actually downloaded. They are stored in the `secrets` dictionary attribute.
An Arkindex-mode exclusive step is done after all that: the cache setup. Some workers benefit a lot, performance-wise, from having a SQLite cache artifact from previous workers. This is mostly used in processes with multiple workers with dependencies, where the second worker needs the results of the first one to work. The database is initialized, the tables created and its version checked as it must match the one supported by the Arkindex instances. The database is then merged with any other database generated by previous worker runs.
......@@ -81,9 +81,9 @@ Once all information is retrieved and stored in the worker, the configuration is
## Developer mode
In the developer mode, the worker execution is not linked to anything on Arkindex. Therefore, the only configuration the worker can use is provided via the `--config` CLI argument. It supports YAML-formatted file and it should be similar to the `configuration` section of the [worker configuration file](../workers/yaml.md#single-worker-configuration), without the `user_configuration` details. More details about how to create the local worker configuration are available in [this section](../workers/run-local.md).
In the developer mode, the worker execution is not linked to anything on Arkindex. Therefore, the only configuration the worker can use is provided via the `--config` CLI argument. It supports YAML-formatted file and it should be similar to the `configuration` section of the [worker configuration file](yaml.md#single-worker-configuration), without the `user_configuration` details. More details about how to create the local worker configuration are available in [this section](../run/local.md).
The multiple configuration sources from the Arkindex-mode are merged into a unique one here. The configuration parameters are parsed as well as the list of required secrets. The secrets are loaded using a local Arkindex client. Again, see the [section about local execution](../workers/run-local.md) for more details.
The multiple configuration sources from the Arkindex-mode are merged into a unique one here. The configuration parameters are parsed as well as the list of required secrets. The secrets are loaded using a local Arkindex client. Again, see the [section about local execution](../run/local.md) for more details.
One information cannot be retrieved directly from the configuration file and is required in some cases: the ID of the Arkindex corpus which the elements processed belong to. This is retrieved via the `ARKINDEX_CORPUS_ID` environment variable.
......
# Usage
# Use secrets in your worker
## Overview
Secrets are text payloads shared securely between the Arkindex instance and any worker. It is generally used to store sensitive values that may give Arkindex users access to any resources that should be private, for example because they cost money or are proprietary.
For more information about secrets, please visit [the Arkindex documentation](https://doc.arkindex.org/secrets/).
## Accessing secrets in the API
......@@ -35,7 +41,7 @@ missing from the Arkindex instance.
```
For more information on the `.arkindex.yml` file,
see [YAML configuration](../workers/yaml.md).
see [YAML configuration](yaml.md).
## Accessing secrets in Python code
......
......@@ -86,18 +86,6 @@ include:
`docker`
: Regroups Docker-related configuration attributes:
<!--
TODO: Make the path relative to the YAML file itself, in the case of a
separate file for a single worker?
https://gitlab.teklia.com/arkindex/tasks/-/issues/95
-->
<!--
TODO: Implement this!
https://gitlab.teklia.com/arkindex/tasks/-/issues/93
`image`: Tag of an existing Docker image to use for this worker instead of building a
custom image from a Dockerfile.
-->
- `build`
: Path towards a Dockerfile used to build this worker, relative to the root of
the repository. Defaults to `Dockerfile`.
......
# Worker Implementation
# Implement your worker
This section presents
- the different stages happening during a worker execution:
- the initialization
- the [configuration](./configure.md)
- the [configuration](configure/index.md)
- the execution
- the conception of a worker
- the architecture
......
# Artifacts
!!! warning
This section is under construction.
# Arkindex API Helpers
!!! warning
This section is under construction.
# Processing
!!! warning
This section is under construction.
# Run your worker in Docker
!!! warning
This section is under construction.
# Run your worker
!!! warning
This section is under construction.
# Running your worker locally
# Run your worker locally
Once you have implemented a worker, you can run it on some Arkindex elements on your own machine to test it.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment