Document Docker deployment for partners

https://redmine.teklia.com/issues/6934

content/deployment/configuration.md

@@ -39,6 +39,9 @@ s3:
   endpoint: https://minio.ark.localhost
   region: local
 
+# Random characters used to salt cryptographi hashes
+secret_key: LkX7et2k5yh2muoCiTcpKCpZBXQ8fmJXXdSuR98lQn
+
 # Connection to the search engine
 # This is only needed if the search feature is enabled
 solr:
@@ -161,10 +164,6 @@ ID of the IIIF image server linked to the Arkindex instance. Defaults to 1.
 
 The ImageServer may be created from the admin panel or a shell instance.
 
-## `ml_classifiers_dir`
-
-Absolute path of machine learning models used in Arkindex workflows.
-
 ## `public_hostname`
 
 *Since Arkindex 1.0.3.* Root URL of the Arkindex instance, including the scheme and hostname (`http://example.com`).
@@ -173,13 +172,33 @@ Absolute path of machine learning models used in Arkindex workflows.
 
 *Since Arkindex 1.3.4.* Timeout for worker activities, in seconds. This timeout is the time without update after which an existing worker activity can be set to `started`. Defaults to `3600`.
 
+## `imports_worker_version`
+
+TODO
+
+## `metrics_port`
+
+TODO
+
+## `robots_txt_disallow`
+
+TODO
+
+## `signup_default_group`
+
+TODO
+
+## `workers_max_chunks`
+
+TODO
+
 ## `features`
 
 This section configures whether or not some optional features of Arkindex are available or not, using *feature flags*. This is available since Arkindex 0.12.3.
 
 Unlike in other parts of the configuration, unknown keys are not allowed here, to prevent unknown configuration items from being potentially shared over the API.
 
-### `search`
+### `features.search`
 
 *Since Arkindex 0.12.3.* Defines whether or not the search feature is available. Boolean, defaults to `false`.
 
@@ -187,23 +206,17 @@ When disabled, this disables all interactions with ElasticSearch or Solr, causes
 
 Since Arkindex 1.1.0, Elasticsearch has been removed in favor of Solr.
 
-### `selection`
+### `featuresselection`
 
 *Since Arkindex 0.13.1.* Defines whether or not the element selection feature is available. Boolean, defaults to `true`.
 
-### `signup`
+### `features.signup`
 
 *Since Arkindex 0.12.3.* Defines whether or not the sign-up feature is available. Boolean, defaults to `true`.
 
 When disabled, the `Register` button on the frontend is not shown when logged out and the `Register` API endpoint always returns HTTP 400 errors. The password reset and email verification features are still available, though the verification email will not be sent automatically anywhere.
 
-### `transkribus`
-
-*Since Arkindex 0.14.0.* Defines whether or not the Transkribus-related features are available. Boolean, defaults to `true`.
-
-When this is set, the `transkribus` options must also all be configured and valid. This could otherwise result in configuration validation errors or in a `arkindex.E012` error in the system checks.
-
-### `workers`
+### `features.workers`
 
 *Introduced in Arkindex 0.13.3. Removed in Arkindex 1.3.5; workers are now always available.*
 
@@ -211,33 +224,29 @@ Defines whether or not the worker system is available. Boolean, defaults to `fal
 
 ## `database`
 
-### `host`
+### `database.host`
 
 Hostname of the PostgreSQL database. Defaults to `localhost`.
 
-### `port`
+### `database.port`
 
 Port of the PostgreSQL database. Defaults to `9100`.
 
-### `name`
+### `database.name`
 
 Name of the PostgreSQL database. Defaults to `arkindex_dev`.
 
-### `user`
+### `database.user`
 
 Name of a PostgreSQL user to use. Defaults to `devuser`. **Please** change this in production.
 
-### `password`
+### `database.password`
 
 Password for the PostgreSQL user. Defaults to `devdata`. **Please** change this in production.
 
-## `elasticsearch`
-
-No longer in use since Arkindex 1.1.0.
-
-### `hosts`
+### `database.replica`
 
-List of hostnames of the Elasticsearch nodes. A port can be optionally added, otherwise it defaults to `80`. This defaults to a single node, `localhost`.
+TODO
 
 ## `email`
 
@@ -249,29 +258,29 @@ The e-mail configuration is used in three cases:
 
 When the whole configuration section is omitted, e-mail messages that would normally be sent will instead be printed to standard output.
 
-### `error_report_recipients`
+### `email.error_report_recipients`
 
 A list of e-mail addresses to send HTTP 500 error reports to. When unset, this will not send any error reports.
 
-### `host`
+### `email.host`
 
 Hostname of the SMTP server to use to send emails. The server must support TLS.
 
-### `port`
+### `email.port`
 
 Port of the SMTP server to use to send emails. Defaults to `25`.
 
-### `user`
+### `email.user`
 
 Email address to use both as the SMTP username and as the sender address.
 
-### `password`
+### `email.password`
 
 Password for the SMTP server.
 
 ## `influxdb`
 
-### `api_url`
+### `influxdb.api_url`
 
 A URL for an InfluxDB API. Defaults to `http://localhost:8086/`.
 
@@ -281,23 +290,23 @@ This is only used by the `manage.py telegraf` command; when this setting is unse
 
 Configuration related to Redis for asynchronous tasks using [Channels](https://channels.readthedocs.io/en/latest/). This is unrelated to the optional Redis cache configuration.
 
-### `host`
+### `redis.host`
 
 Hostname of the Redis server. Defaults to `localhost`.
 
-### `port`
+### `redis.port`
 
 *Since Arkindex 0.14.4.* The port to use to connect to the Redis server. Defaults to `6379`.
 
-### `db`
+### `redis.db`
 
 *Since Arkindex 0.14.4.* The database to use on the Redis server. Defaults to `0`.
 
-### `password`
+### `redis.password`
 
 *Since Arkindex 0.14.4.* Optional password to use when connecting to the Redis server. Defaults to `null` (no password).
 
-### `timeout`
+### `redis.timeout`
 
 *Since Arkindex 0.14.4.* The default asynchronous task timeout to use. Defaults to `1800`.
 
@@ -307,37 +316,49 @@ Since Arkindex 1.1.0, this parameter only applies to tasks that are not defined
 
 *Since Arkindex 1.1.0.* Defines the asynchronous task timeouts for each of the asynchronous task types that Arkindex uses.
 
-### `export_corpus`
+### `job_timeouts.export_corpus`
 
 *Since Arkindex 1.1.0.* Timeout for the corpus export asychronous task, in seconds. Defaults to `7200`.
 
-### `corpus_delete`
+### `job_timeouts.corpus_delete`
 
 *Since Arkindex 1.1.0.* Timeout for the corpus deletion asychronous task, in seconds. Defaults to `7200`.
 
-### `worker_results_delete`
+### `job_timeouts.worker_results_delete`
 
 *Since Arkindex 1.1.0.* Timeout for the worker result deletion asychronous task, in seconds. Defaults to `3600`.
 
-### `element_trash`
+### `job_timeouts.element_trash`
 
 *Since Arkindex 1.1.0.* Timeout for the element list deletion asychronous task, in seconds. Defaults to `3600`.
 
-### `move_element`
+### `job_timeouts.move_element`
 
 *Since Arkindex 1.1.0.* Timeout for the element move asychronous task, in seconds. Defaults to `3600`.
 
-### `initialize_activity`
+### `job_timeouts.initialize_activity`
 
 *Since Arkindex 1.1.0.* Timeout for the worker activity initialization asynchronous task, in seconds. Defaults to `3600`.
 
-### `process_delete`
+### `job_timeouts.process_delete`
 
 *Since Arkindex 1.1.0.* Timeout for the process deletion asychronous task, in seconds. Defaults to `3600`.
 
+### `job_timeouts.reindex_corpus`
+
+TODO
+
+### `job_timeouts.notify_process_completion`
+
+TODO
+
+### `job_timeouts.task`
+
+TODO
+
 ## `cache`
 
-### `type`
+### `cache.type`
 
 Required if `cache` is set; defines the type of cache, implying the possible requirement of other properties on this configuration block. Possibles values are:
 
@@ -349,13 +370,13 @@ Required if `cache` is set; defines the type of cache, implying the possible req
 
 When this is unset, this is set to `memory`. If `arkindex_env` is set to `dev` and this is unset, this is set to `dummy`.
 
-### `url`
+### `cache.url`
 
 Hostname and optional port number for a [memcached](https://www.memcached.org/) or Redis instance.
 
 Required only with `memcached` and `redis` caches; ignored for other cache types.
 
-### `path`
+### `cache.path`
 
 Path to an existing directory in which requests can be cached. Required only with the `filesystem` cache and otherwise ignored.
 
@@ -365,7 +386,7 @@ This section defines configuration items specific to Cross-Origin Resource Shari
 
 To learn more about CORS, browse the [Mozilla docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS).
 
-### `origin_whitelist`
+### `cors.origin_whitelist`
 
 A list of CORS origins, as defined in [RFC 6454](https://tools.ietf.org/html/rfc6454#section-3.2): a URI scheme, hostname and port. It is possible to omit the port for `https://` and `http://`; they will default to `443` and `80` respectively.
 
@@ -376,7 +397,7 @@ This defaults to the following :
 
 Since Arkindex 0.14.0, URI schemes are required for all values of this parameter.
 
-### `suffixes`
+### `cors.suffixes`
 
 A list of regular expressions that can match hostname suffixes. They will be prepended with `^https://.+`.
 
@@ -386,13 +407,13 @@ This may be used in conjunction with `origin_whitelist`: when an origin does not
 
 Configuration related to cross-site request forgery attacks protection.
 
-### `cookie_domain`
+### `csrf.cookie_domain`
 
 This sets the `Domain` option on the CSRF cookie. When unset, this disables the `Domain` option.
 
 See Django's documentation on [`CSRF_COOKIE_DOMAIN`](https://docs.djangoproject.com/en/3.1/ref/settings/#csrf-cookie-domain).
 
-### `cookie_name`
+### `csrf.cookie_name`
 
 Sets a name for the CSRF token cookie. This defaults to `arkindex.csrf`.
 
@@ -400,7 +421,7 @@ Changing this name to a value other than the default may impact authentication o
 
 See Django's documentation on [`CSRF_COOKIE_NAME`](https://docs.djangoproject.com/en/3.1/ref/settings/#csrf-cookie-name).
 
-### `trusted_origins`
+### `csrf.trusted_origins`
 
 A list of hosts where unsafe requests (`POST`, `PUT`, `DELETE`) are allowed over HTTPS. This requires requests to include one of those hosts as the `Referer`.
 
@@ -408,13 +429,13 @@ A subdomain catch-all is accepted, such as `.teklia.com`, to allow any subdomain
 
 See Django's documentation on [`CSRF_TRUSTED_ORIGINS`](https://docs.djangoproject.com/en/3.1/ref/settings/#csrf-trusted-origins).
 
-### `cookie_samesite`
+### `csrf.cookie_samesite`
 
 The value of the `SameSite` flag on the CSRF cookie. This flag prevents the cookie from being sent in cross-site requests. Either `lax`, `strict`, `none` or `false`. `false` removes the flag. Defaults to `Lax`.
 
 See Django's documentation on [`CSRF_COOKIE_SAMESITE`](https://docs.djangoproject.com/en/3.1/ref/settings/#csrf-cookie-samesite).
 
-### `cookie_secure`
+### `csrf.cookie_secure`
 
 *Since Arkindex 0.14.0*
 
@@ -424,25 +445,25 @@ See Django's documentation on [`CSRF_COOKIE_SECURE`](https://docs.djangoproject.
 
 ## `session`
 
-### `cookie_domain`
+### `session.cookie_domain`
 
 This sets the `Domain` option on the session cookie. When unset, this disables the `Domain` option.
 
 See Django's documentation on [`SESSION_COOKIE_DOMAIN`](https://docs.djangoproject.com/en/3.1/ref/settings/#session-cookie-domain).
 
-### `cookie_name`
+### `session.cookie_name`
 
 Sets a name for the session cookie. This defaults to `arkindex.auth`.
 
 See Django's documentation on [`SESSION_COOKIE_NAME`](https://docs.djangoproject.com/en/3.1/ref/settings/#session-cookie-name).
 
-### `cookie_samesite`
+### `session.cookie_samesite`
 
 The value of the `SameSite` flag on the session cookie. This flag prevents the cookie from being sent in cross-site requests. Either `lax`, `strict`, `none` or `false`. `false` removes the flag. Defaults to `Lax`.
 
 See Django's documentation on [`SESSION_COOKIE_SAMESITE`](https://docs.djangoproject.com/en/3.1/ref/settings/#session-cookie-samesite).
 
-### `cookie_secure`
+### `session.cookie_secure`
 
 *Since Arkindex 0.14.0*
 
@@ -454,7 +475,7 @@ See Django's documentation on [`SESSION_COOKIE_SECURE`](https://docs.djangoproje
 
 *Since Arkindex 1.0.1.*
 
-### `api_url`
+### `solr.api_url`
 
 Base URL of the Solr API. Defaults to `http://localhost:8983/solr/`.
 
@@ -462,35 +483,35 @@ Base URL of the Solr API. Defaults to `http://localhost:8983/solr/`.
 
 *Since Arkindex 1.1.1*
 
-### `message`
+### `banner.message`
 
 The markdown message that will be displayed in the frontend.
 
-### `style`
+### `banner.style`
 
 The style used to display the message among `info`, `success`, `warning`, `error`. Defaults to `info`.
 
 ## `static`
 
-### `cdn_assets_url`
+### `static.cdn_assets_url`
 
 URL to the root of an Arkindex frontend assets directory. When this variable is set, the backend will serve an index page to load the frontend, and frontend assets will be looked for in a subdirectory of this URL corresponding to the version number defined in `frontend_version`.
 
-### `frontend_version`
+### `static.frontend_version`
 
 When `cdn_assets_url` is set, this version number will be used when looking up frontend assets. This defaults to the backend's version number.
 
-### `mirador_url`
+### `static.mirador_url`
 
 URL to a [Mirador](https://projectmirador.org/) instance. An absolute URL to an IIIF manifest will be directly appended by the frontend to this URL; you may want to add a query string argument such as `https://universalviewer.io/uv.html?manifest=`. Note that Mirador may need to be modified to use [`XMLHttpRequest.withCredentials`](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/withCredentials), as IIIF manifests on private corpora will require authentication.
 
-### `root_path`
+### `static.root_path`
 
 Absolute path for collected static files during deployment.
 
 See the Django documentation on [`STATIC_ROOT`](https://docs.djangoproject.com/en/3.1/ref/settings/#std:setting-STATIC_ROOT).
 
-### `universal_viewer_url`
+### `static.universal_viewer_url`
 
 URL to a [Universal Viewer](http://universalviewer.io/) instance. An absolute URL to an IIIF manifest will be directly appended by the frontend to this URL; you may want to add a query string argument such as `https://universalviewer.io/uv.html?manifest=`. Note that Universal Viewer may need to be modified to use [`XMLHttpRequest.withCredentials`](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/withCredentials), as IIIF manifests on private corpora will require authentication.
 
@@ -498,41 +519,41 @@ URL to a [Universal Viewer](http://universalviewer.io/) instance. An absolute UR
 
 This section defines configuration items specific to features using Amazon S3 and other S3-compatible APIs.
 
-### `access_key_id`
+### `s3.access_key_id`
 
 The Access Key ID for read/write access to S3 buckets.
 
-### `secret_access_key`
+### `s3.secret_access_key`
 
 The Secret Access Key for read/write access to S3 buckets.
 
-### `endpoint`
+### `s3.endpoint`
 
 An optional custom endpoint to use to access S3 buckets, to use another file storage with an S3-compatible API such as [MinIO](https://min.io/).
 
-### `region`
+### `s3.region`
 
 The AWS region the S3 buckets are located in. This has no effect when `endpoint` is set.
 
-### `thumbnails_bucket`
+### `s3.thumbnails_bucket`
 
 Name of the S3 bucket to use for element thumbnails. This defaults to `thumbnails`.
 
-### `staging_bucket`
+### `s3.staging_bucket`
 
 Name of the S3 bucket to use for DataFile uploads. This defaults to `staging`.
 
 Note that for the frontend to be able to upload files to the staging bucket, you will need to configure CORS support on the bucket. MinIO has it enabled by default, but S3 does not. Learn more about CORS support on S3 buckets [here](https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html).
 
-### `ponos_logs_bucket`
+### `s3.ponos_logs_bucket`
 
 Name of the S3 bucket to use for task logs. This defaults to `ponos-logs`.
 
-### `ponos_artifacts_bucket`
+### `s3.ponos_artifacts_bucket`
 
 Name of the S3 bucket to use for task artifacts. This defaults to `ponos-artifacts`.
 
-### `training_bucket`
+### `s3.training_bucket`
 
 *Since Arkindex 1.2.3.*
 
@@ -544,25 +565,25 @@ Name of the S3 bucket to use for Machine Learning model training. This defaults
 
 This section defines configuration items specific to features using Amazon S3 and other S3-compatible APIs for image ingestion.
 
-### `access_key_id`
+### `ingest.access_key_id`
 
 The Access Key ID for read/write access to S3 buckets.
 
-### `secret_access_key`
+### `ingest.secret_access_key`
 
 The Secret Access Key for read/write access to S3 buckets.
 
-### `endpoint`
+### `ingest.endpoint`
 
 An optional custom endpoint to use to access S3 buckets, to use another file storage with an S3-compatible API such as [MinIO](https://min.io/).
 
-### `region`
+### `ingest.region`
 
 The AWS region the S3 buckets are located in. This has no effect when `endpoint` is set.
 
 ## `docker`
 
-### `tasks_image`
+### `docker.tasks_image`
 
 A Docker tag for the Arkindex tasks image. This defaults to `registry.gitlab.teklia.com/arkindex/tasks`. When the image is specified without a tag, `:latest` is assumed.
 
@@ -570,11 +591,11 @@ This is used to specify which image to use for Ponos tasks when running imports
 
 ## `ponos`
 
-### `private_key`
+### `ponos.private_key`
 
 Path to an elliptic curve private key file to use as the server private key for secure registration of Ponos agents. This defaults to `$BASE_DIR/ponos.key`, where `$BASE_DIR` is the directory of the `arkindex` package.
 
-### `default_env`
+### `ponos.default_env`
 
 Default environment variables sent along with every Ponos workflow the Arkindex backend starts. For default tasks and Arkindex client auto-configuration, the following three variables should be defined:
 
@@ -602,42 +623,20 @@ Allows autoconfiguration of the [Arkindex client](http://pypi.org/project/arkind
 
 The CSRF cookie name is automatically deduced from the `csrf.cookie_name` setting and this variable is always set, but it is possible to override this value explicitly in the configuration file.
 
-## `gitlab`
-
-Configuration related to the GitLab integration.
-
-### `app_id`
-
-An application ID as returned in the [GitLab OAuth application creation](https://docs.gitlab.com/ee/integration/gitlab.html) process.
-
-### `app_secret`
-
-A secret key as returned in the [GitLab OAuth application creation](https://docs.gitlab.com/ee/integration/gitlab.html) process.
-
 ## `sentry`
 
 Configuration related to the Sentry integration.
 
-### `dsn`
+### `sentry.dsn`
 
 Set the Data Source Name for the Sentry project related to the backend. When unset, the backend's Sentry integration is disabled.
 
-### `frontend_dsn`
+### `sentry.frontend_dsn`
 
 Set the Data Source Name for the Sentry project related to the frontend. This will be passed to the frontend when using retrieving it from a CDN using `static.cdn_assets_url`. When unset, the frontend's Sentry integration will be disabled.
 
-## `transkribus`
-
-Configuration related to Transkribus imports. When the `transkribus` feature flag is set, all properties are required to be set and valid. When the feature flag is disabled, this is ignored.
-
-### `email`
-
-Email address of a Transkribus account that Arkindex will use to export collections from. This address will be displayed publicly to allow users to add this account to their collections' editors.
-
-### `password`
-
-Password of the Transkribus account that Arkindex will use to export collections from.
+## `export`
 
-### `worker_version`
+### `export.ttl`
 
-UUID of a `WorkerVersion` that the Transkribus import will use to create elements and transcriptions.
+TODO