Merge version 1.8.0

content/.pages

@@ -7,6 +7,7 @@ nav:
 - processes
 - models
 - secrets
+- budgets
 - training
 - How to: howto
 - deployment

content/budgets/index.md

+---
+title: Budgets
+---
+
+!!! info
+
+    This feature is only available for Enterprise Edition customers.
+
+**Budgets** allow to keep track of and limit the execution costs of [processes](../processes/index.md) when they run [workers with costs](../workers/advanced.md#costs).
+
+A budget can be associated to one or more [projects](../project/index.md), and a project can only be linked to one budget.
+
+A budget has [access rights](../users/rights/index.md#budget-access) distinct from those of projects, allowing for more precise management of user rights, for example by allowing a user to contribute to a project without being able to run workers with costs, or to manage a budget without having access to the documents in the associated projects.
+
+## Budget entries
+
+Within a budget, **budget entries** represent changes in the funds available on the budget, with a description and optionally an associated user or process.
+
+Typically, instance administrators will add budget entries for their user account with positive values through the [administration interface](../deployment/administration.md#administration-interface) to add funds to the budget, and budget entries will be automatically created with negative values when executing a process.
+
+Budget entries can have values ranging from `-999,999,999.999` to `999,999,999.999`. All monetary values are represented in euros (€).
+
+Using euros, rather than a virtual currency or point system, helps users better grasp how much their processes cost in real money. This is however purely informative. Arkindex does not directly handle any financial information, and does not have an invoice or payment system.
+
+## Browsing budgets
+
+In the top-right user menu, the **Budgets** action shows a list of all budgets that the user has access to. Clicking on any of those budgets will show their details page, including the currently available funds computed from all budget entries, the full list of budget entries, and the budget's memberships.
+
+It is also possible to view the budget associated to a project from the [project information](../project/index.md#web-interface) page, under the **Details** tab.
+
+From the status view of a process, the **Budget entries** action of the **Actions** menu offers a list of all the budget entries associated with this specific process, along with the total execution cost for this process.
+
+## Administration
+
+Other than viewing them and [managing their memberships](../users/rights/index.md#budget-access), most operations on budgets are only possible through the [administration interface](../deployment/administration.md#administration-interface). This includes:
+
+* creating a new budget;
+* editing an existing budget;
+* deleting a budget;
+* assigning a budget to a project;
+* creating budget entries.
+
+### Creating budget entries for processes
+
+The `arkindex update_budgets` administrator creates new budget entries for all processes that use workers with costs and have had any activity since the last time they had any budget entry created for them. System administrators need to arrange for this command to be run regularly to create new budget entries automatically.
+
+### Restricting process execution
+
+Instance administrators can enable the [`process_enforce_budgets` setting](../deployment/configuration.md#process_enforce_budgets) to require funds to be available to run processes that use [workers with costs](../workers/advanced.md#costs). When attempting to start a process, retry a process, or restart a task, an error will occur when:
+
+* there is no budget on the project associated with the process;
+* there is a budget, but you do not have the access rights to consume funds from this budget;
+* there is a budget, but it does not have any funds available.
+
+This option will apply for the whole instance, on any budget, project or worker.

content/deployment/checks.md

@@ -22,6 +22,14 @@ Arkindex requires Python 3.10 or later.
 
 Arkindex 1.4.1 or later requires Python 3.10 or later to run. Versions 1.1.4 to 1.4.0 required Python 3.8. Previous versions supported Python 3.7.
 
+### `arkindex.C002`
+
+```
+Arkindex requires PostgreSQL 15 or later.
+```
+
+Arkindex 1.8.0 or later requires the database server to be running PostgreSQL 15 or later to run. Previous versions supported PostgreSQL 14.
+
 ## Errors
 
 !!! info

content/deployment/configuration.md

@@ -329,14 +329,22 @@ Hostname of the SMTP server to use to send emails. The server must support TLS.
 
 Password for the SMTP server.
 
+Since Arkindex 1.8.0, when this parameter is omitted, Arkindex will not attempt authentication to the server.
+
 #### `email.port`
 
 Port of the SMTP server to use to send emails. Defaults to `25`.
 
+#### `email.tls`
+
+*Since Arkindex 1.8.0.* Whether or not to use an explicit TLS connection to the SMTP server. Boolean, defaults to `true`.
+
 #### `email.user`
 
 Username for the SMTP server. When `email.from_address` is not set, this is also used as the sender address.
 
+Since Arkindex 1.8.0, when this parameter is omitted, Arkindex will not attempt authentication to the server.
+
 ### `export`
 
 #### `export.ttl`
@@ -555,6 +563,20 @@ Path to an elliptic curve private key file to use as the server private key for
 
 Once a task has expired, it will be deleted when the `arkindex cleanup` command runs.
 
+### `process_enforce_budgets`
+
+*Enterprise Edition only.*
+
+*Since Arkindex 1.8.0.* Boolean, defaults to `false`. When this is enabled, starting or retrying a process, or restarting a task, may return HTTP 402 Payment Required when:
+
+1. The process or task uses a worker that has costs associated to it, thus running that worker would incur a cost in a budget.
+2. One of the following conditions are true:
+   * The corpus of the process does not have a budget;
+   * The creator of the process, or the user requesting to restart a task, does not have the necessary access rights on the budget;
+   * There are no funds left on the budget.
+
+This option does not affect the creation of budget entries by the `arkindex update_budgets` command.
+
 ### `public_hostname`
 
 *Since Arkindex 1.0.3.* Root URL of the Arkindex instance, including the scheme and hostname (`http://example.com`).

content/develop/tooling.md

@@ -131,9 +131,12 @@ SHELL_PLUS_POST_IMPORTS = [
 ### Common operations
 
 * `npm start`: Start a local development server on `localhost:8080`
-* `npm run test` or `npm t`: Run unit tests
-* `npm run build`: Build for development, no Docker image
-* `npm run production`: Build for production, no Docker image
+* `npm run test` or `npm t`: Run unit tests, then wait for changes to run tests again
+* `npm run test -- --run` or `npm t -- --run`: Run unit tests only once
+* `npm run build`: Build for production, no Docker image
+* `npm run preview`: Show the result of the production build in a local server
+* `npm run lint`: Run ESLint on all files
+* `npm run format`: Run Prettier on all files
 * `make build`: Build to a local Docker image using latest GitLab CI artifact
 * `make clean`: Clean up files from previous builds
 * `make all`: Clean up then build
@@ -144,50 +147,10 @@ SHELL_PLUS_POST_IMPORTS = [
 
 If you want to [contribute a patch to Arkindex's frontend](contributing.md), in order for your contribution to be accepted your code must pass a CI pipeline including not only unit tests but linting rules as well.
 
-These formatting checks are run using [pre-commit](https://pre-commit.com/). You need to install pre-commit by running the following command:
-
-```
-pip install pre-commit
-```
-
-You then need to initialize pre-commit by running the following command, in the frontend repository:
-
-```
-pre-commit install
-```
-
-Once this is done, all the pre-commit checks will run whenever you commit some code. To verify that your code passes the pre-commit checks, you can and should run them using the following command:
-
-```
-pre-commit run -a
-```
+These formatting checks are run automatically on each commit using [Husky](https://typicode.github.io/husky/). It should be installed and configured automatically when running `npm install`.
 
 You cannot commit code if the pre-commit checks fail.
 
-#### Linting rules
-
-See the `.eslintrc.yml` file for the exact ESLint settings. The linting rules are based on the following sets:
-
-* [StandardJS](https://standardjs.com/)
-* [ESLint Recommended](https://eslint.org/docs/rules/)
-* [Vue.js Essential + Strongly Recommended](https://eslint.vuejs.org/rules/)
-* [`eslint-plugin-import`](https://www.npmjs.com/package/eslint-plugin-import)
-
-Some rules have been customized:
-
-* Using `==` and `!=` in templates will show warnings: use `===` and `!==` instead ([vue/eqeqeq](https://eslint.vuejs.org/rules/eqeqeq.html));
-* Using `this` in a template causes errors: use `text` instead of `this.text` ([vue/this-in-template](https://eslint.vuejs.org/rules/this-in-template.html));
-* Single-line HTML elements are allowed to hold content (`<h1>hello</h1>` is allowed);
-* To bind JavaScript expressions to attributes, use `:property=""`, not `v-bind:property=""`, but to bind JavaScript expressions to events, use `v-on:event=""`, not `@event=""` ([vue/v-bind-style](https://eslint.vuejs.org/rules/v-bind-style.html), [vue/v-on-style](https://eslint.vuejs.org/rules/v-on-style.html))
-* Single-line HTML elements can have up to three attributes ; you have to switch to multiline elements with one attribute per line above that ([vue/max-attributes-per-line](https://eslint.vuejs.org/rules/max-attributes-per-line.html))
-* Void elements (HTML elements which do not hold content, like `br`) should be self-closing: `<br />` instead of `<br>` ([vue/html-self-closing](https://eslint.vuejs.org/rules/html-self-closing.html))
-* HTML elements that can hold content like `p` should never be self-closing: `<p></p>` instead of `<p />` ([vue/html-self-closing](https://eslint.vuejs.org/rules/html-self-closing.html))
-* StandardJS-like linting is now applied on all JS code inside `<template>` (linting was only done on the HTML, not the JS bits)
-* Component names must match their file names ([vue/match-component-file-name](https://eslint.vuejs.org/rules/match-component-file-name.html))
-* Component names must use PascalCase ([vue/component-definition-name-casing](https://eslint.vuejs.org/rules/component-definition-name-casing.html) and [vue/component-name-in-template-casing](https://eslint.vuejs.org/rules/component-name-in-template-casing.html))
-* Components cannot be named after existing HTML or SVG tags ([vue/no-reserved-component-names](https://eslint.vuejs.org/rules/no-reserved-component-names.html))
-* The `slot`, `scope` and `slot-scope` attributes are deprecated since Vue.js 2.6.0+ and linting rules forbid them ([vue/no-deprecated-scope-attribute](https://eslint.vuejs.org/rules/no-deprecated-scope-attribute.html), [vue/no-deprecated-slot-attribute](https://eslint.vuejs.org/rules/no-deprecated-slot-attribute.html), [vue/no-deprecated-slot-scope-attribute](https://eslint.vuejs.org/rules/no-deprecated-slot-scope-attribute.html))
-* Use the long form `v-slot:default="..."` instead of `#default="..."` to be more explicit ([vue/v-slot-style](https://eslint.vuejs.org/rules/v-slot-style.html))
-* Static `style` attributes that can be replaced by CSS classes are forbidden: use CSS classes instead ([vue/no-static-inline-styles](https://eslint.vuejs.org/rules/no-static-inline-styles.html))
-* `<template>`, `<script>` and `<style>` must be separated by an empty line ([vue/padding-line-between-blocks](https://eslint.vuejs.org/rules/padding-line-between-blocks.html))
-* Function calls without arguments must not use parentheses: `v-on:click="something"` instead of `v-on:click="something()"` ([vue/v-on-function-call](https://eslint.vuejs.org/rules/v-on-function-call.html))
\ No newline at end of file
+#### Linting
+
+We use ESLint to lint code and Prettier to format it. Running `npm run lint` and `npm run format` will run these on all files, and they will also run automatically on each commit and in CI.

content/processes/advanced.md

@@ -111,4 +111,14 @@ If you **Retry** the process (the "retry" action is available both from the proc
 Two additional actions are available from the Worker activities monitoring page:
 
 - The **Select all failed elements** button adds all the elements in an `error` state to [your selection](../howto/run-process/index.md#from-a-selection);
-- The **Create process** button create a new blank process from all the elements in an `error` state, which you can then [configure and run](../howto/run-process/index.md#configure-workers).
\ No newline at end of file
+- The **Create process** button create a new blank process from all the elements in an `error` state, which you can then [configure and run](../howto/run-process/index.md#configure-workers).
+
+### Execution costs
+
+!!! info
+
+    This feature is only available for Enterprise Edition customers.
+
+When a process includes [workers with costs](../workers/advanced.md#costs), their [budget entries](../budgets/index.md#budget-entries) and total execution cost can be viewed through the **Budget entries** action in the **Actions** menu of the process status view.
+
+If instance administrators [have enabled it](../budgets/index.md#restricting-process-execution), starting a process, retrying a process or restarting a task may cause an error when that involves running a worker with costs and there are no funds available to run it.

content/upgrade/.pages

@@ -10,3 +10,4 @@ nav:
 - 1.7.1.md
 - 1.7.2.md
 - 1.7.3.md
+- 1.8.0.md

content/upgrade/1.8.0.md

+---
+summary: Technical release notes for Arkindex 1.8.0
+title: Arkindex 1.8.0
+---
+A new Arkindex release is available.
+
+To upgrade a **development instance**, follow [this documentation](../develop/upgrade.md).
+
+To upgrade a **production instance**, you need to:
+
+- Deploy this release's Docker image: `registry.gitlab.teklia.com/arkindex/backend:1.8.0`
+- Run the database migrations: `docker exec ark-backend arkindex migrate`
+- Update the system workers: `docker exec ark-backend arkindex update_system_workers`
+
+The release notes for Arkindex 1.8.0 are available [here](https://teklia.com/our-solutions/arkindex/releases/1.8.0).
+
+The main changes impacting developers and system administrators are detailed below.
+
+## Support for PostgreSQL 14 dropped
+
+Arkindex now requires PostgreSQL 15 and later. Official support for PostgreSQL 14 is dropped.
+
+When attempting to run Arkindex on any PostgreSQL database with a version below 15, a new [critical system check issue](../deployment/checks.md#arkindexc002) will be reported.
+
+This has no impact for developers, as the development setup has been running PostgreSQL 17 since [Arkindex 1.7.2](./1.7.2.md).
+
+## New budget update command
+
+!!! info
+
+    This feature is only available in [Arkindex Enterprise Edition](../overview/license.md).
+
+A new `arkindex update_budgets` command has been added to the backend. This command creates new budget entries for all processes that ran using workers with costs. To properly keep track of the current budget usage by processes, system administrators should configure their instances to run this command regularly, similarly to `arkindex cleanup`.
+
+## New setting for process budget enforcement
+
+!!! info
+
+    This feature is only available in [Arkindex Enterprise Edition](../overview/license.md).
+
+A new `process_enforce_budgets` setting has been introduced to restrict running any process or task when workers with costs are involved. It will not be possible to run a worker with costs unless a budget has funds available, and the user has the necessary access rights to use the budget.
+
+This setting is disabled by default to preserve the current behavior, but may be enabled by default in a future release.
+
+## Updated email configuration
+
+Some options in the [backend configuration](../deployment/configuration.md) related to sending emails have been changed:
+
+* [`email.user`](../deployment/configuration.md#emailuser) is now optional and nullable. Omitting it, or setting it to `null`, will disable authentication to the SMTP server.
+* [`email.password`](../deployment/configuration.md#emailpassword) is now optional and nullable. Omitting it, or setting it to `null`, will disable authentication to the SMTP server.
+* A new [`email.tls`](../deployment/configuration.md#emailtls) allows disabling TLS when connecting to the SMTP server. It defaults to `true`, as Arkindex was only ever using TLS before this option was introduced.
+
+## New frontend development tooling
+
+Numerous upgrades have been done on the development and build tools on the Arkindex frontend:
+
+- We now use [Vite](https://vite.dev) instead of [Vue CLI](https://cli.vuejs.org/) as our build tool.
+- Unit tests now use [Vitest](https://vitest.dev) instead of [Mocha](https://mochajs.org/).
+- The [pre-commit](https://pre-commit.com/) hooks have been replaced with [Husky](https://typicode.github.io/husky/).
+- The code is now automatically formatted using [Prettier](https://prettier.io/).
+
+This removes many deprecated dependencies, and brings the project to the current recommendations for modern Vue.js development. Many issues with linting tools not running properly, or only running well when using [Volar](https://marketplace.visualstudio.com/items?itemName=Vue.volar), have been resolved.
+
+Developers should update their frontend repositories and run `npm install` to upgrade their development setup. No other changes are required.
+
+This has **no impact for system administrators**.
+
+## MinIO upgrade
+
+The development setup includes a [MinIO](https://min.io/) container to provide S3 storage, which was pinned to a release from 2021. This older version provided its simple filesystem-based storage using "filesystem mode", a feature that has since been dropped. We have upgraded MinIO to [a more recent release](https://github.com/minio/minio/releases/tag/RELEASE.2025-02-28T09-55-16Z). Developers will need to either start from scratch, losing all of their data on the local S3 buckets, or to migrate manually.
+
+This has **no impact for system administrators**.
+
+### To start from scratch
+
+1. Ensure all Arkindex containers are stopped: `docker compose -p arkindex down`
+2. Erase the previous MinIO data: `docker volume rm arkindex_miniodata`
+3. Restart the backend's [third-party services](../develop/setup.md#run-the-third-party-services). This will recreate the buckets that Arkindex needs.
+
+### To migrate existing data
+
+This is a simplified version of the [official migration guide](https://min.io/docs/minio/container/operations/install-deploy-manage/migrate-fs-gateway.html) adapted to the MinIO instance provided by the Arkindex dev setup.
+During the migration, you will have 2 instances on MinIO running in parallel on your computer. This is required to be able to copy all the data from the old instance towards the new one.
+1. Start the old MinIO server in your previous development setup with `make services`. You can use the `1.7.3` arkindex release for that (`git checkout 1.7.3`)
+2. Verify that your server is correctly running: the [MinIO console](https://minio-console.ark.localhost) should be available with your existing buckets.
+3. Run the following script to perform the migration:
+
+   ```sh
+   # Start a server for this migration with the new MinIO version
+   docker run \
+       --rm \
+       --name ark-minio-migration \
+       --detach \
+       --publish 9002:9002 \
+       --env MINIO_ROOT_USER=minio1234 \
+       --env MINIO_ROOT_PASSWORD=minio1234 \
+       --volume arkindex_miniodata:/olddata \
+       --volume arkindex_minio_migration:/newdata \
+       --network arkindex_default \
+       minio/minio:RELEASE.2025-02-28T09-55-16Z \
+       server /newdata --console-address :9002
+
+   # Configure the MinIO client to perform the migration
+   docker exec -it ark-minio-migration mc alias set old http://ark-minio:9000/ minio1234 minio1234
+   docker exec -it ark-minio-migration mc alias set new http://localhost:9000/ minio1234 minio1234
+
+   # Run the migration on each bucket
+   for bucket in export iiif-cache ingest ponos-artifacts ponos-logs staging thumbnails training uploads; do
+       docker exec -it ark-minio-migration mc mb "new/$bucket"
+       docker exec -it ark-minio-migration mc mirror --preserve "old/$bucket" "new/$bucket"
+   done
+   ```
+
+4. Open the [MinIO console](http://localhost:9002) for the temporary migration server and login with `minio1234` as the username and password.
+5. Check that the same buckets are listed in both consoles, with the correct amount of objects.
+6. Run the following to complete the migration:
+
+   ```sh
+   # Stop the old server
+   docker compose -p arkindex down
+
+   # Overwrite the old server's data with the migrated data
+   docker exec -it ark-minio-migration sh -c 'rm -rf /olddata/.minio.sys /olddata/* && cp -rf /newdata/.minio.sys /newdata/* /olddata/'
+
+   # Stop the migration server
+   docker stop ark-minio-migration
+   ```
+
+7. Update your backend repository to get the newer server in your Docker Compose configuration. You should be on the `release-1.8.0` branch or the `1.8.0` tag once available.
+8. Start the new services with `make services`.
+9. Check that the [MinIO console](https://minio-console.ark.localhost) is now the console of the newer version and still has your buckets.
+10. Delete the migration data: `docker volume rm arkindex_minio_migration`

content/users/rights/index.md

@@ -195,3 +195,14 @@ If you have admin rights on the model, you can delete its versions and manage it
 | Stop a running dataset process        | ❌        | ❌     | ❌           | ✅     |
 | Retry a failed dataset process        | ❌        | ❌     | ❌           | ✅     |
 
+### Budget access
+
+Users can view the [budgets](../../budgets/index.md) that they have access to from the **Budgets** action in the top right user menu. After selecting a budget in the list, the memberships linked to that budget can be managed under the **Members** section.
+
+#### Budget permission table
+
+| action                      | no right | guest | contributor | admin |
+| --------------------------- | -------- | ----- | ----------- | ----- |
+| View a budget               | ❌        | ✅     | ✅           | ✅     |
+| Consume funds from a budget | ❌        | ❌     | ✅           | ✅     |
+| Manage members              | ❌        | ❌     | ❌           | ✅     |

content/workers/advanced.md

@@ -139,6 +139,27 @@ Deleting workers that will not be used anymore is not always possible, because A
 
 Archived workers **can no longer be used in processes**, and are displayed in their own tab in the workers list. It is possible to archive a worker from the Arkindex frontend, but an archived worker can only be made available again by an instance administrator from the admin interface.
 
+## Costs
+
+Multiple execution costs can be defined on workers:
+
+**Hourly CPU cost**
+:   Cost of running any of this worker's versions without a GPU for an hour.
+
+**Hourly GPU cost**
+:   Cost of running any of this worker's versions with a GPU for an hour.
+
+**Cost per 1000 elements**
+:   Cost of running any of this worker's versions on 1000 elements in an inference process.
+
+These costs allow to report the real hardware costs of executing workers to end users.
+
+All costs are expressed in euros, have a precision of up to three decimal places, and can range from `0.000` to `99999.999`. They are set to `0.000` by default, meaning there are no costs for running a worker. They can only be configured through the [administration interface](../deployment/administration.md#administration-interface).
+
+When any of these costs is set to any value above zero, all costs will be displayed when viewing the details of a worker, and a `€` icon will be shown next to the worker name.
+
+In [Arkindex Enterprise Edition](../overview/license.md) only, new entries will be created in a [budget](../budgets/index.md) when running a process that contains any task using a worker with costs.
+
 ## Secrets
 
 Some workers may require the use of **Secrets**, like for example an authentication token to interact with an external API. See the dedicated [secrets documentation](../secrets/index.md).