Apply 5f7a5c2f

README.md

@@ -13,12 +13,14 @@ pip install -e .
 For more details about this package, make sure to see the documentation available at https://teklia.gitlab.io/atr/dan/.
 
 ## Development
+
 For development and tests purpose it may be useful to install the project as a editable package with pip.
 
 * Use a virtualenv (e.g. with virtualenvwrapper `mkvirtualenv -a . dan`)
 * Install `dan` as a package (e.g. `pip install -e .`)
 
 ### Linter
+
 Code syntax is analyzed before submitting the code.\
 To run the linter tools suite you may use pre-commit.
 ```shell
@@ -27,6 +29,7 @@ pre-commit run -a
 ```
 
 ### Run tests
+
 Tests are executed with `tox` using [pytest](https://pytest.org).
 To install `tox`,
 ```shell
@@ -41,6 +44,16 @@ Run a single test: `tox -- <test_path>::<test_function>`
 
 The tests use a large file stored via [Git-LFS](https://docs.gitlab.com/ee/topics/git/lfs/). Make sure to run `git-lfs pull` before running them.
 
+### Update documentation
+
+Please keep the documentation updated when modifying or adding features.
+It's pretty easy to do:
+```shell
+pip install -r doc-requirements.txt
+mkdocs serve
+```
+
+You can then write in Markdown in the relevant `docs/*.md` files, and see live output on http://localhost:8000.
 
 ## Inference
 
@@ -71,6 +84,10 @@ text, confidence_scores = model.predict(image, confidences=True)
 
 This package provides three subcommands. To get more information about any subcommand, use the `--help` option.
 
+### Get started
+
+See the [dedicated section](https://teklia.gitlab.io/atr/dan/get_started/training/) on the official DAN documentation.
+
 ### Data extraction from Arkindex
 
 See the [dedicated section](https://teklia.gitlab.io/atr/dan/usage/datasets/extract/) on the official DAN documentation.

dan/ocr/document/train.py

@@ -163,7 +163,7 @@ def get_config():
         },
         "training_params": {
             "output_folder": "outputs/dan_esposalles_record",  # folder name for checkpoint and results
-            "max_nb_epochs": 710,  # maximum number of epochs before to stop
+            "max_nb_epochs": 800,  # maximum number of epochs before to stop
             "max_training_time": 3600
             * 24
             * 1.9,  # maximum time before to stop (in seconds)

docs/get_started/development.md

+# Development
+
+DAN uses different tools during its development.
+
+## Linter
+
+Code syntax is analyzed before submitting the code.
+
+To run the linter tools suite you may use [pre-commit](https://pre-commit.com).
+
+```shell
+pip install pre-commit
+pre-commit run -a
+```
+
+## Run tests
+
+Tests are executed with [tox](https://tox.wiki) using [pytest](https://pytest.org).
+
+```shell
+pip install tox
+tox
+```
+
+To recreate tox virtual environment (e.g. a dependencies update), you may run `tox -r`.
+
+## Update documentation
+
+Documentation is built with [MkDocs](https://www.mkdocs.org/).
+
+```shell
+pip install -r doc-requirements.txt
+mkdocs serve
+```
+
+You can then write in Markdown in the relevant `docs/*.md` files, and see live output on http://localhost:8000.

docs/get_started/index.md

+# Get started
+
+To use DAN in your own environment, install it using pip:
+
+```shell
+pip install -e .
+```
+
+To learn more about the newly installed `teklia-dan` command, make sure to run:
+```shell
+teklia-dan --help
+```
+
+Get started with:
+
+* [Developments](development.md)
+* [Training workflow](training.md)

docs/get_started/training.md

+# Training workflow
+
+There are a several steps to follow when training a DAN model.
+
+## 1. Extract data
+
+The data must be extracted and formatted for training. To extract the data, DAN uses an Arkindex export database in SQLite format. You will need to:
+
+1. Structure the data into folders (`train` / `val` / `test`) in [Arkindex](https://arkindex.teklia.com/).
+2. [Export the project](https://doc.arkindex.org/howto/export/) in SQLite format.
+3. Extract the data with the [extract command](../usage/datasets/extract.md).
+4. Format the data with the [format command](../usage/datasets/format.md).
+
+At the end, you should have a tree structure like this:
+```
+output/
+├── charset.pkl
+├── labels.json
+├── split.json
+├── images
+│   ├── train
+│   ├── val
+│   └── test
+└── labels
+    ├── train
+    ├── val
+    └── test
+```
+
+## 2. Train
+
+The training command does not take any input parameters for now. To train a DAN model, you will therefore need to:
+
+1. Update the parameters from those listed in the [dedicated page](../usage/train/parameters.md). You will always need to update at least these variables:
+
+  - `dataset_name`, `dataset_level`, `dataset_variant` and `dataset_path`,
+  - `model_params.transfer_learning.*[checkpoint_path]` to finetune an existing model,
+  - `training_params.output_folder`.
+
+2. Train a DAN model with the [train command](../usage/train/index.md).
+
+## 3. Predict
+
+Once the training is complete, you can apply a trained DAN model on an image.
+
+To do this, you will need to:
+
+1. Create a `parameters.yml` file using the parameters saved during training in the `params` file, located in `{training_params.output_folder}/results`. This file should have the following format:
+```yml
+version: 0.0.1
+parameters:
+  mean: [float, float, float]
+  std: [float, float, float]
+  max_char_prediction: int
+  encoder:
+    input_channels: int
+    dropout: float
+  decoder:
+    enc_dim: int
+    l_max: int
+    dec_pred_dropout: float
+    attention_win: int
+    use_1d_pe: bool
+    use_lstm: bool
+    vocab_size: int
+    h_max: int
+    w_max: int
+    dec_num_layers: int
+    dec_dim_feedforward: int
+    dec_num_heads: int
+    dec_att_dropout: float
+    dec_res_dropout: float
+```
+2. Apply a trained DAN model on an image using the [predict command](../usage/predict.md).

docs/index.md

@@ -9,36 +9,4 @@ The model uses a character-level attention to handle slanted lines:
 
 Click [here](original_paper.md) to learn more about the model and how it fares against SOTA models.
 
-## Getting started
-
-To use DAN in your own environment, install it using pip:
-
-```shell
-pip install -e .
-```
-
-To learn more about the newly installed `teklia-dan` command, make sure to run:
-```shell
-teklia-dan --help
-```
-
-## Linter
-
-Code syntax is analyzed before submitting the code.\
-To run the linter tools suite you may use pre-commit.
-
-```shell
-pip install pre-commit
-pre-commit run -a
-```
-
-## Run tests
-
-Tests are executed with tox using [pytest](https://pytest.org).
-
-```shell
-pip install tox
-tox
-```
-
-To recreate tox virtual environment (e.g. a dependencies update), you may run `tox -r`
+[Get started with DAN](get_started/index.md) now!

docs/usage/predict.md

 # Predict
 
-Use the `teklia-dan predict` command to predict a trained DAN model on an image.
+Use the `teklia-dan predict` command to apply a trained DAN model on an image.
 
 ## Description of parameters
 
-| Parameter                   | Description                                                                                  | Type    | Default       |
-| --------------------------- | -------------------------------------------------------------------------------------------- | ------- | ------------- |
-| `--image`                   | Path to the image to predict. Must not be provided with `--image-dir`.                                                                | `Path`  |               |
-| `--image-dir`                   | Path to the folder where the images to predict are stored. Must not be provided with `--image`.                                                                | `Path`  |               |
-| `--image-extension`                   | The extension of the images in the folder. Ignored if `--image-dir` is not provided.                                                                 | `str`  |      .jpg         |
-| `--model`                   | Path to the model to use for prediction                                                      | `Path`  |               |
-| `--parameters`              | Path to the YAML parameters file.                                                            | `Path`  |               |
-| `--charset`                 | Path to the charset file.                                                                    | `Path`  |               |
-| `--output`                  | Path to the output folder. Results will be saved in this directory.                          | `Path`  |               |
-| `--scale`                   | Image scaling factor before feeding it to DAN.                                               | `float` | `1.0`         |
-| `--confidence-score`        | Whether to return confidence scores.                                                         | `bool`  | `False`       |
-| `--confidence-score-levels` | Level to return confidence scores. Should be any combination of `["line", "word", "char"]`.  | `str`   |               |
-| `--attention-map`           | Whether to plot attention maps.                                                              | `bool`  | `False`       |
-| `--attention-map-scale`     | Image scaling factor before creating the GIF.                                                | `float` | `0.5`         |
-| `--attention-map-level`     | Level to plot the attention maps. Should be in `["line", "word", "char"]`.                   | `str`   | `"line"`      |
-| `--predict-objects`         | Whether to return polygons coordinates.                                                      | `bool`  | `False`       |
-| `--word-separators`         | List of word separators.                                                                     | `list`  | `[" ", "\n"]` |
-| `--line-separators`         | List of line separators.                                                                     | `list`  | `["\n"]`      |
-| `--threshold-method`        | Method to use for attention mask thresholding. Should be in `["otsu", "simple"]`.            | `str`   | `"otsu"`      |
-| `--threshold-value `        | Threshold to use for the "simple" thresholding method.                                       | `int`   | `0`      |
+| Parameter                   | Description                                                                                     | Type    | Default       |
+| --------------------------- | ----------------------------------------------------------------------------------------------- | ------- | ------------- |
+| `--image`                   | Path to the image to predict. Must not be provided with `--image-dir`.                          | `Path`  |               |
+| `--image-dir`               | Path to the folder where the images to predict are stored. Must not be provided with `--image`. | `Path`  |               |
+| `--image-extension`         | The extension of the images in the folder. Ignored if `--image-dir` is not provided.            | `str`   | .jpg          |
+| `--model`                   | Path to the model to use for prediction                                                         | `Path`  |               |
+| `--parameters`              | Path to the YAML parameters file.                                                               | `Path`  |               |
+| `--charset`                 | Path to the charset file.                                                                       | `Path`  |               |
+| `--output`                  | Path to the output folder. Results will be saved in this directory.                             | `Path`  |               |
+| `--scale`                   | Image scaling factor before feeding it to DAN.                                                  | `float` | `1.0`         |
+| `--confidence-score`        | Whether to return confidence scores.                                                            | `bool`  | `False`       |
+| `--confidence-score-levels` | Level to return confidence scores. Should be any combination of `["line", "word", "char"]`.     | `str`   |               |
+| `--attention-map`           | Whether to plot attention maps.                                                                 | `bool`  | `False`       |
+| `--attention-map-scale`     | Image scaling factor before creating the GIF.                                                   | `float` | `0.5`         |
+| `--attention-map-level`     | Level to plot the attention maps. Should be in `["line", "word", "char"]`.                      | `str`   | `"line"`      |
+| `--predict-objects`         | Whether to return polygons coordinates.                                                         | `bool`  | `False`       |
+| `--word-separators`         | List of word separators.                                                                        | `list`  | `[" ", "\n"]` |
+| `--line-separators`         | List of line separators.                                                                        | `list`  | `["\n"]`      |
+| `--threshold-method`        | Method to use for attention mask thresholding. Should be in `["otsu", "simple"]`.               | `str`   | `"otsu"`      |
+| `--threshold-value `        | Threshold to use for the "simple" thresholding method.                                          | `int`   | `0`           |
 
 ## Examples
 

docs/usage/train/index.md

@@ -18,7 +18,7 @@ To train DAN on documents:
 
 To train DAN on lines, run `teklia-dan train document` with a line dataset.
 
-## Additional page
+## Additional pages
 
 * [Jean Zay tutorial](jeanzay.md)
 * [Data augmentation](augmentation.md)

mkdocs.yml

@@ -53,6 +53,10 @@ plugins:
 nav:
   - Home: index.md
   - Original implementation: original_paper.md
+  - Get started:
+    - get_started/index.md
+    - Development: get_started/development.md
+    - Training: get_started/training.md
   - Usage:
     - usage/index.md
     - Datasets: