Skip to content
Snippets Groups Projects
Select Git revision
  • ada9e966ffed16bab0973017f01827b06c70504d
  • main default protected
  • bump-boto3
  • bump-wandb
  • bump-nltk
  • bump-pandas
  • bump-PyYAML
  • bump-lxml
  • bump-mlflow-skinny
  • bump-scipy
  • bump-numpy
  • bump-matplotlib
  • bump-torchvision
  • bump-torchaudio
  • bump-torch
  • bump-tensorboard
  • bump-mdutils
  • bump-mkdocs-material
  • bump-mkdocstrings-python
  • bump-albumentations
  • support-nested
  • 0.2.2rc4
  • 0.2.2rc3
  • 0.2.2rc2
  • 0.2.2rc1
  • attention-exp
  • 0.2.2b2
  • 0.2.2b1
  • 0.2.2a2
  • 0.2.2a1
  • 0.2.1
  • 0.2.0
  • 0.2.0rc12
  • 0.2.0rc11
  • 0.2.0rc10
  • 0.2.0rc9
  • 0.2.0rc8
  • 0.2.0rc7
  • 0.2.0rc6
  • 0.2.0rc5
  • 0.2.0rc4
41 results

development.md

Blame
  • Yoann Schneider's avatar
    Yoann Schneider authored and Bastien Abadie committed
    ada9e966
    History

    Development

    DAN uses different tools during its development.

    Linter

    Code syntax is analyzed before submitting the code.

    To run the linter tools suite you may use pre-commit.

    pip install pre-commit
    pre-commit run -a

    Tests

    Unit tests

    Tests are executed with tox using pytest.

    pip install tox
    tox

    To recreate tox virtual environment (e.g. a dependencies update), you may run tox -r.

    Run a single test module: tox -- <test_path> Run a single test: tox -- <test_path>::<test_function>

    The tests use a large file stored via Git-LFS. Make sure to run git-lfs pull before running them.

    Commands

    As unit tests do not test everything, it is sometimes necessary to use DAN commands directly to test developments.

    Training command

    The library already has all the documents needed to run the training command on a minimalist dataset. You can use the configuration available at configs/tests.json. It is already populated with the parameters used in the unit tests.

    teklia-dan train --config configs/tests.json

    Predict command

    The library already has all the documents needed to run the predict command with a minimalist model. In the tests/data/prediction directory, you can run the following command and add any extra parameters you need:

    teklia-dan predict \
        --image-dir images/ \
        --image-extension png \
        --model . \
        --output /tmp/dan-predict

    Evaluation command

    The library already has all the documents needed to run the evaluation command on a minimalist dataset. You can use the configuration available at configs/eval.json. It is already populated with the parameters used in the unit tests.

    teklia-dan evaluate --config configs/eval.json

    Convert command

    If you want to evaluate a NER models with you own scripts, you can convert DAN's predictions in BIO format, using the convert command.

    teklia-dan convert /tmp/dan-predict --tokens tokens.yml --output /tmp/dan-convert

    Documentation

    This documentation uses Sphinx and was generated using MkDocs and mkdocstrings.

    Setup

    Install the needed dependencies through:

    # In a clone of the Git repository
    pip install -r doc-requirements.txt

    Build the documentation using mkdocs serve -v. You can then write in Markdown in the relevant docs/*.md files, and see live output on http://localhost:8000.

    Linter

    This documentation is subject to linting using: