Implement extraction command

Requires !9 (merged)

Closes #9 (closed)

assigned to @schneider-y

added 13 commits

• 825bb4ab - 1 commit from branch main
• 22353a0a - refactoring + packaging
• 6bb96aac - remove obsolete code
• e42d071d - remove not needed files, reorganize inside dan module
• 99285091 - remove Datasets folder
• 0afccab8 - implement extraction
• f48001aa - fix lint
• a6a94a5c - update defaults in doc
• 4d0b43a0 - use the right starting tokens
• 4af68829 - fix tokens retrieval
• f37ae1f0 - final touches
• 46f4c299 - add tests, new params
• dbc99d48 - fix tests, CI, regroupe parser related code in __init__ files

Compare with previous version

marked this merge request as ready

requested review from @starride

added P2 label

changed milestone to %ML Prod - November 2022 n°2

Works great ! I tested these features:

• basic extraction
• extraction with tokens
• extraction with only beginning tokens
• using an existing split
• custom splitting ratio
• custom transcription/entity worker version

You can find the command I used in this note.

approved this merge request

merged