Add predicted objects to predict command

Closes #36 (closed)

changed milestone to %ML Prod - March 2023 n°1

added P2 label

requested review from @schneider-y

By Tristan Faine on 2023-03-09T10:55:00 (imported from GitLab)

assigned to @tfaine

By Tristan Faine on 2023-03-09T10:55:00 (imported from GitLab)

added 4 commits

• 472211ca - Added function to aggregate attention maps
• 23f5346b - fix linting issue
• 74cb91db - First attempt, obtained polygons seem wrong
• 556c40c0 - fix merge issues?

Compare with previous version

By Tristan Faine on 2023-03-09T11:06:09 (imported from GitLab)

new CLI argument --predict-objects is set to False by default, adding it to teklia-dan predict sets it to true.

As an example, command used is the following:

teklia-dan predict \
    --image ~/Stage/gitlab_projects/NP_models_and_stuff/popp_single_page_htr_ner/example.jpg \
    --model ~/Stage/gitlab_projects/NP_models_and_stuff/popp_single_page_htr_ner/model.pt \
    --parameters ~/Stage/gitlab_projects/NP_models_and_stuff/popp_single_page_htr_ner/parameters.yml \
    --charset ~/Stage/gitlab_projects/NP_models_and_stuff/popp_single_page_htr_ner/charset.pkl \
    --output ~/Stage/gitlab_projects/NP_models_and_stuff/popp_single_page_htr_ner/predict_issue36/ \
    --scale 0.5 \
    --confidence-score \
    --attention-map \
    --attention-map-level line \
    --attention-map-scale 0.5 \
    --predict-objects

Taking into account this note https://gitlab.com/teklia/atr/dan/-/issues/36#note_1305985483, i tried different ways to only obtain one contour per object, but couldn't figure out how tune cv2.drawContours() for that. The proposed solution is simply to merge all the contours into a single polygon but these don't seem to be very accurate? Here they are, when selecting all contours or only the first contour:

Here's what the attention map looks at line-level for the first prediction:

Finally, here's what the json output would look like if we integrated this change:

example_all_contours.json

example_1contour.json

By Tristan Faine on 2023-03-09T12:24:24 (imported from GitLab)

changed milestone to %ML Prod - Next

assigned to @starride and unassigned @tfaine

added 1 commit

• 139c1107 - Predict polygons but preserve old behavior

Compare with previous version

added 1 commit

• 4a4ca3e3 - reformat attention.py

Compare with previous version

added 1 commit

• 5a8d7e05 - Simplify polygon

Compare with previous version

added 29 commits

• 5a8d7e05...614fa206 - 23 commits from branch main
• f5e6b784 - Added function to aggregate attention maps
• 66444aba - fix linting issue
• da7e8b3f - First attempt, obtained polygons seem wrong
• 04b15f97 - Predict polygons but preserve old behavior
• 19f6aefa - reformat attention.py
• de6e5b91 - Simplify polygon

Compare with previous version

added 1 commit

• 156ece6c - Convert filename to pathlib.Path

Compare with previous version

added 1 commit

• e5776db4 - Improve thresholding

Compare with previous version

added 1 commit

• 96de1ffb - Fix issue when no contour is found

Compare with previous version

added 1 commit

• 3e53c0d7 - Document usage

Compare with previous version

added 1 commit

• d8106f34 - Fix lint

Compare with previous version

Here is a first version.

At each step:

• binarize the current attention map (blur + OTSU binarization + dilation (OpenCV))
• get the contours using OpenCV
• select the best contour based on attention map intensity and contour area
• extract the polygon coordinates and simplify it (bounding box)

More details in the Redmine issue.

Usage:

teklia-dan predict --image humu_model/page.jpg --model humu_model/model.pt --parameters humu_model/parameters.yml --charset humu_model/charset.pkl --attention-map --attention-map-level line --predict-objects --output humu_model/predict_line/

will create:

• A JSON file including for each polygon its coordinates, confidence, text and text confidence

{
    "text": "Oslo\n39 \nOresden den 24te Rasser!\nH\u00f8jst\u00e6redesherr Hartvig - assert!\nUllereder fra den f\u00f8rste tide da\njeg havder den tilfredsstillelser at vide den ar-\ndistiske ledelser af Kristiania theater i Deres\nhronder, har jeg g\u00e5t hernede med et stille\nh\u00e5b om fra Dem at modtage et forelag, sig -\nsende tils at lade \"K\u00e6rlighedens \u00abKomedie\u00bb\nopf\u00f8re fore det norske purblikum.\nEt s\u00e5dant forslag er imidlertid, imod\nforventning; ikke fremkommet, og jeg n\u00f8des der-\nfor tils self at grivbe initiativet, hvilket hervede\nsker, idet jeg\nbeder\nbet\nragte stigkket some ved denne\nskrivelse officielde indleveret til theatret. No-\nget exemplar af bogen vedlagger jeg ikke da\ndenne (i 2den udgave) med Lethed kan er -\nholdet deroppe.\nDe bet\u00e6nkeligheder, jeg i sin tid n\u00e6-\nrede mod stykkets opf\u00f8relse, er for l\u00e6nge si -\ndem forsvundne. Af mange begn er jeg kom-\nmen til den overbevisning at almenlreden\naru har f\u00e5tt sine \u00f8gne opladte for den sand -\nMed at dette arbejde i sin indersten id\u00e9 hviler\np\u00e5 et ubedinget meralsk grundlag, og brad\nstykkets hele kunstneriske struktuve ang\u00e5r,",
    "objects": [
        {
            "confidence": 0.68,
            "polygon": [
                [
                    264,
                    118
                ],
                [
                    410,
                    118
                ],
                [
                    410,
                    185
                ],
                [
                    264,
                    185
                ]
            ],
            "text": "Oslo",
            "text_confidence": 0.8
        },
        ...
    "attention_gif": "dan_humu_page/predict/example_line.gif"
}

• A GIF image with bounding boxes

changed title from Draft: Resolve "Add predicted objects to predict command" to Draft: Add predicted objects to predict command

changed milestone to %ML Prod - April 2023 n°1