Added function to aggregate attention maps

Closes #35 (closed)

changed milestone to %ML Prod - March 2023 n°1

added P2 label

added 1 commit

• 23f5346b - fix linting issue

Compare with previous version

By Tristan Faine on 2023-03-08T08:10:07 (imported from GitLab)

Unfortunately I haven't found a way to infer height, width of the image from the shape of attentions, a workaround is to resize the coverage vector after creating it in order to still work with the blending code after.

By Tristan Faine on 2023-03-08T08:13:14 (imported from GitLab)

assigned to @tfaine

requested review from @schneider-y

Yes, attentions is shaped like the feature vector: (batch_size, encoder_dim, height//height_divisor, width//width_divisor). You have access to height_divisor and width_divisor from the parameters.yml file, so you could retrieve height and width from there.

But what you did is the most straightforward solution.

Does that refer to the parameters.yml of a worker model from dan? I can't find that information in there.

By Tristan Faine on 2023-03-08T08:41:45 (imported from GitLab)

You are right, I thought that the information was in the parameters.yml file, but it is not.

So you have the correct approach.

As a sidenote, how would one recover the size of the image after performing the // operation?

As an example, one of my images has a width of 253, and 253//8 = 31. However the size of the feature width returned is 32.

But since I'm not sure that height_divisor and width_divisior will always have the values 32 and 8, so should I try hard-coding that or keep the current solution?

By Tristan Faine on 2023-03-08T09:20:09 (imported from GitLab)

height_divisor and width_divisor can vary (depending on the training config) so hard-coding is not the right approach.

Just keep your current solution, and resize the feature vector from the image's size.

Please mark the MR as ready if it is, and add a demo exec of the teklia-dan predict command with attention plots (gifs)

added 1 commit

• c928cbbd - approved suggestion

Compare with previous version

By Tristan Faine on 2023-03-08T10:23:59 (imported from GitLab)

resolved all threads

By Tristan Faine on 2023-03-08T10:24:10 (imported from GitLab)

added 1 commit

• a949bca4 - approved suggestion

Compare with previous version

By Tristan Faine on 2023-03-08T10:24:10 (imported from GitLab)

Just in case there are any issues I'd be unaware of, here is the command used to test the library locally:

teklia-dan predict \
    --image ~/Stage/gitlab_projects/NP_models_and_stuff/popp_single_page_htr_ner/example.jpg \
    --model ~/Stage/gitlab_projects/NP_models_and_stuff/popp_single_page_htr_ner/model.pt \
    --parameters ~/Stage/gitlab_projects/NP_models_and_stuff/popp_single_page_htr_ner/parameters.yml \
    --charset ~/Stage/gitlab_projects/NP_models_and_stuff/popp_single_page_htr_ner/charset.pkl \
    --output ~/Stage/gitlab_projects/NP_models_and_stuff/popp_single_page_htr_ner/predict_issue35/ \
    --scale 0.5 \
    --confidence-score \
    --attention-map \
    --attention-map-level line \
    --attention-map-scale 0.5

The example image and model files used come from the dan worker itself, for popp_single_page_htr_ner

The proposed commits do not change the output as can be seen on these json and gifs, first one shown is the original code, then the modified code.

before_example.json

after_example.json

By Tristan Faine on 2023-03-08T10:30:12 (imported from GitLab)

marked this merge request as ready

merged