Load a language model and decode with it during evaluation
This is needed for #213
Depends #251 (closed)
We will support additional keys in the evaluation configuration, to load a language model and set some hyperparameters.
New keys
{
...
"model": {
...
"lm": {
"path": "", # path to the model i.e. language_model.arpa.gz or language_model.bin
"weight": 1.0, # language_model_weight
"temperature": 1.0 # not exposed in LM inference yet
}
}
...
}
All the language model files should be in the same folder, model.lm.path
's parent is the path to that folder. The expected names should be written in the documentation and a verbose message must be displayed if there is a misnaming.
The LM decoder (named lm_decoder
) must be loaded after the encoder/decoder in GenericTrainingManager.load_model. This decoder is not initialized, is not unpickled so the loading must happen after all that. Isolate the loading in a dedicated method, it will be quite long since you need to check for the file's existence. It should be loaded in the same way as during inference. Expected filenames are:
-
lexicon.txt
forCTCLanguageDecoder.lexicon_path
-
tokens.txt
forCTCLanguageDecoder.tokens_path
Once it's loaded, we need to use it decode the prediction.
- Store the predicted Tensor while predicting on the batch
-
Decode using the lm_decoder and store the output under the
str_lm
key - Store the value in the Inference class