Skip to content
Snippets Groups Projects
Commit b84e25c3 authored by Manon Blanco's avatar Manon Blanco
Browse files

Update logs

parent 4945936c
No related branches found
No related tags found
1 merge request!313Catch runtimeError when formatting LM files
This commit is part of merge request !313. Comments created here will be created in the context of that merge request.
......@@ -193,10 +193,12 @@ class Tokenizer:
vocab_size=self.subword_vocab_size,
model_prefix=self.prefix,
user_defined_symbols=self.special_tokens,
minloglevel=1,
)
except Exception as e:
logger.warning(
f"Failed to train a sentencepiece model for subword tokenization: {e}"
f"Failed to train a sentencepiece model for subword tokenization: {e} "
"Try again by editing the `--subword-vocab-size` parameter."
)
self.sentencepiece_model = None
return
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment