Skip to content
Snippets Groups Projects
Commit ff2f76f9 authored by Solene Tarride's avatar Solene Tarride
Browse files

Use the same space token as sentencepiece

parent 89b49a0c
No related branches found
No related tags found
1 merge request!287Support subword and word language models
......@@ -22,7 +22,7 @@ class Token(NamedTuple):
class LMTokenMapping(NamedTuple):
space: Token = Token("", " ")
space: Token = Token("", " ")
linebreak: Token = Token("", "\n")
ctc: Token = Token("", "<ctc>")
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment