Skip to content
Snippets Groups Projects

Remove new_tokens from OCR code

Merged Thibault Lavigne requested to merge remove-new-tokens-from-ocr-code into main
1 file
+ 0
5
Compare changes
  • Side-by-side
  • Inline
+ 0
5
@@ -42,11 +42,6 @@ class OCRDatasetManager(DatasetManager):
):
self.synthetic_data = self.params["config"]["synthetic_data"]
if "new_tokens" in params:
self.charset = sorted(
list(set(self.charset).union(set(params["new_tokens"])))
)
self.tokens = {
"pad": params["config"]["padding_token"],
}
Loading