2 Commits

Author	SHA1	Message	Date
Lysandre	86b5e09344	Updates the tokenizer configuration file (#62) ... - Adds tokenizer_config.json file (87565a3098077d493d402e1b109e67baf52c5281)	2024-02-19 11:06:12 +00:00
Thomas Wolf	1a7dd4986e	Add tokenizer configurations	2020-11-24 18:37:56 +01:00