tiiuae/falcon-40b
tiiuae/falcon-40b
1
0
Overview Code Discussions Pull Requests Wiki Activity

Upload tokenizer

Browse Source
This commit is contained in:
Daniel Hesslow 2023-05-24 13:49:21 +00:00 committed by huggingface-web
parent e7950c40d6
commit 6064d605bb
3 changed files with 129993 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
special_tokens_map.json Normal file
View File

@ -0,0 +1,16 @@
{
"additional_special_tokens": [
">>TITLE<<",
">>ABSTRACT<<",
">>INTRODUCTION<<",
">>SUMMARY<<",
">>COMMENT<<",
">>ANSWER<<",
">>QUESTION<<",
">>DOMAIN<<",
">>PREFIX<<",
">>SUFFIX<<",
">>MIDDLE<<"
],
"eos_token": "<|endoftext|>"
}

129970
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

7
tokenizer_config.json Normal file
View File

@ -0,0 +1,7 @@
{
"add_prefix_space": false,
"eos_token": "<|endoftext|>",
"model_max_length": 2048,
"special_tokens_map_file": null,
"tokenizer_class": "PreTrainedTokenizerFast"
}