From 25a6b01ef78044a32c6176b6271cf7d9ce3eae1a Mon Sep 17 00:00:00 2001 From: Gustavo de Rosa Date: Wed, 13 Dec 2023 23:02:10 +0000 Subject: [PATCH] Update README.md --- README.md | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index c38af73..773f10b 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,6 @@ Phi-2 is a Transformer with **2.7 billion** parameters. It was trained using the Our model hasn't been fine-tuned through reinforcement learning from human feedback. The intention behind crafting this open-source model is to provide the research community with a non-restricted small model to explore vital safety challenges, such as reducing toxicity, understanding societal biases, enhancing controllability, and more. - ## Intended Uses Phi-2 is intended for research purposes only. Given the nature of the training data, the Phi-2 model is best suited for prompts using the QA format, the chat format, and the code format. @@ -69,13 +68,34 @@ def print_prime(n): ``` where the model generates the text after the comments. -**Notes** +**Notes:** * Phi-2 is intended for research purposes. The model-generated text/code should be treated as a starting point rather than a definitive solution for potential use cases. Users should be cautious when employing these models in their applications. * Direct adoption for production tasks is out of the scope of this research project. As a result, the Phi-2 model has not been tested to ensure that it performs adequately for any production-level application. Please refer to the limitation sections of this document for more details. * If you are using `transformers>=4.36.0`, always load the model with `trust_remote_code=True` to prevent side-effects. ## Sample Code +There are four types of execution mode: + +1. FP16 / Flash-Attention / CUDA: + ```python + model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", flash_attn=True, flash_rotary=True, fused_dense=True, device_map="cuda", trust_remote_code=True) + ``` +2. FP16 / CUDA: + ```python + model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", device_map="cuda", trust_remote_code=True) + ``` +3. FP32 / CUDA: + ```python + model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype=torch.float32, device_map="cuda", trust_remote_code=True) + ``` +4. FP32 / CPU: + ```python + model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype=torch.float32, device_map="cpu", trust_remote_code=True) + ``` + +To ensure the maximum compatibility, we recommend using the second execution mode (FP16 / CUDA), as follows: + ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer @@ -85,8 +105,7 @@ torch.set_default_device("cuda") model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2", trust_remote_code=True) -inputs = tokenizer('''```python -def print_prime(n): +inputs = tokenizer('''def print_prime(n): """ Print all primes between 1 and n """''', return_tensors="pt", return_attention_mask=False) @@ -96,7 +115,7 @@ text = tokenizer.batch_decode(outputs)[0] print(text) ``` -**Remark.** In the generation function, our model currently does not support beam search (`num_beams > 1`). +**Remark:** In the generation function, our model currently does not support beam search (`num_beams > 1`). Furthermore, in the forward pass of the model, we currently do not support outputting hidden states or attention values, or using custom input embeddings. ## Limitations of Phi-2