Merge Allegro into diffusers

README.md

@@ -78,38 +78,54 @@ pipeline_tag: text-to-video
 
 # Quick start
 
-1. Download the [Allegro GitHub code](https://github.com/rhymes-ai/Allegro).
-   
-2. Install the necessary requirements.
+1. Install the necessary requirements.
      
-   - Ensure Python >= 3.10, PyTorch >= 2.4, CUDA >= 12.4. For details, see [requirements.txt](https://github.com/rhymes-ai/Allegro/blob/main/requirements.txt).  
+   - Ensure Python >= 3.10, PyTorch >= 2.4, CUDA >= 12.4.
        
-   - It is recommended to use Anaconda to create a new environment (Python >= 3.10) to run the following example.  
-
-3. Download the [Allegro model weights](https://huggingface.co/rhymes-ai/Allegro). Before diffuser integration, use git lfs or snapshot_download.
- 
-4. Run inference.
-   
-    ```python
-    python single_inference.py \
-    --user_prompt 'A seaside harbor with bright sunlight and sparkling seawater, with many boats in the water. From an aerial view, the boats vary in size and color, some moving and some stationary. Fishing boats in the water suggest that this location might be a popular spot for docking fishing boats.' \
-    --save_path ./output_videos/test_video.mp4
-    --vae your/path/to/vae \
-    --dit your/path/to/transformer \
-    --text_encoder your/path/to/text_encoder \
-    --tokenizer your/path/to/tokenizer \
-    --guidance_scale 7.5 \
-    --num_sampling_steps 100 \
-    --seed 42
-    ```
+   - It is recommended to use Anaconda to create a new environment (Python >= 3.10) `conda create -n rllegro python=3.10 -y` to run the following example.
   
-    Use '--enable_cpu_offload' to offload the model into CPU for less GPU memory cost (about 9.3G, compared to 27.5G if CPU offload is not enabled), but the inference time will increase significantly.
+   - run `pip install git+https://github.com/huggingface/diffusers.git@9214f4a3782a74e510eff7e09b59457fe8b63511 torch==2.4.1 transformers==4.40.1 accelerate sentencepiece imageio imageio-ffmpeg beautifulsoup4` 
+ 
+2. Run inference.
+    ```python
+    import torch
+    from diffusers import AutoencoderKLAllegro, AllegroPipeline
+    from diffusers.utils import export_to_video
+    vae = AutoencoderKLAllegro.from_pretrained("rhymes-ai/Allegro", subfolder="vae", torch_dtype=torch.float32)
+    pipe = AllegroPipeline.from_pretrained(
+        "rhymes-ai/Allegro", vae=vae, torch_dtype=torch.bfloat16
+    )
+    pipe.to("cuda")
+    pipe.vae.enable_tiling()
+    prompt = "A seaside harbor with bright sunlight and sparkling seawater, with many boats in the water. From an aerial view, the boats vary in size and color, some moving and some stationary. Fishing boats in the water suggest that this location might be a popular spot for docking fishing boats."
+    
+    positive_prompt = """
+    (masterpiece), (best quality), (ultra-detailed), (unwatermarked), 
+    {} 
+    emotional, harmonious, vignette, 4k epic detailed, shot on kodak, 35mm photo, 
+    sharp focus, high budget, cinemascope, moody, epic, gorgeous
+    """
+    
+    negative_prompt = """
+    nsfw, lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, 
+    low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry.
+    """
+    
+    prompt = prompt.format(prompt.lower().strip())
+    
+    video = pipe(prompt, negative_prompt=negative_prompt, guidance_scale=7.5, max_sequence_length=512, num_inference_steps=100, generator = torch.Generator(device="cuda:0").manual_seed(42)).frames[0]
+    export_to_video(video, "output.mp4", fps=15)
+    ```
+      
+    Use `pipe.enable_sequential_cpu_offload()` to offload the model into CPU for less GPU memory cost (about 9.3G, compared to 27.5G if CPU offload is not enabled), but the inference time will increase significantly.
 
-5. (Optional) Interpolate the video to 30 FPS.
+3. (Optional) Interpolate the video to 30 FPS.
 
     It is recommended to use [EMA-VFI](https://github.com/MCG-NJU/EMA-VFI) to interpolate the video from 15 FPS to 30 FPS.
   
     For better visual quality, please use imageio to save the video.
 
+4. For faster inference such Context Parallel, PAB, please refer to our [github repo](https://github.com/rhymes-ai/Allegro). 
+
 # License
 This repo is released under the Apache 2.0 License.