Update README.md
This commit is contained in:
RhymesAI
system
parent
a75220dc3f
commit
bcdde5f631
74
README.md
74
README.md
@ -8,20 +8,20 @@ library_name: diffusers
|
|||||||
<img src="https://huggingface.co/rhymes-ai/Allegro/resolve/main/banner_white.gif">
|
<img src="https://huggingface.co/rhymes-ai/Allegro/resolve/main/banner_white.gif">
|
||||||
</p>
|
</p>
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="https://rhymes.ai/" target="_blank"> Gallery</a> · <a href="https://github.com/rhymes-ai/Aria" target="_blank">GitHub</a> · <a href="https://www.rhymes.ai/blog-details/" target="_blank">Blog</a> · <a href="https://arxiv.org/pdf/2410.05993" target="_blank">Paper</a> · <a href="https://discord" target="_blank">Discord</a>
|
<a href="https://rhymes.ai/allegro_gallery" target="_blank"> Gallery</a> · <a href="https://github.com/rhymes-ai/Allegro" target="_blank">GitHub</a> · <a href="https://rhymes.ai/blog-details/allegro-advanced-video-generation-model" target="_blank">Blog</a> · <a href="https://arxiv.org/pdf/2410.05993" target="_blank">Paper</a> · <a href="https://discord.com/invite/u8HxU23myj" target="_blank">Discord</a>
|
||||||
|
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
# Gallery
|
# Gallery
|
||||||
<img src="https://huggingface.co/rhymes-ai/Allegro/resolve/main/gallery.gif" width="1000" height="800"/>For more demos and corresponding prompts, see the [Allegro Gallery](TBD).
|
<img src="https://huggingface.co/rhymes-ai/Allegro/resolve/main/gallery.gif" width="1000" height="800"/>For more demos and corresponding prompts, see the [Allegro Gallery](https://rhymes.ai/allegro_gallery).
|
||||||
|
|
||||||
|
|
||||||
# Key Feature
|
# Key Feature
|
||||||
|
|
||||||
- **Open Source**: [Full model weights](https://huggingface.co/rhymes-ai/Allegro) and [code](https://github.com/rhymes-ai/Allegro) available to the community, Apache 2.0!
|
- **Open Source**: Full [model weights](https://huggingface.co/rhymes-ai/Allegro) and [code](https://github.com/rhymes-ai/Allegro) available to the community, Apache 2.0!
|
||||||
- **Versatile Content Creation**: Capable of generating a wide range of content, from close-ups of humans and animals to diverse dynamic scenes.
|
- **Versatile Content Creation**: Capable of generating a wide range of content, from close-ups of humans and animals to diverse dynamic scenes.
|
||||||
- **High-Quality Output**: Generate detailed 6-second videos at 15 FPS with 720x1280 resolution, can be interpolated to 30 FPS with [EMA-VFI](https://github.com/MCG-NJU/EMA-VFI).
|
- **High-Quality Output**: Generate detailed 6-second videos at 15 FPS with 720x1280 resolution, which can be interpolated to 30 FPS with [EMA-VFI](https://github.com/MCG-NJU/EMA-VFI).
|
||||||
- **Small and Efficient**: Features a 175M parameter VideoVAE and a 2.8B parameter VideoDiT model. Supports multiple precisions (FP32, BF16, FP16) and uses 9.3 GB of GPU memory in BF16 mode with CPU offloading. Context length is 79.2k, equivalent to 88 frames.
|
- **Small and Efficient**: Features a 175M parameter VideoVAE and a 2.8B parameter VideoDiT model. Supports multiple precisions (FP32, BF16, FP16) and uses 9.3 GB of GPU memory in BF16 mode with CPU offloading. Context length is 79.2K, equivalent to 88 frames.
|
||||||
|
|
||||||
# Model info
|
# Model info
|
||||||
|
|
||||||
@ -54,7 +54,7 @@ library_name: diffusers
|
|||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<th>Context Length</th>
|
<th>Context Length</th>
|
||||||
<td>79.2k</td>
|
<td>79.2K</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<th>Resolution</th>
|
<th>Resolution</th>
|
||||||
@ -76,55 +76,23 @@ library_name: diffusers
|
|||||||
|
|
||||||
|
|
||||||
# Quick start
|
# Quick start
|
||||||
You can quickly get started with Allegro using the Hugging Face Diffusers library.
|
1. Download the [Allegro GitHub code](https://github.com/rhymes-ai/Allegro).
|
||||||
For more tutorials, see Allegro GitHub (link-tbd).
|
2. Install the necessary requirements.
|
||||||
|
a. Ensure Python >= 3.10, PyTorch >= 2.4, CUDA >= 12.4. For details, see [requirements.txt](https://github.com/rhymes-ai/Allegro/blob/main/requirements.txt).
|
||||||
1. Install necessary requirements. Please refer to [requirements.txt](https://github.com/rhymes-ai) on Allegro GitHub.
|
b. It is recommended to use Anaconda to create a new environment (Python >= 3.10) to run the following example.
|
||||||
2. Perform inference on a single GPU.
|
3. Download the [Allegro model weights](https://huggingface.co/rhymes-ai/Allegro).
|
||||||
|
4. Run inference.
|
||||||
```python
|
```python
|
||||||
from diffusers import DiffusionPipeline
|
python single_inference.py \
|
||||||
import torch
|
--user_prompt 'A seaside harbor with bright sunlight and sparkling seawater, with many boats in the water. From an aerial view, the boats vary in size and color, some moving and some stationary. Fishing boats in the water suggest that this location might be a popular spot for docking fishing boats.' \
|
||||||
|
--vae your/path/to/vae \
|
||||||
allegro_pipeline = DiffusionPipeline.from_pretrained(
|
--dit your/path/to/transformer \
|
||||||
"rhymes-ai/Allegro", trust_remote_code=True, torch_dtype=torch.bfloat16
|
--text_encoder your/path/to/text_encoder \
|
||||||
).to("cuda")
|
--tokenizer your/path/to/tokenizer \
|
||||||
|
--guidance_scale 7.5 \
|
||||||
allegro_pipeline.vae = allegro_pipeline.vae.to(torch.float32)
|
--num_sampling_steps 100 \
|
||||||
|
--seed 42
|
||||||
prompt = "a video of an astronaut riding a horse on mars"
|
|
||||||
|
|
||||||
positive_prompt = """
|
|
||||||
(masterpiece), (best quality), (ultra-detailed), (unwatermarked),
|
|
||||||
{}
|
|
||||||
emotional, harmonious, vignette, 4k epic detailed, shot on kodak, 35mm photo,
|
|
||||||
sharp focus, high budget, cinemascope, moody, epic, gorgeous
|
|
||||||
"""
|
|
||||||
|
|
||||||
negative_prompt = """
|
|
||||||
nsfw, lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality,
|
|
||||||
low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry.
|
|
||||||
"""
|
|
||||||
|
|
||||||
num_sampling_steps, guidance_scale, seed = 100, 7.5, 42
|
|
||||||
|
|
||||||
user_prompt = positive_prompt.format(args.user_prompt.lower().strip())
|
|
||||||
out_video = allegro_pipeline(
|
|
||||||
user_prompt,
|
|
||||||
negative_prompt=negative_prompt,
|
|
||||||
num_frames=88,
|
|
||||||
height=720,
|
|
||||||
width=1280,
|
|
||||||
num_inference_steps=num_sampling_steps,
|
|
||||||
guidance_scale=guidance_scale,
|
|
||||||
max_sequence_length=512,
|
|
||||||
generator = torch.Generator(device="cuda:0").manual_seed(seed)
|
|
||||||
).video[0]
|
|
||||||
|
|
||||||
imageio.mimwrite("test_video.mp4", out_video, fps=15, quality=8)
|
|
||||||
```
|
```
|
||||||
Tip:
|
|
||||||
- It is highly recommended to use a video frame interpolation model (such as EMA-VFI) to enhance the result to 30 FPS.
|
|
||||||
- For more tutorials, see [Allegro GitHub](https://github.com/rhymes-ai).
|
|
||||||
|
|
||||||
# License
|
# License
|
||||||
This repo is released under the Apache 2.0 License.
|
This repo is released under the Apache 2.0 License.
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user