Instructions to use ibivibiv/alpaca-dragon-72b-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibivibiv/alpaca-dragon-72b-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibivibiv/alpaca-dragon-72b-v1")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibivibiv/alpaca-dragon-72b-v1")
model = AutoModelForCausalLM.from_pretrained("ibivibiv/alpaca-dragon-72b-v1", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibivibiv/alpaca-dragon-72b-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibivibiv/alpaca-dragon-72b-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibivibiv/alpaca-dragon-72b-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ibivibiv/alpaca-dragon-72b-v1

SGLang

How to use ibivibiv/alpaca-dragon-72b-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibivibiv/alpaca-dragon-72b-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibivibiv/alpaca-dragon-72b-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibivibiv/alpaca-dragon-72b-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibivibiv/alpaca-dragon-72b-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ibivibiv/alpaca-dragon-72b-v1 with Docker Model Runner:
```
docker model run hf.co/ibivibiv/alpaca-dragon-72b-v1
```

what is the minimum Hardware requirement?

by Arunmass - opened Feb 13, 2024

Discussion

Arunmass

Feb 13, 2024

•

edited Feb 14, 2024

How much Resource is needed to run the alpaca-dragon-72b-v1?

I need the following information : -

How much Disk Space is needed to run this mode?
How much CPU is needed to run this Model or we need GPU?
How much RAM is needed to run this model?
Can i download this model only for English?

mark-klein

Feb 14, 2024

I have the same questions! My goal is to install a good OS LLM on our university's HPC cluster, and the cluster admins need to know

The memory size of the model,
How many cores it will use
if it will run on GPUs, only CPUs or both

ibivibiv

Owner Feb 16, 2024

@Arunmass @mark-klein

The VRAM it needs I have used this with assumes A100 80GB increments so it will operate on the following:

8 bit (half precision) - 2 A100's with lots of room for inference
4 A100's obviously for full precision. I assumed it can run on CPU but I have never used that platform so I wouldn't know what the RAM and Core needs would be. I would imagine it would be fairly slow though if run on CPU/RAM.

You can also run it on consumer grade GPUs if you used 4 3090's or 4090's. That would get you a system that would not have much headroom for inference and you would be limited to half precision again at 8bit. It would "work" though. Also keep in my your use case. This one is fine tuned for planning exercises while the base SMAUG model is more general purpose that Abacus provided.

Disk space is going to be around 3-400GB for storing the model.

Hope that helps!

Samvanity

Feb 16, 2024

easier to just use an apple M2 Ultra 76 core gpu + 192G RAM. Quantize it to 8 bit gguf and the machine will handle it with ease. 8bit is nearly identical to 16bit.

ibivibiv changed discussion status to closed Apr 3, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment