Instructions to use cyankiwi/Qwen3.5-27B-AWQ-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use cyankiwi/Qwen3.5-27B-AWQ-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="cyankiwi/Qwen3.5-27B-AWQ-4bit")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("cyankiwi/Qwen3.5-27B-AWQ-4bit")
model = AutoModelForMultimodalLM.from_pretrained("cyankiwi/Qwen3.5-27B-AWQ-4bit", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use cyankiwi/Qwen3.5-27B-AWQ-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cyankiwi/Qwen3.5-27B-AWQ-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cyankiwi/Qwen3.5-27B-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/cyankiwi/Qwen3.5-27B-AWQ-4bit

SGLang

How to use cyankiwi/Qwen3.5-27B-AWQ-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "cyankiwi/Qwen3.5-27B-AWQ-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cyankiwi/Qwen3.5-27B-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "cyankiwi/Qwen3.5-27B-AWQ-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cyankiwi/Qwen3.5-27B-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use cyankiwi/Qwen3.5-27B-AWQ-4bit with Docker Model Runner:
```
docker model run hf.co/cyankiwi/Qwen3.5-27B-AWQ-4bit
```

Easy to fall into infinite loop

by dwaynedu - opened Feb 27

Discussion

dwaynedu

Feb 27

•

edited Feb 27

It seems that this quantized model is easier to fall into infinite loop.

Prompt: "请为我设计一个适合团队玩的趣味游戏，游戏目标是通过合作完成一项挑战。"

For this AWQ-4bit model: Falls into infinite loop during thinking, even if I set repetition penalty to 2.0. Engine is vllm latest, kv cache is fp8 quantized.

The same model on OpenRouter does not have this issue.

cpatonn

cyankiwi org Feb 27

Thanks for letting me know. With linear attention layers of this model are quantized to int4, it is alreay prone to errors. I wouldn't recommend using FP8 kv cache for this model.

dwaynedu

Feb 27

•

edited Feb 27

I used non-quantized kv cache and the result is same, still very easy to fall into infinite loop.

I switched to the Qwen/Qwen3.5-27B-FP8 official FP8 quantization and it didn't have this issue, even if with FP8 kv cache.

dwaynedu

Feb 27

Tried several times more, and Qwen/Qwen3.5-27B-FP8 official FP8 quant also falls into infinite loop sometimes.

I'm beginning to speculate it is the model issue.

cpatonn

cyankiwi org Feb 27

I would recommend the following model cyankiwi/Qwen3.5-27B-AWQ-BF16-INT4, as I leave linear attention parameters at BF16. Linear attention params are heavily prone to quantization errors and I always leave linear attention parameters at BF16 in my quantized models.

This model cyankiwi/Qwen3.5-27B-AWQ-4bit is my only model having linear attention layers at INT4.

dwaynedu

Feb 28

Thanks, cyankiwi/Qwen3.5-27B-AWQ-BF16-INT4 is significantly better on the infinite loop issue!

Could you please check if cyankiwi/Qwen3.5-35B-A3B-AWQ-4bit has similar issue -- it doesn't specifically cause infinite loop on this previous prompt, but in our local test run it has shown infinite loop on some of our internal helpdesk chatbot test cases. Similarly, the infinite loop also occurs in the reasoning process.

Thank you very much.

tclf90

Mar 2

coud you try out

https://huggingface.co/QuantTrio/Qwen3.5-27B-AWQ

dwaynedu

Mar 4

Confirmed it is not quantization issue. It is a temperature issue for this model - if temperature is too low (my case is 0.7) and thinking is enabled while the question itself is rather open (such as a brainstorming), it tends to fall into infinite loop, probably because token is never sampled.

I managed to reproduce the issue even if using the original BF16 model.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment