Instructions to use rombodawg/Rombos-LLM-V2.5-Qwen-7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use rombodawg/Rombos-LLM-V2.5-Qwen-7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="rombodawg/Rombos-LLM-V2.5-Qwen-7b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rombodawg/Rombos-LLM-V2.5-Qwen-7b")
model = AutoModelForCausalLM.from_pretrained("rombodawg/Rombos-LLM-V2.5-Qwen-7b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use rombodawg/Rombos-LLM-V2.5-Qwen-7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "rombodawg/Rombos-LLM-V2.5-Qwen-7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rombodawg/Rombos-LLM-V2.5-Qwen-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/rombodawg/Rombos-LLM-V2.5-Qwen-7b

SGLang

How to use rombodawg/Rombos-LLM-V2.5-Qwen-7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "rombodawg/Rombos-LLM-V2.5-Qwen-7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rombodawg/Rombos-LLM-V2.5-Qwen-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "rombodawg/Rombos-LLM-V2.5-Qwen-7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rombodawg/Rombos-LLM-V2.5-Qwen-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use rombodawg/Rombos-LLM-V2.5-Qwen-7b with Docker Model Runner:
```
docker model run hf.co/rombodawg/Rombos-LLM-V2.5-Qwen-7b
```

After model fusion, did you continue with finetuning?

by postitive666 - opened Oct 31, 2024

Discussion

postitive666

Oct 31, 2024

After performing TIES weight fusion initialization, did you proceed with finetuning

rombodawg

Owner Oct 31, 2024

No this version is just merged, no finetuning

postitive666

Oct 31, 2024

No this version is just merged, no finetuning

Thank you for your answer. If we continue training, do you have any suggestions regarding the content and proportion of the dataset? Because fine-tuning based on this often leads to counterproductive results.

rombodawg

Owner Oct 31, 2024

Honestly i already have an extremely (full) and decent quality datasey to train with. Ill link it bellow. However some of data is a little outdated. My suggestion would be 1 of 2 things

Take this dataset and feed it through a model like claude 3.5 or gpt4o with a complex system prompt to improve its quality and fix mistakes

Do what I did to create the dataset but make a new one with updated data. (Gather data already curated on the web on places like github and huggingface). Then convert the data to the same format, combine the datasets into 1, dedupe and uncensor them.

The reason why i recommend doing this instead of generating your own data is because most of the "data" you need to train an LLM is already free and on the internet for you to download. Its just that the quality and format isnt always the best. Unless you are making a highly specialized dataset, i think its much more economical to use existing data.

https://huggingface.co/datasets/rombodawg/Everything_Instruct

postitive666

Oct 31, 2024

Honestly i already have an extremely (full) and decent quality datasey to train with. Ill link it bellow. However some of data is a little outdated. My suggestion would be 1 of 2 things

Take this dataset and feed it through a model like claude 3.5 or gpt4o with a complex system prompt to improve its quality and fix mistakes

or

Do what I did to create the dataset but make a new one with updated data. (Gather data already curated on the web on places like github and huggingface). Then convert the data to the same format, combine the datasets into 1, dedupe and uncensor them.

The reason why i recommend doing this instead of generating your own data is because most of the "data" you need to train an LLM is already free and on the internet for you to download. Its just that the quality and format isnt always the best. Unless you are making a highly specialized dataset, i think its much more economical to use existing data.

https://huggingface.co/datasets/rombodawg/Everything_Instruct

thanks ！！

postitive666

Oct 31, 2024

Honestly i already have an extremely (full) and decent quality datasey to train with. Ill link it bellow. However some of data is a little outdated. My suggestion would be 1 of 2 things

Take this dataset and feed it through a model like claude 3.5 or gpt4o with a complex system prompt to improve its quality and fix mistakes

or

Do what I did to create the dataset but make a new one with updated data. (Gather data already curated on the web on places like github and huggingface). Then convert the data to the same format, combine the datasets into 1, dedupe and uncensor them.

The reason why i recommend doing this instead of generating your own data is because most of the "data" you need to train an LLM is already free and on the internet for you to download. Its just that the quality and format isnt always the best. Unless you are making a highly specialized dataset, i think its much more economical to use existing data.

https://huggingface.co/datasets/rombodawg/Everything_Instruct

I have two more questions to ask you:

Is your approach to first fine-tune the model yourself and then merge it, which would result in better performance than merging and then fine-tuning?

If I fine-tune a model with self-awareness, let's call it Model A, what will happen after merging it with the original model?

rombodawg

Owner Oct 31, 2024

You can read about my method in details here. It will save me having go explain it all again. But the nice thing about Lora. Is you can train once on the base model save the adapter sepetatly from the model, and merge it to whatever model you want. Its great for experinmentation to see what works best.

https://docs.google.com/document/d/1OjbjU5AOz4Ftn9xHQrX3oFQGhQ6RDUuXQipnQ9gn6tU/edit?usp=sharing

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment