DIAL-TFM/shared_embedding
Updated
DIAL-TFM
classroom
AI & ML interests
None defined yet.
Recent Activity
JingzeShi
updated
a
model
about 20 hours ago
JingzeShi
published
a
model
about 21 hours ago
JingzeShi
updated
a
model
19 days ago
JingzeShi
published
a
model
19 days ago
Post
245
Is it time to start developing sparse attention again?
https://github.com/SmallDoges/flash-sparse-attention
https://github.com/SmallDoges/flash-sparse-attention
JingzeShi
authored
a
paper
5 months ago
Post
4077
Trainable selective sampling and sparse attention kernels are indispensable in the era of context engineering. We hope our work will be helpful to everyone! 🤗
Trainable Dynamic Mask Sparse Attention (2508.02124)
Trainable Dynamic Mask Sparse Attention (2508.02124)
JingzeShi
authored
a
paper
7 months ago
Post
2693
@SmallDoge
SmallTalks(
SmallDoge/SmallTalks) is a synthetic dataset designed for supervised fine-tuning of language models. The dataset covers a variety of conversational content, including daily conversations, tool usage, Python programming, encyclopedia Q&A, exam problem-solving, logical reasoning, and more. Each task is provided in both English and Chinese versions.
Post
4838
We distill a more accurate and concise dataset from DeepSeek R1, and also provide a distillation pipeline code repository.🤗
Dataset: SmallDoge/SmallThoughts
Code: https://github.com/SmallDoges/small-thoughts
Dataset: SmallDoge/SmallThoughts
Code: https://github.com/SmallDoges/small-thoughts
Post
2317
Welcome to the Doge Face Open Source Community! 🚀
Our goal is to explore the foundation of embodied intelligence for the next two years, which is indispensable – small language models. 🔬
We aim to open-source code and documentation to give everyone more time to slack off while working or studying! 🤗
👉 Repository name on Github: https://github.com/SmallDoges/small-doge
👉 Organization name on Hugging Face:
SmallDoge
Our goal is to explore the foundation of embodied intelligence for the next two years, which is indispensable – small language models. 🔬
We aim to open-source code and documentation to give everyone more time to slack off while working or studying! 🤗
👉 Repository name on Github: https://github.com/SmallDoges/small-doge
👉 Organization name on Hugging Face:
Post
1735
🤩warmup -> stable -> decay leanring rate scheduler:
😎use the Stable Phase CheckPoints to Continue Training the model on Any New Dataset without spikes of the training!!!
SmallDoge/Doge-20M-checkpoint
SmallDoge/Doge-60M-checkpoint
😎use the Stable Phase CheckPoints to Continue Training the model on Any New Dataset without spikes of the training!!!
SmallDoge/Doge-20M-checkpoint
SmallDoge/Doge-60M-checkpoint
-
- 4 replies
Post
2084
Only a single RTX 4090 running model pre-training is really slow, even for small language models!!! (https://huggingface.co/collections/JingzeShi/doge-slm-677fd879f8c4fd0f43e05458)
-
-
- 2 replies
JingzeShi
authored
a
paper
about 1 year ago