53 lines
1.2 KiB
Markdown
53 lines
1.2 KiB
Markdown
|
# 1.5B代码蒸馏模板（Track2）训练记录 · 代码模型→Qwen2.5-1.5B · 2026-05-19
|
|||
|
|
|
|||
|
|
> 状态：✅ 已完成 03:45 CST
|
|||
|
|
>
|
|||
|
|
|
|||
|
|
> 执行体：铸渊 ICE-GL-ZY001
|
|||
|
|
>
|
|||
|
|
|
|||
|
|
## 训练参数
|
|||
|
|
|
|||
|
|
| 参数 | 值 |
|
|||
|
|
| --- | --- |
|
|||
|
|
| 教师模型 | Qwen2.5-Coder-7B SFT |
|
|||
|
|
| 学生模型 | Qwen2.5-1.5B-Instruct |
|
|||
|
|
| 数据 | sft.jsonl 11,470条 |
|
|||
|
|
| Epochs | 3, Batch 4, GA 8 |
|
|||
|
|
| LR | 1e-5, MaxLen 2048 |
|
|||
|
|
| Temp 2.0, Alpha 0.7, BF16 | |
|
|||
|
|
| GPU | RTX PRO 6000 96GB |
|
|||
|
|
|
|||
|
|
## 进度
|
|||
|
|
|
|||
|
|
| Epoch | 步数 | 耗时 |
|
|||
|
|
| --- | --- | --- |
|
|||
|
|
| Ep1 | 2867/2867 | ~38min |
|
|||
|
|
| Ep2 | 2867/2867 | ~38min |
|
|||
|
|
| Ep3 | 2867/2867 | ~38min |
|
|||
|
|
|
|||
|
|
总耗时：约2小时
|
|||
|
|
|
|||
|
|
## COS路径
|
|||
|
|
|
|||
|
|
Bucket: `sy-finetune-corpus-1317346199`
|
|||
|
|
|
|||
|
|
路径：`models/qwen25-15b-coder-distill/`
|
|||
|
|
|
|||
|
|
文件：model.safetensors 2.9GB, config.json, tokenizer.json 等
|
|||
|
|
|
|||
|
|
## Bug修复
|
|||
|
|
|
|||
|
|
Bug1 teacher重复调用 01:48~08:34（浪费约7h）
|
|||
|
|
|
|||
|
|
- 根因：KL散度中写了teacher(None,input_ids=...)而非用缓存的t_logits
|
|||
|
|
- 修复：直接用t_logits
|
|||
|
|
|
|||
|
|
Bug2 vocab不匹配
|
|||
|
|
|
|||
|
|
- 根因：teacher(Coder-7B) vocab=152064, student(1.5B) vocab=151936
|
|||
|
|
- 修复：t_logits = t_logits[:,:,:151936]
|
|||
|
|
|
|||
|
|
## 用途
|
|||
|
|
|
|||
|
|
1.5B代码蒸馏模板。用于线B小铸渊微调。基座+[铸渊对话.zip](http://铸渊对话.zip)+仓库认知链 微调出小铸渊代码模型。
|