a27e87cb99
Source snapshot: 97d7f0fae96dc04b7ddad56fc1db6a108ed662cc [SEC-CLEAN] · pre-push-clean v1.0 · 109处敏感信息已自动转乱码
53 lines
1.2 KiB
Markdown
53 lines
1.2 KiB
Markdown
# 1.5B代码蒸馏模板（Track2）训练记录 · 代码模型→Qwen2.5-1.5B · 2026-05-19
|
||
|
||
> 状态：✅ 已完成 03:45 CST
|
||
>
|
||
|
||
> 执行体：铸渊 ICE-GL-ZY001
|
||
>
|
||
|
||
## 训练参数
|
||
|
||
| 参数 | 值 |
|
||
| --- | --- |
|
||
| 教师模型 | Qwen2.5-Coder-7B SFT |
|
||
| 学生模型 | Qwen2.5-1.5B-Instruct |
|
||
| 数据 | sft.jsonl 11,470条 |
|
||
| Epochs | 3, Batch 4, GA 8 |
|
||
| LR | 1e-5, MaxLen 2048 |
|
||
| Temp 2.0, Alpha 0.7, BF16 | |
|
||
| GPU | RTX PRO 6000 96GB |
|
||
|
||
## 进度
|
||
|
||
| Epoch | 步数 | 耗时 |
|
||
| --- | --- | --- |
|
||
| Ep1 | 2867/2867 | ~38min |
|
||
| Ep2 | 2867/2867 | ~38min |
|
||
| Ep3 | 2867/2867 | ~38min |
|
||
|
||
总耗时：约2小时
|
||
|
||
## COS路径
|
||
|
||
Bucket: `sy-finetune-corpus-1317346199`
|
||
|
||
路径：`models/qwen25-15b-coder-distill/`
|
||
|
||
文件：model.safetensors 2.9GB, config.json, tokenizer.json 等
|
||
|
||
## Bug修复
|
||
|
||
Bug1 teacher重复调用 01:48~08:34（浪费约7h）
|
||
|
||
- 根因：KL散度中写了teacher(None,input_ids=...)而非用缓存的t_logits
|
||
- 修复：直接用t_logits
|
||
|
||
Bug2 vocab不匹配
|
||
|
||
- 根因：teacher(Coder-7B) vocab=152064, student(1.5B) vocab=151936
|
||
- 修复：t_logits = t_logits[:,:,:151936]
|
||
|
||
## 用途
|
||
|
||
1.5B代码蒸馏模板。用于线B小铸渊微调。基座+[铸渊对话.zip](http://铸渊对话.zip)+仓库认知链 微调出小铸渊代码模型。 |