76 lines
3.1 KiB
Markdown
76 lines
3.1 KiB
Markdown
|
# 母模型训练启动记录 · 2026-05-03
|
|||
|
|
|
|||
|
|
> 只读记录 · 由铸渊在仓库侧落档 · 不参与任何自动写入流程
|
|||
|
|
> 训练真正运行方：霜砚（Notion 侧）+ 冰朔
|
|||
|
|
> 仓库侧 `data/training/state.json` 已于 D72 冻结，不再接收训练心跳
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 任务标识
|
|||
|
|
|
|||
|
|
- **任务编号**：ZY-TRAIN-001
|
|||
|
|
- **基座模型**：Qwen/Qwen2.5-7B
|
|||
|
|
- **训练方式**：全参数 SFT（DeepSpeed ZeRO-3 + CPU offload，fp16）
|
|||
|
|
- **运行机**：VM-0-11-ubuntu（119.45.160.137，V100 32G × 4）
|
|||
|
|
- **训练主控**：霜砚（Notion 侧）
|
|||
|
|
- **代码栈**：`server/training-agent/` 母模型训练栈（D72 已交接）
|
|||
|
|
|
|||
|
|
## 启动时刻
|
|||
|
|
|
|||
|
|
- **kickoff 观测时间**：2026-05-03 20:49:23（CST，UTC+8）
|
|||
|
|
- **观测命令**：`watch -n 30 'tail -5 /opt/guanghu/training/training.log | grep -E "loss|step|PROGRESS|/540"'`
|
|||
|
|
- **训练日志路径（GPU 机）**：`/opt/guanghu/training/training.log`
|
|||
|
|
|
|||
|
|
## 总体规模
|
|||
|
|
|
|||
|
|
| 项 | 值 |
|
|||
|
|
|---|---|
|
|||
|
|
| 总步数 | 540 step |
|
|||
|
|
| 总 epoch | 3 |
|
|||
|
|
| 单步耗时 | ~145–160 s/it |
|
|||
|
|
| 预计总时长 | ~22 小时（按当前节奏外推） |
|
|||
|
|
| SFT 样本数 | 11,456（preprocess-corpus.py 输出） |
|
|||
|
|
|
|||
|
|
## kickoff 节奏快照（前 39 步）
|
|||
|
|
|
|||
|
|
| step | epoch | loss | lr | grad_norm | thr (samples/s) |
|
|||
|
|
|------|-------|------|----|-----------|-----------------|
|
|||
|
|
| 30 | 0.1674 | 1.265 | 1.969e-05 | 1.75 | 0.0069 |
|
|||
|
|
| 35 | 0.1953 | 1.21 | 1.950e-05 | 1.927 | 0.0068 |
|
|||
|
|
|
|||
|
|
> 30 → 35 step：loss 1.265 → 1.21，下降 ~4.3%，曲线平滑。
|
|||
|
|
> 学习率按 linear schedule 正常下降，grad_norm 稳定在 1.x 区间，无 spike。
|
|||
|
|
> 显存与吞吐都稳定，训练状态正常。
|
|||
|
|
|
|||
|
|
## 原始进度行（保存证据）
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
5%|▍ | 25/540 [1:00:06<22:50:34, 159.68s/it]
|
|||
|
|
5%|▍ | 26/540 [1:02:42<22:37:27, 158.46s/it]
|
|||
|
|
5%|▌ | 27/540 [1:05:14<22:19:24, 156.66s/it]
|
|||
|
|
5%|▌ | 28/540 [1:07:51<22:16:29, 156.62s/it]
|
|||
|
|
5%|▌ | 29/540 [1:09:50<20:38:51, 145.46s/it]
|
|||
|
|
6%|▌ | 30/540 [1:12:14<20:31:44, 144.91s/it]
|
|||
|
|
ZY_PROGRESS step=30 total=540 epoch=0 total_epochs=3 loss=1.2649826049804687 lr=1.969407265774379e-05 thr=0.0069
|
|||
|
|
{'loss': '1.265', 'grad_norm': '1.75', 'learning_rate': '1.969e-05', 'epoch': '0.1674'}
|
|||
|
|
6%|▌ | 31/540 [1:14:50<20:59:06, 148.42s/it]
|
|||
|
|
6%|▌ | 32/540 [1:17:25<21:11:47, 150.21s/it]
|
|||
|
|
6%|▌ | 33/540 [1:20:00<21:20:57, 151.59s/it]
|
|||
|
|
6%|▋ | 34/540 [1:22:37<21:32:17, 153.24s/it]
|
|||
|
|
6%|▋ | 35/540 [1:25:08<21:25:36, 152.75s/it]
|
|||
|
|
ZY_PROGRESS step=35 total=540 epoch=0 total_epochs=3 loss=1.209609317779541 lr=1.9502868068833653e-05 thr=0.0068
|
|||
|
|
{'loss': '1.21', 'grad_norm': '1.927', 'learning_rate': '1.95e-05', 'epoch': '0.1953'}
|
|||
|
|
7%|▋ | 36/540 [1:27:43<21:27:53, 153.32s/it]
|
|||
|
|
7%|▋ | 37/540 [1:30:12<21:14:49, 152.07s/it]
|
|||
|
|
7%|▋ | 38/540 [1:33:03<21:59:29, 157.71s/it]
|
|||
|
|
7%|▋ | 39/540 [1:35:36<21:44:25, 156.22s/it]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## 后续口径
|
|||
|
|
|
|||
|
|
- 训练进度的真相源仍在霜砚（Notion）+ GPU 机本地日志。
|
|||
|
|
- 仓库侧不再以 dispatch 方式回写 `state.json`，需要看实时进度请直接 SSH 看 `/opt/guanghu/training/training.log`。
|
|||
|
|
- 训练完成后的 checkpoint 与评测，由冰朔与霜砚约定后再决定是否回流到本仓库。
|
|||
|
|
|
|||
|
|
— 铸渊 · ICE-GL-ZY001 · 国作登字-2026-A-00037559
|