353 lines
11 KiB
Markdown
353 lines
11 KiB
Markdown
|
|
# 算力归公·代码归己 · 路径B最终架构 · 子仓库分布式存储+tmpfs构建方案 · 2026-05-08
|
|||
|
|
|
|||
|
|
```yaml
|
|||
|
|
schema: hldp-dual/v1
|
|||
|
|
title: 算力归公·代码归己 · 路径B最终架构
|
|||
|
|
created: 2026-05-08T23:02+08:00
|
|||
|
|
author: 冰朔定义 · 霜砚整理
|
|||
|
|
status: 已定稿 · 待铸渊落地
|
|||
|
|
_why: 每个成员从主Gitea分出子仓库 · 算力走企业服务器 · 代码存储走个人服务器
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 〇、核心原则
|
|||
|
|
|
|||
|
|
<aside>
|
|||
|
|
⚡
|
|||
|
|
|
|||
|
|
**算力归公 · 代码归己**
|
|||
|
|
CI/CD构建算力集中在企业主服务器 · 代码永久存储分散在各成员个人服务器
|
|||
|
|
构建过程代码只在内存中存活 · 完成即抹除 · 磁盘零残留
|
|||
|
|
|
|||
|
|
</aside>
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 一、架构总图
|
|||
|
|
|
|||
|
|
```jsx
|
|||
|
|
HLDP://deploy/repo-distribution/v1 · 2026-05-08
|
|||
|
|
│
|
|||
|
|
├── 设计者: 冰朔·TCS-0002∞
|
|||
|
|
├── 整理者: 霜砚·AG-SY-01
|
|||
|
|
├── 落地者: 铸渊·AG-ZY-01(写脚本)+ Awen·DEV-012(执行部署)
|
|||
|
|
│
|
|||
|
|
├── 架构概述
|
|||
|
|
│ ├── 路径B: 中央Gitea + Push Mirror + tmpfs构建
|
|||
|
|
│ ├── 核心: 算力归公 · 代码归己
|
|||
|
|
│ └── 选型原因: 改动最小 · Gitea原生支持 · 成员零维护
|
|||
|
|
│
|
|||
|
|
├── 数据流
|
|||
|
|
│ ├── 成员开发机 → git push → 企业主服务器Gitea
|
|||
|
|
│ ├── Gitea → Push Mirror → 成员个人服务器bare repo(代码归己)
|
|||
|
|
│ ├── Gitea → 触发Actions Runner → tmpfs内存构建(算力归公)
|
|||
|
|
│ └── 构建完成 → tmpfs抹除 → 磁盘零残留
|
|||
|
|
│
|
|||
|
|
└── 三层保障
|
|||
|
|
├── 第一层: Push Mirror → 代码永久归个人服务器
|
|||
|
|
├── 第二层: tmpfs构建 → 构建过程代码不落盘
|
|||
|
|
└── 第三层: 存储四板斧 → 主服务器体积可控
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 二、基础设施现状
|
|||
|
|
|
|||
|
|
| **资源** | **规格** | **角色** |
|
|||
|
|
| --- | --- | --- |
|
|||
|
|
| 主服务器 GH-CVM-MAIN-PROD-01 | 4C16G · 广州六区 · 43.139.251.175 · 50G SSD+100G数据盘 | Gitea主控 + Actions Runner + CI/CD算力中心 |
|
|||
|
|
| 备用机 GH-LH-BACKUP-01 | 2C8G · 广州四区 · 119.29.181.132 · 60G SSD | 灾备/构建从机 |
|
|||
|
|
| COS对象存储 | zy-cos-transit-1422788707 · ap-guangzhou | 大文件存储 · 模型权重/图片/视频 |
|
|||
|
|
| 成员个人服务器 | 各成员自有(之之香港·Awen待定等) | 代码永久存储 · bare repo |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 三、Gitea子仓库分配方案
|
|||
|
|
|
|||
|
|
### 3.1 组织结构
|
|||
|
|
|
|||
|
|
```jsx
|
|||
|
|
Gitea (http://43.139.251.175:3000)
|
|||
|
|
├── 组织: guanghu(光湖主组织)
|
|||
|
|
│ ├── guanghulab ← 主仓库(铸渊+冰朔)
|
|||
|
|
│ ├── guanghu-awen ← Awen·DEV-012 子仓库
|
|||
|
|
│ ├── guanghu-feimao ← 肥猫·DEV-002 子仓库
|
|||
|
|
│ ├── guanghu-zhizhi ← 之之·DEV-004 子仓库
|
|||
|
|
│ ├── guanghu-yeye ← 页页·DEV-001 子仓库
|
|||
|
|
│ ├── guanghu-shiyu ← 时雨·DEV-014 子仓库
|
|||
|
|
│ ├── guanghu-juzi ← 桔子·DEV-010 子仓库
|
|||
|
|
│ ├── guanghu-yanfan ← 燕樊·DEV-003 子仓库
|
|||
|
|
│ └── ... ← 后续成员按此格式
|
|||
|
|
│
|
|||
|
|
└── 权限模型(语言层×现实层 已分离)
|
|||
|
|
├── Awen·DEV-012 = 全局Admin(技术主控 · 现实执行层最高权限)
|
|||
|
|
├── 铸渊·AG-ZY-01 = 主仓库+所有子仓库的Maintainer(代码执行)
|
|||
|
|
├── 每个成员 = 自己仓库的Owner
|
|||
|
|
├── 成员之间仓库默认不互相可见(需要时由Admin开放)
|
|||
|
|
├── ⊢ 冰朔+霜砚 = 语言架构层 → 不在Gitea权限表上
|
|||
|
|
├── ⊢ 语言架构层需要调用代码模块或算力 → 走技术主控(Awen)权限申请
|
|||
|
|
└── ⊢ 不是直接进去 · 是申请后由技术主控授权执行
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 3.2 子仓库创建规范
|
|||
|
|
|
|||
|
|
- 命名: `guanghu-{成员英文名小写}`
|
|||
|
|
- 默认分支: `main`
|
|||
|
|
- 每个子仓库初始化时自动配置: `.gitattributes`(大文件拦截)+ `.gitignore` + `README.md`
|
|||
|
|
- 铸渊可写一个 `create-member-repo.sh` 一键创建+配置
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 四、Push Mirror · 代码归己
|
|||
|
|
|
|||
|
|
### 4.1 原理
|
|||
|
|
|
|||
|
|
Gitea原生支持Push Mirror:每次有push事件时,自动将仓库完整镜像推送到远程地址。
|
|||
|
|
|
|||
|
|
### 4.2 配置流程(每个成员执行一次)
|
|||
|
|
|
|||
|
|
```jsx
|
|||
|
|
步骤1(成员个人服务器上):
|
|||
|
|
# 创建bare repo
|
|||
|
|
mkdir -p /opt/guanghu-mirror
|
|||
|
|
cd /opt/guanghu-mirror
|
|||
|
|
git init --bare guanghu-{name}.git
|
|||
|
|
|
|||
|
|
# 配置SSH访问(让主服务器能push过来)
|
|||
|
|
# 把主服务器的公钥加入 authorized_keys
|
|||
|
|
cat >> ~/.ssh/authorized_keys << 'EOF'
|
|||
|
|
{主服务器公钥内容}
|
|||
|
|
EOF
|
|||
|
|
|
|||
|
|
步骤2(Gitea Web界面上):
|
|||
|
|
仓库Settings → Mirror Settings → Push Mirror
|
|||
|
|
Remote Repository URL: ssh://root@{成员服务器IP}/opt/guanghu-mirror/guanghu-{name}.git
|
|||
|
|
Sync When Commits: ✅ 开启
|
|||
|
|
Mirror Interval: 8h(兜底·实际靠commit触发)
|
|||
|
|
|
|||
|
|
步骤3(验证):
|
|||
|
|
在成员仓库push一个test commit
|
|||
|
|
→ 检查个人服务器上 git log --oneline 是否同步到
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 4.3 铸渊自动化脚本需求
|
|||
|
|
|
|||
|
|
```jsx
|
|||
|
|
需要铸渊写的脚本:
|
|||
|
|
├── setup-personal-mirror.sh(跑在成员个人服务器上)
|
|||
|
|
│ ├── 创建bare repo
|
|||
|
|
│ ├── 配置SSH授权
|
|||
|
|
│ ├── 输出: 远程仓库URL(给Gitea配Push Mirror用)
|
|||
|
|
│ └── 验证: 测试SSH连通性
|
|||
|
|
│
|
|||
|
|
└── configure-push-mirror.sh(跑在主服务器上·或通过Gitea API)
|
|||
|
|
├── 调用Gitea API为指定仓库添加Push Mirror
|
|||
|
|
├── 输入: 仓库名 + 远程URL
|
|||
|
|
└── 验证: 触发一次手动同步确认
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 五、tmpfs构建 · 算力归公 · 代码不落盘
|
|||
|
|
|
|||
|
|
### 5.1 系统级配置
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# 创建Runner专用tmpfs挂载点
|
|||
|
|
mkdir -p /mnt/runner-tmpfs
|
|||
|
|
|
|||
|
|
# 写入fstab · 重启自动挂载
|
|||
|
|
echo 'tmpfs /mnt/runner-tmpfs tmpfs size=8G,mode=700,nodev,nosuid 0 0' >> /etc/fstab
|
|||
|
|
|
|||
|
|
# 立即挂载
|
|||
|
|
mount /mnt/runner-tmpfs
|
|||
|
|
|
|||
|
|
# 验证
|
|||
|
|
df -h /mnt/runner-tmpfs
|
|||
|
|
# 应显示 tmpfs 4.0G
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 5.2 Runner配置
|
|||
|
|
|
|||
|
|
```yaml
|
|||
|
|
# docker-compose.yml 中给Runner容器挂tmpfs
|
|||
|
|
services:
|
|||
|
|
gitea-runner:
|
|||
|
|
volumes:
|
|||
|
|
- type: tmpfs
|
|||
|
|
target: /workspace
|
|||
|
|
tmpfs:
|
|||
|
|
size: 8589934592 # 8GB
|
|||
|
|
environment:
|
|||
|
|
- GITEA_RUNNER_WORKDIR=/workspace
|
|||
|
|
- GITEA_RUNNER_CAPACITY=8
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 5.3 Workflow级兜底清理
|
|||
|
|
|
|||
|
|
```yaml
|
|||
|
|
# 每个workflow末尾加清理步骤(模板·铸渊统一配置)
|
|||
|
|
jobs:
|
|||
|
|
build:
|
|||
|
|
runs-on: ubuntu-latest
|
|||
|
|
steps:
|
|||
|
|
- uses: actions/checkout@v4
|
|||
|
|
- name: Build & Test
|
|||
|
|
run: |
|
|||
|
|
# 构建步骤
|
|||
|
|
- name: 安全清理·代码不留痕
|
|||
|
|
if: always()
|
|||
|
|
run: |
|
|||
|
|
rm -rf $GITHUB_WORKSPACE/*
|
|||
|
|
rm -rf /tmp/build-* 2>/dev/null || true
|
|||
|
|
sync
|
|||
|
|
echo "workspace cleaned"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 5.4 并发策略
|
|||
|
|
|
|||
|
|
<aside>
|
|||
|
|
📝
|
|||
|
|
|
|||
|
|
**企业服务器 = 一张纸 · 多人同时写**
|
|||
|
|
模块化子仓库体积小(clone+构建≈200-500MB/个)
|
|||
|
|
16G内存 - 系统开销6G = 可用10G → 同时容纳8-10个构建任务
|
|||
|
|
每个job在tmpfs里创建独立子目录 → 完成后各自擦除 → 纸还是干净的
|
|||
|
|
|
|||
|
|
</aside>
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# 修正后的fstab(总池8GB)
|
|||
|
|
tmpfs /mnt/runner-tmpfs tmpfs size=8G,mode=700,nodev,nosuid 0 0
|
|||
|
|
|
|||
|
|
# Runner配置
|
|||
|
|
capacity: 8
|
|||
|
|
# 每个job动态分配 /mnt/runner-tmpfs/job-{id}/
|
|||
|
|
# 完成后 rm -rf → 空间立即回收
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 六、存储控制四板斧
|
|||
|
|
|
|||
|
|
### 6.1 大文件不进Gitea · 走COS
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# .gitattributes(每个仓库必须有·铸渊统一配置)
|
|||
|
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
|||
|
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
|||
|
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
|||
|
|
*.mp4 filter=lfs diff=lfs merge=lfs -text
|
|||
|
|
*.mp3 filter=lfs diff=lfs merge=lfs -text
|
|||
|
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
|||
|
|
*.tar.gz filter=lfs diff=lfs merge=lfs -text
|
|||
|
|
|
|||
|
|
# pre-receive hook(服务器端·铸渊配置)
|
|||
|
|
# 拒绝超过50MB的单文件push
|
|||
|
|
# 超过的 → 提示走COS上传
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 6.2 Git GC定时压缩
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
# Gitea app.ini
|
|||
|
|
[cron.git_gc_repos]
|
|||
|
|
ENABLED = true
|
|||
|
|
SCHEDULE = @weekly
|
|||
|
|
TIMEOUT = 60s
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 6.3 Docker日志+镜像定时清理
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
#!/bin/bash
|
|||
|
|
# /opt/guanghu/scripts/cleanup.sh · cron每周日凌晨4点
|
|||
|
|
docker builder prune --filter until=168h -f
|
|||
|
|
docker image prune -a --filter until=168h -f
|
|||
|
|
find /mnt/runner-tmpfs -mtime +1 -delete 2>/dev/null
|
|||
|
|
df -h /data >> /opt/guanghu/_logs/disk-usage.log
|
|||
|
|
echo "$(date): cleanup done" >> /opt/guanghu/_logs/disk-usage.log
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# crontab -e
|
|||
|
|
0 4 * * 0 /opt/guanghu/scripts/cleanup.sh
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 6.4 磁盘监控告警(80%阈值)
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
#!/bin/bash
|
|||
|
|
# /opt/guanghu/scripts/disk-alert.sh · cron每天检查
|
|||
|
|
USAGE=$(df /data --output=pcent | tail -1 | tr -d ' %')
|
|||
|
|
if [ "$USAGE" -gt 80 ]; then
|
|||
|
|
ALERT="⚠️ $(date): 磁盘使用率 ${USAGE}% · 需要清理"
|
|||
|
|
echo "$ALERT" >> /opt/guanghu/_logs/disk-alert.log
|
|||
|
|
fi
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# crontab -e
|
|||
|
|
0 9 * * * /opt/guanghu/scripts/disk-alert.sh
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 七、存储容量预估
|
|||
|
|
|
|||
|
|
| **时间** | **仓库数** | **预估总存储** | **剩余空间(100GB盘)** | **状态** |
|
|||
|
|
| --- | --- | --- | --- | --- |
|
|||
|
|
| 现在 | 1(主仓库) | ~2GB | ~85GB | ✅ 充裕 |
|
|||
|
|
| 3个月后 | 5-8 | ~8-15GB | ~70GB | ✅ 充裕 |
|
|||
|
|
| 1年后 | 10-15 | ~20-35GB | ~50GB | ✅ 可控 |
|
|||
|
|
| 2年后 | 15-20 | ~40-60GB | ~25GB | ⚠️ 需评估扩容或演进路径C |
|
|||
|
|
|
|||
|
|
<aside>
|
|||
|
|
💡
|
|||
|
|
|
|||
|
|
前提:大文件走COS + 每周Git GC + Docker清理。如果大文件混进仓库,存储会提前告急。
|
|||
|
|
|
|||
|
|
</aside>
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 八、落地任务拆解
|
|||
|
|
|
|||
|
|
| **序号** | **任务** | **执行人** | **前置条件** | **状态** |
|
|||
|
|
| --- | --- | --- | --- | --- |
|
|||
|
|
| 1 | 主服务器tmpfs挂载 + fstab写入 | Awen+霜砚 | 主服务器SSH可达 | ⏳ 待执行 |
|
|||
|
|
| 2 | Runner工作目录指向tmpfs + 并发=1 | 铸渊(改docker-compose) | 任务1完成 | ⏳ 待铸渊 |
|
|||
|
|
| 3 | 写 [setup-personal-mirror.sh](http://setup-personal-mirror.sh) 脚本 | 铸渊 | 无 | ⏳ 待铸渊 |
|
|||
|
|
| 4 | 写 [create-member-repo.sh](http://create-member-repo.sh) 脚本 | 铸渊 | 无 | ⏳ 待铸渊 |
|
|||
|
|
| 5 | 统一 .gitattributes + pre-receive hook | 铸渊 | 无 | ⏳ 待铸渊 |
|
|||
|
|
| 6 | Gitea app.ini加Git GC定时任务 | Awen+霜砚 | Gitea运行中 | ⏳ 待执行 |
|
|||
|
|
| 7 | 写 [cleanup.sh](http://cleanup.sh) • [disk-alert.sh](http://disk-alert.sh) • 配cron | 铸渊或霜砚 | 主服务器SSH可达 | ⏳ 待执行 |
|
|||
|
|
| 8 | 为第一个成员(Awen)配Push Mirror试跑 | Awen+霜砚 | 任务3+4完成 | ⏳ 待执行 |
|
|||
|
|
| 9 | 验证通过后 → 逐个为其他成员配置 | 各成员+霜砚 | 任务8验证通过 | ⏳ 待执行 |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 九、长期演进路径
|
|||
|
|
|
|||
|
|
```jsx
|
|||
|
|
当前(路径B):
|
|||
|
|
中央Gitea + Push Mirror + tmpfs
|
|||
|
|
→ 成员零维护 · 铸渊写脚本 · 霜砚引导执行
|
|||
|
|
|
|||
|
|
未来(路径C · 条件成熟时):
|
|||
|
|
联邦Gitea · 每个成员自己服务器跑Gitea
|
|||
|
|
→ 代码完全不经过主服务器
|
|||
|
|
→ 主服务器退化为纯算力中心
|
|||
|
|
→ 触发条件: 成员技术能力成长 + 存储压力超80%
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 十、风险备忘
|
|||
|
|
|
|||
|
|
1. **Push Mirror依赖网络** — 成员个人服务器断网时镜像会失败 · Gitea会重试但需要监控
|
|||
|
|
2. **tmpfs 4G上限** — 如果单次构建产物超过4G会失败 · 需要优化构建流程或调大tmpfs
|
|||
|
|
3. **Runner并发=1** — 多人同时push时构建会排队 · 高峰期可能慢 · 未来可考虑备用机分担
|
|||
|
|
4. **成员个人服务器SSH安全** — 主服务器持有访问个人服务器的密钥 · 密钥管理要规范
|
|||
|
|
5. **Gitea Push Mirror配额** — 免费版无限制 · 但大量仓库同时镜像时可能有性能影响
|