Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 98f1479d by DigHuang Committed by GitHub
Options

fix(script): skipError option to skip missing sources in legacy migration (#7417)

parent c769a7c4
Showing with 154 additions and 22 deletions
.agents/design/core/ai/sandbox/user-level-sandbox.md
......@@ -223,6 +223,12 @@ Legacy 预检使用独立的 `LegacySandboxInstanceZodSchema`，不得使用 v2
`stopped`；暂停失败不得提交迁移完成。
7. 发布后把 Legacy 阶段提交为 `completed`，保留旧 S3 和 Legacy Mongo 记录作为迁移备份。
管理员入口支持可选 `skipError=true`，用于兼容业务 source 已不存在或已软删除、但 Legacy Sandbox
记录仍残留的升级场景。该开关只跳过 source fence 返回 `Sandbox source is missing or deleted` 的
整个 source 分组，被跳过的记录不执行归档、资源删除、目标创建或安装，并通过 `skippedCount/skipped`
单独返回。对象存储、Provider、Lease、归档和安装等其他错误仍计入 `failedCount/failures` 并维持
全局归档屏障；省略该参数或传 `false` 时保持原有严格行为。
第一阶段释放单个 Source Lease 后，正常用户请求可以先创建确定性的 v2 目标。第二阶段必须接管或
复用该目标，并按“目标内容优先”规则合并，不能覆盖已经产生的用户文件。
......@@ -364,3 +370,4 @@ interface -> application -> infrastructure -> sandbox-adapter
- [x] 将阶段性技术方案合并到本文并删除重复文档。
- [x] 将 beta6 Sandbox 归一化移入 `initUserSandbox` 并在剩余待处理数归零前阻断归档。
- [x] restore 后保留 v2 S3 归档，仅在业务资源删除时清理。
- [x] 为 `initUserSandbox` 增加可选 `skipError`，仅跳过 source 已缺失的 Legacy Sandbox 分组并返回跳过明细。
document/content/self-host/upgrading/4-16/41601.en.mdx
......@@ -69,9 +69,20 @@ curl -X POST 'https://你的域名/api/admin/4160/initUserSandbox' \
-d '{"dryRun":false}'
```
If every item in `failures` reports `Sandbox source is missing or deleted`, and you have confirmed that the corresponding Apps or Skills no longer exist, you can explicitly skip those stale Sandboxes:
```bash
curl -X POST 'https://your-domain/api/admin/4160/initUserSandbox' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":false,"skipError":true}'
```
`skipError` defaults to `false`, so omitting it preserves strict migration behavior. The switch only skips an entire source group when that source is missing or soft-deleted. Sandboxes in the group are not archived, deleted, or migrated, and are reported through `skippedCount` and `skipped`. Archive, object storage, provider, concurrency-control, and all other errors remain blocking.
The migration first runs all V4.15.0-beta6 normalization steps. It fills in `sourceType/sourceId` for legacy Sandboxes, removes obsolete fields, deletes orphaned resources that cannot be associated, and cleans up the three legacy Skill Debug Chat collections and old private/public Bucket prefixes when `sourceType` is missing. After recounting, the two categories are combined into `normalization.pendingCount`; Workspace archiving does not begin while the count is non-zero. Once normalization is complete, all legacy Workspaces are archived, old compute resources are cleaned up, Skills are migrated, and records are aggregated into user-level Sandboxes by App and user. Installation does not start if any archive operation fails. New Sandboxes are suspended after Workspace installation and start normally on first use. The script is safe to retry: completed archive and migration operations are not repeated. Old archives and MongoDB records are retained as backups after migration.
Check `normalization.pendingCount`, `normalizationBlocked`, `failedCount`, and `failures` in the response. The migration is complete only when `normalization.pendingCount` and `failedCount` are both `0` and `normalizationBlocked` is `false`.
Check `normalization.pendingCount`, `normalizationBlocked`, `failedCount`, `failures`, `skippedCount`, and `skipped` in the response. When both `normalization.pendingCount` and `failedCount` are `0` and `normalizationBlocked` is `false`, every non-skipped Sandbox has been migrated. Legacy records listed in `skipped` remain in place and are not migrated.
### 4. Migrate manual HTTP tool data
......
document/content/self-host/upgrading/4-16/41601.mdx
......@@ -69,9 +69,20 @@ curl -X POST 'https://你的域名/api/admin/4160/initUserSandbox' \
-d '{"dryRun":false}'
```
如果 `failures` 中仅包含 `Sandbox source is missing or deleted`，并且已确认对应 App 或 Skill 确实不再存在，可以显式跳过这些残留 Sandbox：
```bash
curl -X POST 'https://你的域名/api/admin/4160/initUserSandbox' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":false,"skipError":true}'
```
`skipError` 默认为 `false`，省略时保持严格迁移。该开关只跳过 source 已缺失或已软删除的整个分组，不会归档、删除或迁移其中的 Sandbox；跳过明细通过 `skippedCount` 和 `skipped` 返回。归档、对象存储、Provider 和并发控制等其他错误仍会阻断迁移。
迁移会先执行 V4.15.0-beta6 的完整前置逻辑：补齐旧 Sandbox 的 `sourceType/sourceId`、清理遗留字段、删除无法归属的孤立资源，并清理缺失 `sourceType` 的旧 Skill Debug Chat 三表数据及私有、公开 Bucket 旧前缀。与 App 同 ID 的 Skill 会跳过 Chat 清理。两类数据重新统计后合计为 `normalization.pendingCount`；数量不为 `0` 时不会进入 Workspace 归档。归零后直接归档全部旧 Workspace 并清理旧计算资源，再迁移 Skill，最后按 App、用户聚合到用户级 Sandbox。只要归档阶段存在失败，安装阶段就不会开始。新的 Sandbox 会在 Workspace 安装完成后暂停，首次使用时再按正常流程启动。脚本可安全重试，已完成的归档和迁移不会重复执行；迁移完成后会保留旧归档和旧 MongoDB 记录作为备份。
请检查返回结果中的 `normalization.pendingCount`、`normalizationBlocked`、`failedCount` 和 `failures`。只有 `normalization.pendingCount` 和 `failedCount` 均为 `0`，且 `normalizationBlocked` 为 `false` 时，才表示本次迁移全部完成。
请检查返回结果中的 `normalization.pendingCount`、`normalizationBlocked`、`failedCount`、`failures`、`skippedCount` 和 `skipped`。只有 `normalization.pendingCount` 和 `failedCount` 均为 `0`，且 `normalizationBlocked` 为 `false` 时，才表示所有未跳过的 Sandbox 迁移完成；`skipped` 中的 Legacy 记录会保留且不会迁移。
### 4. 迁移手动 HTTP 工具数据
......
packages/service/core/ai/sandbox/application/legacyMigration/service.ts
......@@ -32,7 +32,7 @@ import {
withSandboxSourceMutationLease
} from '../lease';
import { stopSandboxResource } from '../resource';
import { assertSandboxSourceActive } from '../sourceGuard';
import { assertSandboxSourceActive, SandboxSourceMissingError } from '../sourceGuard';
import { archiveLegacyInstanceBeforeMigration, LegacySandboxCleanupError } from './cleanup';
import type {
LegacyMigrationPhase,
......@@ -83,9 +83,11 @@ async function archiveAllLegacySandboxes(params: {
docs: LegacySandboxInstanceSchemaType[];
result: UserSandboxMigrationResult;
runId: string;
skipError: boolean;
}) {
const { docs, result, runId } = params;
const { docs, result, runId, skipError } = params;
const failedIds = new Set<string>();
const skippedIds = new Set<string>();
const groups = new Map<string, LegacySandboxInstanceSchemaType[]>();
for (const doc of docs) {
if (getLegacyMigrationPhase(doc) === 'completed') continue;
......@@ -136,6 +138,13 @@ async function archiveAllLegacySandboxes(params: {
}
});
} catch (error) {
if (skipError && error instanceof SandboxSourceMissingError) {
for (const doc of group) {
skippedIds.add(String(doc._id));
result.skipped.push({ sandboxId: doc.sandboxId, error: getErrText(error) });
}
return;
}
await Promise.all(group.map((doc) => recordFailure(doc, error)));
}
};
......@@ -150,7 +159,7 @@ async function archiveAllLegacySandboxes(params: {
}
await Promise.all(skillGroups.map((group) => skillLimit(() => archiveGroup(group))));
await Promise.all(appGroups.map((group) => appLimit(() => archiveGroup(group))));
return failedIds.size === 0;
return { archiveCompleted: failedIds.size === 0, skippedIds };
}
/** 把单条 Legacy Skill workspace 发布到确定性 v2 Sandbox。 */
......@@ -340,8 +349,9 @@ const migrateAppGroup = async (params: {
group: ResolvedLegacyApp[];
result: UserSandboxMigrationResult;
runId: string;
skipError: boolean;
}) => {
const { group, result, runId } = params;
const { group, result, runId, skipError } = params;
const first = group[0];
const targetSandboxId = getLegacyMigrationTargetSandboxId(first.doc);
......@@ -540,6 +550,12 @@ const migrateAppGroup = async (params: {
}
}).catch(async (error) => {
const errorText = getErrText(error);
if (skipError && error instanceof SandboxSourceMissingError) {
for (const item of group) {
result.skipped.push({ sandboxId: item.doc.sandboxId, error: errorText });
}
return;
}
result.failures.push({ sandboxId: first.doc.sandboxId, error: errorText });
await recordMigrationTrack({
runId,
......@@ -559,6 +575,7 @@ const runLegacySandboxMigration = async (
assertLeaseValid?: () => void
): Promise<UserSandboxMigrationResult> => {
const dryRun = params.dryRun ?? true;
const skipError = params.skipError ?? false;
const startedAt = Date.now();
const normalization = await normalizeLegacySandboxes({ dryRun, assertLeaseValid });
if (normalization.pendingCount > 0) {
......@@ -574,7 +591,9 @@ const runLegacySandboxMigration = async (
appGroupCount: 0,
completedAppGroupCount: 0,
failedCount: normalization.failures.length,
failures: normalization.failures
failures: normalization.failures,
skippedCount: 0,
skipped: []
};
}
......@@ -635,15 +654,23 @@ const runLegacySandboxMigration = async (
appGroupCount: appGroups.size,
completedAppGroupCount: 0,
failedCount: 0,
failures: []
failures: [],
skippedCount: 0,
skipped: []
};
await recordMigrationTrack({ runId, phase: 'started', dryRun });
if (!dryRun) {
const archiveCompleted = await archiveAllLegacySandboxes({ docs, result, runId });
const { archiveCompleted, skippedIds } = await archiveAllLegacySandboxes({
docs,
result,
runId,
skipError
});
if (archiveCompleted) {
const skillLimit = pLimit(SKILL_SANDBOX_MIGRATION_CONCURRENCY);
const skillGroups = new Map<string, ResolvedLegacySkill[]>();
for (const item of skillItems) {
if (skippedIds.has(String(item.doc._id))) continue;
skillGroups.set(item.sourceId, [...(skillGroups.get(item.sourceId) ?? []), item]);
}
await Promise.all(
......@@ -655,6 +682,10 @@ const runLegacySandboxMigration = async (
result.migratedSkillCount += 1;
} catch (error) {
const errorText = getErrText(error);
if (skipError && error instanceof SandboxSourceMissingError) {
result.skipped.push({ sandboxId: item.doc.sandboxId, error: errorText });
continue;
}
result.failures.push({ sandboxId: item.doc.sandboxId, error: errorText });
await recordMigrationTrack({
runId,
......@@ -672,6 +703,7 @@ const runLegacySandboxMigration = async (
const appSourceGroups = new Map<string, ResolvedLegacyApp[][]>();
for (const group of appGroups.values()) {
if (group.some((item) => skippedIds.has(String(item.doc._id)))) continue;
const sourceId = group[0]?.sourceId;
if (!sourceId) continue;
appSourceGroups.set(sourceId, [...(appSourceGroups.get(sourceId) ?? []), group]);
......@@ -680,13 +712,16 @@ const runLegacySandboxMigration = async (
await Promise.all(
Array.from(appSourceGroups.values(), (groups) =>
appLimit(async () => {
for (const group of groups) await migrateAppGroup({ group, result, runId });
for (const group of groups) {
await migrateAppGroup({ group, result, runId, skipError });
}
})
)
);
}
}
result.failedCount = result.failures.length;
result.skippedCount = result.skipped.length;
await recordMigrationTrack({
runId,
phase: 'completed',
......
packages/service/core/ai/sandbox/application/legacyMigration/types.ts
......@@ -3,7 +3,10 @@ import type { LegacySandboxInstanceSchemaType } from '../../infrastructure/insta
import type { VolumeManagerResult } from '../../infrastructure/volume/service';
import type { SandboxRuntimePaths } from '../../utils';
export type UserSandboxMigrationParams = { dryRun?: boolean };
export type UserSandboxMigrationParams = {
dryRun?: boolean;
skipError?: boolean;
};
export type LegacyMigrationPhase =
| 'pending'
......@@ -90,4 +93,6 @@ export type UserSandboxMigrationResult = {
completedAppGroupCount: number;
failedCount: number;
failures: UserSandboxMigrationFailure[];
skippedCount: number;
skipped: UserSandboxMigrationFailure[];
};
packages/service/core/ai/sandbox/application/sourceGuard.ts
......@@ -3,6 +3,14 @@ import { ChatSourceTypeEnum } from '@fastgpt/global/core/chat/constants';
import { MongoApp } from '../../../app/schema';
import { MongoAgentSkills } from '../../skill/model/schema';
/** 业务 source 不存在或已软删除；调用方可据此决定是否显式跳过遗留资源。 */
export class SandboxSourceMissingError extends Error {
constructor(params: { sourceType: ChatSourceTypeEnum; sourceId: string }) {
super(`Sandbox source is missing or deleted: ${params.sourceType}/${params.sourceId}`);
this.name = 'SandboxSourceMissingError';
}
}
/** source 不存在或已经设置 deleteTime 时禁止创建、恢复或迁移 Sandbox。 */
export async function assertSandboxSourceActive(params: {
sourceType: ChatSourceTypeEnum;
......@@ -19,9 +27,7 @@ export async function assertSandboxSourceActive(params: {
})();
if (!active) {
throw new Error(
`Sandbox source is missing or deleted: ${params.sourceType}/${params.sourceId}`
);
throw new SandboxSourceMissingError(params);
}
}
......
packages/service/test/core/ai/sandbox/application/legacyMigration/service.test.ts
......@@ -118,7 +118,8 @@ vi.mock('@fastgpt/service/core/ai/sandbox/application/lease', () => ({
withLegacySandboxMigrationJobLease: mocks.withLegacySandboxMigrationJobLease
}));
vi.mock('@fastgpt/service/core/ai/sandbox/application/sourceGuard', () => ({
vi.mock('@fastgpt/service/core/ai/sandbox/application/sourceGuard', async (importOriginal) => ({
...(await importOriginal()),
assertSandboxSourceActive: mocks.assertSandboxSourceActive,
assertSandboxSourceDeleted: mocks.assertSandboxSourceDeleted
}));
......@@ -130,6 +131,7 @@ vi.mock('@fastgpt/service/core/ai/sandbox/application/legacyMigration/debugChatC
import { deleteAppSandboxesForAppDeletion } from '@fastgpt/service/core/ai/sandbox/application/legacyMigration/cleanup';
import { migrateLegacySandboxesToUserLevel } from '@fastgpt/service/core/ai/sandbox/application/legacyMigration/service';
import { installLegacyWorkspaceArchive } from '@fastgpt/service/core/ai/sandbox/application/legacyMigration/workspace';
import { SandboxSourceMissingError } from '@fastgpt/service/core/ai/sandbox/application/sourceGuard';
import { MongoLegacySandboxInstance } from '@fastgpt/service/core/ai/sandbox/infrastructure/instance/legacySchema';
const createTargetProvider = () => ({
......@@ -415,7 +417,48 @@ describe('legacy sandbox migration', () => {
expect(mocks.claimAppSandboxMigrationTarget).not.toHaveBeenCalled();
});
it('archives every Legacy source and blocks all v2 targets when phase one has a failure', async () => {
it('skips missing sources and continues migrating active sources when skipError is enabled', async () => {
const missingSourceId = 'app-missing-source-skipped';
const activeSourceId = 'app-active-source';
await insertLegacyApp({
sandboxId: 'migration-test-missing-source-skipped',
sourceId: missingSourceId
});
await insertLegacyApp({
sandboxId: 'migration-test-active-source',
sourceId: activeSourceId
});
mocks.assertSandboxSourceActive.mockImplementation(async ({ sourceId, sourceType }: any) => {
if (sourceId === missingSourceId) {
throw new SandboxSourceMissingError({ sourceType, sourceId });
}
});
const result = await migrateLegacySandboxesToUserLevel({
dryRun: false,
skipError: true
});
expect(result).toMatchObject({
migratedAppCount: 1,
failedCount: 0,
failures: [],
skippedCount: 1,
skipped: [
{
sandboxId: 'migration-test-missing-source-skipped',
error: `Sandbox source is missing or deleted: app/${missingSourceId}`
}
]
});
expect(mocks.getSandboxWorkspaceArchiveForMigration).toHaveBeenCalledTimes(1);
expect(mocks.claimAppSandboxMigrationTarget).toHaveBeenCalledTimes(1);
expect(mocks.claimAppSandboxMigrationTarget).toHaveBeenCalledWith(
expect.objectContaining({ sourceId: activeSourceId })
);
});
it('keeps non-source archive errors blocking when skipError is enabled', async () => {
await insertLegacyApp({
sandboxId: 'migration-test-archive-success',
sourceId: 'app-archive-success'
......@@ -433,7 +476,10 @@ describe('legacy sandbox migration', () => {
}
);
const result = await migrateLegacySandboxesToUserLevel({ dryRun: false });
const result = await migrateLegacySandboxesToUserLevel({
dryRun: false,
skipError: true
});
expect(result).toMatchObject({
migratedAppCount: 0,
......
projects/app/src/pages/api/admin/4160/initUserSandbox.ts
......@@ -7,7 +7,8 @@ import { migrateLegacySandboxesToUserLevel } from '@fastgpt/service/core/ai/sand
import z from 'zod';
const InitUserSandboxBodySchema = z.object({
dryRun: BoolSchema.optional().default(true)
dryRun: BoolSchema.optional().default(true),
skipError: BoolSchema.optional().default(false)
});
const InitUserSandboxResponseSchema = z.object({
......@@ -63,6 +64,13 @@ const InitUserSandboxResponseSchema = z.object({
sandboxId: z.string(),
error: z.string()
})
),
skippedCount: z.number().int().nonnegative(),
skipped: z.array(
z.object({
sandboxId: z.string(),
error: z.string()
})
)
});
type InitUserSandboxResponse = z.infer<typeof InitUserSandboxResponseSchema>;
......@@ -70,12 +78,12 @@ type InitUserSandboxResponse = z.infer<typeof InitUserSandboxResponseSchema>;
/** 管理员升级入口；默认 dry-run，真实执行时先完成 beta6 清理再迁移 Legacy Sandbox。 */
async function handler(req: ApiRequestProps): Promise<InitUserSandboxResponse> {
await authCert({ req, authRoot: true });
const { dryRun } = parseApiInput({
const { dryRun, skipError } = parseApiInput({
req,
bodySchema: InitUserSandboxBodySchema
}).body;
const result = await migrateLegacySandboxesToUserLevel({ dryRun });
const result = await migrateLegacySandboxesToUserLevel({ dryRun, skipError });
return InitUserSandboxResponseSchema.parse(result);
}
......
projects/app/test/pages/api/admin/4160/initUserSandbox.test.ts
......@@ -58,7 +58,9 @@ describe('initUserSandbox API', () => {
appGroupCount: 1,
completedAppGroupCount: 0,
failedCount: 0,
failures: []
failures: [],
skippedCount: 0,
skipped: []
});
});
......@@ -68,7 +70,8 @@ describe('initUserSandbox API', () => {
expect(mocks.authCert).toHaveBeenCalledWith({ req, authRoot: true });
expect(mocks.migrateLegacySandboxesToUserLevel).toHaveBeenCalledWith({
dryRun: true
dryRun: true,
skipError: false
});
});
});
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment