Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 402ad3ff by Archer Committed by GitHub
Options

perf: init workflow (#7210) 

* perf: init workflow

* doc

* perf: parse

* fix: ts
parent 9981049b
Showing with 4624 additions and 706 deletions
document/content/self-host/upgrading/4-15/41500.mdx
...@@ -78,14 +78,14 @@ SANDBOX_QUEUE_ID_CONCURRENCY= ...@@ -78,14 +78,14 @@ SANDBOX_QUEUE_ID_CONCURRENCY=
1. 下载所有系统工具的 [zip 包](<https://github.com/labring/fastgpt-img/raw/refs/heads/main/fastgpt-official-plugins(1).zip>)。 1. 下载所有系统工具的 [zip 包](<https://github.com/labring/fastgpt-img/raw/refs/heads/main/fastgpt-official-plugins(1).zip>)。
2. 打开 `fastgpt` 网页 - 点击 `管理员` navbar - 点击添加插件 - 点击 `导入/更新插件` - 上传 zip - 确认。 2. 打开 `fastgpt` 网页 - 点击 `管理员` navbar - 点击添加插件 - 点击 `导入/更新插件` - 上传 zip - 确认。
也可以打开插件市场逐个下载，正式版之前，插件市场地址为: [https://v2.marketplace.fastgpt.cn](https://v2.marketplace.fastgpt.cn) 也可以打开插件市场逐个下载，插件市场地址为: [https://v2.marketplace.fastgpt.cn](https://v2.marketplace.fastgpt.cn)，环境变量默认值已变成该地址，不设置环境变量即可。
### 4. 升级脚本 ### 4. 升级脚本
如使用旧版沙盒 workspace，可通过下面接口将旧沙盒 workspace 归档到 S3，从而更彻底地释放不活跃沙盒。该脚本仅影响旧的沙盒，不影响新生成的沙盒；不执行该脚本，直接移除旧沙盒也可以。 如使用旧版沙盒 workspace，可通过下面接口将旧沙盒 workspace 归档到 S3，从而更彻底地释放不活跃沙盒。该脚本仅影响旧的沙盒，不影响新生成的沙盒；不执行该脚本，直接移除旧沙盒也可以。
```shell ```shell
curl --location --request POST 'https://{{host}}/api/admin/initSandboxArchive' \ curl --location --request POST 'https://{{host}}/api/admin/dataClean/initSandboxArchive' \
--header 'rootkey: {{rootkey}}' \ --header 'rootkey: {{rootkey}}' \
--header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \
-d '{"runArchive":true,"inactiveDays":0}' -d '{"runArchive":true,"inactiveDays":0}'
......
document/content/self-host/upgrading/4-15/41507.en.mdx 0 → 100644
---
title: 'V4.15.0-beta7'
description: 'FastGPT V4.15.0-beta7 Release Notes'
---
## 📦 Upgrade Guide
### 1. Run the Workflow V1 to V2 Migration
Starting from V4.15.0-beta7, workflow save payloads use the V2 structure consistently. Historical `apps.modules` and `app_versions.nodes` records may still use the V1 structure. Run the V1 -> V2 migration first, then run the V2 dirty-data cleanup in the next step.
Migration script path: `projects/app/src/pages/api/admin/dataClean/v1WorkflowToV2.ts`. This endpoint is only for this upgrade migration and is not a public OpenAPI endpoint.
The endpoint uses dry-run mode by default. It scans, converts in memory, and validates with `PublishAppBodySchema` without writing to MongoDB:
```bash
curl -X POST 'https://your-domain/api/admin/dataClean/v1WorkflowToV2' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":true}'
```
After confirming the returned statistics, set `dryRun` to `false` to write the converted data:
```bash
curl -X POST 'https://your-domain/api/admin/dataClean/v1WorkflowToV2' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":false}'
```
Request parameters:
| Parameter | Type | Default | Description |
| --------- | ------- | ------- | ---------------------------------------------------------- |
| `dryRun` | boolean | `true` | Whether to scan and validate only without writing changes. |
Migration behavior:
1. Scans apps where `apps.version != 'v2'` and `type` is not `folder`, `httpPlugin`, or `toolFolder`.
2. For each batch of `apps`, converts and writes related `app_versions` first, then converts and writes `apps`, so a rerun will not miss historical versions after an interruption.
3. Converts V1 node fields to V2 fields, for example `moduleId` -> `nodeId` and `flowType` -> `flowNodeType`.
4. Unknown node types fall back to `emptyNode`, and invalid `valueType` values are converted to `any`.
5. Missing `node.name` falls back to `flowType`, and missing `input.label` falls back to `input.key`.
6. Before writing, the script validates `nodes`, `edges`, and `chatConfig` with `PublishAppBodySchema`. Documents that fail validation are not written and are included in the endpoint response.
### 2. Run the Workflow V2 Enum and Structure Cleanup
Some historical workflow nodes may have stored TypeScript enum expression strings directly in MongoDB, for example:
```json
{
"renderTypeList": ["FlowNodeInputTypeEnum.hidden"],
"valueType": "WorkflowIOValueTypeEnum.any"
}
```
The correct stored values are:
```json
{
"renderTypeList": ["hidden"],
"valueType": "any"
}
```
This dirty data can affect workflow node input rendering and IO type checks. After running the V1 -> V2 migration, continue with the V2 cleanup script to scan and fix `apps.modules` and `app_versions.nodes`.
Migration script path: `projects/app/src/pages/api/admin/dataClean/initWorkflowData.ts`. This endpoint is only for this upgrade migration and is not a public OpenAPI endpoint.
The endpoint uses dry-run mode by default. It formats data in memory and validates with `PublishAppBodySchema` without writing to MongoDB:
```bash
curl -X POST 'https://your-domain/api/admin/dataClean/initWorkflowData' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":true,"batchSize":1000,"writeBatchSize":10}'
```
After confirming the returned statistics, set `dryRun` to `false` to write the cleanup:
```bash
curl -X POST 'https://your-domain/api/admin/dataClean/initWorkflowData' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":false,"batchSize":1000,"writeBatchSize":10}'
```
Request parameters:
| Parameter | Type | Default | Description |
| ---------------- | ------- | ------- | ------------------------------------------------------------------------------- |
| `dryRun` | boolean | `true` | Whether to scan and validate only without writing changes. |
| `batchSize` | number | `1000` | Documents fetched per page. |
| `writeBatchSize` | number | `10` | Documents written per `bulkWrite`. Lower it when online write pressure is high. |
Cleanup behavior:
1. Scans workflow data in `apps` and `app_versions` in batches to reduce read and write pressure.
2. Formats each workflow document once, covering historical dirty fields, null values, enum expressions, and legacy structure compatibility.
3. After formatting, validates the save payload fields `nodes`, `edges`, and `chatConfig` with `PublishAppBodySchema`.
4. Documents that fail Zod validation are only recorded in the response and are not written to MongoDB.
5. In non-dry-run mode, only documents that changed during formatting and passed Zod validation are written. Unchanged documents are not written again.
The response includes separate statistics for `apps`, `appVersions`, and `total`, including scanned documents, fixable documents, Zod error count, successful writes, failed writes, enum expression statistics, format samples, and error samples.
### 3. Update Images
- Update the fastgpt-app (FastGPT main service) image tag to v4.15.0-beta7.
- Update the fastgpt-pro (FastGPT commercial edition) image tag to v4.15.0-beta7.
- Update the fastgpt-plugin image tag to v1.0.0-beta7.
## 🐛 Fixes
1. Fixed historical V1 workflow data that could fail validation under the new save payload structure.
2. Fixed dirty `FlowNodeInputTypeEnum.*`, `FlowNodeOutputTypeEnum.*`, and `WorkflowIOValueTypeEnum.*` expression strings in workflow nodes that could break input rendering and IO type checks.
document/content/self-host/upgrading/4-15/41507.mdx 0 → 100644
---
title: 'V4.15.0-beta7'
description: 'FastGPT V4.15.0-beta7 更新说明'
---
## 📦 升级指南
### 1. 执行工作流 V1 升级 V2 迁移（可选）
该步骤仅需部署过 `<4.8` 版本 FastGPT 的用户执行。
V4.15.0-beta7 后工作流保存结构统一使用 V2。历史 `apps.modules` 与 `app_versions.nodes` 中可能仍存在 V1 结构，升级后建议先执行 V1 -> V2 迁移，再执行后续 V2 脏数据清洗。
迁移脚本位置：`projects/app/src/pages/api/admin/dataClean/v1WorkflowToV2.ts`。该接口仅用于本次升级迁移，不作为 OpenAPI 对外接口。
接口默认 dry-run，只扫描、转换和执行 `PublishAppBodySchema` 校验，不写库：
```bash
curl -X POST 'https://你的域名/api/admin/dataClean/v1WorkflowToV2' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":true}'
```
确认返回统计无误后，将 `dryRun` 改为 `false` 执行写入：
```bash
curl -X POST 'https://你的域名/api/admin/dataClean/v1WorkflowToV2' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":false}'
```
接口参数：
| 参数 | 类型 | 默认值 | 说明 |
| -------- | ------- | ------ | ---------------------- |
| `dryRun` | boolean | `true` | 是否只扫描验证不写库。 |
迁移逻辑：
1. 按 `apps.version != 'v2'` 且 `type` 非 `folder`、`httpPlugin`、`toolFolder` 扫描应用。
2. 对每批 `apps`，先转换并写入对应 `app_versions`，再转换并写入 `apps`，避免中断后遗漏历史版本。
3. 将 V1 节点字段升级为 V2 节点字段，例如 `moduleId` -> `nodeId`、`flowType` -> `flowNodeType`。
4. 未知节点类型会兜底为 `emptyNode`，非法 `valueType` 会转为 `any`。
5. 缺失 `node.name` 时用 `flowType` 兜底，缺失 `input.label` 时用 `input.key` 兜底。
6. 写库前使用 `PublishAppBodySchema` 校验 `nodes`、`edges`、`chatConfig`，校验失败的文档不会写入，并会记录到接口返回结果。
### 2. 执行工作流 V2 枚举与结构脏数据清洗
部分历史工作流节点可能把 TypeScript 枚举表达式字符串直接写入 MongoDB，例如：
```json
{
"renderTypeList": ["FlowNodeInputTypeEnum.hidden"],
"valueType": "WorkflowIOValueTypeEnum.any"
}
```
正确落库值应为：
```json
{
"renderTypeList": ["hidden"],
"valueType": "any"
}
```
该脏数据会影响工作流节点输入渲染和 IO 类型判断。执行 V1 -> V2 迁移后，继续执行 V2 清洗脚本，扫描并修复 `apps.modules` 与 `app_versions.nodes`。
迁移脚本位置：`projects/app/src/pages/api/admin/dataClean/initWorkflowData.ts`。该接口仅用于本次升级迁移，不作为 OpenAPI 对外接口。
接口默认 dry-run，只格式化内存数据并执行 `PublishAppBodySchema` 校验，不写库：
```bash
curl -X POST 'https://你的域名/api/admin/dataClean/initWorkflowData' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":true,"batchSize":1000,"writeBatchSize":10}'
```
确认返回统计无误后，将 `dryRun` 改为 `false` 执行写入：
```bash
curl -X POST 'https://你的域名/api/admin/dataClean/initWorkflowData' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":false,"batchSize":1000,"writeBatchSize":10}'
```
接口参数：
| 参数 | 类型 | 默认值 | 说明 |
| ---------------- | ------- | ------ | ----------------------------------------------------- |
| `dryRun` | boolean | `true` | 是否只扫描验证不写库。 |
| `batchSize` | number | `1000` | 每批读取文档数量。 |
| `writeBatchSize` | number | `10` | 每次 `bulkWrite` 的文档数量。线上写入压力大时可调小。 |
清洗逻辑：
1. 按批扫描 `apps` 和 `app_versions` 中的工作流数据，降低单次读取和写入压力。
2. 对每条工作流数据执行一次格式化，统一修复历史脏字段、空值、枚举表达式和旧结构兼容问题。
3. 格式化后使用 `PublishAppBodySchema` 校验保存接口实际关心的 `nodes`、`edges`、`chatConfig`。
4. Zod 校验失败的文档只记录在返回结果中，不会写入数据库。
5. 非 dry-run 时，只写入“发生过格式化变更，且 Zod 校验通过”的文档；未变化文档不会重复写库。
返回结果会分别展示 `apps`、`appVersions` 和 `total` 的统计，包括扫描文档数、可修复文档数、Zod 错误数量、写入成功数量、写入失败数量、枚举表达式统计、变更样本和错误样本。
### 3. 更新镜像
- 更新 fastgpt-app(fastgpt 主服务) 镜像 tag: v4.15.0-beta7
- 更新 fastgpt-pro(fastgpt 商业版) 镜像 tag: v4.15.0-beta7
- 更新 fastgpt-plugin 镜像 tag: v1.0.0-beta7
## 🐛 修复
1. 修复历史 V1 工作流数据在新版保存结构下无法通过校验的问题。
2. 修复工作流节点配置中 `FlowNodeInputTypeEnum.*`、`FlowNodeOutputTypeEnum.*` 和 `WorkflowIOValueTypeEnum.*` 枚举表达式字符串脏数据导致输入渲染和 IO 类型判断异常的问题。
3. AgentV2 mcp 拿不到 schema。
document/content/self-host/upgrading/4-15/meta.en.json
{ {
"title": "4.15.x", "title": "4.15.x",
"description": "", "description": "",
"pages": ["41500", "41506", "41505", "41504", "41503", "41502", "41501"] "pages": ["41500", "41507", "41506", "41505", "41504", "41503", "41502", "41501"]
} }
document/content/self-host/upgrading/4-15/meta.json
{ {
"title": "4.15.x", "title": "4.15.x",
"description": "", "description": "",
"pages": ["41500", "41506", "41505", "41504", "41503", "41502", "41501"] "pages": ["41500", "41507", "41506", "41505", "41504", "41503", "41502", "41501"]
} }
document/content/toc.en.mdx
...@@ -154,6 +154,7 @@ description: FastGPT Toc ...@@ -154,6 +154,7 @@ description: FastGPT Toc
- [/en/self-host/upgrading/4-15/41504](/en/self-host/upgrading/4-15/41504) - [/en/self-host/upgrading/4-15/41504](/en/self-host/upgrading/4-15/41504)
- [/en/self-host/upgrading/4-15/41505](/en/self-host/upgrading/4-15/41505) - [/en/self-host/upgrading/4-15/41505](/en/self-host/upgrading/4-15/41505)
- [/en/self-host/upgrading/4-15/41506](/en/self-host/upgrading/4-15/41506) - [/en/self-host/upgrading/4-15/41506](/en/self-host/upgrading/4-15/41506)
- [/en/self-host/upgrading/4-15/41507](/en/self-host/upgrading/4-15/41507)
- [/en/self-host/upgrading/outdated/40](/en/self-host/upgrading/outdated/40) - [/en/self-host/upgrading/outdated/40](/en/self-host/upgrading/outdated/40)
- [/en/self-host/upgrading/outdated/41](/en/self-host/upgrading/outdated/41) - [/en/self-host/upgrading/outdated/41](/en/self-host/upgrading/outdated/41)
- [/en/self-host/upgrading/outdated/4100](/en/self-host/upgrading/outdated/4100) - [/en/self-host/upgrading/outdated/4100](/en/self-host/upgrading/outdated/4100)
......
document/content/toc.mdx
...@@ -157,6 +157,7 @@ description: FastGPT 文档目录 ...@@ -157,6 +157,7 @@ description: FastGPT 文档目录
- [/self-host/upgrading/4-15/41504](/self-host/upgrading/4-15/41504) - [/self-host/upgrading/4-15/41504](/self-host/upgrading/4-15/41504)
- [/self-host/upgrading/4-15/41505](/self-host/upgrading/4-15/41505) - [/self-host/upgrading/4-15/41505](/self-host/upgrading/4-15/41505)
- [/self-host/upgrading/4-15/41506](/self-host/upgrading/4-15/41506) - [/self-host/upgrading/4-15/41506](/self-host/upgrading/4-15/41506)
- [/self-host/upgrading/4-15/41507](/self-host/upgrading/4-15/41507)
- [/self-host/upgrading/outdated/40](/self-host/upgrading/outdated/40) - [/self-host/upgrading/outdated/40](/self-host/upgrading/outdated/40)
- [/self-host/upgrading/outdated/41](/self-host/upgrading/outdated/41) - [/self-host/upgrading/outdated/41](/self-host/upgrading/outdated/41)
- [/self-host/upgrading/outdated/4100](/self-host/upgrading/outdated/4100) - [/self-host/upgrading/outdated/4100](/self-host/upgrading/outdated/4100)
......
document/data/doc-last-modified.json
...@@ -294,7 +294,7 @@ ...@@ -294,7 +294,7 @@
"content/self-host/upgrading/4-14/4149.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/4-14/4149.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4149.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/4-14/4149.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-15/41500.en.mdx": "2026-06-29T00:47:43+08:00", "content/self-host/upgrading/4-15/41500.en.mdx": "2026-06-29T00:47:43+08:00",
"content/self-host/upgrading/4-15/41500.mdx": "2026-06-29T00:47:43+08:00", "content/self-host/upgrading/4-15/41500.mdx": "2026-06-29T21:48:35+08:00",
"content/self-host/upgrading/4-15/41501.mdx": "2026-06-23T21:09:39+08:00", "content/self-host/upgrading/4-15/41501.mdx": "2026-06-23T21:09:39+08:00",
"content/self-host/upgrading/4-15/41502.en.mdx": "2026-05-25T11:21:30+08:00", "content/self-host/upgrading/4-15/41502.en.mdx": "2026-05-25T11:21:30+08:00",
"content/self-host/upgrading/4-15/41502.mdx": "2026-06-23T13:54:06+08:00", "content/self-host/upgrading/4-15/41502.mdx": "2026-06-23T13:54:06+08:00",
...@@ -302,8 +302,8 @@ ...@@ -302,8 +302,8 @@
"content/self-host/upgrading/4-15/41503.mdx": "2026-05-28T16:21:09+08:00", "content/self-host/upgrading/4-15/41503.mdx": "2026-05-28T16:21:09+08:00",
"content/self-host/upgrading/4-15/41504.en.mdx": "2026-06-10T19:02:59+08:00", "content/self-host/upgrading/4-15/41504.en.mdx": "2026-06-10T19:02:59+08:00",
"content/self-host/upgrading/4-15/41504.mdx": "2026-06-15T23:34:43+08:00", "content/self-host/upgrading/4-15/41504.mdx": "2026-06-15T23:34:43+08:00",
"content/self-host/upgrading/4-15/41505.en.mdx": "2026-06-29T00:47:43+08:00", "content/self-host/upgrading/4-15/41505.en.mdx": "2026-06-29T10:49:24+08:00",
"content/self-host/upgrading/4-15/41505.mdx": "2026-06-29T10:43:24+08:00", "content/self-host/upgrading/4-15/41505.mdx": "2026-06-29T10:45:58+08:00",
"content/self-host/upgrading/4-15/41506.en.mdx": "2026-06-29T00:47:43+08:00", "content/self-host/upgrading/4-15/41506.en.mdx": "2026-06-29T00:47:43+08:00",
"content/self-host/upgrading/4-15/41506.mdx": "2026-06-29T00:47:43+08:00", "content/self-host/upgrading/4-15/41506.mdx": "2026-06-29T00:47:43+08:00",
"content/self-host/upgrading/outdated/40.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/outdated/40.en.mdx": "2026-04-26T21:08:47+08:00",
...@@ -446,6 +446,6 @@ ...@@ -446,6 +446,6 @@
"content/self-host/upgrading/outdated/499.mdx": "2026-05-07T15:06:40+08:00", "content/self-host/upgrading/outdated/499.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/upgrade-intruction.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/upgrade-intruction.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/upgrade-intruction.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/upgrade-intruction.mdx": "2026-04-26T21:08:47+08:00",
"content/toc.en.mdx": "2026-06-27T22:05:51+08:00", "content/toc.en.mdx": "2026-06-29T21:48:35+08:00",
"content/toc.mdx": "2026-06-27T22:05:51+08:00" "content/toc.mdx": "2026-06-29T21:48:35+08:00"
} }
\ No newline at end of file
packages/global/core/app/type.ts
...@@ -114,7 +114,7 @@ export const AppAutoExecuteConfigTypeSchema = z.object({ ...@@ -114,7 +114,7 @@ export const AppAutoExecuteConfigTypeSchema = z.object({
open: BoolSchema.meta({ open: BoolSchema.meta({
description: '是否在进入会话后自动触发应用执行' description: '是否在进入会话后自动触发应用执行'
}), }),
defaultPrompt: z.string().meta({ defaultPrompt: z.string().default('').meta({
description: '自动执行时注入的默认用户问题' description: '自动执行时注入的默认用户问题'
}) })
}); });
...@@ -168,7 +168,7 @@ export const AppSchemaTypeSchema = z.object({ ...@@ -168,7 +168,7 @@ export const AppSchemaTypeSchema = z.object({
teamId: z.string(), teamId: z.string(),
tmbId: z.string(), tmbId: z.string(),
type: z.enum(AppTypeEnum), type: z.enum(AppTypeEnum),
version: z.enum(['v1', 'v2']).optional(), version: z.enum(['v1', 'v2']).optional().meta({ description: '内容版本，folder 类型不会有' }),
name: z.string(), name: z.string(),
avatar: z.string(), avatar: z.string(),
......
packages/global/core/workflow/utils.ts
...@@ -80,12 +80,7 @@ export const nodeInputIsReference = (input: FlowNodeInputItemType) => { ...@@ -80,12 +80,7 @@ export const nodeInputIsReference = (input: FlowNodeInputItemType) => {
/* node */ /* node */
export const getGuideModule = (nodes: StoreNodeItemType[]) => export const getGuideModule = (nodes: StoreNodeItemType[]) =>
nodes.find( nodes.find((item) => item.flowNodeType === FlowNodeTypeEnum.systemConfig);
(item) =>
item.flowNodeType === FlowNodeTypeEnum.systemConfig ||
// @ts-ignore (adapt v1)
item.flowType === FlowNodeTypeEnum.systemConfig
);
export const splitGuideModule = (guideModules?: StoreNodeItemType) => { export const splitGuideModule = (guideModules?: StoreNodeItemType) => {
const welcomeText: string = const welcomeText: string =
guideModules?.inputs?.find((item) => item.key === NodeInputKeyEnum.welcomeText)?.value ?? ''; guideModules?.inputs?.find((item) => item.key === NodeInputKeyEnum.welcomeText)?.value ?? '';
......
packages/global/test/core/workflow/utils.test.ts
...@@ -172,21 +172,6 @@ describe('getGuideModule', () => { ...@@ -172,21 +172,6 @@ describe('getGuideModule', () => {
const result = getGuideModule([]); const result = getGuideModule([]);
expect(result).toBeUndefined(); expect(result).toBeUndefined();
}); });
it('should find node with v1 flowType (adapt v1)', () => {
const nodes = [
{
nodeId: 'node1',
flowType: FlowNodeTypeEnum.systemConfig,
flowNodeType: FlowNodeTypeEnum.chatNode,
name: 'System Config',
inputs: [],
outputs: []
}
] as any;
const result = getGuideModule(nodes);
expect(result?.nodeId).toBe('node1');
});
}); });
describe('splitGuideModule', () => { describe('splitGuideModule', () => {
......
packages/service/core/ai/llm/utils.ts
...@@ -278,11 +278,19 @@ export const loadRequestMessages = async ({ ...@@ -278,11 +278,19 @@ export const loadRequestMessages = async ({
* 仅从短文本中识别媒体 URL。普通文档 URL 仍作为文本保留，不在这里转成 LLM 媒体输入。 * 仅从短文本中识别媒体 URL。普通文档 URL 仍作为文本保留，不在这里转成 LLM 媒体输入。
*/ */
const extractMediaUrls = (input: string) => { const extractMediaUrls = (input: string) => {
const urlRegex = /(https?:\/\/[^\s]+)/gi; const urlRegex = /(https?:\/\/[^\s<>"'\\]+)/gi;
const normalizeExtractedUrl = (rawUrl: string) => {
// Markdown 链接里的 URL 可能同时出现在 label 和 href 中，避免把 `](...)` 当成 URL 的一部分。
const markdownLinkSeparatorIndex = rawUrl.indexOf('](');
const urlWithoutMarkdownTail =
markdownLinkSeparatorIndex === -1 ? rawUrl : rawUrl.slice(0, markdownLinkSeparatorIndex);
return urlWithoutMarkdownTail.replace(/[)\]\}，。,.!?;:]+$/, '');
};
const normalizedUrls = Array.from(input.matchAll(urlRegex), (m) => normalizeExtractedUrl(m[0]));
return Array.from(new Set(input.matchAll(urlRegex)), (m) => return Array.from(new Set(normalizedUrls)).filter((url) => getFileTypeFromUrl(url) !== 'file');
m[0].replace(/[，。,.!?;:]+$/, '')
).filter((url) => getFileTypeFromUrl(url) !== 'file');
}; };
/** /**
......
packages/service/test/core/ai/llm/utils.test.ts
...@@ -712,6 +712,49 @@ describe('loadRequestMessages function tests', () => { ...@@ -712,6 +712,49 @@ describe('loadRequestMessages function tests', () => {
expect(content.some((item: any) => item.type === 'image_url')).toBe(true); expect(content.some((item: any) => item.type === 'image_url')).toBe(true);
}); });
it('should extract clean image url from markdown link in json-like text', async () => {
serviceEnv.MULTIPLE_DATA_TO_BASE64 = false;
const imageUrl =
'https://proxy.file.fastgpt.io/chat/image.png?X-Amz-Signature=test&x-id=GetObject';
const messages: ChatCompletionMessageParam[] = [
{
role: ChatCompletionRequestMessageRoleEnum.User,
content: `"url": "[${imageUrl}](${imageUrl})\\"]"`
}
];
const result = await loadRequestMessages({ messages, useVision: true });
expect(result).toHaveLength(1);
const content = result[0].content as any[];
const imageParts = content.filter((item: any) => item.type === 'image_url');
expect(imageParts).toHaveLength(1);
expect(imageParts[0].image_url.url).toBe(imageUrl);
expect(mockAxiosHead).toHaveBeenCalledWith(imageUrl, {
timeout: 10000
});
});
it('should extract image url before newline terminators', async () => {
serviceEnv.MULTIPLE_DATA_TO_BASE64 = false;
const imageUrl =
'https://proxy.file.fastgpt.io/chat/%E7%AE%80%E5%8E%86.png?X-Amz-Signature=test&x-id=GetObject';
const messages: ChatCompletionMessageParam[] = [
{
role: ChatCompletionRequestMessageRoleEnum.User,
content: `${imageUrl}\n${imageUrl}\\n`
}
];
const result = await loadRequestMessages({ messages, useVision: true });
expect(result).toHaveLength(1);
const content = result[0].content as any[];
const imageParts = content.filter((item: any) => item.type === 'image_url');
expect(imageParts).toHaveLength(1);
expect(imageParts[0].image_url.url).toBe(imageUrl);
});
it('should not extract images from very long text (>500 chars)', async () => { it('should not extract images from very long text (>500 chars)', async () => {
const longText = 'A'.repeat(600) + ' https://example.com/image.png'; const longText = 'A'.repeat(600) + ' https://example.com/image.png';
const messages: ChatCompletionMessageParam[] = [ const messages: ChatCompletionMessageParam[] = [
......
packages/web/i18n/en/common.json
...@@ -158,13 +158,15 @@ ...@@ -158,13 +158,15 @@
"code_error.error_message.513": "Unauthorized to Read This File", "code_error.error_message.513": "Unauthorized to Read This File",
"code_error.error_message.514": "Invalid API Key", "code_error.error_message.514": "Invalid API Key",
"code_error.openapi_error.api_key_not_exist": "API Key Does Not Exist", "code_error.openapi_error.api_key_not_exist": "API Key Does Not Exist",
"code_error.openapi_error.exceed_limit": "Up to 10 API Keys", "code_error.openapi_error.exceed_limit": "API key limit reached",
"code_error.openapi_error.un_auth": "Unauthorized to Operate This API Key", "code_error.openapi_error.un_auth": "Unauthorized to Operate This API Key",
"code_error.outlink_error.invalid_link": "Invalid Share Link", "code_error.outlink_error.invalid_link": "Invalid Share Link",
"code_error.outlink_error.link_not_exist": "Share Link Does Not Exist", "code_error.outlink_error.link_not_exist": "Share Link Does Not Exist",
"code_error.outlink_error.un_auth_user": "Identity Verification Failed", "code_error.outlink_error.un_auth_user": "Identity Verification Failed",
"code_error.plugin_error.un_auth": "No permission to operate the tool", "code_error.plugin_error.un_auth": "No permission to operate the tool",
"code_error.sandbox_error.agent_sandbox_initializing": "The virtual machine is initializing. Please try again later.",
"code_error.sandbox_error.agent_sandbox_permission_denied": "The current app is not authorized to use the sandbox/VM. Please contact an administrator to configure it.", "code_error.sandbox_error.agent_sandbox_permission_denied": "The current app is not authorized to use the sandbox/VM. Please contact an administrator to configure it.",
"code_error.sandbox_error.runtime_upgrade_failed": "Virtual machine runtime upgrade failed. Please try again.",
"code_error.skill_error.archive_empty": "Archive is empty", "code_error.skill_error.archive_empty": "Archive is empty",
"code_error.skill_error.archive_extraction_failed": "Failed to extract archive", "code_error.skill_error.archive_extraction_failed": "Failed to extract archive",
"code_error.skill_error.archive_too_large": "Archive file size exceeds the maximum allowed limit", "code_error.skill_error.archive_too_large": "Archive file size exceeds the maximum allowed limit",
...@@ -182,10 +184,8 @@ ...@@ -182,10 +184,8 @@
"code_error.skill_error.not_exist": "Skill Does Not Exist", "code_error.skill_error.not_exist": "Skill Does Not Exist",
"code_error.skill_error.skill_name_too_long": "Skill name must be 50 characters or fewer", "code_error.skill_error.skill_name_too_long": "Skill name must be 50 characters or fewer",
"code_error.skill_error.un_auth_skill": "Unauthorized to Operate This Skill", "code_error.skill_error.un_auth_skill": "Unauthorized to Operate This Skill",
"code_error.sandbox_error.agent_sandbox_initializing": "The virtual machine is initializing. Please try again later.",
"code_error.sandbox_error.runtime_upgrade_failed": "Virtual machine runtime upgrade failed. Please try again.",
"code_error.system_error.community_version_num_limit": "Exceeded Open Source Version Limit, Please Upgrade to Commercial Version: https://fastgpt.io",
"code_error.system_error.commercial_feature": "Commercial Edition exclusive feature", "code_error.system_error.commercial_feature": "Commercial Edition exclusive feature",
"code_error.system_error.community_version_num_limit": "Exceeded Open Source Version Limit, Please Upgrade to Commercial Version: https://fastgpt.io",
"code_error.system_error.license_app_amount_limit": "Exceed the maximum number of applications in the system", "code_error.system_error.license_app_amount_limit": "Exceed the maximum number of applications in the system",
"code_error.system_error.license_dataset_amount_limit": "Exceed the maximum number of knowledge bases in the system", "code_error.system_error.license_dataset_amount_limit": "Exceed the maximum number of knowledge bases in the system",
"code_error.system_error.license_user_amount_limit": "Exceed the maximum number of users in the system", "code_error.system_error.license_user_amount_limit": "Exceed the maximum number of users in the system",
......
packages/web/i18n/zh-CN/common.json
...@@ -158,13 +158,15 @@ ...@@ -158,13 +158,15 @@
"code_error.error_message.513": "没有权限读取该文件", "code_error.error_message.513": "没有权限读取该文件",
"code_error.error_message.514": "Api Key 不合法", "code_error.error_message.514": "Api Key 不合法",
"code_error.openapi_error.api_key_not_exist": "Api Key 不存在", "code_error.openapi_error.api_key_not_exist": "Api Key 不存在",
"code_error.openapi_error.exceed_limit": "最多 10 组 API 密钥", "code_error.openapi_error.exceed_limit": "API 密钥数量达到上限",
"code_error.openapi_error.un_auth": "无权操作该 Api Key", "code_error.openapi_error.un_auth": "无权操作该 Api Key",
"code_error.outlink_error.invalid_link": "分享链接无效", "code_error.outlink_error.invalid_link": "分享链接无效",
"code_error.outlink_error.link_not_exist": "分享链接不存在", "code_error.outlink_error.link_not_exist": "分享链接不存在",
"code_error.outlink_error.un_auth_user": "身份校验失败", "code_error.outlink_error.un_auth_user": "身份校验失败",
"code_error.plugin_error.un_auth": "无权操作该工具", "code_error.plugin_error.un_auth": "无权操作该工具",
"code_error.sandbox_error.agent_sandbox_initializing": "虚拟机正在初始化，请稍后重试。",
"code_error.sandbox_error.agent_sandbox_permission_denied": "当前应用无权使用虚拟机，请联系管理员配置。", "code_error.sandbox_error.agent_sandbox_permission_denied": "当前应用无权使用虚拟机，请联系管理员配置。",
"code_error.sandbox_error.runtime_upgrade_failed": "虚拟机运行环境升级失败，请重新尝试。",
"code_error.skill_error.archive_empty": "压缩包内容为空", "code_error.skill_error.archive_empty": "压缩包内容为空",
"code_error.skill_error.archive_extraction_failed": "压缩包解压失败", "code_error.skill_error.archive_extraction_failed": "压缩包解压失败",
"code_error.skill_error.archive_too_large": "压缩包大小超过最大限制", "code_error.skill_error.archive_too_large": "压缩包大小超过最大限制",
...@@ -182,10 +184,8 @@ ...@@ -182,10 +184,8 @@
"code_error.skill_error.not_exist": "技能不存在", "code_error.skill_error.not_exist": "技能不存在",
"code_error.skill_error.skill_name_too_long": "技能名称不能超过 50 字符", "code_error.skill_error.skill_name_too_long": "技能名称不能超过 50 字符",
"code_error.skill_error.un_auth_skill": "无权操作该技能", "code_error.skill_error.un_auth_skill": "无权操作该技能",
"code_error.sandbox_error.agent_sandbox_initializing": "虚拟机正在初始化，请稍后重试。",
"code_error.sandbox_error.runtime_upgrade_failed": "虚拟机运行环境升级失败，请重新尝试。",
"code_error.system_error.community_version_num_limit": "超出社区版数量限制，请升级商业版: https://fastgpt.in",
"code_error.system_error.commercial_feature": "商业版专属功能", "code_error.system_error.commercial_feature": "商业版专属功能",
"code_error.system_error.community_version_num_limit": "超出社区版数量限制，请升级商业版: https://fastgpt.in",
"code_error.system_error.license_app_amount_limit": "超出系统最大应用数量", "code_error.system_error.license_app_amount_limit": "超出系统最大应用数量",
"code_error.system_error.license_dataset_amount_limit": "超出系统最大知识库数量", "code_error.system_error.license_dataset_amount_limit": "超出系统最大知识库数量",
"code_error.system_error.license_user_amount_limit": "超出系统最大用户数量", "code_error.system_error.license_user_amount_limit": "超出系统最大用户数量",
......
packages/web/i18n/zh-Hant/common.json
...@@ -156,13 +156,15 @@ ...@@ -156,13 +156,15 @@
"code_error.error_message.513": "無權讀取此檔案", "code_error.error_message.513": "無權讀取此檔案",
"code_error.error_message.514": "API 金鑰無效", "code_error.error_message.514": "API 金鑰無效",
"code_error.openapi_error.api_key_not_exist": "API 金鑰不存在", "code_error.openapi_error.api_key_not_exist": "API 金鑰不存在",
"code_error.openapi_error.exceed_limit": "最多 10 組 API 金鑰", "code_error.openapi_error.exceed_limit": "API 金鑰數量達到上限",
"code_error.openapi_error.un_auth": "無權操作此 API 金鑰", "code_error.openapi_error.un_auth": "無權操作此 API 金鑰",
"code_error.outlink_error.invalid_link": "分享連結無效", "code_error.outlink_error.invalid_link": "分享連結無效",
"code_error.outlink_error.link_not_exist": "分享連結不存在", "code_error.outlink_error.link_not_exist": "分享連結不存在",
"code_error.outlink_error.un_auth_user": "身份驗證失敗", "code_error.outlink_error.un_auth_user": "身份驗證失敗",
"code_error.plugin_error.un_auth": "無權操作該工具", "code_error.plugin_error.un_auth": "無權操作該工具",
"code_error.sandbox_error.agent_sandbox_initializing": "虛擬機正在初始化，請稍後重試。",
"code_error.sandbox_error.agent_sandbox_permission_denied": "當前應用無權使用虛擬機，請聯絡管理員配置。", "code_error.sandbox_error.agent_sandbox_permission_denied": "當前應用無權使用虛擬機，請聯絡管理員配置。",
"code_error.sandbox_error.runtime_upgrade_failed": "虛擬機運行環境升級失敗，請重新嘗試。",
"code_error.skill_error.archive_empty": "壓縮包內容為空", "code_error.skill_error.archive_empty": "壓縮包內容為空",
"code_error.skill_error.archive_extraction_failed": "壓縮包解壓失敗", "code_error.skill_error.archive_extraction_failed": "壓縮包解壓失敗",
"code_error.skill_error.archive_too_large": "壓縮包大小超過最大限制", "code_error.skill_error.archive_too_large": "壓縮包大小超過最大限制",
...@@ -180,10 +182,8 @@ ...@@ -180,10 +182,8 @@
"code_error.skill_error.not_exist": "技能不存在", "code_error.skill_error.not_exist": "技能不存在",
"code_error.skill_error.skill_name_too_long": "技能名稱不能超過 50 字元", "code_error.skill_error.skill_name_too_long": "技能名稱不能超過 50 字元",
"code_error.skill_error.un_auth_skill": "無權操作該技能", "code_error.skill_error.un_auth_skill": "無權操作該技能",
"code_error.sandbox_error.agent_sandbox_initializing": "虛擬機正在初始化，請稍後重試。",
"code_error.sandbox_error.runtime_upgrade_failed": "虛擬機運行環境升級失敗，請重新嘗試。",
"code_error.system_error.community_version_num_limit": "超出開源版數量限制，請升級商業版：https://fastgpt.io",
"code_error.system_error.commercial_feature": "商業版專屬功能", "code_error.system_error.commercial_feature": "商業版專屬功能",
"code_error.system_error.community_version_num_limit": "超出開源版數量限制，請升級商業版：https://fastgpt.io",
"code_error.system_error.license_app_amount_limit": "超出系統最大應用數量", "code_error.system_error.license_app_amount_limit": "超出系統最大應用數量",
"code_error.system_error.license_dataset_amount_limit": "超出系統最大知識庫數量", "code_error.system_error.license_dataset_amount_limit": "超出系統最大知識庫數量",
"code_error.system_error.license_user_amount_limit": "超出系統最大用戶數量", "code_error.system_error.license_user_amount_limit": "超出系統最大用戶數量",
......
projects/app/package.json
{ {
"name": "@fastgpt/app", "name": "@fastgpt/app",
"version": "4.15.0-6", "version": "4.15.0",
"private": false, "private": false,
"browserslist": [ "browserslist": [
"Chrome >= 80", "Chrome >= 80",
......
projects/app/src/pageComponents/app/detail/Edit/SimpleApp/index.tsx
...@@ -6,7 +6,6 @@ import { Box, Flex } from '@chakra-ui/react'; ...@@ -6,7 +6,6 @@ import { Box, Flex } from '@chakra-ui/react';
import { useTranslation } from 'next-i18next'; import { useTranslation } from 'next-i18next';
import { useSimpleAppSnapshots } from '../FormComponent/useSnapshots'; import { useSimpleAppSnapshots } from '../FormComponent/useSnapshots';
import { useDebounceEffect, useMount } from 'ahooks'; import { useDebounceEffect, useMount } from 'ahooks';
import { v1Workflow2V2 } from '@/web/core/workflow/adapt';
import { defaultAppSelectFileConfig } from '@fastgpt/global/core/app/constants'; import { defaultAppSelectFileConfig } from '@fastgpt/global/core/app/constants';
import { form2AppWorkflow, appWorkflow2Form } from './utils'; import { form2AppWorkflow, appWorkflow2Form } from './utils';
import PublishChannel from '../../Publish'; import PublishChannel from '../../Publish';
...@@ -22,13 +21,6 @@ const SimpleEdit = () => { ...@@ -22,13 +21,6 @@ const SimpleEdit = () => {
); );
const [appForm, setAppForm] = useState(() => { const [appForm, setAppForm] = useState(() => {
if (appDetail.version !== 'v2') {
return appWorkflow2Form({
nodes: v1Workflow2V2((appDetail.modules || []) as any)?.nodes,
chatConfig: appDetail.chatConfig
});
}
if (past.length === 0) { if (past.length === 0) {
return appWorkflow2Form({ return appWorkflow2Form({
nodes: appDetail.modules, nodes: appDetail.modules,
......
projects/app/src/pageComponents/app/detail/Plugin/Header.tsx
...@@ -46,7 +46,6 @@ const Header = () => { ...@@ -46,7 +46,6 @@ const Header = () => {
}); });
const { appDetail, onSaveApp, currentTab } = useContextSelector(AppContext, (v) => v); const { appDetail, onSaveApp, currentTab } = useContextSelector(AppContext, (v) => v);
const isV2Workflow = appDetail?.version === 'v2';
const { const {
isOpen: isOpenBackConfirm, isOpen: isOpenBackConfirm,
onOpen: onOpenBackConfirm, onOpen: onOpenBackConfirm,
...@@ -163,7 +162,7 @@ const Header = () => { ...@@ -163,7 +162,7 @@ const Header = () => {
{/* app info */} {/* app info */}
<Box ml={1}> <Box ml={1}>
<AppCard isSaved={isSaved} showSaveStatus={isV2Workflow} /> <AppCard isSaved={isSaved} showSaveStatus />
</Box> </Box>
{isPc && ( {isPc && (
...@@ -222,7 +221,6 @@ const Header = () => { ...@@ -222,7 +221,6 @@ const Header = () => {
isSaved, isSaved,
onBack, onBack,
onOpenBackConfirm, onOpenBackConfirm,
isV2Workflow,
showHistoryModal, showHistoryModal,
t, t,
loading, loading,
...@@ -235,7 +233,7 @@ const Header = () => { ...@@ -235,7 +233,7 @@ const Header = () => {
return ( return (
<> <>
{Render} {Render}
{showHistoryModal && isV2Workflow && currentTab === TabEnum.appEdit && ( {showHistoryModal && currentTab === TabEnum.appEdit && (
<PublishHistories<WorkflowSnapshotsType> <PublishHistories<WorkflowSnapshotsType>
onClose={() => { onClose={() => {
setShowHistoryModal(false); setShowHistoryModal(false);
......
projects/app/src/pageComponents/app/detail/Plugin/index.tsx
import React from 'react'; import React from 'react';
import { pluginSystemModuleTemplates } from '@fastgpt/global/core/workflow/template/constants'; import { pluginSystemModuleTemplates } from '@fastgpt/global/core/workflow/template/constants';
import { useConfirm } from '@fastgpt/web/hooks/useConfirm';
import { v1Workflow2V2 } from '@/web/core/workflow/adapt';
import { ReactFlowCustomProvider } from '../WorkflowComponents/context'; import { ReactFlowCustomProvider } from '../WorkflowComponents/context';
import { useContextSelector } from 'use-context-selector'; import { useContextSelector } from 'use-context-selector';
import { AppContext, TabEnum } from '../context'; import { AppContext, TabEnum } from '../context';
...@@ -13,7 +11,6 @@ import dynamic from 'next/dynamic'; ...@@ -13,7 +11,6 @@ import dynamic from 'next/dynamic';
import { cloneDeep } from 'lodash'; import { cloneDeep } from 'lodash';
import Flow from '../WorkflowComponents/Flow'; import Flow from '../WorkflowComponents/Flow';
import { useTranslation } from 'next-i18next';
import { WorkflowUtilsContext } from '../WorkflowComponents/context/workflowUtilsContext'; import { WorkflowUtilsContext } from '../WorkflowComponents/context/workflowUtilsContext';
const Logs = dynamic(() => import('../Logs/index')); const Logs = dynamic(() => import('../Logs/index'));
...@@ -21,24 +18,10 @@ const PublishChannel = dynamic(() => import('../Publish')); ...@@ -21,24 +18,10 @@ const PublishChannel = dynamic(() => import('../Publish'));
const WorkflowEdit = () => { const WorkflowEdit = () => {
const { appDetail, currentTab } = useContextSelector(AppContext, (e) => e); const { appDetail, currentTab } = useContextSelector(AppContext, (e) => e);
const isV2Workflow = appDetail?.version === 'v2';
const { t } = useTranslation();
const { openConfirm, ConfirmModal } = useConfirm({
showCancel: false,
content: t('common:info.old_version_attention')
});
const initData = useContextSelector(WorkflowUtilsContext, (v) => v.initData); const initData = useContextSelector(WorkflowUtilsContext, (v) => v.initData);
useMount(() => { useMount(() => {
if (!isV2Workflow) {
openConfirm({
onConfirm: () => {
initData(JSON.parse(JSON.stringify(v1Workflow2V2((appDetail.modules || []) as any))));
}
})();
} else {
initData( initData(
cloneDeep({ cloneDeep({
nodes: appDetail.modules || [], nodes: appDetail.modules || [],
...@@ -46,7 +29,6 @@ const WorkflowEdit = () => { ...@@ -46,7 +29,6 @@ const WorkflowEdit = () => {
}), }),
true true
); );
}
}); });
return ( return (
...@@ -71,8 +53,6 @@ const WorkflowEdit = () => { ...@@ -71,8 +53,6 @@ const WorkflowEdit = () => {
{currentTab === TabEnum.logs && <Logs />} {currentTab === TabEnum.logs && <Logs />}
</Flex> </Flex>
)} )}
{!isV2Workflow && <ConfirmModal countDown={0} />}
</Flex> </Flex>
); );
}; };
......
projects/app/src/pageComponents/app/detail/Workflow/Header.tsx
...@@ -46,7 +46,6 @@ const Header = () => { ...@@ -46,7 +46,6 @@ const Header = () => {
}); });
const { appDetail, onSaveApp, currentTab } = useContextSelector(AppContext, (v) => v); const { appDetail, onSaveApp, currentTab } = useContextSelector(AppContext, (v) => v);
const isV2Workflow = appDetail?.version === 'v2';
const { const {
isOpen: isOpenBackConfirm, isOpen: isOpenBackConfirm,
onOpen: onOpenBackConfirm, onOpen: onOpenBackConfirm,
...@@ -163,7 +162,7 @@ const Header = () => { ...@@ -163,7 +162,7 @@ const Header = () => {
{/* app info */} {/* app info */}
<Box ml={1}> <Box ml={1}>
<AppCard isSaved={isSaved} showSaveStatus={isV2Workflow} /> <AppCard isSaved={isSaved} showSaveStatus />
</Box> </Box>
{isPc && ( {isPc && (
...@@ -222,7 +221,6 @@ const Header = () => { ...@@ -222,7 +221,6 @@ const Header = () => {
isSaved, isSaved,
onBack, onBack,
onOpenBackConfirm, onOpenBackConfirm,
isV2Workflow,
showHistoryModal, showHistoryModal,
t, t,
loading, loading,
...@@ -235,7 +233,7 @@ const Header = () => { ...@@ -235,7 +233,7 @@ const Header = () => {
return ( return (
<> <>
{Render} {Render}
{showHistoryModal && isV2Workflow && currentTab === TabEnum.appEdit && ( {showHistoryModal && currentTab === TabEnum.appEdit && (
<PublishHistories<WorkflowSnapshotsType> <PublishHistories<WorkflowSnapshotsType>
onClose={() => { onClose={() => {
setShowHistoryModal(false); setShowHistoryModal(false);
......
projects/app/src/pageComponents/app/detail/Workflow/index.tsx
import React from 'react'; import React from 'react';
import { appSystemModuleTemplates } from '@fastgpt/global/core/workflow/template/constants'; import { appSystemModuleTemplates } from '@fastgpt/global/core/workflow/template/constants';
import { useConfirm } from '@fastgpt/web/hooks/useConfirm';
import { v1Workflow2V2 } from '@/web/core/workflow/adapt';
import { useContextSelector } from 'use-context-selector'; import { useContextSelector } from 'use-context-selector';
import { AppContext, TabEnum } from '../context'; import { AppContext, TabEnum } from '../context';
...@@ -11,7 +9,6 @@ import { Flex } from '@chakra-ui/react'; ...@@ -11,7 +9,6 @@ import { Flex } from '@chakra-ui/react';
import { workflowBoxStyles } from '../constants'; import { workflowBoxStyles } from '../constants';
import dynamic from 'next/dynamic'; import dynamic from 'next/dynamic';
import { cloneDeep } from 'lodash'; import { cloneDeep } from 'lodash';
import { useTranslation } from 'next-i18next';
import Flow from '../WorkflowComponents/Flow'; import Flow from '../WorkflowComponents/Flow';
import { ReactFlowCustomProvider } from '../WorkflowComponents/context/index'; import { ReactFlowCustomProvider } from '../WorkflowComponents/context/index';
...@@ -24,27 +21,9 @@ const WorkflowEdit = () => { ...@@ -24,27 +21,9 @@ const WorkflowEdit = () => {
const appDetail = useContextSelector(AppContext, (v) => v.appDetail); const appDetail = useContextSelector(AppContext, (v) => v.appDetail);
const currentTab = useContextSelector(AppContext, (v) => v.currentTab); const currentTab = useContextSelector(AppContext, (v) => v.currentTab);
const isV2Workflow = appDetail?.version === 'v2';
const { t } = useTranslation();
const { openConfirm, ConfirmModal } = useConfirm({
showCancel: false,
content: t('common:info.old_version_attention')
});
const initData = useContextSelector(WorkflowUtilsContext, (v) => v.initData); const initData = useContextSelector(WorkflowUtilsContext, (v) => v.initData);
useMount(() => { useMount(() => {
if (!isV2Workflow) {
openConfirm({
onConfirm: () => {
initData(
JSON.parse(JSON.stringify(v1Workflow2V2((appDetail.modules || []) as any))),
true
);
}
})();
} else {
initData( initData(
cloneDeep({ cloneDeep({
nodes: appDetail.modules || [], nodes: appDetail.modules || [],
...@@ -52,7 +31,6 @@ const WorkflowEdit = () => { ...@@ -52,7 +31,6 @@ const WorkflowEdit = () => {
}), }),
true true
); );
}
}); });
return ( return (
...@@ -77,8 +55,6 @@ const WorkflowEdit = () => { ...@@ -77,8 +55,6 @@ const WorkflowEdit = () => {
{currentTab === TabEnum.logs && <Logs />} {currentTab === TabEnum.logs && <Logs />}
</Flex> </Flex>
)} )}
{!isV2Workflow && <ConfirmModal countDown={0} />}
</Flex> </Flex>
); );
}; };
......
projects/app/src/pages/api/admin/clearInvalidData.ts deleted 100644 → 0
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { addHours } from 'date-fns';
import { checkInvalidDatasetData, checkInvalidVector } from '@/service/common/system/cronTask';
import dayjs from 'dayjs';
import { retryFn } from '@fastgpt/global/common/system/utils';
import { NextAPI } from '@/service/middleware/entry';
import { useIPFrequencyLimit } from '@fastgpt/service/common/middle/reqFrequencyLimit';
import { MongoImage } from '@fastgpt/service/common/file/image/schema';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import { getLogger, LogCategories } from '@fastgpt/service/common/logger';
import z from 'zod';
const logger = getLogger(LogCategories.SYSTEM);
const BodySchema = z
.object({
start: z.number().int().min(-8760).max(0).default(-2),
end: z
.number()
.int()
.min(-8760)
.max(-1)
.default(-360 * 24)
})
.refine((data) => data.end < data.start, {
message: 'end 必须小于 start（end 时间点更早）'
});
const MAX_CHUNKS = 1000;
let deleteImageAmount = 0;
async function checkInvalidImg(start: Date, end: Date) {
const images = await MongoImage.find(
{
createTime: {
$gte: start,
$lte: end
},
'metadata.relatedId': { $exists: true }
},
'_id teamId metadata'
);
logger.info('Start invalid image cleanup', { totalImages: images.length });
let index = 0;
for await (const image of images) {
try {
// 1. 检测是否有对应的集合
const collection = await MongoDatasetCollection.findOne(
{
teamId: image.teamId,
'metadata.relatedImgId': image.metadata?.relatedId
},
'_id'
).lean();
if (!collection) {
await image.deleteOne();
deleteImageAmount++;
}
index++;
if (index % 100 === 0) {
logger.debug('Invalid image cleanup progress', {
processed: index,
total: images.length,
deleted: deleteImageAmount
});
}
} catch (error) {
logger.error('Invalid data cleanup task failed', { error });
}
}
logger.info(`检测完成，共删除 ${deleteImageAmount} 个无效图片`);
}
async function handler(req: NextApiRequest, res: NextApiResponse) {
deleteImageAmount = 0;
try {
await authCert({ req, authRoot: true });
const { start, end } = BodySchema.parse(req.body);
(async () => {
try {
logger.info('执行脏数据清理任务');
// Split time range into 6-hour chunks to avoid processing too much data at once
const totalHours = Math.abs(start - end);
const chunkHours = 6;
const chunks = Math.min(Math.ceil(totalHours / chunkHours), MAX_CHUNKS);
logger.info(
`Total time range: ${totalHours} hours, split into ${chunks} chunks of ${chunkHours} hours each`
);
for (let i = 0; i < chunks; i++) {
const chunkStart = start - i * chunkHours;
const chunkEnd = Math.max(start - (i + 1) * chunkHours, end);
const chunkEndTime = addHours(new Date(), chunkStart);
const chunkStartTime = addHours(new Date(), chunkEnd);
logger.info(
`Processing chunk ${i + 1}/${chunks}: ${dayjs(chunkStartTime).format(
'YYYY-MM-DD HH:mm'
)} to ${dayjs(chunkEndTime).format('YYYY-MM-DD HH:mm')}`
);
await retryFn(() => checkInvalidImg(chunkStartTime, chunkEndTime));
await retryFn(() => checkInvalidDatasetData(chunkStartTime, chunkEndTime));
await retryFn(() => checkInvalidVector(chunkStartTime, chunkEndTime));
logger.info(`Chunk ${i + 1}/${chunks} completed`);
}
logger.info('执行脏数据清理任务完毕');
} catch (error) {
logger.info('执行脏数据清理任务出错了');
}
})();
jsonRes(res, {
message: 'success'
});
} catch (error) {
logger.error('Invalid data cleanup task failed', { error });
jsonRes(res, {
code: 500,
error
});
}
}
export default NextAPI(useIPFrequencyLimit({ id: 'admin-api', seconds: 60, limit: 1 }), handler);
projects/app/src/pages/api/admin/dataClean/initWorkflowData.ts 0 → 100644
import { NextAPI } from '@/service/middleware/entry';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { parseApiInput } from '@fastgpt/service/common/zod/requestParseError';
import type { ApiRequestProps } from '@fastgpt/service/type/next';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { MongoAppVersion } from '@fastgpt/service/core/app/version/schema';
import type { AnyBulkWriteOperation, Model } from '@fastgpt/service/common/mongo';
import { AppFolderTypeList } from '@fastgpt/global/core/app/constants';
import {
FlowNodeInputTypeEnum,
FlowNodeOutputTypeEnum,
FlowNodeTypeEnum
} from '@fastgpt/global/core/workflow/node/constant';
import {
NodeInputKeyEnum,
VariableInputEnum,
WorkflowIOValueTypeEnum
} from '@fastgpt/global/core/workflow/constants';
import { PluginStatusEnum, type PluginStatusType } from '@fastgpt/global/core/plugin/type';
import { PublishAppBodySchema } from '@fastgpt/global/openapi/core/app/version/api';
import { BoolSchema, IntSchema } from '@fastgpt/global/common/zod';
import { getLogger } from '@fastgpt/service/common/logger';
import z from 'zod';
/* ============================================================================
* API: 初始化工作流 V2 枚举与结构脏数据
* Route: POST /api/admin/dataClean/initWorkflowData
* Method: POST
* Description: 管理员数据清洗接口，扫描 apps.modules 与 app_versions.nodes，按保存接口结构格式化工作流数据并在 Zod 校验通过后写库。
* Tags: ['Admin', 'DataClean', 'Workflow', 'Write']
* ============================================================================ */
const logger = getLogger(['initWorkflowData']);
const PROGRESS_LOG_EVERY = 1000;
const DEFAULT_BATCH_SIZE = 1000;
const DEFAULT_WRITE_BATCH_SIZE = 10;
const validFlowNodeTypes = new Set(Object.values(FlowNodeTypeEnum));
const validFlowNodeInputTypes = new Set(Object.values(FlowNodeInputTypeEnum));
const validFlowNodeOutputTypes = new Set(Object.values(FlowNodeOutputTypeEnum));
const validWorkflowIOValueTypes = new Set(Object.values(WorkflowIOValueTypeEnum));
const validVariableInputTypes = new Set(Object.values(VariableInputEnum));
const validPluginStatusValues = new Set(Object.values(PluginStatusEnum));
const pluginStatusNumberMap = {
1: PluginStatusEnum.Normal,
2: PluginStatusEnum.SoonOffline,
3: PluginStatusEnum.Offline
} as const;
const optionalNodeStringFields = [
'parentNodeId',
'avatar',
'avatarLinear',
'intro',
'toolDescription',
'version',
'versionLabel',
'pluginId',
'source',
'readmeUrl'
] as const;
const optionalToolDataStringFields = [
'diagram',
'userGuide',
'courseUrl',
'readmeUrl',
'name',
'avatar',
'error'
] as const;
const optionalWorkflowIOStringFields = [
'referencePlaceholder',
'placeholder',
'valueDesc',
'debugLabel',
'description',
'toolDescription',
'enum'
] as const;
const optionalWorkflowIOArrayFields = ['list', 'markList'] as const;
const optionalWorkflowIONumberFields = [
'maxLength',
'minLength',
'step',
'max',
'min',
'precision'
] as const;
const optionalWorkflowIOBooleanFields = [
'required',
'canEdit',
'isPro',
'isToolOutput',
'deprecated'
] as const;
const optionalNodeBooleanFields = [
'abandon',
'showStatus',
'isLatestVersion',
'catchError',
'isFolder',
'hasTokenFee',
'hasSystemSecret'
] as const;
const optionalNodeNumberFields = ['currentCost', 'systemKeyCost'] as const;
const optionalNodeObjectFields = ['position'] as const;
const legacyVariableInputTypeMap: Record<string, VariableInputEnum> = {
string: VariableInputEnum.input,
text: VariableInputEnum.input,
number: VariableInputEnum.numberInput,
boolean: VariableInputEnum.switch,
multiSelect: VariableInputEnum.multipleSelect
};
const saveApiSchema = PublishAppBodySchema.pick({ nodes: true, edges: true, chatConfig: true });
const enumConfigs = {
renderTypeList: {
enumName: 'FlowNodeInputTypeEnum',
enumObject: FlowNodeInputTypeEnum,
regexp: /^FlowNodeInputTypeEnum\./
},
outputType: {
enumName: 'FlowNodeOutputTypeEnum',
enumObject: FlowNodeOutputTypeEnum,
regexp: /^FlowNodeOutputTypeEnum\./
},
valueType: {
enumName: 'WorkflowIOValueTypeEnum',
enumObject: WorkflowIOValueTypeEnum,
regexp: /^WorkflowIOValueTypeEnum\./
}
} as const;
const InitWorkflowDataBodySchema = z
.object({
dryRun: BoolSchema.optional().meta({
example: true,
description: '是否只扫描验证不写库'
}),
dryrun: BoolSchema.optional().meta({
example: true,
description: '是否只扫描验证不写库，兼容小写参数'
}),
batchSize: IntSchema.refine((value) => value >= 1)
.optional()
.meta({
example: DEFAULT_BATCH_SIZE,
description: '每批读取文档数量'
}),
batchsize: IntSchema.refine((value) => value >= 1)
.optional()
.meta({
example: DEFAULT_BATCH_SIZE,
description: '每批读取文档数量，兼容小写参数'
}),
writeBatchSize: IntSchema.refine((value) => value >= 1)
.optional()
.meta({
example: DEFAULT_WRITE_BATCH_SIZE,
description: '每次 bulkWrite 的文档数量'
})
})
.transform((body) => ({
dryRun: body.dryRun ?? body.dryrun ?? true,
batchSize: body.batchSize ?? body.batchsize ?? DEFAULT_BATCH_SIZE,
writeBatchSize: body.writeBatchSize ?? DEFAULT_WRITE_BATCH_SIZE
}));
export type InitWorkflowDataBodyType = z.infer<typeof InitWorkflowDataBodySchema>;
const EnumExpressionEntrySchema = z.object({
field: z.enum(['renderTypeList', 'outputType', 'valueType']).meta({ description: '字段名' }),
expression: z.string().meta({ description: '历史枚举表达式' }),
enumKey: z.string().meta({ description: '枚举 key' }),
known: z.boolean().meta({ description: '是否能映射到当前枚举' }),
fixedValue: z.string().optional().meta({ description: '修复后的值' }),
count: z.number().int().nonnegative().meta({ description: '出现次数' })
});
const ValidationIssueSchema = z.object({
code: z.string().meta({ description: 'Zod 错误码' }),
path: z.string().meta({ description: '错误字段路径' }),
message: z.string().meta({ description: '错误信息' }),
expected: z.unknown().optional().meta({ description: '期望值' }),
received: z.unknown().optional().meta({ description: '实际类型' }),
actualValue: z.unknown().optional().meta({ description: '压缩后的实际值' })
});
const ValidationErrorRecordSchema = z.object({
collectionName: z.string().meta({ description: '集合名' }),
fieldName: z.string().meta({ description: '工作流字段名' }),
documentId: z.string().optional().meta({ description: '文档 ID' }),
appId: z.string().optional().meta({ description: '应用 ID' }),
appVersion: z.string().optional().meta({ description: '应用版本号' }),
name: z.string().optional().meta({ description: '应用名称' }),
schemaName: z.string().meta({ description: 'Schema 名称' }),
stage: z.enum(['saveApi', 'clean', 'write']).meta({ description: '报错阶段' }),
issueCount: z.number().int().nonnegative().meta({ description: '错误数量' }),
issues: z.array(ValidationIssueSchema).meta({ description: '错误明细' })
});
const CollectionStatsSchema = z.object({
collectionName: z.string().meta({ description: '集合名' }),
fieldName: z.string().meta({ description: '工作流字段名' }),
queryMatchedDocumentCount: z.number().int().nonnegative().nullable().meta({
description: '查询命中文档数量；固定全量扫描时为 null'
}),
scannedDocumentCount: z.number().int().nonnegative().meta({ description: '已扫描文档数' }),
fixableDocumentCount: z
.number()
.int()
.nonnegative()
.meta({ description: '存在可修复枚举的文档数' }),
unknownDocumentCount: z
.number()
.int()
.nonnegative()
.meta({ description: '存在未知枚举表达式的文档数' }),
enumExpressionCount: z.number().int().nonnegative().meta({ description: '枚举表达式总数' }),
renderTypeListFixableCount: z.number().int().nonnegative().meta({
description: '可修复 renderTypeList 数量'
}),
outputTypeFixableCount: z
.number()
.int()
.nonnegative()
.meta({ description: '可修复 output.type 数量' }),
valueTypeFixableCount: z
.number()
.int()
.nonnegative()
.meta({ description: '可修复 valueType 数量' }),
unknownEnumExpressionCount: z
.number()
.int()
.nonnegative()
.meta({ description: '未知枚举表达式数量' }),
saveApiValidationErrorDocumentCount: z.number().int().nonnegative().meta({
description: '保存接口 Schema 校验失败文档数'
}),
cleanErrorDocumentCount: z.number().int().nonnegative().meta({ description: '清洗异常文档数' }),
formatChangedDocumentCount: z
.number()
.int()
.nonnegative()
.meta({ description: '存在 format 变更的文档数' }),
writeSuccessDocumentCount: z.number().int().nonnegative().meta({ description: '写入成功文档数' }),
writeBlockedDocumentCount: z
.number()
.int()
.nonnegative()
.meta({ description: '因校验失败阻断写入文档数' }),
writeErrorDocumentCount: z.number().int().nonnegative().meta({ description: '写入失败文档数' }),
byExpression: z.array(EnumExpressionEntrySchema).meta({ description: '枚举表达式分布' })
});
export type CollectionStatsType = z.infer<typeof CollectionStatsSchema>;
const InitWorkflowDataResponseSchema = z.object({
dryRun: z.boolean().meta({ description: '是否 dryRun' }),
batchSize: z.number().int().positive().meta({ description: '每批读取文档数量' }),
writeBatchSize: z.number().int().positive().meta({ description: '每次 bulkWrite 的文档数量' }),
apps: CollectionStatsSchema,
appVersions: CollectionStatsSchema,
total: CollectionStatsSchema
});
export type InitWorkflowDataResponseType = z.infer<typeof InitWorkflowDataResponseSchema>;
type CollectionKey = 'apps' | 'appVersions';
type EnumField = keyof typeof enumConfigs;
type CollectionConfig = {
key: CollectionKey;
collectionName: 'apps' | 'app_versions';
fieldName: 'modules' | 'nodes';
saveSchemaName: 'PublishAppBodySchema.nodes/edges/chatConfig';
};
type WorkflowIOItem = {
renderTypeList?: unknown;
type?: unknown;
valueType?: unknown;
[key: string]: unknown;
};
type WorkflowNode = {
flowNodeType?: unknown;
inputs?: unknown;
outputs?: unknown;
[key: string]: unknown;
};
type WorkflowDocument = {
_id?: unknown;
appId?: unknown;
name?: unknown;
version?: unknown;
modules?: unknown;
nodes?: unknown;
edges?: unknown;
chatConfig?: unknown;
};
type DocumentContext = {
collectionName: string;
documentId?: string;
appId?: string;
appVersion?: string;
name?: string;
};
type EnumExpressionEntry = z.infer<typeof EnumExpressionEntrySchema>;
type ValidationIssue = z.infer<typeof ValidationIssueSchema>;
type ValidationErrorRecord = z.infer<typeof ValidationErrorRecordSchema>;
type FormatChangeTracker = {
count: number;
};
type MutableCollectionStats = Omit<CollectionStatsType, 'byExpression'> & {
byExpression: Record<string, EnumExpressionEntry>;
};
type CleanResult = {
nodes: unknown[];
edges: unknown;
chatConfig: unknown;
renderTypeListFixedCount: number;
outputTypeFixedCount: number;
valueTypeFixedCount: number;
unknownEnumExpressionCount: number;
formatChanges: FormatChangeTracker;
};
type RuntimeContext = Pick<InitWorkflowDataBodyType, 'dryRun'> & {
writeBatchSize: number;
};
type PendingWriteOperation = {
docContext: DocumentContext;
operation: AnyBulkWriteOperation<any>;
};
type ProcessDocumentResult = {
writeOperation?: PendingWriteOperation;
};
const collectionConfigs = {
apps: {
key: 'apps',
collectionName: 'apps',
fieldName: 'modules',
saveSchemaName: 'PublishAppBodySchema.nodes/edges/chatConfig'
},
appVersions: {
key: 'appVersions',
collectionName: 'app_versions',
fieldName: 'nodes',
saveSchemaName: 'PublishAppBodySchema.nodes/edges/chatConfig'
}
} as const satisfies Record<CollectionKey, CollectionConfig>;
const isRecord = (value: unknown): value is Record<string, unknown> =>
typeof value === 'object' && value !== null && !Array.isArray(value);
const stringifyId = (value: unknown) => {
if (value == null) return undefined;
if (typeof value === 'object' && 'toString' in value && typeof value.toString === 'function') {
return value.toString();
}
return String(value);
};
const pathToString = (issuePath: PropertyKey[]) => issuePath.map((item) => String(item)).join('.');
const getValueByPath = ({ value, issuePath }: { value: unknown; issuePath: PropertyKey[] }) =>
issuePath.reduce<unknown>((current, key) => {
if (current == null) return undefined;
if (Array.isArray(current) && typeof key === 'number') return current[key];
if (isRecord(current) && (typeof key === 'string' || typeof key === 'number')) {
return current[key];
}
return undefined;
}, value);
const compactIssueValue = (value: unknown): unknown => {
if (value == null) return value;
if (typeof value === 'string' || typeof value === 'number' || typeof value === 'boolean') {
return value;
}
if (value instanceof Date) return value.toISOString();
if (Array.isArray(value)) {
return {
type: 'array',
length: value.length
};
}
if (typeof value === 'object') {
const stringValue = stringifyId(value);
if (stringValue && /^[0-9a-fA-F]{24}$/.test(stringValue)) return stringValue;
return {
type: 'object',
keys: Object.keys(value).slice(0, 20)
};
}
return String(value);
};
const recordFormatChange = ({
changes
}: {
changes: FormatChangeTracker;
path: string;
before: unknown;
after: unknown;
reason: string;
}) => {
changes.count += 1;
};
const deleteNullOptionalField = ({
target,
key,
path: fieldPath,
changes,
reason
}: {
target: Record<string, unknown>;
key: string;
path: string;
changes: FormatChangeTracker;
reason: string;
}) => {
if (target[key] !== null) return;
recordFormatChange({
changes,
path: fieldPath,
before: null,
after: undefined,
reason
});
delete target[key];
};
const normalizeStringFallback = ({
target,
key,
fallback,
path: fieldPath,
changes,
reason
}: {
target: Record<string, unknown>;
key: string;
fallback: string;
path: string;
changes: FormatChangeTracker;
reason: string;
}) => {
if (typeof target[key] === 'string') return;
recordFormatChange({
changes,
path: fieldPath,
before: target[key],
after: fallback,
reason
});
target[key] = fallback;
};
const normalizeBooleanOptionalField = ({
target,
key,
path: fieldPath,
changes,
reason
}: {
target: Record<string, unknown>;
key: string;
path: string;
changes: FormatChangeTracker;
reason: string;
}) => {
if (target[key] !== null) return;
recordFormatChange({
changes,
path: fieldPath,
before: null,
after: undefined,
reason
});
delete target[key];
};
const normalizeStringOptionalFields = ({
target,
keys,
basePath,
changes,
reason
}: {
target: Record<string, unknown>;
keys: readonly string[];
basePath: string;
changes: FormatChangeTracker;
reason: string;
}) => {
keys.forEach((key) =>
deleteNullOptionalField({
target,
key,
path: `${basePath}.${key}`,
changes,
reason
})
);
};
const normalizeNullOptionalFields = ({
target,
keys,
basePath,
changes,
reason
}: {
target: Record<string, unknown>;
keys: readonly string[];
basePath: string;
changes: FormatChangeTracker;
reason: string;
}) => {
keys.forEach((key) =>
deleteNullOptionalField({
target,
key,
path: `${basePath}.${key}`,
changes,
reason
})
);
};
const normalizeValueType = ({
value,
path: fieldPath,
changes,
reason
}: {
value: unknown;
path: string;
changes: FormatChangeTracker;
reason: string;
}) => {
if (value === undefined) return value;
if (
typeof value === 'string' &&
validWorkflowIOValueTypes.has(value as WorkflowIOValueTypeEnum)
) {
return value;
}
recordFormatChange({
changes,
path: fieldPath,
before: value,
after: WorkflowIOValueTypeEnum.any,
reason
});
return WorkflowIOValueTypeEnum.any;
};
const normalizeValueTypeList = ({
list,
path: fieldPath,
changes
}: {
list: unknown;
path: string;
changes: FormatChangeTracker;
}) => {
if (!Array.isArray(list)) return list;
return list.map((valueType, valueTypeIndex) =>
normalizeValueType({
value: valueType,
path: `${fieldPath}[${valueTypeIndex}]`,
changes,
reason: 'legacy invalid workflow value type option converted to any'
})
);
};
const normalizeSelectValueTypeList = ({
item,
basePath,
changes
}: {
item: Record<string, unknown>;
basePath: string;
changes: FormatChangeTracker;
}) => {
['customInputConfig', 'customFieldConfig'].forEach((configKey) => {
const configValue = item[configKey];
if (!isRecord(configValue)) return;
if (configValue.selectValueTypeList === null) {
recordFormatChange({
changes,
path: `${basePath}.${configKey}.selectValueTypeList`,
before: null,
after: undefined,
reason: 'legacy optional selectValueTypeList is null'
});
delete configValue.selectValueTypeList;
}
const nextConfigValue = { ...configValue };
const normalizedValueTypeList = normalizeValueTypeList({
list: configValue.selectValueTypeList,
path: `${basePath}.${configKey}.selectValueTypeList`,
changes
});
if (normalizedValueTypeList === undefined) {
delete nextConfigValue.selectValueTypeList;
} else {
nextConfigValue.selectValueTypeList = normalizedValueTypeList;
}
item[configKey] = nextConfigValue;
});
};
const normalizeRenderTypeList = ({
list,
path: fieldPath,
changes
}: {
list: unknown;
path: string;
changes: FormatChangeTracker;
}) => {
if (!Array.isArray(list)) return list;
return list
.map((renderType, renderTypeIndex) => {
if (
typeof renderType === 'string' &&
validFlowNodeInputTypes.has(renderType as FlowNodeInputTypeEnum)
) {
return renderType;
}
recordFormatChange({
changes,
path: `${fieldPath}[${renderTypeIndex}]`,
before: renderType,
after: undefined,
reason: 'legacy invalid render type removed'
});
return undefined;
})
.filter((renderType): renderType is FlowNodeInputTypeEnum => renderType !== undefined);
};
const normalizePluginStatus = ({
value,
path: fieldPath,
changes
}: {
value: unknown;
path: string;
changes: FormatChangeTracker;
}) => {
const normalizedStatus = (() => {
if (typeof value === 'string' && validPluginStatusValues.has(value as PluginStatusType)) {
return value;
}
if (
typeof value === 'number' &&
Object.prototype.hasOwnProperty.call(pluginStatusNumberMap, value)
) {
return pluginStatusNumberMap[value as keyof typeof pluginStatusNumberMap];
}
return undefined;
})();
if (normalizedStatus === value) return value;
recordFormatChange({
changes,
path: fieldPath,
before: value,
after: normalizedStatus,
reason: 'legacy plugin status normalized'
});
return normalizedStatus;
};
const getNodeIdFallback = ({
node,
nodeIndex,
documentId
}: {
node: Record<string, unknown>;
nodeIndex: number;
documentId?: string;
}) => {
if (typeof node.moduleId === 'string' && node.moduleId.length > 0) return node.moduleId;
const seed = `${documentId ?? 'doc'}-${nodeIndex}`;
let hash = 0;
for (let index = 0; index < seed.length; index += 1) {
hash = (hash * 31 + seed.charCodeAt(index)) >>> 0;
}
return hash.toString(36).slice(0, 6).padStart(6, '0');
};
const normalizeOptionListLabels = ({
item,
listKey,
basePath,
changes
}: {
item: Record<string, unknown>;
listKey: 'list' | 'enums';
basePath: string;
changes: FormatChangeTracker;
}) => {
const list = item[listKey];
if (!Array.isArray(list)) return;
item[listKey] = list
.map((option, optionIndex) => {
if (!isRecord(option)) return option;
const nextOption: Record<string, unknown> = { ...option };
if (nextOption.value === undefined) {
recordFormatChange({
changes,
path: `${basePath}.${listKey}[${optionIndex}].value`,
before: nextOption.value,
after: '',
reason: 'legacy option missing value'
});
nextOption.value = '';
}
if (typeof nextOption.label !== 'string' && typeof nextOption.value === 'string') {
recordFormatChange({
changes,
path: `${basePath}.${listKey}[${optionIndex}].label`,
before: nextOption.label,
after: nextOption.value,
reason: 'legacy option missing label'
});
nextOption.label = nextOption.value;
}
return nextOption;
})
.filter((option) => option !== undefined);
};
const normalizeJsonStringArrayField = ({
item,
key,
path: fieldPath,
changes,
reason
}: {
item: Record<string, unknown>;
key: string;
path: string;
changes: FormatChangeTracker;
reason: string;
}) => {
if (typeof item[key] !== 'string') return;
try {
const parsedValue = JSON.parse(item[key]);
if (!Array.isArray(parsedValue)) throw new Error('parsed value is not array');
recordFormatChange({
changes,
path: fieldPath,
before: item[key],
after: parsedValue,
reason
});
item[key] = parsedValue;
} catch {
recordFormatChange({
changes,
path: fieldPath,
before: item[key],
after: undefined,
reason: `${reason}, parse failed`
});
delete item[key];
}
};
const normalizeNonNegativeIntOptionalField = ({
item,
key,
path: fieldPath,
changes,
reason
}: {
item: Record<string, unknown>;
key: string;
path: string;
changes: FormatChangeTracker;
reason: string;
}) => {
if (item[key] === undefined) return;
if (Number.isInteger(item[key]) && Number(item[key]) >= 0) return;
recordFormatChange({
changes,
path: fieldPath,
before: item[key],
after: undefined,
reason
});
delete item[key];
};
const normalizeVariableType = ({
variable,
variableIndex,
changes
}: {
variable: Record<string, unknown>;
variableIndex: number;
changes: FormatChangeTracker;
}) => {
const currentType = variable.type;
const normalizedType = (() => {
if (
typeof currentType === 'string' &&
validVariableInputTypes.has(currentType as VariableInputEnum)
) {
return currentType;
}
if (typeof currentType === 'string' && legacyVariableInputTypeMap[currentType]) {
return legacyVariableInputTypeMap[currentType];
}
if (variable.valueType === WorkflowIOValueTypeEnum.number) return VariableInputEnum.numberInput;
if (variable.valueType === WorkflowIOValueTypeEnum.boolean) return VariableInputEnum.switch;
if (Array.isArray(variable.list) || Array.isArray(variable.enums))
return VariableInputEnum.select;
return VariableInputEnum.input;
})();
if (currentType === normalizedType) return;
recordFormatChange({
changes,
path: `chatConfig.variables[${variableIndex}].type`,
before: currentType,
after: normalizedType,
reason: 'legacy variable input type normalized'
});
variable.type = normalizedType;
};
const normalizeVariableBaseFields = ({
variable,
variableIndex,
changes
}: {
variable: Record<string, unknown>;
variableIndex: number;
changes: FormatChangeTracker;
}) => {
if (typeof variable.key !== 'string' || variable.key.length === 0) {
const fallbackKey =
typeof variable.label === 'string' && variable.label.length > 0
? variable.label
: `variable_${variableIndex}`;
recordFormatChange({
changes,
path: `chatConfig.variables[${variableIndex}].key`,
before: variable.key,
after: fallbackKey,
reason: 'legacy variable missing key'
});
variable.key = fallbackKey;
}
if (typeof variable.label !== 'string') {
recordFormatChange({
changes,
path: `chatConfig.variables[${variableIndex}].label`,
before: variable.label,
after: variable.key,
reason: 'legacy variable missing label'
});
variable.label = variable.key;
}
variable.valueType = normalizeValueType({
value: variable.valueType,
path: `chatConfig.variables[${variableIndex}].valueType`,
changes,
reason: 'legacy invalid variable value type converted to any'
});
};
const removePropertyRequiredFlags = ({
value,
basePath,
changes
}: {
value: unknown;
basePath: string;
changes: FormatChangeTracker;
}) => {
if (Array.isArray(value)) {
value.forEach((item, index) =>
removePropertyRequiredFlags({ value: item, basePath: `${basePath}[${index}]`, changes })
);
return;
}
if (!isRecord(value)) return;
Object.entries(value).forEach(([key, itemValue]) => {
const itemPath = `${basePath}.${key}`;
if ((key === 'properties' || key === '$defs' || key === 'definitions') && isRecord(itemValue)) {
Object.entries(itemValue).forEach(([propertyKey, propertyValue]) => {
if (!isRecord(propertyValue) || typeof propertyValue.required !== 'boolean') return;
recordFormatChange({
changes,
path: `${itemPath}.${propertyKey}.required`,
before: propertyValue.required,
after: undefined,
reason: 'legacy JSON schema property required flag removed'
});
delete propertyValue.required;
});
}
removePropertyRequiredFlags({ value: itemValue, basePath: itemPath, changes });
});
};
const normalizeToolConfig = ({
toolConfig,
basePath,
changes
}: {
toolConfig: unknown;
basePath: string;
changes: FormatChangeTracker;
}) => {
if (toolConfig === null) {
recordFormatChange({
changes,
path: basePath,
before: null,
after: undefined,
reason: 'legacy optional toolConfig is null'
});
return undefined;
}
if (!isRecord(toolConfig)) return toolConfig;
removePropertyRequiredFlags({ value: toolConfig, basePath, changes });
const systemToolSet = toolConfig.systemToolSet;
if (isRecord(systemToolSet) && Array.isArray(systemToolSet.toolList)) {
systemToolSet.toolList = systemToolSet.toolList
.map((tool, toolIndex) => {
if (!isRecord(tool)) {
recordFormatChange({
changes,
path: `${basePath}.systemToolSet.toolList[${toolIndex}]`,
before: tool,
after: undefined,
reason: 'legacy invalid system tool set child removed'
});
return undefined;
}
const nextTool: Record<string, unknown> = { ...tool };
normalizeStringFallback({
target: nextTool,
key: 'toolId',
fallback:
typeof tool.key === 'string' && tool.key.length > 0
? tool.key
: typeof tool.name === 'string' && tool.name.length > 0
? tool.name
: `${String(systemToolSet.toolId ?? 'systemTool')}_${toolIndex}`,
path: `${basePath}.systemToolSet.toolList[${toolIndex}].toolId`,
changes,
reason: 'legacy system tool set child missing toolId'
});
normalizeStringFallback({
target: nextTool,
key: 'name',
fallback: typeof nextTool.toolId === 'string' ? nextTool.toolId : `tool_${toolIndex}`,
path: `${basePath}.systemToolSet.toolList[${toolIndex}].name`,
changes,
reason: 'legacy system tool set child missing name'
});
normalizeStringFallback({
target: nextTool,
key: 'description',
fallback: '',
path: `${basePath}.systemToolSet.toolList[${toolIndex}].description`,
changes,
reason: 'legacy system tool set child missing description'
});
return nextTool;
})
.filter((tool) => tool !== undefined);
}
const httpToolSet = toolConfig.httpToolSet;
if (isRecord(httpToolSet) && typeof httpToolSet.customHeaders !== 'string') {
recordFormatChange({
changes,
path: `${basePath}.httpToolSet.customHeaders`,
before: httpToolSet.customHeaders,
after: undefined,
reason: 'legacy invalid httpToolSet customHeaders removed'
});
delete httpToolSet.customHeaders;
}
return toolConfig;
};
const normalizeInputListValue = ({
inputList,
basePath,
changes
}: {
inputList: unknown;
basePath: string;
changes: FormatChangeTracker;
}) => {
if (inputList === null) {
recordFormatChange({
changes,
path: basePath,
before: null,
after: undefined,
reason: 'legacy optional inputList is null'
});
return undefined;
}
if (!Array.isArray(inputList)) return inputList;
return inputList.map((inputConfig, inputIndex) => {
if (!isRecord(inputConfig)) return inputConfig;
const nextInputConfig: Record<string, unknown> = { ...inputConfig };
if (
nextInputConfig.value !== undefined &&
!isRecord(nextInputConfig.value) &&
(typeof nextInputConfig.value === 'string' || typeof nextInputConfig.value === 'number')
) {
const nextValue = { value: String(nextInputConfig.value) };
recordFormatChange({
changes,
path: `${basePath}[${inputIndex}].value`,
before: nextInputConfig.value,
after: nextValue,
reason: 'legacy input config secret value normalized'
});
nextInputConfig.value = nextValue;
}
return nextInputConfig;
});
};
const normalizeWorkflowIOItem = ({
item,
basePath,
changes,
kind,
itemIndex
}: {
item: Record<string, unknown>;
basePath: string;
changes: FormatChangeTracker;
kind: 'inputs' | 'outputs';
itemIndex: number;
}) => {
normalizeStringFallback({
target: item,
key: 'key',
fallback:
typeof item.id === 'string'
? item.id
: typeof item.label === 'string'
? item.label
: `${kind}_${itemIndex}`,
path: `${basePath}.key`,
changes,
reason: 'legacy workflow IO missing key'
});
normalizeStringFallback({
target: item,
key: 'label',
fallback: typeof item.key === 'string' ? item.key : '',
path: `${basePath}.label`,
changes,
reason: 'legacy workflow IO missing label'
});
normalizeStringOptionalFields({
target: item,
keys: optionalWorkflowIOStringFields,
basePath,
changes,
reason: 'legacy optional workflow IO string field is null'
});
normalizeNullOptionalFields({
target: item,
keys: optionalWorkflowIOArrayFields,
basePath,
changes,
reason: 'legacy optional workflow IO array field is null'
});
normalizeNullOptionalFields({
target: item,
keys: optionalWorkflowIONumberFields,
basePath,
changes,
reason: 'legacy optional workflow IO number field is null'
});
optionalWorkflowIOBooleanFields.forEach((key) =>
normalizeBooleanOptionalField({
target: item,
key,
path: `${basePath}.${key}`,
changes,
reason: 'legacy optional workflow IO boolean field is null'
})
);
normalizeOptionListLabels({ item, listKey: 'list', basePath, changes });
normalizeOptionListLabels({ item, listKey: 'enums', basePath, changes });
normalizeSelectValueTypeList({ item, basePath, changes });
const normalizedInputList = normalizeInputListValue({
inputList: item.inputList,
basePath: `${basePath}.inputList`,
changes
});
if (normalizedInputList === undefined) {
delete item.inputList;
} else {
item.inputList = normalizedInputList;
}
if (kind === 'inputs') {
item.renderTypeList = normalizeRenderTypeList({
list: item.renderTypeList,
path: `${basePath}.renderTypeList`,
changes
});
if (!Array.isArray(item.renderTypeList) || item.renderTypeList.length === 0) {
recordFormatChange({
changes,
path: `${basePath}.renderTypeList`,
before: item.renderTypeList,
after: [FlowNodeInputTypeEnum.reference],
reason: 'legacy input missing render type'
});
item.renderTypeList = [FlowNodeInputTypeEnum.reference];
}
}
if (kind === 'outputs') {
normalizeStringFallback({
target: item,
key: 'id',
fallback: typeof item.key === 'string' ? item.key : `outputs_${itemIndex}`,
path: `${basePath}.id`,
changes,
reason: 'legacy output missing id'
});
if (!validFlowNodeOutputTypes.has(item.type as FlowNodeOutputTypeEnum)) {
recordFormatChange({
changes,
path: `${basePath}.type`,
before: item.type,
after: FlowNodeOutputTypeEnum.static,
reason: 'legacy invalid output type converted to static'
});
item.type = FlowNodeOutputTypeEnum.static;
}
}
};
const normalizePluginData = ({
pluginData,
basePath,
changes
}: {
pluginData: unknown;
basePath: string;
changes: FormatChangeTracker;
}) => {
if (pluginData === null) {
recordFormatChange({
changes,
path: basePath,
before: null,
after: undefined,
reason: 'legacy optional pluginData is null'
});
return undefined;
}
if (!isRecord(pluginData)) return pluginData;
const nextPluginData: Record<string, unknown> = { ...pluginData };
normalizeStringOptionalFields({
target: nextPluginData,
keys: optionalToolDataStringFields,
basePath,
changes,
reason: 'legacy optional plugin data string field is null'
});
if (nextPluginData.status !== undefined) {
const normalizedStatus = normalizePluginStatus({
value: nextPluginData.status,
path: `${basePath}.status`,
changes
});
if (normalizedStatus === undefined) {
delete nextPluginData.status;
} else {
nextPluginData.status = normalizedStatus;
}
}
return nextPluginData;
};
const buildEmptyNode = ({
node,
nodeIndex,
rootPath,
docContext,
changes,
reason
}: {
node: unknown;
nodeIndex: number;
rootPath: string;
docContext: DocumentContext;
changes: FormatChangeTracker;
reason: string;
}): WorkflowNode => {
const fallbackNode = {
nodeId: getNodeIdFallback({
node: isRecord(node) ? node : {},
nodeIndex,
documentId: docContext.documentId
}),
name:
isRecord(node) && typeof node.flowType === 'string'
? node.flowType
: FlowNodeTypeEnum.emptyNode,
flowNodeType: FlowNodeTypeEnum.emptyNode,
inputs: [],
outputs: []
};
recordFormatChange({
changes,
path: `${rootPath}[${nodeIndex}]`,
before: node,
after: fallbackNode,
reason
});
return fallbackNode;
};
const emptyStats = ({
collectionName,
fieldName
}: {
collectionName: string;
fieldName: string;
}): MutableCollectionStats => ({
collectionName,
fieldName,
queryMatchedDocumentCount: null,
scannedDocumentCount: 0,
fixableDocumentCount: 0,
unknownDocumentCount: 0,
enumExpressionCount: 0,
renderTypeListFixableCount: 0,
outputTypeFixableCount: 0,
valueTypeFixableCount: 0,
unknownEnumExpressionCount: 0,
saveApiValidationErrorDocumentCount: 0,
cleanErrorDocumentCount: 0,
formatChangedDocumentCount: 0,
writeSuccessDocumentCount: 0,
writeBlockedDocumentCount: 0,
writeErrorDocumentCount: 0,
byExpression: {}
});
const createDocContext = ({
collectionName,
doc
}: {
collectionName: string;
doc: WorkflowDocument;
}): DocumentContext => ({
collectionName,
documentId: stringifyId(doc._id),
appId: stringifyId(doc.appId),
appVersion: typeof doc.version === 'string' ? doc.version : undefined,
name: typeof doc.name === 'string' ? doc.name : undefined
});
const parseEnumExpressionValue = <T extends Record<string, string>>({
enumObject,
enumName,
value
}: {
enumObject: T;
enumName: string;
value: unknown;
}) => {
if (typeof value !== 'string') return undefined;
const prefix = `${enumName}.`;
if (!value.startsWith(prefix)) return undefined;
const enumKey = value.slice(prefix.length);
const known = Object.prototype.hasOwnProperty.call(enumObject, enumKey);
return {
expression: value,
enumKey,
known,
fixedValue: known ? enumObject[enumKey as keyof T] : undefined
};
};
const recordExpression = ({
stats,
docResult,
path: fieldPath,
field,
value
}: {
stats: MutableCollectionStats;
docResult: Pick<
CleanResult,
'renderTypeListFixedCount' | 'outputTypeFixedCount' | 'valueTypeFixedCount' | 'formatChanges'
> & {
unknownEnumExpressionCount: number;
};
path: string;
field: EnumField;
value: unknown;
}) => {
const enumConfig = enumConfigs[field];
const parsed = parseEnumExpressionValue({
enumObject: enumConfig.enumObject,
enumName: enumConfig.enumName,
value
});
if (!parsed) return value;
const expressionKey = `${field}:${parsed.expression}`;
const existing = stats.byExpression[expressionKey];
stats.byExpression[expressionKey] = existing || {
field,
expression: parsed.expression,
enumKey: parsed.enumKey,
known: parsed.known,
fixedValue: parsed.fixedValue,
count: 0
};
stats.byExpression[expressionKey].count += 1;
stats.enumExpressionCount += 1;
if (parsed.known) {
if (field === 'renderTypeList') {
docResult.renderTypeListFixedCount += 1;
stats.renderTypeListFixableCount += 1;
} else if (field === 'outputType') {
docResult.outputTypeFixedCount += 1;
stats.outputTypeFixableCount += 1;
} else {
docResult.valueTypeFixedCount += 1;
stats.valueTypeFixableCount += 1;
}
} else {
docResult.unknownEnumExpressionCount += 1;
stats.unknownEnumExpressionCount += 1;
}
if (parsed.known && parsed.fixedValue !== value) {
recordFormatChange({
changes: docResult.formatChanges,
path: fieldPath,
before: value,
after: parsed.fixedValue,
reason: `${enumConfig.enumName} expression`
});
}
return parsed.known ? parsed.fixedValue : value;
};
const fixWorkflowIOList = ({
list,
stats,
docResult,
basePath,
kind,
node
}: {
list: unknown;
stats: MutableCollectionStats;
docResult: CleanResult;
basePath: string;
kind: 'inputs' | 'outputs';
node: WorkflowNode;
}) => {
if (!Array.isArray(list)) {
recordFormatChange({
changes: docResult.formatChanges,
path: basePath,
before: list,
after: [],
reason: 'legacy workflow IO list is not array'
});
return [];
}
return list.map((item, itemIndex) => {
if (!isRecord(item)) {
const fallbackItem =
kind === 'inputs'
? {
key: `${kind}_${itemIndex}`,
label: `${kind}_${itemIndex}`,
renderTypeList: [FlowNodeInputTypeEnum.reference],
valueType: WorkflowIOValueTypeEnum.any
}
: {
id: `${kind}_${itemIndex}`,
key: `${kind}_${itemIndex}`,
label: `${kind}_${itemIndex}`,
type: FlowNodeOutputTypeEnum.static,
valueType: WorkflowIOValueTypeEnum.any
};
recordFormatChange({
changes: docResult.formatChanges,
path: `${basePath}[${itemIndex}]`,
before: item,
after: fallbackItem,
reason: 'legacy workflow IO item is not object'
});
return fallbackItem;
}
const nextItem: WorkflowIOItem = { ...item };
if (Array.isArray(item.renderTypeList)) {
nextItem.renderTypeList = item.renderTypeList.map((renderType, renderTypeIndex) =>
recordExpression({
stats,
docResult,
path: `${basePath}[${itemIndex}].renderTypeList[${renderTypeIndex}]`,
field: 'renderTypeList',
value: renderType
})
);
}
nextItem.renderTypeList = normalizeRenderTypeList({
list: nextItem.renderTypeList,
path: `${basePath}[${itemIndex}].renderTypeList`,
changes: docResult.formatChanges
});
if (kind === 'outputs') {
nextItem.type = recordExpression({
stats,
docResult,
path: `${basePath}[${itemIndex}].type`,
field: 'outputType',
value: item.type
});
}
nextItem.valueType = recordExpression({
stats,
docResult,
path: `${basePath}[${itemIndex}].valueType`,
field: 'valueType',
value: item.valueType
});
nextItem.valueType = normalizeValueType({
value: nextItem.valueType,
path: `${basePath}[${itemIndex}].valueType`,
changes: docResult.formatChanges,
reason: 'legacy invalid workflow value type converted to any'
});
if (
kind === 'inputs' &&
nextItem.valueType == null &&
node.flowNodeType === FlowNodeTypeEnum.code &&
(item.key === NodeInputKeyEnum.codeType || item.key === NodeInputKeyEnum.code)
) {
nextItem.valueType = WorkflowIOValueTypeEnum.string;
recordFormatChange({
changes: docResult.formatChanges,
path: `${basePath}[${itemIndex}].valueType`,
before: item.valueType,
after: nextItem.valueType,
reason: 'legacy code node input missing valueType'
});
}
normalizeWorkflowIOItem({
item: nextItem,
basePath: `${basePath}[${itemIndex}]`,
changes: docResult.formatChanges,
kind,
itemIndex
});
return nextItem;
});
};
/**
* 兼容旧版 chatConfig 数据，保证 dry-run 和写库前校验检查的是最终待保存结构。
*/
const formatChatConfig = ({
chatConfig,
formatChanges
}: {
chatConfig: unknown;
formatChanges: FormatChangeTracker;
}) => {
if (chatConfig == null) {
recordFormatChange({
changes: formatChanges,
path: 'chatConfig',
before: chatConfig,
after: {},
reason: 'legacy empty chatConfig'
});
return {};
}
if (!isRecord(chatConfig)) return chatConfig;
const nextChatConfig: Record<string, unknown> = { ...chatConfig };
const optionalChatConfigKeys = [
'welcomeText',
'variables',
'autoExecute',
'questionGuide',
'ttsConfig',
'whisperConfig',
'scheduledTriggerConfig',
'chatInputGuide',
'fileSelectConfig',
'instruction'
];
optionalChatConfigKeys.forEach((key) => {
if (nextChatConfig[key] === null) {
recordFormatChange({
changes: formatChanges,
path: `chatConfig.${key}`,
before: null,
after: undefined,
reason: 'legacy optional chatConfig field is null'
});
delete nextChatConfig[key];
}
});
if (typeof nextChatConfig.questionGuide === 'boolean') {
recordFormatChange({
changes: formatChanges,
path: 'chatConfig.questionGuide',
before: nextChatConfig.questionGuide,
after: { open: nextChatConfig.questionGuide },
reason: 'legacy boolean questionGuide'
});
nextChatConfig.questionGuide = { open: nextChatConfig.questionGuide };
} else if (
nextChatConfig.questionGuide !== undefined &&
!isRecord(nextChatConfig.questionGuide)
) {
recordFormatChange({
changes: formatChanges,
path: 'chatConfig.questionGuide',
before: nextChatConfig.questionGuide,
after: undefined,
reason: 'legacy invalid questionGuide removed'
});
delete nextChatConfig.questionGuide;
}
if (nextChatConfig.chatInputGuide !== undefined) {
const shouldRemoveChatInputGuide =
!isRecord(nextChatConfig.chatInputGuide) ||
typeof nextChatConfig.chatInputGuide.customUrl !== 'string';
if (shouldRemoveChatInputGuide) {
recordFormatChange({
changes: formatChanges,
path: 'chatConfig.chatInputGuide',
before: nextChatConfig.chatInputGuide,
after: undefined,
reason: 'legacy invalid chatInputGuide removed'
});
delete nextChatConfig.chatInputGuide;
}
}
if (isRecord(nextChatConfig.autoExecute)) {
const autoExecute = nextChatConfig.autoExecute;
if (autoExecute.defaultPrompt === null) {
recordFormatChange({
changes: formatChanges,
path: 'chatConfig.autoExecute.defaultPrompt',
before: null,
after: undefined,
reason: 'legacy auto execute defaultPrompt is null'
});
delete autoExecute.defaultPrompt;
}
if (typeof autoExecute.open !== 'boolean') {
recordFormatChange({
changes: formatChanges,
path: 'chatConfig.autoExecute',
before: autoExecute,
after: undefined,
reason: 'legacy incomplete auto execute config removed'
});
delete nextChatConfig.autoExecute;
}
}
if (isRecord(nextChatConfig.scheduledTriggerConfig)) {
const scheduledTriggerConfig = nextChatConfig.scheduledTriggerConfig;
const missingRequiredField =
typeof scheduledTriggerConfig.cronString !== 'string' ||
typeof scheduledTriggerConfig.timezone !== 'string' ||
typeof scheduledTriggerConfig.defaultPrompt !== 'string';
if (missingRequiredField) {
recordFormatChange({
changes: formatChanges,
path: 'chatConfig.scheduledTriggerConfig',
before: scheduledTriggerConfig,
after: undefined,
reason: 'legacy incomplete scheduled trigger config removed'
});
delete nextChatConfig.scheduledTriggerConfig;
}
}
if (Array.isArray(nextChatConfig.variables)) {
nextChatConfig.variables = nextChatConfig.variables.map((variable, variableIndex) => {
if (!isRecord(variable)) return variable;
const nextVariable: Record<string, unknown> = { ...variable };
normalizeVariableBaseFields({
variable: nextVariable,
variableIndex,
changes: formatChanges
});
if (nextVariable.description === undefined) {
recordFormatChange({
changes: formatChanges,
path: `chatConfig.variables[${variableIndex}].description`,
before: undefined,
after: '',
reason: 'legacy variable missing description'
});
nextVariable.description = '';
}
normalizeNonNegativeIntOptionalField({
item: nextVariable,
key: 'maxLength',
path: `chatConfig.variables[${variableIndex}].maxLength`,
changes: formatChanges,
reason: 'legacy variable maxLength is not non-negative integer'
});
normalizeOptionListLabels({
item: nextVariable,
listKey: 'list',
basePath: `chatConfig.variables[${variableIndex}]`,
changes: formatChanges
});
normalizeJsonStringArrayField({
item: nextVariable,
key: 'enums',
path: `chatConfig.variables[${variableIndex}].enums`,
changes: formatChanges,
reason: 'legacy variable enums JSON string converted to array'
});
normalizeVariableType({
variable: nextVariable,
variableIndex,
changes: formatChanges
});
if (Array.isArray(nextVariable.enums)) {
nextVariable.enums = nextVariable.enums.map((enumItem, enumIndex) => {
if (!isRecord(enumItem)) return enumItem;
const nextEnumItem: Record<string, unknown> = { ...enumItem };
if (nextEnumItem.label === undefined && typeof nextEnumItem.value === 'string') {
recordFormatChange({
changes: formatChanges,
path: `chatConfig.variables[${variableIndex}].enums[${enumIndex}].label`,
before: undefined,
after: nextEnumItem.value,
reason: 'legacy enum missing label'
});
nextEnumItem.label = nextEnumItem.value;
}
return nextEnumItem;
});
}
return nextVariable;
});
}
return nextChatConfig;
};
const formatEdges = ({
edges,
formatChanges
}: {
edges: unknown;
formatChanges: FormatChangeTracker;
}) => {
if (edges == null) {
recordFormatChange({
changes: formatChanges,
path: 'edges',
before: edges,
after: [],
reason: 'legacy empty edges'
});
return [];
}
if (!Array.isArray(edges)) return edges;
return edges.filter((edge, edgeIndex) => {
if (!isRecord(edge)) return true;
const hasInvalidEndpoint = typeof edge.source !== 'string' || typeof edge.target !== 'string';
if (hasInvalidEndpoint) {
recordFormatChange({
changes: formatChanges,
path: `edges[${edgeIndex}]`,
before: edge,
after: undefined,
reason: 'legacy edge missing source or target removed'
});
return false;
}
const hasInvalidHandle =
typeof edge.sourceHandle !== 'string' || typeof edge.targetHandle !== 'string';
if (!hasInvalidHandle) return true;
recordFormatChange({
changes: formatChanges,
path: `edges[${edgeIndex}]`,
before: edge,
after: undefined,
reason: 'legacy edge missing sourceHandle or targetHandle removed'
});
return false;
});
};
/**
* 与本地 scan-workflow-enum-dirty-data 脚本保持一致：只在内存中 format，
* 写库前必须通过 PublishAppBodySchema 的 nodes/edges/chatConfig 校验。
*/
export const formatWorkflowDocument = ({
doc,
fieldName,
stats,
docContext,
rootPath
}: {
doc: WorkflowDocument;
fieldName: CollectionConfig['fieldName'];
stats: MutableCollectionStats;
docContext: DocumentContext;
rootPath: string;
}): CleanResult => {
const docResult: CleanResult = {
nodes: [],
edges: doc.edges,
chatConfig: doc.chatConfig,
renderTypeListFixedCount: 0,
outputTypeFixedCount: 0,
valueTypeFixedCount: 0,
unknownEnumExpressionCount: 0,
formatChanges: {
count: 0
}
};
const nodes = doc[fieldName];
if (!Array.isArray(nodes)) {
recordFormatChange({
changes: docResult.formatChanges,
path: rootPath,
before: nodes,
after: [],
reason: 'legacy workflow nodes is not array'
});
docResult.chatConfig = formatChatConfig({
chatConfig: doc.chatConfig,
formatChanges: docResult.formatChanges
});
docResult.edges = formatEdges({
edges: docResult.edges,
formatChanges: docResult.formatChanges
});
return docResult;
}
docResult.nodes = nodes.map((node, nodeIndex) => {
if (!isRecord(node)) {
return buildEmptyNode({
node,
nodeIndex,
rootPath,
docContext,
changes: docResult.formatChanges,
reason: 'legacy workflow node is not object'
});
}
const nextNode: WorkflowNode = { ...node };
normalizeStringFallback({
target: nextNode,
key: 'nodeId',
fallback: getNodeIdFallback({ node, nodeIndex, documentId: docContext.documentId }),
path: `${rootPath}[${nodeIndex}].nodeId`,
changes: docResult.formatChanges,
reason: 'legacy node missing nodeId'
});
normalizeStringFallback({
target: nextNode,
key: 'name',
fallback: typeof node.flowType === 'string' ? node.flowType : String(nextNode.nodeId),
path: `${rootPath}[${nodeIndex}].name`,
changes: docResult.formatChanges,
reason: 'legacy node missing name'
});
normalizeStringOptionalFields({
target: nextNode,
keys: optionalNodeStringFields,
basePath: `${rootPath}[${nodeIndex}]`,
changes: docResult.formatChanges,
reason: 'legacy optional node string field is null'
});
normalizeNullOptionalFields({
target: nextNode,
keys: optionalNodeBooleanFields,
basePath: `${rootPath}[${nodeIndex}]`,
changes: docResult.formatChanges,
reason: 'legacy optional node boolean field is null'
});
normalizeNullOptionalFields({
target: nextNode,
keys: optionalNodeNumberFields,
basePath: `${rootPath}[${nodeIndex}]`,
changes: docResult.formatChanges,
reason: 'legacy optional node number field is null'
});
normalizeNullOptionalFields({
target: nextNode,
keys: optionalNodeObjectFields,
basePath: `${rootPath}[${nodeIndex}]`,
changes: docResult.formatChanges,
reason: 'legacy optional node object field is null'
});
if (typeof nextNode.version === 'number') {
recordFormatChange({
changes: docResult.formatChanges,
path: `${rootPath}[${nodeIndex}].version`,
before: nextNode.version,
after: String(nextNode.version),
reason: 'legacy numeric node version converted to string'
});
nextNode.version = String(nextNode.version);
}
if (
node.flowNodeType === 'lafModule' ||
!validFlowNodeTypes.has(node.flowNodeType as FlowNodeTypeEnum)
) {
nextNode.flowNodeType = FlowNodeTypeEnum.emptyNode;
recordFormatChange({
changes: docResult.formatChanges,
path: `${rootPath}[${nodeIndex}].flowNodeType`,
before: node.flowNodeType,
after: nextNode.flowNodeType,
reason: 'legacy unknown node converted to emptyNode'
});
}
nextNode.inputs = fixWorkflowIOList({
list: node.inputs,
stats,
docResult,
basePath: `${rootPath}[${nodeIndex}].inputs`,
kind: 'inputs',
node
});
nextNode.outputs = fixWorkflowIOList({
list: node.outputs,
stats,
docResult,
basePath: `${rootPath}[${nodeIndex}].outputs`,
kind: 'outputs',
node
});
const normalizedPluginData = normalizePluginData({
pluginData: node.pluginData,
basePath: `${rootPath}[${nodeIndex}].pluginData`,
changes: docResult.formatChanges
});
if (normalizedPluginData === undefined) {
delete nextNode.pluginData;
} else {
nextNode.pluginData = normalizedPluginData;
}
const normalizedToolConfig = normalizeToolConfig({
toolConfig: node.toolConfig,
basePath: `${rootPath}[${nodeIndex}].toolConfig`,
changes: docResult.formatChanges
});
if (normalizedToolConfig === undefined) {
delete nextNode.toolConfig;
} else {
nextNode.toolConfig = normalizedToolConfig;
}
return nextNode;
});
docResult.chatConfig = formatChatConfig({
chatConfig: doc.chatConfig,
formatChanges: docResult.formatChanges
});
docResult.edges = formatEdges({
edges: docResult.edges,
formatChanges: docResult.formatChanges
});
return docResult;
};
const normalizeZodIssue = ({
issue,
data
}: {
issue: z.core.$ZodIssue;
data: unknown;
}): ValidationIssue => {
const issueWithDetails = issue as z.core.$ZodIssue & {
expected?: unknown;
received?: unknown;
};
const actualValue = compactIssueValue(getValueByPath({ value: data, issuePath: issue.path }));
return {
code: issue.code,
path: pathToString(issue.path),
message: issue.message,
expected: issueWithDetails.expected,
received: issueWithDetails.received,
actualValue
};
};
const recordValidationError = ({
record,
stats
}: {
record: ValidationErrorRecord;
stats: MutableCollectionStats;
}) => {
if (record.stage === 'saveApi') {
stats.saveApiValidationErrorDocumentCount += 1;
} else if (record.stage === 'clean') {
stats.cleanErrorDocumentCount += 1;
} else {
stats.writeErrorDocumentCount += 1;
}
logger.warn('Workflow data clean validation blocked', {
collectionName: record.collectionName,
fieldName: record.fieldName,
documentId: record.documentId,
appId: record.appId,
appVersion: record.appVersion,
name: record.name,
schemaName: record.schemaName,
stage: record.stage,
issueCount: record.issueCount,
issues: record.issues
});
};
const validateAndRecord = ({
schemaName,
stage,
data,
config,
docContext,
stats
}: {
schemaName: string;
stage: ValidationErrorRecord['stage'];
data: unknown;
config: CollectionConfig;
docContext: DocumentContext;
stats: MutableCollectionStats;
}) => {
const result = saveApiSchema.safeParse(data);
if (result.success) return true;
const issues = result.error.issues.map((issue) => normalizeZodIssue({ issue, data }));
recordValidationError({
stats,
record: {
...docContext,
collectionName: config.collectionName,
fieldName: config.fieldName,
schemaName,
stage,
issueCount: issues.length,
issues
}
});
return false;
};
const recordFormatChanges = ({
cleanResult,
stats
}: {
cleanResult: CleanResult;
stats: MutableCollectionStats;
}) => {
if (cleanResult.formatChanges.count === 0) return;
stats.formatChangedDocumentCount += 1;
};
const buildUpdatePayload = ({
config,
cleanResult
}: {
config: CollectionConfig;
cleanResult: CleanResult;
}) => ({
[config.fieldName]: cleanResult.nodes,
edges: cleanResult.edges,
chatConfig: cleanResult.chatConfig
});
const recordWriteError = ({
config,
docContext,
stats,
message
}: {
config: CollectionConfig;
docContext: DocumentContext;
stats: MutableCollectionStats;
message: string;
}) => {
recordValidationError({
stats,
record: {
...docContext,
collectionName: config.collectionName,
fieldName: config.fieldName,
schemaName: 'bulkWriteWorkflowDirtyData',
stage: 'write',
issueCount: 1,
issues: [
{
code: 'write_error',
path: config.fieldName,
message
}
]
}
});
};
const flushWriteOperations = async ({
model,
operations,
config,
stats,
runtime
}: {
model: Model<any>;
operations: PendingWriteOperation[];
config: CollectionConfig;
stats: MutableCollectionStats;
runtime: RuntimeContext;
}) => {
if (operations.length === 0) return;
const writeOperationBatch = async ({
batchOperations,
reason
}: {
batchOperations: PendingWriteOperation[];
reason: string;
}) => {
logger.info('Workflow data clean write start', {
collectionName: config.collectionName,
fieldName: config.fieldName,
reason,
batchSize: batchOperations.length,
firstDocumentId: batchOperations[0]?.docContext.documentId,
lastDocumentId: batchOperations[batchOperations.length - 1]?.docContext.documentId
});
try {
const result = await model.bulkWrite(
batchOperations.map(({ operation }) => operation),
{ ordered: false }
);
stats.writeSuccessDocumentCount += result.matchedCount;
logger.info('Workflow data clean write success', {
collectionName: config.collectionName,
fieldName: config.fieldName,
batchSize: batchOperations.length,
matchedCount: result.matchedCount,
totalWriteSuccessDocumentCount: stats.writeSuccessDocumentCount,
totalWriteErrorDocumentCount: stats.writeErrorDocumentCount
});
const unmatchedCount = batchOperations.length - result.matchedCount;
if (unmatchedCount <= 0) return;
stats.writeErrorDocumentCount += unmatchedCount;
logger.warn('Workflow data clean write unmatched', {
collectionName: config.collectionName,
fieldName: config.fieldName,
batchSize: batchOperations.length,
matchedCount: result.matchedCount,
unmatchedCount,
firstDocumentId: batchOperations[0]?.docContext.documentId,
lastDocumentId: batchOperations[batchOperations.length - 1]?.docContext.documentId
});
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
logger.error('Workflow data clean write error', {
collectionName: config.collectionName,
fieldName: config.fieldName,
batchSize: batchOperations.length,
firstDocumentId: batchOperations[0]?.docContext.documentId,
lastDocumentId: batchOperations[batchOperations.length - 1]?.docContext.documentId,
totalWriteSuccessDocumentCount: stats.writeSuccessDocumentCount,
totalWriteErrorDocumentCount: stats.writeErrorDocumentCount + batchOperations.length,
error: message
});
batchOperations.forEach(({ docContext }) => {
recordWriteError({
config,
docContext,
stats,
message
});
});
}
};
logger.info('Workflow data clean write flush', {
collectionName: config.collectionName,
fieldName: config.fieldName,
operationCount: operations.length,
writeBatchSize: runtime.writeBatchSize,
firstDocumentId: operations[0]?.docContext.documentId,
lastDocumentId: operations[operations.length - 1]?.docContext.documentId
});
for (let start = 0; start < operations.length; start += runtime.writeBatchSize) {
const batchOperations = operations.slice(start, start + runtime.writeBatchSize);
await writeOperationBatch({
batchOperations,
reason: `offset:${start}`
});
}
};
const processWorkflowDocument = ({
doc,
config,
stats,
runtime
}: {
doc: WorkflowDocument;
config: CollectionConfig;
stats: MutableCollectionStats;
runtime: RuntimeContext;
}): ProcessDocumentResult => {
stats.scannedDocumentCount += 1;
if (stats.scannedDocumentCount % PROGRESS_LOG_EVERY === 0) {
logger.info('Workflow data clean progress', {
collectionName: config.collectionName,
fieldName: config.fieldName,
scannedDocumentCount: stats.scannedDocumentCount,
formatChangedDocumentCount: stats.formatChangedDocumentCount,
saveApiValidationErrorDocumentCount: stats.saveApiValidationErrorDocumentCount,
writeSuccessDocumentCount: stats.writeSuccessDocumentCount,
writeBlockedDocumentCount: stats.writeBlockedDocumentCount,
writeErrorDocumentCount: stats.writeErrorDocumentCount
});
}
const docContext = createDocContext({ collectionName: config.collectionName, doc });
try {
const cleanResult = formatWorkflowDocument({
doc,
fieldName: config.fieldName,
stats,
docContext,
rootPath: config.fieldName
});
if (
cleanResult.renderTypeListFixedCount +
cleanResult.outputTypeFixedCount +
cleanResult.valueTypeFixedCount >
0
) {
stats.fixableDocumentCount += 1;
}
if (cleanResult.unknownEnumExpressionCount > 0) {
stats.unknownDocumentCount += 1;
}
recordFormatChanges({
cleanResult,
stats
});
const saveApiValid = validateAndRecord({
schemaName: config.saveSchemaName,
stage: 'saveApi',
data: {
nodes: cleanResult.nodes,
edges: cleanResult.edges,
chatConfig: cleanResult.chatConfig
},
config,
docContext,
stats
});
const valid = saveApiValid && cleanResult.unknownEnumExpressionCount === 0;
if (!valid) {
stats.writeBlockedDocumentCount += 1;
return {};
}
if (runtime.dryRun || cleanResult.formatChanges.count === 0) {
return {};
}
return {
writeOperation: {
docContext,
operation: {
updateOne: {
filter: { _id: doc._id },
update: {
$set: buildUpdatePayload({ config, cleanResult })
}
}
}
}
};
} catch (error) {
recordValidationError({
stats,
record: {
...docContext,
collectionName: config.collectionName,
fieldName: config.fieldName,
schemaName: runtime.dryRun ? 'formatWorkflowDirtyData' : 'formatOrWriteWorkflowDirtyData',
stage: 'clean',
issueCount: 1,
issues: [
{
code: 'clean_error',
path: config.fieldName,
message: error instanceof Error ? error.message : String(error)
}
]
}
});
return {};
}
};
const processWorkflowDocumentBatch = async ({
model,
docs,
config,
stats,
runtime
}: {
model: Model<any>;
docs: WorkflowDocument[];
config: CollectionConfig;
stats: MutableCollectionStats;
runtime: RuntimeContext;
}) => {
const writeOperations: PendingWriteOperation[] = [];
docs.forEach((doc) => {
const result = processWorkflowDocument({
doc,
config,
stats,
runtime
});
if (result.writeOperation) {
writeOperations.push(result.writeOperation);
}
});
await flushWriteOperations({
model,
operations: writeOperations,
config,
stats,
runtime
});
};
const serializeStats = (stats: MutableCollectionStats): CollectionStatsType => ({
...stats,
byExpression: Object.values(stats.byExpression).sort((left, right) => {
if (right.count !== left.count) return right.count - left.count;
return left.expression.localeCompare(right.expression);
})
});
const mergeTotalStats = (statsList: MutableCollectionStats[]) => {
const total = emptyStats({ collectionName: 'total', fieldName: '*' });
const hasCompleteQueryMatchedCount = statsList.every(
(stats) => typeof stats.queryMatchedDocumentCount === 'number'
);
total.queryMatchedDocumentCount = hasCompleteQueryMatchedCount ? 0 : null;
statsList.forEach((stats) => {
if (hasCompleteQueryMatchedCount && typeof stats.queryMatchedDocumentCount === 'number') {
total.queryMatchedDocumentCount =
(total.queryMatchedDocumentCount ?? 0) + stats.queryMatchedDocumentCount;
}
total.scannedDocumentCount += stats.scannedDocumentCount;
total.fixableDocumentCount += stats.fixableDocumentCount;
total.unknownDocumentCount += stats.unknownDocumentCount;
total.enumExpressionCount += stats.enumExpressionCount;
total.renderTypeListFixableCount += stats.renderTypeListFixableCount;
total.outputTypeFixableCount += stats.outputTypeFixableCount;
total.valueTypeFixableCount += stats.valueTypeFixableCount;
total.unknownEnumExpressionCount += stats.unknownEnumExpressionCount;
total.saveApiValidationErrorDocumentCount += stats.saveApiValidationErrorDocumentCount;
total.cleanErrorDocumentCount += stats.cleanErrorDocumentCount;
total.formatChangedDocumentCount += stats.formatChangedDocumentCount;
total.writeSuccessDocumentCount += stats.writeSuccessDocumentCount;
total.writeBlockedDocumentCount += stats.writeBlockedDocumentCount;
total.writeErrorDocumentCount += stats.writeErrorDocumentCount;
Object.values(stats.byExpression).forEach((entry) => {
const expressionKey = `${entry.field}:${entry.expression}`;
const existing = total.byExpression[expressionKey];
total.byExpression[expressionKey] = existing || {
field: entry.field,
expression: entry.expression,
enumKey: entry.enumKey,
known: entry.known,
fixedValue: entry.fixedValue,
count: 0
};
total.byExpression[expressionKey].count += entry.count;
});
});
return total;
};
const scanCollection = async ({
model,
config,
runtime,
batchSize
}: {
model: Model<any>;
config: CollectionConfig;
runtime: RuntimeContext;
batchSize: number;
}) => {
const query: Record<string, unknown> =
config.key === 'apps' ? { type: { $nin: AppFolderTypeList } } : {};
const stats = emptyStats({
collectionName: config.collectionName,
fieldName: config.fieldName
});
let lastId: unknown;
while (true) {
const pageQueryConditions = [query];
if (lastId) {
pageQueryConditions.push({ _id: { $gt: lastId } });
}
const pageQuery = pageQueryConditions.length === 1 ? query : { $and: pageQueryConditions };
const docs = await model
.find(pageQuery, {
_id: 1,
appId: 1,
chatConfig: 1,
edges: 1,
name: 1,
version: 1,
[config.fieldName]: 1
})
.sort({ _id: 1 })
.limit(batchSize)
.lean<WorkflowDocument[]>();
if (docs.length === 0) break;
await processWorkflowDocumentBatch({
model,
docs,
config,
stats,
runtime
});
docs.forEach((doc) => {
lastId = doc._id;
});
}
return stats;
};
/**
* 执行工作流 V2 数据结构清洗。
*
* 流程与本地 scan-workflow-enum-dirty-data 脚本一致：批量读取，逐条 format，
* 逐条用保存接口 Schema 校验；非 dryRun 时只批量更新校验通过且确实被 format 的数据。
*/
export async function runInitWorkflowDataMigration(
options: InitWorkflowDataBodyType
): Promise<InitWorkflowDataResponseType> {
const normalizedOptions = {
dryRun: options.dryRun,
batchSize: options.batchSize ?? DEFAULT_BATCH_SIZE,
writeBatchSize: options.writeBatchSize ?? DEFAULT_WRITE_BATCH_SIZE
};
const runtime: RuntimeContext = {
dryRun: normalizedOptions.dryRun,
writeBatchSize: normalizedOptions.writeBatchSize
};
const appsStats = await scanCollection({
model: MongoApp,
config: collectionConfigs.apps,
runtime,
batchSize: normalizedOptions.batchSize
});
const appVersionsStats = await scanCollection({
model: MongoAppVersion,
config: collectionConfigs.appVersions,
runtime,
batchSize: normalizedOptions.batchSize
});
const totalStats = mergeTotalStats([appsStats, appVersionsStats]);
return InitWorkflowDataResponseSchema.parse({
dryRun: normalizedOptions.dryRun,
batchSize: normalizedOptions.batchSize,
writeBatchSize: normalizedOptions.writeBatchSize,
apps: serializeStats(appsStats),
appVersions: serializeStats(appVersionsStats),
total: serializeStats(totalStats)
});
}
/**
* 管理员工作流数据清洗接口。
*
* 默认 dryRun。真正执行时会批量修复 `apps.modules` 和 `app_versions.nodes`
* 中的 V2 工作流枚举表达式、空值和旧结构字段；失败文档只记录，不写库。
*/
async function handler(req: ApiRequestProps): Promise<InitWorkflowDataResponseType> {
await authCert({ req, authRoot: true });
const { body } = parseApiInput({
req,
bodySchema: InitWorkflowDataBodySchema
});
return runInitWorkflowDataMigration(body);
}
export default NextAPI(handler);
projects/app/src/pages/api/admin/dataClean/v1WorkflowToV2.ts 0 → 100644
import { NextAPI } from '@/service/middleware/entry';
import { parseApiInput } from '@fastgpt/service/common/zod/requestParseError';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import type { ApiRequestProps } from '@fastgpt/service/type/next';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { MongoAppVersion } from '@fastgpt/service/core/app/version/schema';
import type { AnyBulkWriteOperation, Model } from '@fastgpt/service/common/mongo';
import { AppFolderTypeList, type AppTypeEnum } from '@fastgpt/global/core/app/constants';
import {
FlowNodeInputTypeEnum,
FlowNodeOutputTypeEnum,
FlowNodeTypeEnum
} from '@fastgpt/global/core/workflow/node/constant';
import {
NodeInputKeyEnum,
NodeOutputKeyEnum,
WorkflowIOValueTypeEnum
} from '@fastgpt/global/core/workflow/constants';
import { getHandleId } from '@fastgpt/global/core/workflow/utils';
import { PublishAppBodySchema } from '@fastgpt/global/openapi/core/app/version/api';
import { Types } from '@fastgpt/service/common/mongo';
import z from 'zod';
/* ============================================================================
* API: 工作流 V1 数据升级 V2
* Route: POST /api/admin/dataClean/v1WorkflowToV2
* Method: POST
* Description: 管理员数据清洗接口，将历史 apps.modules 与 app_versions.nodes 的 V1 工作流结构升级为 V2。
* Tags: ['Admin', 'DataClean', 'Workflow', 'Write']
* ============================================================================ */
const BATCH_SIZE = 1000;
const MAX_ZOD_ERRORS = 50;
const V1WorkflowToV2BodySchema = z.object({
dryRun: z.boolean().default(true).meta({
example: true,
description: '是否只扫描验证不写库'
})
});
export type V1WorkflowToV2BodyType = z.infer<typeof V1WorkflowToV2BodySchema>;
const ValidationIssueSchema = z.object({
code: z.string().meta({ description: 'Zod 错误码' }),
path: z.string().meta({ description: '错误字段路径' }),
message: z.string().meta({ description: '错误信息' }),
actualValue: z.unknown().optional().meta({ description: '压缩后的实际值' })
});
const IssueSummarySchema = z.object({
path: z.string().meta({ description: '错误字段路径' }),
message: z.string().meta({ description: '错误信息' }),
actualValue: z.unknown().optional().meta({ description: '压缩后的实际值' }),
count: z.number().int().nonnegative().meta({ description: '出现次数' })
});
export type IssueSummaryType = z.infer<typeof IssueSummarySchema>;
const ValidationErrorRecordSchema = z.object({
collectionName: z.string().meta({ description: '集合名' }),
fieldName: z.string().meta({ description: '工作流字段名' }),
documentId: z.string().optional().meta({ description: '文档 ID' }),
appId: z.string().optional().meta({ description: '应用 ID' }),
name: z.string().optional().meta({ description: '应用名称' }),
issueCount: z.number().int().nonnegative().meta({ description: '错误数量' }),
issues: z.array(ValidationIssueSchema).meta({ description: '错误明细' })
});
export type ValidationErrorRecordType = z.infer<typeof ValidationErrorRecordSchema>;
const CollectionStatsSchema = z.object({
collectionName: z.string().meta({ description: '集合名' }),
fieldName: z.string().meta({ description: '工作流字段名' }),
scannedDocumentCount: z.number().int().nonnegative().meta({ description: '扫描文档数' }),
skippedDocumentCount: z.number().int().nonnegative().meta({ description: '跳过文档数' }),
convertedDocumentCount: z.number().int().nonnegative().meta({ description: '转换文档数' }),
zodErrorDocumentCount: z.number().int().nonnegative().meta({ description: 'Zod 失败文档数' }),
writeSuccessDocumentCount: z.number().int().nonnegative().meta({ description: '写入成功文档数' }),
writeBlockedDocumentCount: z
.number()
.int()
.nonnegative()
.meta({ description: '因 Zod 失败阻断写入文档数' }),
writeErrorDocumentCount: z.number().int().nonnegative().meta({ description: '写入失败文档数' }),
issuesByPath: z.array(IssueSummarySchema).meta({ description: 'Zod 错误聚合' })
});
export type CollectionStatsType = z.infer<typeof CollectionStatsSchema>;
const V1WorkflowToV2ResponseSchema = z.object({
dryRun: z.boolean().meta({ description: '是否 dryRun' }),
apps: CollectionStatsSchema,
appVersions: CollectionStatsSchema,
total: CollectionStatsSchema,
zodErrors: z
.array(ValidationErrorRecordSchema)
.meta({ description: 'Zod 错误明细，按固定样本数量截断' })
});
export type V1WorkflowToV2ResponseType = z.infer<typeof V1WorkflowToV2ResponseSchema>;
type CollectionKey = 'apps' | 'appVersions';
type CollectionConfig = {
key: CollectionKey;
collectionName: 'apps' | 'app_versions';
fieldName: 'modules' | 'nodes';
};
type WorkflowDocument = {
_id?: unknown;
appId?: unknown;
name?: unknown;
type?: unknown;
version?: unknown;
modules?: unknown;
nodes?: unknown;
edges?: unknown;
chatConfig?: unknown;
};
type LegacyV1WorkflowNode = {
moduleId?: string;
flowType?: string;
name?: string;
avatar?: string;
intro?: string;
position?: unknown;
showStatus?: boolean;
parentId?: string;
inputs?: Array<Record<string, unknown>>;
outputs?: Array<Record<string, unknown>>;
[key: string]: unknown;
};
type UpgradeChange = {
path: string;
before: unknown;
after: unknown;
reason: string;
};
type UpgradeResult = {
converted: boolean;
nodes: unknown[];
edges: unknown;
chatConfig: unknown;
changes: UpgradeChange[];
};
type MutableCollectionStats = Omit<CollectionStatsType, 'issuesByPath'> & {
issuesByPath: Record<string, IssueSummaryType>;
};
type PendingWriteOperation = {
operation: AnyBulkWriteOperation<any>;
};
const appConfig = {
key: 'apps',
collectionName: 'apps',
fieldName: 'modules'
} as const satisfies CollectionConfig;
const appVersionConfig = {
key: 'appVersions',
collectionName: 'app_versions',
fieldName: 'nodes'
} as const satisfies CollectionConfig;
const inputTypeMap: Record<string, FlowNodeInputTypeEnum> = {
systemInput: FlowNodeInputTypeEnum.input,
input: FlowNodeInputTypeEnum.input,
numberInput: FlowNodeInputTypeEnum.numberInput,
select: FlowNodeInputTypeEnum.select,
target: FlowNodeInputTypeEnum.reference,
switch: FlowNodeInputTypeEnum.switch,
textarea: FlowNodeInputTypeEnum.textarea,
JSONEditor: FlowNodeInputTypeEnum.JSONEditor,
addInputParam: FlowNodeInputTypeEnum.addInputParam,
selectApp: FlowNodeInputTypeEnum.selectApp,
selectLLMModel: FlowNodeInputTypeEnum.selectLLMModel,
settingLLMModel: FlowNodeInputTypeEnum.settingLLMModel,
selectDataset: FlowNodeInputTypeEnum.selectDataset,
selectDatasetParamsModal: FlowNodeInputTypeEnum.selectDatasetParamsModal,
settingDatasetQuotePrompt: FlowNodeInputTypeEnum.settingDatasetQuotePrompt,
hidden: FlowNodeInputTypeEnum.hidden,
custom: FlowNodeInputTypeEnum.custom
};
const outputTypeMap: Record<string, FlowNodeOutputTypeEnum> = {
addOutputParam: FlowNodeOutputTypeEnum.dynamic,
answer: FlowNodeOutputTypeEnum.static,
source: FlowNodeOutputTypeEnum.static,
hidden: FlowNodeOutputTypeEnum.hidden
};
const flowTypeMap: Record<string, FlowNodeTypeEnum> = {
userGuide: FlowNodeTypeEnum.systemConfig,
questionInput: FlowNodeTypeEnum.workflowStart,
chatNode: FlowNodeTypeEnum.chatNode,
datasetSearchNode: FlowNodeTypeEnum.datasetSearchNode,
datasetConcatNode: FlowNodeTypeEnum.datasetConcatNode,
answerNode: FlowNodeTypeEnum.answerNode,
classifyQuestion: FlowNodeTypeEnum.classifyQuestion,
contentExtract: FlowNodeTypeEnum.contentExtract,
httpRequest468: FlowNodeTypeEnum.httpRequest468,
app: FlowNodeTypeEnum.runApp,
pluginModule: FlowNodeTypeEnum.pluginModule,
pluginInput: FlowNodeTypeEnum.pluginInput,
pluginOutput: FlowNodeTypeEnum.pluginOutput,
cfr: FlowNodeTypeEnum.queryExtension,
tools: FlowNodeTypeEnum.toolCall,
stopTool: FlowNodeTypeEnum.stopTool
};
const legacyWorkflowValueTypeMap: Record<string, WorkflowIOValueTypeEnum> = {
chat_history: WorkflowIOValueTypeEnum.chatHistory,
kb_quote: WorkflowIOValueTypeEnum.datasetQuote
};
const isRecord = (value: unknown): value is Record<string, unknown> =>
typeof value === 'object' && value !== null && !Array.isArray(value);
const stringifyId = (value: unknown) => {
if (value == null) return undefined;
if (typeof value === 'object' && 'toString' in value && typeof value.toString === 'function') {
return value.toString();
}
return String(value);
};
const toObjectId = (value: unknown) => {
if (value instanceof Types.ObjectId) return value;
const id = stringifyId(value);
return id && Types.ObjectId.isValid(id) ? new Types.ObjectId(id) : undefined;
};
const isFolderAppType = (type: unknown) =>
typeof type === 'string' && AppFolderTypeList.includes(type as AppTypeEnum);
const isKnownWorkflowValueType = (value: unknown): value is WorkflowIOValueTypeEnum =>
typeof value === 'string' &&
Object.values(WorkflowIOValueTypeEnum).includes(value as WorkflowIOValueTypeEnum);
const normalizeWorkflowValueType = (value: unknown) => {
if (value == null) return undefined;
if (typeof value === 'string' && legacyWorkflowValueTypeMap[value]) {
return legacyWorkflowValueTypeMap[value];
}
return isKnownWorkflowValueType(value) ? value : WorkflowIOValueTypeEnum.any;
};
const isLegacyV1WorkflowNodes = (nodes: unknown[]): nodes is LegacyV1WorkflowNode[] =>
nodes.some(
(node) =>
isRecord(node) &&
typeof node.flowType === 'string' &&
(typeof node.moduleId === 'string' || typeof node.nodeId !== 'string')
);
const randomNodeId = () => Math.random().toString(36).slice(2, 8).padEnd(6, '0');
const getLegacyNodeId = ({ node, index }: { node: LegacyV1WorkflowNode; index: number }) =>
typeof node.moduleId === 'string' && node.moduleId
? node.moduleId
: randomNodeId() || `node${index}`;
const recordChange = ({
changes,
path,
before,
after,
reason
}: {
changes: UpgradeChange[];
path: string;
before: unknown;
after: unknown;
reason: string;
}) => {
changes.push({ path, before, after, reason });
};
/**
* 将历史 V1 workflow 节点结构转换为当前 V2 保存结构。
*
* 兼容策略与本地清洗脚本保持一致：未知 flowType 统一转 emptyNode，非法 valueType
* 转 any，缺失 valueType 保持 undefined，避免旧脏数据在保存前触发 schema 错误。
*/
const convertV1WorkflowToV2 = ({
nodes,
changes,
rootPath
}: {
nodes: LegacyV1WorkflowNode[];
changes: UpgradeChange[];
rootPath: string;
}) => {
const copyNodes = nodes
.map((node, index) => ({
...node,
moduleId: getLegacyNodeId({ node, index }),
inputs: Array.isArray(node.inputs) ? node.inputs : [],
outputs: Array.isArray(node.outputs) ? node.outputs : []
}))
.filter((node, index, self) => {
if (node.flowType === 'questionInput') {
return index === self.findIndex((item) => item.flowType === 'questionInput');
}
return true;
});
const newNodes = copyNodes
.map((node) => {
let pluginId: string | undefined;
const flowNodeType =
typeof node.flowType === 'string'
? flowTypeMap[node.flowType] || FlowNodeTypeEnum.emptyNode
: FlowNodeTypeEnum.emptyNode;
const inputs = (node.inputs || [])
.map((input) => {
const inputType = typeof input.type === 'string' ? inputTypeMap[input.type] : undefined;
const newInput: Record<string, unknown> = {
...input,
selectedTypeIndex: 0,
renderTypeList: !input.type
? [FlowNodeInputTypeEnum.custom]
: inputType
? [inputType]
: [],
key: input.key,
value: input.value,
valueType: normalizeWorkflowValueType(input.valueType),
label: typeof input.label === 'string' ? input.label : String(input.key || ''),
description: input.description,
required: input.required,
toolDescription: input.toolDescription,
canEdit: input.edit,
placeholder: input.placeholder,
list: input.list,
markList: input.markList,
step: input.step,
max: input.max,
min: input.min
};
if (input.key === 'userChatInput') {
newInput.label = '问题输入';
} else if (input.key === 'quoteQA') {
newInput.label = '';
} else if (input.key === 'pluginId' && typeof input.value === 'string') {
pluginId = input.value;
}
return newInput;
})
.filter((input) => Array.isArray(input.renderTypeList) && input.renderTypeList.length > 0)
.filter((input) => {
if (input.key === 'pluginId') return false;
if (input.key === 'switch') return false;
if (input.key === 'pluginStart') return false;
if (input.key === 'DYNAMIC_INPUT_KEY') return false;
if (input.key === 'system_addInputParam') return false;
return true;
});
const outputs = (node.outputs || [])
.map((output) => ({
id: output.key,
type:
typeof output.type === 'string'
? outputTypeMap[output.type] || FlowNodeOutputTypeEnum.static
: FlowNodeOutputTypeEnum.static,
key: output.key,
valueType: normalizeWorkflowValueType(output.valueType),
label: typeof output.label === 'string' ? output.label : String(output.key || ''),
description: output.description,
required: output.required,
defaultValue: output.defaultValue,
canEdit: output.edit,
editField: output.editField
}))
.filter((output) => {
if (node.flowType === 'pluginOutput') return false;
if (output.key === 'finish') return false;
if (output.key === 'isEmpty') return false;
if (output.key === 'unEmpty') return false;
if (output.key === 'pluginStart') return false;
if (node.flowType !== 'questionInput' && output.key === 'userChatInput') return false;
if (
node.flowType === 'contentExtract' &&
(output.key === 'success' || output.key === 'failed')
) {
return false;
}
return true;
});
if (node.flowType === 'questionInput') {
node.name = '流程开始';
} else if (node.flowType === 'pluginOutput') {
(node.outputs || []).forEach((output) => {
inputs.push({
key: output.key,
valueType: normalizeWorkflowValueType(output.valueType),
renderTypeList: [FlowNodeInputTypeEnum.reference],
label: typeof output.key === 'string' ? output.key : '',
canEdit: true
});
});
}
return {
nodeId: node.moduleId,
position: node.position,
flowNodeType,
avatar: node.flowType === 'pluginModule' ? node.avatar : undefined,
name:
node.flowType === 'questionInput'
? node.name
: typeof node.name === 'string'
? node.name
: String(node.flowType || ''),
intro: node.intro,
showStatus: node.showStatus,
pluginId,
parentId: node.parentId,
version: '481',
inputs,
outputs
};
})
.filter((node) => node.nodeId);
let newEdges: Record<string, unknown>[] = [];
copyNodes.forEach((node) => {
(node.outputs || []).forEach((output) => {
const targets = Array.isArray(output.targets) ? output.targets : [];
targets.forEach((target) => {
if (!isRecord(target) || typeof target.moduleId !== 'string') return;
if (output.key === 'finish') return;
if (output.key === 'isEmpty') return;
if (output.key === 'unEmpty') return;
if (node.flowType !== 'questionInput' && output.key === 'userChatInput') return;
if (output.key === NodeOutputKeyEnum.selectedTools) {
newEdges.push({
source: node.moduleId,
sourceHandle: NodeOutputKeyEnum.selectedTools,
target: target.moduleId,
targetHandle: NodeOutputKeyEnum.selectedTools
});
} else if (node.flowType === 'classifyQuestion') {
newEdges.push({
source: node.moduleId,
sourceHandle: getHandleId(node.moduleId || '', 'source', String(output.key)),
target: target.moduleId,
targetHandle: getHandleId(target.moduleId, 'target', 'left')
});
} else if (node.flowType !== 'contentExtract') {
newEdges.push({
source: node.moduleId,
sourceHandle: getHandleId(node.moduleId || '', 'source', 'right'),
target: target.moduleId,
targetHandle: getHandleId(target.moduleId, 'target', 'left')
});
}
});
});
});
newEdges = newEdges.filter(
(edge, index, self) =>
self.findIndex((item) => item.source === edge.source && item.target === edge.target) === index
);
const workflowStart = newNodes.find(
(node) => node.flowNodeType === FlowNodeTypeEnum.workflowStart
);
copyNodes.forEach((node) => {
(node.outputs || []).forEach((output) => {
const targets = Array.isArray(output.targets) ? output.targets : [];
targets.forEach((target) => {
if (!isRecord(target) || typeof target.moduleId !== 'string') return;
const targetNode = newNodes.find((item) => item.nodeId === target.moduleId);
if (!targetNode) return;
const targetInput = targetNode.inputs.find((item) => item.key === target.key);
if (!targetInput) return;
targetInput.value = [node.moduleId, output.key];
});
});
});
newNodes.forEach((node) => {
node.inputs.forEach((input) => {
if (!workflowStart) return;
if (
node.flowNodeType === FlowNodeTypeEnum.datasetSearchNode &&
input.key === NodeInputKeyEnum.datasetSearchInput
) {
input.value = [
[workflowStart.nodeId, NodeOutputKeyEnum.userChatInput],
[workflowStart.nodeId, NodeOutputKeyEnum.userFiles]
];
input.valueType = WorkflowIOValueTypeEnum.arrayString;
return;
}
if (input.key !== NodeInputKeyEnum.userChatInput) return;
input.value = [workflowStart.nodeId, NodeOutputKeyEnum.userChatInput];
});
});
recordChange({
changes,
path: rootPath,
before: { type: 'array', length: nodes.length },
after: { type: 'array', length: newNodes.length },
reason: 'legacy v1 workflow converted to v2'
});
copyNodes.forEach((node, nodeIndex) => {
if (typeof node.flowType !== 'string' || !flowTypeMap[node.flowType]) {
recordChange({
changes,
path: `${rootPath}[${nodeIndex}].flowType`,
before: node.flowType,
after: FlowNodeTypeEnum.emptyNode,
reason: 'legacy unknown flowType converted to emptyNode'
});
}
});
return { nodes: newNodes, edges: newEdges };
};
const formatConvertedWorkflowNodes = ({
nodes,
changes,
rootPath
}: {
nodes: unknown[];
changes: UpgradeChange[];
rootPath: string;
}) =>
nodes.map((node, nodeIndex) => {
if (!isRecord(node)) return node;
const nextNode: Record<string, unknown> = { ...node };
if (nextNode.flowNodeType === 'lafModule') {
recordChange({
changes,
path: `${rootPath}[${nodeIndex}].flowNodeType`,
before: nextNode.flowNodeType,
after: FlowNodeTypeEnum.emptyNode,
reason: 'legacy lafModule node converted to emptyNode'
});
nextNode.flowNodeType = FlowNodeTypeEnum.emptyNode;
}
const formatIOList = ({ list, path }: { list: unknown; path: string }) => {
if (!Array.isArray(list)) return list;
return list.map((item, itemIndex) => {
if (!isRecord(item)) return item;
const nextItem: Record<string, unknown> = { ...item };
const normalizedValueType = normalizeWorkflowValueType(nextItem.valueType);
if (normalizedValueType !== nextItem.valueType) {
recordChange({
changes,
path: `${path}[${itemIndex}].valueType`,
before: nextItem.valueType,
after: normalizedValueType,
reason: 'legacy valueType normalized'
});
nextItem.valueType = normalizedValueType;
}
if (nextItem.label === undefined) {
const label = String(nextItem.key || '');
recordChange({
changes,
path: `${path}[${itemIndex}].label`,
before: undefined,
after: label,
reason: 'legacy IO missing label'
});
nextItem.label = label;
}
(['description', 'toolDescription'] as const).forEach((key) => {
if (nextItem[key] === null) {
recordChange({
changes,
path: `${path}[${itemIndex}].${key}`,
before: null,
after: '',
reason: `legacy IO ${key} is null`
});
nextItem[key] = '';
}
});
return nextItem;
});
};
nextNode.inputs = formatIOList({
list: nextNode.inputs,
path: `${rootPath}[${nodeIndex}].inputs`
});
nextNode.outputs = formatIOList({
list: nextNode.outputs,
path: `${rootPath}[${nodeIndex}].outputs`
});
return nextNode;
});
const formatChatConfig = ({
chatConfig,
changes
}: {
chatConfig: unknown;
changes: UpgradeChange[];
}) => {
if (chatConfig == null) {
recordChange({
changes,
path: 'chatConfig',
before: chatConfig,
after: {},
reason: 'legacy empty chatConfig'
});
return {};
}
if (!isRecord(chatConfig)) return chatConfig;
const nextChatConfig: Record<string, unknown> = { ...chatConfig };
[
'welcomeText',
'variables',
'autoExecute',
'questionGuide',
'ttsConfig',
'whisperConfig',
'scheduledTriggerConfig',
'chatInputGuide',
'fileSelectConfig',
'instruction'
].forEach((key) => {
if (nextChatConfig[key] === null) {
recordChange({
changes,
path: `chatConfig.${key}`,
before: null,
after: undefined,
reason: 'legacy optional chatConfig field is null'
});
delete nextChatConfig[key];
}
});
if (typeof nextChatConfig.questionGuide === 'boolean') {
recordChange({
changes,
path: 'chatConfig.questionGuide',
before: nextChatConfig.questionGuide,
after: { open: nextChatConfig.questionGuide },
reason: 'legacy boolean questionGuide'
});
nextChatConfig.questionGuide = { open: nextChatConfig.questionGuide };
}
if (Array.isArray(nextChatConfig.variables)) {
nextChatConfig.variables = nextChatConfig.variables.map((variable, variableIndex) => {
if (!isRecord(variable)) return variable;
const nextVariable: Record<string, unknown> = { ...variable };
if (nextVariable.description === undefined) {
recordChange({
changes,
path: `chatConfig.variables[${variableIndex}].description`,
before: undefined,
after: '',
reason: 'legacy variable missing description'
});
nextVariable.description = '';
}
if (Array.isArray(nextVariable.enums)) {
nextVariable.enums = nextVariable.enums.map((enumItem, enumIndex) => {
if (!isRecord(enumItem)) return enumItem;
const nextEnumItem: Record<string, unknown> = { ...enumItem };
if (nextEnumItem.label === undefined && typeof nextEnumItem.value === 'string') {
recordChange({
changes,
path: `chatConfig.variables[${variableIndex}].enums[${enumIndex}].label`,
before: undefined,
after: nextEnumItem.value,
reason: 'legacy enum missing label'
});
nextEnumItem.label = nextEnumItem.value;
}
return nextEnumItem;
});
}
if (Array.isArray(nextVariable.list)) {
nextVariable.list = nextVariable.list.map((listItem, listIndex) => {
if (!isRecord(listItem)) return listItem;
const nextListItem: Record<string, unknown> = { ...listItem };
if (nextListItem.label === undefined && typeof nextListItem.value === 'string') {
recordChange({
changes,
path: `chatConfig.variables[${variableIndex}].list[${listIndex}].label`,
before: undefined,
after: nextListItem.value,
reason: 'legacy variable list item missing label'
});
nextListItem.label = nextListItem.value;
}
return nextListItem;
});
}
return nextVariable;
});
}
return nextChatConfig;
};
/**
* 对单个 apps/app_versions 文档执行 V1->V2 转换和兼容格式化。
*
* apps 只有 version 非 v2 且非 folder/httpPlugin/toolFolder 才转换；app_versions
* 通过节点结构判断是否为 V1，避免依赖版本号导致漏处理。
*/
export const upgradeV1WorkflowDocument = ({
doc,
config
}: {
doc: WorkflowDocument;
config: CollectionConfig;
}) => {
const changes: UpgradeChange[] = [];
const rawNodes = doc[config.fieldName];
if (config.key === 'apps') {
if (isFolderAppType(doc.type) || doc.version === 'v2') {
return {
converted: false,
nodes: Array.isArray(rawNodes) ? rawNodes : [],
edges: doc.edges,
chatConfig: doc.chatConfig,
changes
} satisfies UpgradeResult;
}
if (!Array.isArray(rawNodes)) {
recordChange({
changes,
path: config.fieldName,
before: rawNodes,
after: [],
reason: 'legacy app workflow field is not array'
});
return {
converted: true,
nodes: [],
edges: [],
chatConfig: formatChatConfig({ chatConfig: doc.chatConfig, changes }),
changes
} satisfies UpgradeResult;
}
} else if (!Array.isArray(rawNodes) || !isLegacyV1WorkflowNodes(rawNodes)) {
return {
converted: false,
nodes: Array.isArray(rawNodes) ? rawNodes : [],
edges: doc.edges,
chatConfig: doc.chatConfig,
changes
} satisfies UpgradeResult;
}
if (!Array.isArray(rawNodes)) {
return {
converted: false,
nodes: [],
edges: doc.edges,
chatConfig: doc.chatConfig,
changes
} satisfies UpgradeResult;
}
const converted = isLegacyV1WorkflowNodes(rawNodes)
? convertV1WorkflowToV2({ nodes: rawNodes, changes, rootPath: config.fieldName })
: { nodes: rawNodes, edges: doc.edges };
const normalizedNodes = formatConvertedWorkflowNodes({
nodes: converted.nodes,
changes,
rootPath: config.fieldName
});
return {
converted: true,
nodes: normalizedNodes,
edges: converted.edges,
chatConfig: formatChatConfig({ chatConfig: doc.chatConfig, changes }),
changes
} satisfies UpgradeResult;
};
const compactValue = (value: unknown): unknown => {
if (value == null) return value;
if (typeof value === 'string' || typeof value === 'number' || typeof value === 'boolean') {
return value;
}
if (Array.isArray(value)) return { type: 'array', length: value.length };
if (typeof value === 'object') return { type: 'object', keys: Object.keys(value).slice(0, 20) };
return String(value);
};
const getValueByPath = ({ value, issuePath }: { value: unknown; issuePath: PropertyKey[] }) =>
issuePath.reduce<unknown>((current, key) => {
if (current == null) return undefined;
if (Array.isArray(current) && typeof key === 'number') return current[key];
if (isRecord(current) && (typeof key === 'string' || typeof key === 'number'))
return current[key];
return undefined;
}, value);
const normalizeIssue = ({ issue, data }: { issue: z.core.$ZodIssue; data: unknown }) => ({
code: issue.code,
path: issue.path.map((item) => String(item)).join('.'),
message: issue.message,
actualValue: compactValue(getValueByPath({ value: data, issuePath: issue.path }))
});
const emptyStats = (config: CollectionConfig): MutableCollectionStats => ({
collectionName: config.collectionName,
fieldName: config.fieldName,
scannedDocumentCount: 0,
skippedDocumentCount: 0,
convertedDocumentCount: 0,
zodErrorDocumentCount: 0,
writeSuccessDocumentCount: 0,
writeBlockedDocumentCount: 0,
writeErrorDocumentCount: 0,
issuesByPath: {}
});
const recordIssueSummary = ({
stats,
issues
}: {
stats: MutableCollectionStats;
issues: ValidationErrorRecordType['issues'];
}) => {
issues.forEach((issue) => {
const key = `${issue.path}|${issue.message}|${JSON.stringify(issue.actualValue)}`;
const existing = stats.issuesByPath[key];
stats.issuesByPath[key] = existing || {
path: issue.path,
message: issue.message,
actualValue: issue.actualValue,
count: 0
};
stats.issuesByPath[key].count += 1;
});
};
const serializeStats = (stats: MutableCollectionStats): CollectionStatsType => ({
...stats,
issuesByPath: Object.values(stats.issuesByPath).sort((left, right) => right.count - left.count)
});
const mergeStats = (statsList: MutableCollectionStats[]) => {
const total = emptyStats({
key: 'apps',
collectionName: 'total' as 'apps',
fieldName: '*' as 'modules'
});
total.collectionName = 'total';
total.fieldName = '*';
statsList.forEach((stats) => {
total.scannedDocumentCount += stats.scannedDocumentCount;
total.skippedDocumentCount += stats.skippedDocumentCount;
total.convertedDocumentCount += stats.convertedDocumentCount;
total.zodErrorDocumentCount += stats.zodErrorDocumentCount;
total.writeSuccessDocumentCount += stats.writeSuccessDocumentCount;
total.writeBlockedDocumentCount += stats.writeBlockedDocumentCount;
total.writeErrorDocumentCount += stats.writeErrorDocumentCount;
Object.values(stats.issuesByPath).forEach((issue) => {
const key = `${issue.path}|${issue.message}|${JSON.stringify(issue.actualValue)}`;
const existing = total.issuesByPath[key];
total.issuesByPath[key] = existing || { ...issue, count: 0 };
total.issuesByPath[key].count += issue.count;
});
});
return total;
};
const createErrorRecord = ({
doc,
config,
issues
}: {
doc: WorkflowDocument;
config: CollectionConfig;
issues: ValidationErrorRecordType['issues'];
}): ValidationErrorRecordType => ({
collectionName: config.collectionName,
fieldName: config.fieldName,
documentId: stringifyId(doc._id),
appId: stringifyId(doc.appId),
name: typeof doc.name === 'string' ? doc.name : undefined,
issueCount: issues.length,
issues
});
const validateUpgrade = ({
data,
doc,
config,
stats,
zodErrors,
maxZodErrors
}: {
data: unknown;
doc: WorkflowDocument;
config: CollectionConfig;
stats: MutableCollectionStats;
zodErrors: ValidationErrorRecordType[];
maxZodErrors: number;
}) => {
const result = PublishAppBodySchema.pick({
nodes: true,
edges: true,
chatConfig: true
}).safeParse(data);
if (result.success) return true;
const issues = result.error.issues.map((issue) => normalizeIssue({ issue, data }));
stats.zodErrorDocumentCount += 1;
recordIssueSummary({ stats, issues });
if (zodErrors.length < maxZodErrors) {
zodErrors.push(createErrorRecord({ doc, config, issues }));
}
return false;
};
const buildUpdatePayload = ({
config,
result
}: {
config: CollectionConfig;
result: UpgradeResult;
}) => ({
[config.fieldName]: result.nodes,
edges: result.edges,
chatConfig: result.chatConfig,
...(config.key === 'apps' ? { version: 'v2' } : {})
});
const processBatch = async ({
model,
docs,
config,
dryRun,
stats,
zodErrors,
maxZodErrors
}: {
model: Model<any>;
docs: WorkflowDocument[];
config: CollectionConfig;
dryRun: boolean;
stats: MutableCollectionStats;
zodErrors: ValidationErrorRecordType[];
maxZodErrors: number;
}) => {
const operations: PendingWriteOperation[] = [];
for (const doc of docs) {
stats.scannedDocumentCount += 1;
const upgradeResult = upgradeV1WorkflowDocument({ doc, config });
if (!upgradeResult.converted) {
stats.skippedDocumentCount += 1;
continue;
}
stats.convertedDocumentCount += 1;
const valid = validateUpgrade({
data: {
nodes: upgradeResult.nodes,
edges: upgradeResult.edges,
chatConfig: upgradeResult.chatConfig
},
doc,
config,
stats,
zodErrors,
maxZodErrors
});
if (!valid) {
stats.writeBlockedDocumentCount += 1;
continue;
}
if (dryRun) continue;
operations.push({
operation: {
updateOne: {
filter: { _id: doc._id },
update: {
$set: buildUpdatePayload({ config, result: upgradeResult })
}
}
}
});
}
if (operations.length === 0) return;
try {
const result = await model.bulkWrite(
operations.map(({ operation }) => operation),
{ ordered: false }
);
stats.writeSuccessDocumentCount += result.matchedCount;
stats.writeErrorDocumentCount += operations.length - result.matchedCount;
} catch {
stats.writeErrorDocumentCount += operations.length;
}
};
const appProjection = {
_id: 1,
appId: 1,
chatConfig: 1,
edges: 1,
name: 1,
type: 1,
version: 1,
modules: 1
};
const appVersionProjection = {
_id: 1,
appId: 1,
chatConfig: 1,
edges: 1,
name: 1,
version: 1,
nodes: 1
};
const buildAppScanQuery = ({ lastId }: { lastId?: unknown }) => {
const conditions: Record<string, unknown>[] = [
{
version: { $ne: 'v2' },
type: { $nin: AppFolderTypeList }
}
];
if (lastId) conditions.push({ _id: { $gt: lastId } });
return conditions.length === 1 ? conditions[0] : { $and: conditions };
};
/**
* 执行 V1 workflow 到 V2 的数据清洗。
*
* 流程与本地脚本一致：每批拉取 apps，先处理这些 app 对应的 app_versions，
* app_versions 写完后再写 apps，防止中断后 apps 已标记 v2 但版本漏处理。
*/
export async function runV1WorkflowToV2Migration({
dryRun = true
}: V1WorkflowToV2BodyType): Promise<V1WorkflowToV2ResponseType> {
const apps = emptyStats(appConfig);
const appVersions = emptyStats(appVersionConfig);
const zodErrors: ValidationErrorRecordType[] = [];
const processAppDocsWithVersions = async (appDocs: WorkflowDocument[]) => {
const appIds = appDocs
.map((doc) => toObjectId(doc._id))
.filter((id): id is Types.ObjectId => !!id);
if (appIds.length > 0) {
const appVersionDocs = await MongoAppVersion.find(
{
appId: { $in: appIds }
},
appVersionProjection
)
.sort({ _id: 1 })
.lean<WorkflowDocument[]>();
await processBatch({
model: MongoAppVersion,
docs: appVersionDocs,
config: appVersionConfig,
dryRun,
stats: appVersions,
zodErrors,
maxZodErrors: MAX_ZOD_ERRORS
});
}
await processBatch({
model: MongoApp,
docs: appDocs,
config: appConfig,
dryRun,
stats: apps,
zodErrors,
maxZodErrors: MAX_ZOD_ERRORS
});
};
let lastId: unknown;
while (true) {
const docs = await MongoApp.find(
buildAppScanQuery({
lastId
}),
appProjection
)
.sort({ _id: 1 })
.limit(BATCH_SIZE)
.lean<WorkflowDocument[]>();
if (docs.length === 0) break;
await processAppDocsWithVersions(docs);
lastId = docs[docs.length - 1]?._id;
}
const total = mergeStats([apps, appVersions]);
const result = {
dryRun,
apps: serializeStats(apps),
appVersions: serializeStats(appVersions),
total: serializeStats(total),
zodErrors
};
return V1WorkflowToV2ResponseSchema.parse(result);
}
async function handler(req: ApiRequestProps): Promise<V1WorkflowToV2ResponseType> {
await authCert({ req, authRoot: true });
const body = parseApiInput({
req,
bodySchema: V1WorkflowToV2BodySchema
}).body;
return runV1WorkflowToV2Migration(body);
}
export default NextAPI(handler);
projects/app/src/pages/api/v1/chat/completions.ts
...@@ -328,9 +328,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse) { ...@@ -328,9 +328,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
customFeedbacks, customFeedbacks,
nodeResponseSummary, nodeResponseSummary,
flatNodeResponses flatNodeResponses
} = await (async () => { } = await dispatchWorkFlow({
if (app.version === 'v2') {
return dispatchWorkFlow({
apiVersion: 'v1', apiVersion: 'v1',
req, req,
res, res,
...@@ -368,9 +366,6 @@ async function handler(req: NextApiRequest, res: NextApiResponse) { ...@@ -368,9 +366,6 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
retainInMemory: shouldCollectFinalResponseData retainInMemory: shouldCollectFinalResponseData
} }
}); });
}
return Promise.reject('您的工作流版本过低，请重新发布一次');
})();
const aiResponse: AIChatItemType & { dataId?: string } = { const aiResponse: AIChatItemType & { dataId?: string } = {
dataId: finalResponseChatItemId, dataId: finalResponseChatItemId,
......
projects/app/src/pages/api/v2/chat/completions.ts
...@@ -332,9 +332,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse) { ...@@ -332,9 +332,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
customFeedbacks, customFeedbacks,
nodeResponseSummary, nodeResponseSummary,
flatNodeResponses flatNodeResponses
} = await (async () => { } = await dispatchWorkFlow({
if (app.version === 'v2') {
return dispatchWorkFlow({
apiVersion: 'v2', apiVersion: 'v2',
res, res,
lang: getLocale(req), lang: getLocale(req),
...@@ -373,9 +371,6 @@ async function handler(req: NextApiRequest, res: NextApiResponse) { ...@@ -373,9 +371,6 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
retainInMemory: shouldCollectFinalResponseData retainInMemory: shouldCollectFinalResponseData
} }
}); });
}
return Promise.reject('您的工作流版本过低，请重新发布一次');
})();
const aiResponse: AIChatItemType & { dataId?: string } = { const aiResponse: AIChatItemType & { dataId?: string } = {
dataId: finalResponseChatItemId, dataId: finalResponseChatItemId,
......
projects/app/src/web/core/workflow/adapt.ts
import { import {
FlowNodeTemplateTypeEnum, FlowNodeTemplateTypeEnum,
NodeInputKeyEnum,
NodeOutputKeyEnum,
WorkflowIOValueTypeEnum WorkflowIOValueTypeEnum
} from '@fastgpt/global/core/workflow/constants'; } from '@fastgpt/global/core/workflow/constants';
import { import {
FlowNodeInputTypeEnum,
FlowNodeOutputTypeEnum, FlowNodeOutputTypeEnum,
FlowNodeTypeEnum FlowNodeTypeEnum
} from '@fastgpt/global/core/workflow/node/constant'; } from '@fastgpt/global/core/workflow/node/constant';
import type { FlowNodeItemType, StoreNodeItemType } from '@fastgpt/global/core/workflow/type/node'; import type { FlowNodeItemType, StoreNodeItemType } from '@fastgpt/global/core/workflow/type/node';
import type { FlowNodeTemplateType } from '@fastgpt/global/core/workflow/type/node'; import type { FlowNodeTemplateType } from '@fastgpt/global/core/workflow/type/node';
import { VARIABLE_NODE_ID } from '@fastgpt/global/core/workflow/constants'; import { VARIABLE_NODE_ID } from '@fastgpt/global/core/workflow/constants';
import { getHandleId } from '@fastgpt/global/core/workflow/utils';
import type { StoreEdgeItemType } from '@fastgpt/global/core/workflow/type/edge';
import type {
FlowNodeInputItemType,
FlowNodeOutputItemType
} from '@fastgpt/global/core/workflow/type/io';
import { getWorkflowGlobalVariables } from './utils'; import { getWorkflowGlobalVariables } from './utils';
import type { TFunction } from 'next-i18next'; import type { TFunction } from 'next-i18next';
import type { AppChatConfigType } from '@fastgpt/global/core/app/type'; import type { AppChatConfigType } from '@fastgpt/global/core/app/type';
...@@ -63,441 +54,3 @@ export const getGlobalVariableNode = ({ ...@@ -63,441 +54,3 @@ export const getGlobalVariableNode = ({
return variableNode; return variableNode;
}; };
/* adapt v1 workfwlo */
enum InputTypeEnum {
triggerAndFinish = 'triggerAndFinish',
systemInput = 'systemInput', // history, userChatInput, variableInput
input = 'input', // one line input
numberInput = 'numberInput',
select = 'select',
slider = 'slider',
target = 'target', // data input
switch = 'switch',
// editor
textarea = 'textarea',
JSONEditor = 'JSONEditor',
addInputParam = 'addInputParam', // params input
selectApp = 'selectApp',
// chat special input
aiSettings = 'aiSettings',
// ai model select
selectLLMModel = 'selectLLMModel',
settingLLMModel = 'settingLLMModel',
// dataset special input
selectDataset = 'selectDataset',
selectDatasetParamsModal = 'selectDatasetParamsModal',
settingDatasetQuotePrompt = 'settingDatasetQuotePrompt',
hidden = 'hidden',
custom = 'custom'
}
enum FlowTypeEnum {
userGuide = 'userGuide',
questionInput = 'questionInput',
chatNode = 'chatNode',
datasetSearchNode = 'datasetSearchNode',
datasetConcatNode = 'datasetConcatNode',
answerNode = 'answerNode',
classifyQuestion = 'classifyQuestion',
contentExtract = 'contentExtract',
httpRequest468 = 'httpRequest468',
runApp = 'app',
pluginModule = 'pluginModule',
pluginInput = 'pluginInput',
pluginOutput = 'pluginOutput',
queryExtension = 'cfr',
tools = 'tools',
stopTool = 'stopTool'
}
enum OutputTypeEnum {
answer = 'answer',
source = 'source',
hidden = 'hidden',
addOutputParam = 'addOutputParam'
}
type V1WorkflowType = {
name: string;
avatar?: string;
intro?: string;
moduleId: string;
position?: {
x: number;
y: number;
};
flowType: FlowTypeEnum;
showStatus?: boolean;
inputs: {
valueType?: WorkflowIOValueTypeEnum; // data type
type: InputTypeEnum; // Node Type. Decide on a render style
key: `${NodeInputKeyEnum}` | string;
value?: any;
label: string;
description?: string;
required?: boolean;
toolDescription?: string; // If this field is not empty, it is entered as a tool
edit?: boolean; // Whether to allow editing
editField?: {
inputType?: boolean;
required?: boolean;
isToolInput?: boolean;
name?: boolean;
key?: boolean;
description?: boolean;
dataType?: boolean;
defaultValue?: boolean;
};
defaultEditField?: {
inputType?: InputTypeEnum; // input type
outputType?: FlowNodeOutputTypeEnum;
required?: boolean;
key?: string;
label?: string;
description?: string;
valueType?: WorkflowIOValueTypeEnum;
isToolInput?: boolean;
defaultValue?: string;
};
connected?: boolean; // There are incoming data
showTargetInApp?: boolean;
showTargetInPlugin?: boolean;
hideInApp?: boolean;
hideInPlugin?: boolean;
placeholder?: string; // input,textarea
list?: { label: string; value: any }[]; // select
markList?: { label: string; value: any }[]; // slider
step?: number; // slider
max?: number; // slider, number input
min?: number; // slider, number input
}[];
outputs: {
type?: OutputTypeEnum;
key: `${NodeOutputKeyEnum}` | string;
valueType?: WorkflowIOValueTypeEnum;
label?: string;
description?: string;
required?: boolean;
defaultValue?: any;
edit?: boolean;
editField?: {
inputType?: boolean;
required?: boolean;
isToolInput?: boolean;
name?: boolean;
key?: boolean;
description?: boolean;
dataType?: boolean;
defaultValue?: boolean;
};
defaultEditField?: {
inputType?: `${FlowNodeInputTypeEnum}`; // input type
outputType?: FlowNodeOutputTypeEnum;
required?: boolean;
key?: string;
label?: string;
description?: string;
valueType?: `${WorkflowIOValueTypeEnum}`;
isToolInput?: boolean;
defaultValue?: string;
};
targets: { moduleId: string; key: string }[];
}[];
// runTime field
isEntry?: boolean;
parentId?: string;
};
export const v1Workflow2V2 = (
nodes: V1WorkflowType[]
): {
nodes: StoreNodeItemType[];
edges: StoreEdgeItemType[];
} => {
let copyNodes = JSON.parse(JSON.stringify(nodes)) as V1WorkflowType[];
// 只保留1个开始节点
copyNodes = copyNodes.filter((node, index, self) => {
if (node.flowType === FlowTypeEnum.questionInput) {
return index === self.findIndex((item) => item.flowType === FlowTypeEnum.questionInput);
}
return true;
});
const newNodes: StoreNodeItemType[] = copyNodes.map((node) => {
// flowNodeType adapt
const nodeTypeMap = {
[FlowTypeEnum.userGuide]: FlowNodeTypeEnum.systemConfig,
[FlowTypeEnum.questionInput]: FlowNodeTypeEnum.workflowStart,
[FlowTypeEnum.chatNode]: FlowNodeTypeEnum.chatNode,
[FlowTypeEnum.datasetSearchNode]: FlowNodeTypeEnum.datasetSearchNode,
[FlowTypeEnum.datasetConcatNode]: FlowNodeTypeEnum.datasetConcatNode,
[FlowTypeEnum.answerNode]: FlowNodeTypeEnum.answerNode,
[FlowTypeEnum.classifyQuestion]: FlowNodeTypeEnum.classifyQuestion,
[FlowTypeEnum.contentExtract]: FlowNodeTypeEnum.contentExtract,
[FlowTypeEnum.httpRequest468]: FlowNodeTypeEnum.httpRequest468,
[FlowTypeEnum.runApp]: FlowNodeTypeEnum.runApp,
[FlowTypeEnum.pluginModule]: FlowNodeTypeEnum.pluginModule,
[FlowTypeEnum.pluginInput]: FlowNodeTypeEnum.pluginInput,
[FlowTypeEnum.pluginOutput]: FlowNodeTypeEnum.pluginOutput,
[FlowTypeEnum.queryExtension]: FlowNodeTypeEnum.queryExtension,
[FlowTypeEnum.tools]: FlowNodeTypeEnum.toolCall,
[FlowTypeEnum.stopTool]: FlowNodeTypeEnum.stopTool
};
const inputTypeMap: Record<any, FlowNodeInputTypeEnum> = {
[InputTypeEnum.systemInput]: FlowNodeInputTypeEnum.input,
[InputTypeEnum.input]: FlowNodeInputTypeEnum.input,
[InputTypeEnum.numberInput]: FlowNodeInputTypeEnum.numberInput,
[InputTypeEnum.select]: FlowNodeInputTypeEnum.select,
[InputTypeEnum.target]: FlowNodeInputTypeEnum.reference,
[InputTypeEnum.switch]: FlowNodeInputTypeEnum.switch,
[InputTypeEnum.textarea]: FlowNodeInputTypeEnum.textarea,
[InputTypeEnum.JSONEditor]: FlowNodeInputTypeEnum.JSONEditor,
[InputTypeEnum.addInputParam]: FlowNodeInputTypeEnum.addInputParam,
[InputTypeEnum.selectApp]: FlowNodeInputTypeEnum.selectApp,
[InputTypeEnum.selectLLMModel]: FlowNodeInputTypeEnum.selectLLMModel,
[InputTypeEnum.settingLLMModel]: FlowNodeInputTypeEnum.settingLLMModel,
[InputTypeEnum.selectDataset]: FlowNodeInputTypeEnum.selectDataset,
[InputTypeEnum.selectDatasetParamsModal]: FlowNodeInputTypeEnum.selectDatasetParamsModal,
[InputTypeEnum.settingDatasetQuotePrompt]: FlowNodeInputTypeEnum.settingDatasetQuotePrompt,
[InputTypeEnum.hidden]: FlowNodeInputTypeEnum.hidden,
[InputTypeEnum.custom]: FlowNodeInputTypeEnum.custom
};
let pluginId: string | undefined = undefined;
const inputs = node.inputs
.map<FlowNodeInputItemType>((input) => {
const newInput: FlowNodeInputItemType = {
...input,
selectedTypeIndex: 0,
renderTypeList: !input.type
? [FlowNodeInputTypeEnum.custom]
: inputTypeMap[input.type]
? [inputTypeMap[input.type]]
: [],
key: input.key,
value: input.value,
valueType: input.valueType,
label: input.label,
description: input.description,
required: input.required,
toolDescription: input.toolDescription,
canEdit: input.edit,
placeholder: input.placeholder,
list: input.list,
markList: input.markList,
step: input.step,
max: input.max,
min: input.min
};
if (input.key === 'userChatInput') {
newInput.label = '问题输入';
} else if (input.key === 'quoteQA') {
newInput.label = '';
} else if (input.key === 'pluginId') {
pluginId = input.value;
}
return newInput;
})
.filter((input) => input.renderTypeList.length > 0)
.filter((input) => {
if (input.key === 'pluginId') {
return false;
}
if (input.key === 'switch') {
return false;
}
if (input.key === 'pluginStart') {
return false;
}
if (input.key === 'DYNAMIC_INPUT_KEY') return;
if (input.key === 'system_addInputParam') return;
return true;
});
const outputTypeMap: Record<any, FlowNodeOutputTypeEnum> = {
[OutputTypeEnum.addOutputParam]: FlowNodeOutputTypeEnum.dynamic,
[OutputTypeEnum.answer]: FlowNodeOutputTypeEnum.static,
[OutputTypeEnum.source]: FlowNodeOutputTypeEnum.static,
[OutputTypeEnum.hidden]: FlowNodeOutputTypeEnum.hidden
};
const outputs = node.outputs
.map<FlowNodeOutputItemType>((output) => ({
id: output.key,
type: output.type ? outputTypeMap[output.type] : FlowNodeOutputTypeEnum.static,
key: output.key,
valueType: output.valueType,
label: output.label,
description: output.description,
required: output.required,
defaultValue: output.defaultValue,
canEdit: output.edit,
editField: output.editField
}))
.filter((output) => {
if (node.flowType === FlowTypeEnum.pluginOutput) return false;
if (output.key === 'finish') return false;
if (output.key === 'isEmpty') return false;
if (output.key === 'unEmpty') return false;
if (output.key === 'pluginStart') return false;
if (node.flowType !== FlowTypeEnum.questionInput && output.key === 'userChatInput')
return false;
if (
node.flowType === FlowTypeEnum.contentExtract &&
(output.key === 'success' || output.key === 'failed')
)
return;
return true;
});
// special node
if (node.flowType === FlowTypeEnum.questionInput) {
node.name = '流程开始';
} else if (node.flowType === FlowTypeEnum.pluginOutput) {
node.outputs.forEach((output) => {
inputs.push({
key: output.key,
valueType: output.valueType,
renderTypeList: [FlowNodeInputTypeEnum.reference],
label: output.key,
canEdit: true
});
});
}
return {
nodeId: node.moduleId,
position: node.position,
flowNodeType: nodeTypeMap[node.flowType],
avatar: node.flowType === FlowTypeEnum.pluginModule ? node.avatar : undefined,
name: node.name,
intro: node.intro,
showStatus: node.showStatus,
pluginId,
parentId: node.parentId,
version: '481',
inputs,
outputs
};
});
let newEdges: StoreEdgeItemType[] = [];
// 遍历output，连线
copyNodes.forEach((node) => {
node.outputs.forEach((output) => {
output.targets?.forEach((target) => {
if (output.key === 'finish') return;
if (output.key === 'isEmpty') return;
if (output.key === 'unEmpty') return;
if (node.flowType !== FlowTypeEnum.questionInput && output.key === 'userChatInput') return;
if (output.key === NodeOutputKeyEnum.selectedTools) {
newEdges.push({
source: node.moduleId,
sourceHandle: NodeOutputKeyEnum.selectedTools,
target: target.moduleId,
targetHandle: NodeOutputKeyEnum.selectedTools
});
} else if (node.flowType === FlowTypeEnum.classifyQuestion) {
newEdges.push({
source: node.moduleId,
sourceHandle: getHandleId(node.moduleId, 'source', output.key),
target: target.moduleId,
targetHandle: getHandleId(target.moduleId, 'target', 'left')
});
} else if (node.flowType === FlowTypeEnum.contentExtract) {
} else {
newEdges.push({
source: node.moduleId,
sourceHandle: getHandleId(node.moduleId, 'source', 'right'),
target: target.moduleId,
targetHandle: getHandleId(target.moduleId, 'target', 'left')
});
}
});
});
});
// 去除相同source和target的线
newEdges = newEdges.filter((edge, index, self) => {
return (
self.findIndex((item) => item.source === edge.source && item.target === edge.target) === index
);
});
const workflowStart = newNodes.find(
(node) => node.flowNodeType === FlowNodeTypeEnum.workflowStart
);
/* 更新input的取值 */
copyNodes.forEach((node) => {
node.outputs.forEach((output) => {
output.targets?.forEach((target) => {
const targetNode = newNodes.find((item) => item.nodeId === target.moduleId);
if (!targetNode) return;
const targetInput = targetNode.inputs.find((item) => item.key === target.key);
if (!targetInput) return;
targetInput.value = [node.moduleId, output.key];
});
});
});
// 更新特殊的输入(输入全部从开始取)
newNodes.forEach((node) => {
node.inputs.forEach((input) => {
if (workflowStart) {
if (
node.flowNodeType === FlowNodeTypeEnum.datasetSearchNode &&
input.key === NodeInputKeyEnum.datasetSearchInput
) {
input.value = [
[workflowStart.nodeId, NodeOutputKeyEnum.userChatInput],
[workflowStart.nodeId, NodeOutputKeyEnum.userFiles]
];
input.valueType = WorkflowIOValueTypeEnum.arrayString;
return;
}
if (input.key !== NodeInputKeyEnum.userChatInput) return;
input.value = [workflowStart.nodeId, NodeOutputKeyEnum.userChatInput];
}
});
});
console.log({
nodes: newNodes.filter((node) => node.nodeId),
edges: newEdges
});
return {
nodes: newNodes.filter((node) => node.nodeId),
edges: newEdges
};
};
projects/app/test/pages/api/admin/dataClean/initWorkflowData.test.ts 0 → 100644
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { AppTypeEnum } from '@fastgpt/global/core/app/constants';
import {
FlowNodeInputTypeEnum,
FlowNodeOutputTypeEnum,
FlowNodeTypeEnum
} from '@fastgpt/global/core/workflow/node/constant';
import { WorkflowIOValueTypeEnum } from '@fastgpt/global/core/workflow/constants';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { MongoAppVersion } from '@fastgpt/service/core/app/version/schema';
vi.mock('@/service/middleware/entry', () => ({
NextAPI: (handler: unknown) => handler
}));
vi.mock('@fastgpt/service/support/permission/auth/common', () => ({
authCert: vi.fn()
}));
import {
formatWorkflowDocument,
runInitWorkflowDataMigration
} from '@/pages/api/admin/dataClean/initWorkflowData';
const teamId = '65f000000000000000000051';
const tmbId = '65f000000000000000000052';
const appId = '65f000000000000000000053';
const dirtyV2Nodes = [
{
nodeId: 'start',
flowNodeType: FlowNodeTypeEnum.workflowStart,
name: 'Start',
inputs: null,
outputs: [
{
key: 'userChatInput',
type: 'FlowNodeOutputTypeEnum.static',
valueType: 'WorkflowIOValueTypeEnum.string'
}
]
},
{
moduleId: 'legacy-laf',
flowNodeType: 'lafModule',
name: null,
inputs: [
{
key: 'hidden',
label: null,
renderTypeList: ['FlowNodeInputTypeEnum.hidden'],
valueType: 'WorkflowIOValueTypeEnum.string',
description: null
}
],
outputs: []
}
];
describe('initWorkflowData data clean API', () => {
beforeEach(async () => {
await Promise.all([MongoApp.deleteMany({}), MongoAppVersion.deleteMany({})]);
});
it('formats dirty v2 workflow data and keeps save schema valid', () => {
const stats = {
collectionName: 'apps',
fieldName: 'modules',
queryMatchedDocumentCount: null,
scannedDocumentCount: 0,
fixableDocumentCount: 0,
unknownDocumentCount: 0,
enumExpressionCount: 0,
renderTypeListFixableCount: 0,
outputTypeFixableCount: 0,
valueTypeFixableCount: 0,
unknownEnumExpressionCount: 0,
saveApiValidationErrorDocumentCount: 0,
cleanErrorDocumentCount: 0,
formatChangedDocumentCount: 0,
writeSuccessDocumentCount: 0,
writeBlockedDocumentCount: 0,
writeErrorDocumentCount: 0,
byExpression: {}
};
const result = formatWorkflowDocument({
doc: {
_id: appId,
modules: dirtyV2Nodes,
edges: null,
chatConfig: {
questionGuide: true,
variables: [
{
key: 'topic',
label: 'topic',
type: 'string',
enums: '[{\"value\":\"a\"}]',
maxLength: -1
}
],
scheduledTriggerConfig: null
}
},
fieldName: 'modules',
stats,
docContext: {
collectionName: 'apps',
documentId: appId
},
rootPath: 'modules'
});
expect(result.nodes).toEqual(
expect.arrayContaining([
expect.objectContaining({
nodeId: 'legacy-laf',
flowNodeType: FlowNodeTypeEnum.emptyNode,
inputs: expect.arrayContaining([
expect.objectContaining({
renderTypeList: [FlowNodeInputTypeEnum.hidden],
valueType: WorkflowIOValueTypeEnum.string,
label: 'hidden'
})
])
}),
expect.objectContaining({
outputs: expect.arrayContaining([
expect.objectContaining({
type: FlowNodeOutputTypeEnum.static,
valueType: WorkflowIOValueTypeEnum.string
})
])
})
])
);
expect(result.edges).toEqual([]);
expect(result.chatConfig).toMatchObject({
questionGuide: { open: true },
variables: [
{
type: 'input',
description: '',
enums: [{ value: 'a', label: 'a' }]
}
]
});
expect(result.chatConfig).not.toHaveProperty('scheduledTriggerConfig');
expect(result.formatChanges.count).toBeGreaterThan(0);
});
it('dry-runs without writing formatted data', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'dirty app',
type: AppTypeEnum.workflow,
version: 'v2',
modules: dirtyV2Nodes,
edges: null,
chatConfig: null
});
const result = await runInitWorkflowDataMigration({
dryRun: true
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 1,
fixableDocumentCount: 1,
saveApiValidationErrorDocumentCount: 0,
formatChangedDocumentCount: 1,
writeSuccessDocumentCount: 0
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
modules: expect.arrayContaining([expect.objectContaining({ flowNodeType: 'lafModule' })]),
chatConfig: null
});
});
it('writes only formatted and zod-valid documents', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'dirty app',
type: AppTypeEnum.workflow,
version: 'v2',
modules: dirtyV2Nodes,
edges: null,
chatConfig: null
});
const result = await runInitWorkflowDataMigration({
dryRun: false
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 1,
fixableDocumentCount: 1,
saveApiValidationErrorDocumentCount: 0,
writeSuccessDocumentCount: 1
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
modules: expect.arrayContaining([
expect.objectContaining({
nodeId: 'legacy-laf',
flowNodeType: FlowNodeTypeEnum.emptyNode
})
]),
edges: [],
chatConfig: {}
});
});
it('skips folder-like apps when cleaning app workflow data', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'folder app',
type: AppTypeEnum.folder,
modules: null,
edges: null,
chatConfig: null
});
const result = await runInitWorkflowDataMigration({
dryRun: false
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 0,
formatChangedDocumentCount: 0,
writeSuccessDocumentCount: 0
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
type: AppTypeEnum.folder,
modules: null,
edges: null,
chatConfig: null
});
});
it('fills missing system tool set child toolId before zod validation', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'system tool set app',
type: AppTypeEnum.workflow,
version: 'v2',
modules: [
...dirtyV2Nodes,
{
nodeId: 'toolSet',
flowNodeType: FlowNodeTypeEnum.toolSet,
name: 'Tool Set',
inputs: [],
outputs: [],
toolConfig: {
systemToolSet: {
toolId: 'system-tool-set',
toolList: [
{
key: 'searchByKey',
description: 'Search tool'
},
{
name: 'read',
description: 'Read tool'
}
]
}
}
}
],
edges: [],
chatConfig: null
});
const result = await runInitWorkflowDataMigration({
dryRun: false
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 1,
saveApiValidationErrorDocumentCount: 0,
writeSuccessDocumentCount: 1
});
const app = await MongoApp.findById(appId).lean();
const toolSetNode = app?.modules.find((node) => node.nodeId === 'toolSet');
expect(toolSetNode?.toolConfig?.systemToolSet?.toolList).toEqual([
expect.objectContaining({
name: 'searchByKey',
toolId: 'searchByKey'
}),
expect.objectContaining({
name: 'read',
toolId: 'read'
})
]);
});
it('blocks writes when formatted data still fails zod parse', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'invalid app',
type: AppTypeEnum.workflow,
version: 'v2',
modules: dirtyV2Nodes,
edges: [],
chatConfig: {
variables: ['invalid']
}
});
const result = await runInitWorkflowDataMigration({
dryRun: false
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 1,
saveApiValidationErrorDocumentCount: 1,
writeBlockedDocumentCount: 1,
writeSuccessDocumentCount: 0
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
modules: expect.arrayContaining([expect.objectContaining({ flowNodeType: 'lafModule' })])
});
});
});
projects/app/test/pages/api/admin/dataClean/v1WorkflowToV2.test.ts 0 → 100644
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { AppTypeEnum } from '@fastgpt/global/core/app/constants';
import {
FlowNodeInputTypeEnum,
FlowNodeTypeEnum
} from '@fastgpt/global/core/workflow/node/constant';
import { WorkflowIOValueTypeEnum } from '@fastgpt/global/core/workflow/constants';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { MongoAppVersion } from '@fastgpt/service/core/app/version/schema';
vi.mock('@/service/middleware/entry', () => ({
NextAPI: (handler: unknown) => handler
}));
vi.mock('@fastgpt/service/support/permission/auth/common', () => ({
authCert: vi.fn()
}));
import {
runV1WorkflowToV2Migration,
upgradeV1WorkflowDocument
} from '@/pages/api/admin/dataClean/v1WorkflowToV2';
const teamId = '65f000000000000000000041';
const tmbId = '65f000000000000000000042';
const appId = '65f000000000000000000043';
const legacyV1Nodes = [
{
moduleId: 'start',
flowType: 'questionInput',
name: '开始',
inputs: [
{
key: 'userChatInput',
type: 'input',
valueType: 'string'
}
],
outputs: [
{
key: 'userChatInput',
type: 'source',
valueType: 'string',
targets: [{ moduleId: 'chat', key: 'userChatInput' }]
}
]
},
{
moduleId: 'chat',
flowType: 'chatNode',
inputs: [
{
key: 'userChatInput',
type: 'target',
valueType: 'chat_history'
},
{
key: 'dirty',
type: 'hidden',
valueType: 'tools',
description: null,
toolDescription: null
}
],
outputs: [
{
key: 'answer',
type: 'answer',
valueType: 'kb_quote'
}
]
},
{
moduleId: 'legacy-laf',
flowType: 'lafModule',
inputs: [],
outputs: []
}
];
describe('v1WorkflowToV2 data clean API', () => {
beforeEach(async () => {
await Promise.all([MongoApp.deleteMany({}), MongoAppVersion.deleteMany({})]);
});
it('converts one legacy v1 workflow document and keeps it publish-schema valid', () => {
const result = upgradeV1WorkflowDocument({
config: {
key: 'apps',
collectionName: 'apps',
fieldName: 'modules'
},
doc: {
_id: appId,
type: AppTypeEnum.workflow,
version: 'v1',
modules: legacyV1Nodes,
edges: [],
chatConfig: {
questionGuide: true,
variables: [
{
key: 'topic',
label: 'topic',
type: 'select',
list: [{ value: 'a' }]
}
],
scheduledTriggerConfig: null
}
}
});
expect(result.converted).toBe(true);
expect(result.nodes).toEqual(
expect.arrayContaining([
expect.objectContaining({
nodeId: 'start',
flowNodeType: FlowNodeTypeEnum.workflowStart
}),
expect.objectContaining({
nodeId: 'chat',
flowNodeType: FlowNodeTypeEnum.chatNode,
inputs: expect.arrayContaining([
expect.objectContaining({
key: 'userChatInput',
valueType: WorkflowIOValueTypeEnum.chatHistory,
value: ['start', 'userChatInput']
}),
expect.objectContaining({
key: 'dirty',
renderTypeList: [FlowNodeInputTypeEnum.hidden],
valueType: WorkflowIOValueTypeEnum.any,
description: '',
toolDescription: ''
})
]),
outputs: expect.arrayContaining([
expect.objectContaining({
key: 'answer',
valueType: WorkflowIOValueTypeEnum.datasetQuote
})
])
}),
expect.objectContaining({
nodeId: 'legacy-laf',
flowNodeType: FlowNodeTypeEnum.emptyNode
})
])
);
expect(result.chatConfig).toMatchObject({
questionGuide: { open: true },
variables: [
{
description: '',
list: [{ value: 'a', label: 'a' }]
}
]
});
expect(result.chatConfig).not.toHaveProperty('scheduledTriggerConfig');
});
it('dry-runs apps and app_versions without writing converted data', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'legacy app',
type: AppTypeEnum.workflow,
version: 'v1',
modules: legacyV1Nodes,
edges: [],
chatConfig: null
});
await MongoAppVersion.create({
appId,
tmbId,
nodes: legacyV1Nodes,
edges: [],
chatConfig: null
});
const result = await runV1WorkflowToV2Migration({
dryRun: true
});
expect(result).toMatchObject({
dryRun: true,
apps: {
scannedDocumentCount: 1,
convertedDocumentCount: 1,
zodErrorDocumentCount: 0,
writeSuccessDocumentCount: 0
},
appVersions: {
scannedDocumentCount: 1,
convertedDocumentCount: 1,
zodErrorDocumentCount: 0,
writeSuccessDocumentCount: 0
}
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
version: 'v1',
modules: expect.arrayContaining([expect.objectContaining({ flowType: 'questionInput' })]),
chatConfig: null
});
});
it('writes app_versions before apps and marks apps v2 only after zod validation', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'legacy app',
type: AppTypeEnum.workflow,
version: 'v1',
modules: legacyV1Nodes,
edges: [],
chatConfig: {
variables: [{ key: 'invalid-variable' }]
}
});
await MongoAppVersion.create({
appId,
tmbId,
nodes: legacyV1Nodes,
edges: [],
chatConfig: null
});
const result = await runV1WorkflowToV2Migration({
dryRun: false
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 1,
convertedDocumentCount: 1,
zodErrorDocumentCount: 1,
writeBlockedDocumentCount: 1,
writeSuccessDocumentCount: 0
});
expect(result.appVersions).toMatchObject({
scannedDocumentCount: 1,
convertedDocumentCount: 1,
zodErrorDocumentCount: 0,
writeSuccessDocumentCount: 1
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
version: 'v1',
modules: expect.arrayContaining([expect.objectContaining({ flowType: 'questionInput' })])
});
await expect(MongoAppVersion.findOne({ appId }).lean()).resolves.toMatchObject({
nodes: expect.arrayContaining([
expect.objectContaining({
nodeId: 'start',
flowNodeType: FlowNodeTypeEnum.workflowStart
})
]),
chatConfig: {}
});
});
});
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment