Commit 402ad3ff by Archer Committed by GitHub

perf: init workflow (#7210)

* perf: init workflow

* doc

* perf: parse

* fix: ts
parent 9981049b
...@@ -78,14 +78,14 @@ SANDBOX_QUEUE_ID_CONCURRENCY= ...@@ -78,14 +78,14 @@ SANDBOX_QUEUE_ID_CONCURRENCY=
1. 下载所有系统工具的 [zip 包](<https://github.com/labring/fastgpt-img/raw/refs/heads/main/fastgpt-official-plugins(1).zip>)。 1. 下载所有系统工具的 [zip 包](<https://github.com/labring/fastgpt-img/raw/refs/heads/main/fastgpt-official-plugins(1).zip>)。
2. 打开 `fastgpt` 网页 - 点击 `管理员` navbar - 点击添加插件 - 点击 `导入/更新插件` - 上传 zip - 确认。 2. 打开 `fastgpt` 网页 - 点击 `管理员` navbar - 点击添加插件 - 点击 `导入/更新插件` - 上传 zip - 确认。
也可以打开插件市场逐个下载,正式版之前,插件市场地址为: [https://v2.marketplace.fastgpt.cn](https://v2.marketplace.fastgpt.cn) 也可以打开插件市场逐个下载,插件市场地址为: [https://v2.marketplace.fastgpt.cn](https://v2.marketplace.fastgpt.cn),环境变量默认值已变成该地址,不设置环境变量即可。
### 4. 升级脚本 ### 4. 升级脚本
如使用旧版沙盒 workspace,可通过下面接口将旧沙盒 workspace 归档到 S3,从而更彻底地释放不活跃沙盒。该脚本仅影响旧的沙盒,不影响新生成的沙盒;不执行该脚本,直接移除旧沙盒也可以。 如使用旧版沙盒 workspace,可通过下面接口将旧沙盒 workspace 归档到 S3,从而更彻底地释放不活跃沙盒。该脚本仅影响旧的沙盒,不影响新生成的沙盒;不执行该脚本,直接移除旧沙盒也可以。
```shell ```shell
curl --location --request POST 'https://{{host}}/api/admin/initSandboxArchive' \ curl --location --request POST 'https://{{host}}/api/admin/dataClean/initSandboxArchive' \
--header 'rootkey: {{rootkey}}' \ --header 'rootkey: {{rootkey}}' \
--header 'Content-Type: application/json' \ --header 'Content-Type: application/json' \
-d '{"runArchive":true,"inactiveDays":0}' -d '{"runArchive":true,"inactiveDays":0}'
......
---
title: 'V4.15.0-beta7'
description: 'FastGPT V4.15.0-beta7 Release Notes'
---
## 📦 Upgrade Guide
### 1. Run the Workflow V1 to V2 Migration
Starting from V4.15.0-beta7, workflow save payloads use the V2 structure consistently. Historical `apps.modules` and `app_versions.nodes` records may still use the V1 structure. Run the V1 -> V2 migration first, then run the V2 dirty-data cleanup in the next step.
Migration script path: `projects/app/src/pages/api/admin/dataClean/v1WorkflowToV2.ts`. This endpoint is only for this upgrade migration and is not a public OpenAPI endpoint.
The endpoint uses dry-run mode by default. It scans, converts in memory, and validates with `PublishAppBodySchema` without writing to MongoDB:
```bash
curl -X POST 'https://your-domain/api/admin/dataClean/v1WorkflowToV2' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":true}'
```
After confirming the returned statistics, set `dryRun` to `false` to write the converted data:
```bash
curl -X POST 'https://your-domain/api/admin/dataClean/v1WorkflowToV2' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":false}'
```
Request parameters:
| Parameter | Type | Default | Description |
| --------- | ------- | ------- | ---------------------------------------------------------- |
| `dryRun` | boolean | `true` | Whether to scan and validate only without writing changes. |
Migration behavior:
1. Scans apps where `apps.version != 'v2'` and `type` is not `folder`, `httpPlugin`, or `toolFolder`.
2. For each batch of `apps`, converts and writes related `app_versions` first, then converts and writes `apps`, so a rerun will not miss historical versions after an interruption.
3. Converts V1 node fields to V2 fields, for example `moduleId` -> `nodeId` and `flowType` -> `flowNodeType`.
4. Unknown node types fall back to `emptyNode`, and invalid `valueType` values are converted to `any`.
5. Missing `node.name` falls back to `flowType`, and missing `input.label` falls back to `input.key`.
6. Before writing, the script validates `nodes`, `edges`, and `chatConfig` with `PublishAppBodySchema`. Documents that fail validation are not written and are included in the endpoint response.
### 2. Run the Workflow V2 Enum and Structure Cleanup
Some historical workflow nodes may have stored TypeScript enum expression strings directly in MongoDB, for example:
```json
{
"renderTypeList": ["FlowNodeInputTypeEnum.hidden"],
"valueType": "WorkflowIOValueTypeEnum.any"
}
```
The correct stored values are:
```json
{
"renderTypeList": ["hidden"],
"valueType": "any"
}
```
This dirty data can affect workflow node input rendering and IO type checks. After running the V1 -> V2 migration, continue with the V2 cleanup script to scan and fix `apps.modules` and `app_versions.nodes`.
Migration script path: `projects/app/src/pages/api/admin/dataClean/initWorkflowData.ts`. This endpoint is only for this upgrade migration and is not a public OpenAPI endpoint.
The endpoint uses dry-run mode by default. It formats data in memory and validates with `PublishAppBodySchema` without writing to MongoDB:
```bash
curl -X POST 'https://your-domain/api/admin/dataClean/initWorkflowData' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":true,"batchSize":1000,"writeBatchSize":10}'
```
After confirming the returned statistics, set `dryRun` to `false` to write the cleanup:
```bash
curl -X POST 'https://your-domain/api/admin/dataClean/initWorkflowData' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":false,"batchSize":1000,"writeBatchSize":10}'
```
Request parameters:
| Parameter | Type | Default | Description |
| ---------------- | ------- | ------- | ------------------------------------------------------------------------------- |
| `dryRun` | boolean | `true` | Whether to scan and validate only without writing changes. |
| `batchSize` | number | `1000` | Documents fetched per page. |
| `writeBatchSize` | number | `10` | Documents written per `bulkWrite`. Lower it when online write pressure is high. |
Cleanup behavior:
1. Scans workflow data in `apps` and `app_versions` in batches to reduce read and write pressure.
2. Formats each workflow document once, covering historical dirty fields, null values, enum expressions, and legacy structure compatibility.
3. After formatting, validates the save payload fields `nodes`, `edges`, and `chatConfig` with `PublishAppBodySchema`.
4. Documents that fail Zod validation are only recorded in the response and are not written to MongoDB.
5. In non-dry-run mode, only documents that changed during formatting and passed Zod validation are written. Unchanged documents are not written again.
The response includes separate statistics for `apps`, `appVersions`, and `total`, including scanned documents, fixable documents, Zod error count, successful writes, failed writes, enum expression statistics, format samples, and error samples.
### 3. Update Images
- Update the fastgpt-app (FastGPT main service) image tag to v4.15.0-beta7.
- Update the fastgpt-pro (FastGPT commercial edition) image tag to v4.15.0-beta7.
- Update the fastgpt-plugin image tag to v1.0.0-beta7.
## 🐛 Fixes
1. Fixed historical V1 workflow data that could fail validation under the new save payload structure.
2. Fixed dirty `FlowNodeInputTypeEnum.*`, `FlowNodeOutputTypeEnum.*`, and `WorkflowIOValueTypeEnum.*` expression strings in workflow nodes that could break input rendering and IO type checks.
---
title: 'V4.15.0-beta7'
description: 'FastGPT V4.15.0-beta7 更新说明'
---
## 📦 升级指南
### 1. 执行工作流 V1 升级 V2 迁移(可选)
该步骤仅需部署过 `<4.8` 版本 FastGPT 的用户执行。
V4.15.0-beta7 后工作流保存结构统一使用 V2。历史 `apps.modules` 与 `app_versions.nodes` 中可能仍存在 V1 结构,升级后建议先执行 V1 -> V2 迁移,再执行后续 V2 脏数据清洗。
迁移脚本位置:`projects/app/src/pages/api/admin/dataClean/v1WorkflowToV2.ts`。该接口仅用于本次升级迁移,不作为 OpenAPI 对外接口。
接口默认 dry-run,只扫描、转换和执行 `PublishAppBodySchema` 校验,不写库:
```bash
curl -X POST 'https://你的域名/api/admin/dataClean/v1WorkflowToV2' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":true}'
```
确认返回统计无误后,将 `dryRun` 改为 `false` 执行写入:
```bash
curl -X POST 'https://你的域名/api/admin/dataClean/v1WorkflowToV2' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":false}'
```
接口参数:
| 参数 | 类型 | 默认值 | 说明 |
| -------- | ------- | ------ | ---------------------- |
| `dryRun` | boolean | `true` | 是否只扫描验证不写库。 |
迁移逻辑:
1. 按 `apps.version != 'v2'` 且 `type` 非 `folder`、`httpPlugin`、`toolFolder` 扫描应用。
2. 对每批 `apps`,先转换并写入对应 `app_versions`,再转换并写入 `apps`,避免中断后遗漏历史版本。
3. 将 V1 节点字段升级为 V2 节点字段,例如 `moduleId` -> `nodeId`、`flowType` -> `flowNodeType`。
4. 未知节点类型会兜底为 `emptyNode`,非法 `valueType` 会转为 `any`。
5. 缺失 `node.name` 时用 `flowType` 兜底,缺失 `input.label` 时用 `input.key` 兜底。
6. 写库前使用 `PublishAppBodySchema` 校验 `nodes`、`edges`、`chatConfig`,校验失败的文档不会写入,并会记录到接口返回结果。
### 2. 执行工作流 V2 枚举与结构脏数据清洗
部分历史工作流节点可能把 TypeScript 枚举表达式字符串直接写入 MongoDB,例如:
```json
{
"renderTypeList": ["FlowNodeInputTypeEnum.hidden"],
"valueType": "WorkflowIOValueTypeEnum.any"
}
```
正确落库值应为:
```json
{
"renderTypeList": ["hidden"],
"valueType": "any"
}
```
该脏数据会影响工作流节点输入渲染和 IO 类型判断。执行 V1 -> V2 迁移后,继续执行 V2 清洗脚本,扫描并修复 `apps.modules` 与 `app_versions.nodes`。
迁移脚本位置:`projects/app/src/pages/api/admin/dataClean/initWorkflowData.ts`。该接口仅用于本次升级迁移,不作为 OpenAPI 对外接口。
接口默认 dry-run,只格式化内存数据并执行 `PublishAppBodySchema` 校验,不写库:
```bash
curl -X POST 'https://你的域名/api/admin/dataClean/initWorkflowData' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":true,"batchSize":1000,"writeBatchSize":10}'
```
确认返回统计无误后,将 `dryRun` 改为 `false` 执行写入:
```bash
curl -X POST 'https://你的域名/api/admin/dataClean/initWorkflowData' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":false,"batchSize":1000,"writeBatchSize":10}'
```
接口参数:
| 参数 | 类型 | 默认值 | 说明 |
| ---------------- | ------- | ------ | ----------------------------------------------------- |
| `dryRun` | boolean | `true` | 是否只扫描验证不写库。 |
| `batchSize` | number | `1000` | 每批读取文档数量。 |
| `writeBatchSize` | number | `10` | 每次 `bulkWrite` 的文档数量。线上写入压力大时可调小。 |
清洗逻辑:
1. 按批扫描 `apps` 和 `app_versions` 中的工作流数据,降低单次读取和写入压力。
2. 对每条工作流数据执行一次格式化,统一修复历史脏字段、空值、枚举表达式和旧结构兼容问题。
3. 格式化后使用 `PublishAppBodySchema` 校验保存接口实际关心的 `nodes`、`edges`、`chatConfig`。
4. Zod 校验失败的文档只记录在返回结果中,不会写入数据库。
5. 非 dry-run 时,只写入“发生过格式化变更,且 Zod 校验通过”的文档;未变化文档不会重复写库。
返回结果会分别展示 `apps`、`appVersions` 和 `total` 的统计,包括扫描文档数、可修复文档数、Zod 错误数量、写入成功数量、写入失败数量、枚举表达式统计、变更样本和错误样本。
### 3. 更新镜像
- 更新 fastgpt-app(fastgpt 主服务) 镜像 tag: v4.15.0-beta7
- 更新 fastgpt-pro(fastgpt 商业版) 镜像 tag: v4.15.0-beta7
- 更新 fastgpt-plugin 镜像 tag: v1.0.0-beta7
## 🐛 修复
1. 修复历史 V1 工作流数据在新版保存结构下无法通过校验的问题。
2. 修复工作流节点配置中 `FlowNodeInputTypeEnum.*`、`FlowNodeOutputTypeEnum.*` 和 `WorkflowIOValueTypeEnum.*` 枚举表达式字符串脏数据导致输入渲染和 IO 类型判断异常的问题。
3. AgentV2 mcp 拿不到 schema。
{ {
"title": "4.15.x", "title": "4.15.x",
"description": "", "description": "",
"pages": ["41500", "41506", "41505", "41504", "41503", "41502", "41501"] "pages": ["41500", "41507", "41506", "41505", "41504", "41503", "41502", "41501"]
} }
{ {
"title": "4.15.x", "title": "4.15.x",
"description": "", "description": "",
"pages": ["41500", "41506", "41505", "41504", "41503", "41502", "41501"] "pages": ["41500", "41507", "41506", "41505", "41504", "41503", "41502", "41501"]
} }
...@@ -154,6 +154,7 @@ description: FastGPT Toc ...@@ -154,6 +154,7 @@ description: FastGPT Toc
- [/en/self-host/upgrading/4-15/41504](/en/self-host/upgrading/4-15/41504) - [/en/self-host/upgrading/4-15/41504](/en/self-host/upgrading/4-15/41504)
- [/en/self-host/upgrading/4-15/41505](/en/self-host/upgrading/4-15/41505) - [/en/self-host/upgrading/4-15/41505](/en/self-host/upgrading/4-15/41505)
- [/en/self-host/upgrading/4-15/41506](/en/self-host/upgrading/4-15/41506) - [/en/self-host/upgrading/4-15/41506](/en/self-host/upgrading/4-15/41506)
- [/en/self-host/upgrading/4-15/41507](/en/self-host/upgrading/4-15/41507)
- [/en/self-host/upgrading/outdated/40](/en/self-host/upgrading/outdated/40) - [/en/self-host/upgrading/outdated/40](/en/self-host/upgrading/outdated/40)
- [/en/self-host/upgrading/outdated/41](/en/self-host/upgrading/outdated/41) - [/en/self-host/upgrading/outdated/41](/en/self-host/upgrading/outdated/41)
- [/en/self-host/upgrading/outdated/4100](/en/self-host/upgrading/outdated/4100) - [/en/self-host/upgrading/outdated/4100](/en/self-host/upgrading/outdated/4100)
......
...@@ -157,6 +157,7 @@ description: FastGPT 文档目录 ...@@ -157,6 +157,7 @@ description: FastGPT 文档目录
- [/self-host/upgrading/4-15/41504](/self-host/upgrading/4-15/41504) - [/self-host/upgrading/4-15/41504](/self-host/upgrading/4-15/41504)
- [/self-host/upgrading/4-15/41505](/self-host/upgrading/4-15/41505) - [/self-host/upgrading/4-15/41505](/self-host/upgrading/4-15/41505)
- [/self-host/upgrading/4-15/41506](/self-host/upgrading/4-15/41506) - [/self-host/upgrading/4-15/41506](/self-host/upgrading/4-15/41506)
- [/self-host/upgrading/4-15/41507](/self-host/upgrading/4-15/41507)
- [/self-host/upgrading/outdated/40](/self-host/upgrading/outdated/40) - [/self-host/upgrading/outdated/40](/self-host/upgrading/outdated/40)
- [/self-host/upgrading/outdated/41](/self-host/upgrading/outdated/41) - [/self-host/upgrading/outdated/41](/self-host/upgrading/outdated/41)
- [/self-host/upgrading/outdated/4100](/self-host/upgrading/outdated/4100) - [/self-host/upgrading/outdated/4100](/self-host/upgrading/outdated/4100)
......
...@@ -294,7 +294,7 @@ ...@@ -294,7 +294,7 @@
"content/self-host/upgrading/4-14/4149.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/4-14/4149.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4149.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/4-14/4149.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-15/41500.en.mdx": "2026-06-29T00:47:43+08:00", "content/self-host/upgrading/4-15/41500.en.mdx": "2026-06-29T00:47:43+08:00",
"content/self-host/upgrading/4-15/41500.mdx": "2026-06-29T00:47:43+08:00", "content/self-host/upgrading/4-15/41500.mdx": "2026-06-29T21:48:35+08:00",
"content/self-host/upgrading/4-15/41501.mdx": "2026-06-23T21:09:39+08:00", "content/self-host/upgrading/4-15/41501.mdx": "2026-06-23T21:09:39+08:00",
"content/self-host/upgrading/4-15/41502.en.mdx": "2026-05-25T11:21:30+08:00", "content/self-host/upgrading/4-15/41502.en.mdx": "2026-05-25T11:21:30+08:00",
"content/self-host/upgrading/4-15/41502.mdx": "2026-06-23T13:54:06+08:00", "content/self-host/upgrading/4-15/41502.mdx": "2026-06-23T13:54:06+08:00",
...@@ -302,8 +302,8 @@ ...@@ -302,8 +302,8 @@
"content/self-host/upgrading/4-15/41503.mdx": "2026-05-28T16:21:09+08:00", "content/self-host/upgrading/4-15/41503.mdx": "2026-05-28T16:21:09+08:00",
"content/self-host/upgrading/4-15/41504.en.mdx": "2026-06-10T19:02:59+08:00", "content/self-host/upgrading/4-15/41504.en.mdx": "2026-06-10T19:02:59+08:00",
"content/self-host/upgrading/4-15/41504.mdx": "2026-06-15T23:34:43+08:00", "content/self-host/upgrading/4-15/41504.mdx": "2026-06-15T23:34:43+08:00",
"content/self-host/upgrading/4-15/41505.en.mdx": "2026-06-29T00:47:43+08:00", "content/self-host/upgrading/4-15/41505.en.mdx": "2026-06-29T10:49:24+08:00",
"content/self-host/upgrading/4-15/41505.mdx": "2026-06-29T10:43:24+08:00", "content/self-host/upgrading/4-15/41505.mdx": "2026-06-29T10:45:58+08:00",
"content/self-host/upgrading/4-15/41506.en.mdx": "2026-06-29T00:47:43+08:00", "content/self-host/upgrading/4-15/41506.en.mdx": "2026-06-29T00:47:43+08:00",
"content/self-host/upgrading/4-15/41506.mdx": "2026-06-29T00:47:43+08:00", "content/self-host/upgrading/4-15/41506.mdx": "2026-06-29T00:47:43+08:00",
"content/self-host/upgrading/outdated/40.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/outdated/40.en.mdx": "2026-04-26T21:08:47+08:00",
...@@ -446,6 +446,6 @@ ...@@ -446,6 +446,6 @@
"content/self-host/upgrading/outdated/499.mdx": "2026-05-07T15:06:40+08:00", "content/self-host/upgrading/outdated/499.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/upgrade-intruction.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/upgrade-intruction.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/upgrade-intruction.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/upgrade-intruction.mdx": "2026-04-26T21:08:47+08:00",
"content/toc.en.mdx": "2026-06-27T22:05:51+08:00", "content/toc.en.mdx": "2026-06-29T21:48:35+08:00",
"content/toc.mdx": "2026-06-27T22:05:51+08:00" "content/toc.mdx": "2026-06-29T21:48:35+08:00"
} }
\ No newline at end of file
...@@ -114,7 +114,7 @@ export const AppAutoExecuteConfigTypeSchema = z.object({ ...@@ -114,7 +114,7 @@ export const AppAutoExecuteConfigTypeSchema = z.object({
open: BoolSchema.meta({ open: BoolSchema.meta({
description: '是否在进入会话后自动触发应用执行' description: '是否在进入会话后自动触发应用执行'
}), }),
defaultPrompt: z.string().meta({ defaultPrompt: z.string().default('').meta({
description: '自动执行时注入的默认用户问题' description: '自动执行时注入的默认用户问题'
}) })
}); });
...@@ -168,7 +168,7 @@ export const AppSchemaTypeSchema = z.object({ ...@@ -168,7 +168,7 @@ export const AppSchemaTypeSchema = z.object({
teamId: z.string(), teamId: z.string(),
tmbId: z.string(), tmbId: z.string(),
type: z.enum(AppTypeEnum), type: z.enum(AppTypeEnum),
version: z.enum(['v1', 'v2']).optional(), version: z.enum(['v1', 'v2']).optional().meta({ description: '内容版本,folder 类型不会有' }),
name: z.string(), name: z.string(),
avatar: z.string(), avatar: z.string(),
......
...@@ -80,12 +80,7 @@ export const nodeInputIsReference = (input: FlowNodeInputItemType) => { ...@@ -80,12 +80,7 @@ export const nodeInputIsReference = (input: FlowNodeInputItemType) => {
/* node */ /* node */
export const getGuideModule = (nodes: StoreNodeItemType[]) => export const getGuideModule = (nodes: StoreNodeItemType[]) =>
nodes.find( nodes.find((item) => item.flowNodeType === FlowNodeTypeEnum.systemConfig);
(item) =>
item.flowNodeType === FlowNodeTypeEnum.systemConfig ||
// @ts-ignore (adapt v1)
item.flowType === FlowNodeTypeEnum.systemConfig
);
export const splitGuideModule = (guideModules?: StoreNodeItemType) => { export const splitGuideModule = (guideModules?: StoreNodeItemType) => {
const welcomeText: string = const welcomeText: string =
guideModules?.inputs?.find((item) => item.key === NodeInputKeyEnum.welcomeText)?.value ?? ''; guideModules?.inputs?.find((item) => item.key === NodeInputKeyEnum.welcomeText)?.value ?? '';
......
...@@ -172,21 +172,6 @@ describe('getGuideModule', () => { ...@@ -172,21 +172,6 @@ describe('getGuideModule', () => {
const result = getGuideModule([]); const result = getGuideModule([]);
expect(result).toBeUndefined(); expect(result).toBeUndefined();
}); });
it('should find node with v1 flowType (adapt v1)', () => {
const nodes = [
{
nodeId: 'node1',
flowType: FlowNodeTypeEnum.systemConfig,
flowNodeType: FlowNodeTypeEnum.chatNode,
name: 'System Config',
inputs: [],
outputs: []
}
] as any;
const result = getGuideModule(nodes);
expect(result?.nodeId).toBe('node1');
});
}); });
describe('splitGuideModule', () => { describe('splitGuideModule', () => {
......
...@@ -278,11 +278,19 @@ export const loadRequestMessages = async ({ ...@@ -278,11 +278,19 @@ export const loadRequestMessages = async ({
* 仅从短文本中识别媒体 URL。普通文档 URL 仍作为文本保留,不在这里转成 LLM 媒体输入。 * 仅从短文本中识别媒体 URL。普通文档 URL 仍作为文本保留,不在这里转成 LLM 媒体输入。
*/ */
const extractMediaUrls = (input: string) => { const extractMediaUrls = (input: string) => {
const urlRegex = /(https?:\/\/[^\s]+)/gi; const urlRegex = /(https?:\/\/[^\s<>"'\\]+)/gi;
const normalizeExtractedUrl = (rawUrl: string) => {
// Markdown 链接里的 URL 可能同时出现在 label 和 href 中,避免把 `](...)` 当成 URL 的一部分。
const markdownLinkSeparatorIndex = rawUrl.indexOf('](');
const urlWithoutMarkdownTail =
markdownLinkSeparatorIndex === -1 ? rawUrl : rawUrl.slice(0, markdownLinkSeparatorIndex);
return urlWithoutMarkdownTail.replace(/[)\]\},。,.!?;:]+$/, '');
};
const normalizedUrls = Array.from(input.matchAll(urlRegex), (m) => normalizeExtractedUrl(m[0]));
return Array.from(new Set(input.matchAll(urlRegex)), (m) => return Array.from(new Set(normalizedUrls)).filter((url) => getFileTypeFromUrl(url) !== 'file');
m[0].replace(/[,。,.!?;:]+$/, '')
).filter((url) => getFileTypeFromUrl(url) !== 'file');
}; };
/** /**
......
...@@ -712,6 +712,49 @@ describe('loadRequestMessages function tests', () => { ...@@ -712,6 +712,49 @@ describe('loadRequestMessages function tests', () => {
expect(content.some((item: any) => item.type === 'image_url')).toBe(true); expect(content.some((item: any) => item.type === 'image_url')).toBe(true);
}); });
it('should extract clean image url from markdown link in json-like text', async () => {
serviceEnv.MULTIPLE_DATA_TO_BASE64 = false;
const imageUrl =
'https://proxy.file.fastgpt.io/chat/image.png?X-Amz-Signature=test&x-id=GetObject';
const messages: ChatCompletionMessageParam[] = [
{
role: ChatCompletionRequestMessageRoleEnum.User,
content: `"url": "[${imageUrl}](${imageUrl})\\"]"`
}
];
const result = await loadRequestMessages({ messages, useVision: true });
expect(result).toHaveLength(1);
const content = result[0].content as any[];
const imageParts = content.filter((item: any) => item.type === 'image_url');
expect(imageParts).toHaveLength(1);
expect(imageParts[0].image_url.url).toBe(imageUrl);
expect(mockAxiosHead).toHaveBeenCalledWith(imageUrl, {
timeout: 10000
});
});
it('should extract image url before newline terminators', async () => {
serviceEnv.MULTIPLE_DATA_TO_BASE64 = false;
const imageUrl =
'https://proxy.file.fastgpt.io/chat/%E7%AE%80%E5%8E%86.png?X-Amz-Signature=test&x-id=GetObject';
const messages: ChatCompletionMessageParam[] = [
{
role: ChatCompletionRequestMessageRoleEnum.User,
content: `${imageUrl}\n${imageUrl}\\n`
}
];
const result = await loadRequestMessages({ messages, useVision: true });
expect(result).toHaveLength(1);
const content = result[0].content as any[];
const imageParts = content.filter((item: any) => item.type === 'image_url');
expect(imageParts).toHaveLength(1);
expect(imageParts[0].image_url.url).toBe(imageUrl);
});
it('should not extract images from very long text (>500 chars)', async () => { it('should not extract images from very long text (>500 chars)', async () => {
const longText = 'A'.repeat(600) + ' https://example.com/image.png'; const longText = 'A'.repeat(600) + ' https://example.com/image.png';
const messages: ChatCompletionMessageParam[] = [ const messages: ChatCompletionMessageParam[] = [
......
...@@ -158,13 +158,15 @@ ...@@ -158,13 +158,15 @@
"code_error.error_message.513": "Unauthorized to Read This File", "code_error.error_message.513": "Unauthorized to Read This File",
"code_error.error_message.514": "Invalid API Key", "code_error.error_message.514": "Invalid API Key",
"code_error.openapi_error.api_key_not_exist": "API Key Does Not Exist", "code_error.openapi_error.api_key_not_exist": "API Key Does Not Exist",
"code_error.openapi_error.exceed_limit": "Up to 10 API Keys", "code_error.openapi_error.exceed_limit": "API key limit reached",
"code_error.openapi_error.un_auth": "Unauthorized to Operate This API Key", "code_error.openapi_error.un_auth": "Unauthorized to Operate This API Key",
"code_error.outlink_error.invalid_link": "Invalid Share Link", "code_error.outlink_error.invalid_link": "Invalid Share Link",
"code_error.outlink_error.link_not_exist": "Share Link Does Not Exist", "code_error.outlink_error.link_not_exist": "Share Link Does Not Exist",
"code_error.outlink_error.un_auth_user": "Identity Verification Failed", "code_error.outlink_error.un_auth_user": "Identity Verification Failed",
"code_error.plugin_error.un_auth": "No permission to operate the tool", "code_error.plugin_error.un_auth": "No permission to operate the tool",
"code_error.sandbox_error.agent_sandbox_initializing": "The virtual machine is initializing. Please try again later.",
"code_error.sandbox_error.agent_sandbox_permission_denied": "The current app is not authorized to use the sandbox/VM. Please contact an administrator to configure it.", "code_error.sandbox_error.agent_sandbox_permission_denied": "The current app is not authorized to use the sandbox/VM. Please contact an administrator to configure it.",
"code_error.sandbox_error.runtime_upgrade_failed": "Virtual machine runtime upgrade failed. Please try again.",
"code_error.skill_error.archive_empty": "Archive is empty", "code_error.skill_error.archive_empty": "Archive is empty",
"code_error.skill_error.archive_extraction_failed": "Failed to extract archive", "code_error.skill_error.archive_extraction_failed": "Failed to extract archive",
"code_error.skill_error.archive_too_large": "Archive file size exceeds the maximum allowed limit", "code_error.skill_error.archive_too_large": "Archive file size exceeds the maximum allowed limit",
...@@ -182,10 +184,8 @@ ...@@ -182,10 +184,8 @@
"code_error.skill_error.not_exist": "Skill Does Not Exist", "code_error.skill_error.not_exist": "Skill Does Not Exist",
"code_error.skill_error.skill_name_too_long": "Skill name must be 50 characters or fewer", "code_error.skill_error.skill_name_too_long": "Skill name must be 50 characters or fewer",
"code_error.skill_error.un_auth_skill": "Unauthorized to Operate This Skill", "code_error.skill_error.un_auth_skill": "Unauthorized to Operate This Skill",
"code_error.sandbox_error.agent_sandbox_initializing": "The virtual machine is initializing. Please try again later.",
"code_error.sandbox_error.runtime_upgrade_failed": "Virtual machine runtime upgrade failed. Please try again.",
"code_error.system_error.community_version_num_limit": "Exceeded Open Source Version Limit, Please Upgrade to Commercial Version: https://fastgpt.io",
"code_error.system_error.commercial_feature": "Commercial Edition exclusive feature", "code_error.system_error.commercial_feature": "Commercial Edition exclusive feature",
"code_error.system_error.community_version_num_limit": "Exceeded Open Source Version Limit, Please Upgrade to Commercial Version: https://fastgpt.io",
"code_error.system_error.license_app_amount_limit": "Exceed the maximum number of applications in the system", "code_error.system_error.license_app_amount_limit": "Exceed the maximum number of applications in the system",
"code_error.system_error.license_dataset_amount_limit": "Exceed the maximum number of knowledge bases in the system", "code_error.system_error.license_dataset_amount_limit": "Exceed the maximum number of knowledge bases in the system",
"code_error.system_error.license_user_amount_limit": "Exceed the maximum number of users in the system", "code_error.system_error.license_user_amount_limit": "Exceed the maximum number of users in the system",
......
...@@ -158,13 +158,15 @@ ...@@ -158,13 +158,15 @@
"code_error.error_message.513": "没有权限读取该文件", "code_error.error_message.513": "没有权限读取该文件",
"code_error.error_message.514": "Api Key 不合法", "code_error.error_message.514": "Api Key 不合法",
"code_error.openapi_error.api_key_not_exist": "Api Key 不存在", "code_error.openapi_error.api_key_not_exist": "Api Key 不存在",
"code_error.openapi_error.exceed_limit": "最多 10 组 API 密钥", "code_error.openapi_error.exceed_limit": "API 密钥数量达到上限",
"code_error.openapi_error.un_auth": "无权操作该 Api Key", "code_error.openapi_error.un_auth": "无权操作该 Api Key",
"code_error.outlink_error.invalid_link": "分享链接无效", "code_error.outlink_error.invalid_link": "分享链接无效",
"code_error.outlink_error.link_not_exist": "分享链接不存在", "code_error.outlink_error.link_not_exist": "分享链接不存在",
"code_error.outlink_error.un_auth_user": "身份校验失败", "code_error.outlink_error.un_auth_user": "身份校验失败",
"code_error.plugin_error.un_auth": "无权操作该工具", "code_error.plugin_error.un_auth": "无权操作该工具",
"code_error.sandbox_error.agent_sandbox_initializing": "虚拟机正在初始化,请稍后重试。",
"code_error.sandbox_error.agent_sandbox_permission_denied": "当前应用无权使用虚拟机,请联系管理员配置。", "code_error.sandbox_error.agent_sandbox_permission_denied": "当前应用无权使用虚拟机,请联系管理员配置。",
"code_error.sandbox_error.runtime_upgrade_failed": "虚拟机运行环境升级失败,请重新尝试。",
"code_error.skill_error.archive_empty": "压缩包内容为空", "code_error.skill_error.archive_empty": "压缩包内容为空",
"code_error.skill_error.archive_extraction_failed": "压缩包解压失败", "code_error.skill_error.archive_extraction_failed": "压缩包解压失败",
"code_error.skill_error.archive_too_large": "压缩包大小超过最大限制", "code_error.skill_error.archive_too_large": "压缩包大小超过最大限制",
...@@ -182,10 +184,8 @@ ...@@ -182,10 +184,8 @@
"code_error.skill_error.not_exist": "技能不存在", "code_error.skill_error.not_exist": "技能不存在",
"code_error.skill_error.skill_name_too_long": "技能名称不能超过 50 字符", "code_error.skill_error.skill_name_too_long": "技能名称不能超过 50 字符",
"code_error.skill_error.un_auth_skill": "无权操作该技能", "code_error.skill_error.un_auth_skill": "无权操作该技能",
"code_error.sandbox_error.agent_sandbox_initializing": "虚拟机正在初始化,请稍后重试。",
"code_error.sandbox_error.runtime_upgrade_failed": "虚拟机运行环境升级失败,请重新尝试。",
"code_error.system_error.community_version_num_limit": "超出社区版数量限制,请升级商业版: https://fastgpt.in",
"code_error.system_error.commercial_feature": "商业版专属功能", "code_error.system_error.commercial_feature": "商业版专属功能",
"code_error.system_error.community_version_num_limit": "超出社区版数量限制,请升级商业版: https://fastgpt.in",
"code_error.system_error.license_app_amount_limit": "超出系统最大应用数量", "code_error.system_error.license_app_amount_limit": "超出系统最大应用数量",
"code_error.system_error.license_dataset_amount_limit": "超出系统最大知识库数量", "code_error.system_error.license_dataset_amount_limit": "超出系统最大知识库数量",
"code_error.system_error.license_user_amount_limit": "超出系统最大用户数量", "code_error.system_error.license_user_amount_limit": "超出系统最大用户数量",
......
...@@ -156,13 +156,15 @@ ...@@ -156,13 +156,15 @@
"code_error.error_message.513": "無權讀取此檔案", "code_error.error_message.513": "無權讀取此檔案",
"code_error.error_message.514": "API 金鑰無效", "code_error.error_message.514": "API 金鑰無效",
"code_error.openapi_error.api_key_not_exist": "API 金鑰不存在", "code_error.openapi_error.api_key_not_exist": "API 金鑰不存在",
"code_error.openapi_error.exceed_limit": "最多 10 組 API 金鑰", "code_error.openapi_error.exceed_limit": "API 金鑰數量達到上限",
"code_error.openapi_error.un_auth": "無權操作此 API 金鑰", "code_error.openapi_error.un_auth": "無權操作此 API 金鑰",
"code_error.outlink_error.invalid_link": "分享連結無效", "code_error.outlink_error.invalid_link": "分享連結無效",
"code_error.outlink_error.link_not_exist": "分享連結不存在", "code_error.outlink_error.link_not_exist": "分享連結不存在",
"code_error.outlink_error.un_auth_user": "身份驗證失敗", "code_error.outlink_error.un_auth_user": "身份驗證失敗",
"code_error.plugin_error.un_auth": "無權操作該工具", "code_error.plugin_error.un_auth": "無權操作該工具",
"code_error.sandbox_error.agent_sandbox_initializing": "虛擬機正在初始化,請稍後重試。",
"code_error.sandbox_error.agent_sandbox_permission_denied": "當前應用無權使用虛擬機,請聯絡管理員配置。", "code_error.sandbox_error.agent_sandbox_permission_denied": "當前應用無權使用虛擬機,請聯絡管理員配置。",
"code_error.sandbox_error.runtime_upgrade_failed": "虛擬機運行環境升級失敗,請重新嘗試。",
"code_error.skill_error.archive_empty": "壓縮包內容為空", "code_error.skill_error.archive_empty": "壓縮包內容為空",
"code_error.skill_error.archive_extraction_failed": "壓縮包解壓失敗", "code_error.skill_error.archive_extraction_failed": "壓縮包解壓失敗",
"code_error.skill_error.archive_too_large": "壓縮包大小超過最大限制", "code_error.skill_error.archive_too_large": "壓縮包大小超過最大限制",
...@@ -180,10 +182,8 @@ ...@@ -180,10 +182,8 @@
"code_error.skill_error.not_exist": "技能不存在", "code_error.skill_error.not_exist": "技能不存在",
"code_error.skill_error.skill_name_too_long": "技能名稱不能超過 50 字元", "code_error.skill_error.skill_name_too_long": "技能名稱不能超過 50 字元",
"code_error.skill_error.un_auth_skill": "無權操作該技能", "code_error.skill_error.un_auth_skill": "無權操作該技能",
"code_error.sandbox_error.agent_sandbox_initializing": "虛擬機正在初始化,請稍後重試。",
"code_error.sandbox_error.runtime_upgrade_failed": "虛擬機運行環境升級失敗,請重新嘗試。",
"code_error.system_error.community_version_num_limit": "超出開源版數量限制,請升級商業版:https://fastgpt.io",
"code_error.system_error.commercial_feature": "商業版專屬功能", "code_error.system_error.commercial_feature": "商業版專屬功能",
"code_error.system_error.community_version_num_limit": "超出開源版數量限制,請升級商業版:https://fastgpt.io",
"code_error.system_error.license_app_amount_limit": "超出系統最大應用數量", "code_error.system_error.license_app_amount_limit": "超出系統最大應用數量",
"code_error.system_error.license_dataset_amount_limit": "超出系統最大知識庫數量", "code_error.system_error.license_dataset_amount_limit": "超出系統最大知識庫數量",
"code_error.system_error.license_user_amount_limit": "超出系統最大用戶數量", "code_error.system_error.license_user_amount_limit": "超出系統最大用戶數量",
......
{ {
"name": "@fastgpt/app", "name": "@fastgpt/app",
"version": "4.15.0-6", "version": "4.15.0",
"private": false, "private": false,
"browserslist": [ "browserslist": [
"Chrome >= 80", "Chrome >= 80",
......
...@@ -6,7 +6,6 @@ import { Box, Flex } from '@chakra-ui/react'; ...@@ -6,7 +6,6 @@ import { Box, Flex } from '@chakra-ui/react';
import { useTranslation } from 'next-i18next'; import { useTranslation } from 'next-i18next';
import { useSimpleAppSnapshots } from '../FormComponent/useSnapshots'; import { useSimpleAppSnapshots } from '../FormComponent/useSnapshots';
import { useDebounceEffect, useMount } from 'ahooks'; import { useDebounceEffect, useMount } from 'ahooks';
import { v1Workflow2V2 } from '@/web/core/workflow/adapt';
import { defaultAppSelectFileConfig } from '@fastgpt/global/core/app/constants'; import { defaultAppSelectFileConfig } from '@fastgpt/global/core/app/constants';
import { form2AppWorkflow, appWorkflow2Form } from './utils'; import { form2AppWorkflow, appWorkflow2Form } from './utils';
import PublishChannel from '../../Publish'; import PublishChannel from '../../Publish';
...@@ -22,13 +21,6 @@ const SimpleEdit = () => { ...@@ -22,13 +21,6 @@ const SimpleEdit = () => {
); );
const [appForm, setAppForm] = useState(() => { const [appForm, setAppForm] = useState(() => {
if (appDetail.version !== 'v2') {
return appWorkflow2Form({
nodes: v1Workflow2V2((appDetail.modules || []) as any)?.nodes,
chatConfig: appDetail.chatConfig
});
}
if (past.length === 0) { if (past.length === 0) {
return appWorkflow2Form({ return appWorkflow2Form({
nodes: appDetail.modules, nodes: appDetail.modules,
......
...@@ -46,7 +46,6 @@ const Header = () => { ...@@ -46,7 +46,6 @@ const Header = () => {
}); });
const { appDetail, onSaveApp, currentTab } = useContextSelector(AppContext, (v) => v); const { appDetail, onSaveApp, currentTab } = useContextSelector(AppContext, (v) => v);
const isV2Workflow = appDetail?.version === 'v2';
const { const {
isOpen: isOpenBackConfirm, isOpen: isOpenBackConfirm,
onOpen: onOpenBackConfirm, onOpen: onOpenBackConfirm,
...@@ -163,7 +162,7 @@ const Header = () => { ...@@ -163,7 +162,7 @@ const Header = () => {
{/* app info */} {/* app info */}
<Box ml={1}> <Box ml={1}>
<AppCard isSaved={isSaved} showSaveStatus={isV2Workflow} /> <AppCard isSaved={isSaved} showSaveStatus />
</Box> </Box>
{isPc && ( {isPc && (
...@@ -222,7 +221,6 @@ const Header = () => { ...@@ -222,7 +221,6 @@ const Header = () => {
isSaved, isSaved,
onBack, onBack,
onOpenBackConfirm, onOpenBackConfirm,
isV2Workflow,
showHistoryModal, showHistoryModal,
t, t,
loading, loading,
...@@ -235,7 +233,7 @@ const Header = () => { ...@@ -235,7 +233,7 @@ const Header = () => {
return ( return (
<> <>
{Render} {Render}
{showHistoryModal && isV2Workflow && currentTab === TabEnum.appEdit && ( {showHistoryModal && currentTab === TabEnum.appEdit && (
<PublishHistories<WorkflowSnapshotsType> <PublishHistories<WorkflowSnapshotsType>
onClose={() => { onClose={() => {
setShowHistoryModal(false); setShowHistoryModal(false);
......
import React from 'react'; import React from 'react';
import { pluginSystemModuleTemplates } from '@fastgpt/global/core/workflow/template/constants'; import { pluginSystemModuleTemplates } from '@fastgpt/global/core/workflow/template/constants';
import { useConfirm } from '@fastgpt/web/hooks/useConfirm';
import { v1Workflow2V2 } from '@/web/core/workflow/adapt';
import { ReactFlowCustomProvider } from '../WorkflowComponents/context'; import { ReactFlowCustomProvider } from '../WorkflowComponents/context';
import { useContextSelector } from 'use-context-selector'; import { useContextSelector } from 'use-context-selector';
import { AppContext, TabEnum } from '../context'; import { AppContext, TabEnum } from '../context';
...@@ -13,7 +11,6 @@ import dynamic from 'next/dynamic'; ...@@ -13,7 +11,6 @@ import dynamic from 'next/dynamic';
import { cloneDeep } from 'lodash'; import { cloneDeep } from 'lodash';
import Flow from '../WorkflowComponents/Flow'; import Flow from '../WorkflowComponents/Flow';
import { useTranslation } from 'next-i18next';
import { WorkflowUtilsContext } from '../WorkflowComponents/context/workflowUtilsContext'; import { WorkflowUtilsContext } from '../WorkflowComponents/context/workflowUtilsContext';
const Logs = dynamic(() => import('../Logs/index')); const Logs = dynamic(() => import('../Logs/index'));
...@@ -21,24 +18,10 @@ const PublishChannel = dynamic(() => import('../Publish')); ...@@ -21,24 +18,10 @@ const PublishChannel = dynamic(() => import('../Publish'));
const WorkflowEdit = () => { const WorkflowEdit = () => {
const { appDetail, currentTab } = useContextSelector(AppContext, (e) => e); const { appDetail, currentTab } = useContextSelector(AppContext, (e) => e);
const isV2Workflow = appDetail?.version === 'v2';
const { t } = useTranslation();
const { openConfirm, ConfirmModal } = useConfirm({
showCancel: false,
content: t('common:info.old_version_attention')
});
const initData = useContextSelector(WorkflowUtilsContext, (v) => v.initData); const initData = useContextSelector(WorkflowUtilsContext, (v) => v.initData);
useMount(() => { useMount(() => {
if (!isV2Workflow) {
openConfirm({
onConfirm: () => {
initData(JSON.parse(JSON.stringify(v1Workflow2V2((appDetail.modules || []) as any))));
}
})();
} else {
initData( initData(
cloneDeep({ cloneDeep({
nodes: appDetail.modules || [], nodes: appDetail.modules || [],
...@@ -46,7 +29,6 @@ const WorkflowEdit = () => { ...@@ -46,7 +29,6 @@ const WorkflowEdit = () => {
}), }),
true true
); );
}
}); });
return ( return (
...@@ -71,8 +53,6 @@ const WorkflowEdit = () => { ...@@ -71,8 +53,6 @@ const WorkflowEdit = () => {
{currentTab === TabEnum.logs && <Logs />} {currentTab === TabEnum.logs && <Logs />}
</Flex> </Flex>
)} )}
{!isV2Workflow && <ConfirmModal countDown={0} />}
</Flex> </Flex>
); );
}; };
......
...@@ -46,7 +46,6 @@ const Header = () => { ...@@ -46,7 +46,6 @@ const Header = () => {
}); });
const { appDetail, onSaveApp, currentTab } = useContextSelector(AppContext, (v) => v); const { appDetail, onSaveApp, currentTab } = useContextSelector(AppContext, (v) => v);
const isV2Workflow = appDetail?.version === 'v2';
const { const {
isOpen: isOpenBackConfirm, isOpen: isOpenBackConfirm,
onOpen: onOpenBackConfirm, onOpen: onOpenBackConfirm,
...@@ -163,7 +162,7 @@ const Header = () => { ...@@ -163,7 +162,7 @@ const Header = () => {
{/* app info */} {/* app info */}
<Box ml={1}> <Box ml={1}>
<AppCard isSaved={isSaved} showSaveStatus={isV2Workflow} /> <AppCard isSaved={isSaved} showSaveStatus />
</Box> </Box>
{isPc && ( {isPc && (
...@@ -222,7 +221,6 @@ const Header = () => { ...@@ -222,7 +221,6 @@ const Header = () => {
isSaved, isSaved,
onBack, onBack,
onOpenBackConfirm, onOpenBackConfirm,
isV2Workflow,
showHistoryModal, showHistoryModal,
t, t,
loading, loading,
...@@ -235,7 +233,7 @@ const Header = () => { ...@@ -235,7 +233,7 @@ const Header = () => {
return ( return (
<> <>
{Render} {Render}
{showHistoryModal && isV2Workflow && currentTab === TabEnum.appEdit && ( {showHistoryModal && currentTab === TabEnum.appEdit && (
<PublishHistories<WorkflowSnapshotsType> <PublishHistories<WorkflowSnapshotsType>
onClose={() => { onClose={() => {
setShowHistoryModal(false); setShowHistoryModal(false);
......
import React from 'react'; import React from 'react';
import { appSystemModuleTemplates } from '@fastgpt/global/core/workflow/template/constants'; import { appSystemModuleTemplates } from '@fastgpt/global/core/workflow/template/constants';
import { useConfirm } from '@fastgpt/web/hooks/useConfirm';
import { v1Workflow2V2 } from '@/web/core/workflow/adapt';
import { useContextSelector } from 'use-context-selector'; import { useContextSelector } from 'use-context-selector';
import { AppContext, TabEnum } from '../context'; import { AppContext, TabEnum } from '../context';
...@@ -11,7 +9,6 @@ import { Flex } from '@chakra-ui/react'; ...@@ -11,7 +9,6 @@ import { Flex } from '@chakra-ui/react';
import { workflowBoxStyles } from '../constants'; import { workflowBoxStyles } from '../constants';
import dynamic from 'next/dynamic'; import dynamic from 'next/dynamic';
import { cloneDeep } from 'lodash'; import { cloneDeep } from 'lodash';
import { useTranslation } from 'next-i18next';
import Flow from '../WorkflowComponents/Flow'; import Flow from '../WorkflowComponents/Flow';
import { ReactFlowCustomProvider } from '../WorkflowComponents/context/index'; import { ReactFlowCustomProvider } from '../WorkflowComponents/context/index';
...@@ -24,27 +21,9 @@ const WorkflowEdit = () => { ...@@ -24,27 +21,9 @@ const WorkflowEdit = () => {
const appDetail = useContextSelector(AppContext, (v) => v.appDetail); const appDetail = useContextSelector(AppContext, (v) => v.appDetail);
const currentTab = useContextSelector(AppContext, (v) => v.currentTab); const currentTab = useContextSelector(AppContext, (v) => v.currentTab);
const isV2Workflow = appDetail?.version === 'v2';
const { t } = useTranslation();
const { openConfirm, ConfirmModal } = useConfirm({
showCancel: false,
content: t('common:info.old_version_attention')
});
const initData = useContextSelector(WorkflowUtilsContext, (v) => v.initData); const initData = useContextSelector(WorkflowUtilsContext, (v) => v.initData);
useMount(() => { useMount(() => {
if (!isV2Workflow) {
openConfirm({
onConfirm: () => {
initData(
JSON.parse(JSON.stringify(v1Workflow2V2((appDetail.modules || []) as any))),
true
);
}
})();
} else {
initData( initData(
cloneDeep({ cloneDeep({
nodes: appDetail.modules || [], nodes: appDetail.modules || [],
...@@ -52,7 +31,6 @@ const WorkflowEdit = () => { ...@@ -52,7 +31,6 @@ const WorkflowEdit = () => {
}), }),
true true
); );
}
}); });
return ( return (
...@@ -77,8 +55,6 @@ const WorkflowEdit = () => { ...@@ -77,8 +55,6 @@ const WorkflowEdit = () => {
{currentTab === TabEnum.logs && <Logs />} {currentTab === TabEnum.logs && <Logs />}
</Flex> </Flex>
)} )}
{!isV2Workflow && <ConfirmModal countDown={0} />}
</Flex> </Flex>
); );
}; };
......
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { addHours } from 'date-fns';
import { checkInvalidDatasetData, checkInvalidVector } from '@/service/common/system/cronTask';
import dayjs from 'dayjs';
import { retryFn } from '@fastgpt/global/common/system/utils';
import { NextAPI } from '@/service/middleware/entry';
import { useIPFrequencyLimit } from '@fastgpt/service/common/middle/reqFrequencyLimit';
import { MongoImage } from '@fastgpt/service/common/file/image/schema';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import { getLogger, LogCategories } from '@fastgpt/service/common/logger';
import z from 'zod';
const logger = getLogger(LogCategories.SYSTEM);
const BodySchema = z
.object({
start: z.number().int().min(-8760).max(0).default(-2),
end: z
.number()
.int()
.min(-8760)
.max(-1)
.default(-360 * 24)
})
.refine((data) => data.end < data.start, {
message: 'end 必须小于 start(end 时间点更早)'
});
const MAX_CHUNKS = 1000;
let deleteImageAmount = 0;
async function checkInvalidImg(start: Date, end: Date) {
const images = await MongoImage.find(
{
createTime: {
$gte: start,
$lte: end
},
'metadata.relatedId': { $exists: true }
},
'_id teamId metadata'
);
logger.info('Start invalid image cleanup', { totalImages: images.length });
let index = 0;
for await (const image of images) {
try {
// 1. 检测是否有对应的集合
const collection = await MongoDatasetCollection.findOne(
{
teamId: image.teamId,
'metadata.relatedImgId': image.metadata?.relatedId
},
'_id'
).lean();
if (!collection) {
await image.deleteOne();
deleteImageAmount++;
}
index++;
if (index % 100 === 0) {
logger.debug('Invalid image cleanup progress', {
processed: index,
total: images.length,
deleted: deleteImageAmount
});
}
} catch (error) {
logger.error('Invalid data cleanup task failed', { error });
}
}
logger.info(`检测完成,共删除 ${deleteImageAmount} 个无效图片`);
}
async function handler(req: NextApiRequest, res: NextApiResponse) {
deleteImageAmount = 0;
try {
await authCert({ req, authRoot: true });
const { start, end } = BodySchema.parse(req.body);
(async () => {
try {
logger.info('执行脏数据清理任务');
// Split time range into 6-hour chunks to avoid processing too much data at once
const totalHours = Math.abs(start - end);
const chunkHours = 6;
const chunks = Math.min(Math.ceil(totalHours / chunkHours), MAX_CHUNKS);
logger.info(
`Total time range: ${totalHours} hours, split into ${chunks} chunks of ${chunkHours} hours each`
);
for (let i = 0; i < chunks; i++) {
const chunkStart = start - i * chunkHours;
const chunkEnd = Math.max(start - (i + 1) * chunkHours, end);
const chunkEndTime = addHours(new Date(), chunkStart);
const chunkStartTime = addHours(new Date(), chunkEnd);
logger.info(
`Processing chunk ${i + 1}/${chunks}: ${dayjs(chunkStartTime).format(
'YYYY-MM-DD HH:mm'
)} to ${dayjs(chunkEndTime).format('YYYY-MM-DD HH:mm')}`
);
await retryFn(() => checkInvalidImg(chunkStartTime, chunkEndTime));
await retryFn(() => checkInvalidDatasetData(chunkStartTime, chunkEndTime));
await retryFn(() => checkInvalidVector(chunkStartTime, chunkEndTime));
logger.info(`Chunk ${i + 1}/${chunks} completed`);
}
logger.info('执行脏数据清理任务完毕');
} catch (error) {
logger.info('执行脏数据清理任务出错了');
}
})();
jsonRes(res, {
message: 'success'
});
} catch (error) {
logger.error('Invalid data cleanup task failed', { error });
jsonRes(res, {
code: 500,
error
});
}
}
export default NextAPI(useIPFrequencyLimit({ id: 'admin-api', seconds: 60, limit: 1 }), handler);
...@@ -328,9 +328,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse) { ...@@ -328,9 +328,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
customFeedbacks, customFeedbacks,
nodeResponseSummary, nodeResponseSummary,
flatNodeResponses flatNodeResponses
} = await (async () => { } = await dispatchWorkFlow({
if (app.version === 'v2') {
return dispatchWorkFlow({
apiVersion: 'v1', apiVersion: 'v1',
req, req,
res, res,
...@@ -368,9 +366,6 @@ async function handler(req: NextApiRequest, res: NextApiResponse) { ...@@ -368,9 +366,6 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
retainInMemory: shouldCollectFinalResponseData retainInMemory: shouldCollectFinalResponseData
} }
}); });
}
return Promise.reject('您的工作流版本过低,请重新发布一次');
})();
const aiResponse: AIChatItemType & { dataId?: string } = { const aiResponse: AIChatItemType & { dataId?: string } = {
dataId: finalResponseChatItemId, dataId: finalResponseChatItemId,
......
...@@ -332,9 +332,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse) { ...@@ -332,9 +332,7 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
customFeedbacks, customFeedbacks,
nodeResponseSummary, nodeResponseSummary,
flatNodeResponses flatNodeResponses
} = await (async () => { } = await dispatchWorkFlow({
if (app.version === 'v2') {
return dispatchWorkFlow({
apiVersion: 'v2', apiVersion: 'v2',
res, res,
lang: getLocale(req), lang: getLocale(req),
...@@ -373,9 +371,6 @@ async function handler(req: NextApiRequest, res: NextApiResponse) { ...@@ -373,9 +371,6 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
retainInMemory: shouldCollectFinalResponseData retainInMemory: shouldCollectFinalResponseData
} }
}); });
}
return Promise.reject('您的工作流版本过低,请重新发布一次');
})();
const aiResponse: AIChatItemType & { dataId?: string } = { const aiResponse: AIChatItemType & { dataId?: string } = {
dataId: finalResponseChatItemId, dataId: finalResponseChatItemId,
......
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { AppTypeEnum } from '@fastgpt/global/core/app/constants';
import {
FlowNodeInputTypeEnum,
FlowNodeOutputTypeEnum,
FlowNodeTypeEnum
} from '@fastgpt/global/core/workflow/node/constant';
import { WorkflowIOValueTypeEnum } from '@fastgpt/global/core/workflow/constants';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { MongoAppVersion } from '@fastgpt/service/core/app/version/schema';
vi.mock('@/service/middleware/entry', () => ({
NextAPI: (handler: unknown) => handler
}));
vi.mock('@fastgpt/service/support/permission/auth/common', () => ({
authCert: vi.fn()
}));
import {
formatWorkflowDocument,
runInitWorkflowDataMigration
} from '@/pages/api/admin/dataClean/initWorkflowData';
const teamId = '65f000000000000000000051';
const tmbId = '65f000000000000000000052';
const appId = '65f000000000000000000053';
const dirtyV2Nodes = [
{
nodeId: 'start',
flowNodeType: FlowNodeTypeEnum.workflowStart,
name: 'Start',
inputs: null,
outputs: [
{
key: 'userChatInput',
type: 'FlowNodeOutputTypeEnum.static',
valueType: 'WorkflowIOValueTypeEnum.string'
}
]
},
{
moduleId: 'legacy-laf',
flowNodeType: 'lafModule',
name: null,
inputs: [
{
key: 'hidden',
label: null,
renderTypeList: ['FlowNodeInputTypeEnum.hidden'],
valueType: 'WorkflowIOValueTypeEnum.string',
description: null
}
],
outputs: []
}
];
describe('initWorkflowData data clean API', () => {
beforeEach(async () => {
await Promise.all([MongoApp.deleteMany({}), MongoAppVersion.deleteMany({})]);
});
it('formats dirty v2 workflow data and keeps save schema valid', () => {
const stats = {
collectionName: 'apps',
fieldName: 'modules',
queryMatchedDocumentCount: null,
scannedDocumentCount: 0,
fixableDocumentCount: 0,
unknownDocumentCount: 0,
enumExpressionCount: 0,
renderTypeListFixableCount: 0,
outputTypeFixableCount: 0,
valueTypeFixableCount: 0,
unknownEnumExpressionCount: 0,
saveApiValidationErrorDocumentCount: 0,
cleanErrorDocumentCount: 0,
formatChangedDocumentCount: 0,
writeSuccessDocumentCount: 0,
writeBlockedDocumentCount: 0,
writeErrorDocumentCount: 0,
byExpression: {}
};
const result = formatWorkflowDocument({
doc: {
_id: appId,
modules: dirtyV2Nodes,
edges: null,
chatConfig: {
questionGuide: true,
variables: [
{
key: 'topic',
label: 'topic',
type: 'string',
enums: '[{\"value\":\"a\"}]',
maxLength: -1
}
],
scheduledTriggerConfig: null
}
},
fieldName: 'modules',
stats,
docContext: {
collectionName: 'apps',
documentId: appId
},
rootPath: 'modules'
});
expect(result.nodes).toEqual(
expect.arrayContaining([
expect.objectContaining({
nodeId: 'legacy-laf',
flowNodeType: FlowNodeTypeEnum.emptyNode,
inputs: expect.arrayContaining([
expect.objectContaining({
renderTypeList: [FlowNodeInputTypeEnum.hidden],
valueType: WorkflowIOValueTypeEnum.string,
label: 'hidden'
})
])
}),
expect.objectContaining({
outputs: expect.arrayContaining([
expect.objectContaining({
type: FlowNodeOutputTypeEnum.static,
valueType: WorkflowIOValueTypeEnum.string
})
])
})
])
);
expect(result.edges).toEqual([]);
expect(result.chatConfig).toMatchObject({
questionGuide: { open: true },
variables: [
{
type: 'input',
description: '',
enums: [{ value: 'a', label: 'a' }]
}
]
});
expect(result.chatConfig).not.toHaveProperty('scheduledTriggerConfig');
expect(result.formatChanges.count).toBeGreaterThan(0);
});
it('dry-runs without writing formatted data', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'dirty app',
type: AppTypeEnum.workflow,
version: 'v2',
modules: dirtyV2Nodes,
edges: null,
chatConfig: null
});
const result = await runInitWorkflowDataMigration({
dryRun: true
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 1,
fixableDocumentCount: 1,
saveApiValidationErrorDocumentCount: 0,
formatChangedDocumentCount: 1,
writeSuccessDocumentCount: 0
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
modules: expect.arrayContaining([expect.objectContaining({ flowNodeType: 'lafModule' })]),
chatConfig: null
});
});
it('writes only formatted and zod-valid documents', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'dirty app',
type: AppTypeEnum.workflow,
version: 'v2',
modules: dirtyV2Nodes,
edges: null,
chatConfig: null
});
const result = await runInitWorkflowDataMigration({
dryRun: false
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 1,
fixableDocumentCount: 1,
saveApiValidationErrorDocumentCount: 0,
writeSuccessDocumentCount: 1
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
modules: expect.arrayContaining([
expect.objectContaining({
nodeId: 'legacy-laf',
flowNodeType: FlowNodeTypeEnum.emptyNode
})
]),
edges: [],
chatConfig: {}
});
});
it('skips folder-like apps when cleaning app workflow data', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'folder app',
type: AppTypeEnum.folder,
modules: null,
edges: null,
chatConfig: null
});
const result = await runInitWorkflowDataMigration({
dryRun: false
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 0,
formatChangedDocumentCount: 0,
writeSuccessDocumentCount: 0
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
type: AppTypeEnum.folder,
modules: null,
edges: null,
chatConfig: null
});
});
it('fills missing system tool set child toolId before zod validation', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'system tool set app',
type: AppTypeEnum.workflow,
version: 'v2',
modules: [
...dirtyV2Nodes,
{
nodeId: 'toolSet',
flowNodeType: FlowNodeTypeEnum.toolSet,
name: 'Tool Set',
inputs: [],
outputs: [],
toolConfig: {
systemToolSet: {
toolId: 'system-tool-set',
toolList: [
{
key: 'searchByKey',
description: 'Search tool'
},
{
name: 'read',
description: 'Read tool'
}
]
}
}
}
],
edges: [],
chatConfig: null
});
const result = await runInitWorkflowDataMigration({
dryRun: false
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 1,
saveApiValidationErrorDocumentCount: 0,
writeSuccessDocumentCount: 1
});
const app = await MongoApp.findById(appId).lean();
const toolSetNode = app?.modules.find((node) => node.nodeId === 'toolSet');
expect(toolSetNode?.toolConfig?.systemToolSet?.toolList).toEqual([
expect.objectContaining({
name: 'searchByKey',
toolId: 'searchByKey'
}),
expect.objectContaining({
name: 'read',
toolId: 'read'
})
]);
});
it('blocks writes when formatted data still fails zod parse', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'invalid app',
type: AppTypeEnum.workflow,
version: 'v2',
modules: dirtyV2Nodes,
edges: [],
chatConfig: {
variables: ['invalid']
}
});
const result = await runInitWorkflowDataMigration({
dryRun: false
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 1,
saveApiValidationErrorDocumentCount: 1,
writeBlockedDocumentCount: 1,
writeSuccessDocumentCount: 0
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
modules: expect.arrayContaining([expect.objectContaining({ flowNodeType: 'lafModule' })])
});
});
});
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { AppTypeEnum } from '@fastgpt/global/core/app/constants';
import {
FlowNodeInputTypeEnum,
FlowNodeTypeEnum
} from '@fastgpt/global/core/workflow/node/constant';
import { WorkflowIOValueTypeEnum } from '@fastgpt/global/core/workflow/constants';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { MongoAppVersion } from '@fastgpt/service/core/app/version/schema';
vi.mock('@/service/middleware/entry', () => ({
NextAPI: (handler: unknown) => handler
}));
vi.mock('@fastgpt/service/support/permission/auth/common', () => ({
authCert: vi.fn()
}));
import {
runV1WorkflowToV2Migration,
upgradeV1WorkflowDocument
} from '@/pages/api/admin/dataClean/v1WorkflowToV2';
const teamId = '65f000000000000000000041';
const tmbId = '65f000000000000000000042';
const appId = '65f000000000000000000043';
const legacyV1Nodes = [
{
moduleId: 'start',
flowType: 'questionInput',
name: '开始',
inputs: [
{
key: 'userChatInput',
type: 'input',
valueType: 'string'
}
],
outputs: [
{
key: 'userChatInput',
type: 'source',
valueType: 'string',
targets: [{ moduleId: 'chat', key: 'userChatInput' }]
}
]
},
{
moduleId: 'chat',
flowType: 'chatNode',
inputs: [
{
key: 'userChatInput',
type: 'target',
valueType: 'chat_history'
},
{
key: 'dirty',
type: 'hidden',
valueType: 'tools',
description: null,
toolDescription: null
}
],
outputs: [
{
key: 'answer',
type: 'answer',
valueType: 'kb_quote'
}
]
},
{
moduleId: 'legacy-laf',
flowType: 'lafModule',
inputs: [],
outputs: []
}
];
describe('v1WorkflowToV2 data clean API', () => {
beforeEach(async () => {
await Promise.all([MongoApp.deleteMany({}), MongoAppVersion.deleteMany({})]);
});
it('converts one legacy v1 workflow document and keeps it publish-schema valid', () => {
const result = upgradeV1WorkflowDocument({
config: {
key: 'apps',
collectionName: 'apps',
fieldName: 'modules'
},
doc: {
_id: appId,
type: AppTypeEnum.workflow,
version: 'v1',
modules: legacyV1Nodes,
edges: [],
chatConfig: {
questionGuide: true,
variables: [
{
key: 'topic',
label: 'topic',
type: 'select',
list: [{ value: 'a' }]
}
],
scheduledTriggerConfig: null
}
}
});
expect(result.converted).toBe(true);
expect(result.nodes).toEqual(
expect.arrayContaining([
expect.objectContaining({
nodeId: 'start',
flowNodeType: FlowNodeTypeEnum.workflowStart
}),
expect.objectContaining({
nodeId: 'chat',
flowNodeType: FlowNodeTypeEnum.chatNode,
inputs: expect.arrayContaining([
expect.objectContaining({
key: 'userChatInput',
valueType: WorkflowIOValueTypeEnum.chatHistory,
value: ['start', 'userChatInput']
}),
expect.objectContaining({
key: 'dirty',
renderTypeList: [FlowNodeInputTypeEnum.hidden],
valueType: WorkflowIOValueTypeEnum.any,
description: '',
toolDescription: ''
})
]),
outputs: expect.arrayContaining([
expect.objectContaining({
key: 'answer',
valueType: WorkflowIOValueTypeEnum.datasetQuote
})
])
}),
expect.objectContaining({
nodeId: 'legacy-laf',
flowNodeType: FlowNodeTypeEnum.emptyNode
})
])
);
expect(result.chatConfig).toMatchObject({
questionGuide: { open: true },
variables: [
{
description: '',
list: [{ value: 'a', label: 'a' }]
}
]
});
expect(result.chatConfig).not.toHaveProperty('scheduledTriggerConfig');
});
it('dry-runs apps and app_versions without writing converted data', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'legacy app',
type: AppTypeEnum.workflow,
version: 'v1',
modules: legacyV1Nodes,
edges: [],
chatConfig: null
});
await MongoAppVersion.create({
appId,
tmbId,
nodes: legacyV1Nodes,
edges: [],
chatConfig: null
});
const result = await runV1WorkflowToV2Migration({
dryRun: true
});
expect(result).toMatchObject({
dryRun: true,
apps: {
scannedDocumentCount: 1,
convertedDocumentCount: 1,
zodErrorDocumentCount: 0,
writeSuccessDocumentCount: 0
},
appVersions: {
scannedDocumentCount: 1,
convertedDocumentCount: 1,
zodErrorDocumentCount: 0,
writeSuccessDocumentCount: 0
}
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
version: 'v1',
modules: expect.arrayContaining([expect.objectContaining({ flowType: 'questionInput' })]),
chatConfig: null
});
});
it('writes app_versions before apps and marks apps v2 only after zod validation', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'legacy app',
type: AppTypeEnum.workflow,
version: 'v1',
modules: legacyV1Nodes,
edges: [],
chatConfig: {
variables: [{ key: 'invalid-variable' }]
}
});
await MongoAppVersion.create({
appId,
tmbId,
nodes: legacyV1Nodes,
edges: [],
chatConfig: null
});
const result = await runV1WorkflowToV2Migration({
dryRun: false
});
expect(result.apps).toMatchObject({
scannedDocumentCount: 1,
convertedDocumentCount: 1,
zodErrorDocumentCount: 1,
writeBlockedDocumentCount: 1,
writeSuccessDocumentCount: 0
});
expect(result.appVersions).toMatchObject({
scannedDocumentCount: 1,
convertedDocumentCount: 1,
zodErrorDocumentCount: 0,
writeSuccessDocumentCount: 1
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
version: 'v1',
modules: expect.arrayContaining([expect.objectContaining({ flowType: 'questionInput' })])
});
await expect(MongoAppVersion.findOne({ appId }).lean()).resolves.toMatchObject({
nodes: expect.arrayContaining([
expect.objectContaining({
nodeId: 'start',
flowNodeType: FlowNodeTypeEnum.workflowStart
})
]),
chatConfig: {}
});
});
});
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment