Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 132093f1 by Archer Committed by GitHub
Options

fix: standardize manual HTTP tool array schemas (#7416) 

* fix http tool array json schema

* optimize http tool schema migration queries

* submodule

* document http tool schema migration

* fix: external
parent 7b1c53fa
Showing with 818 additions and 21 deletions
document/content/self-host/upgrading/4-16/41601.en.mdx
...@@ -62,6 +62,30 @@ The E2B Sandbox Provider has been removed. Deployments previously using E2B must ...@@ -62,6 +62,30 @@ The E2B Sandbox Provider has been removed. Deployments previously using E2B must
> The preview protocol has changed across FastGPT and the `fastgpt-agent-sandbox-proxy` and `fastgpt-agent-sandbox` services. If Agent Sandbox is enabled, use the matching images for this release. Mixed-version deployments are not supported. > The preview protocol has changed across FastGPT and the `fastgpt-agent-sandbox-proxy` and `fastgpt-agent-sandbox` services. If Agent Sandbox is enabled, use the matching images for this release. Mixed-version deployments are not supported.
### 3. Migrate Manual HTTP Tool Schemas
This release converts array parameters in manually configured HTTP tools to standard JSON Schema. Deployments with manual HTTP tools created before this upgrade must run this migration. OpenAPI-based HTTP tools do not require migration and are skipped automatically.
First, run a dry run to inspect pending data in current Apps and historical versions. A dry run does not modify data:
```bash
curl -X POST 'https://your-domain/api/admin/4160/initHttpToolSchema' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":true}'
```
After reviewing the result, run the migration:
```bash
curl -X POST 'https://your-domain/api/admin/4160/initHttpToolSchema' \
-H 'Content-Type: application/json' \
-H 'rootkey: YOUR_ROOT_KEY' \
-d '{"dryRun":false}'
```
The script first selects HTTP tool Apps by type, then migrates historical versions matching those App IDs. It only processes manual configurations where `apiSchemaStr` is absent; other Apps and OpenAPI-based tools remain unchanged. The migration runs in batches and is safe to retry. `total.changedDocumentCount` reports the number of documents requiring migration. After the formal run, run another dry run and confirm that this value is `0`.
## 🚀 New Features ## 🚀 New Features
1. Agent Sandbox now uses App-level, per-user instances. Chats for the same App and user reuse one Sandbox while keeping chat files in separate session directories. 1. Agent Sandbox now uses App-level, per-user instances. Chats for the same App and user reuse one Sandbox while keeping chat files in separate session directories.
......
document/content/self-host/upgrading/4-16/41601.mdx
...@@ -62,6 +62,30 @@ curl -X POST 'https://你的域名/api/admin/4160/initUserSandbox' \ ...@@ -62,6 +62,30 @@ curl -X POST 'https://你的域名/api/admin/4160/initUserSandbox' \
请检查返回结果中的 `normalization.pendingCount`、`normalizationBlocked`、`failedCount` 和 `failures`。只有 `normalization.pendingCount` 和 `failedCount` 均为 `0`，且 `normalizationBlocked` 为 `false` 时，才表示本次迁移全部完成。 请检查返回结果中的 `normalization.pendingCount`、`normalizationBlocked`、`failedCount` 和 `failures`。只有 `normalization.pendingCount` 和 `failedCount` 均为 `0`，且 `normalizationBlocked` 为 `false` 时，才表示本次迁移全部完成。
### 3. 迁移手动 HTTP 工具 Schema
本版本将手动模式 HTTP 工具的数组参数改为标准 JSON Schema。升级前创建过手动 HTTP 工具的环境需要执行此迁移；OpenAPI 模式的 HTTP 工具无需迁移，脚本会自动跳过。
先执行 dry-run，查看当前应用及历史版本中的待处理数据。dry-run 不会修改数据：
```bash
curl -X POST 'https://你的域名/api/admin/4160/initHttpToolSchema' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":true}'
```
确认结果后执行正式迁移：
```bash
curl -X POST 'https://你的域名/api/admin/4160/initHttpToolSchema' \
-H 'Content-Type: application/json' \
-H 'rootkey: 你的ROOT_KEY' \
-d '{"dryRun":false}'
```
脚本会先按 HTTP 工具类型筛选应用，再根据这些应用的 `appId` 迁移对应的历史版本。仅 `apiSchemaStr` 不存在的手动模式会被处理，其他应用及 OpenAPI 模式不会修改。迁移按批次执行且可安全重试；返回结果中 `total.changedDocumentCount` 表示发现的待处理文档数，正式执行后可再次 dry-run，确认该值为 `0`。
## 🚀 新增内容 ## 🚀 新增内容
1. Agent Sandbox 改为 App 用户级实例，同一 App、同一用户的多个对话复用 Sandbox，并通过独立 session 目录隔离各对话文件。 1. Agent Sandbox 改为 App 用户级实例，同一 App、同一用户的多个对话复用 Sandbox，并通过独立 session 目录隔离各对话文件。
......
document/data/doc-last-modified.json
...@@ -167,8 +167,8 @@ ...@@ -167,8 +167,8 @@
"content/plugin/model-presets.mdx": "2026-06-04T16:10:15+08:00", "content/plugin/model-presets.mdx": "2026-06-04T16:10:15+08:00",
"content/plugin/system-tool-development.en.mdx": "2026-07-02T11:54:55+08:00", "content/plugin/system-tool-development.en.mdx": "2026-07-02T11:54:55+08:00",
"content/plugin/system-tool-development.mdx": "2026-07-02T11:54:55+08:00", "content/plugin/system-tool-development.mdx": "2026-07-02T11:54:55+08:00",
"content/self-host/config/env.en.mdx": "2026-07-30T14:44:03+08:00", "content/self-host/config/env.en.mdx": "2026-07-30T15:22:52+08:00",
"content/self-host/config/env.mdx": "2026-07-30T14:44:03+08:00", "content/self-host/config/env.mdx": "2026-07-30T15:22:52+08:00",
"content/self-host/config/model/intro.en.mdx": "2026-06-04T16:10:15+08:00", "content/self-host/config/model/intro.en.mdx": "2026-06-04T16:10:15+08:00",
"content/self-host/config/model/intro.mdx": "2026-06-04T16:10:15+08:00", "content/self-host/config/model/intro.mdx": "2026-06-04T16:10:15+08:00",
"content/self-host/config/model/minimax.en.mdx": "2026-06-03T10:40:17+08:00", "content/self-host/config/model/minimax.en.mdx": "2026-06-03T10:40:17+08:00",
...@@ -179,12 +179,12 @@ ...@@ -179,12 +179,12 @@
"content/self-host/config/object-storage.mdx": "2026-07-26T21:56:28+08:00", "content/self-host/config/object-storage.mdx": "2026-07-26T21:56:28+08:00",
"content/self-host/config/remote-debug-suite.en.mdx": "2026-06-27T22:05:51+08:00", "content/self-host/config/remote-debug-suite.en.mdx": "2026-06-27T22:05:51+08:00",
"content/self-host/config/remote-debug-suite.mdx": "2026-06-27T22:05:51+08:00", "content/self-host/config/remote-debug-suite.mdx": "2026-06-27T22:05:51+08:00",
"content/self-host/config/sandbox/common.en.mdx": "2026-07-30T14:44:03+08:00", "content/self-host/config/sandbox/common.en.mdx": "2026-07-30T15:22:52+08:00",
"content/self-host/config/sandbox/common.mdx": "2026-07-30T14:44:03+08:00", "content/self-host/config/sandbox/common.mdx": "2026-07-30T15:22:52+08:00",
"content/self-host/config/sandbox/opensandbox.en.mdx": "2026-07-30T14:44:03+08:00", "content/self-host/config/sandbox/opensandbox.en.mdx": "2026-07-30T15:22:52+08:00",
"content/self-host/config/sandbox/opensandbox.mdx": "2026-07-30T14:44:03+08:00", "content/self-host/config/sandbox/opensandbox.mdx": "2026-07-30T15:22:52+08:00",
"content/self-host/config/sandbox/sealosdevbox.en.mdx": "2026-07-30T14:44:03+08:00", "content/self-host/config/sandbox/sealosdevbox.en.mdx": "2026-07-30T15:22:52+08:00",
"content/self-host/config/sandbox/sealosdevbox.mdx": "2026-07-30T14:44:03+08:00", "content/self-host/config/sandbox/sealosdevbox.mdx": "2026-07-30T15:22:52+08:00",
"content/self-host/config/signoz.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/config/signoz.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/config/signoz.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/config/signoz.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/custom-models/bge-rerank.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/custom-models/bge-rerank.en.mdx": "2026-04-26T21:08:47+08:00",
...@@ -329,8 +329,8 @@ ...@@ -329,8 +329,8 @@
"content/self-host/upgrading/4-15/4154.mdx": "2026-07-30T11:22:58+08:00", "content/self-host/upgrading/4-15/4154.mdx": "2026-07-30T11:22:58+08:00",
"content/self-host/upgrading/4-15/4155.en.mdx": "2026-07-30T11:22:58+08:00", "content/self-host/upgrading/4-15/4155.en.mdx": "2026-07-30T11:22:58+08:00",
"content/self-host/upgrading/4-15/4155.mdx": "2026-07-30T11:22:58+08:00", "content/self-host/upgrading/4-15/4155.mdx": "2026-07-30T11:22:58+08:00",
"content/self-host/upgrading/4-16/41601.en.mdx": "2026-07-30T14:44:03+08:00", "content/self-host/upgrading/4-16/41601.en.mdx": "2026-07-30T15:22:52+08:00",
"content/self-host/upgrading/4-16/41601.mdx": "2026-07-30T14:44:03+08:00", "content/self-host/upgrading/4-16/41601.mdx": "2026-07-30T15:22:52+08:00",
"content/self-host/upgrading/outdated/40.en.mdx": "2026-07-25T00:27:20+08:00", "content/self-host/upgrading/outdated/40.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/40.mdx": "2026-07-25T00:27:20+08:00", "content/self-host/upgrading/outdated/40.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/41.en.mdx": "2026-07-25T00:27:20+08:00", "content/self-host/upgrading/outdated/41.en.mdx": "2026-07-25T00:27:20+08:00",
......
packages/global/core/app/tool/httpTool/utils.ts
...@@ -3,13 +3,64 @@ import type { PathDataType, HttpToolConfigType } from './type'; ...@@ -3,13 +3,64 @@ import type { PathDataType, HttpToolConfigType } from './type';
import { type RuntimeNodeItemType } from '../../../workflow/runtime/type'; import { type RuntimeNodeItemType } from '../../../workflow/runtime/type';
import { FlowNodeOutputTypeEnum, FlowNodeTypeEnum } from '../../../workflow/node/constant'; import { FlowNodeOutputTypeEnum, FlowNodeTypeEnum } from '../../../workflow/node/constant';
import { AppToolSourceEnum } from '../constants'; import { AppToolSourceEnum } from '../constants';
import { jsonSchema2NodeInput, jsonSchema2NodeOutput } from '../../jsonschema'; import {
getNodeInputTypeFromSchemaInputType,
jsonSchema2NodeInput,
jsonSchema2NodeOutput
} from '../../jsonschema';
import { type StoreSecretValueType } from '../../../../common/secret/type'; import { type StoreSecretValueType } from '../../../../common/secret/type';
import { type JsonSchemaPropertiesItemType } from '../../jsonschema'; import { type JsonSchemaPropertiesItemType } from '../../jsonschema';
import { NodeOutputKeyEnum, WorkflowIOValueTypeEnum } from '../../../workflow/constants'; import {
NodeOutputKeyEnum,
WorkflowIOValueTypeEnum,
valueTypeJsonSchemaMap
} from '../../../workflow/constants';
import { i18nT } from '../../../../common/i18n/utils'; import { i18nT } from '../../../../common/i18n/utils';
import type { NodeToolConfigType } from '../../../workflow/type/node'; import type { NodeToolConfigType } from '../../../workflow/type/node';
const legacyManualHttpToolArrayTypes = new Set<string>([
WorkflowIOValueTypeEnum.arrayString,
WorkflowIOValueTypeEnum.arrayNumber,
WorkflowIOValueTypeEnum.arrayBoolean
]);
/** 判断 JSON Schema type 是否为手工 HTTP 工具历史上写入的数组值类型。 */
export const isLegacyManualHttpToolArrayType = (
value: unknown
): value is
| WorkflowIOValueTypeEnum.arrayString
| WorkflowIOValueTypeEnum.arrayNumber
| WorkflowIOValueTypeEnum.arrayBoolean =>
typeof value === 'string' && legacyManualHttpToolArrayTypes.has(value);
/** 将手工 HTTP 工具编辑器的工作流值类型转换为标准 JSON Schema property。 */
export const manualHttpToolValueType2JsonSchema = (
valueType: WorkflowIOValueTypeEnum
): JsonSchemaPropertiesItemType => {
const schema =
valueTypeJsonSchemaMap[valueType] ?? valueTypeJsonSchemaMap[WorkflowIOValueTypeEnum.string];
return {
...schema,
...(schema.items && typeof schema.items === 'object' ? { items: { ...schema.items } } : {})
};
};
/** 将标准或历史 HTTP 工具 property Schema 还原为手工编辑器使用的工作流值类型。 */
export const jsonSchemaProperty2ManualHttpToolValueType = (
schema: JsonSchemaPropertiesItemType
): WorkflowIOValueTypeEnum => {
if (isLegacyManualHttpToolArrayType(schema.type)) {
return schema.type;
}
return getNodeInputTypeFromSchemaInputType({
type: typeof schema.type === 'string' ? schema.type : undefined,
arrayItems: schema.items,
schema
});
};
export const getHTTPToolSetRuntimeNode = ({ export const getHTTPToolSetRuntimeNode = ({
name, name,
avatar, avatar,
......
packages/global/test/core/app/tool/httpTool/utils.test.ts
...@@ -2,6 +2,9 @@ import { describe, expect, it } from 'vitest'; ...@@ -2,6 +2,9 @@ import { describe, expect, it } from 'vitest';
import { import {
getHTTPToolSetRuntimeNode, getHTTPToolSetRuntimeNode,
getHTTPToolRuntimeNode, getHTTPToolRuntimeNode,
isLegacyManualHttpToolArrayType,
jsonSchemaProperty2ManualHttpToolValueType,
manualHttpToolValueType2JsonSchema,
parseHttpToolConfig, parseHttpToolConfig,
pathData2ToolList pathData2ToolList
} from '@fastgpt/global/core/app/tool/httpTool/utils'; } from '@fastgpt/global/core/app/tool/httpTool/utils';
...@@ -17,6 +20,45 @@ import { AppToolSourceEnum } from '@fastgpt/global/core/app/tool/constants'; ...@@ -17,6 +20,45 @@ import { AppToolSourceEnum } from '@fastgpt/global/core/app/tool/constants';
import type { HttpToolConfigType, PathDataType } from '@fastgpt/global/core/app/tool/httpTool/type'; import type { HttpToolConfigType, PathDataType } from '@fastgpt/global/core/app/tool/httpTool/type';
describe('httpTool utils', () => { describe('httpTool utils', () => {
describe('manual HTTP tool schema conversion', () => {
it.each([
[
WorkflowIOValueTypeEnum.arrayString,
{
type: 'array',
items: { type: 'string' }
}
],
[
WorkflowIOValueTypeEnum.arrayNumber,
{
type: 'array',
items: { type: 'number' }
}
],
[
WorkflowIOValueTypeEnum.arrayBoolean,
{
type: 'array',
items: { type: 'boolean' }
}
]
] as const)('converts %s to a standard JSON Schema array', (valueType, expectedSchema) => {
expect(manualHttpToolValueType2JsonSchema(valueType)).toEqual(expectedSchema);
expect(jsonSchemaProperty2ManualHttpToolValueType(expectedSchema)).toBe(valueType);
});
it('keeps legacy array value types available for editing before migration', () => {
expect(isLegacyManualHttpToolArrayType('arrayString')).toBe(true);
expect(isLegacyManualHttpToolArrayType('array')).toBe(false);
expect(
jsonSchemaProperty2ManualHttpToolValueType({
type: WorkflowIOValueTypeEnum.arrayString
})
).toBe(WorkflowIOValueTypeEnum.arrayString);
});
});
describe('getHTTPToolSetRuntimeNode', () => { describe('getHTTPToolSetRuntimeNode', () => {
it('should create runtime node with minimal params', () => { it('should create runtime node with minimal params', () => {
const result = getHTTPToolSetRuntimeNode({}); const result = getHTTPToolSetRuntimeNode({});
......
packages/service/core/workflow/dispatch/utils/variables.ts
...@@ -81,8 +81,8 @@ const getVariableInputValue = ({ ...@@ -81,8 +81,8 @@ const getVariableInputValue = ({
variables: Record<string, unknown>; variables: Record<string, unknown>;
item: VariableItemType; item: VariableItemType;
}) => { }) => {
// 内部变量只读自身默认值；外部动态变量只接受 externalVariables 注入。 // 内部变量只读自身默认值，不能被客户端输入或外部变量覆盖。
if (item.type === VariableInputEnum.internal || item.type === VariableInputEnum.custom) { if (item.type === VariableInputEnum.internal) {
return item.defaultValue; return item.defaultValue;
} }
if (item.label && variables[item.label] !== undefined) return variables[item.label]; if (item.label && variables[item.label] !== undefined) return variables[item.label];
......
packages/service/test/core/workflow/dispatch/variables.test.ts
...@@ -71,7 +71,24 @@ describe('WorkflowVariableState', () => { ...@@ -71,7 +71,24 @@ describe('WorkflowVariableState', () => {
expect(state.toStoreRecord()).toEqual({ internalToken: 'internal-default' }); expect(state.toStoreRecord()).toEqual({ internalToken: 'internal-default' });
}); });
it('should only let the external provider override an external dynamic variable', async () => { it('should initialize an external dynamic variable from input variables', async () => {
const state = await createState({
variablesConfig: [
{
key: 'externalToken',
type: VariableInputEnum.custom,
valueType: WorkflowIOValueTypeEnum.string,
defaultValue: 'external-default'
} as any
],
inputVariables: { externalToken: 'input-value' }
});
expect(state.get('externalToken')).toBe('input-value');
expect(state.toStoreRecord()).toEqual({ externalToken: 'input-value' });
});
it('should let the external provider override an input external dynamic variable', async () => {
const state = await createState({ const state = await createState({
variablesConfig: [ variablesConfig: [
{ {
......
pro @ 01fcfa13
Subproject commit 8052827606acad0588c2b497ce205e562657ee6e Subproject commit 01fcfa13c0961733c90673b92f31402d69b8aa8a
projects/app/src/pageComponents/app/detail/Edit/HTTPTools/ManualToolModal.tsx
...@@ -36,7 +36,8 @@ import { ...@@ -36,7 +36,8 @@ import {
type HttpMethod, type HttpMethod,
toolValueTypeList, toolValueTypeList,
ContentTypes, ContentTypes,
VARIABLE_NODE_ID VARIABLE_NODE_ID,
WorkflowIOValueTypeEnum
} from '@fastgpt/global/core/workflow/constants'; } from '@fastgpt/global/core/workflow/constants';
import { import {
headerValue2StoreHeader, headerValue2StoreHeader,
...@@ -53,6 +54,10 @@ import CurlImportModal from './CurlImportModal'; ...@@ -53,6 +54,10 @@ import CurlImportModal from './CurlImportModal';
import type { EditorVariableLabelPickerType } from '@fastgpt/web/components/common/Textarea/PromptEditor/type'; import type { EditorVariableLabelPickerType } from '@fastgpt/web/components/common/Textarea/PromptEditor/type';
import PromptEditor from '@fastgpt/web/components/common/Textarea/PromptEditor'; import PromptEditor from '@fastgpt/web/components/common/Textarea/PromptEditor';
import QuestionTip from '@fastgpt/web/components/common/MyTooltip/QuestionTip'; import QuestionTip from '@fastgpt/web/components/common/MyTooltip/QuestionTip';
import {
jsonSchemaProperty2ManualHttpToolValueType,
manualHttpToolValueType2JsonSchema
} from '@fastgpt/global/core/app/tool/httpTool/utils';
type ManualToolFormType = { type ManualToolFormType = {
name: string; name: string;
...@@ -71,7 +76,7 @@ type ManualToolFormType = { ...@@ -71,7 +76,7 @@ type ManualToolFormType = {
type CustomParamItemType = { type CustomParamItemType = {
key: string; key: string;
description: string; description: string;
type: string; type: WorkflowIOValueTypeEnum;
required: boolean; required: boolean;
isTool: boolean; isTool: boolean;
}; };
...@@ -107,7 +112,7 @@ const ManualToolModal = ({ ...@@ -107,7 +112,7 @@ const ManualToolModal = ({
([key, value]: [string, any]) => ({ ([key, value]: [string, any]) => ({
key, key,
description: value.description || '', description: value.description || '',
type: value.type || 'string', type: jsonSchemaProperty2ManualHttpToolValueType(value),
required: editingTool.inputSchema?.required?.includes(key) || false, required: editingTool.inputSchema?.required?.includes(key) || false,
isTool: !!value['x-tool-description'] isTool: !!value['x-tool-description']
}) })
...@@ -152,7 +157,7 @@ const ManualToolModal = ({ ...@@ -152,7 +157,7 @@ const ManualToolModal = ({
const inputRequired: string[] = []; const inputRequired: string[] = [];
customParams.forEach((param) => { customParams.forEach((param) => {
inputProperties[param.key] = { inputProperties[param.key] = {
type: param.type, ...manualHttpToolValueType2JsonSchema(param.type),
description: param.description || '', description: param.description || '',
'x-tool-description': param.isTool ? param.description : '' 'x-tool-description': param.isTool ? param.description : ''
}; };
...@@ -281,7 +286,7 @@ const ManualToolModal = ({ ...@@ -281,7 +286,7 @@ const ManualToolModal = ({
setEditingParam({ setEditingParam({
key: '', key: '',
description: '', description: '',
type: 'string', type: WorkflowIOValueTypeEnum.string,
required: false, required: false,
isTool: true isTool: true
}); });
......
projects/app/src/pages/api/admin/4160/initHttpToolSchema.ts 0 → 100644
import { NextAPI } from '@/service/middleware/entry';
import type { ApiRequestProps } from '@fastgpt/next/type';
import { BoolSchema, IntSchema } from '@fastgpt/global/common/zod';
import { AppTypeEnum } from '@fastgpt/global/core/app/constants';
import {
isLegacyManualHttpToolArrayType,
manualHttpToolValueType2JsonSchema
} from '@fastgpt/global/core/app/tool/httpTool/utils';
import type { AnyBulkWriteOperation, Model } from '@fastgpt/service/common/mongo';
import { parseApiInput } from '@fastgpt/service/common/zod/requestParseError';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { MongoAppVersion } from '@fastgpt/service/core/app/version/schema';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import z from 'zod';
type UnknownRecord = Record<string, any>;
type WorkflowDocument = {
_id: unknown;
appId?: unknown;
modules?: unknown;
nodes?: unknown;
};
type WorkflowFieldName = 'modules' | 'nodes';
const InitHttpToolSchemaBodySchema = z.object({
dryRun: BoolSchema.optional().default(true),
batchSize: IntSchema.min(1).max(5000).optional().default(500)
});
type InitHttpToolSchemaBodyType = z.infer<typeof InitHttpToolSchemaBodySchema>;
const HttpToolSchemaMigrationStatsSchema = z.object({
scannedDocumentCount: z.number().int().nonnegative(),
changedDocumentCount: z.number().int().nonnegative(),
modifiedDocumentCount: z.number().int().nonnegative(),
convertedPropertyCount: z.number().int().nonnegative()
});
type HttpToolSchemaMigrationStatsType = z.infer<typeof HttpToolSchemaMigrationStatsSchema>;
const InitHttpToolSchemaResponseSchema = z.object({
dryRun: z.boolean(),
batchSize: z.number().int().positive(),
apps: HttpToolSchemaMigrationStatsSchema,
appVersions: HttpToolSchemaMigrationStatsSchema,
total: HttpToolSchemaMigrationStatsSchema
});
type InitHttpToolSchemaResponseType = z.infer<typeof InitHttpToolSchemaResponseSchema>;
const createStats = (): HttpToolSchemaMigrationStatsType => ({
scannedDocumentCount: 0,
changedDocumentCount: 0,
modifiedDocumentCount: 0,
convertedPropertyCount: 0
});
const isRecord = (value: unknown): value is UnknownRecord =>
!!value && typeof value === 'object' && !Array.isArray(value);
/**
* 将工作流节点中手工 HTTP 工具的历史数组类型转换为标准 JSON Schema。
*
* 仅处理 `apiSchemaStr` 未定义的手工模式；返回新节点数组和转换数量，
* 未发生转换时保留原数组引用，便于迁移层跳过无效写入。
*/
export function formatManualHttpToolSchemas(nodes: unknown): {
nodes: unknown;
changed: boolean;
convertedPropertyCount: number;
} {
if (!Array.isArray(nodes)) {
return {
nodes,
changed: false,
convertedPropertyCount: 0
};
}
let convertedPropertyCount = 0;
const formattedNodes = nodes.map((node) => {
if (!isRecord(node) || !isRecord(node.toolConfig)) return node;
const httpToolSet = node.toolConfig.httpToolSet;
if (!isRecord(httpToolSet) || httpToolSet.apiSchemaStr !== undefined) return node;
if (!Array.isArray(httpToolSet.toolList)) return node;
let nodeChanged = false;
const toolList = httpToolSet.toolList.map((tool: unknown) => {
if (!isRecord(tool) || !isRecord(tool.inputSchema)) return tool;
const properties = tool.inputSchema.properties;
if (!isRecord(properties)) return tool;
let toolChanged = false;
const formattedProperties = Object.fromEntries(
Object.entries(properties).map(([key, property]) => {
if (!isRecord(property) || !isLegacyManualHttpToolArrayType(property.type)) {
return [key, property];
}
toolChanged = true;
nodeChanged = true;
convertedPropertyCount += 1;
return [
key,
{
...property,
...manualHttpToolValueType2JsonSchema(property.type)
}
];
})
);
if (!toolChanged) return tool;
return {
...tool,
inputSchema: {
...tool.inputSchema,
properties: formattedProperties
}
};
});
if (!nodeChanged) return node;
return {
...node,
toolConfig: {
...node.toolConfig,
httpToolSet: {
...httpToolSet,
toolList
}
}
};
});
return {
nodes: convertedPropertyCount > 0 ? formattedNodes : nodes,
changed: convertedPropertyCount > 0,
convertedPropertyCount
};
}
/** 按筛选条件流式扫描单个工作流集合，并在非 dry-run 时批量写回已转换文档。 */
async function migrateCollection({
model,
fieldName,
baseQuery,
onDocuments,
options
}: {
model: Model<any>;
fieldName: WorkflowFieldName;
baseQuery: Record<string, unknown>;
onDocuments?: (documents: WorkflowDocument[]) => void;
options: InitHttpToolSchemaBodyType;
}): Promise<HttpToolSchemaMigrationStatsType> {
const stats = createStats();
const migrateBatch = async (documents: WorkflowDocument[]) => {
onDocuments?.(documents);
const operations: AnyBulkWriteOperation<any>[] = [];
stats.scannedDocumentCount += documents.length;
for (const document of documents) {
const result = formatManualHttpToolSchemas(document[fieldName]);
if (!result.changed) continue;
stats.changedDocumentCount += 1;
stats.convertedPropertyCount += result.convertedPropertyCount;
operations.push({
updateOne: {
filter: { _id: document._id },
update: { $set: { [fieldName]: result.nodes } }
}
});
}
if (!options.dryRun && operations.length > 0) {
const result = await model.bulkWrite(operations, { ordered: false });
stats.modifiedDocumentCount += result.modifiedCount;
}
};
const cursor = model
.find(baseQuery, { _id: 1, appId: 1, [fieldName]: 1 })
.lean()
.cursor({ batchSize: options.batchSize });
let documents: WorkflowDocument[] = [];
for await (const document of cursor) {
documents.push(document as WorkflowDocument);
if (documents.length < options.batchSize) continue;
await migrateBatch(documents);
documents = [];
}
if (documents.length > 0) await migrateBatch(documents);
return stats;
}
const mergeStats = (
statsList: HttpToolSchemaMigrationStatsType[]
): HttpToolSchemaMigrationStatsType =>
statsList.reduce(
(total, stats) => ({
scannedDocumentCount: total.scannedDocumentCount + stats.scannedDocumentCount,
changedDocumentCount: total.changedDocumentCount + stats.changedDocumentCount,
modifiedDocumentCount: total.modifiedDocumentCount + stats.modifiedDocumentCount,
convertedPropertyCount: total.convertedPropertyCount + stats.convertedPropertyCount
}),
createStats()
);
/** 扫描当前应用和应用版本，清洗手工 HTTP 工具的历史数组 Schema。 */
export async function runInitHttpToolSchemaMigration(
options: InitHttpToolSchemaBodyType
): Promise<InitHttpToolSchemaResponseType> {
const httpToolAppIds: unknown[] = [];
const apps = await migrateCollection({
model: MongoApp,
fieldName: 'modules',
baseQuery: {
type: AppTypeEnum.httpToolSet
},
onDocuments: (documents) => {
httpToolAppIds.push(...documents.map((document) => document._id));
},
options
});
const appVersions =
httpToolAppIds.length > 0
? await migrateCollection({
model: MongoAppVersion,
fieldName: 'nodes',
baseQuery: {
appId: { $in: httpToolAppIds }
},
options
})
: createStats();
return InitHttpToolSchemaResponseSchema.parse({
dryRun: options.dryRun,
batchSize: options.batchSize,
apps,
appVersions,
total: mergeStats([apps, appVersions])
});
}
/** 4.16.0 HTTP 工具 Schema 管理员迁移入口，默认仅执行 dry-run。 */
async function handler(req: ApiRequestProps): Promise<InitHttpToolSchemaResponseType> {
await authCert({ req, authRoot: true });
const { body } = parseApiInput({
req,
bodySchema: InitHttpToolSchemaBodySchema
});
return runInitHttpToolSchemaMigration(body);
}
export default NextAPI(handler);
projects/app/test/pages/api/admin/4160/initHttpToolSchema.test.ts 0 → 100644
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { AppTypeEnum } from '@fastgpt/global/core/app/constants';
import { WorkflowIOValueTypeEnum } from '@fastgpt/global/core/workflow/constants';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { MongoAppVersion } from '@fastgpt/service/core/app/version/schema';
const mocks = vi.hoisted(() => ({
authCert: vi.fn()
}));
vi.mock('@/service/middleware/entry', () => ({
NextAPI: (handler: unknown) => handler
}));
vi.mock('@fastgpt/service/support/permission/auth/common', () => ({
authCert: mocks.authCert
}));
import handler, {
formatManualHttpToolSchemas,
runInitHttpToolSchemaMigration
} from '@/pages/api/admin/4160/initHttpToolSchema';
const teamId = '65f000000000000000000061';
const tmbId = '65f000000000000000000062';
const appId = '65f000000000000000000063';
const workflowAppId = '65f000000000000000000064';
const createHttpToolSetNode = ({
apiSchemaStr,
propertyType
}: {
apiSchemaStr?: string;
propertyType: string;
}) => ({
nodeId: `http-tool-${propertyType}`,
toolConfig: {
httpToolSet: {
...(apiSchemaStr !== undefined ? { apiSchemaStr } : {}),
toolList: [
{
name: 'search',
description: 'Search',
path: '/search',
method: 'POST',
inputSchema: {
type: 'object',
required: ['values'],
properties: {
values: {
type: propertyType,
description: 'Search values',
'x-tool-description': 'Values provided by the model'
}
}
}
}
]
}
}
});
describe('initHttpToolSchema migration', () => {
beforeEach(async () => {
vi.clearAllMocks();
mocks.authCert.mockResolvedValue(undefined);
await Promise.all([MongoApp.deleteMany({}), MongoAppVersion.deleteMany({})]);
});
it('formats only manual HTTP tool legacy arrays and preserves property metadata', () => {
const manualNode = createHttpToolSetNode({
propertyType: WorkflowIOValueTypeEnum.arrayString
});
const batchNode = createHttpToolSetNode({
apiSchemaStr: '',
propertyType: WorkflowIOValueTypeEnum.arrayNumber
});
const nodes = [manualNode, batchNode];
const result = formatManualHttpToolSchemas(nodes);
expect(result).toMatchObject({
changed: true,
convertedPropertyCount: 1
});
expect((result.nodes as any[])[0]).toMatchObject({
toolConfig: {
httpToolSet: {
toolList: [
{
inputSchema: {
properties: {
values: {
type: 'array',
items: { type: 'string' },
description: 'Search values',
'x-tool-description': 'Values provided by the model'
}
}
}
}
]
}
}
});
expect((result.nodes as any[])[1]).toBe(batchNode);
});
it('dry-runs without writing either collection', async () => {
await MongoApp.create({
_id: appId,
teamId,
tmbId,
name: 'HTTP tools',
type: AppTypeEnum.httpToolSet,
version: 'v2',
modules: [
createHttpToolSetNode({
propertyType: WorkflowIOValueTypeEnum.arrayString
})
],
edges: [],
chatConfig: {}
});
await MongoAppVersion.create({
appId,
tmbId,
versionName: 'v1',
nodes: [
createHttpToolSetNode({
propertyType: WorkflowIOValueTypeEnum.arrayBoolean
})
],
edges: [],
chatConfig: {}
});
const result = await runInitHttpToolSchemaMigration({
dryRun: true,
batchSize: 1
});
expect(result).toMatchObject({
apps: {
scannedDocumentCount: 1,
changedDocumentCount: 1,
modifiedDocumentCount: 0,
convertedPropertyCount: 1
},
appVersions: {
scannedDocumentCount: 1,
changedDocumentCount: 1,
modifiedDocumentCount: 0,
convertedPropertyCount: 1
},
total: {
changedDocumentCount: 2,
convertedPropertyCount: 2
}
});
await expect(MongoApp.findById(appId).lean()).resolves.toMatchObject({
modules: [
{
toolConfig: {
httpToolSet: {
toolList: [
{
inputSchema: {
properties: {
values: { type: WorkflowIOValueTypeEnum.arrayString }
}
}
}
]
}
}
}
]
});
});
it('writes current and version schemas and is idempotent', async () => {
await MongoApp.create([
{
_id: appId,
teamId,
tmbId,
name: 'HTTP tools',
type: AppTypeEnum.httpToolSet,
version: 'v2',
modules: [
createHttpToolSetNode({
propertyType: WorkflowIOValueTypeEnum.arrayNumber
}),
createHttpToolSetNode({
apiSchemaStr: '{"openapi":"3.0.0"}',
propertyType: WorkflowIOValueTypeEnum.arrayBoolean
})
],
edges: [],
chatConfig: {}
},
{
_id: workflowAppId,
teamId,
tmbId,
name: 'Workflow app',
type: AppTypeEnum.workflow,
version: 'v2',
modules: [
createHttpToolSetNode({
propertyType: WorkflowIOValueTypeEnum.arrayBoolean
})
],
edges: [],
chatConfig: {}
}
]);
await MongoAppVersion.create([
{
appId,
tmbId,
versionName: 'v1',
nodes: [
createHttpToolSetNode({
propertyType: WorkflowIOValueTypeEnum.arrayString
})
],
edges: [],
chatConfig: {}
},
{
appId: workflowAppId,
tmbId,
versionName: 'workflow-v1',
nodes: [
createHttpToolSetNode({
propertyType: WorkflowIOValueTypeEnum.arrayBoolean
})
],
edges: [],
chatConfig: {}
}
]);
const firstResult = await runInitHttpToolSchemaMigration({
dryRun: false,
batchSize: 10
});
expect(firstResult.total).toMatchObject({
scannedDocumentCount: 2,
changedDocumentCount: 2,
modifiedDocumentCount: 2,
convertedPropertyCount: 2
});
const app = await MongoApp.findById(appId).lean();
expect(app?.modules[0]).toMatchObject({
toolConfig: {
httpToolSet: {
toolList: [
{
inputSchema: {
properties: {
values: {
type: 'array',
items: { type: 'number' }
}
}
}
}
]
}
}
});
expect(app?.modules[1]).toMatchObject({
toolConfig: {
httpToolSet: {
toolList: [
{
inputSchema: {
properties: {
values: { type: WorkflowIOValueTypeEnum.arrayBoolean }
}
}
}
]
}
}
});
await expect(MongoAppVersion.findOne({ appId }).lean()).resolves.toMatchObject({
nodes: [
{
toolConfig: {
httpToolSet: {
toolList: [
{
inputSchema: {
properties: {
values: {
type: 'array',
items: { type: 'string' }
}
}
}
}
]
}
}
}
]
});
await expect(MongoApp.findById(workflowAppId).lean()).resolves.toMatchObject({
modules: [
{
toolConfig: {
httpToolSet: {
toolList: [
{
inputSchema: {
properties: {
values: { type: WorkflowIOValueTypeEnum.arrayBoolean }
}
}
}
]
}
}
}
]
});
await expect(MongoAppVersion.findOne({ appId: workflowAppId }).lean()).resolves.toMatchObject({
nodes: [
{
toolConfig: {
httpToolSet: {
toolList: [
{
inputSchema: {
properties: {
values: { type: WorkflowIOValueTypeEnum.arrayBoolean }
}
}
}
]
}
}
}
]
});
const secondResult = await runInitHttpToolSchemaMigration({
dryRun: false,
batchSize: 10
});
expect(secondResult.total).toMatchObject({
changedDocumentCount: 0,
modifiedDocumentCount: 0,
convertedPropertyCount: 0
});
});
it('authenticates root and defaults the request to dry-run', async () => {
const req = { body: {} } as any;
await expect(handler(req)).resolves.toMatchObject({
dryRun: true,
batchSize: 500
});
expect(mocks.authCert).toHaveBeenCalledWith({ req, authRoot: true });
});
});
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment