Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit c483a563 by Archer Committed by GitHub
Options

sandbox tool inject (#6836) 

* sandbox tool inject

* perf: read file prompt

* submodule
parent 83c02c97
Showing with 567 additions and 193 deletions
.claude/design/bug/chat-file-remap-功能开发文档.md
# 功能开发文档
# 功能开发文档
......@@ -38,7 +38,7 @@
| `packages/service/core/workflow/dispatch/ai/chat.ts` | 修改 | 并行处理 human messages，逐条重写 user query，文件内容不进 system | `Promise.all(...rewriteUserQueryWithFileContent(...))` | T2/T3 |
| `packages/service/core/workflow/dispatch/ai/tool/index.ts` | 修改 | Tool LLM messages 同步并行重写；保留 `hasReadFilesTool` skip | `skip: hasReadFilesTool` | T2/T4 |
| `packages/service/core/workflow/utils/context.ts` | 修改/复用 | 承载单条 user query 文件内容重写 helper | `rewriteUserQueryWithFileContent(...)` | T2 |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | 修改/复用 | 保留可读文件 URL 标准化、读文件与解析文件能力，供 readFiles tool 和重写 helper 复用 | `normalizeReadableFileUrl(...)` / `getFileContentFromLinks(...)` | T2 |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | 修改/复用 | 保留可读文件 URL 标准化、读文件与解析文件能力，供 readFiles tool 和重写 helper 复用 | `normalizeReadableFileUrl(...)` / `parseFileContentFromUrls(...)` | T2 |
| `packages/service/core/ai/llm/utils.ts` | 修改/测试驱动 | 保持 `file_url` 过滤，确保同条 text 保留 | 不改协议行为 | T5 |
| `test/cases/...` | 修改/新增 | 替换保存前增强测试，新增运行时逐条注入测试 | 当前轮/历史/Tool/maxFiles | T5 |
......@@ -65,7 +65,7 @@ const userMessages = await Promise.all(
requestOrigin,
maxFiles,
customPdfParse,
getFileContentFromLinks,
parseFileContentFromUrls,
teamId,
tmbId
})
......@@ -95,11 +95,11 @@ N/A（无对外接口结构变化）。
| 模块 | 函数/类型 | 具体改动 | 依赖关系 |
|---|---|---|---|
| `packages/service/core/workflow/dispatch/ai/chat.ts` | `getChatMessages` 附近 | 构造 LLM messages 前，对历史 human 与当前轮 user 做文件内容注入 | 依赖 `getFileContentFromLinks` |
| `packages/service/core/workflow/dispatch/ai/chat.ts` | `getChatMessages` 附近 | 构造 LLM messages 前，对历史 human 与当前轮 user 做文件内容注入 | 依赖 `parseFileContentFromUrls` |
| `packages/service/core/workflow/dispatch/ai/chat.ts` | `getMultiInput` | 不再把文件正文作为 system quote；当前轮文件参与逐条注入 | 与 token 裁剪链路协同 |
| `packages/service/core/workflow/dispatch/ai/tool/index.ts` | `dispatchRunTools` | 与 Chat 路径一致；无 `readFiles` tool 时注入，有则跳过 | 避免与 readFiles tool 重复预解析 |
| `packages/service/core/workflow/utils/context.ts` | `rewriteUserQueryWithFileContent` | 单条 user query 重写 `<FilesContent>`，外层负责并行处理 history/current messages | 通过入参复用 `getFileContentFromLinks` |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | `normalizeReadableFileUrl` / `getFileContentFromLinks` | 统一负责 URL 标准化、过滤、文件读取与解析；按单条 query URL 顺序与 `maxFiles` 控制解析量 | 保持现有错误兜底 |
| `packages/service/core/workflow/utils/context.ts` | `rewriteUserQueryWithFileContent` | 单条 user query 重写 `<FilesContent>`，外层负责并行处理 history/current messages | 通过入参复用 `parseFileContentFromUrls` |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | `normalizeReadableFileUrl` / `parseFileContentFromUrls` | 统一负责 URL 标准化、过滤、文件读取与解析；按单条 query URL 顺序与 `maxFiles` 控制解析量 | 保持现有错误兜底 |
| `packages/service/core/ai/llm/utils.ts` | `loadRequestMessages` | 保持 `file_url` 过滤；回归验证 text part 不丢 | 最终模型请求安全过滤 |
### 3.3 运行时注入算法
......@@ -108,7 +108,7 @@ N/A（无对外接口结构变化）。
2. Chat/Tool 外层通过 `Promise.all` 并行处理运行时 messages。
3. 非 human message 原样返回；human message 调用 `rewriteUserQueryWithFileContent`。
4. 单条 user query 内只收集本条 `file.url`，不做跨 message URL 去重或共享缓存。
5. 调用 `getFileContentFromLinks` 统一完成 URL 标准化、过滤、`maxFiles` 截断与文件解析。
5. 调用 `parseFileContentFromUrls` 统一完成 URL 标准化、过滤、`maxFiles` 截断与文件解析。
6. 将解析结果回填到当前 user query：
- message 原本有 text：追加分隔符和 `<FilesContent>`。
- message 原本无 text：新增 text part。
......@@ -144,7 +144,7 @@ const userMessages = await Promise.all(
maxFiles,
requestOrigin,
customPdfParse,
getFileContentFromLinks,
parseFileContentFromUrls,
teamId,
tmbId
})
......
.claude/design/bug/chat-file-remap-需求设计文档.md
# 需求设计文档
# 需求设计文档
......@@ -129,7 +129,7 @@
| `packages/service/core/workflow/dispatch/ai/chat.ts` | `getMultiInput/getChatMessages` | 构造 LLM messages 前增强运行时副本：历史和当前轮每条 user message 注入自己的文件内容；文件内容不进 system | Chat node 满足历史逐条注入 |
| `packages/service/core/workflow/dispatch/ai/tool/index.ts` | `getMultiInput/dispatchRunTools` | 无 `readFiles` tool 时同 Chat；有 `readFiles` tool 时跳过预解析 | 避免与 readFiles tool 职责冲突 |
| `packages/service/core/workflow/utils/context.ts` | `rewriteUserQueryWithFileContent` | 承载单条 user query 的文件内容重写逻辑，外层并行处理 history/current messages | 不污染 readFiles tool 职责 |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | `normalizeReadableFileUrl` / `getFileContentFromLinks` | `getFileContentFromLinks` 统一负责 URL 标准化、过滤、文件读取与解析；`normalizeReadableFileUrl` 仅作为底层清洗工具 | 不改对外 API |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | `normalizeReadableFileUrl` / `parseFileContentFromUrls` | `parseFileContentFromUrls` 统一负责 URL 标准化、过滤、文件读取与解析；`normalizeReadableFileUrl` 仅作为底层清洗工具 | 不改对外 API |
| `packages/service/core/ai/llm/utils.ts` | `loadRequestMessages` | 保持 `file_url` 过滤逻辑；确保同条消息 text 不被过滤 | 回归保障 |
### 6.4 运行时注入规则
......@@ -137,7 +137,7 @@
1. 使用消息副本，不修改 `histories`、`query`、`userQuestion` 原对象。
2. Chat/Tool 外层用 `Promise.all` 并行处理运行时 messages。
3. 单条 user query 只收集本条 `file.url`；不做跨 message URL 去重，不共享解析缓存。
4. `getFileContentFromLinks` 负责 URL 标准化、过滤、`maxFiles` 截断和文件解析。
4. `parseFileContentFromUrls` 负责 URL 标准化、过滤、`maxFiles` 截断和文件解析。
5. 文件解析结果回填到原本所属的 user message：
- 原 message 已有 text：追加 `\n\n===---===---===\n\n<FilesContent>...`。
- 原 message 只有 file：新增一个 text part 存放 `<FilesContent>`。
......
.codex/design/bug/chat-file-remap-功能开发文档.md
# 功能开发文档
# 功能开发文档
......@@ -38,7 +38,7 @@
| `packages/service/core/workflow/dispatch/ai/chat.ts` | 修改 | 并行处理 human messages，逐条重写 user query，文件内容不进 system | `Promise.all(...rewriteUserQueryWithFileContent(...))` | T2/T3 |
| `packages/service/core/workflow/dispatch/ai/tool/index.ts` | 修改 | Tool LLM messages 同步并行重写；保留 `hasReadFilesTool` skip | `skip: hasReadFilesTool` | T2/T4 |
| `packages/service/core/workflow/utils/context.ts` | 修改/复用 | 承载单条 user query 文件内容重写 helper | `rewriteUserQueryWithFileContent(...)` | T2 |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | 修改/复用 | 保留可读文件 URL 标准化、读文件与解析文件能力，供 readFiles tool 和重写 helper 复用 | `normalizeReadableFileUrl(...)` / `getFileContentFromLinks(...)` | T2 |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | 修改/复用 | 保留可读文件 URL 标准化、读文件与解析文件能力，供 readFiles tool 和重写 helper 复用 | `normalizeReadableFileUrl(...)` / `parseFileContentFromUrls(...)` | T2 |
| `packages/service/core/ai/llm/utils.ts` | 修改/测试驱动 | 保持 `file_url` 过滤，确保同条 text 保留 | 不改协议行为 | T5 |
| `test/cases/...` | 修改/新增 | 替换保存前增强测试，新增运行时逐条注入测试 | 当前轮/历史/Tool/maxFiles | T5 |
......@@ -65,7 +65,7 @@ const userMessages = await Promise.all(
requestOrigin,
maxFiles,
customPdfParse,
getFileContentFromLinks,
parseFileContentFromUrls,
teamId,
tmbId
})
......@@ -95,11 +95,11 @@ N/A（无对外接口结构变化）。
| 模块 | 函数/类型 | 具体改动 | 依赖关系 |
|---|---|---|---|
| `packages/service/core/workflow/dispatch/ai/chat.ts` | `getChatMessages` 附近 | 构造 LLM messages 前，对历史 human 与当前轮 user 做文件内容注入 | 依赖 `getFileContentFromLinks` |
| `packages/service/core/workflow/dispatch/ai/chat.ts` | `getChatMessages` 附近 | 构造 LLM messages 前，对历史 human 与当前轮 user 做文件内容注入 | 依赖 `parseFileContentFromUrls` |
| `packages/service/core/workflow/dispatch/ai/chat.ts` | `getMultiInput` | 不再把文件正文作为 system quote；当前轮文件参与逐条注入 | 与 token 裁剪链路协同 |
| `packages/service/core/workflow/dispatch/ai/tool/index.ts` | `dispatchRunTools` | 与 Chat 路径一致；无 `readFiles` tool 时注入，有则跳过 | 避免与 readFiles tool 重复预解析 |
| `packages/service/core/workflow/utils/context.ts` | `rewriteUserQueryWithFileContent` | 单条 user query 重写 `<FilesContent>`，外层负责并行处理 history/current messages | 通过入参复用 `getFileContentFromLinks` |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | `normalizeReadableFileUrl` / `getFileContentFromLinks` | 统一负责 URL 标准化、过滤、文件读取与解析；按单条 query URL 顺序与 `maxFiles` 控制解析量 | 保持现有错误兜底 |
| `packages/service/core/workflow/utils/context.ts` | `rewriteUserQueryWithFileContent` | 单条 user query 重写 `<FilesContent>`，外层负责并行处理 history/current messages | 通过入参复用 `parseFileContentFromUrls` |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | `normalizeReadableFileUrl` / `parseFileContentFromUrls` | 统一负责 URL 标准化、过滤、文件读取与解析；按单条 query URL 顺序与 `maxFiles` 控制解析量 | 保持现有错误兜底 |
| `packages/service/core/ai/llm/utils.ts` | `loadRequestMessages` | 保持 `file_url` 过滤；回归验证 text part 不丢 | 最终模型请求安全过滤 |
### 3.3 运行时注入算法
......@@ -108,7 +108,7 @@ N/A（无对外接口结构变化）。
2. Chat/Tool 外层通过 `Promise.all` 并行处理运行时 messages。
3. 非 human message 原样返回；human message 调用 `rewriteUserQueryWithFileContent`。
4. 单条 user query 内只收集本条 `file.url`，不做跨 message URL 去重或共享缓存。
5. 调用 `getFileContentFromLinks` 统一完成 URL 标准化、过滤、`maxFiles` 截断与文件解析。
5. 调用 `parseFileContentFromUrls` 统一完成 URL 标准化、过滤、`maxFiles` 截断与文件解析。
6. 将解析结果回填到当前 user query：
- message 原本有 text：追加分隔符和 `<FilesContent>`。
- message 原本无 text：新增 text part。
......@@ -144,7 +144,7 @@ const userMessages = await Promise.all(
maxFiles,
requestOrigin,
customPdfParse,
getFileContentFromLinks,
parseFileContentFromUrls,
teamId,
tmbId
})
......
.codex/design/bug/chat-file-remap-需求设计文档.md
# 需求设计文档
# 需求设计文档
......@@ -129,7 +129,7 @@
| `packages/service/core/workflow/dispatch/ai/chat.ts` | `getMultiInput/getChatMessages` | 构造 LLM messages 前增强运行时副本：历史和当前轮每条 user message 注入自己的文件内容；文件内容不进 system | Chat node 满足历史逐条注入 |
| `packages/service/core/workflow/dispatch/ai/tool/index.ts` | `getMultiInput/dispatchRunTools` | 无 `readFiles` tool 时同 Chat；有 `readFiles` tool 时跳过预解析 | 避免与 readFiles tool 职责冲突 |
| `packages/service/core/workflow/utils/context.ts` | `rewriteUserQueryWithFileContent` | 承载单条 user query 的文件内容重写逻辑，外层并行处理 history/current messages | 不污染 readFiles tool 职责 |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | `normalizeReadableFileUrl` / `getFileContentFromLinks` | `getFileContentFromLinks` 统一负责 URL 标准化、过滤、文件读取与解析；`normalizeReadableFileUrl` 仅作为底层清洗工具 | 不改对外 API |
| `packages/service/core/workflow/dispatch/tools/readFiles.ts` | `normalizeReadableFileUrl` / `parseFileContentFromUrls` | `parseFileContentFromUrls` 统一负责 URL 标准化、过滤、文件读取与解析；`normalizeReadableFileUrl` 仅作为底层清洗工具 | 不改对外 API |
| `packages/service/core/ai/llm/utils.ts` | `loadRequestMessages` | 保持 `file_url` 过滤逻辑；确保同条消息 text 不被过滤 | 回归保障 |
### 6.4 运行时注入规则
......@@ -137,7 +137,7 @@
1. 使用消息副本，不修改 `histories`、`query`、`userQuestion` 原对象。
2. Chat/Tool 外层用 `Promise.all` 并行处理运行时 messages。
3. 单条 user query 只收集本条 `file.url`；不做跨 message URL 去重，不共享解析缓存。
4. `getFileContentFromLinks` 负责 URL 标准化、过滤、`maxFiles` 截断和文件解析。
4. `parseFileContentFromUrls` 负责 URL 标准化、过滤、`maxFiles` 截断和文件解析。
5. 文件解析结果回填到原本所属的 user message：
- 原 message 已有 text：追加 `\n\n===---===---===\n\n<FilesContent>...`。
- 原 message 只有 file：新增一个 text part 存放 `<FilesContent>`。
......
document/content/self-host/upgrading/4-14/41417.mdx 0 → 100644
---
title: 'V4.14.17(处理中)'
description: 'FastGPT V4.14.17 更新说明'
---
## 升级指南
### 1. 更新镜像 tag
- 更新 fastgpt-app(fastgpt 主服务) 镜像 tag: v4.14.17
- 更新 fastgpt-pro(fastgpt 商业版) 镜像 tag: v4.14.17
## 🐛 修复
1. API 知识库 parentId 类型校验错误。
2. 门户页对话无法上传文件。
3. 商业版未包含内部文件解析接口，如果未配置 S3 External Endpoint，会导致文件解析失败。
\ No newline at end of file
document/content/self-host/upgrading/4-14/meta.en.json
......@@ -2,6 +2,7 @@
"title": "4.14.x",
"description": "",
"pages": [
"41417",
"41416",
"41415",
"41414",
......
document/content/self-host/upgrading/4-14/meta.json
......@@ -2,6 +2,7 @@
"title": "4.14.x",
"description": "",
"pages": [
"41417",
"41416",
"41415",
"41414",
......
document/content/self-host/upgrading/4-15/4150.mdx
......@@ -7,6 +7,7 @@ description: 'FastGPT V4.15.0 更新说明'
1. 新增循环节点，弃用旧的批量执行。
2. 全局变量输入框支持输入 object 类型数据。
3. 工具调用模式下，如果开启了虚拟机功能，用户对话框上传的文件会直接注入到虚拟机中。
## ⚙️ 优化
......
document/content/toc.mdx
......@@ -118,6 +118,7 @@ description: FastGPT 文档目录
- [/self-host/upgrading/4-14/41414](/self-host/upgrading/4-14/41414)
- [/self-host/upgrading/4-14/41415](/self-host/upgrading/4-14/41415)
- [/self-host/upgrading/4-14/41416](/self-host/upgrading/4-14/41416)
- [/self-host/upgrading/4-14/41417](/self-host/upgrading/4-14/41417)
- [/self-host/upgrading/4-14/4142](/self-host/upgrading/4-14/4142)
- [/self-host/upgrading/4-14/4143](/self-host/upgrading/4-14/4143)
- [/self-host/upgrading/4-14/4144](/self-host/upgrading/4-14/4144)
......
document/data/doc-last-modified.json
......@@ -251,7 +251,7 @@
"content/self-host/upgrading/4-14/41481.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4149.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4149.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-15/4150.mdx": "2026-04-28T13:31:00+08:00",
"content/self-host/upgrading/4-15/4150.mdx": "2026-04-28T15:10:52+08:00",
"content/self-host/upgrading/outdated/40.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/40.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/41.en.mdx": "2026-04-26T21:08:47+08:00",
......
packages/global/core/ai/sandbox/constants.ts
......@@ -77,11 +77,14 @@ export const SANDBOX_GET_FILE_URL_TOOL: ChatCompletionTool = {
};
// Prompt
export const SANDBOX_SYSTEM_PROMPT = `你拥有一个独立的 Linux 沙盒环境（Ubuntu 22.04），可通过 ${SANDBOX_TOOL_NAME} 工具执行命令：
- 预装：bash / python3 / node / bun / git / curl
export const SANDBOX_USER_FILES_PATH = 'user_files/';
export const SANDBOX_SYSTEM_PROMPT = `## 沙盒能力
你拥有一个独立的 Linux 沙盒环境（Ubuntu 22.04），可通过 ${SANDBOX_TOOL_NAME} 工具执行命令。
- 系统预装：bash / python3 / node / bun / git / curl
- 可自行安装软件包（apt / pip / npm）
- 生成的文件内容都保存在当前目录下即可
- 若需要将生成的文件分享给用户，可使用 ${SANDBOX_GET_FILE_URL_TOOL_NAME} 工具获取文件的临时访问链接`;
- 用户主动上传的文件存储在 ${SANDBOX_USER_FILES_PATH} 目录下
- 若需要将生成的文件链接，可使用 ${SANDBOX_GET_FILE_URL_TOOL_NAME} 工具获取文件的临时访问链接`;
// 聚合
export const sandboxToolMap: Record<
......
packages/service/core/ai/llm/agentLoop/prompt.ts
......@@ -46,16 +46,18 @@ ${list}
/* ===== Inject user query ===== */
export const getUserFilesPrompt = (
files: { id: string; name: string; content?: string }[] = []
files: { id?: string; name: string; sandboxPath?: string; content?: string }[] = []
) => {
if (files.length === 0) return '';
return `# Input Files
本次用户上传的文件：
用户本次上传的文件：
${files
.map((file) =>
`<file>
${file.id ? `<id>${file.id}</id>` : ''}
<name>${file.name}</name>
${file.sandboxPath ? `<sandboxPath>${file.sandboxPath}</sandboxPath>` : ''}
${file.content ? `<content>${file.content}</content>` : ''}
</file>`.trim()
)
......
packages/service/core/ai/sandbox/toolCall/index.ts
......@@ -6,6 +6,9 @@ import { toolMap as getFileUrlToolMap } from './getFileUrl.tool';
import { toolMap as shellToolMap } from './shell.tool';
import { getSandboxClient } from '../controller';
import { parseJsonArgs } from '../../utils';
import { axios } from '../../../../common/api/axios';
import { serverRequestBaseUrl } from '../../../../common/api/serverRequest';
import type { FileWriteEntry } from '@fastgpt-sdk/sandbox-adapter';
const ToolMap = {
...getFileUrlToolMap,
......@@ -74,6 +77,39 @@ export const runSandboxTools = async ({
};
};
export const injectSandboxFiles = async ({
appId,
userId,
chatId,
files
}: {
appId: string;
userId: string;
chatId: string;
files: { path: string; url: string }[];
}) => {
const instance = await getSandboxClient({ appId, userId, chatId });
await instance.ensureAvailable();
const writeFilesData = await Promise.all(
files
.filter((file) => file.path)
.map(async ({ path, url }): Promise<FileWriteEntry> => {
const response = await axios.get<ArrayBuffer>(url, {
baseURL: serverRequestBaseUrl,
responseType: 'arraybuffer'
});
return {
path,
data: response.data
};
})
);
await instance.provider.writeFiles(writeFilesData);
};
export const getSandboxToolInfo = (name: string, lang: localeType = LangEnum.en) => {
if (name in sandboxToolMap) {
const info = sandboxToolMap[name];
......
packages/service/core/workflow/dispatch/ai/chat.ts
......@@ -30,9 +30,9 @@ import { getHistoryPreview } from '@fastgpt/global/core/chat/utils';
import { computedMaxToken } from '../../../ai/utils';
import { formatTime2YMDHM } from '@fastgpt/global/common/string/time';
import type { AiChatQuoteRoleType } from '@fastgpt/global/core/workflow/template/system/aiChat/type';
import { getFileContentFromLinks } from '../../utils/file';
import { parseFileContentFromUrls } from '../../utils/file';
import { parseUrlToFileType } from '../../utils/context';
import { rewriteUserQueryWithFiles } from '../../utils/file';
import { formatUserQueryWithFiles } from '../../utils/file';
import { i18nT } from '../../../../../web/i18n/utils';
import { postTextCensor } from '../../../chat/postTextCensor';
import { createLLMResponse } from '../../../ai/llm/request';
......@@ -167,9 +167,6 @@ export const dispatchChatCompletion = async (props: ChatProps): Promise<ChatResp
})()
]);
console.log(111111);
console.dir(filterMessages, { depth: null });
const {
completeMessages,
reasoningText,
......@@ -422,18 +419,29 @@ const getChatMessages = async ({
return message;
}
const query = await formatUserQueryWithFiles({
userQuery: message.value,
parseFileFn: async (urls) => {
const files = await parseFileContentFromUrls({
urls,
requestOrigin,
maxFiles,
teamId: runningUserInfo.teamId,
tmbId: runningUserInfo.tmbId,
customPdfParse,
usageId
});
return files.map((file) => ({
name: file.name,
content: file.content
}));
}
});
return {
...message,
value: await rewriteUserQueryWithFiles({
queryId: message.dataId || `${index}`,
userQuery: message.value,
requestOrigin,
maxFiles,
customPdfParse,
usageId,
teamId: runningUserInfo.teamId,
tmbId: runningUserInfo.tmbId
})
value: query
};
})
);
......
packages/service/core/workflow/dispatch/ai/toolcall/constants.ts
import { replaceVariable } from '@fastgpt/global/common/string/tools';
import { FlowNodeTypeEnum } from '@fastgpt/global/core/workflow/node/constant';
import { getNanoid } from '@fastgpt/global/common/string/tools';
import type { ChildResponseItemType } from './type';
......
packages/service/core/workflow/dispatch/ai/toolcall/index.ts
......@@ -4,6 +4,7 @@ import type { DispatchNodeResultType } from '@fastgpt/global/core/workflow/runti
import { getLLMModel } from '../../../../ai/model';
import { filterToolNodeIdByEdges, getNodeErrResponse, getHistories } from '../../utils';
import { runToolCall } from './toolCall';
import type { FileInputType } from './type';
import { type DispatchToolModuleProps, type ToolNodeItemType } from './type';
import type { UserChatItemFileItemType, ChatItemMiniType } from '@fastgpt/global/core/chat/type';
import { ChatRoleEnum } from '@fastgpt/global/core/chat/constants';
......@@ -16,11 +17,12 @@ import {
import { getHistoryPreview } from '@fastgpt/global/core/chat/utils';
import { filterToolResponseToPreview } from './utils';
import { parseUrlToFileType } from '../../../utils/context';
import { rewriteUserQueryWithFiles } from '../../../utils/file';
import { formatUserQueryWithFiles, parseFileInfoFromUrls } from '../../../utils/file';
import { postTextCensor } from '../../../../chat/postTextCensor';
import type { FlowNodeInputItemType } from '@fastgpt/global/core/workflow/type/io';
import type { McpToolDataType } from '@fastgpt/global/core/app/tool/mcpTool/type';
import { getToolConfigStatus } from '@fastgpt/global/core/app/formEdit/utils';
import { SANDBOX_USER_FILES_PATH } from '@fastgpt/global/core/ai/sandbox/constants';
type Response = DispatchNodeResultType<{
[NodeOutputKeyEnum.answerText]: string;
......@@ -38,6 +40,7 @@ export const dispatchRunTools = async (props: DispatchToolModuleProps): Promise<
runningUserInfo,
externalProvider,
usageId,
responseChatItemId,
params: {
model,
systemPrompt,
......@@ -46,10 +49,13 @@ export const dispatchRunTools = async (props: DispatchToolModuleProps): Promise<
fileUrlList: fileLinks,
aiChatVision,
aiChatReasoning,
isResponseAnswerText = true
isResponseAnswerText = true,
useAgentSandbox
}
} = props;
const useSandbox = !!useAgentSandbox && !!global.feConfigs?.show_agent_sandbox;
try {
const toolModel = getLLMModel(model);
const useVision = aiChatVision && toolModel.vision;
......@@ -120,11 +126,14 @@ export const dispatchRunTools = async (props: DispatchToolModuleProps): Promise<
.filter(Boolean)
.join('\n\n-----\n\n');
const allFiles = new Map<string, FileInputType>();
const currentInputFiles: FileInputType[] = [];
const messages = await (async () => {
const value: ChatItemMiniType[] = [
...getSystemPrompt_ChatItemType(concatenateSystemPrompt),
...chatHistories,
{
dataId: responseChatItemId,
obj: ChatRoleEnum.Human,
value: runtimePrompt2ChatsValue({
text: userChatInput,
......@@ -142,18 +151,40 @@ export const dispatchRunTools = async (props: DispatchToolModuleProps): Promise<
return message;
}
const prefixId = message.dataId || `${index}`;
const query = await formatUserQueryWithFiles({
userQuery: message.value,
parseFileFn: async (urls) => {
const files = await parseFileInfoFromUrls({
urls,
requestOrigin,
maxFiles,
teamId: runningUserInfo.teamId
}).then((res) =>
res
.filter((item) => item.success)
.map((item, index) => ({
id: `${prefixId}-${index}`,
name: item.name,
url: item.url,
sandboxPath: useSandbox ? `${SANDBOX_USER_FILES_PATH}${item.name}` : undefined
}))
);
files.forEach((file) => {
allFiles.set(file.id, file);
});
if (index === runtimeMessages.length - 1) {
currentInputFiles.push(...files);
}
return files;
}
});
return {
...message,
value: await rewriteUserQueryWithFiles({
queryId: message.dataId || `${index}`,
userQuery: message.value,
requestOrigin,
maxFiles,
customPdfParse: chatConfig?.fileSelectConfig?.customPdfParse,
usageId,
teamId: runningUserInfo.teamId,
tmbId: runningUserInfo.tmbId
})
value: query
};
})
);
......@@ -188,6 +219,8 @@ export const dispatchRunTools = async (props: DispatchToolModuleProps): Promise<
return runToolCall({
...props,
allFiles,
currentInputFiles,
runtimeNodes,
runtimeEdges,
toolNodes,
......
packages/service/core/workflow/dispatch/ai/toolcall/toolCall.ts
......@@ -19,7 +19,18 @@ import type { ToolCallChildrenInteractive } from '@fastgpt/global/core/workflow/
import type { JsonSchemaPropertiesItemType } from '@fastgpt/global/core/app/jsonschema';
import { SANDBOX_SYSTEM_PROMPT, SANDBOX_TOOLS } from '@fastgpt/global/core/ai/sandbox/constants';
import { getSandboxToolWorkflowResponse } from './constants';
import { getSandboxToolInfo, runSandboxTools } from '../../../../ai/sandbox/toolCall';
import {
getSandboxToolInfo,
injectSandboxFiles,
runSandboxTools
} from '../../../../ai/sandbox/toolCall';
import {
dispatchReadFileTool,
ReadFileTooData,
ReadFileToolParamsSchema,
ReadFileToolSchema
} from './tools/file';
import { parseI18nString } from '@fastgpt/global/common/i18n/utils';
type ResponseType = {
requestIds: string[];
......@@ -40,6 +51,8 @@ export const runToolCall = async (props: DispatchToolModuleProps): Promise<Respo
toolNodes,
toolModel,
childrenInteractiveParams,
allFiles,
currentInputFiles,
...workflowProps
} = props;
......@@ -114,7 +127,12 @@ export const runToolCall = async (props: DispatchToolModuleProps): Promise<Respo
};
});
// 注入 sandbox 提示
// 注入 readFile tool
if (allFiles.size > 0) {
tools.push(ReadFileToolSchema);
}
// 注入 sandbox tool
if (useAgentSandbox && global.feConfigs?.show_agent_sandbox) {
// 注入 sandbox_shell 工具
tools.push(...SANDBOX_TOOLS);
......@@ -128,9 +146,27 @@ export const runToolCall = async (props: DispatchToolModuleProps): Promise<Respo
} else {
finalMessages = [{ role: 'system', content: SANDBOX_SYSTEM_PROMPT }, ...messages];
}
// 注入文件到沙盒里
await injectSandboxFiles({
appId: workflowProps.runningAppInfo.id,
userId: workflowProps.uid,
chatId: workflowProps.chatId,
files: currentInputFiles.map((file) => ({
path: file.sandboxPath!,
url: file.url
}))
});
}
const getToolInfo = (name: string) => {
if (name === ReadFileTooData.id) {
return {
type: 'file' as const,
name: parseI18nString(ReadFileTooData.name, workflowProps.lang),
avatar: ReadFileTooData.avatar
};
}
const sandboxToolInfo = getSandboxToolInfo(name, workflowProps.lang);
if (sandboxToolInfo) {
return {
......@@ -276,6 +312,20 @@ export const runToolCall = async (props: DispatchToolModuleProps): Promise<Respo
});
return { response, flowResponse };
} else if (toolInfo.type === 'file') {
const { ids } = ReadFileToolParamsSchema.parse(parseJsonArgs(call.function.arguments));
const { response, usages, nodeResponse } = await dispatchReadFileTool({
files: ids.map((id) => ({ id, url: allFiles.get(id)?.url! })),
teamId: workflowProps.runningUserInfo.teamId,
tmbId: workflowProps.runningUserInfo.tmbId,
customPdfParse: workflowProps.chatConfig?.fileSelectConfig?.customPdfParse,
usageId: workflowProps.usageId
});
return {
response,
usages,
nodeResponse
};
} else {
const toolNode = toolInfo.rawData;
......
packages/service/core/workflow/dispatch/ai/toolcall/tools/file.ts 0 → 100644
import type { ChatCompletionTool } from '@fastgpt/global/core/ai/llm/type';
import type { ChatNodeUsageType } from '@fastgpt/global/support/wallet/bill/type';
import { getFileContentByUrl } from '../../../../utils/file';
import { getErrText } from '@fastgpt/global/common/error/utils';
import { getLogger } from '@fastgpt-sdk/otel/logger';
import { LogCategories } from '../../../../../../common/logger';
import { FlowNodeTypeEnum } from '@fastgpt/global/core/workflow/node/constant';
import { i18nT } from '../../../../../../../web/i18n/utils';
import z from 'zod';
const logger = getLogger(LogCategories.MODULE.AI.TOOL_CALL);
export const ReadFileTooData = {
id: 'read_files',
name: {
'zh-CN': '文件解析',
en: 'File parse',
'zh-Hant': '文件解析'
},
avatar: 'core/workflow/template/readFiles'
};
export const ReadFileToolSchema: ChatCompletionTool = {
type: 'function',
function: {
name: ReadFileTooData.id,
description: '解析文件内容，获取文本。',
parameters: {
type: 'object',
properties: {
ids: { type: 'array', items: { type: 'string' } }
},
required: ['ids']
}
}
};
export const ReadFileToolParamsSchema = z.object({
ids: z.array(z.string())
});
type FileReadParams = {
files: { id: string; url: string }[];
teamId: string;
tmbId: string;
customPdfParse?: boolean;
usageId?: string;
};
export const dispatchReadFileTool = async ({
files,
teamId,
tmbId,
customPdfParse,
usageId
}: FileReadParams) => {
try {
const usages: ChatNodeUsageType[] = [];
const readFilesResult = await Promise.all(
files.map(async ({ url, id }) => {
try {
const { name, content } = await getFileContentByUrl({
url,
teamId,
tmbId,
customPdfParse,
usageId
});
return {
id,
name,
content
};
} catch (error) {
return {
id,
name: url,
content: getErrText(error, 'Load file error')
};
}
})
);
// Stringify the result
const response = readFilesResult
.map(
(file) => `<file>
<id>${file.id}</id>
<content>${file.content}</content>
</file>`
)
.join('\n');
return {
response,
usages,
nodeResponse: {
moduleType: FlowNodeTypeEnum.readFiles,
moduleName: i18nT('chat:read_file')
}
};
} catch (error) {
logger.error('[File Read] Compression failed, using original content', { error });
return {
response: `Failed to read file: ${getErrText(error)}`,
usages: [],
nodeResponse: {
moduleType: FlowNodeTypeEnum.readFiles,
moduleName: i18nT('chat:read_file'),
errorText: `Failed to read file: ${getErrText(error)}`
}
};
}
};
packages/service/core/workflow/dispatch/ai/toolcall/type.ts
......@@ -30,6 +30,8 @@ export type DispatchToolModuleProps = ModuleDispatchProps<{
toolNodes: ToolNodeItemType[];
toolModel: LLMModelItemType;
childrenInteractiveParams?: ToolCallChildrenInteractive['params'];
allFiles: Map<string, FileInputType>;
currentInputFiles: FileInputType[];
};
export type ToolNodeItemType = {
......@@ -49,3 +51,10 @@ export type ChildResponseItemType = {
runTimes: DispatchFlowResponse['runTimes'];
flowUsages: DispatchFlowResponse['flowUsages'];
};
export type FileInputType = {
id: string;
name: string;
url: string;
sandboxPath?: string;
};
packages/service/core/workflow/dispatch/tools/readFiles.ts
......@@ -6,7 +6,7 @@ import { type DispatchNodeResultType } from '@fastgpt/global/core/workflow/runti
import { ChatRoleEnum } from '@fastgpt/global/core/chat/constants';
import { type ChatItemMiniType } from '@fastgpt/global/core/chat/type';
import { getNodeErrResponse } from '../utils';
import { getFileContentFromLinks } from '../../utils/file';
import { parseFileContentFromUrls } from '../../utils/file';
import { getUserFilesPrompt } from '../../../ai/llm/agentLoop/prompt';
import { sliceStrStartEnd } from '@fastgpt/global/common/string/tools';
......@@ -35,7 +35,7 @@ export const dispatchReadFiles = async (props: Props): Promise<Response> => {
const filesFromHistories = version !== '489' ? [] : getHistoryFileLinks(histories);
try {
const readFilesResult = await getFileContentFromLinks({
const readFilesResult = await parseFileContentFromUrls({
// Concat fileUrlList and filesFromHistories; remove not supported files
urls: [...fileUrlList, ...filesFromHistories],
requestOrigin,
......@@ -47,7 +47,7 @@ export const dispatchReadFiles = async (props: Props): Promise<Response> => {
});
const files = readFilesResult.map((item, index) => ({
id: `${index}`,
name: item.filename,
name: item.name,
content: item.content
}));
......@@ -61,14 +61,14 @@ export const dispatchReadFiles = async (props: Props): Promise<Response> => {
data: {
[NodeOutputKeyEnum.text]: text,
[NodeOutputKeyEnum.rawResponse]: readFilesResult.map((item) => ({
filename: item.filename,
filename: item.name,
url: item.url,
text: item.content
}))
},
[DispatchNodeResponseKeyEnum.nodeResponse]: {
readFiles: readFilesResult.map((item) => ({
name: item.filename,
name: item.name,
url: item.url
})),
readFilesResult: getPreviewResponse
......
packages/service/test/core/workflow/dispatch/tools/readFiles.test.ts
......@@ -4,10 +4,10 @@ import type { ChatItemMiniType } from '@fastgpt/global/core/chat/type';
import { NodeOutputKeyEnum } from '@fastgpt/global/core/workflow/constants';
import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/workflow/runtime/constants';
const mockGetFileContentFromLinks = vi.hoisted(() => vi.fn());
const mockparseFileContentFromUrls = vi.hoisted(() => vi.fn());
vi.mock('@fastgpt/service/core/workflow/utils/file', () => ({
getFileContentFromLinks: mockGetFileContentFromLinks
parseFileContentFromUrls: mockparseFileContentFromUrls
}));
import {
......@@ -28,13 +28,13 @@ const baseProps = {
describe('dispatchReadFiles', () => {
beforeEach(() => {
vi.clearAllMocks();
mockGetFileContentFromLinks.mockResolvedValue([]);
mockparseFileContentFromUrls.mockResolvedValue([]);
});
it('成功读取并返回文本/原始响应/节点响应/工具响应结构', async () => {
mockGetFileContentFromLinks.mockResolvedValue([
{ success: true, filename: 'a.pdf', url: '/a.pdf', content: 'Alpha' },
{ success: true, filename: 'b.pdf', url: '/b.pdf', content: 'Beta' }
mockparseFileContentFromUrls.mockResolvedValue([
{ success: true, name: 'a.pdf', url: '/a.pdf', content: 'Alpha' },
{ success: true, name: 'b.pdf', url: '/b.pdf', content: 'Beta' }
]);
const result = await dispatchReadFiles({
......@@ -42,7 +42,7 @@ describe('dispatchReadFiles', () => {
params: { fileUrlList: ['/a.pdf', '/b.pdf'] }
});
expect(mockGetFileContentFromLinks).toHaveBeenCalledWith({
expect(mockparseFileContentFromUrls).toHaveBeenCalledWith({
urls: ['/a.pdf', '/b.pdf'],
requestOrigin: 'http://localhost:3000',
maxFiles: 20,
......@@ -90,7 +90,7 @@ describe('dispatchReadFiles', () => {
params: { fileUrlList: ['/a.pdf'] }
});
expect(mockGetFileContentFromLinks).toHaveBeenCalledWith(
expect(mockparseFileContentFromUrls).toHaveBeenCalledWith(
expect.objectContaining({
maxFiles: 5,
customPdfParse: true
......@@ -105,7 +105,7 @@ describe('dispatchReadFiles', () => {
params: { fileUrlList: ['/a.pdf'] }
});
expect(mockGetFileContentFromLinks).toHaveBeenCalledWith(
expect(mockparseFileContentFromUrls).toHaveBeenCalledWith(
expect.objectContaining({ maxFiles: 20, customPdfParse: false })
);
});
......@@ -117,7 +117,7 @@ describe('dispatchReadFiles', () => {
params: { fileUrlList: ['/a.pdf'] }
});
expect(mockGetFileContentFromLinks).toHaveBeenCalledWith(
expect(mockparseFileContentFromUrls).toHaveBeenCalledWith(
expect.objectContaining({ maxFiles: 20 })
);
});
......@@ -145,7 +145,7 @@ describe('dispatchReadFiles', () => {
params: { fileUrlList: ['/current.pdf'] }
});
expect(mockGetFileContentFromLinks).toHaveBeenCalledWith(
expect(mockparseFileContentFromUrls).toHaveBeenCalledWith(
expect.objectContaining({
urls: ['/current.pdf', '/history.pdf']
})
......@@ -175,7 +175,7 @@ describe('dispatchReadFiles', () => {
params: { fileUrlList: ['/current.pdf'] }
});
expect(mockGetFileContentFromLinks).toHaveBeenCalledWith(
expect(mockparseFileContentFromUrls).toHaveBeenCalledWith(
expect.objectContaining({
urls: ['/current.pdf']
})
......@@ -188,11 +188,13 @@ describe('dispatchReadFiles', () => {
params: {}
});
expect(mockGetFileContentFromLinks).toHaveBeenCalledWith(expect.objectContaining({ urls: [] }));
expect(mockparseFileContentFromUrls).toHaveBeenCalledWith(
expect.objectContaining({ urls: [] })
);
});
it('空文件结果返回空文本和空数组结构', async () => {
mockGetFileContentFromLinks.mockResolvedValue([]);
mockparseFileContentFromUrls.mockResolvedValue([]);
const result = await dispatchReadFiles({
...baseProps,
......@@ -209,8 +211,8 @@ describe('dispatchReadFiles', () => {
it('超大内容下预览仍按 sliceStrStartEnd 截断 (start/end 各 1000)', async () => {
const huge = 'x'.repeat(5000);
mockGetFileContentFromLinks.mockResolvedValue([
{ success: true, filename: 'big.txt', url: '/big.txt', content: huge }
mockparseFileContentFromUrls.mockResolvedValue([
{ success: true, name: 'big.txt', url: '/big.txt', content: huge }
]);
const result = await dispatchReadFiles({
......@@ -226,8 +228,8 @@ describe('dispatchReadFiles', () => {
expect(preview).toContain('## big.txt');
});
it('getFileContentFromLinks 抛错时通过 getNodeErrResponse 返回错误结构', async () => {
mockGetFileContentFromLinks.mockRejectedValue(new Error('boom'));
it('parseFileContentFromUrls 抛错时通过 getNodeErrResponse 返回错误结构', async () => {
mockparseFileContentFromUrls.mockRejectedValue(new Error('boom'));
const result = await dispatchReadFiles({
...baseProps,
......
packages/service/type/env.ts
......@@ -7,7 +7,6 @@ declare global {
PRO_URL: string;
LOG_DEPTH: string;
DB_MAX_LINK: string;
FILE_TOKEN_KEY: string;
STORAGE_VENDOR?: 'minio' | 'aws-s3' | 'cos' | 'oss';
STORAGE_PUBLIC_BUCKET?: string;
......
pro @ 41720ca1
Subproject commit 1d38337167baeed33ece061772c84b0ccce71333
Subproject commit 41720ca13d5c9c85a6f135bb8c71567308ef3d96
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment