Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit bdee2db7 by Archer Committed by GitHub
Options

V4.14.4 dev (#6058) 

* perf: faq

* index

* delete dataset

* delete dataset

* perf: delete dataset

* init

* fix: outLink UID (#6048)

* perf: query extension

* fix: s3 configs (#6050)

* fix: s3 configs

* s3

---------

Co-authored-by: archer <545436317@qq.com>

* s3 valid string check

* perf: completion api

* fix: model test

* perf: init

* fix: init

* fix: init shell

* fix: faq

---------

Co-authored-by: Roy <whoeverimf5@gmail.com>
parent 44f95038
Showing with 160 additions and 59 deletions
document/content/docs/introduction/development/openapi/share.mdx
......@@ -15,20 +15,20 @@ description: FastGPT 分享链接身份鉴权
### 接口统一响应格式
```json
```jsonc
{
"success": true,
"message": "错误提示",
"msg": "同message, 错误提示",
"data": {
"uid": "用户唯一凭证"
"uid": "用户唯一凭证" // 必须返回
}
}
```
`FastGPT` 将会判断`success`是否为`true`决定是允许用户继续操作。`message`与`msg`是等同的，你可以选择返回其中一个，当`success`不为`true`时，将会提示这个错误。
`uid`是用户的唯一凭证，将会用于拉取对话记录以及保存对话记录。可参考下方实践案例。
`uid` 是用户的唯一凭证，必须返回该 ID 且 ID 的格式为不包含 "|"、"/“、"\" 字符的、长度小于等于 255 的字符串，否则会返回 `Invalid UID` 的错误。`uid` 将会用于拉取对话记录以及保存对话记录，可参考下方实践案例。
### 触发流程
......
document/content/docs/upgrading/4-14/4144.mdx
......@@ -30,6 +30,7 @@ curl --location --request POST 'https://{{host}}/api/admin/initv4144' \
4. 通过 API 上传本地文件至知识库，保存至 S3。同时将旧版 Gridfs 代码全部移除。
5. 新版订阅套餐逻辑。
6. 支持配置对话文件白名单。
7. S3 支持 pathStyle 配置。
## ⚙️ 优化
......@@ -38,6 +39,8 @@ curl --location --request POST 'https://{{host}}/api/admin/initv4144' \
3. 用户通知，支持中英文，以及优化模板。
4. 删除知识库采用队列异步删除模式。
5. LLM 请求时，图片无效报错提示。
6. completions 接口，非 stream 模式， detail=false 时，增加返回 reason_content。
7. 增加对于无效的 S3 key 检测。
## 🐛 修复
......@@ -47,6 +50,9 @@ curl --location --request POST 'https://{{host}}/api/admin/initv4144' \
4. 工作台卡片在名字过长时错位。
5. 分享链接中url query 中携带全局变量时，前端 UI 不会加载该值。
6. window 下判断 CSV 文件异常。
7. 模型测试时，如果模型未启动，会导致无法被测试。
8. MCP header 中带特殊内容时，会抛错。
9. 工作流引用其他 Agent 时，切换版本号后未及时更新 UI。
## 插件
document/data/doc-last-modified.json
......@@ -34,7 +34,7 @@
"document/content/docs/introduction/development/openapi/chat.mdx": "2025-11-14T13:21:17+08:00",
"document/content/docs/introduction/development/openapi/dataset.mdx": "2025-09-29T11:34:11+08:00",
"document/content/docs/introduction/development/openapi/intro.mdx": "2025-09-29T11:34:11+08:00",
"document/content/docs/introduction/development/openapi/share.mdx": "2025-08-05T23:20:39+08:00",
"document/content/docs/introduction/development/openapi/share.mdx": "2025-12-08T16:10:51+08:00",
"document/content/docs/introduction/development/proxy/cloudflare.mdx": "2025-07-23T21:35:03+08:00",
"document/content/docs/introduction/development/proxy/http_proxy.mdx": "2025-07-23T21:35:03+08:00",
"document/content/docs/introduction/development/proxy/nginx.mdx": "2025-07-23T21:35:03+08:00",
......@@ -118,7 +118,7 @@
"document/content/docs/upgrading/4-14/4141.mdx": "2025-11-19T10:15:27+08:00",
"document/content/docs/upgrading/4-14/4142.mdx": "2025-11-18T19:27:14+08:00",
"document/content/docs/upgrading/4-14/4143.mdx": "2025-11-26T20:52:05+08:00",
"document/content/docs/upgrading/4-14/4144.mdx": "2025-12-08T01:44:15+08:00",
"document/content/docs/upgrading/4-14/4144.mdx": "2025-12-08T17:57:59+08:00",
"document/content/docs/upgrading/4-8/40.mdx": "2025-08-02T19:38:37+08:00",
"document/content/docs/upgrading/4-8/41.mdx": "2025-08-02T19:38:37+08:00",
"document/content/docs/upgrading/4-8/42.mdx": "2025-08-02T19:38:37+08:00",
......
packages/service/common/s3/buckets/base.ts
......@@ -50,6 +50,7 @@ export class S3BaseBucket {
port: externalPort,
accessKey: options.accessKey,
secretKey: options.secretKey,
pathStyle: options.pathStyle,
transportAgent: options.transportAgent
});
}
......
packages/service/common/s3/constants.ts
......@@ -52,3 +52,9 @@ export const getSystemMaxFileSize = () => {
const config = global.feConfigs?.uploadFileMaxSize || 1024; // MB, default 1024MB
return config; // bytes
};
export const S3_KEY_PATH_INVALID_CHARS_MAP: Record<string, boolean> = {
'/': true,
'\\': true,
'|': true
};
packages/service/common/s3/controller.ts
......@@ -4,6 +4,7 @@ import { setCron } from '../system/cron';
import { checkTimerLock } from '../system/timerLock/utils';
import { TimerIdEnum } from '../system/timerLock/constants';
import path from 'node:path';
import { S3Error } from 'minio';
export async function clearExpiredMinioFiles() {
try {
......@@ -56,6 +57,12 @@ export async function clearExpiredMinioFiles() {
addLog.warn(`Bucket not found: ${file.bucketName}`);
}
} catch (error) {
if (
error instanceof S3Error &&
error.message.includes('Object name contains unsupported characters.')
) {
await MongoS3TTL.deleteOne({ _id: file._id });
}
fail++;
addLog.error(`Failed to delete minio file: ${file.minioKey}`, error);
}
......
packages/service/core/ai/functions/queryExtension.ts
......@@ -13,7 +13,7 @@ import { useTextCosine } from '../hooks/useTextCosine';
This module can eliminate referential ambiguity and expand queries based on context to improve retrieval.
Submodular Optimization Mode: Generate multiple candidate queries, then use submodular algorithm to select the optimal query combination
*/
const title = global.feConfigs?.systemTitle || 'FastAI';
const title = global.feConfigs?.systemTitle || 'Nginx';
const defaultPrompt = `## 你的任务
你作为一个向量检索助手，你的任务是结合历史记录，为"原问题"生成{{count}}个不同版本的"检索词"。这些检索词应该从不同角度探索主题，以提高向量检索的语义丰富度和精度。
......@@ -230,7 +230,7 @@ assistant: ${chatBg}
.replace(/ /g, '');
try {
const queries = json5.parse(jsonStr) as string[];
let queries = json5.parse(jsonStr) as string[];
if (!Array.isArray(queries) || queries.length === 0) {
return {
......@@ -248,6 +248,8 @@ assistant: ${chatBg}
const { lazyGreedyQuerySelection, embeddingModel: useEmbeddingModel } = useTextCosine({
embeddingModel
});
queries = queries.map((item) => String(item));
const { selectedData: selectedQueries, embeddingTokens } = await lazyGreedyQuerySelection({
originalText: query,
candidates: queries,
......
packages/service/core/ai/llm/request.ts
......@@ -81,7 +81,7 @@ export const createLLMResponse = async <T extends CompletionsBodyType>(
return requestMessages;
})();
const requestBody = await llmCompletionsBodyFormat({
const { requestBody, modelData } = await llmCompletionsBodyFormat({
...body,
messages: rewriteMessages
});
......@@ -89,6 +89,7 @@ export const createLLMResponse = async <T extends CompletionsBodyType>(
// console.log(JSON.stringify(requestBody, null, 2));
const { response, isStreamResponse, getEmptyResponseTip } = await createChatCompletion({
body: requestBody,
modelData,
userKey,
options: {
headers: {
......@@ -491,10 +492,16 @@ const llmCompletionsBodyFormat = async <T extends CompletionsBodyType>({
parallel_tool_calls,
toolCallMode,
...body
}: LLMRequestBodyType<T>): Promise<InferCompletionsBody<T>> => {
}: LLMRequestBodyType<T>): Promise<{
requestBody: InferCompletionsBody<T>;
modelData: LLMModelItemType;
}> => {
const modelData = getLLMModel(body.model);
if (!modelData) {
return body as unknown as InferCompletionsBody<T>;
return {
requestBody: body as unknown as InferCompletionsBody<T>,
modelData
};
}
const response_format = (() => {
......@@ -548,7 +555,10 @@ const llmCompletionsBodyFormat = async <T extends CompletionsBodyType>({
});
}
return requestBody as unknown as InferCompletionsBody<T>;
return {
requestBody: requestBody as unknown as InferCompletionsBody<T>,
modelData
};
};
const createChatCompletion = async ({
modelData,
......@@ -579,6 +589,7 @@ const createChatCompletion = async ({
try {
// Rewrite model
const modelConstantsData = modelData || getLLMModel(body.model);
if (!modelConstantsData) {
return Promise.reject(`${body.model} not found`);
}
......
packages/service/core/dataset/controller.ts
......@@ -115,6 +115,25 @@ export async function delDatasetRelevantData({
// Delete vector data
await deleteDatasetDataVector({ teamId, datasetIds });
// Delete dataset_data_texts in batches by datasetId
for (const datasetId of datasetIds) {
await MongoDatasetDataText.deleteMany({
teamId,
datasetId
}).maxTimeMS(300000); // Reduce timeout for single batch
}
// Delete dataset_datas in batches by datasetId
for (const datasetId of datasetIds) {
await MongoDatasetData.deleteMany({
teamId,
datasetId
}).maxTimeMS(300000);
}
await delCollectionRelatedSource({ collections });
// Delete vector data
await deleteDatasetDataVector({ teamId, datasetIds });
// delete collections
await MongoDatasetCollection.deleteMany({
teamId,
......
packages/service/core/dataset/data/controller.ts
......@@ -5,15 +5,11 @@ import { addDays } from 'date-fns';
import { isS3ObjectKey, jwtSignS3ObjectKey } from '../../../common/s3/utils';
export const formatDatasetDataValue = ({
teamId,
datasetId,
q,
a,
imageId,
imageDescMap
}: {
teamId: string;
datasetId: string;
q: string;
a?: string;
imageId?: string;
......@@ -73,8 +69,6 @@ export const getFormatDatasetCiteList = (list: DatasetDataSchemaType[]) => {
return list.map((item) => ({
_id: item._id,
...formatDatasetDataValue({
teamId: item.teamId,
datasetId: item.datasetId,
q: item.q,
a: item.a,
imageId: item.imageId
......
packages/service/core/dataset/search/controller.ts
......@@ -555,8 +555,6 @@ export async function searchDatasetData(
id: String(data._id),
updateTime: data.updateTime,
...formatDatasetDataValue({
teamId,
datasetId: data.datasetId,
q: data.q,
a: data.a,
imageId: data.imageId,
......@@ -727,8 +725,6 @@ export async function searchDatasetData(
collectionId: String(data.collectionId),
updateTime: data.updateTime,
...formatDatasetDataValue({
teamId,
datasetId: data.datasetId,
q: data.q,
a: data.a,
imageId: data.imageId,
......
packages/service/core/workflow/dispatch/dataset/search.ts
......@@ -14,6 +14,7 @@ import { i18nT } from '../../../../../web/i18n/utils';
import { filterDatasetsByTmbId } from '../../../dataset/utils';
import { getDatasetSearchToolResponsePrompt } from '../../../../../global/core/ai/prompt/dataset';
import { getNodeErrResponse } from '../utils';
import { addLog } from '../../../../common/system/log';
type DatasetSearchProps = ModuleDispatchProps<{
[NodeInputKeyEnum.datasetSelectList]: SelectedDatasetType[];
......@@ -49,7 +50,6 @@ export async function dispatchDatasetSearch(
const {
runningAppInfo: { teamId },
runningUserInfo: { tmbId },
uid,
histories,
node,
params: {
......@@ -281,6 +281,7 @@ export async function dispatchDatasetSearch(
: 'No results'
};
} catch (error) {
addLog.error(`[Dataset search] error`, error);
return getNodeErrResponse({ error });
}
}
projects/app/.env.template
......@@ -43,7 +43,7 @@ S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin
S3_PUBLIC_BUCKET=fastgpt-public # 插件文件存储公开桶
S3_PRIVATE_BUCKET=fastgpt-private # 插件文件存储公开桶
S3_PATH_STYLE=false # forcePathStyle 默认为 true, 当且仅当设置为 false 时关闭, 其他值都为 true
S3_PATH_STYLE=true # forcePathStyle 默认为 true, 当且仅当设置为 false 时关闭, 其他值都为 true
# Redis URL
REDIS_URL=redis://default:mypassword@127.0.0.1:6379
......
projects/app/src/pageComponents/price/FAQ.tsx
......@@ -10,10 +10,6 @@ const FAQ = () => {
desc: t('common:FAQ.switch_package_a')
},
{
title: t('common:FAQ.year_day_q'),
desc: t('common:FAQ.year_day_a')
},
{
title: t('common:FAQ.check_subscription_q'),
desc: t('common:FAQ.check_subscription_a')
},
......@@ -43,6 +39,10 @@ const FAQ = () => {
desc: t('common:FAQ.qpm_a')
},
{
title: t('common:FAQ.year_day_q'),
desc: t('common:FAQ.year_day_a')
},
{
title: t('common:FAQ.free_user_clean_q'),
desc: t('common:FAQ.free_user_clean_a')
}
......
projects/app/src/pages/api/admin/initv4143.ts
......@@ -244,7 +244,7 @@ async function processCollectionBatch({
{
projection: {
_id: 1,
metadata: { teamId: 1 }
metadata: 1
}
}
)
......@@ -259,7 +259,10 @@ async function processCollectionBatch({
// 2. 查找对应的 collections
const fileIds = files.map((f) => f._id);
const collections = await MongoDatasetCollection.find(
{ fileId: { $in: fileIds } },
{
teamId: { $in: Array.from(new Set(files.map((f) => f.metadata?.teamId).filter(Boolean))) },
fileId: { $in: fileIds }
},
'_id fileId teamId datasetId type parentId name updateTime'
).lean();
......@@ -531,6 +534,11 @@ async function processImageBatch({
const imageIds = imageFiles.map((file) => file._id.toString());
const dataList = await MongoDatasetData.find(
{
teamId: { $in: Array.from(new Set(imageFiles.map((file) => file.metadata?.teamId))) },
datasetId: { $in: Array.from(new Set(imageFiles.map((file) => file.metadata?.datasetId))) },
collectionId: {
$in: Array.from(new Set(imageFiles.map((file) => file.metadata?.collectionId)))
},
imageId: { $in: imageIds }
},
'_id imageId teamId datasetId collectionId updateTime'
......
projects/app/src/pages/api/core/ai/model/test.ts
......@@ -71,7 +71,7 @@ export default NextAPI(handler);
const testLLMModel = async (model: LLMModelItemType, headers: Record<string, string>) => {
const { answerText } = await createLLMResponse({
body: {
model: model.model,
model,
messages: [{ role: 'user', content: 'hi' }],
stream: true
},
......
projects/app/src/pages/api/core/dataset/data/getQuoteData.ts
......@@ -80,8 +80,6 @@ async function handler(req: ApiRequestProps<GetQuoteDataProps>): Promise<GetQuot
return {
collection,
...formatDatasetDataValue({
teamId: datasetData.teamId,
datasetId: datasetData.datasetId,
q: datasetData.q,
a: datasetData.a,
imageId: datasetData.imageId
......@@ -98,8 +96,6 @@ async function handler(req: ApiRequestProps<GetQuoteDataProps>): Promise<GetQuot
return {
collection,
...formatDatasetDataValue({
teamId: datasetData.teamId,
datasetId: datasetData.datasetId,
q: datasetData.q,
a: datasetData.a,
imageId: datasetData.imageId
......
projects/app/src/pages/api/v1/chat/completions.ts
......@@ -406,32 +406,59 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
res.end();
} else {
const responseContent = (() => {
if (assistantResponses.length === 0) return '';
if (assistantResponses.length === 1 && assistantResponses[0].text?.content)
return assistantResponses[0].text?.content;
const formatResponseContent = removeAIResponseCite(assistantResponses, retainDatasetCite);
const formattdResponse = (() => {
if (formatResponseContent.length === 0)
return {
reasoning: '',
content: ''
};
if (formatResponseContent.length === 1) {
return {
reasoning: formatResponseContent[0].reasoning?.content,
content: formatResponseContent[0].text?.content
};
}
if (!detail) {
return assistantResponses
.map((item) => item?.text?.content)
.filter(Boolean)
.join('\n');
return {
reasoning: formatResponseContent
.map((item) => item?.reasoning?.content)
.filter(Boolean)
.join('\n'),
content: formatResponseContent
.map((item) => item?.text?.content)
.filter(Boolean)
.join('\n')
};
}
return assistantResponses;
return formatResponseContent;
})();
const formatResponseContent = removeAIResponseCite(responseContent, retainDatasetCite);
const error = flowResponses[flowResponses.length - 1]?.error;
const error =
flowResponses[flowResponses.length - 1]?.error ||
flowResponses[flowResponses.length - 1]?.errorText;
res.json({
...(detail ? { responseData: feResponseData, newVariables } : {}),
error,
id: saveChatId,
id: chatId || '',
model: '',
usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 1 },
choices: [
{
message: { role: 'assistant', content: formatResponseContent },
message: {
role: 'assistant',
...(Array.isArray(formattdResponse)
? { content: formattdResponse }
: {
content: formattdResponse.content,
...(formattdResponse.reasoning && {
reasoning_content: formattdResponse.reasoning
})
})
},
finish_reason: 'stop',
index: 0
}
......
projects/app/src/pages/api/v2/chat/completions.ts
......@@ -402,22 +402,39 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
res.end();
} else {
const responseContent = (() => {
if (assistantResponses.length === 0) return '';
if (assistantResponses.length === 1 && assistantResponses[0].text?.content)
return assistantResponses[0].text?.content;
const formatResponseContent = removeAIResponseCite(assistantResponses, retainDatasetCite);
const formattdResponse = (() => {
if (formatResponseContent.length === 0)
return {
reasoning: '',
content: ''
};
if (formatResponseContent.length === 1) {
return {
reasoning: formatResponseContent[0].reasoning?.content,
content: formatResponseContent[0].text?.content
};
}
if (!detail) {
return assistantResponses
.map((item) => item?.text?.content)
.filter(Boolean)
.join('\n');
return {
reasoning: formatResponseContent
.map((item) => item?.reasoning?.content)
.filter(Boolean)
.join('\n'),
content: formatResponseContent
.map((item) => item?.text?.content)
.filter(Boolean)
.join('\n')
};
}
return assistantResponses;
return formatResponseContent;
})();
const formatResponseContent = removeAIResponseCite(responseContent, retainDatasetCite);
const error = flowResponses[flowResponses.length - 1]?.error;
const error =
flowResponses[flowResponses.length - 1]?.error ||
flowResponses[flowResponses.length - 1]?.errorText;
res.json({
...(detail ? { responseData: feResponseData, newVariables } : {}),
......@@ -427,7 +444,17 @@ async function handler(req: NextApiRequest, res: NextApiResponse) {
usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 1 },
choices: [
{
message: { role: 'assistant', content: formatResponseContent },
message: {
role: 'assistant',
...(Array.isArray(formattdResponse)
? { content: formattdResponse }
: {
content: formattdResponse.content,
...(formattdResponse.reasoning && {
reasoning_content: formattdResponse.reasoning
})
})
},
finish_reason: 'stop',
index: 0
}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment