Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit e927b94c by Archer Committed by GitHub
Options

perf: quote code (#6977) 

* perf: quote code

* fix: type

* doc

* doc

* add zod schema
parent 22ebfacb
Showing with 339 additions and 201 deletions
document/content/self-host/upgrading/4-15/41502.mdx
......@@ -24,6 +24,7 @@ description: 'FastGPT V4.15.0-beta2 更新说明'
1. 支持 Skill 编辑，Agent 支持 Skill 使用，目前仅支持静态 Skill，无法反向调用系统工具。
2. 重写 agentV2 loop 逻辑。
3. 知识库搜索支持原生多模态 embedding 模型以及图搜图。
4. Chat API dataId 校验：`/v1/chat/completions`、`/v2/chat/completions` 与 `chatTest` 在工作流执行前校验本轮 `dataId` 是否与请求内或当前会话已有记录重复；重复时直接返回业务错误，避免脏数据进入工作流与流恢复合并逻辑。
## ⚙️ 优化
......@@ -44,13 +45,14 @@ description: 'FastGPT V4.15.0-beta2 更新说明'
1. 工作流，单节点调试，存在异常默认值。
2. 模型配置，defaultConfig 覆盖异常。
3. 切换团队时，清除本地 chat 缓存。
4. 流恢复表单输入：刷新或断线续传后，已提交的表单输入值（含 `fileSelect` 文件列表）能正确回填到交互节点内，不再出现空表单或文件消失。
5. 流恢复内容保留：自动续传开始时保留已加载的 AI 输出与节点响应；completed 记录覆盖时不再丢失已恢复的交互表单值与 flow 节点响应。
6. 流恢复交互状态：已提交表单后不再重复追加过期未提交交互；恢复过程中表单默认值能随 `formInputResult` 同步更新。
7. 流恢复历史标题：新对话发起后，侧栏临时历史项优先展示用户输入生成的标题，服务端标题落库后再覆盖，避免长时间显示「新对话」。
8. 切换应用历史串线：修复切换不同应用时，侧栏或会话内容短暂展示其他应用聊天记录的问题。
9. 停止对话提示：移除停止时的 warning toast，改为与后端生成态同步的状态提示。
10. Chat API dataId 校验：`/v1/chat/completions`、`/v2/chat/completions` 与 `chatTest` 在工作流执行前校验本轮 `dataId` 是否与请求内或当前会话已有记录重复；重复时直接返回业务错误，避免脏数据进入工作流与流恢复合并逻辑。
4. 对话流恢复：
- 刷新或断线续传后，已提交的表单输入值（含 `fileSelect` 文件列表）能正确回填到交互节点内，不再出现空表单或文件消失。
- 自动续传开始时保留已加载的 AI 输出与节点响应；completed 记录覆盖时不再丢失已恢复的交互表单值与 flow 节点响应。
- 已提交表单后不再重复追加过期未提交交互；恢复过程中表单默认值能随 `formInputResult` 同步更新。
- 新对话发起后，侧栏临时历史项优先展示用户输入生成的标题，服务端标题落库后再覆盖，避免长时间显示「新对话」。
- 切换不同应用时，侧栏或会话内容短暂展示其他应用聊天记录的问题。
5. 停止对话提示：移除停止时的 warning toast，改为与后端生成态同步的状态提示。
6. v1/completions 接口，nodeResponse 中，quoteList 未返回 `q` , `a`。
## 代码优化
......
document/data/doc-last-modified.json
......@@ -187,8 +187,8 @@
"content/self-host/custom-models/ollama.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/custom-models/xinference.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/custom-models/xinference.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/deploy/docker.en.mdx": "2026-05-07T15:08:21+08:00",
"content/self-host/deploy/docker.mdx": "2026-05-07T15:08:21+08:00",
"content/self-host/deploy/docker.en.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/deploy/docker.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/deploy/sealos.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/deploy/sealos.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/design/dataset.en.mdx": "2026-04-26T21:08:47+08:00",
......@@ -233,8 +233,8 @@
"content/self-host/upgrading/4-14/4140.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4141.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4141.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/41410.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/41410.mdx": "2026-05-23T22:47:02+08:00",
"content/self-host/upgrading/4-14/41410.en.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/4-14/41410.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/4-14/41411.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/41411.mdx": "2026-04-26T21:28:27+08:00",
"content/self-host/upgrading/4-14/41412.en.mdx": "2026-04-26T21:08:47+08:00",
......@@ -278,13 +278,13 @@
"content/self-host/upgrading/4-14/4149.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4149.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-15/4150.mdx": "2026-05-20T17:52:26+08:00",
"content/self-host/upgrading/4-15/41502.mdx": "2026-05-23T23:33:02+08:00",
"content/self-host/upgrading/4-15/41502.mdx": "2026-05-24T17:01:11+08:00",
"content/self-host/upgrading/outdated/40.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/40.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/41.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/41.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4100.en.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/outdated/4100.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/outdated/4100.en.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/4100.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/4101.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4101.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4110.en.mdx": "2026-04-26T21:08:47+08:00",
......@@ -363,8 +363,8 @@
"content/self-host/upgrading/outdated/4818.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4819.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4819.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/482.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/482.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/482.en.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/482.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/4820.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4820.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4821.en.mdx": "2026-04-26T21:08:47+08:00",
......@@ -373,8 +373,8 @@
"content/self-host/upgrading/outdated/4822.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4823.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4823.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/483.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/483.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/483.en.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/483.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/484.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/484.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/485.en.mdx": "2026-04-26T21:08:47+08:00",
......@@ -387,8 +387,8 @@
"content/self-host/upgrading/outdated/488.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/489.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/489.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/490.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/490.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/490.en.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/490.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/491.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/491.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4910.en.mdx": "2026-04-26T21:08:47+08:00",
......
packages/global/common/error/code/common.ts
......@@ -22,19 +22,19 @@ const datasetErr = [
},
{
statusText: CommonErrEnum.fileNotFound,
message: 'error.fileNotFound'
message: i18nT('common:error.fileNotFound')
},
{
statusText: CommonErrEnum.unAuthFile,
message: 'error.unAuthFile'
message: i18nT('common:error.unAuthFile')
},
{
statusText: CommonErrEnum.missingParams,
message: 'error.missingParams'
message: i18nT('common:error.missingParams')
},
{
statusText: CommonErrEnum.inheritPermissionError,
message: 'error.inheritPermissionError'
message: i18nT('common:error.inheritPermissionError')
}
];
export default datasetErr.reduce((acc, cur, index) => {
......
packages/global/common/error/code/dataset.ts
......@@ -20,11 +20,11 @@ export enum DatasetErrEnum {
const datasetErr = [
{
statusText: DatasetErrEnum.sameApiCollection,
message: i18nT('dataset:same_api_collection')
message: i18nT('common:core.dataset.error.sameApiCollection')
},
{
statusText: DatasetErrEnum.notSupportSync,
message: i18nT('dataset:collection_not_support_sync')
message: i18nT('common:core.dataset.error.notSupportSync')
},
{
statusText: DatasetErrEnum.unExist,
......@@ -36,35 +36,39 @@ const datasetErr = [
},
{
statusText: DatasetErrEnum.unAuthDataset,
message: 'core.dataset.error.unAuthDataset'
message: i18nT('common:core.dataset.error.unAuthDataset')
},
{
statusText: DatasetErrEnum.unAuthDatasetCollection,
message: 'core.dataset.error.unAuthDatasetCollection'
message: i18nT('common:core.dataset.error.unAuthDatasetCollection')
},
{
statusText: DatasetErrEnum.unAuthDatasetData,
message: 'core.dataset.error.unAuthDatasetData'
message: i18nT('common:core.dataset.error.unAuthDatasetData')
},
{
statusText: DatasetErrEnum.unAuthDatasetFile,
message: 'core.dataset.error.unAuthDatasetFile'
message: i18nT('common:core.dataset.error.unAuthDatasetFile')
},
{
statusText: DatasetErrEnum.unCreateCollection,
message: 'core.dataset.error.unCreateCollection'
message: i18nT('common:core.dataset.error.unCreateCollection')
},
{
statusText: DatasetErrEnum.unLinkCollection,
message: 'core.dataset.error.unLinkCollection'
message: i18nT('common:core.dataset.error.unLinkCollection')
},
{
statusText: DatasetErrEnum.invalidVectorModelOrQAModel,
message: 'core.dataset.error.invalidVectorModelOrQAModel'
message: i18nT('common:core.dataset.error.invalidVectorModelOrQAModel')
},
{
statusText: DatasetErrEnum.canNotEditAdminPermission,
message: 'core.dataset.error.canNotEditAdminPermission'
message: i18nT('common:core.dataset.error.canNotEditAdminPermission')
},
{
statusText: DatasetErrEnum.noApiServer,
message: i18nT('common:core.dataset.error.noApiServer')
}
];
export default datasetErr.reduce((acc, cur, index) => {
......
packages/global/core/chat/type.ts
import { SearchDataResponseItemSchema } from '../dataset/type';
import { SearchDataResponseQuoteListItemSchema } from '../dataset/type';
import {
ChatFileTypeEnum,
ChatGenerateStatusEnum,
ChatRoleEnum,
type ChatSourceEnum
ChatSourceEnum
} from './constants';
import { FlowNodeTypeEnum } from '../workflow/node/constant';
import { DispatchNodeResponseKeyEnum } from '../workflow/runtime/constants';
import { AppSchemaTypeSchema, type AppSchemaType, type VariableItemType } from '../app/type';
import { AppSchemaTypeSchema, VariableItemTypeSchema } from '../app/type';
import { DispatchNodeResponseSchema } from '../workflow/runtime/type';
import { WorkflowInteractiveResponseTypeSchema } from '../workflow/template/system/interactive/type';
import type { FlowNodeInputItemType } from '../workflow/type/io';
import { FlowNodeInputItemTypeSchema } from '../workflow/type/io';
import z from 'zod';
import {
AgentLoopAskSchema,
......@@ -19,6 +19,7 @@ import {
AgentPlanSchema,
AgentPlanStatusSchema
} from '../ai/agent/type';
import { ObjectIdSchema } from '../../common/type/mongo';
export const ChatHistoryItemResSchema = DispatchNodeResponseSchema.extend({
nodeId: z.string(),
......@@ -80,49 +81,56 @@ export const SkillModuleResponseItemSchema = z.object({
export type SkillModuleResponseItemType = z.infer<typeof SkillModuleResponseItemSchema>;
/* --------- chat ---------- */
export type ChatSchemaType = {
_id: string;
chatId: string;
userId: string;
teamId: string;
tmbId: string;
appId: string;
appVersionId?: string;
createTime: Date;
updateTime: Date;
title: string;
customTitle: string;
top: boolean;
source: `${ChatSourceEnum}`;
sourceName?: string;
shareId?: string;
outLinkUid?: string;
variableList?: VariableItemType[];
welcomeText?: string;
variables: Record<string, any>;
pluginInputs?: FlowNodeInputItemType[];
metadata?: Record<string, any>;
export const ChatSchema = z.object({
_id: ObjectIdSchema,
chatId: z.string(),
userId: ObjectIdSchema,
teamId: ObjectIdSchema,
tmbId: ObjectIdSchema,
appId: ObjectIdSchema.meta({
description: '目前已经变成 sourceId，可能是 app 的，也可能是 skill 的'
}),
appVersionId: ObjectIdSchema.optional().meta({ description: 'appId 为 app 时候才有' }),
createTime: z.coerce.date(),
updateTime: z.coerce.date(),
title: z.string(),
customTitle: z.string().optional(),
top: z.boolean().default(false),
source: z.enum(ChatSourceEnum),
sourceName: z.string().optional(),
shareId: z.string().optional(),
outLinkUid: z.string().optional(),
variableList: z.array(VariableItemTypeSchema).optional(),
welcomeText: z.string().optional(),
variables: z.record(z.string(), z.any()),
pluginInputs: z.array(FlowNodeInputItemTypeSchema).optional(),
metadata: z.record(z.string(), z.any()).optional(),
// Boolean flags for efficient filtering
hasGoodFeedback?: boolean;
hasBadFeedback?: boolean;
hasUnreadGoodFeedback?: boolean;
hasUnreadBadFeedback?: boolean;
hasGoodFeedback: z.boolean().optional(),
hasBadFeedback: z.boolean().optional(),
hasUnreadGoodFeedback: z.boolean().optional(),
hasUnreadBadFeedback: z.boolean().optional(),
// Error count (redundant field for performance)
errorCount?: number;
errorCount: z.number().optional(),
/** 旧数据可能无此字段；业务上按 done 处理 */
chatGenerateStatus?: ChatGenerateStatusEnum;
hasBeenRead: boolean;
chatGenerateStatus: z
.enum(ChatGenerateStatusEnum)
.default(ChatGenerateStatusEnum.done)
.meta({ description: '生成状态' }),
hasBeenRead: z.boolean().default(true),
deleteTime?: Date | null;
};
deleteTime: z.coerce.date().nullish()
});
export type ChatSchemaType = z.infer<typeof ChatSchema>;
export type ChatWithAppSchema = Omit<ChatSchemaType, 'appId'> & {
appId: AppSchemaType;
};
export const ChatWithAppSchema = ChatSchema.omit({ appId: true }).extend({
appId: AppSchemaTypeSchema
});
export type ChatWithAppSchema = z.infer<typeof ChatWithAppSchema>;
/* --------- chat item ---------- */
// User
......@@ -320,7 +328,7 @@ export type ToolCiteLinksType = z.infer<typeof ToolCiteLinksSchema>;
export const ResponseTagItemSchema = z.object({
useAgentSandbox: z.boolean().optional(),
totalQuoteList: z.array(SearchDataResponseItemSchema).optional(),
totalQuoteList: z.array(SearchDataResponseQuoteListItemSchema).optional(),
toolCiteLinks: z.array(ToolCiteLinksSchema).optional(),
errorText: ErrorTextItemSchema.optional(),
llmModuleAccount: z.number().optional().meta({ deprecated: true }),
......
packages/global/core/dataset/type.ts
......@@ -428,6 +428,25 @@ export const SearchDataResponseItemSchema = DatasetDataItemSchema.omit({
.meta({ description: '搜索数据响应项' });
export type SearchDataResponseItemType = z.infer<typeof SearchDataResponseItemSchema>;
export const SearchDataResponseQuoteItemSchema = SearchDataResponseItemSchema.pick({
id: true,
chunkIndex: true,
datasetId: true,
collectionId: true,
sourceId: true,
sourceName: true,
score: true
}).meta({ description: '搜索数据引用响应项（精简）' });
export type SearchDataResponseQuoteItemType = z.infer<typeof SearchDataResponseQuoteItemSchema>;
export const SearchDataResponseQuoteListItemSchema = z.union([
SearchDataResponseItemSchema,
SearchDataResponseQuoteItemSchema
]);
export type SearchDataResponseQuoteListItemType = z.infer<
typeof SearchDataResponseQuoteListItemSchema
>;
export const DatasetCiteItemSchema = z
.object({
_id: ObjectIdSchema.meta({ description: '数据 ID' }),
......
packages/global/core/workflow/runtime/type.ts
......@@ -24,7 +24,7 @@ import type {
InteractiveNodeResponseType,
WorkflowInteractiveResponseType
} from '../template/system/interactive/type';
import { SearchDataResponseItemSchema } from '../../dataset/type';
import { SearchDataResponseQuoteListItemSchema } from '../../dataset/type';
import type { localeType } from '../../../common/i18n/type';
import { type ChatFileStoreValue, type UserChatItemValueItemType } from '../../chat/type';
import { DatasetSearchModeEnum } from '../../dataset/constants';
......@@ -198,7 +198,7 @@ export const DispatchNodeResponseSchema = z
temperature: z.number().optional().meta({ description: '温度' }),
maxToken: z.number().optional().meta({ description: '最大 token' }),
quoteList: z
.array(SearchDataResponseItemSchema)
.array(SearchDataResponseQuoteListItemSchema)
.optional()
.meta({ description: '知识库引用列表' }),
reasoningText: z.string().optional().meta({ description: '思考文本' }),
......
packages/global/test/core/chat/type.test.ts
import { describe, expect, it } from 'vitest';
import {
ChatSchema,
ToolModuleResponseItemSchema,
UserChatItemFileItemSchema,
UserChatItemValueItemSchema,
......@@ -12,7 +13,66 @@ import {
ToolCiteLinksSchema,
RuntimeUserPromptSchema
} from '@fastgpt/global/core/chat/type';
import { ChatRoleEnum, ChatFileTypeEnum } from '@fastgpt/global/core/chat/constants';
import {
ChatRoleEnum,
ChatFileTypeEnum,
ChatGenerateStatusEnum,
ChatSourceEnum
} from '@fastgpt/global/core/chat/constants';
describe('ChatSchema', () => {
const chatId = 'chat-1';
const objectId = '68ee0bd23d17260b7829b137';
it('should validate chat document and coerce date fields', () => {
const result = ChatSchema.safeParse({
_id: objectId,
chatId,
userId: objectId,
teamId: objectId,
tmbId: objectId,
appId: objectId,
createTime: '2026-05-24T00:00:00.000Z',
updateTime: new Date('2026-05-24T00:00:01.000Z'),
title: '历史记录',
customTitle: '',
top: false,
source: ChatSourceEnum.online,
variables: {},
errorCount: 0,
chatGenerateStatus: ChatGenerateStatusEnum.done,
hasBeenRead: false,
deleteTime: null
});
expect(result.success).toBe(true);
if (result.success) {
expect(result.data.createTime).toBeInstanceOf(Date);
expect(result.data.deleteTime).toBeNull();
}
});
it('should reject invalid chat source', () => {
const result = ChatSchema.safeParse({
_id: objectId,
chatId,
userId: objectId,
teamId: objectId,
tmbId: objectId,
appId: objectId,
createTime: new Date(),
updateTime: new Date(),
title: '历史记录',
customTitle: '',
top: false,
source: 'invalid',
variables: {},
hasBeenRead: false
});
expect(result.success).toBe(false);
});
});
describe('ToolModuleResponseItemSchema', () => {
it('should validate valid tool response', () => {
......
packages/service/core/chat/saveChat.ts
......@@ -38,6 +38,7 @@ import { getFlatAppResponses } from '@fastgpt/global/core/chat/utils';
import { getErrText } from '@fastgpt/global/common/error/utils';
import { getNanoid } from '@fastgpt/global/common/string/tools';
import { normalizeChatFileStoreValues } from './fileStoreValue';
import { cloneDeep } from 'lodash';
export type Props = {
chatId: string;
......@@ -178,10 +179,10 @@ const formatAiContent = ({
const { responseData, ...aiResponse } = aiContent;
const citeCollectionIds = new Set<string>();
const cloneResponseData = cloneDeep(responseData || []);
const dealResponseData = (responseItem: ChatHistoryItemResType) => {
getFlatAppResponses(cloneResponseData).forEach((responseItem: ChatHistoryItemResType) => {
if (responseItem.moduleType === FlowNodeTypeEnum.datasetSearchNode && responseItem.quoteList) {
// @ts-ignore
responseItem.quoteList = responseItem.quoteList.map((quote) => {
citeCollectionIds.add(quote.collectionId);
return {
......@@ -195,10 +196,9 @@ const formatAiContent = ({
};
});
}
};
getFlatAppResponses(responseData || []).forEach(dealResponseData);
});
const errorCount = responseData?.filter((item) => item.errorText).length ?? 0;
const errorCount = cloneResponseData?.filter((item) => item.errorText).length ?? 0;
return {
aiResponse: {
......@@ -207,7 +207,7 @@ const formatAiContent = ({
errorMsg,
citeCollectionIds: Array.from(citeCollectionIds)
},
nodeResponses: responseData,
nodeResponses: cloneResponseData,
citeCollectionIds,
errorCount
};
......
packages/service/test/core/chat/saveChat.test.ts
......@@ -349,6 +349,11 @@ describe('pushChatRecords', () => {
await pushChatRecords(props);
expect(props.aiContent.responseData?.[0]?.quoteList?.[0]).toMatchObject({
q: quote.q,
a: quote.a
});
const responses = await MongoChatItemResponse.find({
appId: testAppId,
chatId: props.chatId
......
packages/web/hooks/useLinkedScroll.tsx
......@@ -2,7 +2,7 @@ import { useCallback, useEffect, useRef, useState, type ReactNode } from 'react'
import { type LinkedListResponse, type LinkedPaginationProps } from '@fastgpt/global/openapi/api';
import { Box, type BoxProps } from '@chakra-ui/react';
import { useTranslation } from 'next-i18next';
import { useScroll, useMemoizedFn, useDebounceEffect, useLatest } from 'ahooks';
import { useScroll, useDebounceEffect } from 'ahooks';
import MyBox from '../components/common/MyBox';
import { useRequest } from './useRequest';
......@@ -42,37 +42,44 @@ export function useLinkedScroll<
const isInit = useRef(false);
const scrollToItem = useCallback(
async (id?: string) => {
if (!id) {
id = defaultScroll === 'top' ? dataList[0]?.id : dataList[dataList.length - 1]?.id;
}
(id?: string) => {
const targetId =
id || (defaultScroll === 'top' ? dataList[0]?.id : dataList[dataList.length - 1]?.id);
const itemIndex = dataList.findIndex((item) => item.id === id);
if (itemIndex === -1) {
if (!targetId) {
return;
}
const element = itemRefs.current.get(id);
if (!element || !containerRef.current) {
requestAnimationFrame(() => scrollToItem(id));
const itemIndex = dataList.findIndex((item) => item.id === targetId);
if (itemIndex === -1) {
return;
}
const elementRect = element.getBoundingClientRect();
const containerRect = containerRef.current.getBoundingClientRect();
const tryScroll = () => {
const element = itemRefs.current.get(targetId);
if (!element || !containerRef.current) {
requestAnimationFrame(tryScroll);
return;
}
const elementRect = element.getBoundingClientRect();
const containerRect = containerRef.current.getBoundingClientRect();
const scrollTop = containerRef.current.scrollTop + elementRect.top - containerRect.top;
const scrollTop = containerRef.current.scrollTop + elementRect.top - containerRect.top;
containerRef.current.scrollTo({
top: scrollTop
});
};
containerRef.current.scrollTo({
top: scrollTop
});
tryScroll();
},
[dataList, defaultScroll]
);
const { runAsync: callApi, loading: isLoading } = useRequest(api);
const { runAsync: callApi, loading: isLoading } = useRequest(api, { errorToast: '' });
let scrollSign = useRef(false);
const scrollSign = useRef(false);
const { runAsync: loadInitData } = useRequest(
async ({ scrollWhenFinish, refresh } = { scrollWhenFinish: true, refresh: false }) => {
if (isLoading) return;
......
projects/app/src/components/core/chat/ChatContainer/ChatBox/components/QuoteList.tsx
import React, { useMemo } from 'react';
import { Box, useTheme } from '@chakra-ui/react';
import type { SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type';
import type { SearchDataResponseQuoteListItemType } from '@fastgpt/global/core/dataset/type';
import QuoteItem, { formatScore } from '@/components/core/dataset/QuoteItem';
import { useContextSelector } from 'use-context-selector';
import { WorkflowRuntimeContext } from '../../context/workflowRuntimeContext';
......@@ -15,7 +15,7 @@ const QuoteList = React.memo(function QuoteList({
rawSearch = []
}: {
chatItemDataId?: string;
rawSearch: SearchDataResponseItemType[];
rawSearch: SearchDataResponseQuoteListItemType[];
}) {
const theme = useTheme();
const { appId, outLinkAuthData } = useChatStore();
......@@ -62,7 +62,7 @@ const QuoteList = React.memo(function QuoteList({
};
}
return { ...item, q: item.q || '' };
return { ...item, q: 'q' in item ? item.q : '' };
});
return processedData.sort((a, b) => {
......
projects/app/src/components/core/chat/ChatContainer/ChatBox/components/ResponseTags.tsx
import React, { useMemo, useState } from 'react';
import { Flex, useDisclosure, Box } from '@chakra-ui/react';
import { useTranslation } from 'next-i18next';
import type { SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type';
import type { SearchDataResponseQuoteListItemType } from '@fastgpt/global/core/dataset/type';
import dynamic from 'next/dynamic';
import MyTag from '@fastgpt/web/components/common/Tag/index';
import MyTooltip from '@fastgpt/web/components/common/MyTooltip';
......@@ -95,7 +95,7 @@ const ResponseTags = ({
// Dataset citations
const datasetItems = Object.values(
quoteList.reduce((acc: Record<string, SearchDataResponseItemType[]>, cur) => {
quoteList.reduce((acc: Record<string, SearchDataResponseQuoteListItemType[]>, cur) => {
if (!acc[cur.collectionId]) {
acc[cur.collectionId] = [cur];
}
......@@ -107,9 +107,10 @@ const ResponseTags = ({
type: 'dataset' as const,
key: item.collectionId,
displayText: item.sourceName,
icon: item.imageId
? 'core/dataset/imageFill'
: getSourceNameIcon({ sourceId: item.sourceId, sourceName: item.sourceName }),
icon:
'imageId' in item && item.imageId
? 'core/dataset/imageFill'
: getSourceNameIcon({ sourceId: item.sourceId, sourceName: item.sourceName }),
onClick: () => {
onOpenCiteModal({
collectionId: item.collectionId,
......
projects/app/src/components/core/dataset/QuoteItem.tsx
import React, { useMemo, useState } from 'react';
import { Box, Flex, Link, Progress } from '@chakra-ui/react';
import RawSourceBox from '@/components/core/dataset/RawSourceBox';
import type { SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type';
import type { SearchDataResponseQuoteItemType } from '@fastgpt/global/core/dataset/type';
import NextLink from 'next/link';
import MyIcon from '@fastgpt/web/components/common/Icon';
import { useTranslation } from 'next-i18next';
......@@ -85,7 +85,11 @@ const QuoteItem = ({
canEditDataset,
...RawSourceBoxProps
}: {
quoteItem: SearchDataResponseItemType;
quoteItem: SearchDataResponseQuoteItemType & {
q: string;
a?: string;
imagePreivewUrl?: string;
};
canDownloadSource?: boolean;
canEditData?: boolean;
canEditDataset?: boolean;
......
projects/app/src/global/core/chat/utils.ts
......@@ -5,7 +5,7 @@ import type {
ToolCiteLinksType,
ErrorTextItemType
} from '@fastgpt/global/core/chat/type';
import type { SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type';
import type { SearchDataResponseQuoteListItemType } from '@fastgpt/global/core/dataset/type';
import { FlowNodeTypeEnum } from '@fastgpt/global/core/workflow/node/constant';
import { getFlatAppResponses } from '@fastgpt/global/core/chat/utils';
import { sandboxToolMap } from '@fastgpt/global/core/ai/sandbox/tools';
......@@ -32,8 +32,8 @@ export function transformPreviewHistories(
const extractCitationIdsFromText = (text: string): string[] => {
if (!text) return [];
// Match [24-bit hexadecimal ID](CITE) format
const citeRegex = /\[([a-f0-9]{24})\]\(CITE\)/gi;
// Match [24-bit hexadecimal ID](CITE|QUOTE) format. Markdown rendering supports both.
const citeRegex = /\[([a-f0-9]{24})\]\((?:CITE|QUOTE)\)/gi;
const matches = text.match(citeRegex);
if (!matches) return [];
......@@ -111,7 +111,7 @@ export function addStatisticalDataToHistoryItem(historyItem: ChatItemMiniType) {
},
{
useAgentSandbox: false,
totalQuoteList: [] as SearchDataResponseItemType[],
totalQuoteList: [] as SearchDataResponseQuoteListItemType[],
toolCiteLinks: [] as ToolCiteLinksType[],
linkDedupe: new Set<string>(),
errorText: undefined as ErrorTextItemType | undefined,
......
projects/app/src/pageComponents/chat/ChatQuoteList/CollectionQuoteItem.tsx
import Markdown from '@/components/Markdown';
import { Box, Flex } from '@chakra-ui/react';
import MyTooltip from '@fastgpt/web/components/common/MyTooltip';
import { type Dispatch, type MutableRefObject, type SetStateAction, useState } from 'react';
import { type MutableRefObject, useState } from 'react';
import { useTranslation } from 'next-i18next';
import MyIcon from '@fastgpt/web/components/common/Icon';
import { useCopyData } from '@fastgpt/web/hooks/useCopyData';
......@@ -23,7 +23,7 @@ const CollectionQuoteItem = ({
}: {
quoteRefs: MutableRefObject<Map<string, HTMLDivElement | null>>;
quoteIndex: number;
setQuoteIndex: Dispatch<SetStateAction<number>>;
setQuoteIndex: (quoteIndex: number) => void;
refreshList: () => void;
canEdit: boolean;
......
projects/app/src/pageComponents/chat/ChatQuoteList/CollectionQuoteReader.tsx
import { Box, Flex, HStack } from '@chakra-ui/react';
import { type SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type';
import { type SearchDataResponseQuoteListItemType } from '@fastgpt/global/core/dataset/type';
import { getSourceNameIcon } from '@fastgpt/global/core/dataset/utils';
import MyIcon from '@fastgpt/web/components/common/Icon';
import { useRouter } from 'next/router';
......@@ -28,7 +28,7 @@ const CollectionReader = ({
metadata,
onClose
}: {
rawSearch: SearchDataResponseItemType[];
rawSearch: SearchDataResponseQuoteListItemType[];
metadata: GetCollectionQuoteDataProps;
onClose: () => void;
}) => {
......@@ -39,7 +39,7 @@ const CollectionReader = ({
const canDownloadSource = useContextSelector(ChatItemContext, (v) => v.canDownloadSource);
const { collectionId, datasetId, chatItemDataId, sourceId, sourceName, quoteId } = metadata;
const [quoteIndex, setQuoteIndex] = useState(0);
const [selectedQuote, setSelectedQuote] = useState<{ sourceQuoteId?: string; id: string }>();
// Get dataset permission
const { data: datasetData } = useRequest(async () => await getDatasetPermission(datasetId), {
......@@ -48,7 +48,7 @@ const CollectionReader = ({
});
const filterResults = useMemo(() => {
const res = rawSearch
return rawSearch
.filter((item) => item.collectionId === collectionId)
.sort((a, b) => {
const chunkDiff = (a.chunkIndex || 0) - (b.chunkIndex || 0);
......@@ -56,15 +56,28 @@ const CollectionReader = ({
return a.id.localeCompare(b.id);
});
}, [collectionId, rawSearch]);
if (quoteId) {
setQuoteIndex(res.findIndex((item) => item.id === quoteId));
} else {
setQuoteIndex(0);
}
const quoteIndex = useMemo(() => {
const selectedQuoteId =
selectedQuote && selectedQuote.sourceQuoteId === quoteId ? selectedQuote.id : quoteId;
if (!selectedQuoteId) return 0;
return Math.max(
filterResults.findIndex((item) => item.id === selectedQuoteId),
0
);
}, [filterResults, quoteId, selectedQuote]);
return res;
}, [collectionId, quoteId, rawSearch]);
const setQuoteIndex = (index: number) => {
const nextQuote = filterResults[index];
if (!nextQuote) return;
setSelectedQuote({
sourceQuoteId: quoteId,
id: nextQuote.id
});
};
const currentQuoteItem = useMemo(() => {
const item = filterResults[quoteIndex];
......@@ -281,7 +294,7 @@ const CollectionReader = ({
{isLoading || datasetDataList.length > 0 ? (
<ScrollData flex={'1 0 0'} mt={2} px={5} py={1}>
<Flex flexDir={'column'}>
{formatedDataList.map((item, index) => (
{formatedDataList.map((item) => (
<CollectionQuoteItem
key={item._id}
quoteRefs={itemRefs as React.MutableRefObject<Map<string, HTMLDivElement | null>>}
......
projects/app/src/pageComponents/chat/ChatQuoteList/QuoteReader.tsx
import { Box, Flex } from '@chakra-ui/react';
import { type SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type';
import { type SearchDataResponseQuoteListItemType } from '@fastgpt/global/core/dataset/type';
import MyIcon from '@fastgpt/web/components/common/Icon';
import MyBox from '@fastgpt/web/components/common/MyBox';
import { useTranslation } from 'next-i18next';
......@@ -16,7 +16,7 @@ const QuoteReader = ({
metadata,
onClose
}: {
rawSearch: SearchDataResponseItemType[];
rawSearch: SearchDataResponseQuoteListItemType[];
metadata: GetAllQuoteDataProps;
onClose: () => void;
}) => {
......
projects/app/src/pageComponents/chat/ChatQuoteList/index.tsx
import React from 'react';
import { type SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type';
import { type SearchDataResponseQuoteListItemType } from '@fastgpt/global/core/dataset/type';
import { type GetQuoteProps } from '@/web/core/chat/context/chatItemContext';
import CollectionQuoteReader from './CollectionQuoteReader';
import QuoteReader from './QuoteReader';
......@@ -9,7 +9,7 @@ const ChatQuoteList = ({
metadata,
onClose
}: {
rawSearch: SearchDataResponseItemType[];
rawSearch: SearchDataResponseQuoteListItemType[];
metadata: GetQuoteProps;
onClose: () => void;
}) => {
......
projects/app/src/pages/api/core/chat/record/getCollectionQuote.ts
......@@ -50,7 +50,7 @@ async function handler(req: ApiRequestProps): Promise<GetCollectionQuoteResType>
teamToken
}),
MongoChatItem.findOne({ appId, chatId, dataId: chatItemDataId }, 'time').lean(),
authCollectionInChat({ appId, chatId, chatItemDataId, collectionIds: [collectionId] })
authCollectionInChat({ appId, chatId, collectionIds: [collectionId] })
]);
if (!showFullText || !chat || !chatItem || initialAnchor === undefined) {
......
projects/app/src/pages/api/core/chat/record/getQuote.ts
......@@ -38,7 +38,7 @@ async function handler(req: ApiRequestProps): Promise<GetQuoteResponseType> {
teamToken
}),
MongoChatItem.findOne({ appId, chatId, dataId: chatItemDataId }, 'time').lean(),
authCollectionInChat({ appId, chatId, chatItemDataId, collectionIds: collectionIdList })
authCollectionInChat({ appId, chatId, collectionIds: collectionIdList })
]);
if (!chat || !chatItem || !showCite) return Promise.reject(ChatErrEnum.unAuthChat);
......
projects/app/src/pages/api/core/dataset/collection/export.ts
......@@ -41,8 +41,7 @@ async function handler(req: ApiRequestProps, res: NextApiResponse) {
};
}
const { appId, chatId, chatItemDataId, shareId, outLinkUid, teamId, teamToken, chatTime } =
parseBody;
const { appId, chatId, shareId, outLinkUid, teamId, teamToken, chatTime } = parseBody;
/*
1. auth chat read permission
2. auth collection quote in chat
......@@ -60,7 +59,7 @@ async function handler(req: ApiRequestProps, res: NextApiResponse) {
teamToken
}),
getCollectionWithDataset(collectionId),
authCollectionInChat({ appId, chatId, chatItemDataId, collectionIds: [collectionId] })
authCollectionInChat({ appId, chatId, collectionIds: [collectionId] })
]);
if (!authRes.canDownloadSource) {
......
projects/app/src/pages/api/core/dataset/collection/read.ts
......@@ -48,7 +48,7 @@ async function handler(req: ApiRequestProps): Promise<ReadCollectionSourceRespon
teamToken
}),
getCollectionWithDataset(collectionId),
authCollectionInChat({ appId, chatId, chatItemDataId, collectionIds: [collectionId] })
authCollectionInChat({ appId, chatId, collectionIds: [collectionId] })
]);
if (!authRes.canDownloadSource) {
......
projects/app/src/pages/api/core/dataset/data/getQuoteData.ts
......@@ -23,7 +23,7 @@ async function handler(req: ApiRequestProps): Promise<GetQuoteDataResponse> {
// Auth
const { collection, q, a } = await (async () => {
if (body.chatId && body.appId && body.chatItemDataId) {
const { appId, chatId, shareId, outLinkUid, teamId, teamToken, chatItemDataId } = body;
const { appId, chatId, shareId, outLinkUid, teamId, teamToken } = body;
await authChatCrud({
req,
authToken: true,
......@@ -55,7 +55,6 @@ async function handler(req: ApiRequestProps): Promise<GetQuoteDataResponse> {
authCollectionInChat({
appId,
chatId,
chatItemDataId,
collectionIds: [datasetData.collectionId]
})
]);
......
projects/app/src/service/support/permission/auth/chat.ts
import { type ChatHistoryItemResType, type ChatSchemaType } from '@fastgpt/global/core/chat/type';
import { type ChatSchemaType } from '@fastgpt/global/core/chat/type';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { type AuthModeType } from '@fastgpt/service/support/permission/type';
import { authOutLink } from './outLink';
......@@ -8,13 +8,12 @@ import { AuthUserTypeEnum, ReadPermissionVal } from '@fastgpt/global/support/per
import { authApp } from '@fastgpt/service/support/permission/app/auth';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset';
import { getFlatAppResponses } from '@fastgpt/global/core/chat/utils';
import { MongoChatItemResponse } from '@fastgpt/service/core/chat/chatItemResponseSchema';
import { ChatRoleEnum } from '@fastgpt/global/core/chat/constants';
import type { HelperBotTypeEnum } from '@fastgpt/global/core/chat/helperBot/type';
import { MongoHelperBotChat } from '@fastgpt/service/core/chat/HelperBot/chatSchema';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { MongoApp } from '@fastgpt/service/core/app/schema';
import { Types } from 'mongoose';
/*
检查chat的权限：
......@@ -237,69 +236,54 @@ export async function authChatCrud({
return Promise.reject(ChatErrEnum.unAuthChat);
}
/**
* 校验文档是否来自当前会话引用。
*
* 只依赖 ChatItem 上的 citeCollectionIds 判断 collection 是否在当前会话中被引用，
* 避免读取和解析完整 responseData。
*/
export const authCollectionInChat = async ({
collectionIds,
appId,
chatId,
chatItemDataId
chatId
}: {
collectionIds: string[];
appId: string;
chatId: string;
chatItemDataId: string;
}) => {
try {
// 1. 使用 citeCollectionIds 字段来判断
const chatItems = await MongoChatItem.find(
{
appId,
const appObjectId = Types.ObjectId.isValid(String(appId))
? new Types.ObjectId(String(appId))
: appId;
const targetCollectionIds = collectionIds.map(String);
const [authResult] = await MongoChatItem.aggregate<{ isAuthorized: boolean }>([
{
$match: {
appId: appObjectId,
chatId,
obj: ChatRoleEnum.AI
},
'citeCollectionIds'
)
.sort({ _id: -1 })
.limit(50)
.lean();
const citeCollectionIds = new Set(
chatItems.map((item) => ('citeCollectionIds' in item ? item.citeCollectionIds : [])).flat()
);
if (collectionIds.every((id) => citeCollectionIds.has(id))) {
return;
}
// Adapt <=4.13.0
const chatItem = (await MongoChatItem.findOne(
{
appId,
chatId,
dataId: chatItemDataId
},
'responseData'
).lean()) as { time: Date; responseData?: ChatHistoryItemResType[] };
if (!chatItem) return Promise.reject(DatasetErrEnum.unAuthDatasetFile);
// Concat response data
if (!chatItem.responseData || chatItem.responseData.length === 0) {
const chatItemResponses = await MongoChatItemResponse.find(
{ appId, chatId, chatItemDataId },
{ data: 1 }
).lean();
chatItem.responseData = chatItemResponses.map((item) => item.data);
}
},
{ $sort: { _id: -1 } },
{ $limit: 50 },
{ $unwind: '$citeCollectionIds' },
{
$group: {
_id: null,
citeCollectionIds: { $addToSet: { $toString: '$citeCollectionIds' } }
}
},
{
$project: {
_id: 0,
isAuthorized: { $setIsSubset: [targetCollectionIds, '$citeCollectionIds'] }
}
}
]);
// 找 responseData 里，是否有该文档 id
const flatResData = getFlatAppResponses(chatItem.responseData || []);
const quoteListSet = new Set(
flatResData.map((item) => item.quoteList?.map((quote) => quote.collectionId) || []).flat()
);
if (collectionIds.every((id) => quoteListSet.has(id))) {
return;
}
} catch (error) {}
if (authResult?.isAuthorized) {
return;
}
return Promise.reject(DatasetErrEnum.unAuthDatasetFile);
};
......
projects/app/src/web/core/chat/context/chatItemContext.tsx
import { type ChatBoxInputFormType } from '@/components/core/chat/ChatContainer/ChatBox/type';
import { PluginRunBoxTabEnum } from '@/components/core/chat/ChatContainer/PluginRunBox/constants';
import React, { type ReactNode, useCallback, useEffect, useMemo, useRef, useState } from 'react';
import React, { type ReactNode, useCallback, useMemo, useRef, useState } from 'react';
import { createContext } from 'use-context-selector';
import { type ComponentRef as ChatComponentRef } from '@/components/core/chat/ChatContainer/ChatBox/type';
import { useForm, type UseFormReturn } from 'react-hook-form';
......@@ -8,7 +8,7 @@ import { defaultChatData } from '@/global/core/chat/constants';
import { AppTypeEnum } from '@fastgpt/global/core/app/constants';
import { type AppChatConfigType, type VariableItemType } from '@fastgpt/global/core/app/type';
import { type FlowNodeInputItemType } from '@fastgpt/global/core/workflow/type/io';
import { type SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type';
import { type SearchDataResponseQuoteListItemType } from '@fastgpt/global/core/dataset/type';
import { type OutLinkChatAuthProps } from '@fastgpt/global/support/permission/chat';
import type { ChatGenerateStatusEnum } from '@fastgpt/global/core/chat/constants';
......@@ -65,7 +65,7 @@ export type GetAllQuoteDataProps = GetQuoteDataBasicProps & {
};
export type GetQuoteProps = GetAllQuoteDataProps | GetCollectionQuoteDataProps;
export type QuoteDataType = {
rawSearch: SearchDataResponseItemType[];
rawSearch: SearchDataResponseQuoteListItemType[];
metadata: GetQuoteProps;
};
export type OnOpenCiteModalProps = {
......@@ -131,7 +131,7 @@ export const ChatItemContext = createContext<ChatItemContextType>({
}
});
/*
/*
Chat 对象的上下文
*/
const ChatItemContextProvider = ({
......
projects/app/test/api/core/dataset/data/getQuoteData.test.ts
......@@ -201,7 +201,6 @@ describe('getQuoteData handler', () => {
expect(authCollectionInChatMock).toHaveBeenCalledWith({
appId: VALID_APP_ID,
chatId: 'chat_123',
chatItemDataId: 'item_456',
collectionIds: ['col_1']
});
expect(authDatasetDataMock).not.toHaveBeenCalled();
......
projects/app/test/global/core/chat/utils.test.ts
......@@ -85,4 +85,38 @@ describe('addStatisticalDataToHistoryItem', () => {
expect(addStatisticalDataToHistoryItem(historyItem).useAgentSandbox).toBe(false);
});
it('includes dataset quote tags that use QUOTE markdown links', () => {
const quoteId = '507f1f77bcf86cd799439011';
const historyItem: ChatItemMiniType = {
obj: ChatRoleEnum.AI,
value: [
{
text: {
content: `done [${quoteId}](QUOTE)`
}
}
],
responseData: [
{
id: 'dataset-response',
nodeId: 'dataset-node',
moduleName: 'Dataset Search',
moduleType: FlowNodeTypeEnum.datasetSearchNode,
quoteList: [
{
id: quoteId,
chunkIndex: 0,
datasetId: 'dataset-1',
collectionId: 'collection-1',
sourceName: 'doc.pdf',
score: [{ type: 'embedding', value: 0.9, index: 0 }]
}
]
}
]
};
expect(addStatisticalDataToHistoryItem(historyItem).totalQuoteList).toHaveLength(1);
});
});
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment