Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 13b10720 by Archer Committed by GitHub
Options

v4.6.6-search test adapt diff search mode (#685)

parent c766a0ed
Showing with 703 additions and 350 deletions
.vscode/settings.json
......@@ -5,7 +5,6 @@
"prettier.prettierPath": "./node_modules/prettier",
"i18n-ally.localesPaths": [
"projects/app/public/locales",
"projects/home/public/locales"
],
"i18n-ally.enabledParsers": ["json"],
"i18n-ally.keystyle": "nested",
......
README.md
......@@ -48,19 +48,20 @@ https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409b
<img src="https://img.shields.io/badge/-返回顶部-7d09f1.svg" alt="#" align="right">
</a>
## 💡 功能
## 💡 RoadMap
`1` 强大的可视化编排，轻松构建 AI 应用
`1` 应用编排能力
- [x] 提供简易模式，无需操作编排
- [x] 对话下一步指引
- [x] 工作流编排
- [x] 对话下一步指引
- [x] 源文件引用追踪
- [x] 模块封装，实现多级复用
- [x] 混合检索 & 重排
- [ ] 自查询规划
- [ ] 嵌入 [Laf](https://github.com/labring/laf)，实现在线编写 HTTP 模块
- [ ] 混合检索 & 重排
- [ ] 插件封装功能
`2` 丰富的知识库预处理
`2` 知识库能力
- [x] 多库复用，混用
- [x] chunk 记录修改和删除
- [x] 支持知识库单独设置向量模型
......@@ -68,22 +69,26 @@ https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409b
- [x] 支持手动输入，直接分段，QA 拆分导入
- [x] 支持 pdf、word、txt、md 等常用文件，支持 url 读取、CSV 批量导入
- [ ] 支持 HTML、csv、PPT、Excel 导入
- [ ] 支持文件阅读器
- [ ] 支持差异性文件同步
- [ ] 更多的数据预处理方案
`3` 多种效果测试渠道
`3` 应用调试能力
- [x] 知识库单点搜索测试
- [x] 对话时反馈引用并可修改与删除
- [x] 完整上下文呈现
- [x] 完整模块中间值呈现
- [ ] 高级编排 DeBug 模式
`4` OpenAPI
`4` OpenAPI 接口
- [x] completions 接口 (对齐 GPT 接口)
- [ ] 知识库 CRUD
- [ ] 对话 CRUD
`5` 运营功能
`5` 运营能力
- [x] 免登录分享窗口
- [x] Iframe 一键嵌入
- [x] 聊天窗口嵌入支持自定义 Icon，默认打开，拖拽等功能
- [x] 统一查阅对话记录，并对数据进行标注
<a href="#readme">
......
docSite/content/docs/development/configuration.md
......@@ -307,3 +307,27 @@ weight: 708
}
}
```
## 特殊模型
### ReRank 接入
请使用 4.6.6-alpha 以上版本，配置文件中的 `reRankModels` 为重排模型，虽然是数组，不过目前仅有第1个生效。
1. [部署 ReRank 模型](/docs/development/custom-models/reranker/)
1. 找到 FastGPT 的配置文件中的 `reRankModels`， 4.6.6 以前是 `ReRankModels`。
2. 修改对应的值：（记得去掉注释）
```json
{
"reRankModels": [
{
"model": "bge-reranker-base", // 随意
"name": "检索重排-base", // 随意
"inputPrice": 0,
"requestUrl": "{{host}}/api/v1/rerank",
"requestAuth": "安全凭证，已自动补 Bearer"
}
]
}
```
docSite/content/docs/development/custom-models/reranker.md 0 → 100644
---
title: '接入 ReRank 重排模型'
description: '接入 ReRank 重排模型'
icon: 'sort'
draft: false
toc: true
weight: 910
---
## 推荐配置
推荐配置如下：
{{< table "table-hover table-striped-columns" >}}
| 类型 | 内存 | 显存 | 硬盘空间 | 启动命令 |
|------|---------|---------|----------|--------------------------|
| base | >=4GB | >=3GB | >=8GB | python app.py |
{{< /table >}}
## 部署
### 环境要求
- Python 3.10.11
- CUDA 11.7
- 科学上网环境
### 源码部署
1. 根据上面的环境配置配置好环境，具体教程自行 GPT；
2. 下载 [python 文件](app.py)
3. 在命令行输入命令 `pip install -r requirments.txt`；
4. 按照[https://huggingface.co/BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base)下载模型仓库到app.py同级目录
5. 添加环境变量 `export ACCESS_TOKEN=XXXXXX` 配置 token，这里的 token 只是加一层验证，防止接口被人盗用，默认值为 `ACCESS_TOKEN` ；
6. 执行命令 `python app.py`。
然后等待模型下载，直到模型加载完毕为止。如果出现报错先问 GPT。
启动成功后应该会显示如下地址：
![](/imgs/chatglm2.png)
> 这里的 `http://0.0.0.0:6006` 就是连接地址。
### docker 部署
+ 镜像名: `luanshaotong/reranker:v0.1`
+ 端口号: 6006
+ 大小：约8GB
**设置安全凭证（即oneapi中的渠道密钥）**
```
ACCESS_TOKEN=mytoken
```
**运行命令示例**
```sh
docker run -d --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken luanshaotong/reranker:v0.1
```
## 接入 FastGPT
参考 [ReRank模型接入](/docs/development/configuration/#rerank-接入)，host 变量为部署的域名。
\ No newline at end of file
docSite/content/docs/development/openapi/dataset.md
......@@ -157,7 +157,7 @@ A2:
## 搜索测试
{{< tabs tabTotal="2" >}}
{{< tabs tabTotal="3" >}}
{{< tab tabName="请求示例" >}}
{{< markdownify >}}
......@@ -168,18 +168,33 @@ curl --location --request POST 'https://api.fastgpt.in/api/core/dataset/searchTe
--data-raw '{
"datasetId": "知识库的ID",
"text": "导演是谁",
"rarank": true,
"limit": 20
"limit": 5000,
"similarity": 0,
"searchMode": "embedding",
"usingReRank": false
}'
```
{{< /markdownify >}}
{{< /tab >}}
{{< tab tabName="参数说明" >}}
{{< markdownify >}}
- datasetId - 知识库ID
- text - 需要测试的文本
- limit - 最大 tokens 数量
- similarity - 最低相关度（0~1，可选）
- searchMode - 搜索模式：embedding | fullTextRecall | mixedRecall
- usingReRank - 使用重排
{{< /markdownify >}}
{{< /tab >}}
{{< tab tabName="响应示例" >}}
{{< markdownify >}}
返回 top k 结果， limit 为预估条数，会按每条数据 800 tokens 的长度进行预估，20条也就是返回 16000 tokens 长度的数据，最多测试 30000 tokens 的数据。
返回 top k 结果， limit 为最大 Tokens 数量，最多 20000 tokens。
```bash
{
......@@ -190,15 +205,6 @@ curl --location --request POST 'https://api.fastgpt.in/api/core/dataset/searchTe
"id": "65599c54a5c814fb803363cb",
"q": "你是谁",
"a": "我是FastGPT助手",
"indexes": [
{
"defaultIndex": true,
"type": "qa",
"dataId": "3645952",
"text": "你是谁\n我是FastGPT助手",
"_id": "65599c5588271af95b019862"
}
],
"datasetId": "6554684f7f9ed18a39a4d15c",
"collectionId": "6556cd795e4b663e770bb66d",
"sourceName": "GBT 15104-2021 装饰单板贴面人造板.pdf",
......
docSite/content/docs/development/qa.md
......@@ -7,12 +7,29 @@ draft: false
images: []
---
## 错误排查方式
遇到问题先按下面方式排查。
1. `docker ps -a` 查看所有容器运行状态，检查是否全部 running，如有异常，尝试`docker logs 容器名`查看对应日志。
2. 不懂 docker 不要瞎改端口，只需要改`OPENAI_BASE_URL`和`CHAT_API_KEY`即可。
3. 容器都运行正常的，`docker logs 容器名` 查看报错日志
4. 无法解决时，可以找找[Issue](https://github.com/labring/FastGPT/issues)，或新提 Issue，私有部署错误，务必提供详细的日志，否则很难排查。
## 通用问题
### insufficient_user_quota user quota is not enough
OneAPI 账号的余额不足，默认 root 用户只有 200 刀，可以手动修改。
### xxx渠道找不到
OneAPI 中没有配置该模型渠道。
### 页面中可以正常回复，API 报错
页面中是用 stream=true 模式，所以API也需要设置 stream=true 来进行测试。部分模型接口（国产居多）非 Stream 的兼容有点垃圾。
## Docker 部署常见问题
......@@ -66,11 +83,3 @@ PG 数据库没有连接上/初始化失败，可以查看日志。FastGPT 会
mongo连接失败，检查
1. mongo 服务有没有起来(有些 cpu 不支持 AVX，无法用 mongo5，需要换成 mongo4.x，可以dockerhub找个最新的4.x，修改镜像版本，重新运行）
2. 环境变量（账号密码，注意host和port）
### 错误排查方式
遇到问题先按下面方式排查。
1. `docker ps -a` 查看所有容器运行状态，检查是否全部 running，如有异常，尝试`docker logs 容器名`查看对应日志。
2. 不懂 docker 不要瞎改端口，只需要改`OPENAI_BASE_URL`和`CHAT_API_KEY`即可。
docSite/content/docs/development/upgrading/466.md
......@@ -13,10 +13,9 @@ weight: 830
为了减少代码重复度，我们对配置文件做了一些修改：[点击查看最新的配置文件](/docs/development/configuration/)
## V4.6.6 更新说明
## V4.6.6 即将更新
1. UI 优化，未来将逐步替换新的UI设计。
1. 新增 - 搜索方式：分离向量语义检索，全文检索和重排，通过 RRF 进行排序合并。
2. 优化 - 问题分类提示词，id引导。测试国产商用 api 模型（百度阿里智谱讯飞）使用 Prompt 模式均可分类。
3. UI 优化，未来将逐步替换新的UI设计。
4. 查看 [FastGPT 2024 RoadMap](https://github.com/labring/FastGPT?tab=readme-ov-file#-%E5%9C%A8%E7%BA%BF%E4%BD%BF%E7%94%A8)
\ No newline at end of file
packages/global/core/dataset/constant.ts
......@@ -148,5 +148,19 @@ export enum SearchScoreTypeEnum {
reRank = 'reRank',
rrf = 'rrf'
}
export const SearchScoreTypeMap = {
[SearchScoreTypeEnum.embedding]: {
label: 'core.dataset.search.score.embedding'
},
[SearchScoreTypeEnum.fullText]: {
label: 'core.dataset.search.score.fullText'
},
[SearchScoreTypeEnum.reRank]: {
label: 'core.dataset.search.score.reRank'
},
[SearchScoreTypeEnum.rrf]: {
label: 'core.dataset.search.score.rrf'
}
};
export const FolderAvatarSrc = '/imgs/files/folder.svg';
packages/global/core/dataset/type.d.ts
......@@ -162,7 +162,10 @@ export type DatasetFileSchema = {
};
/* ============= search =============== */
export type SearchDataResponseItemType = Omit<DatasetDataItemType, 'isOwner' | 'canWrite'> & {
export type SearchDataResponseItemType = Omit<
DatasetDataItemType,
'indexes' | 'isOwner' | 'canWrite'
> & {
score: { type: `${SearchScoreTypeEnum}`; value: number; index: number }[];
// score: number;
};
packages/global/core/module/template/input.ts
......@@ -27,7 +27,7 @@ export const Input_Template_History: FlowNodeInputItemType = {
export const Input_Template_UserChatInput: FlowNodeInputItemType = {
key: ModuleInputKeyEnum.userChatInput,
type: FlowNodeInputTypeEnum.target,
type: FlowNodeInputTypeEnum.hidden,
label: 'core.module.input.label.user question',
required: true,
valueType: ModuleIOValueTypeEnum.string,
......
packages/global/core/module/template/output.ts
......@@ -3,6 +3,14 @@ import { ModuleOutputKeyEnum } from '../constants';
import { FlowNodeOutputTypeEnum } from '../node/constant';
import { ModuleIOValueTypeEnum } from '../constants';
export const Output_Template_UserChatInput: FlowNodeOutputItemType = {
key: ModuleOutputKeyEnum.userChatInput,
label: 'core.module.input.label.user question',
type: FlowNodeOutputTypeEnum.hidden,
valueType: ModuleIOValueTypeEnum.string,
targets: []
};
export const Output_Template_Finish: FlowNodeOutputItemType = {
key: ModuleOutputKeyEnum.finish,
label: 'core.module.output.label.running done',
......
packages/global/core/module/template/system/aiChat.ts
......@@ -16,7 +16,7 @@ import {
Input_Template_UserChatInput
} from '../input';
import { chatNodeSystemPromptTip } from '../tip';
import { Output_Template_Finish } from '../output';
import { Output_Template_Finish, Output_Template_UserChatInput } from '../output';
export const AiChatModule: FlowModuleTemplateType = {
id: FlowNodeTypeEnum.chatNode,
......@@ -131,6 +131,7 @@ export const AiChatModule: FlowModuleTemplateType = {
Input_Template_UserChatInput
],
outputs: [
Output_Template_UserChatInput,
{
key: ModuleOutputKeyEnum.history,
label: '新的上下文',
......
packages/global/core/module/template/system/classifyQuestion.ts
......@@ -10,6 +10,7 @@ import {
Input_Template_Switch,
Input_Template_UserChatInput
} from '../input';
import { Output_Template_UserChatInput } from '../output';
export const ClassifyQuestionModule: FlowModuleTemplateType = {
id: FlowNodeTypeEnum.classifyQuestion,
......@@ -72,6 +73,7 @@ export const ClassifyQuestionModule: FlowModuleTemplateType = {
}
],
outputs: [
Output_Template_UserChatInput,
// custom output
{
key: 'wqre',
......
packages/global/core/module/template/system/coreferenceResolution.ts
......@@ -15,6 +15,7 @@ import {
Input_Template_Switch,
Input_Template_UserChatInput
} from '../input';
import { Output_Template_UserChatInput } from '../output';
export const AiCFR: FlowModuleTemplateType = {
id: FlowNodeTypeEnum.chatNode,
......@@ -50,6 +51,7 @@ export const AiCFR: FlowModuleTemplateType = {
Input_Template_UserChatInput
],
outputs: [
Output_Template_UserChatInput,
{
key: ModuleOutputKeyEnum.text,
label: 'core.module.output.label.cfr result',
......
packages/global/core/module/template/system/datasetSearch.ts
......@@ -11,7 +11,7 @@ import {
ModuleTemplateTypeEnum
} from '../../constants';
import { Input_Template_Switch, Input_Template_UserChatInput } from '../input';
import { Output_Template_Finish } from '../output';
import { Output_Template_Finish, Output_Template_UserChatInput } from '../output';
import { DatasetSearchModeEnum } from '../../../dataset/constant';
export const DatasetSearchModule: FlowModuleTemplateType = {
......@@ -90,6 +90,7 @@ export const DatasetSearchModule: FlowModuleTemplateType = {
Input_Template_UserChatInput
],
outputs: [
Output_Template_UserChatInput,
{
key: ModuleOutputKeyEnum.datasetIsEmpty,
label: '搜索结果为空',
......
packages/global/core/module/template/system/runApp.ts
......@@ -15,7 +15,7 @@ import {
Input_Template_Switch,
Input_Template_UserChatInput
} from '../input';
import { Output_Template_Finish } from '../output';
import { Output_Template_Finish, Output_Template_UserChatInput } from '../output';
export const RunAppModule: FlowModuleTemplateType = {
id: FlowNodeTypeEnum.runApp,
......@@ -41,6 +41,7 @@ export const RunAppModule: FlowModuleTemplateType = {
Input_Template_UserChatInput
],
outputs: [
Output_Template_UserChatInput,
{
key: ModuleOutputKeyEnum.history,
label: '新的上下文',
......
packages/service/common/vectorStore/pg/controller.ts
......@@ -24,10 +24,9 @@ export async function initPg() {
);
`);
await PgClient.query(`
CREATE INDEX CONCURRENTLY IF NOT EXISTS vector_index ON ${PgDatasetTableName}
USING hnsw (vector vector_ip_ops) WITH (m = 32, ef_construction = 64);
`);
await PgClient.query(
`CREATE INDEX CONCURRENTLY IF NOT EXISTS vector_index ON ${PgDatasetTableName} USING hnsw (vector vector_ip_ops) WITH (m = 32, ef_construction = 64);`
);
console.log('init pg successful');
} catch (error) {
......
packages/service/core/chat/chatItemSchema.ts
......@@ -80,8 +80,8 @@ const ChatItemSchema = new Schema({
});
try {
ChatItemSchema.index({ dataId: -1 });
ChatItemSchema.index({ time: -1 });
ChatItemSchema.index({ userId: 1 });
ChatItemSchema.index({ appId: 1 });
ChatItemSchema.index({ chatId: 1 });
ChatItemSchema.index({ userGoodFeedback: 1 });
......
packages/service/core/dataset/collection/controller.ts
......@@ -86,6 +86,8 @@ export const getSameRawTextCollection = async ({
datasetId: string;
hashRawText?: string;
}) => {
if (!hashRawText) return undefined;
const collection = await MongoDatasetCollection.findOne({
datasetId,
hashRawText
......
projects/app/public/docs/versionIntro.md
### Fast GPT V4.6.5
### Fast GPT V4.6.6
1. 新增 - [问题补全模块](https://doc.fastgpt.in/docs/workflow/modules/coreferenceresolution/)
2. 新增 - [文本编辑模块](https://doc.fastgpt.in/docs/workflow/modules/text_editor/)
......
projects/app/public/locales/en/common.json
......@@ -296,6 +296,7 @@
"Select Image": "Select Image",
"Send Message": "Send Message",
"Speaking": "I'm listening...",
"Start Chat": "Start Chat",
"Stop Speak": "Stop Speak",
"Type a message": "Input problem",
"error": {
......@@ -379,6 +380,7 @@
"Name": "Name",
"Quote Length": "Quote Length",
"Read Dataset": "Read Dataset",
"Search score tip": "{{scoreText}}Here are the rankings and scores:\n----\n{{detailScore}}",
"Set Empty Result Tip": ",Response empty text",
"Set Website Config": "Configuring Website",
"Similarity": "Similarity",
......@@ -435,6 +437,7 @@
"Too Long": "Content is too long",
"Total Amount": "{{total}} Chunks",
"data is deleted": "Data is deleted",
"get data error": "Get data error",
"id": "Data ID"
},
"error": {
......@@ -482,6 +485,7 @@
"link": "Link",
"search": {
"Dataset Search Params": "Dataset Search Params",
"Embedding score": "Embedding score",
"Empty result response": "Empty Response",
"Empty result response Tips": "If you fill in the content, if no suitable content is found, you will directly reply to the content.",
"Max Tokens": "Max Tokens",
......@@ -489,8 +493,14 @@
"Min Similarity": "Min Similarity",
"Min Similarity Tips": "The similarity of different index models is different, please use the search test to select the appropriate value",
"Params Setting": "Params Setting",
"Rank": "Rank",
"Rank Tip": "Ranking in all data",
"ReRank": "ReRank",
"ReRank desc": "The rearrangement model is used for secondary ranking to enhance the overall ranking.",
"Read score": "Read score",
"Rerank score": "ReRank score",
"Score": "Score",
"Search type": "Type",
"Top K": "Top K",
"mode": {
"embedding": "Vector search",
......@@ -500,6 +510,12 @@
"mixedRecall": "Mixedrecall",
"mixedRecall desc": "Returns the combined results of vector and full-text searches, sorted using the RRF algorithm."
},
"score": {
"embedding": "Embedding",
"fullText": "Full text",
"reRank": "ReRank",
"rrf": "RRF Merge"
},
"search mode": "Search Mode"
},
"status": {
......@@ -507,11 +523,15 @@
"syncing": "Syncing"
},
"test": {
"Batch test": "Batch test",
"Batch test Placeholder": "Select one csv file",
"Search Test": "Search Test",
"Test": "Start",
"Test File": "",
"Test Result": "Results",
"Test Text": "Text",
"Test Text Placeholder": "Enter the text you want to test",
"Test params": "Search Params",
"delete test history": "Delete the test result",
"test history": "Test History",
"test result placeholder": "The test results will be presented here",
......
projects/app/public/locales/zh/common.json
......@@ -296,6 +296,7 @@
"Select Image": "选择图片",
"Send Message": "发送",
"Speaking": "我在听，请说...",
"Start Chat": "开始对话",
"Stop Speak": "停止录音",
"Type a message": "输入问题",
"error": {
......@@ -379,6 +380,7 @@
"Name": "知识库名称",
"Quote Length": "引用内容长度",
"Read Dataset": "查看知识库详情",
"Search score tip": "{{scoreText}}下面是详细排名和得分情况:\n----\n{{detailScore}}",
"Set Empty Result Tip": ",未搜索到内容时回复指定内容",
"Set Website Config": "开始配置网站信息",
"Similarity": "相关度",
......@@ -435,6 +437,7 @@
"Too Long": "总长度超长了",
"Total Amount": "{{total}} 组",
"data is deleted": "该数据已被删除",
"get data error": "获取数据异常",
"id": "数据ID"
},
"error": {
......@@ -482,6 +485,7 @@
"link": "链接",
"search": {
"Dataset Search Params": "搜索参数",
"Embedding score": "语意检索得分",
"Empty result response": "空搜索回复",
"Empty result response Tips": "若填写该内容，没有搜索到合适内容时，将直接回复填写的内容。",
"Max Tokens": "引用上限",
......@@ -489,8 +493,14 @@
"Min Similarity": "最低相关度",
"Min Similarity Tips": "不同索引模型的相关度有区别，请通过搜索测试来选择合适的数值，使用 ReRank 时，相关度可能会很低。",
"Params Setting": "搜索参数设置",
"Rank": "排名",
"Rank Tip": "在所有数据中的排名",
"ReRank": "结果重排",
"ReRank desc": "使用重排模型来进行二次排序，可增强综合排名。",
"Read score": "查看得分",
"Rerank score": "结果重排得分",
"Score": "得分",
"Search type": "类型",
"Top K": "单次搜索上限",
"mode": {
"embedding": "语义检索",
......@@ -500,6 +510,12 @@
"mixedRecall": "混合检索",
"mixedRecall desc": "使用向量检索与全文检索的综合结果返回，使用RRF算法进行排序。"
},
"score": {
"embedding": "语义检索",
"fullText": "全文检索",
"reRank": "结果重排",
"rrf": "RRF 合并"
},
"search mode": "搜索模式"
},
"status": {
......@@ -507,11 +523,15 @@
"syncing": "同步中"
},
"test": {
"Batch test": "批量测试",
"Batch test Placeholder": "选择一个 Csv 文件",
"Search Test": "搜索测试",
"Test": "测试",
"Test File": "批量测试",
"Test Result": "测试结果",
"Test Text": "测试文本",
"Test Text": "单个文本测试",
"Test Text Placeholder": "输入需要测试的文本",
"Test params": "测试参数",
"delete test history": "删除该测试结果",
"test history": "测试历史",
"test result placeholder": "测试结果将在这里展示",
......@@ -847,7 +867,7 @@
"Response Detail": "返回详情",
"Response Detail tips": "是否需要返回详情（引用内容，调用时间等，不会返回预设提示词和完整上下文）",
"token auth": "身份验证",
"token auth Tips": "身份校验服务器地址，如填写该值，每次对话前都会想指定服务器发送一个请求，进行身份校验",
"token auth Tips": "身份校验服务器地址，如填写该值，每次对话前都会向指定服务器发送一个请求，进行身份校验",
"token auth use cases": "查看身份验证使用说明"
},
"permission": {
......
projects/app/src/components/ChatBox/ContextModal.tsx
......@@ -32,7 +32,7 @@ const ContextModal = ({
<Box
key={i}
p={2}
borderRadius={'lg'}
borderRadius={'md'}
border={theme.borders.base}
_notLast={{ mb: 2 }}
position={'relative'}
......
projects/app/src/components/ChatBox/FeedbackModal.tsx
......@@ -9,12 +9,16 @@ const FeedbackModal = ({
appId,
chatId,
chatItemId,
shareId,
outLinkUid,
onSuccess,
onClose
}: {
appId: string;
chatId: string;
chatItemId: string;
shareId?: string;
outLinkUid?: string;
onSuccess: (e: string) => void;
onClose: () => void;
}) => {
......@@ -28,6 +32,8 @@ const FeedbackModal = ({
appId,
chatId,
chatItemId,
shareId,
outLinkUid,
userBadFeedback: val
});
},
......
projects/app/src/components/ChatBox/QuoteModal.tsx
import React, { useCallback, useState } from 'react';
import { ModalBody, Box, useTheme, Flex, Progress, Link } from '@chakra-ui/react';
import { getDatasetDataItemById } from '@/web/core/dataset/api';
import { useLoading } from '@/web/common/hooks/useLoading';
import { useToast } from '@/web/common/hooks/useToast';
import { getErrText } from '@fastgpt/global/common/error/utils';
import MyIcon from '@/components/Icon';
import InputDataModal, {
RawSourceText,
type InputDataType
} from '@/pages/dataset/detail/components/InputDataModal';
import React, { useMemo, useState } from 'react';
import { ModalBody, Box, useTheme } from '@chakra-ui/react';
import MyModal from '../MyModal';
import { useTranslation } from 'next-i18next';
import type { SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type';
import MyTooltip from '../MyTooltip';
import NextLink from 'next/link';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import QuoteItem from '../core/dataset/QuoteItem';
import { RawSourceText } from '@/pages/dataset/detail/components/InputDataModal';
const QuoteModal = ({
rawSearch = [],
onClose,
isShare
isShare,
metadata
}: {
rawSearch: SearchDataResponseItemType[];
onClose: () => void;
isShare: boolean;
metadata?: {
collectionId: string;
sourceId?: string;
sourceName: string;
};
}) => {
const { t } = useTranslation();
const filterResults = useMemo(
() =>
metadata
? rawSearch.filter(
(item) =>
item.collectionId === metadata.collectionId && item.sourceId === metadata.sourceId
)
: rawSearch,
[metadata, rawSearch]
);
return (
<>
......@@ -35,18 +42,22 @@ const QuoteModal = ({
h={['90vh', '80vh']}
isCentered
minW={['90vw', '600px']}
iconSrc="/imgs/modal/quote.svg"
iconSrc={!!metadata ? undefined : '/imgs/modal/quote.svg'}
title={
<Box>
{t('core.chat.Quote Amount', { amount: rawSearch.length })}
<Box fontSize={'sm'} color={'myGray.500'} fontWeight={'normal'}>
{metadata ? (
<RawSourceText {...metadata} canView={false} />
) : (
<>{t('core.chat.Quote Amount', { amount: rawSearch.length })}</>
)}
<Box fontSize={'xs'} color={'myGray.500'} fontWeight={'normal'}>
{t('core.chat.quote.Quote Tip')}
</Box>
</Box>
}
>
<ModalBody whiteSpace={'pre-wrap'} textAlign={'justify'} wordBreak={'break-all'}>
<QuoteList rawSearch={rawSearch} isShare={isShare} />
<ModalBody>
<QuoteList rawSearch={filterResults} isShare={isShare} />
</ModalBody>
</MyModal>
</>
......@@ -62,38 +73,7 @@ export const QuoteList = React.memo(function QuoteList({
rawSearch: SearchDataResponseItemType[];
isShare: boolean;
}) {
const { t } = useTranslation();
const { isPc } = useSystemStore();
const theme = useTheme();
const { toast } = useToast();
const { setIsLoading, Loading } = useLoading();
const [editInputData, setEditInputData] = useState<InputDataType & { collectionId: string }>();
/**
* click edit, get new DataItem
*/
const onclickEdit = useCallback(
async (item: InputDataType) => {
if (!item.id) return;
try {
setIsLoading(true);
const data = await getDatasetDataItemById(item.id);
if (!data) {
throw new Error('该数据已被删除');
}
setEditInputData(data);
} catch (err) {
toast({
status: 'warning',
title: getErrText(err)
});
}
setIsLoading(false);
},
[setIsLoading, toast]
);
return (
<>
......@@ -102,113 +82,15 @@ export const QuoteList = React.memo(function QuoteList({
key={i}
flex={'1 0 0'}
p={2}
borderRadius={'lg'}
borderRadius={'sm'}
border={theme.borders.base}
_notLast={{ mb: 2 }}
position={'relative'}
overflow={'hidden'}
_hover={{ '& .hover-data': { display: 'flex' } }}
bg={i % 2 === 0 ? 'white' : 'myWhite.500'}
>
<Flex alignItems={'flex-end'} mb={3} fontSize={'sm'}>
<RawSourceText
fontWeight={'bold'}
color={'black'}
sourceName={item.sourceName}
sourceId={item.sourceId}
canView={!isShare}
/>
<Box flex={1} />
{!isShare && (
<Link
as={NextLink}
className="hover-data"
display={'none'}
alignItems={'center'}
color={'primary.500'}
href={`/dataset/detail?datasetId=${item.datasetId}&currentTab=dataCard&collectionId=${item.collectionId}`}
>
{t('core.dataset.Go Dataset')}
<MyIcon name={'common/rightArrowLight'} w={'10px'} />
</Link>
)}
</Flex>
<Box color={'black'}>{item.q}</Box>
<Box color={'myGray.600'}>{item.a}</Box>
{!isShare && (
<Flex alignItems={'center'} fontSize={'sm'} mt={3} gap={4} color={'myGray.500'}>
{isPc && (
<MyTooltip label={t('core.dataset.data.id')}>
<Flex border={theme.borders.base} py={'1px'} px={3} borderRadius={'3px'}>
# {item.id}
</Flex>
</MyTooltip>
)}
<MyTooltip label={t('core.dataset.Quote Length')}>
<Flex alignItems={'center'}>
<MyIcon name="common/text/t" w={'14px'} mr={1} color={'myGray.500'} />
{item.q.length + (item.a?.length || 0)}
</Flex>
</MyTooltip>
{/* {!isShare && item.score && (
<MyTooltip label={t('core.dataset.Similarity')}>
<Flex alignItems={'center'}>
<MyIcon name={'kbTest'} w={'12px'} />
<Progress
mx={2}
w={['60px', '90px']}
value={item.score * 100}
size="sm"
borderRadius={'20px'}
colorScheme="myGray"
border={theme.borders.base}
/>
<Box>{item.score.toFixed(4)}</Box>
</Flex>
</MyTooltip>
)} */}
<Box flex={1} />
{item.id && (
<MyTooltip label={t('core.dataset.data.Edit')}>
<Box
bg={'rgba(255,255,255,0.9)'}
alignItems={'center'}
justifyContent={'center'}
boxShadow={'-10px 0 10px rgba(255,255,255,1)'}
>
<MyIcon
name={'edit'}
w={['16px', '18px']}
h={['16px', '18px']}
cursor={'pointer'}
color={'myGray.600'}
_hover={{
color: 'primary.600'
}}
onClick={() => onclickEdit(item)}
/>
</Box>
</MyTooltip>
)}
</Flex>
)}
<QuoteItem quoteItem={item} canViewSource={!isShare} linkToDataset={!isShare} />
</Box>
))}
{editInputData && editInputData.id && (
<InputDataModal
onClose={() => setEditInputData(undefined)}
onSuccess={() => {
console.log('更新引用成功');
}}
onDelete={() => {
console.log('删除引用成功');
}}
defaultValue={editInputData}
collectionId={editInputData.collectionId}
/>
)}
<Loading fixed={false} />
</>
);
});
projects/app/src/components/ChatBox/ResponseTags.tsx
......@@ -11,8 +11,6 @@ import MyTooltip from '../MyTooltip';
import { FlowNodeTypeEnum } from '@fastgpt/global/core/module/node/constant';
import { getSourceNameIcon } from '@fastgpt/global/core/dataset/utils';
import ChatBoxDivider from '@/components/core/chat/Divider';
import MyIcon from '../Icon';
import { getFileAndOpen } from '@/web/core/dataset/utils';
import { strIsLink } from '@fastgpt/global/common/string/tools';
const QuoteModal = dynamic(() => import('./QuoteModal'), { ssr: false });
......@@ -29,7 +27,14 @@ const ResponseTags = ({
const theme = useTheme();
const { isPc } = useSystemStore();
const { t } = useTranslation();
const [quoteModalData, setQuoteModalData] = useState<SearchDataResponseItemType[]>();
const [quoteModalData, setQuoteModalData] = useState<{
rawSearch: SearchDataResponseItemType[];
metadata?: {
collectionId: string;
sourceId?: string;
sourceName: string;
};
}>();
const [contextModalData, setContextModalData] = useState<ChatItemType[]>();
const {
isOpen: isOpenWholeModal,
......@@ -52,8 +57,8 @@ const ResponseTags = ({
.filter(Boolean) as SearchDataResponseItemType[];
const sourceList = quoteList.reduce(
(acc: Record<string, SearchDataResponseItemType[]>, cur) => {
if (!acc[cur.sourceName]) {
acc[cur.sourceName] = [cur];
if (!acc[cur.collectionId]) {
acc[cur.collectionId] = [cur];
}
return acc;
},
......@@ -70,7 +75,8 @@ const ResponseTags = ({
sourceName: item.sourceName,
sourceId: item.sourceId,
icon: getSourceNameIcon({ sourceId: item.sourceId, sourceName: item.sourceName }),
canReadQuote: !isShare || strIsLink(item.sourceId)
canReadQuote: !isShare || strIsLink(item.sourceId),
collectionId: item.collectionId
})),
historyPreview: chatData?.historyPreview,
runningTime: +responseData.reduce((sum, item) => sum + (item.runningTime || 0), 0).toFixed(2)
......@@ -89,88 +95,52 @@ const ResponseTags = ({
<ChatBoxDivider icon="core/chat/quoteFill" text={t('chat.Quote')} />
<Flex alignItems={'center'} flexWrap={'wrap'} gap={2}>
{sourceList.map((item) => (
<Flex
key={item.sourceName}
alignItems={'center'}
flexWrap={'wrap'}
fontSize={'sm'}
border={theme.borders.sm}
py={1}
px={2}
borderRadius={'md'}
_hover={{
'.controller': {
display: 'flex'
}
}}
overflow={'hidden'}
position={'relative'}
>
<Image src={item.icon} alt={''} mr={1} w={'12px'} />
<Box className="textEllipsis" flex={'1 0 0'}>
{item.sourceName}
</Box>
<Box
className="controller"
display={'none'}
pr={2}
position={'absolute'}
right={0}
left={0}
justifyContent={'flex-end'}
<MyTooltip key={item.sourceName} label={t('core.chat.quote.Read Quote')}>
<Flex
alignItems={'center'}
h={'100%'}
lineHeight={0}
bg={`linear-gradient(to left, white,white ${
item.sourceId ? '60px' : '30px'
}, rgba(255,255,255,0) 80%)`}
fontSize={'sm'}
border={theme.borders.sm}
py={1}
px={2}
borderRadius={'sm'}
_hover={{
'.controller': {
display: 'flex'
}
}}
overflow={'hidden'}
position={'relative'}
cursor={'pointer'}
onClick={(e) => {
e.stopPropagation();
setQuoteModalData({
rawSearch: quoteList,
metadata: {
collectionId: item.collectionId,
sourceId: item.sourceId,
sourceName: item.sourceName
}
});
}}
>
<MyTooltip label={t('core.chat.quote.Read Quote')}>
<MyIcon
name="common/viewLight"
w={'14px'}
cursor={'pointer'}
_hover={{
color: 'green.600'
}}
onClick={(e) => {
e.stopPropagation();
setQuoteModalData(quoteList);
}}
/>
</MyTooltip>
{item.sourceId && item.canReadQuote && (
<MyTooltip label={t('core.chat.quote.Read Source')}>
<MyIcon
ml={4}
name="common/routePushLight"
w={'14px'}
cursor={'pointer'}
_hover={{ color: 'primary.500' }}
onClick={async (e) => {
e.stopPropagation();
if (!item.sourceId) return;
await getFileAndOpen(item.sourceId);
}}
/>
</MyTooltip>
)}
</Box>
</Flex>
<Image src={item.icon} alt={''} mr={1} flexShrink={0} w={'12px'} />
<Box className="textEllipsis3" wordBreak={'break-all'} flex={'1 0 0'}>
{item.sourceName}
</Box>
</Flex>
</MyTooltip>
))}
</Flex>
</>
)}
<Flex alignItems={'center'} mt={2} flexWrap={'wrap'}>
<Flex alignItems={'center'} mt={3} flexWrap={'wrap'}>
{quoteList.length > 0 && (
<MyTooltip label="查看引用">
<Tag
colorSchema="blue"
cursor={'pointer'}
{...TagStyles}
onClick={() => setQuoteModalData(quoteList)}
onClick={() => setQuoteModalData({ rawSearch: quoteList })}
>
{quoteList.length}条引用
</Tag>
......@@ -213,7 +183,7 @@ const ResponseTags = ({
{!!quoteModalData && (
<QuoteModal
rawSearch={quoteModalData}
{...quoteModalData}
isShare={isShare}
onClose={() => setQuoteModalData(undefined)}
/>
......
projects/app/src/components/ChatBox/index.tsx
......@@ -632,14 +632,13 @@ const ChatBox = (
leftIcon={<MyIcon name={'chatFill'} w={'16px'} />}
size={'sm'}
maxW={'100px'}
borderRadius={'lg'}
onClick={handleSubmit((data) => {
onUpdateVariable?.(data);
setVariables(data);
setVariableInputFinish(true);
})}
>
{'开始对话'}
{t('core.chat.Start Chat')}
</Button>
)}
</Card>
......@@ -952,6 +951,8 @@ const ChatBox = (
appId={appId}
chatId={chatId}
chatItemId={feedbackId}
shareId={shareId}
outLinkUid={outLinkUid}
onClose={() => setFeedbackId(undefined)}
onSuccess={(content: string) => {
setChatHistory((state) =>
......@@ -1142,7 +1143,7 @@ function ChatAvatar({ src, type }: { src?: string; type: 'Human' | 'AI' }) {
w={['28px', '34px']}
h={['28px', '34px']}
p={'2px'}
borderRadius={'lg'}
borderRadius={'sm'}
border={theme.borders.base}
boxShadow={'0 0 5px rgba(0,0,0,0.1)'}
bg={type === 'Human' ? 'white' : 'primary.50'}
......@@ -1208,7 +1209,7 @@ function ChatController({
cursor: 'pointer',
p: 1,
bg: 'white',
borderRadius: 'lg',
borderRadius: 'md',
boxShadow: '0 0 5px rgba(0,0,0,0.1)',
border: theme.borders.base,
mr: 3
......
projects/app/src/components/Markdown/img/Image.tsx
......@@ -9,7 +9,6 @@ import {
Skeleton,
useDisclosure
} from '@chakra-ui/react';
import MyModal from '@/components/MyModal';
const MdImage = ({ src }: { src?: string }) => {
const [isLoading, setIsLoading] = useState(true);
......@@ -34,6 +33,7 @@ const MdImage = ({ src }: { src?: string }) => {
cursor={succeed ? 'pointer' : 'default'}
loading="eager"
objectFit={'contain'}
referrerPolicy="no-referrer"
onLoad={() => {
setIsLoading(false);
setSucceed(true);
......@@ -53,6 +53,7 @@ const MdImage = ({ src }: { src?: string }) => {
alt={''}
w={'100%'}
maxH={'80vh'}
referrerPolicy="no-referrer"
fallbackSrc={'/imgs/errImg.png'}
fallbackStrategy={'onError'}
objectFit={'contain'}
......
projects/app/src/components/Tag/index.tsx
......@@ -39,7 +39,7 @@ const Tag = ({ children, colorSchema = 'blue', ...props }: Props) => {
px={2}
lineHeight={1}
py={1}
borderRadius={'md'}
borderRadius={'sm'}
fontSize={'xs'}
alignItems={'center'}
{...theme}
......
projects/app/src/components/common/Textarea/PromptTextarea/index.tsx
......@@ -77,7 +77,7 @@ const Editor = React.memo(function Editor({
return (
<Box h={'100%'} w={'100%'} position={'relative'}>
<Textarea ref={textareaRef} textAlign={'justify'} maxW={'100%'} {...props} />
<Textarea ref={textareaRef} maxW={'100%'} {...props} />
{onOpenModal && (
<Box
zIndex={1}
......
projects/app/src/components/core/dataset/QuoteItem.tsx 0 → 100644
import React, { useMemo, useState } from 'react';
import { Box, Flex, Link, Progress, useTheme } from '@chakra-ui/react';
import {
type InputDataType,
RawSourceText
} from '@/pages/dataset/detail/components/InputDataModal';
import type { SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type.d';
import NextLink from 'next/link';
import MyIcon from '@/components/Icon';
import { useTranslation } from 'next-i18next';
import MyTooltip from '@/components/MyTooltip';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import dynamic from 'next/dynamic';
import MyBox from '@/components/common/MyBox';
import { getDatasetDataItemById } from '@/web/core/dataset/api';
import { useRequest } from '@/web/common/hooks/useRequest';
import { DatasetDataItemType } from '@fastgpt/global/core/dataset/type';
import { SearchScoreTypeEnum, SearchScoreTypeMap } from '@fastgpt/global/core/dataset/constant';
const InputDataModal = dynamic(() => import('@/pages/dataset/detail/components/InputDataModal'));
const QuoteItem = ({
quoteItem,
canViewSource,
linkToDataset
}: {
quoteItem: SearchDataResponseItemType;
canViewSource?: boolean;
linkToDataset?: boolean;
}) => {
const { t } = useTranslation();
const { isPc } = useSystemStore();
const theme = useTheme();
const [editInputData, setEditInputData] = useState<InputDataType & { collectionId: string }>();
const { mutate: onclickEdit, isLoading } = useRequest({
mutationFn: async (id: string) => {
return getDatasetDataItemById(id);
},
onSuccess(data: DatasetDataItemType) {
setEditInputData(data);
},
errorToast: t('core.dataset.data.get data error')
});
const rank = useMemo(() => {
if (quoteItem.score.length === 1) {
return quoteItem.score[0].index;
}
const rrf = quoteItem.score?.find((item) => item.type === SearchScoreTypeEnum.rrf);
if (rrf) return rrf.index;
return 0;
}, [quoteItem.score]);
const score = useMemo(() => {
let searchScore: number | undefined = undefined;
let text = '';
const reRankScore = quoteItem.score?.find((item) => item.type === SearchScoreTypeEnum.reRank);
if (reRankScore) {
searchScore = reRankScore.value;
text = t('core.dataset.search.Rerank score');
}
const embScore = quoteItem.score?.find((item) => item.type === SearchScoreTypeEnum.embedding);
if (embScore && quoteItem.score.length === 1) {
searchScore = embScore.value;
text = t('core.dataset.search.Embedding score');
}
const detailScore = (() => {
if (Array.isArray(quoteItem.score)) {
return quoteItem.score
.map(
(item) =>
`${t('core.dataset.search.Search type')}: ${t(SearchScoreTypeMap[item.type]?.label)}
${t('core.dataset.search.Rank')}: ${item.index + 1}
${t('core.dataset.search.Score')}: ${item.value.toFixed(4)}`
)
.join('\n----\n');
}
return 'null';
})();
return {
value: searchScore,
tip: t('core.dataset.Search score tip', {
scoreText: text ? `${text}。\n` : text,
detailScore
})
};
}, [quoteItem.score, t]);
return (
<>
<MyBox
isLoading={isLoading}
position={'relative'}
overflow={'hidden'}
fontSize={'sm'}
whiteSpace={'pre-wrap'}
_hover={{ '& .hover-data': { display: 'flex' } }}
>
<Flex alignItems={'flex-end'} mb={3}>
{rank !== undefined && (
<MyTooltip label={t('core.dataset.search.Rank Tip')}>
<Box px={2} py={'3px'} mr={3} bg={'myGray.200'} borderRadius={'md'}>
# {rank + 1}
</Box>
</MyTooltip>
)}
<RawSourceText
fontWeight={'bold'}
color={'black'}
sourceName={quoteItem.sourceName}
sourceId={quoteItem.sourceId}
canView={canViewSource}
/>
<Box flex={1} />
{linkToDataset && (
<Link
as={NextLink}
className="hover-data"
display={'none'}
alignItems={'center'}
color={'primary.500'}
href={`/dataset/detail?datasetId=${quoteItem.datasetId}&currentTab=dataCard&collectionId=${quoteItem.collectionId}`}
>
{t('core.dataset.Go Dataset')}
<MyIcon name={'common/rightArrowLight'} w={'10px'} />
</Link>
)}
</Flex>
<Box color={'black'}>{quoteItem.q}</Box>
<Box color={'myGray.600'}>{quoteItem.a}</Box>
{canViewSource && (
<Flex alignItems={'center'} mt={3} gap={4} color={'myGray.500'} fontSize={'xs'}>
{isPc && (
<Flex border={theme.borders.base} px={3} borderRadius={'xs'} lineHeight={'16px'}>
ID: {quoteItem.id}
</Flex>
)}
<MyTooltip label={t('core.dataset.Quote Length')}>
<Flex alignItems={'center'}>
<MyIcon name="common/text/t" w={'14px'} mr={1} color={'myGray.500'} />
{quoteItem.q.length + (quoteItem.a?.length || 0)}
</Flex>
</MyTooltip>
{canViewSource && score && (
<MyTooltip label={score.tip}>
<Flex alignItems={'center'}>
<MyIcon name={'kbTest'} w={'12px'} />
{score.value ? (
<>
<Progress
mx={2}
w={['60px', '90px']}
value={score?.value * 100}
size="sm"
borderRadius={'20px'}
colorScheme="myGray"
border={theme.borders.base}
/>
<Box>{score?.value.toFixed(4)}</Box>
</>
) : (
<Box ml={1} cursor={'pointer'}>
{t('core.dataset.search.Read score')}
</Box>
)}
</Flex>
</MyTooltip>
)}
<Box flex={1} />
{quoteItem.id && (
<MyTooltip label={t('core.dataset.data.Edit')}>
<Box
className="hover-data"
display={['flex', 'none']}
bg={'rgba(255,255,255,0.9)'}
alignItems={'center'}
justifyContent={'center'}
boxShadow={'-10px 0 10px rgba(255,255,255,1)'}
>
<MyIcon
name={'edit'}
w={['16px', '18px']}
h={['16px', '18px']}
cursor={'pointer'}
color={'myGray.600'}
_hover={{
color: 'primary.600'
}}
onClick={() => onclickEdit(quoteItem.id)}
/>
</Box>
</MyTooltip>
)}
</Flex>
)}
</MyBox>
{editInputData && editInputData.id && (
<InputDataModal
onClose={() => setEditInputData(undefined)}
onSuccess={() => {
console.log('更新引用成功');
}}
onDelete={() => {
console.log('删除引用成功');
}}
defaultValue={editInputData}
collectionId={editInputData.collectionId}
/>
)}
</>
);
};
export default React.memo(QuoteItem);
projects/app/src/components/core/module/Flow/components/modules/VariableEdit.tsx
......@@ -128,7 +128,7 @@ const VariableEdit = ({
</Flex>
</Flex>
{formatVariables.length > 0 && (
<Box mt={2} borderRadius={'lg'} overflow={'hidden'} borderWidth={'1px'} borderBottom="none">
<Box mt={2} borderRadius={'md'} overflow={'hidden'} borderWidth={'1px'} borderBottom="none">
<TableContainer>
<Table bg={'white'}>
<Thead>
......@@ -266,7 +266,7 @@ const VariableEdit = ({
w={'16px'}
cursor={'pointer'}
p={2}
borderRadius={'lg'}
borderRadius={'md'}
_hover={{ bg: 'red.100' }}
onClick={() => removeEnums(i)}
/>
......
projects/app/src/components/core/module/Flow/components/render/RenderInput/index.tsx
......@@ -7,6 +7,7 @@ import dynamic from 'next/dynamic';
import InputLabel from './Label';
import type { RenderInputProps } from './type.d';
import { getFlowStore, type useFlowProviderStoreType } from '../../../FlowProvider';
import { ModuleInputKeyEnum } from '@fastgpt/global/core/module/constants';
const RenderList: {
types: `${FlowNodeInputTypeEnum}`[];
......@@ -65,6 +66,7 @@ const RenderList: {
Component: dynamic(() => import('./templates/AddInputParam'))
}
];
const UserChatInput = dynamic(() => import('./templates/UserChatInput'));
type Props = {
flowInputList: FlowNodeInputItemType[];
......@@ -124,18 +126,23 @@ const RenderInput = ({ flowInputList, moduleId, CustomComponent = {} }: Props) =
})();
return (
input.type !== FlowNodeInputTypeEnum.hidden && (
<Box key={input.key} _notLast={{ mb: 7 }} position={'relative'}>
{!!input.label && (
<InputLabel moduleId={moduleId} inputKey={input.key} mode={mode} {...input} />
)}
{!!RenderComponent && (
<Box mt={2} className={'nodrag'}>
{RenderComponent}
</Box>
)}
</Box>
)
<Box key={input.key} _notLast={{ mb: 7 }} position={'relative'}>
{input.key === ModuleInputKeyEnum.userChatInput && (
<UserChatInput inputs={filterInputs} item={input} moduleId={moduleId} />
)}
{input.type !== FlowNodeInputTypeEnum.hidden && (
<>
{!!input.label && (
<InputLabel moduleId={moduleId} inputKey={input.key} mode={mode} {...input} />
)}
{!!RenderComponent && (
<Box mt={2} className={'nodrag'}>
{RenderComponent}
</Box>
)}
</>
)}
</Box>
);
})}
</>
......
projects/app/src/components/core/module/Flow/components/render/RenderInput/templates/UserChatInput.tsx 0 → 100644
import React from 'react';
import type { RenderInputProps } from '../type';
import { Box, Flex } from '@chakra-ui/react';
import { useTranslation } from 'next-i18next';
import TargetHandle from '../../TargetHandle';
import SourceHandle from '../../SourceHandle';
import { ModuleInputKeyEnum, ModuleOutputKeyEnum } from '@fastgpt/global/core/module/constants';
const UserChatInput = ({ item }: RenderInputProps) => {
const { t } = useTranslation();
return (
<Flex
className="nodrag"
cursor={'default'}
alignItems={'center'}
justifyContent={'space-between'}
position={'relative'}
>
<Box position={'relative'}>
<TargetHandle handleKey={ModuleInputKeyEnum.userChatInput} valueType={item.valueType} />
{t('core.module.input.label.user question')}
<Box
position={'absolute'}
top={'-2px'}
right={'-8px'}
color={'red.500'}
fontWeight={'bold'}
>
*
</Box>
</Box>
<Box position={'relative'}>
{t('core.module.input.label.user question')}
<SourceHandle handleKey={ModuleOutputKeyEnum.userChatInput} valueType={item.valueType} />
</Box>
</Flex>
);
};
export default React.memo(UserChatInput);
projects/app/src/global/core/dataset/api.d.ts
......@@ -59,8 +59,13 @@ export type SearchTestProps = {
limit?: number;
searchMode?: `${DatasetSearchModeEnum}`;
usingReRank: boolean;
similarity?: number;
};
export type SearchTestResponse = {
list: SearchDataResponseItemType[];
duration: string;
limit: number;
searchMode: `${DatasetSearchModeEnum}`;
usingReRank: boolean;
similarity: number;
};
projects/app/src/global/core/prompt/agent.ts
......@@ -36,11 +36,7 @@ export const Prompt_ExtractJson = `你可以从 <对话记录></对话记录>
</对话记录>
`;
export const Prompt_CQJson = `我会给你几个问题类型，请参考额外的背景知识（可能为空）和对话内容，判断我本次的问题类型，并返回对应类型的 ID，格式为 JSON 字符串:
"""
'{"问题类型":"类型的 ID"}'
"""
export const Prompt_CQJson = `我会给你几个问题类型，请参考背景知识（可能为空）和对话记录，判断我“本次问题”的类型，并返回一个问题“类型ID”:
<问题类型>
{{typeList}}
</问题类型>
......@@ -49,9 +45,13 @@ export const Prompt_CQJson = `我会给你几个问题类型，请参考额外
{{systemPrompt}}
</背景知识>
<对话内容>
{{text}}
</对话内容>
<对话记录>
{{history}}
</对话记录>
Human："{{question}}"
类型ID=
`;
export const Prompt_QuestionGuide = `我不太清楚问你什么问题，请帮我生成 3 个问题，引导我继续提问。问题的长度应小于20个字符，按 JSON 格式返回: ["问题1", "问题2", "问题3"]`;
projects/app/src/pages/api/core/dataset/searchTest.ts
......@@ -14,7 +14,14 @@ import { searchQueryExtension } from '@fastgpt/service/core/ai/functions/queryEx
export default withNextCors(async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
const { datasetId, text, limit = 20, searchMode, usingReRank } = req.body as SearchTestProps;
const {
datasetId,
text,
limit = 1500,
similarity,
searchMode,
usingReRank
} = req.body as SearchTestProps;
if (!datasetId || !text) {
throw new Error('缺少参数');
......@@ -40,11 +47,12 @@ export default withNextCors(async function handler(req: NextApiRequest, res: Nex
// model: global.chatModels[0].model
// });
const { searchRes, tokens } = await searchDatasetData({
const { searchRes, tokens, ...result } = await searchDatasetData({
rawQuery: text,
queries: [text],
model: dataset.vectorModel,
limit: Math.min(limit * 800, 30000),
limit: Math.min(limit, 20000),
similarity,
datasetIds: [datasetId],
searchMode,
usingReRank
......@@ -68,7 +76,8 @@ export default withNextCors(async function handler(req: NextApiRequest, res: Nex
jsonRes<SearchTestResponse>(res, {
data: {
list: searchRes,
duration: `${((Date.now() - start) / 1000).toFixed(3)}s`
duration: `${((Date.now() - start) / 1000).toFixed(3)}s`,
...result
}
});
} catch (err) {
......
projects/app/src/pages/app/detail/components/InfoModal.tsx
......@@ -133,7 +133,7 @@ const InfoModal = ({
w={['26px', '34px']}
h={['26px', '34px']}
cursor={'pointer'}
borderRadius={'lg'}
borderRadius={'md'}
mr={4}
title={'点击切换头像'}
onClick={() => onOpenSelectFile()}
......
projects/app/src/pages/app/detail/components/Logs.tsx
......@@ -128,7 +128,7 @@ const Logs = ({ appId }: { appId: string }) => {
py={1}
alignItems={'center'}
justifyContent={'center'}
borderRadius={'lg'}
borderRadius={'md'}
fontWeight={'bold'}
>
<MyIcon
......@@ -148,7 +148,7 @@ const Logs = ({ appId }: { appId: string }) => {
py={1}
alignItems={'center'}
justifyContent={'center'}
borderRadius={'lg'}
borderRadius={'md'}
fontWeight={'bold'}
>
<MyIcon
......
projects/app/src/pages/app/detail/components/OutLink/index.tsx
......@@ -20,7 +20,7 @@ const OutLink = ({ appId }: { appId: string }) => {
</Box>
<Box pb={[5, 7]} px={[4, 8]} borderBottom={theme.borders.base}>
<MyRadio
gridTemplateColumns={['repeat(1,1fr)', 'repeat(auto-fill, minmax(0, 360px))']}
gridTemplateColumns={['repeat(1,1fr)', 'repeat(auto-fill, minmax(0, 400px))']}
iconSize={'20px'}
list={[
{
......
projects/app/src/pages/app/detail/components/SimpleEdit/index.tsx
......@@ -597,7 +597,7 @@ function Settings({ appId }: { appId: string }) {
<Box
borderWidth={'1px'}
borderColor={'primary.1'}
borderRadius={'lg'}
borderRadius={'md'}
mt={2}
px={5}
py={4}
......
projects/app/src/pages/app/detail/index.tsx
......@@ -108,7 +108,7 @@ const AppDetail = ({ currentTab }: { currentTab: `${TabEnum}` }) => {
borderRight={theme.borders.base}
>
<Flex mb={4} alignItems={'center'}>
<Avatar src={appDetail.avatar} w={'34px'} borderRadius={'lg'} />
<Avatar src={appDetail.avatar} w={'34px'} borderRadius={'md'} />
<Box ml={2} fontWeight={'bold'}>
{appDetail.name}
</Box>
......
projects/app/src/pages/app/list/index.tsx
......@@ -99,7 +99,7 @@ const MyApps = () => {
borderWidth={'1.5px'}
borderColor={'borderColor.low'}
bg={'white'}
borderRadius={'lg'}
borderRadius={'md'}
userSelect={'none'}
position={'relative'}
display={'flex'}
......
projects/app/src/pages/chat/components/ChatHistorySlider.tsx
......@@ -170,7 +170,7 @@ const ChatHistorySlider = ({
variant={'whiteDanger'}
size={'mdSquare'}
aria-label={''}
borderRadius={'xl'}
borderRadius={'50%'}
onClick={openConfirm(onClearHistory)}
>
<MyIcon name={'clear'} w={'16px'} />
......@@ -191,7 +191,7 @@ const ChatHistorySlider = ({
px={4}
cursor={'pointer'}
userSelect={'none'}
borderRadius={'lg'}
borderRadius={'md'}
mb={2}
_hover={{
bg: 'myGray.100',
......@@ -287,7 +287,7 @@ const ChatHistorySlider = ({
py={2}
px={3}
mb={3}
borderRadius={'lg'}
borderRadius={'md'}
alignItems={'center'}
{...(item._id === appId
? {
......
projects/app/src/pages/chat/components/SliderApps.tsx
......@@ -46,7 +46,7 @@ const SliderApps = ({ appId }: { appId: string }) => {
px={3}
mb={3}
cursor={'pointer'}
borderRadius={'lg'}
borderRadius={'md'}
alignItems={'center'}
{...(item._id === appId
? {
......
projects/app/src/pages/dataset/detail/components/Import/FileSelect.tsx
......@@ -380,7 +380,7 @@ const FileSelect = ({
textAlign={'center'}
bg={'myWhite.400'}
p={5}
borderRadius={'lg'}
borderRadius={'md'}
border={'1px dashed'}
borderColor={'myGray.300'}
w={'100%'}
......
projects/app/src/pages/dataset/detail/index.tsx
......@@ -156,7 +156,7 @@ const Detail = ({ datasetId, currentTab }: { datasetId: string; currentTab: `${T
borderRight={theme.borders.base}
>
<Flex mb={4} alignItems={'center'}>
<Avatar src={datasetDetail.avatar} w={'34px'} borderRadius={'lg'} />
<Avatar src={datasetDetail.avatar} w={'34px'} borderRadius={'md'} />
<Box ml={2}>
<Box fontWeight={'bold'}>{datasetDetail.name}</Box>
</Box>
......
projects/app/src/pages/dataset/list/index.tsx
......@@ -211,7 +211,7 @@ const Kb = () => {
borderWidth={1.5}
borderColor={dragTargetId === dataset._id ? 'primary.600' : 'borderColor.low'}
bg={'white'}
borderRadius={'lg'}
borderRadius={'md'}
minH={'130px'}
position={'relative'}
data-drag-id={dataset.type === DatasetTypeEnum.folder ? dataset._id : undefined}
......@@ -390,7 +390,7 @@ const Kb = () => {
/>
)}
<Flex alignItems={'center'} h={'38px'}>
<Avatar src={dataset.avatar} borderRadius={'lg'} w={'28px'} />
<Avatar src={dataset.avatar} borderRadius={'md'} w={'28px'} />
<Box mx={3} className="textEllipsis3">
{dataset.name}
</Box>
......
projects/app/src/pages/plugin/list/index.tsx
......@@ -60,7 +60,7 @@ const MyModules = () => {
borderWidth={'1.5px'}
borderColor={'borderColor.low'}
bg={'white'}
borderRadius={'lg'}
borderRadius={'md'}
userSelect={'none'}
position={'relative'}
_hover={{
......
projects/app/src/service/core/ai/rerank.ts
......@@ -18,7 +18,8 @@ export function reRankRecall({ query, inputs }: PostReRankProps) {
{
headers: {
Authorization: `Bearer ${model.requestAuth}`
}
},
timeout: 120000
}
)
.then((data) => {
......
projects/app/src/service/core/dataset/data/controller.ts
......@@ -275,25 +275,20 @@ export async function searchDatasetData(props: {
const oneChunkToken = 50;
const estimatedLen = Math.max(20, Math.ceil(maxTokens / oneChunkToken));
// Increase search range, reduce hnsw loss. 20 ~ 100
if (searchMode === DatasetSearchModeEnum.embedding) {
return {
embeddingLimit: Math.min(estimatedLen, 100),
embeddingLimit: Math.min(estimatedLen, 80),
fullTextLimit: 0
};
}
// 50 < 2*limit < value < 100
if (searchMode === DatasetSearchModeEnum.fullTextRecall) {
return {
embeddingLimit: 0,
fullTextLimit: Math.min(estimatedLen, 50)
};
}
// mixed
// 50 < 2*limit < embedding < 80
// 20 < limit < fullTextLimit < 40
return {
embeddingLimit: Math.min(estimatedLen, 80),
embeddingLimit: Math.min(estimatedLen, 60),
fullTextLimit: Math.min(estimatedLen, 40)
};
};
......@@ -340,7 +335,6 @@ export async function searchDatasetData(props: {
q: data.q,
a: data.a,
chunkIndex: data.chunkIndex,
indexes: data.indexes,
datasetId: String(data.datasetId),
collectionId: String(data.collectionId),
sourceName: collection.name || '',
......@@ -389,7 +383,6 @@ export async function searchDatasetData(props: {
collectionId: 1,
q: 1,
a: 1,
indexes: 1,
chunkIndex: 1
}
)
......@@ -464,6 +457,7 @@ export async function searchDatasetData(props: {
return mergeResult;
} catch (error) {
usingReRank = false;
return [];
}
};
......@@ -553,6 +547,11 @@ export async function searchDatasetData(props: {
const rrfConcat = (
arr: { k: number; list: SearchDataResponseItemType[] }[]
): SearchDataResponseItemType[] => {
arr = arr.filter((item) => item.list.length > 0);
if (arr.length === 0) return [];
if (arr.length === 1) return arr[0].list;
const map = new Map<string, SearchDataResponseItemType & { rrfScore: number }>();
// rrf
......@@ -643,7 +642,7 @@ export async function searchDatasetData(props: {
// embedding recall and fullText recall rrf concat
const rrfConcatResults = rrfConcat([
{ k: 60, list: embeddingRecallResults },
{ k: 60, list: fullTextRecallResults },
{ k: 64, list: fullTextRecallResults },
{ k: 60, list: reRankResults }
]);
......@@ -685,6 +684,10 @@ export async function searchDatasetData(props: {
return {
searchRes: filterResultsByMaxTokens(scoreFilter, maxTokens),
tokens,
searchMode,
limit: maxTokens,
similarity,
usingReRank,
usingSimilarityFilter
};
}
projects/app/src/service/moduleDispatch/agent/classifyQuestion.ts
......@@ -176,10 +176,12 @@ async function completions({
{
obj: ChatRoleEnum.Human,
value: replaceVariable(cqModel.functionPrompt || Prompt_CQJson, {
systemPrompt,
typeList: agents.map((item) => `{"${item.value}": ${item.key}}`).join('\n'),
text: `${histories.map((item) => `${item.obj}:${item.value}`).join('\n')}
Human:${userChatInput}`
systemPrompt: systemPrompt || 'null',
typeList: agents
.map((item) => `{"questionType": "${item.value}", "typeId": "${item.key}"}`)
.join('\n'),
history: histories.map((item) => `${item.obj}:${item.value}`).join('\n'),
question: userChatInput
})
}
];
......@@ -194,7 +196,8 @@ Human:${userChatInput}`
});
const answer = data.choices?.[0].message?.content || '';
const id = agents.find((item) => answer.includes(item.key))?.key || '';
const id =
agents.find((item) => answer.includes(item.key) || answer.includes(item.value))?.key || '';
return {
inputTokens: data.usage?.prompt_tokens || 0,
......
projects/app/src/service/moduleDispatch/dataset/search.ts
......@@ -52,7 +52,12 @@ export async function dispatchDatasetSearch(
const concatQueries = [userChatInput];
// start search
const { searchRes, tokens, usingSimilarityFilter } = await searchDatasetData({
const {
searchRes,
tokens,
usingSimilarityFilter,
usingReRank: searchUsingReRank
} = await searchDatasetData({
rawQuery: userChatInput,
queries: concatQueries,
model: vectorModel.model,
......@@ -81,7 +86,7 @@ export async function dispatchDatasetSearch(
similarity: usingSimilarityFilter ? similarity : undefined,
limit,
searchMode,
searchUsingReRank: usingReRank
searchUsingReRank: searchUsingReRank
}
};
}
projects/app/src/service/moduleDispatch/index.ts
......@@ -181,7 +181,7 @@ export async function dispatchModules({
});
}
// get fetch params
// get module running params
const params: Record<string, any> = {};
module.inputs.forEach((item: any) => {
params[item.key] = item.value;
......@@ -198,6 +198,7 @@ export async function dispatchModules({
inputs: params
};
// run module
const dispatchRes: Record<string, any> = await (async () => {
if (callbackMap[module.flowType]) {
return callbackMap[module.flowType](dispatchData);
......@@ -205,10 +206,13 @@ export async function dispatchModules({
return {};
})();
// format response data. Add modulename and moduletype
const formatResponseData = (() => {
if (!dispatchRes[ModuleOutputKeyEnum.responseData]) return undefined;
if (Array.isArray(dispatchRes[ModuleOutputKeyEnum.responseData]))
if (Array.isArray(dispatchRes[ModuleOutputKeyEnum.responseData])) {
return dispatchRes[ModuleOutputKeyEnum.responseData];
}
return {
moduleName: module.name,
moduleType: module.flowType,
......@@ -216,8 +220,16 @@ export async function dispatchModules({
};
})();
// Pass userChatInput
const hasUserChatInputTarget = !!module.outputs.find(
(item) => item.key === ModuleOutputKeyEnum.userChatInput
)?.targets?.length;
return moduleOutput(module, {
[ModuleOutputKeyEnum.finish]: true,
[ModuleOutputKeyEnum.userChatInput]: hasUserChatInputTarget
? params[ModuleOutputKeyEnum.userChatInput]
: undefined,
...dispatchRes,
[ModuleOutputKeyEnum.responseData]: formatResponseData
});
......
projects/app/src/web/core/dataset/store/searchTest.ts
......@@ -12,6 +12,9 @@ export type SearchTestStoreItemType = {
duration: string;
results: SearchDataResponseItemType[];
searchMode: `${DatasetSearchModeEnum}`;
limit: number;
usingReRank: boolean;
similarity: number;
};
type State = {
......
projects/app/src/web/styles/theme.ts
......@@ -417,8 +417,8 @@ export const theme = extendTheme({
body: 'PingFang,Noto Sans,-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol"'
},
fontSizes: {
xs: '10',
sm: '12px',
xs: '12px',
sm: '13px',
md: '14px',
lg: '16px',
xl: '18px',
......@@ -440,7 +440,7 @@ export const theme = extendTheme({
md: '1px solid #DAE0E2',
lg: '1px solid #D0E0E2'
},
borderRadius: {
radii: {
xs: '4px',
sm: '6px',
md: '8px',
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment