Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 8a463724 by Archer Committed by GitHub
Options

4.7.1-alpha (#1120) 

Co-authored-by: heheer <71265218+newfish-cmyk@users.noreply.github.com>
parent 9ae581e0
Showing with 1709 additions and 544 deletions
README.md
...@@ -60,7 +60,7 @@ fastgpt.run 域名会弃用。 ...@@ -60,7 +60,7 @@ fastgpt.run 域名会弃用。
- [x] 混合检索 & 重排 - [x] 混合检索 & 重排
- [x] Tool 模块 - [x] Tool 模块
- [ ] 嵌入 [Laf](https://github.com/labring/laf)，实现在线编写 HTTP 模块 - [ ] 嵌入 [Laf](https://github.com/labring/laf)，实现在线编写 HTTP 模块
- [ ] 插件封装功能 - [ ] 插件封装功能，支持低代码渲染
`2` 知识库能力 `2` 知识库能力
- [x] 多库复用，混用 - [x] 多库复用，混用
...@@ -68,9 +68,8 @@ fastgpt.run 域名会弃用。 ...@@ -68,9 +68,8 @@ fastgpt.run 域名会弃用。
- [x] 支持知识库单独设置向量模型 - [x] 支持知识库单独设置向量模型
- [x] 源文件存储 - [x] 源文件存储
- [x] 支持手动输入，直接分段，QA 拆分导入 - [x] 支持手动输入，直接分段，QA 拆分导入
- [x] 支持 pdf，docx，txt，html，md，csv - [x] 支持。txt， 。md， 。html， 。pdf， 。docx，pptx， 。csv， 。xlsx (有需要更多可 PR file loader)
- [x] 支持 url 读取、CSV 批量导入 - [x] 支持 url 读取、CSV 批量导入
- [ ] 支持 PPT、Excel 导入
- [ ] 支持文件阅读器 - [ ] 支持文件阅读器
- [ ] 更多的数据预处理方案 - [ ] 更多的数据预处理方案
...@@ -114,7 +113,7 @@ fastgpt.run 域名会弃用。 ...@@ -114,7 +113,7 @@ fastgpt.run 域名会弃用。
* [多模型配置](https://doc.fastgpt.in/docs/development/one-api/) * [多模型配置](https://doc.fastgpt.in/docs/development/one-api/)
* [版本更新/升级介绍](https://doc.fastgpt.in/docs/development/upgrading) * [版本更新/升级介绍](https://doc.fastgpt.in/docs/development/upgrading)
* [OpenAPI API 文档](https://doc.fastgpt.in/docs/development/openapi/) * [OpenAPI API 文档](https://doc.fastgpt.in/docs/development/openapi/)
* [知识库结构详解](https://doc.fastgpt.in/docs/course/datasetengine/) * [知识库结构详解](https://doc.fastgpt.in/docs/course/dataset_engine/)
<a href="#readme"> <a href="#readme">
<img src="https://img.shields.io/badge/-返回顶部-7d09f1.svg" alt="#" align="right"> <img src="https://img.shields.io/badge/-返回顶部-7d09f1.svg" alt="#" align="right">
......
docSite/content/docs/course/ai_settings.md
...@@ -64,7 +64,7 @@ Tips: 可以通过点击上下文按键查看完整的上下文组成，便于 ...@@ -64,7 +64,7 @@ Tips: 可以通过点击上下文按键查看完整的上下文组成，便于
FastGPT 知识库采用 QA 对(不一定都是问答格式，仅代表两个变量)的格式存储，在转义成字符串时候会根据**引用模板**来进行格式化。知识库包含多个可用变量： q, a, sourceId（数据的ID）, index(第n个数据), source(数据的集合名、文件名)，score(距离得分，0-1) 可以通过 {{q}} {{a}} {{sourceId}} {{index}} {{source}} {{score}} 按需引入。下面一个模板例子： FastGPT 知识库采用 QA 对(不一定都是问答格式，仅代表两个变量)的格式存储，在转义成字符串时候会根据**引用模板**来进行格式化。知识库包含多个可用变量： q, a, sourceId（数据的ID）, index(第n个数据), source(数据的集合名、文件名)，score(距离得分，0-1) 可以通过 {{q}} {{a}} {{sourceId}} {{index}} {{source}} {{score}} 按需引入。下面一个模板例子：
可以通过 [知识库结构讲解](/docs/course/datasetEngine/) 了解详细的知识库的结构。 可以通过 [知识库结构讲解](/docs/course/dataset_engine/) 了解详细的知识库的结构。
#### 引用模板 #### 引用模板
......
docSite/content/docs/course/datasetEngine.md deleted 100644 → 0
---
title: "知识库结构讲解"
description: "本节会详细介绍 FastGPT 知识库结构设计，理解其 QA 的存储格式和多向量映射，以便更好的构建知识库。这篇介绍主要以使用为主，详细原理不多介绍。"
icon: "dataset"
draft: false
toc: true
weight: 102
---
## 理解向量
FastGPT 采用了 RAG 中的 Embedding 方案构建知识库，要使用好 FastGPT 需要简单的理解`Embedding`向量是如何工作的及其特点。
人类的文字、图片、视频等媒介是无法直接被计算机理解的，要想让计算机理解两段文字是否有相似性、相关性，通常需要将它们转成计算机可以理解的语言，向量是其中的一种方式。
向量可以简单理解为一个数字数组，两个向量之间可以通过数学公式得出一个`距离`，距离越小代表两个向量的相似度越大。从而映射到文字、图片、视频等媒介上，可以用来判断两个媒介之间的相似度。向量搜索便是利用了这个原理。
而由于文字是有多种类型，并且拥有成千上万种组合方式，因此在转成向量进行相似度匹配时，很难保障其精确性。在向量方案构建的知识库中，通常使用`topk`召回的方式，也就是查找前`k`个最相似的内容，丢给大模型去做更进一步的`语义判断`、`逻辑推理`和`归纳总结`，从而实现知识库问答。因此，在知识库问答中，向量搜索的环节是最为重要的。
影响向量搜索精度的因素非常多，主要包括：向量模型的质量、数据的质量（长度，完整性，多样性）、检索器的精度（速度与精度之间的取舍）。与数据质量对应的就是检索词的质量。
检索器的精度比较容易解决，向量模型的训练略复杂，因此数据和检索词质量优化成了一个重要的环节。
## FastGPT 中向量的结构设计
FastGPT 采用了 `PostgresSQL` 的 `PG Vector` 插件作为向量检索器，索引为`HNSW`。且`PostgresSQL`仅用于向量检索，`MongoDB`用于其他数据的存取。
在`MongoDB`的`dataset.datas`表中，会存储向量原数据的信息，同时有一个`indexes`字段，会记录其对应的向量ID，这是一个数组，也就是说，一组向量可以对应多组数据。
在`PostgresSQL`的表中，设置一个 `index` 字段用于存储向量。在检索时，会先召回向量，再根据向量的ID，去`MongoDB`中寻找原数据内容，如果对应了同一组原数据，则进行合并，向量得分取最高得分。
![](/imgs/datasetSetting1.png)
### 多向量的目的和使用方式
在一组向量中，内容的长度和语义的丰富度通常是矛盾的，无法兼得。因此，FastGPT 采用了多向量映射的方式，将一组数据映射到多组向量中，从而保障数据的完整性和语义的丰富度。
你可以为一组较长的文本，添加多组向量，从而在检索时，只要其中一组向量被检索到，该数据也将被召回。
### 提高向量搜索精度的方法
1. 更好分词分段：当一段话的结构和语义是完整的，并且是单一的，精度也会提高。因此，许多系统都会优化分词器，尽可能的保障每组数据的完整性。
2. 精简`index`的内容，减少向量内容的长度：当`index`的内容更少，更准确时，检索精度自然会提高。但与此同时，会牺牲一定的检索范围，适合答案较为严格的场景。
3. 丰富`index`的数量，可以为同一个`chunk`内容增加多组`index`。
4. 优化检索词：在实际使用过程中，用户的问题通常是模糊的或是缺失的，并不一定是完整清晰的问题。因此优化用户的问题（检索词）很大程度上也可以提高精度。
5. 微调向量模型：由于市面上直接使用的向量模型都是通用型模型，在特定领域的检索精度并不高，因此微调向量模型可以很大程度上提高专业领域的检索效果。
## FastGPT 构建知识库方案
在 FastGPT 中，整个知识库由库、集合和数据 3 部分组成。集合可以简单理解为一个`文件`。一个`库`中可以包含多个`集合`，一个`集合`中可以包含多组`数据`。最小的搜索单位是`库`，也就是说，知识库搜索时，是对整个`库`进行搜索，而集合仅是为了对数据进行分类管理，与搜索效果无关。（起码目前还是）
| 库 | 集合 | 数据 |
| --- | --- | --- |
| ![](/imgs/datasetEngine1.jpg) | ![](/imgs/datasetEngine2.jpg) | ![](/imgs/datasetEngine3.webp) |
### 导入数据方案1 - 直接分段导入
选择文件导入时，可以选择直接分段方案。直接分段会利用`句子分词器`对文本进行一定长度拆分，最终分割中多组的`q`。如果使用了直接分段方案，我们建议在`应用`设置`引用提示词`时，使用`通用模板`即可，无需选择`问答模板`。
| 交互 | 结果 |
| --- | --- |
| ![](/imgs/datasetEngine4.webp) | ![](/imgs/datasetEngine5.webp) |
### 导入数据方案2 - QA导入
选择文件导入时，可以选择QA拆分方案。仍然需要使用到`句子分词器`对文本进行拆分，但长度比直接分段大很多。在导入后，会先调用`大模型`对分段进行学习，并给出一些`问题`和`答案`，最终问题和答案会一起被存储到`q`中。注意，新版的 FastGPT 为了提高搜索的范围，不再将问题和答案分别存储到 qa 中。
| 交互 | 结果 |
| --- | --- |
| ![](/imgs/datasetEngine6.webp) | ![](/imgs/datasetEngine7.webp) |
### 导入数据方案3 - 手动录入
在 FastGPT 中，你可以在任何一个`集合`中点击右上角的`插入`手动录入知识点，或者使用`标注`功能手动录入。被搜索的内容为`q`，补充内容(可选)为`a`。
| | | |
| --- | --- | --- |
| ![](/imgs/datasetEngine8.jpg) | ![](/imgs/datasetEngine9.jpg) | ![](/imgs/datasetEngine10.jpg) |
### 导入数据方案4 - CSV录入
有些数据较为独特，可能需要单独的进行预处理分割后再导入 FastGPT，此时可以选择 csv 导入，可批量的将处理好的数据导入。
![](/imgs/datasetEngine11.jpg)
### 导入数据方案5 - API导入
参考[FastGPT OpenAPI使用](/docs/development/openapi)。
## QA的组合与引用提示词构建
参考[引用模板与引用提示词示例](/docs/course/ai_settings/#示例)
docSite/content/docs/course/dataset_engine.md 0 → 100644
---
title: '知识库搜索方案和参数'
description: '本节会详细介绍 FastGPT 知识库结构设计，理解其 QA 的存储格式和多向量映射，以便更好的构建知识库。同时会介绍每个搜索参数的功能。这篇介绍主要以使用为主，详细原理不多介绍。'
icon: 'language'
draft: false
toc: true
weight: 106
---
## 理解向量
FastGPT 采用了 RAG 中的 Embedding 方案构建知识库，要使用好 FastGPT 需要简单的理解`Embedding`向量是如何工作的及其特点。
人类的文字、图片、视频等媒介是无法直接被计算机理解的，要想让计算机理解两段文字是否有相似性、相关性，通常需要将它们转成计算机可以理解的语言，向量是其中的一种方式。
向量可以简单理解为一个数字数组，两个向量之间可以通过数学公式得出一个`距离`，距离越小代表两个向量的相似度越大。从而映射到文字、图片、视频等媒介上，可以用来判断两个媒介之间的相似度。向量搜索便是利用了这个原理。
而由于文字是有多种类型，并且拥有成千上万种组合方式，因此在转成向量进行相似度匹配时，很难保障其精确性。在向量方案构建的知识库中，通常使用`topk`召回的方式，也就是查找前`k`个最相似的内容，丢给大模型去做更进一步的`语义判断`、`逻辑推理`和`归纳总结`，从而实现知识库问答。因此，在知识库问答中，向量搜索的环节是最为重要的。
影响向量搜索精度的因素非常多，主要包括：向量模型的质量、数据的质量（长度，完整性，多样性）、检索器的精度（速度与精度之间的取舍）。与数据质量对应的就是检索词的质量。
检索器的精度比较容易解决，向量模型的训练略复杂，因此数据和检索词质量优化成了一个重要的环节。
### 提高向量搜索精度的方法
1. 更好分词分段：当一段话的结构和语义是完整的，并且是单一的，精度也会提高。因此，许多系统都会优化分词器，尽可能的保障每组数据的完整性。
2. 精简`index`的内容，减少向量内容的长度：当`index`的内容更少，更准确时，检索精度自然会提高。但与此同时，会牺牲一定的检索范围，适合答案较为严格的场景。
3. 丰富`index`的数量，可以为同一个`chunk`内容增加多组`index`。
4. 优化检索词：在实际使用过程中，用户的问题通常是模糊的或是缺失的，并不一定是完整清晰的问题。因此优化用户的问题（检索词）很大程度上也可以提高精度。
5. 微调向量模型：由于市面上直接使用的向量模型都是通用型模型，在特定领域的检索精度并不高，因此微调向量模型可以很大程度上提高专业领域的检索效果。
## FastGPT 构建知识库方案
### 数据存储结构
在 FastGPT 中，整个知识库由库、集合和数据 3 部分组成。集合可以简单理解为一个`文件`。一个`库`中可以包含多个`集合`，一个`集合`中可以包含多组`数据`。最小的搜索单位是`库`，也就是说，知识库搜索时，是对整个`库`进行搜索，而集合仅是为了对数据进行分类管理，与搜索效果无关。（起码目前还是）
![](/imgs/dataset_tree.png)
### 向量存储结构
FastGPT 采用了`PostgresSQL`的`PG Vector`插件作为向量检索器，索引为`HNSW`。且`PostgresSQL`仅用于向量检索（该引擎可以替换成其它数据库），`MongoDB`用于其他数据的存取。
在`MongoDB`的`dataset.datas`表中，会存储向量原数据的信息，同时有一个`indexes`字段，会记录其对应的向量ID，这是一个数组，也就是说，一组向量可以对应多组数据。
在`PostgresSQL`的表中，设置一个`vector`字段用于存储向量。在检索时，会先召回向量，再根据向量的ID，去`MongoDB`中寻找原数据内容，如果对应了同一组原数据，则进行合并，向量得分取最高得分。
![](/imgs/datasetSetting1.png)
### 多向量的目的和使用方式
在一组向量中，内容的长度和语义的丰富度通常是矛盾的，无法兼得。因此，FastGPT 采用了多向量映射的方式，将一组数据映射到多组向量中，从而保障数据的完整性和语义的丰富度。
你可以为一组较长的文本，添加多组向量，从而在检索时，只要其中一组向量被检索到，该数据也将被召回。
意味着，你可以通过标注数据块的方式，不断提高数据块的精度。
### 检索方案
1. 通过`问题优化`实现指代消除和问题扩展，从而增加连续对话的检索能力以及语义丰富度。
2. 通过`Concat query`来增加`Rerank`连续对话的时，排序的准确性。
3. 通过`RRF`合并方式，综合多个渠道的检索效果。
4. 通过`Rerank`来二次排序，提高精度。
![](/imgs/dataset_search_process.png)
## 搜索参数
| | | |
| --- |---| --- |
|![](/imgs/dataset_search_params1.png)| ![](/imgs/dataset_search_params2.png) | ![](/imgs/dataset_search_params3.png) |
### 搜索模式
#### 语义检索
语义检索是通过向量距离，计算用户问题与知识库内容的距离，从而得出“相似度”，当然这并不是语文上的相似度，而是数学上的。
优点：
- 相近语义理解
- 跨多语言理解（例如输入中文问题匹配英文知识点）
- 多模态理解（文本，图片，音视频等）
缺点：
- 依赖模型训练效果
- 精度不稳定
- 受关键词和句子完整度影响
#### 全文检索
采用传统的全文检索方式。适合查找关键的主谓语等。
#### 混合检索
同时使用向量检索和全文检索，并通过 RRF 公式进行两个搜索结果合并，一般情况下搜索结果会更加丰富准确。
由于混合检索后的查找范围很大，并且无法直接进行相似度过滤，通常需要进行利用重排模型进行一次结果重新排序，并利用重排的得分进行过滤。
#### 结果重排
利用`ReRank`模型对搜索结果进行重排，绝大多数情况下，可以有效提高搜索结果的准确率。不过，重排模型与问题的完整度（主谓语齐全）有一些关系，通常会先走问题优化后再进行搜索-重排。重排后可以得到一个`0-1`的得分，代表着搜索内容与问题的相关度，该分数通常比向量的得分更加精确，可以根据得分进行过滤。
FastGPT 会使用 `RRF` 对重排结果、向量搜索结果、全文检索结果进行合并，得到最终的搜索结果。
### 搜索过滤
#### 引用上限
每次搜索最多引用`n`个`tokens`的内容。
之所以不采用`top k`，是发现在混合知识库（问答库、文档库）时，不同`chunk`的长度差距很大，会导致`top k`的结果不稳定，因此采用了`tokens`的方式进行引用上限的控制。
#### 最低相关度
一个`0-1`的数值，会过滤掉一些低相关度的搜索结果。
该值仅在`语义检索`或使用`结果重排`时生效。
### 问题优化
#### 背景
在 RAG 中，我们需要根据输入的问题去数据库里执行 embedding 搜索，查找相关的内容，从而查找到相似的内容（简称知识库搜索）。
在搜索的过程中，尤其是连续对话的搜索，我们通常会发现后续的问题难以搜索到合适的内容，其中一个原因是知识库搜索只会使用“当前”的问题去执行。看下面的例子：
![](/imgs/coreferenceResolution2.webp)
用户在提问“第二点是什么”的时候，只会去知识库里查找“第二点是什么”，压根查不到内容。实际上需要查询的是“QA结构是什么”。因此我们需要引入一个【问题优化】模块，来对用户当前的问题进行补全，从而使得知识库搜索能够搜索到合适的内容。使用补全后效果如下：
![](/imgs/coreferenceResolution3.webp)
#### 实现方式
在进行`数据检索`前，会先让模型进行`指代消除`与`问题扩展`，一方面可以可以解决指代对象不明确问题，同时可以扩展问题的语义丰富度。你可以通过每次对话后的对话详情，查看补全的结果。
docSite/content/docs/development/configuration.md
...@@ -19,6 +19,9 @@ llm模型全部合并 ...@@ -19,6 +19,9 @@ llm模型全部合并
```json ```json
{ {
"feConfigs": {
"lafEnv": "https://laf.dev" // laf环境
},
"systemEnv": { "systemEnv": {
"vectorMaxProcess": 15, "vectorMaxProcess": 15,
"qaMaxProcess": 15, "qaMaxProcess": 15,
...@@ -164,7 +167,7 @@ llm模型全部合并 ...@@ -164,7 +167,7 @@ llm模型全部合并
"model": "bge-reranker-base", // 随意 "model": "bge-reranker-base", // 随意
"name": "检索重排-base", // 随意 "name": "检索重排-base", // 随意
"charsPointsPrice": 0, "charsPointsPrice": 0,
"requestUrl": "{{host}}/api/v1/rerank", "requestUrl": "{{host}}/v1/rerank",
"requestAuth": "安全凭证，已自动补 Bearer" "requestAuth": "安全凭证，已自动补 Bearer"
} }
] ]
......
docSite/content/docs/development/custom-models/reranker.md
...@@ -44,7 +44,7 @@ weight: 910 ...@@ -44,7 +44,7 @@ weight: 910
### docker 部署 ### docker 部署
+ 镜像名: `luanshaotong/reranker:v0.2` + 镜像名: `registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2`
+ 端口号: 6006 + 端口号: 6006
+ 大小：约8GB + 大小：约8GB
...@@ -56,12 +56,12 @@ ACCESS_TOKEN=mytoken ...@@ -56,12 +56,12 @@ ACCESS_TOKEN=mytoken
**运行命令示例** **运行命令示例**
- 无需GPU环境，使用CPU运行 - 无需GPU环境，使用CPU运行
```sh ```sh
docker run -d --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken luanshaotong/reranker:v0.2 docker run -d --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
``` ```
- 需要CUDA 11.7环境 - 需要CUDA 11.7环境
```sh ```sh
docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken luanshaotong/reranker:v0.2 docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
``` ```
**docker-compose.yml示例** **docker-compose.yml示例**
...@@ -69,7 +69,7 @@ docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken lu ...@@ -69,7 +69,7 @@ docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken lu
version: "3" version: "3"
services: services:
reranker: reranker:
image: luanshaotong/reranker:v0.2 image: registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
container_name: reranker container_name: reranker
# GPU运行环境，如果宿主机未安装，将deploy配置隐藏即可 # GPU运行环境，如果宿主机未安装，将deploy配置隐藏即可
deploy: deploy:
......
docSite/content/docs/development/upgrading/47.md
--- ---
title: 'V4.7(进行中)' title: 'V4.7'
description: 'FastGPT V4.7更新说明' description: 'FastGPT V4.7更新说明'
icon: 'upgrade' icon: 'upgrade'
draft: false draft: false
...@@ -26,7 +26,7 @@ curl --location --request POST 'https://{{host}}/api/admin/initv47' \ ...@@ -26,7 +26,7 @@ curl --location --request POST 'https://{{host}}/api/admin/initv47' \
## 3. 升级 ReRank 模型 ## 3. 升级 ReRank 模型
4.7对ReRank模型进行了格式变动，兼容 cohere 的格式，可以直接使用 cohere 提供的 API。如果是本地的 ReRank 模型，需要修改镜像为：`luanshaotong/reranker:v0.2` 。 4.7对ReRank模型进行了格式变动，兼容 cohere 的格式，可以直接使用 cohere 提供的 API。如果是本地的 ReRank 模型，需要修改镜像为：`registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2` 。
cohere的重排模型对中文不是很好，感觉不如 bge 的好用，接入教程如下： cohere的重排模型对中文不是很好，感觉不如 bge 的好用，接入教程如下：
......
docSite/content/docs/development/upgrading/471.md 0 → 100644
---
title: 'V4.7.1'
description: 'FastGPT V4.7.1 更新说明'
icon: 'upgrade'
draft: false
toc: true
weight: 825
---
## 初始化脚本
从任意终端，发起 1 个 HTTP 请求。其中 {{rootkey}} 替换成环境变量里的 `rootkey`；{{host}} 替换成FastGPT的域名。
```bash
curl --location --request POST 'https://{{host}}/api/admin/clearInvalidData' \
--header 'rootkey: {{rootkey}}' \
--header 'Content-Type: application/json'
```
该请求会执行脏数据清理（清理无效的文件、清理无效的图片、清理无效的知识库集合、清理无效的向量）
## V4.7.1 更新说明
1. 新增 - Pptx 和 xlsx 文件读取。但所有文件读取都放服务端，会消耗更多的服务器资源，以及无法在上传时预览更多内容。
2. 新增 - 集成 Laf 云函数，可以读取 Laf 账号中的云函数作为 HTTP 模块。
3. 新增 - 定时器，清理垃圾数据。（采用小范围清理，会清理最近n个小时的，所以请保证服务持续运行，长时间不允许，可以继续执行 clearInvalidData 的接口进行全量清理。）
4. 修改 - csv导入模板，取消 header 校验，自动获取前两列。
5. 修复 - 工具调用模块连线数据类型校验错误。
\ No newline at end of file
packages/global/common/file/icon.ts
export const fileImgs = [ export const fileImgs = [
{ suffix: 'pdf', src: 'file/fill/pdf' }, { suffix: 'pdf', src: 'file/fill/pdf' },
{ suffix: 'ppt', src: 'file/fill/ppt' },
{ suffix: 'xlsx', src: 'file/fill/xlsx' },
{ suffix: 'csv', src: 'file/fill/csv' }, { suffix: 'csv', src: 'file/fill/csv' },
{ suffix: '(doc|docs)', src: 'file/fill/doc' }, { suffix: '(doc|docs)', src: 'file/fill/doc' },
{ suffix: 'txt', src: 'file/fill/txt' }, { suffix: 'txt', src: 'file/fill/txt' },
......
packages/global/common/file/tools.ts
...@@ -11,5 +11,5 @@ export const formatFileSize = (bytes: number): string => { ...@@ -11,5 +11,5 @@ export const formatFileSize = (bytes: number): string => {
}; };
export const detectFileEncoding = (buffers: string | Buffer) => { export const detectFileEncoding = (buffers: string | Buffer) => {
return detect(buffers)?.encoding || 'utf-8'; return (detect(buffers)?.encoding || 'utf-8') as BufferEncoding;
}; };
packages/global/common/system/types/index.d.ts
...@@ -57,6 +57,7 @@ export type FastGPTFeConfigsType = { ...@@ -57,6 +57,7 @@ export type FastGPTFeConfigsType = {
uploadFileMaxAmount?: number; uploadFileMaxAmount?: number;
uploadFileMaxSize?: number; uploadFileMaxSize?: number;
lafEnv?: string;
}; };
export type SystemEnvType = { export type SystemEnvType = {
......
packages/global/core/module/node/constant.ts
...@@ -61,7 +61,8 @@ export enum FlowNodeTypeEnum { ...@@ -61,7 +61,8 @@ export enum FlowNodeTypeEnum {
pluginOutput = 'pluginOutput', pluginOutput = 'pluginOutput',
queryExtension = 'cfr', queryExtension = 'cfr',
tools = 'tools', tools = 'tools',
stopTool = 'stopTool' stopTool = 'stopTool',
lafModule = 'lafModule'
// abandon // abandon
} }
......
packages/global/core/module/template/constants.ts
...@@ -20,6 +20,7 @@ import { AiQueryExtension } from './system/queryExtension'; ...@@ -20,6 +20,7 @@ import { AiQueryExtension } from './system/queryExtension';
import type { FlowNodeTemplateType, moduleTemplateListType } from '../../module/type.d'; import type { FlowNodeTemplateType, moduleTemplateListType } from '../../module/type.d';
import { FlowNodeTemplateTypeEnum } from '../../module/constants'; import { FlowNodeTemplateTypeEnum } from '../../module/constants';
import { lafModule } from './system/laf';
/* app flow module templates */ /* app flow module templates */
export const appSystemModuleTemplates: FlowNodeTemplateType[] = [ export const appSystemModuleTemplates: FlowNodeTemplateType[] = [
...@@ -35,7 +36,8 @@ export const appSystemModuleTemplates: FlowNodeTemplateType[] = [ ...@@ -35,7 +36,8 @@ export const appSystemModuleTemplates: FlowNodeTemplateType[] = [
ClassifyQuestionModule, ClassifyQuestionModule,
ContextExtractModule, ContextExtractModule,
HttpModule468, HttpModule468,
AiQueryExtension AiQueryExtension,
lafModule
]; ];
/* plugin flow module templates */ /* plugin flow module templates */
export const pluginSystemModuleTemplates: FlowNodeTemplateType[] = [ export const pluginSystemModuleTemplates: FlowNodeTemplateType[] = [
...@@ -51,7 +53,8 @@ export const pluginSystemModuleTemplates: FlowNodeTemplateType[] = [ ...@@ -51,7 +53,8 @@ export const pluginSystemModuleTemplates: FlowNodeTemplateType[] = [
ClassifyQuestionModule, ClassifyQuestionModule,
ContextExtractModule, ContextExtractModule,
HttpModule468, HttpModule468,
AiQueryExtension AiQueryExtension,
lafModule
]; ];
/* all module */ /* all module */
...@@ -73,7 +76,8 @@ export const moduleTemplatesFlat: FlowNodeTemplateType[] = [ ...@@ -73,7 +76,8 @@ export const moduleTemplatesFlat: FlowNodeTemplateType[] = [
PluginInputModule, PluginInputModule,
PluginOutputModule, PluginOutputModule,
RunPluginModule, RunPluginModule,
AiQueryExtension AiQueryExtension,
lafModule
]; ];
export const moduleTemplatesList: moduleTemplateListType = [ export const moduleTemplatesList: moduleTemplateListType = [
......
packages/global/core/module/template/system/laf.ts 0 → 100644
import {
FlowNodeInputTypeEnum,
FlowNodeOutputTypeEnum,
FlowNodeTypeEnum
} from '../../node/constant';
import { FlowNodeTemplateType } from '../../type';
import {
ModuleIOValueTypeEnum,
ModuleInputKeyEnum,
ModuleOutputKeyEnum,
FlowNodeTemplateTypeEnum
} from '../../constants';
import {
Input_Template_DynamicInput,
Input_Template_Switch,
Input_Template_AddInputParam
} from '../input';
import { Output_Template_Finish, Output_Template_AddOutput } from '../output';
export const lafModule: FlowNodeTemplateType = {
id: FlowNodeTypeEnum.lafModule,
templateType: FlowNodeTemplateTypeEnum.externalCall,
flowType: FlowNodeTypeEnum.lafModule,
avatar: '/imgs/module/laf.png',
name: 'Laf 函数调用（测试）',
intro: '可以调用Laf账号下的云函数。',
showStatus: true,
isTool: true,
inputs: [
Input_Template_Switch,
{
key: ModuleInputKeyEnum.httpReqUrl,
type: FlowNodeInputTypeEnum.hidden,
valueType: ModuleIOValueTypeEnum.string,
label: '',
description: 'core.module.input.description.Http Request Url',
placeholder: 'https://api.ai.com/getInventory',
required: false,
showTargetInApp: false,
showTargetInPlugin: false
},
Input_Template_DynamicInput,
{
...Input_Template_AddInputParam,
editField: {
key: true,
description: true,
dataType: true
},
defaultEditField: {
label: '',
key: '',
description: '',
inputType: FlowNodeInputTypeEnum.target,
valueType: ModuleIOValueTypeEnum.string
}
}
],
outputs: [
{
key: ModuleOutputKeyEnum.httpRawResponse,
label: '原始响应',
description: 'HTTP请求的原始响应。只能接受字符串或JSON类型响应数据。',
valueType: ModuleIOValueTypeEnum.any,
type: FlowNodeOutputTypeEnum.source,
targets: []
},
{
...Output_Template_AddOutput,
editField: {
key: true,
description: true,
dataType: true,
defaultValue: true
},
defaultEditField: {
label: '',
key: '',
description: '',
outputType: FlowNodeOutputTypeEnum.source,
valueType: ModuleIOValueTypeEnum.string
}
},
Output_Template_Finish
]
};
packages/global/core/module/type.d.ts
...@@ -9,6 +9,7 @@ import { DispatchNodeResponseKeyEnum } from './runtime/constants'; ...@@ -9,6 +9,7 @@ import { DispatchNodeResponseKeyEnum } from './runtime/constants';
import { FlowNodeInputItemType, FlowNodeOutputItemType } from './node/type'; import { FlowNodeInputItemType, FlowNodeOutputItemType } from './node/type';
import { UserModelSchema } from 'support/user/type'; import { UserModelSchema } from 'support/user/type';
import { import {
ChatItemType,
ChatItemValueItemType, ChatItemValueItemType,
ToolRunResponseItemType, ToolRunResponseItemType,
UserChatItemValueItemType UserChatItemValueItemType
......
packages/global/core/plugin/httpPlugin/utils.ts
...@@ -41,7 +41,7 @@ export const str2OpenApiSchema = async (yamlStr = ''): Promise<OpenApiJsonSchema ...@@ -41,7 +41,7 @@ export const str2OpenApiSchema = async (yamlStr = ''): Promise<OpenApiJsonSchema
path, path,
method, method,
name: methodInfo.operationId || path, name: methodInfo.operationId || path,
description: methodInfo.description, description: methodInfo.description || methodInfo.summary,
params: methodInfo.parameters, params: methodInfo.parameters,
request: methodInfo?.requestBody request: methodInfo?.requestBody
}; };
......
packages/global/support/user/team/controller.d.ts
import { TeamMemberRoleEnum } from './constant'; import { TeamMemberRoleEnum } from './constant';
import { TeamMemberSchema } from './type'; import { LafAccountType, TeamMemberSchema } from './type';
export type AuthTeamRoleProps = { export type AuthTeamRoleProps = {
teamId: string; teamId: string;
...@@ -10,12 +10,14 @@ export type CreateTeamProps = { ...@@ -10,12 +10,14 @@ export type CreateTeamProps = {
name: string; name: string;
avatar?: string; avatar?: string;
defaultTeam?: boolean; defaultTeam?: boolean;
lafAccount?: LafAccountType;
}; };
export type UpdateTeamProps = { export type UpdateTeamProps = {
teamId: string; teamId: string;
name?: string; name?: string;
avatar?: string; avatar?: string;
teamDomain?: string; teamDomain?: string;
lafAccount?: null | LafAccountType;
}; };
/* ------------- member ----------- */ /* ------------- member ----------- */
......
packages/global/support/user/team/type.d.ts
import type { UserModelSchema } from '../type'; import type { UserModelSchema } from '../type';
import type { TeamMemberRoleEnum, TeamMemberStatusEnum } from './constant'; import type { TeamMemberRoleEnum, TeamMemberStatusEnum } from './constant';
import { LafAccountType } from './type';
export type TeamSchema = { export type TeamSchema = {
_id: string; _id: string;
...@@ -13,6 +14,7 @@ export type TeamSchema = { ...@@ -13,6 +14,7 @@ export type TeamSchema = {
lastExportDatasetTime: Date; lastExportDatasetTime: Date;
lastWebsiteSyncTime: Date; lastWebsiteSyncTime: Date;
}; };
lafAccount: LafAccountType;
}; };
export type tagsType = { export type tagsType = {
label: string; label: string;
...@@ -58,6 +60,7 @@ export type TeamItemType = { ...@@ -58,6 +60,7 @@ export type TeamItemType = {
role: `${TeamMemberRoleEnum}`; role: `${TeamMemberRoleEnum}`;
status: `${TeamMemberStatusEnum}`; status: `${TeamMemberStatusEnum}`;
canWrite: boolean; canWrite: boolean;
lafAccount?: LafAccountType;
}; };
export type TeamMemberItemType = { export type TeamMemberItemType = {
...@@ -74,3 +77,8 @@ export type TeamTagItemType = { ...@@ -74,3 +77,8 @@ export type TeamTagItemType = {
label: string; label: string;
key: string; key: string;
}; };
export type LafAccountType = {
token: string;
appid: string;
};
packages/service/common/file/gridfs/controller.ts
...@@ -6,16 +6,9 @@ import { DatasetFileSchema } from '@fastgpt/global/core/dataset/type'; ...@@ -6,16 +6,9 @@ import { DatasetFileSchema } from '@fastgpt/global/core/dataset/type';
import { MongoFileSchema } from './schema'; import { MongoFileSchema } from './schema';
import { detectFileEncoding } from '@fastgpt/global/common/file/tools'; import { detectFileEncoding } from '@fastgpt/global/common/file/tools';
import { CommonErrEnum } from '@fastgpt/global/common/error/code/common'; import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
import { readFileRawText } from '../read/rawText';
import { ReadFileByBufferParams } from '../read/type'; import { ReadFileByBufferParams } from '../read/type';
import { readMarkdown } from '../read/markdown';
import { readHtmlRawText } from '../read/html';
import { readPdfFile } from '../read/pdf';
import { readWordFile } from '../read/word';
import { readCsvRawText } from '../read/csv';
import { MongoRwaTextBuffer } from '../../buffer/rawText/schema'; import { MongoRwaTextBuffer } from '../../buffer/rawText/schema';
import { readPptxRawText } from '../read/pptx'; import { readFileRawContent } from '../read/utils';
import { readXlsxRawText } from '../read/xlsx';
export function getGFSCollection(bucket: `${BucketNameEnum}`) { export function getGFSCollection(bucket: `${BucketNameEnum}`) {
MongoFileSchema; MongoFileSchema;
...@@ -146,7 +139,7 @@ export const readFileEncode = async ({ ...@@ -146,7 +139,7 @@ export const readFileEncode = async ({
return encoding as BufferEncoding; return encoding as BufferEncoding;
}; };
export const readFileContent = async ({ export const readFileContentFromMongo = async ({
teamId, teamId,
bucketName, bucketName,
fileId, fileId,
...@@ -205,47 +198,14 @@ export const readFileContent = async ({ ...@@ -205,47 +198,14 @@ export const readFileContent = async ({
} }
}; };
const { rawText } = await (async () => { const { rawText } = await readFileRawContent({
switch (extension) { extension,
case 'txt': csvFormat,
return readFileRawText(params); params
case 'md': });
return readMarkdown(params);
case 'html':
return readHtmlRawText(params);
case 'pdf':
return readPdfFile(params);
case 'docx':
return readWordFile(params);
case 'pptx':
return readPptxRawText(params);
case 'xlsx':
const xlsxResult = await readXlsxRawText(params);
if (csvFormat) {
return {
rawText: xlsxResult.formatText || ''
};
}
return {
rawText: xlsxResult.rawText
};
case 'csv':
const csvResult = await readCsvRawText(params);
if (csvFormat) {
return {
rawText: csvResult.formatText || ''
};
}
return {
rawText: csvResult.rawText
};
default:
return Promise.reject('Only support .txt, .md, .html, .pdf, .docx, pptx, .csv, .xlsx');
}
})();
if (rawText.trim()) { if (rawText.trim()) {
await MongoRwaTextBuffer.create({ MongoRwaTextBuffer.create({
sourceId: fileId, sourceId: fileId,
rawText, rawText,
metadata: { metadata: {
......
packages/service/common/file/read/utils.ts
...@@ -2,6 +2,15 @@ import { markdownProcess } from '@fastgpt/global/common/string/markdown'; ...@@ -2,6 +2,15 @@ import { markdownProcess } from '@fastgpt/global/common/string/markdown';
import { uploadMongoImg } from '../image/controller'; import { uploadMongoImg } from '../image/controller';
import { MongoImageTypeEnum } from '@fastgpt/global/common/file/image/constants'; import { MongoImageTypeEnum } from '@fastgpt/global/common/file/image/constants';
import { addHours } from 'date-fns'; import { addHours } from 'date-fns';
import { ReadFileByBufferParams } from './type';
import { readFileRawText } from '../read/rawText';
import { readMarkdown } from '../read/markdown';
import { readHtmlRawText } from '../read/html';
import { readPdfFile } from '../read/pdf';
import { readWordFile } from '../read/word';
import { readCsvRawText } from '../read/csv';
import { readPptxRawText } from '../read/pptx';
import { readXlsxRawText } from '../read/xlsx';
export const initMarkdownText = ({ export const initMarkdownText = ({
teamId, teamId,
...@@ -23,3 +32,50 @@ export const initMarkdownText = ({ ...@@ -23,3 +32,50 @@ export const initMarkdownText = ({
expiredTime: addHours(new Date(), 2) expiredTime: addHours(new Date(), 2)
}) })
}); });
export const readFileRawContent = async ({
extension,
csvFormat,
params
}: {
csvFormat?: boolean;
extension: string;
params: ReadFileByBufferParams;
}) => {
switch (extension) {
case 'txt':
return readFileRawText(params);
case 'md':
return readMarkdown(params);
case 'html':
return readHtmlRawText(params);
case 'pdf':
return readPdfFile(params);
case 'docx':
return readWordFile(params);
case 'pptx':
return readPptxRawText(params);
case 'xlsx':
const xlsxResult = await readXlsxRawText(params);
if (csvFormat) {
return {
rawText: xlsxResult.formatText || ''
};
}
return {
rawText: xlsxResult.rawText
};
case 'csv':
const csvResult = await readCsvRawText(params);
if (csvFormat) {
return {
rawText: csvResult.formatText || ''
};
}
return {
rawText: csvResult.rawText
};
default:
return Promise.reject('Only support .txt, .md, .html, .pdf, .docx, pptx, .csv, .xlsx');
}
};
packages/service/common/file/utils.ts
...@@ -55,8 +55,8 @@ export const clearTmpUploadFiles = () => { ...@@ -55,8 +55,8 @@ export const clearTmpUploadFiles = () => {
fs.stat(filePath, (err, stats) => { fs.stat(filePath, (err, stats) => {
if (err) return; if (err) return;
// 如果文件是在1小时前上传的，则认为是临时文件并删除它 // 如果文件是在2小时前上传的，则认为是临时文件并删除它
if (Date.now() - stats.mtime.getTime() > 1 * 60 * 60 * 1000) { if (Date.now() - stats.mtime.getTime() > 2 * 60 * 60 * 1000) {
fs.unlink(filePath, (err) => { fs.unlink(filePath, (err) => {
if (err) return; if (err) return;
console.log(`Deleted temp file: ${filePath}`); console.log(`Deleted temp file: ${filePath}`);
......
packages/service/common/system/timerLock/constants.ts 0 → 100644
export enum TimerIdEnum {
checkInValidDatasetFiles = 'checkInValidDatasetFiles',
checkInvalidDatasetData = 'checkInvalidDatasetData',
checkInvalidVector = 'checkInvalidVector',
clearExpiredSubPlan = 'clearExpiredSubPlan',
updateStandardPlan = 'updateStandardPlan'
}
export const timerIdMap = {
[TimerIdEnum.checkInValidDatasetFiles]: 'checkInValidDatasetFiles',
[TimerIdEnum.checkInvalidDatasetData]: 'checkInvalidDatasetData',
[TimerIdEnum.checkInvalidVector]: 'checkInvalidVector',
[TimerIdEnum.clearExpiredSubPlan]: 'clearExpiredSubPlan',
[TimerIdEnum.updateStandardPlan]: 'updateStandardPlan'
};
packages/service/common/system/timerLock/schema.ts 0 → 100644
import { connectionMongo, type Model } from '../../mongo';
import { timerIdMap } from './constants';
const { Schema, model, models } = connectionMongo;
import { TimerLockSchemaType } from './type.d';
export const collectionName = 'systemtimerlocks';
const TimerLockSchema = new Schema({
timerId: {
type: String,
required: true,
unique: true,
enum: Object.keys(timerIdMap)
},
expiredTime: {
type: Date,
required: true
}
});
try {
TimerLockSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 5 });
} catch (error) {
console.log(error);
}
export const MongoTimerLock: Model<TimerLockSchemaType> =
models[collectionName] || model(collectionName, TimerLockSchema);
MongoTimerLock.syncIndexes();
packages/service/common/system/timerLock/type.d.ts 0 → 100644
export type TimerLockSchemaType = {
_id: string;
timerId: string;
expiredTime: Date;
};
packages/service/common/system/timerLock/utils.ts 0 → 100644
import { TimerIdEnum } from './constants';
import { MongoTimerLock } from './schema';
import { addMinutes } from 'date-fns';
/*
利用唯一健，使得同一时间只有一个任务在执行，后创建的锁，会因唯一健创建失败，从而无法继续执行任务
*/
export const checkTimerLock = async ({
timerId,
lockMinuted
}: {
timerId: `${TimerIdEnum}`;
lockMinuted: number;
}) => {
try {
await MongoTimerLock.create({
timerId,
expiredTime: addMinutes(new Date(), lockMinuted)
});
return true;
} catch (error) {
return false;
}
};
packages/service/core/workflow/dispatch/agent/runTool/toolChoice.ts
...@@ -210,7 +210,6 @@ export const runToolWithToolChoice = async ( ...@@ -210,7 +210,6 @@ export const runToolWithToolChoice = async (
).filter(Boolean) as ToolRunResponseType; ).filter(Boolean) as ToolRunResponseType;
const flatToolsResponseData = toolsRunResponse.map((item) => item.moduleRunResponse).flat(); const flatToolsResponseData = toolsRunResponse.map((item) => item.moduleRunResponse).flat();
if (toolCalls.length > 0 && !res.closed) { if (toolCalls.length > 0 && !res.closed) {
// Run the tool, combine its results, and perform another round of AI calls // Run the tool, combine its results, and perform another round of AI calls
const assistantToolMsgParams: ChatCompletionAssistantToolParam = { const assistantToolMsgParams: ChatCompletionAssistantToolParam = {
......
packages/service/core/workflow/dispatch/index.ts
...@@ -37,6 +37,7 @@ import { dispatchRunTools } from './agent/runTool/index'; ...@@ -37,6 +37,7 @@ import { dispatchRunTools } from './agent/runTool/index';
import { ChatItemValueTypeEnum } from '@fastgpt/global/core/chat/constants'; import { ChatItemValueTypeEnum } from '@fastgpt/global/core/chat/constants';
import { DispatchFlowResponse } from './type'; import { DispatchFlowResponse } from './type';
import { dispatchStopToolCall } from './agent/runTool/stopTool'; import { dispatchStopToolCall } from './agent/runTool/stopTool';
import { dispatchLafRequest } from './tools/runLaf';
const callbackMap: Record<`${FlowNodeTypeEnum}`, Function> = { const callbackMap: Record<`${FlowNodeTypeEnum}`, Function> = {
[FlowNodeTypeEnum.historyNode]: dispatchHistory, [FlowNodeTypeEnum.historyNode]: dispatchHistory,
...@@ -56,6 +57,7 @@ const callbackMap: Record<`${FlowNodeTypeEnum}`, Function> = { ...@@ -56,6 +57,7 @@ const callbackMap: Record<`${FlowNodeTypeEnum}`, Function> = {
[FlowNodeTypeEnum.queryExtension]: dispatchQueryExtension, [FlowNodeTypeEnum.queryExtension]: dispatchQueryExtension,
[FlowNodeTypeEnum.tools]: dispatchRunTools, [FlowNodeTypeEnum.tools]: dispatchRunTools,
[FlowNodeTypeEnum.stopTool]: dispatchStopToolCall, [FlowNodeTypeEnum.stopTool]: dispatchStopToolCall,
[FlowNodeTypeEnum.lafModule]: dispatchLafRequest,
// none // none
[FlowNodeTypeEnum.userGuide]: () => Promise.resolve() [FlowNodeTypeEnum.userGuide]: () => Promise.resolve()
......
packages/service/core/workflow/dispatch/tools/http468.ts
...@@ -61,7 +61,7 @@ export const dispatchHttp468Request = async (props: HttpRequestProps): Promise<H ...@@ -61,7 +61,7 @@ export const dispatchHttp468Request = async (props: HttpRequestProps): Promise<H
chatId, chatId,
responseChatItemId, responseChatItemId,
...variables, ...variables,
histories: histories.slice(0, 10), histories: histories.slice(-10),
...body ...body
}; };
...@@ -165,6 +165,7 @@ async function fetchData({ ...@@ -165,6 +165,7 @@ async function fetchData({
'Content-Type': 'application/json', 'Content-Type': 'application/json',
...headers ...headers
}, },
timeout: 120000,
params: params, params: params,
data: ['POST', 'PUT', 'PATCH'].includes(method) ? body : undefined data: ['POST', 'PUT', 'PATCH'].includes(method) ? body : undefined
}); });
......
packages/service/core/workflow/dispatch/tools/runLaf.ts 0 → 100644
import type { ModuleDispatchProps } from '@fastgpt/global/core/module/type.d';
import {
DYNAMIC_INPUT_KEY,
ModuleInputKeyEnum,
ModuleOutputKeyEnum
} from '@fastgpt/global/core/module/constants';
import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/module/runtime/constants';
import axios from 'axios';
import { valueTypeFormat } from '../utils';
import { SERVICE_LOCAL_HOST } from '../../../../common/system/tools';
import { addLog } from '../../../../common/system/log';
import { DispatchNodeResultType } from '@fastgpt/global/core/module/runtime/type';
type LafRequestProps = ModuleDispatchProps<{
[ModuleInputKeyEnum.httpReqUrl]: string;
[DYNAMIC_INPUT_KEY]: Record<string, any>;
[key: string]: any;
}>;
type LafResponse = DispatchNodeResultType<{
[ModuleOutputKeyEnum.failed]?: boolean;
[key: string]: any;
}>;
const UNDEFINED_SIGN = 'UNDEFINED_SIGN';
export const dispatchLafRequest = async (props: LafRequestProps): Promise<LafResponse> => {
let {
appId,
chatId,
responseChatItemId,
variables,
module: { outputs },
histories,
params: { system_httpReqUrl: httpReqUrl, [DYNAMIC_INPUT_KEY]: dynamicInput, ...body }
} = props;
if (!httpReqUrl) {
return Promise.reject('Http url is empty');
}
const concatVariables = {
appId,
chatId,
responseChatItemId,
...variables,
...body
};
httpReqUrl = replaceVariable(httpReqUrl, concatVariables);
const requestBody = {
systemParams: {
appId,
chatId,
responseChatItemId,
histories: histories.slice(0, 10)
},
variables,
...dynamicInput,
...body
};
try {
const { formatResponse, rawResponse } = await fetchData({
method: 'POST',
url: httpReqUrl,
body: requestBody
});
// format output value type
const results: Record<string, any> = {};
for (const key in formatResponse) {
const output = outputs.find((item) => item.key === key);
if (!output) continue;
results[key] = valueTypeFormat(formatResponse[key], output.valueType);
}
return {
assistantResponses: [],
[DispatchNodeResponseKeyEnum.nodeResponse]: {
totalPoints: 0,
body: Object.keys(requestBody).length > 0 ? requestBody : undefined,
httpResult: rawResponse
},
[DispatchNodeResponseKeyEnum.toolResponses]: rawResponse,
[ModuleOutputKeyEnum.httpRawResponse]: rawResponse,
...results
};
} catch (error) {
addLog.error('Http request error', error);
return {
[ModuleOutputKeyEnum.failed]: true,
[DispatchNodeResponseKeyEnum.nodeResponse]: {
totalPoints: 0,
body: Object.keys(requestBody).length > 0 ? requestBody : undefined,
httpResult: { error: formatHttpError(error) }
}
};
}
};
async function fetchData({
method,
url,
body
}: {
method: string;
url: string;
body: Record<string, any>;
}): Promise<Record<string, any>> {
const { data: response } = await axios({
method,
baseURL: `http://${SERVICE_LOCAL_HOST}`,
url,
headers: {
'Content-Type': 'application/json'
},
data: body
});
const parseJson = (obj: Record<string, any>, prefix = '') => {
let result: Record<string, any> = {};
if (Array.isArray(obj)) {
for (let i = 0; i < obj.length; i++) {
result[`${prefix}[${i}]`] = obj[i];
if (Array.isArray(obj[i])) {
result = {
...result,
...parseJson(obj[i], `${prefix}[${i}]`)
};
} else if (typeof obj[i] === 'object') {
result = {
...result,
...parseJson(obj[i], `${prefix}[${i}].`)
};
}
}
} else if (typeof obj == 'object') {
for (const key in obj) {
result[`${prefix}${key}`] = obj[key];
if (Array.isArray(obj[key])) {
result = {
...result,
...parseJson(obj[key], `${prefix}${key}`)
};
} else if (typeof obj[key] === 'object') {
result = {
...result,
...parseJson(obj[key], `${prefix}${key}.`)
};
}
}
}
return result;
};
return {
formatResponse:
typeof response === 'object' && !Array.isArray(response) ? parseJson(response) : {},
rawResponse: response
};
}
function replaceVariable(text: string, obj: Record<string, any>) {
for (const [key, value] of Object.entries(obj)) {
if (value === undefined) {
text = text.replace(new RegExp(`{{${key}}}`, 'g'), UNDEFINED_SIGN);
} else {
const replacement = JSON.stringify(value);
const unquotedReplacement =
replacement.startsWith('"') && replacement.endsWith('"')
? replacement.slice(1, -1)
: replacement;
text = text.replace(new RegExp(`{{${key}}}`, 'g'), unquotedReplacement);
}
}
return text || '';
}
function removeUndefinedSign(obj: Record<string, any>) {
for (const key in obj) {
if (obj[key] === UNDEFINED_SIGN) {
obj[key] = undefined;
} else if (Array.isArray(obj[key])) {
obj[key] = obj[key].map((item: any) => {
if (item === UNDEFINED_SIGN) {
return undefined;
} else if (typeof item === 'object') {
removeUndefinedSign(item);
}
return item;
});
} else if (typeof obj[key] === 'object') {
removeUndefinedSign(obj[key]);
}
}
return obj;
}
function formatHttpError(error: any) {
return {
message: error?.message,
name: error?.name,
method: error?.config?.method,
baseURL: error?.config?.baseURL,
url: error?.config?.url,
code: error?.code,
status: error?.status
};
}
packages/service/support/user/team/controller.ts
...@@ -10,7 +10,6 @@ import { MongoTeam } from './teamSchema'; ...@@ -10,7 +10,6 @@ import { MongoTeam } from './teamSchema';
async function getTeamMember(match: Record<string, any>): Promise<TeamItemType> { async function getTeamMember(match: Record<string, any>): Promise<TeamItemType> {
const tmb = (await MongoTeamMember.findOne(match).populate('teamId')) as TeamMemberWithTeamSchema; const tmb = (await MongoTeamMember.findOne(match).populate('teamId')) as TeamMemberWithTeamSchema;
if (!tmb) { if (!tmb) {
return Promise.reject('member not exist'); return Promise.reject('member not exist');
} }
...@@ -27,7 +26,8 @@ async function getTeamMember(match: Record<string, any>): Promise<TeamItemType> ...@@ -27,7 +26,8 @@ async function getTeamMember(match: Record<string, any>): Promise<TeamItemType>
role: tmb.role, role: tmb.role,
status: tmb.status, status: tmb.status,
defaultTeam: tmb.defaultTeam, defaultTeam: tmb.defaultTeam,
canWrite: tmb.role !== TeamMemberRoleEnum.visitor canWrite: tmb.role !== TeamMemberRoleEnum.visitor,
lafAccount: tmb.teamId.lafAccount
}; };
} }
......
packages/service/support/user/team/teamSchema.ts
...@@ -35,6 +35,14 @@ const TeamSchema = new Schema({ ...@@ -35,6 +35,14 @@ const TeamSchema = new Schema({
lastWebsiteSyncTime: { lastWebsiteSyncTime: {
type: Date type: Date
} }
},
lafAccount: {
token: {
type: String
},
appid: {
type: String
}
} }
}); });
......
packages/web/components/common/Icon/constants.ts
...@@ -125,7 +125,9 @@ export const iconPaths = { ...@@ -125,7 +125,9 @@ export const iconPaths = {
'file/fill/manual': () => import('./icons/file/fill/manual.svg'), 'file/fill/manual': () => import('./icons/file/fill/manual.svg'),
'file/fill/markdown': () => import('./icons/file/fill/markdown.svg'), 'file/fill/markdown': () => import('./icons/file/fill/markdown.svg'),
'file/fill/pdf': () => import('./icons/file/fill/pdf.svg'), 'file/fill/pdf': () => import('./icons/file/fill/pdf.svg'),
'file/fill/ppt': () => import('./icons/file/fill/ppt.svg'),
'file/fill/txt': () => import('./icons/file/fill/txt.svg'), 'file/fill/txt': () => import('./icons/file/fill/txt.svg'),
'file/fill/xlsx': () => import('./icons/file/fill/xlsx.svg'),
'file/html': () => import('./icons/file/html.svg'), 'file/html': () => import('./icons/file/html.svg'),
'file/indexImport': () => import('./icons/file/indexImport.svg'), 'file/indexImport': () => import('./icons/file/indexImport.svg'),
'file/manualImport': () => import('./icons/file/manualImport.svg'), 'file/manualImport': () => import('./icons/file/manualImport.svg'),
......
packages/web/components/common/Icon/icons/file/fill/ppt.svg 0 → 100644
<svg t="1712108366314" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="1576"
width="128" height="128">
<path
d="M145.6 0C100.8 0 64 36.8 64 81.6v860.8C64 987.2 100.8 1024 145.6 1024h732.8c44.8 0 81.6-36.8 81.6-81.6V324.8L657.6 0h-512z"
fill="#E34221" p-id="1577"></path>
<path d="M960 326.4v16H755.2s-100.8-20.8-99.2-108.8c0 0 4.8 92.8 97.6 92.8H960z" fill="#DC3119" p-id="1578"></path>
<path d="M657.6 0v233.6c0 25.6 17.6 92.8 97.6 92.8H960L657.6 0z" fill="#FFFFFF" p-id="1579"></path>
<path
d="M304 784h-54.4v67.2c0 6.4-4.8 11.2-11.2 11.2-6.4 0-12.8-4.8-12.8-11.2V686.4c0-9.6 8-17.6 17.6-17.6H304c38.4 0 59.2 25.6 59.2 57.6S340.8 784 304 784z m-3.2-94.4h-51.2v73.6h51.2c22.4 0 38.4-16 38.4-36.8 0-22.4-16-36.8-38.4-36.8zM480 784h-54.4v67.2c0 6.4-4.8 11.2-11.2 11.2-6.4 0-11.2-4.8-11.2-11.2V686.4c0-9.6 6.4-17.6 16-17.6H480c38.4 0 59.2 25.6 59.2 57.6S518.4 784 480 784z m-3.2-94.4h-49.6v73.6h49.6c22.4 0 38.4-16 38.4-36.8 0-22.4-16-36.8-38.4-36.8z m225.6 0h-52.8v161.6c0 6.4-4.8 11.2-11.2 11.2-6.4 0-12.8-4.8-12.8-11.2V689.6h-51.2c-6.4 0-11.2-4.8-11.2-11.2 0-4.8 4.8-9.6 11.2-9.6h128c6.4 0 11.2 4.8 11.2 11.2 0 4.8-4.8 9.6-11.2 9.6z"
fill="#FFFFFF" p-id="1580"></path>
</svg>
\ No newline at end of file
packages/web/components/common/Icon/icons/file/fill/xlsx.svg 0 → 100644
<svg t="1712108380525" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="2654"
width="128" height="128">
<path
d="M145.6 0C100.8 0 64 36.8 64 81.6v860.8C64 987.2 100.8 1024 145.6 1024h732.8c44.8 0 81.6-36.8 81.6-81.6V324.8L657.6 0h-512z"
fill="#45B058" p-id="2655"></path>
<path
d="M374.4 862.4c-3.2 0-6.4-1.6-8-3.2l-59.2-80-60.8 80c-1.6 1.6-4.8 3.2-8 3.2-6.4 0-11.2-4.8-11.2-11.2 0-1.6 0-4.8 1.6-6.4l62.4-81.6-57.6-78.4c-1.6-1.6-3.2-3.2-3.2-6.4 0-4.8 4.8-11.2 11.2-11.2 4.8 0 8 1.6 9.6 4.8l56 73.6 54.4-73.6c1.6-3.2 4.8-4.8 8-4.8 6.4 0 12.8 4.8 12.8 11.2 0 3.2-1.6 4.8-1.6 6.4l-59.2 76.8 62.4 83.2c1.6 1.6 3.2 4.8 3.2 6.4 0 6.4-6.4 11.2-12.8 11.2z m160-1.6H448c-9.6 0-17.6-8-17.6-17.6V678.4c0-6.4 4.8-11.2 12.8-11.2 6.4 0 11.2 4.8 11.2 11.2v161.6h80c6.4 0 11.2 4.8 11.2 9.6 0 6.4-4.8 11.2-11.2 11.2z m112 3.2c-28.8 0-51.2-9.6-67.2-24-3.2-1.6-3.2-4.8-3.2-8 0-6.4 3.2-12.8 11.2-12.8 1.6 0 4.8 1.6 6.4 3.2 12.8 11.2 32 20.8 54.4 20.8 33.6 0 44.8-19.2 44.8-33.6 0-49.6-113.6-22.4-113.6-89.6 0-32 27.2-54.4 65.6-54.4 24 0 46.4 8 60.8 20.8 3.2 1.6 4.8 4.8 4.8 8 0 6.4-4.8 12.8-11.2 12.8-1.6 0-4.8-1.6-6.4-3.2-14.4-11.2-32-16-49.6-16-24 0-40 11.2-40 30.4 0 43.2 113.6 17.6 113.6 89.6 0 27.2-19.2 56-70.4 56z"
fill="#FFFFFF" p-id="2656"></path>
<path d="M960 326.4v16H755.2s-102.4-20.8-99.2-108.8c0 0 3.2 92.8 96 92.8h208z" fill="#349C42" p-id="2657"></path>
<path d="M656 0v233.6c0 25.6 19.2 92.8 99.2 92.8H960L656 0z" fill="#FFFFFF" p-id="2658"></path>
</svg>
\ No newline at end of file
packages/web/components/common/Input/HttpInput/Editor.tsx
...@@ -36,7 +36,7 @@ export default function Editor({ ...@@ -36,7 +36,7 @@ export default function Editor({
hasVariablePlugin?: boolean; hasVariablePlugin?: boolean;
hasDropDownPlugin?: boolean; hasDropDownPlugin?: boolean;
variables: EditorVariablePickerType[]; variables: EditorVariablePickerType[];
onChange?: (editorState: EditorState) => void; onChange?: (editorState: EditorState, editor: LexicalEditor) => void;
onBlur?: (editor: LexicalEditor) => void; onBlur?: (editor: LexicalEditor) => void;
value?: string; value?: string;
currentValue?: string; currentValue?: string;
...@@ -119,9 +119,9 @@ export default function Editor({ ...@@ -119,9 +119,9 @@ export default function Editor({
<HistoryPlugin /> <HistoryPlugin />
<FocusPlugin focus={focus} setFocus={setFocus} /> <FocusPlugin focus={focus} setFocus={setFocus} />
<OnChangePlugin <OnChangePlugin
onChange={(e) => { onChange={(editorState: EditorState, editor: LexicalEditor) => {
startSts(() => { startSts(() => {
onChange?.(e); onChange?.(editorState, editor);
}); });
}} }}
/> />
......
packages/web/components/common/Input/HttpInput/index.tsx
...@@ -32,15 +32,13 @@ const HttpInput = ({ ...@@ -32,15 +32,13 @@ const HttpInput = ({
const [, startSts] = useTransition(); const [, startSts] = useTransition();
const onChangeInput = useCallback((editorState: EditorState) => { const onChangeInput = useCallback((editorState: EditorState, editor: LexicalEditor) => {
const text = editorState.read(() => $getRoot().getTextContent()); const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
const formatValue = text.replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{'); onChange?.(text);
setCurrentValue(formatValue);
onChange?.(formatValue);
}, []); }, []);
const onBlurInput = useCallback((editor: LexicalEditor) => { const onBlurInput = useCallback((editor: LexicalEditor) => {
startSts(() => { startSts(() => {
const text = editorStateToText(editor).replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{'); const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
onBlur?.(text); onBlur?.(text);
}); });
}, []); }, []);
......
packages/web/components/common/MySelect/index.tsx
...@@ -7,10 +7,13 @@ import { ...@@ -7,10 +7,13 @@ import {
useDisclosure, useDisclosure,
MenuButton, MenuButton,
Box, Box,
css css,
Flex
} from '@chakra-ui/react'; } from '@chakra-ui/react';
import type { ButtonProps, MenuItemProps } from '@chakra-ui/react'; import type { ButtonProps, MenuItemProps } from '@chakra-ui/react';
import { ChevronDownIcon } from '@chakra-ui/icons'; import { ChevronDownIcon } from '@chakra-ui/icons';
import { useLoading } from '../../../hooks/useLoading';
import MyIcon from '../Icon';
export type SelectProps = ButtonProps & { export type SelectProps = ButtonProps & {
value?: string; value?: string;
...@@ -20,14 +23,16 @@ export type SelectProps = ButtonProps & { ...@@ -20,14 +23,16 @@ export type SelectProps = ButtonProps & {
label: string | React.ReactNode; label: string | React.ReactNode;
value: string; value: string;
}[]; }[];
isLoading?: boolean;
onchange?: (val: any) => void; onchange?: (val: any) => void;
}; };
const MySelect = ( const MySelect = (
{ placeholder, value, width = '100%', list, onchange, ...props }: SelectProps, { placeholder, value, width = '100%', list, onchange, isLoading = false, ...props }: SelectProps,
selectRef: any selectRef: any
) => { ) => {
const ref = useRef<HTMLButtonElement>(null); const ref = useRef<HTMLButtonElement>(null);
const { Loading } = useLoading();
const menuItemStyles: MenuItemProps = { const menuItemStyles: MenuItemProps = {
borderRadius: 'sm', borderRadius: 'sm',
py: 2, py: 2,
...@@ -78,7 +83,10 @@ const MySelect = ( ...@@ -78,7 +83,10 @@ const MySelect = (
: {})} : {})}
{...props} {...props}
> >
<Flex alignItems={'center'}>
{isLoading && <MyIcon mr={2} name={'common/loading'} w={'16px'} />}
{selectItem?.alias || selectItem?.label || placeholder} {selectItem?.alias || selectItem?.label || placeholder}
</Flex>
</MenuButton> </MenuButton>
<MenuList <MenuList
......
packages/web/components/common/Textarea/PromptEditor/index.tsx
import { Button, ModalBody, ModalFooter, useDisclosure } from '@chakra-ui/react'; import { Button, ModalBody, ModalFooter, useDisclosure } from '@chakra-ui/react';
import React, { useEffect, useState } from 'react'; import React from 'react';
import { editorStateToText } from './utils'; import { editorStateToText } from './utils';
import Editor from './Editor'; import Editor from './Editor';
import MyModal from '../../MyModal'; import MyModal from '../../MyModal';
import { useTranslation } from 'next-i18next'; import { useTranslation } from 'next-i18next';
import { $getRoot, EditorState, type LexicalEditor } from 'lexical'; import { EditorState, type LexicalEditor } from 'lexical';
import { EditorVariablePickerType } from './type.d'; import { EditorVariablePickerType } from './type.d';
import { useCallback, useTransition } from 'react'; import { useCallback, useTransition } from 'react';
...@@ -34,16 +34,12 @@ const PromptEditor = ({ ...@@ -34,16 +34,12 @@ const PromptEditor = ({
const { t } = useTranslation(); const { t } = useTranslation();
const onChangeInput = useCallback((editorState: EditorState, editor: LexicalEditor) => { const onChangeInput = useCallback((editorState: EditorState, editor: LexicalEditor) => {
const stringifiedEditorState = JSON.stringify(editorState.toJSON()); const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
const parsedEditorState = editor.parseEditorState(stringifiedEditorState); onChange?.(text);
const editorStateTextString = parsedEditorState.read(() => $getRoot().getTextContent());
const formatValue = editorStateTextString.replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
onChange?.(formatValue);
}, []); }, []);
const onBlurInput = useCallback((editor: LexicalEditor) => { const onBlurInput = useCallback((editor: LexicalEditor) => {
startSts(() => { startSts(() => {
const text = editorStateToText(editor).replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{'); const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
onBlur?.(text); onBlur?.(text);
}); });
}, []); }, []);
......
packages/web/components/common/Textarea/PromptEditor/plugins/OnBlurPlugin/index.tsx
import { useEffect } from 'react'; import { useEffect } from 'react';
import { BLUR_COMMAND, COMMAND_PRIORITY_EDITOR, LexicalEditor } from 'lexical'; import { BLUR_COMMAND, COMMAND_PRIORITY_EDITOR, LexicalEditor } from 'lexical';
import { mergeRegister } from '@lexical/utils';
import { useLexicalComposerContext } from '@lexical/react/LexicalComposerContext'; import { useLexicalComposerContext } from '@lexical/react/LexicalComposerContext';
export default function OnBlurPlugin({ onBlur }: { onBlur?: (editor: LexicalEditor) => void }) { export default function OnBlurPlugin({ onBlur }: { onBlur?: (editor: LexicalEditor) => void }) {
......
packages/web/components/common/Textarea/PromptEditor/utils.ts
...@@ -209,9 +209,8 @@ export function editorStateToText(editor: LexicalEditor) { ...@@ -209,9 +209,8 @@ export function editorStateToText(editor: LexicalEditor) {
const stringifiedEditorState = JSON.stringify(editor.getEditorState().toJSON()); const stringifiedEditorState = JSON.stringify(editor.getEditorState().toJSON());
const parsedEditorState = editor.parseEditorState(stringifiedEditorState); const parsedEditorState = editor.parseEditorState(stringifiedEditorState);
const editorStateTextString = parsedEditorState.read(() => $getRoot().getTextContent()); const editorStateTextString = parsedEditorState.read(() => $getRoot().getTextContent());
const compressedText = editorStateTextString.replace(/\n+/g, '\n\n');
return compressedText; return editorStateTextString;
} }
const varRegex = /\{\{([a-zA-Z_][a-zA-Z0-9_]*)\}\}/g; const varRegex = /\{\{([a-zA-Z_][a-zA-Z0-9_]*)\}\}/g;
......
pnpm-lock.yaml
This source diff could not be displayed because it is too large. You can view the blob instead.
projects/app/data/config.json
{ {
"feConfigs": {
"lafEnv": "https://laf.dev"
},
"systemEnv": { "systemEnv": {
"openapiPrefix": "fastgpt", "openapiPrefix": "fastgpt",
"vectorMaxProcess": 15, "vectorMaxProcess": 15,
......
projects/app/public/js/iframe.js
...@@ -23,7 +23,7 @@ function embedChatbot() { ...@@ -23,7 +23,7 @@ function embedChatbot() {
const ChatBtn = document.createElement('div'); const ChatBtn = document.createElement('div');
ChatBtn.id = chatBtnId; ChatBtn.id = chatBtnId;
ChatBtn.style.cssText = ChatBtn.style.cssText =
'position: fixed; bottom: 1rem; right: 1rem; width: 40px; height: 40px; cursor: pointer; z-index: 2147483647; transition: 0;'; 'position: fixed; bottom: 30px; right: 60px; width: 40px; height: 40px; cursor: pointer; z-index: 2147483647; transition: 0;';
// btn icon // btn icon
const ChatBtnDiv = document.createElement('img'); const ChatBtnDiv = document.createElement('img');
...@@ -39,7 +39,7 @@ function embedChatbot() { ...@@ -39,7 +39,7 @@ function embedChatbot() {
iframe.id = chatWindowId; iframe.id = chatWindowId;
iframe.src = botSrc; iframe.src = botSrc;
iframe.style.cssText = iframe.style.cssText =
'border: none; position: fixed; flex-direction: column; justify-content: space-between; box-shadow: rgba(150, 150, 150, 0.2) 0px 10px 30px 0px, rgba(150, 150, 150, 0.2) 0px 0px 0px 1px; bottom: 4rem; right: 1rem; width: 24rem; height: 40rem; max-width: 90vw; max-height: 85vh; border-radius: 0.75rem; display: flex; z-index: 2147483647; overflow: hidden; left: unset; background-color: #F3F4F6;'; 'border: none; position: fixed; flex-direction: column; justify-content: space-between; box-shadow: rgba(150, 150, 150, 0.2) 0px 10px 30px 0px, rgba(150, 150, 150, 0.2) 0px 0px 0px 1px; bottom: 80px; right: 60px; width: 375px; height: 667px; max-width: 90vw; max-height: 85vh; border-radius: 0.75rem; display: flex; z-index: 2147483647; overflow: hidden; left: unset; background-color: #F3F4F6;';
iframe.style.visibility = defaultOpen ? 'unset' : 'hidden'; iframe.style.visibility = defaultOpen ? 'unset' : 'hidden';
document.body.appendChild(iframe); document.body.appendChild(iframe);
......
projects/app/public/locales/en/common.json
...@@ -148,6 +148,7 @@ ...@@ -148,6 +148,7 @@
"Status": "Status", "Status": "Status",
"Submit failed": "Submit failed", "Submit failed": "Submit failed",
"Submit success": "Update Success", "Submit success": "Update Success",
"Sync success": "",
"System version": "System version", "System version": "System version",
"Team": "Team", "Team": "Team",
"Team Tags Set": "Team Tags", "Team Tags Set": "Team Tags",
...@@ -819,6 +820,7 @@ ...@@ -819,6 +820,7 @@
"Http request props": "Request props", "Http request props": "Request props",
"Http request settings": "Request settings", "Http request settings": "Request settings",
"Input Type": "Input Type", "Input Type": "Input Type",
"Laf sync params": "Sync params",
"Plugin output must connect": "Custom outputs must all be connected", "Plugin output must connect": "Custom outputs must all be connected",
"QueryExtension": { "QueryExtension": {
"placeholder": "Questions about python introduction and usage, etc. The current conversation is related to the game GTA5.", "placeholder": "Questions about python introduction and usage, etc. The current conversation is related to the game GTA5.",
...@@ -912,6 +914,9 @@ ...@@ -912,6 +914,9 @@
"target": "Target Data", "target": "Target Data",
"textarea": "Textarea" "textarea": "Textarea"
}, },
"laf": {
"Select laf function": ""
},
"output": { "output": {
"Add Output": "Add Output", "Add Output": "Add Output",
"Output Number": "Output: {{length}}", "Output Number": "Output: {{length}}",
...@@ -1226,19 +1231,26 @@ ...@@ -1226,19 +1231,26 @@
"Set Public": "Set to public" "Set Public": "Set to public"
}, },
"plugin": { "plugin": {
"App": "Choose App",
"Auth Header Prefix": "Auth header prefix", "Auth Header Prefix": "Auth header prefix",
"Auth Method": "Auth method", "Auth Method": "Auth method",
"Auth Type": "Auth Type", "Auth Type": "Auth Type",
"Confirm Delete": "Confirm to delete the plugin?", "Confirm Delete": "Confirm to delete the plugin?",
"Create Your Plugin": "Create Plugin", "Create Your Plugin": "Create Plugin",
"Currentapp": "CurrentApp",
"Custom Plugin": "Custom plugin", "Custom Plugin": "Custom plugin",
"Description": "Description", "Description": "Description",
"Edit Http Plugin": "Edit HTTP plugin", "Edit Http Plugin": "Edit HTTP plugin",
"Enter Env": "Enter laf environment",
"Enter PAT": "Please enter personal access token (PAT)",
"Func": "Choose Function",
"Get Plugin Module Detail Failed": "Get plugin detail failed", "Get Plugin Module Detail Failed": "Get plugin detail failed",
"HTTP Plugin": "HTTP plugin", "HTTP Plugin": "HTTP plugin",
"Import Plugin": "Import HTTP plugin", "Import Plugin": "Import HTTP plugin",
"Import from URL": "Import from URL. https://xxxx", "Import from URL": "Import from URL. https://xxxx",
"Intro": "Plugin Intro", "Intro": "Plugin Intro",
"Invalid Appid": "Invalid appid",
"Invalid Env": "Invalid Env",
"Invalid Schema": "Invalid Schema", "Invalid Schema": "Invalid Schema",
"Invalid URL": "Invalid URL", "Invalid URL": "Invalid URL",
"Key": "Key", "Key": "Key",
...@@ -1248,16 +1260,20 @@ ...@@ -1248,16 +1260,20 @@
"No Intro": "This plugin is not introduced", "No Intro": "This plugin is not introduced",
"None": "None", "None": "None",
"Path": "Path", "Path": "Path",
"Please bind laf accout first": "Please bind laf accout first",
"Plugin List": "Plugin list", "Plugin List": "Plugin list",
"Plugin Module": "Plugin", "Plugin Module": "Plugin",
"Privacy Agreement": "privacy agreement", "Privacy Agreement": "privacy agreement",
"Search plugin": "Search plugins", "Search plugin": "Search plugins",
"Set Name": "Plugin Name", "Set Name": "Plugin Name",
"Synchronous app": "Sync App",
"Synchronous version": "Sync Version", "Synchronous version": "Sync Version",
"To Edit Plugin": "To Edit", "To Edit Plugin": "To Edit",
"Update Your Plugin": "Update Plugin", "Update Your Plugin": "Update Plugin",
"Value": "Value", "Value": "Value",
"path": "" "go to laf": "go to laf",
"path": "",
"update params": "update params"
}, },
"support": { "support": {
"account": { "account": {
...@@ -1303,6 +1319,9 @@ ...@@ -1303,6 +1319,9 @@
"user": { "user": {
"AI point standard": "AI points price", "AI point standard": "AI points price",
"Avatar": "Avatar", "Avatar": "Avatar",
"Go laf env": "Click to go to laf to get PAT certificate.",
"Laf account course": "Check out the laf account binding tutorial.",
"Laf account intro": "After binding your laf account, you will be able to use the laf module in your workflow to write code online.",
"Need to login": "Please log in first", "Need to login": "Please log in first",
"Price": "Price", "Price": "Price",
"User self info": "My info", "User self info": "My info",
...@@ -1495,10 +1514,13 @@ ...@@ -1495,10 +1514,13 @@
"Bill Detail": "Bill Detail", "Bill Detail": "Bill Detail",
"Change": "Change", "Change": "Change",
"Copy invite url": "Copy invitation link", "Copy invite url": "Copy invitation link",
"Current laf Env": "Current laf Env",
"Edit name": "Click to modify nickname", "Edit name": "Click to modify nickname",
"Invite Url": "Invite Url", "Invite Url": "Invite Url",
"Invite url tip": "Friends who register through this link will be permanently bound to you, and you will get a certain balance reward when they recharge. In addition, when friends register with their mobile phone number, you will get 5 yuan reward immediately.", "Invite url tip": "Friends who register through this link will be permanently bound to you, and you will get a certain balance reward when they recharge. In addition, when friends register with their mobile phone number, you will get 5 yuan reward immediately.",
"Laf Account Setting": "laf account setting",
"Language": "Language", "Language": "Language",
"Learn More": "Learn More",
"Member Name": "Name", "Member Name": "Name",
"Notice": "Notice", "Notice": "Notice",
"Old password is error": "Old password is error", "Old password is error": "Old password is error",
...@@ -1513,6 +1535,7 @@ ...@@ -1513,6 +1535,7 @@
"Promotion rate tip": "You will be rewarded with a percentage of the balance when your friends top up", "Promotion rate tip": "You will be rewarded with a percentage of the balance when your friends top up",
"Recharge Record": "Recharge", "Recharge Record": "Recharge",
"Replace": "Replace", "Replace": "Replace",
"Set Laf Account Failed": "set laf accout failed",
"Set OpenAI Account Failed": "Set OpenAI account failed", "Set OpenAI Account Failed": "Set OpenAI account failed",
"Sign Out": "Sign Out", "Sign Out": "Sign Out",
"Source": "Source", "Source": "Source",
......
projects/app/public/locales/zh/common.json
...@@ -148,6 +148,7 @@ ...@@ -148,6 +148,7 @@
"Status": "状态", "Status": "状态",
"Submit failed": "提交失败", "Submit failed": "提交失败",
"Submit success": "提交成功", "Submit success": "提交成功",
"Sync success": "同步成功",
"System version": "系统版本", "System version": "系统版本",
"Team": "团队", "Team": "团队",
"Team Tags Set": "标签", "Team Tags Set": "标签",
...@@ -821,6 +822,7 @@ ...@@ -821,6 +822,7 @@
"Http request props": "请求参数", "Http request props": "请求参数",
"Http request settings": "请求配置", "Http request settings": "请求配置",
"Input Type": "输入类型", "Input Type": "输入类型",
"Laf sync params": "同步参数",
"Plugin output must connect": "自定义输出必须全部连接", "Plugin output must connect": "自定义输出必须全部连接",
"QueryExtension": { "QueryExtension": {
"placeholder": "例如:\n关于 python 的介绍和使用等问题。\n当前对话与游戏《GTA5》有关。", "placeholder": "例如:\n关于 python 的介绍和使用等问题。\n当前对话与游戏《GTA5》有关。",
...@@ -914,6 +916,9 @@ ...@@ -914,6 +916,9 @@
"target": "外部数据", "target": "外部数据",
"textarea": "段落输入" "textarea": "段落输入"
}, },
"laf": {
"Select laf function": "选择laf函数"
},
"output": { "output": {
"Add Output": "添加出参", "Add Output": "添加出参",
"Output Number": "出参: {{length}}", "Output Number": "出参: {{length}}",
...@@ -1228,19 +1233,26 @@ ...@@ -1228,19 +1233,26 @@
"Set Public": "设为团队可用" "Set Public": "设为团队可用"
}, },
"plugin": { "plugin": {
"App": "选择应用",
"Auth Header Prefix": "鉴权头部前缀", "Auth Header Prefix": "鉴权头部前缀",
"Auth Method": "鉴权方法", "Auth Method": "鉴权方法",
"Auth Type": "鉴权类型", "Auth Type": "鉴权类型",
"Confirm Delete": "确认删除该插件？", "Confirm Delete": "确认删除该插件？",
"Create Your Plugin": "创建你的插件", "Create Your Plugin": "创建你的插件",
"Currentapp": "当前应用",
"Custom Plugin": "自定义插件", "Custom Plugin": "自定义插件",
"Description": "描述", "Description": "描述",
"Edit Http Plugin": "编辑 HTTP 插件", "Edit Http Plugin": "编辑 HTTP 插件",
"Enter Env": "输入 laf 环境",
"Enter PAT": "请输入访问凭证（PAT）",
"Func": "选择函数",
"Get Plugin Module Detail Failed": "获取插件信息异常", "Get Plugin Module Detail Failed": "获取插件信息异常",
"HTTP Plugin": "HTTP 插件", "HTTP Plugin": "HTTP 插件",
"Import Plugin": "导入 HTTP 插件", "Import Plugin": "导入 HTTP 插件",
"Import from URL": "从URL导入。https://xxxx", "Import from URL": "从URL导入。https://xxxx",
"Intro": "插件介绍", "Intro": "插件介绍",
"Invalid Appid": "appid 无效",
"Invalid Env": "laf 环境错误",
"Invalid Schema": "Schema 无效", "Invalid Schema": "Schema 无效",
"Invalid URL": "URL 无效", "Invalid URL": "URL 无效",
"Key": "键", "Key": "键",
...@@ -1250,16 +1262,20 @@ ...@@ -1250,16 +1262,20 @@
"No Intro": "这个插件没有介绍~", "No Intro": "这个插件没有介绍~",
"None": "无", "None": "无",
"Path": "路径", "Path": "路径",
"Please bind laf accout first": "请先绑定 laf 账号",
"Plugin List": "插件列表", "Plugin List": "插件列表",
"Plugin Module": "插件模块", "Plugin Module": "插件模块",
"Privacy Agreement": "隐私协议", "Privacy Agreement": "隐私协议",
"Search plugin": "搜索插件", "Search plugin": "搜索插件",
"Set Name": "给插件取个名字", "Set Name": "给插件取个名字",
"Synchronous app": "同步应用",
"Synchronous version": "同步版本", "Synchronous version": "同步版本",
"To Edit Plugin": "去编辑", "To Edit Plugin": "去编辑",
"Update Your Plugin": "更新插件", "Update Your Plugin": "更新插件",
"Value": "值", "Value": "值",
"path": "" "go to laf": "去编写",
"path": "",
"update params": "更新参数"
}, },
"support": { "support": {
"account": { "account": {
...@@ -1305,6 +1321,9 @@ ...@@ -1305,6 +1321,9 @@
"user": { "user": {
"AI point standard": "AI积分标准", "AI point standard": "AI积分标准",
"Avatar": "头像", "Avatar": "头像",
"Go laf env": "点击前往 laf 获取 PAT 凭证。",
"Laf account course": "查看绑定 laf 账号教程。",
"Laf account intro": "绑定你的laf账号后，你将可以在工作流中使用 laf 模块，实现在线编写代码。",
"Need to login": "请先登录", "Need to login": "请先登录",
"Price": "计费标准", "Price": "计费标准",
"User self info": "个人信息", "User self info": "个人信息",
...@@ -1497,10 +1516,13 @@ ...@@ -1497,10 +1516,13 @@
"Bill Detail": "账单详情", "Bill Detail": "账单详情",
"Change": "变更", "Change": "变更",
"Copy invite url": "复制邀请链接", "Copy invite url": "复制邀请链接",
"Current laf Env": "当前 laf 环境",
"Edit name": "点击修改昵称", "Edit name": "点击修改昵称",
"Invite Url": "邀请链接", "Invite Url": "邀请链接",
"Invite url tip": "通过该链接注册的好友将永久与你绑定，其充值时你会获得一定余额奖励。\n此外，好友使用手机号注册时，你将立即获得 5 元奖励。\n奖励会发送到您的默认团队中。", "Invite url tip": "通过该链接注册的好友将永久与你绑定，其充值时你会获得一定余额奖励。\n此外，好友使用手机号注册时，你将立即获得 5 元奖励。\n奖励会发送到您的默认团队中。",
"Laf Account Setting": "laf 账号配置",
"Language": "语言", "Language": "语言",
"Learn More": "查看文档",
"Member Name": "昵称", "Member Name": "昵称",
"Notice": "通知", "Notice": "通知",
"Old password is error": "旧密码错误", "Old password is error": "旧密码错误",
...@@ -1515,6 +1537,7 @@ ...@@ -1515,6 +1537,7 @@
"Promotion rate tip": "好友充值时你将获得一定比例的余额奖励", "Promotion rate tip": "好友充值时你将获得一定比例的余额奖励",
"Recharge Record": "支付记录", "Recharge Record": "支付记录",
"Replace": "更换", "Replace": "更换",
"Set Laf Account Failed": "设置 laf 账号异常",
"Set OpenAI Account Failed": "设置 OpenAI 账号异常", "Set OpenAI Account Failed": "设置 OpenAI 账号异常",
"Sign Out": "登出", "Sign Out": "登出",
"Source": "来源", "Source": "来源",
......
projects/app/src/components/core/module/Flow/FlowProvider.tsx
...@@ -214,9 +214,6 @@ export const FlowProvider = ({ ...@@ -214,9 +214,6 @@ export const FlowProvider = ({
if (source?.flowType === FlowNodeTypeEnum.classifyQuestion && !type) { if (source?.flowType === FlowNodeTypeEnum.classifyQuestion && !type) {
return ModuleIOValueTypeEnum.boolean; return ModuleIOValueTypeEnum.boolean;
} }
if (source?.flowType === FlowNodeTypeEnum.tools) {
return ModuleIOValueTypeEnum.tools;
}
if (source?.flowType === FlowNodeTypeEnum.pluginInput) { if (source?.flowType === FlowNodeTypeEnum.pluginInput) {
return source?.inputs.find((input) => input.key === connect.sourceHandle)?.valueType; return source?.inputs.find((input) => input.key === connect.sourceHandle)?.valueType;
} }
......
projects/app/src/components/core/module/Flow/ModuleTemplateList.tsx
...@@ -52,6 +52,7 @@ const ModuleTemplateList = ({ isOpen, onClose }: ModuleTemplateListProps) => { ...@@ -52,6 +52,7 @@ const ModuleTemplateList = ({ isOpen, onClose }: ModuleTemplateListProps) => {
const router = useRouter(); const router = useRouter();
const [currentParent, setCurrentParent] = useState<RenderListProps['currentParent']>(); const [currentParent, setCurrentParent] = useState<RenderListProps['currentParent']>();
const [searchKey, setSearchKey] = useState(''); const [searchKey, setSearchKey] = useState('');
const { feConfigs } = useSystemStore();
const { const {
basicNodeTemplates, basicNodeTemplates,
...@@ -64,7 +65,12 @@ const ModuleTemplateList = ({ isOpen, onClose }: ModuleTemplateListProps) => { ...@@ -64,7 +65,12 @@ const ModuleTemplateList = ({ isOpen, onClose }: ModuleTemplateListProps) => {
const templates = useMemo(() => { const templates = useMemo(() => {
const map = { const map = {
[TemplateTypeEnum.basic]: basicNodeTemplates, [TemplateTypeEnum.basic]: basicNodeTemplates.filter((item) => {
if (item.flowType === FlowNodeTypeEnum.lafModule && !feConfigs.lafEnv) {
return false;
}
return true;
}),
[TemplateTypeEnum.systemPlugin]: systemNodeTemplates, [TemplateTypeEnum.systemPlugin]: systemNodeTemplates,
[TemplateTypeEnum.teamPlugin]: teamPluginNodeTemplates.filter((item) => [TemplateTypeEnum.teamPlugin]: teamPluginNodeTemplates.filter((item) =>
searchKey ? item.pluginType !== PluginTypeEnum.folder : true searchKey ? item.pluginType !== PluginTypeEnum.folder : true
......
projects/app/src/components/core/module/Flow/components/nodes/NodeHttp/index.tsx
...@@ -88,7 +88,7 @@ enum TabEnum { ...@@ -88,7 +88,7 @@ enum TabEnum {
headers = 'headers', headers = 'headers',
body = 'body' body = 'body'
} }
type PropsArrType = { export type PropsArrType = {
key: string; key: string;
type: string; type: string;
value: string; value: string;
...@@ -245,7 +245,7 @@ const RenderHttpMethodAndUrl = React.memo(function RenderHttpMethodAndUrl({ ...@@ -245,7 +245,7 @@ const RenderHttpMethodAndUrl = React.memo(function RenderHttpMethodAndUrl({
); );
}); });
function RenderHttpProps({ export function RenderHttpProps({
moduleId, moduleId,
inputs inputs
}: { }: {
......
projects/app/src/components/core/module/Flow/components/nodes/NodeLaf.tsx 0 → 100644
import React, { useCallback, useMemo } from 'react';
import { NodeProps } from 'reactflow';
import NodeCard from '../render/NodeCard';
import { FlowModuleItemType } from '@fastgpt/global/core/module/type.d';
import Container from '../modules/Container';
import { Box, Button, Center, Flex, useDisclosure } from '@chakra-ui/react';
import { ModuleIOValueTypeEnum, ModuleInputKeyEnum } from '@fastgpt/global/core/module/constants';
import { onChangeNode, useFlowProviderStore } from '../../FlowProvider';
import { useTranslation } from 'next-i18next';
import { getLafAppDetail } from '@/web/support/laf/api';
import MySelect from '@fastgpt/web/components/common/MySelect';
import { getApiSchemaByUrl } from '@/web/core/plugin/api';
import { str2OpenApiSchema } from '@fastgpt/global/core/plugin/httpPlugin/utils';
import { useUserStore } from '@/web/support/user/useUserStore';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import { ChevronRightIcon } from '@chakra-ui/icons';
import { useQuery } from '@tanstack/react-query';
import dynamic from 'next/dynamic';
import { FlowNodeInputTypeEnum } from '@fastgpt/global/core/module/node/constant';
import { useToast } from '@fastgpt/web/hooks/useToast';
import Divider from '../modules/Divider';
import RenderToolInput from '../render/RenderToolInput';
import RenderInput from '../render/RenderInput';
import RenderOutput from '../render/RenderOutput';
import { getErrText } from '@fastgpt/global/common/error/utils';
const LafAccountModal = dynamic(() => import('@/components/support/laf/LafAccountModal'));
const NodeLaf = (props: NodeProps<FlowModuleItemType>) => {
const { t } = useTranslation();
const { toast } = useToast();
const { feConfigs } = useSystemStore();
const { data, selected } = props;
const { moduleId, inputs } = data;
const requestUrl = inputs.find((item) => item.key === ModuleInputKeyEnum.httpReqUrl);
const { userInfo } = useUserStore();
const token = userInfo?.team.lafAccount?.token;
const appid = userInfo?.team.lafAccount?.appid;
// not config laf
if (!token || !appid) {
return (
<NodeCard minW={'350px'} selected={selected} {...data}>
<ConfigLaf />
</NodeCard>
);
}
const { data: lafData, isLoading: isLoadingFunctions } = useQuery(
['getLafFunctionList'],
async () => {
// load laf app detail
const appDetail = await getLafAppDetail(appid);
// load laf app functions
const schemaUrl = `https://${appDetail?.domain.domain}/_/api-docs?token=${appDetail?.openapi_token}`;
const schema = await getApiSchemaByUrl(schemaUrl);
const openApiSchema = await str2OpenApiSchema(JSON.stringify(schema));
const filterPostSchema = openApiSchema.pathData.filter((item) => item.method === 'post');
return {
lafApp: appDetail,
lafFunctions: filterPostSchema.map((item) => ({
...item,
requestUrl: `https://${appDetail?.domain.domain}${item.path}`
}))
};
},
{
onError(err) {
toast({
status: 'error',
title: getErrText(err, '获取Laf函数列表失败')
});
}
}
);
const lafFunctionSelectList = useMemo(
() =>
lafData?.lafFunctions.map((item) => ({
label: item.description ? `${item.name} (${item.description})` : item.name,
value: item.requestUrl
})) || [],
[lafData?.lafFunctions]
);
const selectedFunction = useMemo(
() => lafFunctionSelectList.find((item) => item.value === requestUrl?.value)?.value,
[lafFunctionSelectList, requestUrl?.value]
);
const onSyncParams = useCallback(() => {
const lafFunction = lafData?.lafFunctions.find((item) => item.requestUrl === selectedFunction);
if (!lafFunction) return;
const bodyParams =
lafFunction?.request?.content?.['application/json']?.schema?.properties || {};
const requiredParams =
lafFunction?.request?.content?.['application/json']?.schema?.required || [];
const allParams = [
...Object.keys(bodyParams).map((key) => ({
name: key,
desc: bodyParams[key].description,
required: requiredParams?.includes(key) || false,
value: `{{${key}}}`,
type: 'string'
}))
].filter((item) => !inputs.find((input) => input.key === item.name));
// add params
allParams.forEach((param) => {
onChangeNode({
moduleId,
type: 'addInput',
key: param.name,
value: {
key: param.name,
valueType: ModuleIOValueTypeEnum.string,
label: param.name,
type: FlowNodeInputTypeEnum.target,
required: param.required,
description: param.desc || '',
toolDescription: param.desc || '未设置参数描述',
edit: true,
editField: {
key: true,
name: true,
description: true,
required: true,
dataType: true,
inputType: true,
isToolInput: true
},
connected: false
}
});
});
toast({
status: 'success',
title: t('common.Sync success')
});
}, [inputs, lafData?.lafFunctions, moduleId, selectedFunction, t, toast]);
return (
<NodeCard minW={'350px'} selected={selected} {...data}>
<Container>
{/* select function */}
<MySelect
isLoading={isLoadingFunctions}
list={lafFunctionSelectList}
placeholder={t('core.module.laf.Select laf function')}
onchange={(e) => {
onChangeNode({
moduleId,
type: 'updateInput',
key: ModuleInputKeyEnum.httpReqUrl,
value: {
...requestUrl,
value: e
}
});
}}
value={selectedFunction}
/>
{/* auto set params and go to edit */}
{!!selectedFunction && (
<Flex justifyContent={'flex-end'} mt={2} gap={2}>
{/* <Button variant={'whiteBase'} size={'sm'} onClick={onSyncParams}>
{t('core.module.Laf sync params')}
</Button> */}
<Button
variant={'grayBase'}
size={'sm'}
onClick={() => {
const lafFunction = lafData?.lafFunctions.find(
(item) => item.requestUrl === selectedFunction
);
if (!lafFunction) return;
const url = `${feConfigs.lafEnv}/app/${lafData?.lafApp?.appid}/function${lafFunction?.path}`;
window.open(url, '_blank');
}}
>
{t('plugin.go to laf')}
</Button>
</Flex>
)}
</Container>
{!!selectedFunction && <RenderIO {...props} />}
</NodeCard>
);
};
export default React.memo(NodeLaf);
const ConfigLaf = () => {
const { t } = useTranslation();
const { userInfo } = useUserStore();
const { feConfigs } = useSystemStore();
const {
isOpen: isOpenLafConfig,
onOpen: onOpenLafConfig,
onClose: onCloseLafConfig
} = useDisclosure();
return !!feConfigs?.lafEnv ? (
<Center minH={150}>
<Button onClick={onOpenLafConfig} variant={'whitePrimary'}>
{t('plugin.Please bind laf accout first')} <ChevronRightIcon />
</Button>
{isOpenLafConfig && feConfigs?.lafEnv && (
<LafAccountModal defaultData={userInfo?.team.lafAccount} onClose={onCloseLafConfig} />
)}
</Center>
) : (
<Box>系统未配置Laf环境</Box>
);
};
const RenderIO = ({ data, selected }: NodeProps<FlowModuleItemType>) => {
const { t } = useTranslation();
const { moduleId, inputs, outputs } = data;
const { splitToolInputs, hasToolNode } = useFlowProviderStore();
const { commonInputs, toolInputs } = splitToolInputs(inputs, moduleId);
return (
<>
{hasToolNode && (
<>
<Divider text={t('core.module.tool.Tool input')} />
<Container>
<RenderToolInput moduleId={moduleId} inputs={toolInputs} canEdit />
</Container>
</>
)}
<>
<Divider text={t('common.Input')} />
<Container>
<Box mb={3}>自定义Body参数</Box>
<RenderInput moduleId={moduleId} flowInputList={commonInputs} />
</Container>
</>
<>
<Divider text={t('common.Output')} />
<Container>
<RenderOutput moduleId={moduleId} flowOutputList={outputs} />
</Container>
</>
</>
);
};
projects/app/src/components/core/module/Flow/components/render/NodeCard.tsx
...@@ -164,7 +164,7 @@ const NodeCard = (props: Props) => { ...@@ -164,7 +164,7 @@ const NodeCard = (props: Props) => {
top={'-20px'} top={'-20px'}
right={0} right={0}
transform={'translateX(90%)'} transform={'translateX(90%)'}
pl={'17px'} pl={'20px'}
pr={'10px'} pr={'10px'}
pb={'20px'} pb={'20px'}
pt={'20px'} pt={'20px'}
......
projects/app/src/components/core/module/Flow/components/render/RenderToolInput/constants.ts
import { FlowNodeInputItemType } from '@fastgpt/global/core/module/node/type'; import { FlowNodeInputItemType } from '@fastgpt/global/core/module/node/type';
import { FlowNodeInputTypeEnum } from '@fastgpt/global/core/module/node/constant'; import { FlowNodeInputTypeEnum } from '@fastgpt/global/core/module/node/constant';
import { ModuleIOValueTypeEnum } from '@fastgpt/global/core/module/constants';
export const defaultEditFormData: FlowNodeInputItemType = { export const defaultEditFormData: FlowNodeInputItemType = {
valueType: 'string', valueType: 'string',
type: FlowNodeInputTypeEnum.hidden, type: FlowNodeInputTypeEnum.target,
key: '', key: '',
label: '', label: '',
toolDescription: '', toolDescription: '',
required: true required: true,
edit: true,
editField: {
key: true,
description: true,
dataType: true
},
defaultEditField: {
label: '',
key: '',
description: '',
inputType: FlowNodeInputTypeEnum.target,
valueType: ModuleIOValueTypeEnum.string
}
}; };
projects/app/src/components/core/module/Flow/index.tsx
...@@ -44,7 +44,8 @@ const nodeTypes: Record<`${FlowNodeTypeEnum}`, any> = { ...@@ -44,7 +44,8 @@ const nodeTypes: Record<`${FlowNodeTypeEnum}`, any> = {
[FlowNodeTypeEnum.tools]: dynamic(() => import('./components/nodes/NodeTools')), [FlowNodeTypeEnum.tools]: dynamic(() => import('./components/nodes/NodeTools')),
[FlowNodeTypeEnum.stopTool]: (data: NodeProps<FlowModuleItemType>) => ( [FlowNodeTypeEnum.stopTool]: (data: NodeProps<FlowModuleItemType>) => (
<NodeSimple {...data} minW={'100px'} maxW={'300px'} /> <NodeSimple {...data} minW={'100px'} maxW={'300px'} />
) ),
[FlowNodeTypeEnum.lafModule]: dynamic(() => import('./components/nodes/NodeLaf'))
}; };
const edgeTypes = { const edgeTypes = {
[EDGE_TYPE]: ButtonEdge [EDGE_TYPE]: ButtonEdge
......
projects/app/src/components/support/laf/LafAccountModal.tsx 0 → 100644
import React, { useCallback } from 'react';
import { ModalBody, Box, Flex, Input, ModalFooter, Button, Link } from '@chakra-ui/react';
import MyModal from '@fastgpt/web/components/common/MyModal';
import { useTranslation } from 'next-i18next';
import { useForm } from 'react-hook-form';
import { useRequest } from '@fastgpt/web/hooks/useRequest';
import { useQuery } from '@tanstack/react-query';
import MySelect from '@fastgpt/web/components/common/MySelect';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import { useToast } from '@fastgpt/web/hooks/useToast';
import { putUpdateTeam } from '@/web/support/user/team/api';
import { useUserStore } from '@/web/support/user/useUserStore';
import type { LafAccountType } from '@fastgpt/global/support/user/team/type.d';
import { postLafPat2Token, getLafApplications } from '@/web/support/laf/api';
import { getErrText } from '@fastgpt/global/common/error/utils';
const LafAccountModal = ({
defaultData = {
token: '',
appid: ''
},
onClose
}: {
defaultData?: LafAccountType;
onClose: () => void;
}) => {
const { t } = useTranslation();
const { register, handleSubmit, setValue, getValues, watch, reset } = useForm({
defaultValues: {
...defaultData,
pat: ''
}
});
const lafToken = watch('token');
const pat = watch('pat');
const appid = watch('appid');
const { feConfigs } = useSystemStore();
const { toast } = useToast();
const { userInfo, initUserInfo } = useUserStore();
const onResetForm = useCallback(() => {
reset({
token: '',
appid: '',
pat: ''
});
}, [reset]);
const { mutate: authLafPat, isLoading: isPatLoading } = useRequest({
mutationFn: async (pat) => {
const token = await postLafPat2Token(pat);
setValue('token', token);
},
errorToast: t('plugin.Invalid Env')
});
const { data: appListData = [] } = useQuery(
['appList', lafToken],
() => {
return getLafApplications(lafToken);
},
{
enabled: !!lafToken,
onSuccess: (data) => {
if (!getValues('appid') && data.length > 0) {
setValue('appid', data[0].appid);
}
},
onError: (err) => {
onResetForm();
toast({
title: getErrText(err, '获取应用列表失败'),
status: 'error'
});
}
}
);
const { mutate: onSubmit, isLoading: isUpdating } = useRequest({
mutationFn: async (data: LafAccountType) => {
if (!userInfo?.team.teamId) return;
return putUpdateTeam({
teamId: userInfo?.team.teamId,
lafAccount: data
});
},
onSuccess() {
initUserInfo();
onClose();
},
successToast: t('common.Update Success'),
errorToast: t('common.Update Failed')
});
return (
<MyModal isOpen iconSrc="/imgs/module/laf.png" title={t('user.Laf Account Setting')}>
<ModalBody>
<Box fontSize={'sm'} color={'myGray.500'}>
<Box>{t('support.user.Laf account intro')}</Box>
<Box textDecoration={'underline'}>
<Link href={`https://doc.laf.run/zh/`} isExternal>
{t('support.user.Laf account course')}
</Link>
</Box>
<Box>
<Link textDecoration={'underline'} href={`${feConfigs.lafEnv}/`} isExternal>
{t('support.user.Go laf env')}
</Link>
</Box>
</Box>
<Flex alignItems={'center'} mt={5}>
<Box flex={'0 0 70px'}>PAT:</Box>
{!lafToken ? (
<>
<Input
flex={'1 0 0'}
size={'sm'}
{...register('pat')}
placeholder={t('plugin.Enter PAT')}
/>
<Button
ml={2}
variant={'whitePrimary'}
isDisabled={!pat}
onClick={() => {
authLafPat(pat);
}}
isLoading={isPatLoading}
>
验证
</Button>
</>
) : (
<Button
variant={'whitePrimary'}
onClick={() => {
onResetForm();
putUpdateTeam({
teamId: userInfo?.team.teamId || '',
lafAccount: { token: '', appid: '' }
});
}}
>
已验证，点击取消绑定
</Button>
)}
</Flex>
{!!lafToken && (
<Flex alignItems={'center'} mt={5}>
<Box flex={'0 0 70px'}>{t('plugin.Currentapp')}</Box>
<MySelect
minW={'200px'}
list={
appListData
.filter((app) => app.state === 'Running')
.map((app) => ({
label: `${app.name}`,
value: app.appid
})) || []
}
placeholder={t('plugin.App')}
value={watch('appid')}
onchange={(e) => {
setValue('appid', e);
}}
{...(register('appid'), { required: true })}
/>
</Flex>
)}
</ModalBody>
<ModalFooter>
<Button mr={3} variant={'whiteBase'} onClick={onClose}>
{t('common.Close')}
</Button>
<Button isLoading={isUpdating} onClick={handleSubmit((data) => onSubmit(data))}>
{t('common.Update')}
</Button>
</ModalFooter>
</MyModal>
);
};
export default LafAccountModal;
projects/app/src/pages/account/components/Info.tsx
...@@ -8,7 +8,8 @@ import { ...@@ -8,7 +8,8 @@ import {
Input, Input,
Link, Link,
Progress, Progress,
Grid Grid,
Image
} from '@chakra-ui/react'; } from '@chakra-ui/react';
import { useForm } from 'react-hook-form'; import { useForm } from 'react-hook-form';
import { UserUpdateParams } from '@/types/user'; import { UserUpdateParams } from '@/types/user';
...@@ -41,12 +42,14 @@ import { ...@@ -41,12 +42,14 @@ import {
} from '@/web/support/wallet/sub/constants'; } from '@/web/support/wallet/sub/constants';
import StandardPlanContentList from '@/components/support/wallet/StandardPlanContentList'; import StandardPlanContentList from '@/components/support/wallet/StandardPlanContentList';
import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
const StandDetailModal = dynamic(() => import('./standardDetailModal')); const StandDetailModal = dynamic(() => import('./standardDetailModal'));
const TeamMenu = dynamic(() => import('@/components/support/user/team/TeamMenu')); const TeamMenu = dynamic(() => import('@/components/support/user/team/TeamMenu'));
const PayModal = dynamic(() => import('./PayModal')); const PayModal = dynamic(() => import('./PayModal'));
const UpdatePswModal = dynamic(() => import('./UpdatePswModal')); const UpdatePswModal = dynamic(() => import('./UpdatePswModal'));
const OpenAIAccountModal = dynamic(() => import('./OpenAIAccountModal')); const OpenAIAccountModal = dynamic(() => import('./OpenAIAccountModal'));
const LafAccountModal = dynamic(() => import('@/components/support/laf/LafAccountModal'));
const CommunityModal = dynamic(() => import('@/components/CommunityModal')); const CommunityModal = dynamic(() => import('@/components/CommunityModal'));
const Account = () => { const Account = () => {
...@@ -518,7 +521,7 @@ const Other = () => { ...@@ -518,7 +521,7 @@ const Other = () => {
const { reset } = useForm<UserUpdateParams>({ const { reset } = useForm<UserUpdateParams>({
defaultValues: userInfo as UserType defaultValues: userInfo as UserType
}); });
const { isOpen: isOpenLaf, onClose: onCloseLaf, onOpen: onOpenLaf } = useDisclosure();
const { isOpen: isOpenOpenai, onClose: onCloseOpenai, onOpen: onOpenOpenai } = useDisclosure(); const { isOpen: isOpenOpenai, onClose: onCloseOpenai, onOpen: onOpenOpenai } = useDisclosure();
const { isOpen: isOpenConcat, onClose: onCloseConcat, onOpen: onOpenConcat } = useDisclosure(); const { isOpen: isOpenConcat, onClose: onCloseConcat, onOpen: onOpenConcat } = useDisclosure();
...@@ -537,7 +540,6 @@ const Other = () => { ...@@ -537,7 +540,6 @@ const Other = () => {
}, },
[reset, toast, updateUserInfo] [reset, toast, updateUserInfo]
); );
return ( return (
<Box> <Box>
<Grid gridGap={4} mt={3}> <Grid gridGap={4} mt={3}>
...@@ -582,6 +584,32 @@ const Other = () => { ...@@ -582,6 +584,32 @@ const Other = () => {
</Box> </Box>
</Link> </Link>
{feConfigs?.lafEnv && userInfo?.team.role === TeamMemberRoleEnum.owner && (
<Flex
bg={'white'}
py={4}
px={6}
border={theme.borders.sm}
borderWidth={'1.5px'}
borderRadius={'md'}
alignItems={'center'}
cursor={'pointer'}
userSelect={'none'}
onClick={onOpenLaf}
>
<Image src="/imgs/module/laf.png" w={'18px'} alt="laf" />
<Box ml={2} flex={1}>
laf 账号
</Box>
<Box
w={'9px'}
h={'9px'}
borderRadius={'50%'}
bg={userInfo?.team.lafAccount?.token ? '#67c13b' : 'myGray.500'}
/>
</Flex>
)}
{feConfigs?.show_openai_account && ( {feConfigs?.show_openai_account && (
<Flex <Flex
bg={'white'} bg={'white'}
...@@ -620,6 +648,9 @@ const Other = () => { ...@@ -620,6 +648,9 @@ const Other = () => {
)} )}
</Grid> </Grid>
{isOpenLaf && userInfo && (
<LafAccountModal defaultData={userInfo?.team.lafAccount} onClose={onCloseLaf} />
)}
{isOpenOpenai && userInfo && ( {isOpenOpenai && userInfo && (
<OpenAIAccountModal <OpenAIAccountModal
defaultData={userInfo?.openaiAccount} defaultData={userInfo?.openaiAccount}
......
projects/app/src/pages/api/admin/clearInvalidData.ts
...@@ -2,12 +2,14 @@ import type { NextApiRequest, NextApiResponse } from 'next'; ...@@ -2,12 +2,14 @@ import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo'; import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common'; import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { checkFiles } from '../timerTask/dataset/checkInValidDatasetFiles';
import { addHours } from 'date-fns'; import { addHours } from 'date-fns';
import { checkInvalidCollection } from '../timerTask/dataset/checkInvalidMongoCollection';
import { checkInvalidVector } from '../timerTask/dataset/checkInvalidVector';
import { MongoImage } from '@fastgpt/service/common/file/image/schema'; import { MongoImage } from '@fastgpt/service/common/file/image/schema';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema'; import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import {
checkInvalidDatasetFiles,
checkInvalidDatasetData,
checkInvalidVector
} from '@/service/common/system/cronTask';
let deleteImageAmount = 0; let deleteImageAmount = 0;
async function checkInvalidImg(start: Date, end: Date, limit = 50) { async function checkInvalidImg(start: Date, end: Date, limit = 50) {
...@@ -60,11 +62,12 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse) ...@@ -60,11 +62,12 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
(async () => { (async () => {
try { try {
console.log('执行脏数据清理任务'); console.log('执行脏数据清理任务');
const end = addHours(new Date(), -1); // 360天 ~ 2小时前
const end = addHours(new Date(), -2);
const start = addHours(new Date(), -360 * 24); const start = addHours(new Date(), -360 * 24);
await checkFiles(start, end); await checkInvalidDatasetFiles(start, end);
await checkInvalidImg(start, end); await checkInvalidImg(start, end);
await checkInvalidCollection(start, end); await checkInvalidDatasetData(start, end);
await checkInvalidVector(start, end); await checkInvalidVector(start, end);
console.log('执行脏数据清理任务完毕'); console.log('执行脏数据清理任务完毕');
} catch (error) { } catch (error) {
......
projects/app/src/pages/api/admin/initv469.ts deleted 100644 → 0
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { MongoUsage } from '@fastgpt/service/support/wallet/usage/schema';
import { connectionMongo } from '@fastgpt/service/common/mongo';
import { checkFiles } from '../timerTask/dataset/checkInValidDatasetFiles';
import { addHours } from 'date-fns';
import { checkInvalidCollection } from '../timerTask/dataset/checkInvalidMongoCollection';
import { checkInvalidVector } from '../timerTask/dataset/checkInvalidVector';
import { MongoImage } from '@fastgpt/service/common/file/image/schema';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
let deleteImageAmount = 0;
export async function checkInvalidImg(start: Date, end: Date, limit = 50) {
const images = await MongoImage.find(
{
createTime: {
$gte: start,
$lte: end
},
'metadata.relatedId': { $exists: true }
},
'_id teamId metadata'
);
console.log('total images', images.length);
let index = 0;
for await (const image of images) {
try {
// 1. 检测是否有对应的集合
const collection = await MongoDatasetCollection.findOne(
{
teamId: image.teamId,
'metadata.relatedImgId': image.metadata?.relatedId
},
'_id'
);
if (!collection) {
await image.deleteOne();
deleteImageAmount++;
}
index++;
index % 100 === 0 && console.log(index);
} catch (error) {
console.log(error);
}
}
console.log(`检测完成，共删除 ${deleteImageAmount} 个无效图片`);
}
/* pg 中的数据搬到 mongo dataset.datas 中，并做映射 */
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
await authCert({ req, authRoot: true });
// 检查 usage 是否有记录
const totalUsages = await MongoUsage.countDocuments();
if (totalUsages === 0) {
// 重命名 bills 集合成 usages
await connectionMongo.connection.db.renameCollection('bills', 'usages', {
// 强制
dropTarget: true
});
}
(async () => {
try {
console.log('执行脏数据清理任务');
const end = addHours(new Date(), -1);
const start = addHours(new Date(), -360 * 24);
await checkFiles(start, end);
await checkInvalidImg(start, end);
await checkInvalidCollection(start, end);
await checkInvalidVector(start, end);
console.log('执行脏数据清理任务完毕');
} catch (error) {
console.log('执行脏数据清理任务出错了');
}
})();
jsonRes(res, {
message: 'success'
});
} catch (error) {
console.log(error);
jsonRes(res, {
code: 500,
error
});
}
}
projects/app/src/pages/api/common/file/previewContent.ts
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
import type { NextApiRequest, NextApiResponse } from 'next'; import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo'; import { connectToDatabase } from '@/service/mongo';
import { readFileContent } from '@fastgpt/service/common/file/gridfs/controller'; import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller';
import { authFile } from '@fastgpt/service/support/permission/auth/file'; import { authFile } from '@fastgpt/service/support/permission/auth/file';
import { BucketNameEnum } from '@fastgpt/global/common/file/constants'; import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
...@@ -19,7 +19,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse< ...@@ -19,7 +19,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
const { teamId } = await authFile({ req, authToken: true, fileId }); const { teamId } = await authFile({ req, authToken: true, fileId });
const { rawText } = await readFileContent({ const { rawText } = await readFileContentFromMongo({
teamId, teamId,
bucketName: BucketNameEnum.dataset, bucketName: BucketNameEnum.dataset,
fileId, fileId,
......
projects/app/src/pages/api/common/system/getInitData.ts
...@@ -90,6 +90,7 @@ export async function initSystemConfig() { ...@@ -90,6 +90,7 @@ export async function initSystemConfig() {
// get config from database // get config from database
const config: FastGPTConfigFileType = { const config: FastGPTConfigFileType = {
feConfigs: { feConfigs: {
...fileRes?.feConfigs,
...defaultFeConfigs, ...defaultFeConfigs,
...(dbConfig.feConfigs || {}), ...(dbConfig.feConfigs || {}),
isPlus: !!FastGPTProUrl isPlus: !!FastGPTProUrl
......
projects/app/src/pages/api/core/dataset/collection/create/csvTable.ts
import type { NextApiRequest, NextApiResponse } from 'next'; import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo'; import { connectToDatabase } from '@/service/mongo';
import { readFileContent } from '@fastgpt/service/common/file/gridfs/controller'; import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset'; import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { FileIdCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api'; import { FileIdCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api';
import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller'; import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller';
...@@ -36,7 +36,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse< ...@@ -36,7 +36,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
}); });
// 1. read file // 1. read file
const { rawText, filename } = await readFileContent({ const { rawText, filename } = await readFileContentFromMongo({
teamId, teamId,
bucketName: BucketNameEnum.dataset, bucketName: BucketNameEnum.dataset,
fileId fileId
......
projects/app/src/pages/api/core/dataset/collection/create/file.ts
...@@ -3,7 +3,7 @@ import { jsonRes } from '@fastgpt/service/common/response'; ...@@ -3,7 +3,7 @@ import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo'; import { connectToDatabase } from '@/service/mongo';
import { import {
delFileByFileIdList, delFileByFileIdList,
readFileContent readFileContentFromMongo
} from '@fastgpt/service/common/file/gridfs/controller'; } from '@fastgpt/service/common/file/gridfs/controller';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset'; import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { FileIdCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api'; import { FileIdCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api';
...@@ -47,7 +47,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse< ...@@ -47,7 +47,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
}); });
// 1. read file // 1. read file
const { rawText, filename } = await readFileContent({ const { rawText, filename } = await readFileContentFromMongo({
teamId, teamId,
bucketName: BucketNameEnum.dataset, bucketName: BucketNameEnum.dataset,
fileId fileId
......
projects/app/src/pages/api/core/dataset/file/getPreviewChunks.ts
...@@ -4,7 +4,7 @@ import { connectToDatabase } from '@/service/mongo'; ...@@ -4,7 +4,7 @@ import { connectToDatabase } from '@/service/mongo';
import { BucketNameEnum } from '@fastgpt/global/common/file/constants'; import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { authFile } from '@fastgpt/service/support/permission/auth/file'; import { authFile } from '@fastgpt/service/support/permission/auth/file';
import { PostPreviewFilesChunksProps } from '@/global/core/dataset/api'; import { PostPreviewFilesChunksProps } from '@/global/core/dataset/api';
import { readFileContent } from '@fastgpt/service/common/file/gridfs/controller'; import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller';
import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter'; import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter';
import { ImportDataSourceEnum } from '@fastgpt/global/core/dataset/constants'; import { ImportDataSourceEnum } from '@fastgpt/global/core/dataset/constants';
import { parseCsvTable2Chunks } from '@fastgpt/service/core/dataset/training/utils'; import { parseCsvTable2Chunks } from '@fastgpt/service/core/dataset/training/utils';
...@@ -28,7 +28,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse< ...@@ -28,7 +28,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
const { file, teamId } = await authFile({ req, authToken: true, fileId: sourceId }); const { file, teamId } = await authFile({ req, authToken: true, fileId: sourceId });
const fileId = String(file._id); const fileId = String(file._id);
const { rawText } = await readFileContent({ const { rawText } = await readFileContentFromMongo({
teamId, teamId,
bucketName: BucketNameEnum.dataset, bucketName: BucketNameEnum.dataset,
fileId, fileId,
...@@ -53,7 +53,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse< ...@@ -53,7 +53,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
if (type === ImportDataSourceEnum.csvTable) { if (type === ImportDataSourceEnum.csvTable) {
const { file, teamId } = await authFile({ req, authToken: true, fileId: sourceId }); const { file, teamId } = await authFile({ req, authToken: true, fileId: sourceId });
const fileId = String(file._id); const fileId = String(file._id);
const { rawText } = await readFileContent({ const { rawText } = await readFileContentFromMongo({
teamId, teamId,
bucketName: BucketNameEnum.dataset, bucketName: BucketNameEnum.dataset,
fileId, fileId,
......
projects/app/src/pages/api/lafApi/[...path].ts 0 → 100644
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { request } from 'https';
import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';
import url from 'url';
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { path = [], ...query } = req.query as any;
const queryStr = new URLSearchParams(query).toString();
const requestPath = queryStr
? `/${path?.join('/')}?${new URLSearchParams(query).toString()}`
: `/${path?.join('/')}`;
if (!requestPath) {
throw new Error('url is empty');
}
const lafEnv = global.feConfigs?.lafEnv;
if (!lafEnv) {
throw new Error('lafEnv is empty');
}
const parsedUrl = url.parse(lafEnv);
delete req.headers?.cookie;
delete req.headers?.host;
delete req.headers?.origin;
const requestResult = request({
protocol: parsedUrl.protocol,
hostname: parsedUrl.hostname,
port: parsedUrl.port,
path: requestPath,
method: req.method,
headers: req.headers,
timeout: 30000
});
req.pipe(requestResult);
requestResult.on('response', (response) => {
Object.keys(response.headers).forEach((key) => {
// @ts-ignore
res.setHeader(key, response.headers[key]);
});
response.statusCode && res.writeHead(response.statusCode);
response.pipe(res);
});
requestResult.on('error', (e) => {
res.send(e);
res.end();
});
} catch (error) {
jsonRes(res, {
code: 500,
error
});
}
}
export const config = {
api: {
bodyParser: false
}
};
projects/app/src/pages/api/support/user/account/update.ts
...@@ -11,7 +11,7 @@ import { MongoTeamMember } from '@fastgpt/service/support/user/team/teamMemberSc ...@@ -11,7 +11,7 @@ import { MongoTeamMember } from '@fastgpt/service/support/user/team/teamMemberSc
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) { export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try { try {
await connectToDatabase(); await connectToDatabase();
const { avatar, timezone, openaiAccount } = req.body as UserUpdateParams; const { avatar, timezone, openaiAccount, lafAccount } = req.body as UserUpdateParams;
const { tmbId } = await authCert({ req, authToken: true }); const { tmbId } = await authCert({ req, authToken: true });
const tmb = await MongoTeamMember.findById(tmbId); const tmb = await MongoTeamMember.findById(tmbId);
...@@ -47,7 +47,8 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse< ...@@ -47,7 +47,8 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
{ {
...(avatar && { avatar }), ...(avatar && { avatar }),
...(timezone && { timezone }), ...(timezone && { timezone }),
openaiAccount: openaiAccount?.key ? openaiAccount : null openaiAccount: openaiAccount?.key ? openaiAccount : null,
lafAccount: lafAccount?.token ? lafAccount : null
} }
); );
......
projects/app/src/pages/api/timerTask/dataset/checkInValidDatasetFiles.ts deleted 100644 → 0
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import {
delFileByFileIdList,
getGFSCollection
} from '@fastgpt/service/common/file/gridfs/controller';
import { addLog } from '@fastgpt/service/common/system/log';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import { addHours } from 'date-fns';
/*
check dataset.files data. If there is no match in dataset.collections, delete it
可能异常情况
1. 上传了文件，未成功创建集合
*/
let deleteFileAmount = 0;
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
const { startHour = 24, endHour = 1 } = req.body as {
startHour?: number;
endHour?: number;
limit?: number;
};
await authCert({ req, authRoot: true });
await connectToDatabase();
// start: now - maxDay, end: now - 3 day
const start = addHours(new Date(), -startHour);
const end = addHours(new Date(), -endHour);
deleteFileAmount = 0;
console.log(start, end);
await checkFiles(start, end);
jsonRes(res, {
data: deleteFileAmount,
message: 'success'
});
} catch (error) {
addLog.error(`check valid dataset files error`, error);
jsonRes(res, {
code: 500,
error
});
}
}
export async function checkFiles(start: Date, end: Date) {
const collection = getGFSCollection('dataset');
const where = {
uploadDate: { $gte: start, $lte: end }
};
// 1. get all file _id
const files = await collection
.find(where, {
projection: {
metadata: 1,
_id: 1
}
})
.toArray();
console.log('total files', files.length);
let index = 0;
for await (const file of files) {
try {
// 2. find fileId in dataset.collections
const hasCollection = await MongoDatasetCollection.countDocuments({
teamId: file.metadata.teamId,
fileId: file._id
});
// 3. if not found, delete file
if (hasCollection === 0) {
await delFileByFileIdList({ bucketName: 'dataset', fileIdList: [String(file._id)] });
console.log('delete file', file._id);
deleteFileAmount++;
}
index++;
index % 100 === 0 && console.log(index);
} catch (error) {
console.log(error);
}
}
console.log(`检测完成，共删除 ${deleteFileAmount} 个无效文件`);
}
projects/app/src/pages/api/timerTask/dataset/checkInvalidVector.ts deleted 100644 → 0
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { addLog } from '@fastgpt/service/common/system/log';
import {
deleteDatasetDataVector,
getVectorDataByTime
} from '@fastgpt/service/common/vectorStore/controller';
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
import { addHours } from 'date-fns';
/*
检测无效的 Vector 数据.
异常情况：
1. 插入数据时，vector成功，mongo失败
2. 更新数据，也会有插入 vector
*/
let deletedVectorAmount = 0;
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
const { startHour = 5, endHour = 1 } = req.body as { startHour?: number; endHour?: number };
await authCert({ req, authRoot: true });
await connectToDatabase();
// start: now - maxDay, end: now - endHour
const start = addHours(new Date(), -startHour);
const end = addHours(new Date(), -endHour);
deletedVectorAmount = 0;
await checkInvalidVector(start, end);
jsonRes(res, {
data: deletedVectorAmount,
message: 'success'
});
} catch (error) {
addLog.error(`check Invalid user error`, error);
jsonRes(res, {
code: 500,
error
});
}
}
export async function checkInvalidVector(start: Date, end: Date) {
// 1. get all vector data
const rows = await getVectorDataByTime(start, end);
console.log('total data', rows.length);
let index = 0;
for await (const item of rows) {
if (!item.teamId || !item.datasetId || !item.id) {
console.log('error data', item);
continue;
}
try {
// 2. find dataset.data
const hasData = await MongoDatasetData.countDocuments({
teamId: item.teamId,
datasetId: item.datasetId,
'indexes.dataId': item.id
});
// 3. if not found, delete vector
if (hasData === 0) {
await deleteDatasetDataVector({
teamId: item.teamId,
id: item.id
});
console.log('delete vector data', item.id);
deletedVectorAmount++;
}
index++;
index % 100 === 0 && console.log(index);
} catch (error) {
console.log(error);
}
}
console.log(`检测完成，共删除 ${deletedVectorAmount} 个无效 向量 数据`);
}
projects/app/src/pages/dataset/detail/components/Import/components/RenderFiles.tsx
...@@ -117,3 +117,5 @@ export const RenderUploadFiles = ({ ...@@ -117,3 +117,5 @@ export const RenderUploadFiles = ({
</> </>
) : null; ) : null;
}; };
export default RenderUploadFiles;
projects/app/src/pages/dataset/detail/components/InputDataModal.tsx
...@@ -249,7 +249,7 @@ const InputDataModal = ({ ...@@ -249,7 +249,7 @@ const InputDataModal = ({
return openConfirm(onDeleteData)(); return openConfirm(onDeleteData)();
} }
if (e === TabEnum.doc) { if (e === TabEnum.doc) {
return window.open(getDocPath('/docs/course/datasetengine'), '_blank'); return window.open(getDocPath('/docs/course/dataset_engine'), '_blank');
} }
setCurrentTab(e); setCurrentTab(e);
}} }}
......
projects/app/src/service/common/system/cron.ts
import { setCron } from '@fastgpt/service/common/system/cron'; import { setCron } from '@fastgpt/service/common/system/cron';
import { startTrainingQueue } from '@/service/core/dataset/training/utils'; import { startTrainingQueue } from '@/service/core/dataset/training/utils';
import { clearTmpUploadFiles } from '@fastgpt/service/common/file/utils'; import { clearTmpUploadFiles } from '@fastgpt/service/common/file/utils';
import { checkInvalidDatasetFiles, checkInvalidDatasetData, checkInvalidVector } from './cronTask';
import { checkTimerLock } from '@fastgpt/service/common/system/timerLock/utils';
import { TimerIdEnum } from '@fastgpt/service/common/system/timerLock/constants';
import { addHours } from 'date-fns';
export const startCron = () => { const setTrainingQueueCron = () => {
setTrainingQueueCron();
setClearTmpUploadFilesCron();
};
export const setTrainingQueueCron = () => {
setCron('*/1 * * * *', () => { setCron('*/1 * * * *', () => {
startTrainingQueue(); startTrainingQueue();
}); });
}; };
export const setClearTmpUploadFilesCron = () => { const setClearTmpUploadFilesCron = () => {
clearTmpUploadFiles();
// Clear tmp upload files every ten minutes // Clear tmp upload files every ten minutes
setCron('*/10 * * * *', () => { setCron('*/10 * * * *', () => {
clearTmpUploadFiles(); clearTmpUploadFiles();
}); });
}; };
const clearInvalidDataCron = () => {
setCron('0 */1 * * *', async () => {
if (
await checkTimerLock({
timerId: TimerIdEnum.checkInValidDatasetFiles,
lockMinuted: 59
})
) {
checkInvalidDatasetFiles(addHours(new Date(), 2), addHours(new Date(), 6));
}
});
setCron('10 */1 * * *', async () => {
if (
await checkTimerLock({
timerId: TimerIdEnum.checkInvalidDatasetData,
lockMinuted: 59
})
) {
checkInvalidDatasetData(addHours(new Date(), 2), addHours(new Date(), 6));
}
});
setCron('30 */1 * * *', async () => {
if (
await checkTimerLock({
timerId: TimerIdEnum.checkInvalidVector,
lockMinuted: 59
})
) {
checkInvalidVector(addHours(new Date(), 2), addHours(new Date(), 6));
}
});
};
export const startCron = () => {
setTrainingQueueCron();
setClearTmpUploadFilesCron();
clearInvalidDataCron();
};
projects/app/src/pages/api/timerTask/dataset/checkInvalidMongoCollection.ts → projects/app/src/service/common/system/cronTask.ts
import type { NextApiRequest, NextApiResponse } from 'next'; import {
import { jsonRes } from '@fastgpt/service/common/response'; delFileByFileIdList,
import { connectToDatabase } from '@/service/mongo'; getGFSCollection
import { authCert } from '@fastgpt/service/support/permission/auth/common'; } from '@fastgpt/service/common/file/gridfs/controller';
import { addLog } from '@fastgpt/service/common/system/log'; import { addLog } from '@fastgpt/service/common/system/log';
import { deleteDatasetDataVector } from '@fastgpt/service/common/vectorStore/controller'; import {
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema'; deleteDatasetDataVector,
import { addHours } from 'date-fns'; getVectorDataByTime
} from '@fastgpt/service/common/vectorStore/controller';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema'; import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
import { MongoDatasetTraining } from '@fastgpt/service/core/dataset/training/schema'; import { MongoDatasetTraining } from '@fastgpt/service/core/dataset/training/schema';
/* /*
检测无效的 Mongo 数据 check dataset.files data. If there is no match in dataset.collections, delete it
异常情况： 可能异常情况
1. 训练过程删除知识库，可能导致还会有新的数据插入，导致无效。 1. 上传了文件，未成功创建集合
*/ */
export async function checkInvalidDatasetFiles(start: Date, end: Date) {
let deleteFileAmount = 0;
const collection = getGFSCollection('dataset');
const where = {
uploadDate: { $gte: start, $lte: end }
};
let deleteAmount = 0; // 1. get all file _id
export default async function handler(req: NextApiRequest, res: NextApiResponse) { const files = await collection
try { .find(where, {
const { startHour = 3, endHour = 1 } = req.body as { startHour?: number; endHour?: number }; projection: {
await authCert({ req, authRoot: true }); metadata: 1,
await connectToDatabase(); _id: 1
}
// start: now - maxDay, end: now - endHour })
const start = addHours(new Date(), -startHour); .toArray();
const end = addHours(new Date(), -endHour); addLog.info(`Clear invalid dataset files, total files: ${files.length}`);
deleteAmount = 0;
await checkInvalidCollection(start, end);
jsonRes(res, { let index = 0;
data: deleteAmount, for await (const file of files) {
message: 'success' try {
// 2. find fileId in dataset.collections
const hasCollection = await MongoDatasetCollection.countDocuments({
teamId: file.metadata.teamId,
fileId: file._id
}); });
} catch (error) {
addLog.error(`check Invalid user error`, error);
jsonRes(res, { // 3. if not found, delete file
code: 500, if (hasCollection === 0) {
error await delFileByFileIdList({ bucketName: 'dataset', fileIdList: [String(file._id)] });
}); console.log('delete file', file._id);
deleteFileAmount++;
}
index++;
index % 100 === 0 && console.log(index);
} catch (error) {
console.log(error);
} }
}
addLog.info(`Clear invalid dataset files finish, remove ${deleteFileAmount} files`);
} }
export async function checkInvalidCollection(start: Date, end: Date) { /*
检测无效的 Mongo 数据
异常情况：
1. 训练过程删除知识库，可能导致还会有新的数据继续插入，导致无效。
*/
export async function checkInvalidDatasetData(start: Date, end: Date) {
// 1. 获取时间范围的所有data // 1. 获取时间范围的所有data
const rows = await MongoDatasetData.find( const rows = await MongoDatasetData.find(
{ {
...@@ -64,7 +84,7 @@ export async function checkInvalidCollection(start: Date, end: Date) { ...@@ -64,7 +84,7 @@ export async function checkInvalidCollection(start: Date, end: Date) {
} }
} }
const list = Array.from(map.values()); const list = Array.from(map.values());
console.log('total collections', list.length); addLog.info(`Clear invalid dataset data, total collections: ${list.length}`);
let index = 0; let index = 0;
for await (const item of list) { for await (const item of list) {
...@@ -94,3 +114,44 @@ export async function checkInvalidCollection(start: Date, end: Date) { ...@@ -94,3 +114,44 @@ export async function checkInvalidCollection(start: Date, end: Date) {
console.log(++index); console.log(++index);
} }
} }
export async function checkInvalidVector(start: Date, end: Date) {
let deletedVectorAmount = 0;
// 1. get all vector data
const rows = await getVectorDataByTime(start, end);
addLog.info(`Clear invalid vector, total vector data: ${rows.length}`);
let index = 0;
for await (const item of rows) {
if (!item.teamId || !item.datasetId || !item.id) {
addLog.error('error data', item);
continue;
}
try {
// 2. find dataset.data
const hasData = await MongoDatasetData.countDocuments({
teamId: item.teamId,
datasetId: item.datasetId,
'indexes.dataId': item.id
});
// 3. if not found, delete vector
if (hasData === 0) {
await deleteDatasetDataVector({
teamId: item.teamId,
id: item.id
});
console.log('delete vector data', item.id);
deletedVectorAmount++;
}
index++;
index % 100 === 0 && console.log(index);
} catch (error) {
console.log(error);
}
}
addLog.info(`Clear invalid vector finish, remove ${deletedVectorAmount} data`);
}
projects/app/src/service/core/dataset/data/controller.ts
...@@ -44,8 +44,7 @@ export async function insertData2Dataset({ ...@@ -44,8 +44,7 @@ export async function insertData2Dataset({
indexes = indexes =
Array.isArray(indexes) && indexes.length > 0 Array.isArray(indexes) && indexes.length > 0
? indexes.map((index) => ({ ? indexes.map((index) => ({
// @ts-ignore text: index.text,
...index.toObject(),
dataId: undefined, dataId: undefined,
defaultIndex: index.text.trim() === qaStr defaultIndex: index.text.trim() === qaStr
})) }))
......
projects/app/src/types/user.d.ts
import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants'; import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants';
import type { UserModelSchema } from '@fastgpt/global/support/user/type'; import type { UserModelSchema } from '@fastgpt/global/support/user/type';
import { LafAccountType } from '@fastgpt/global/support/user/team/type.d';
export interface UserUpdateParams { export interface UserUpdateParams {
balance?: number; balance?: number;
avatar?: string; avatar?: string;
timezone?: string; timezone?: string;
openaiAccount?: UserModelSchema['openaiAccount']; openaiAccount?: UserModelSchema['openaiAccount'];
lafAccount?: LafAccountType;
} }
projects/app/src/web/common/api/lafRequest.ts 0 → 100644
import axios, {
Method,
InternalAxiosRequestConfig,
AxiosResponse,
AxiosProgressEvent
} from 'axios';
import { useUserStore } from '@/web/support/user/useUserStore';
interface ConfigType {
headers?: { [key: string]: string };
timeout?: number;
onUploadProgress?: (progressEvent: AxiosProgressEvent) => void;
cancelToken?: AbortController;
maxQuantity?: number;
}
interface ResponseDataType {
error: string | null;
data: any;
}
const maxQuantityMap: Record<
string,
{
amount: number;
sign: AbortController;
}
> = {};
function requestStart({ url, maxQuantity }: { url: string; maxQuantity?: number }) {
if (!maxQuantity) return;
const item = maxQuantityMap[url];
if (item) {
if (item.amount >= maxQuantity && item.sign) {
item.sign.abort();
delete maxQuantityMap[url];
}
} else {
maxQuantityMap[url] = {
amount: 1,
sign: new AbortController()
};
}
}
function requestFinish({ url }: { url: string }) {
const item = maxQuantityMap[url];
if (item) {
item.amount--;
if (item.amount <= 0) {
delete maxQuantityMap[url];
}
}
}
/**
* 请求开始
*/
function startInterceptors(config: InternalAxiosRequestConfig): InternalAxiosRequestConfig {
if (config.headers && !config.headers.Authorization) {
config.headers.Authorization = `Bearer ${useUserStore.getState().userInfo?.team?.lafAccount?.token || ''}`;
}
return config;
}
/**
* 请求成功,检查请求头
*/
function responseSuccess(response: AxiosResponse<ResponseDataType>) {
return response;
}
/**
* 响应数据检查
*/
function checkRes(data: ResponseDataType) {
if (data === undefined) {
console.log('error->', data, 'data is empty');
return Promise.reject('服务器异常');
} else if (data.error) {
return responseError(data.error);
}
return data.data;
}
/**
* 响应错误
*/
function responseError(err: any) {
console.log('error->', '请求错误', err);
if (!err) {
return Promise.reject({ message: '未知错误' });
}
if (typeof err === 'string') {
return Promise.reject({ message: err });
}
if (err?.response?.data) {
return Promise.reject(err?.response?.data);
}
return Promise.reject(err);
}
/* 创建请求实例 */
const instance = axios.create({
timeout: 60000, // 超时时间
headers: {
'content-type': 'application/json'
}
});
/* 请求拦截 */
instance.interceptors.request.use(startInterceptors, (err) => Promise.reject(err));
/* 响应拦截 */
instance.interceptors.response.use(responseSuccess, (err) => Promise.reject(err));
function request(
url: string,
data: any,
{ cancelToken, maxQuantity, ...config }: ConfigType,
method: Method
): any {
/* 去空 */
for (const key in data) {
if (data[key] === null || data[key] === undefined) {
delete data[key];
}
}
requestStart({ url, maxQuantity });
return instance
.request({
baseURL: '/api/lafApi',
url,
method,
data: ['POST', 'PUT'].includes(method) ? data : null,
params: !['POST', 'PUT'].includes(method) ? data : null,
signal: cancelToken?.signal,
...config // 用户自定义配置，可以覆盖前面的配置
})
.then((res) => checkRes(res.data))
.catch((err) => responseError(err))
.finally(() => requestFinish({ url }));
}
/**
* api请求方式
* @param {String} url
* @param {Any} params
* @param {Object} config
* @returns
*/
export function GET<T = undefined>(url: string, params = {}, config: ConfigType = {}): Promise<T> {
return request(url, params, config, 'GET');
}
export function POST<T = undefined>(url: string, data = {}, config: ConfigType = {}): Promise<T> {
return request(url, data, config, 'POST');
}
export function PUT<T = undefined>(url: string, data = {}, config: ConfigType = {}): Promise<T> {
return request(url, data, config, 'PUT');
}
export function DELETE<T = undefined>(url: string, data = {}, config: ConfigType = {}): Promise<T> {
return request(url, data, config, 'DELETE');
}
projects/app/src/web/support/laf/api.ts 0 → 100644
import { GET, POST, PUT } from '@/web/common/api/lafRequest';
export const postLafPat2Token = (pat: string) => POST<string>(`/v1/auth/pat2token`, { pat });
export const getLafApplications = (token: string) =>
GET<
{
appid: string;
name: string;
state: 'Running' | 'Failed' | 'Stopped';
}[]
>(
`/v1/applications`,
{},
{
headers: {
Authorization: `Bearer ${token}`
}
}
);
export const getLafAppDetail = (appid: string) =>
GET<{
appid: string;
name: string;
openapi_token: string;
domain: {
_id: string;
appid: string;
domain: string;
state: string;
phase: string;
};
}>(`/v1/applications/${appid}`);
python/reranker/bge-reranker-base/README.md
...@@ -38,7 +38,7 @@ ...@@ -38,7 +38,7 @@
**镜像和端口** **镜像和端口**
+ 镜像名: `luanshaotong/reranker:v0.2` + 镜像名: `registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2`
+ 端口号: 6006 + 端口号: 6006
``` ```
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment