Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 62e87551 by Archer Committed by GitHub
Options

New dpcs structure and dataset i18n (#551) 

* perf: check balance

* md

* lock way

* i18n

* docs

* doc

* i18n

* update doc

* feat: one link sync

* feat: one link sync

* feat: one link sync

* feat: one link sync

* feat: one link sync

* feat: one link sync

* feat: one link sync
parent c3ae38df
Showing with 966 additions and 471 deletions
README.md
......@@ -100,10 +100,10 @@ https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409b
由于需要部署数据库，部署完后需要等待 2~4 分钟才能正常访问。默认用了最低配置，首次访问时会有些慢。
* [快开始本地开发](https://doc.fastgpt.in/docs/development/intro/)
* [部署 FastGPT](https://doc.fastgpt.in/docs/installation)
* [部署 FastGPT](https://doc.fastgpt.in/docs/development/sealos)
* [系统配置文件说明](https://doc.fastgpt.in/docs/development/configuration/)
* [多模型配置](https://doc.fastgpt.in/docs/installation/one-api/)
* [版本更新/升级介绍](https://doc.fastgpt.in/docs/installation/upgrading)
* [多模型配置](https://doc.fastgpt.in/docs/development/one-api/)
* [版本更新/升级介绍](https://doc.fastgpt.in/docs/development/upgrading)
* [OpenAPI API 文档](https://doc.fastgpt.in/docs/development/openapi/)
* [知识库结构详解](https://doc.fastgpt.in/docs/use-cases/datasetengine/)
......
docSite/content/docs/_index.md
---
weight: 10
weight: 0
title: '文档'
description: 'FastGPT 官方文档'
icon: menu_book
......
docSite/content/docs/community/_index.md → docSite/content/docs/agreement/_index.md
---
weight: 1100
title: '社区'
weight: 1200
title: '协议'
description: '社区相关内容'
icon: 'forum'
icon: 'handshake'
draft: false
images: []
---
\ No newline at end of file
docSite/content/docs/agreement/disclaimer.md 0 → 100644
---
title: '免责声明'
description: ' FastGPT 免责声明'
icon: 'gavel'
draft: false
toc: true
weight: 1220
---
由于生成式 AI 的特性，其在不同国家的管控措施也会有所不同。请所有使用者务必遵守所在地的相关法律。
免责声明：以任何违反 FastGPT 可接受使用政策的方式使用，包括但不限于法律、法规、政府命令或法令禁止的任何用途，或任何侵犯他人权利的使用；由使用者自行承担。我们对由客户使用产生的问题概不负责。
下面是各国对生成式AI的管控条例的链接：
[中国生成式人工智能服务管理办法（征求意见稿）](http://www.cac.gov.cn/2023-04/11/c_1682854275475410.htm)
## 内容要求
我们禁止使用我们对接的模型服务生成可能对个人或社会造成伤害的内容。保障平台的安全性，是长期稳定运营的关键。如发现任何利用平台接入模型能力进行违规内容生成和使用，将立即封号，账号余额不退。
- 剥削和虐待
- 禁止描述、展示或宣扬儿童性剥削或性虐待的内容，无论法律是否禁止。这包括涉及儿童或使儿童色情的内容。
- 禁止描述或用于培养儿童的内容。修饰是成年人以剥削，特别是性剥削为目的与儿童建立关系的行为。这包括以性剥削、贩运或其他形式剥削为目的与儿童交流。
- 未经同意的私密内容
- 服务禁止描述、提供或宣传未经同意的亲密活动的内容。
- 禁止描述、提供特征或宣传或用于招揽商业性活动和性服务的内容。这包括鼓励和协调真正的性活动。
- 禁止描述或用于人口贩运目的的内容。这包括招募人员、便利交通、支付和助长对人的剥削，如强迫劳动、家庭奴役、役、强迫婚姻和强迫医疗程序。
- 自杀和自残，禁止描述、赞美、支持、促进、美化、鼓励和/或指导个人自残或自杀的内容。
- 暴力内容和行为
- 禁止描述、展示或宣扬血腥暴力或血腥的内容。
- 禁止描绘恐怖主义行为的内容；赞扬或支持恐怖组织、恐怖行为者或暴力恐怖意识形态；鼓励恐怖活动；向恐怖组织或恐怖事业提供援助；或协助恐怖组织招募成员。
- 禁止通过暴力威胁或煽动来鼓吹或宣扬对他人的暴力行为的内容。
- 仇恨言论和歧视
- 禁止基于实际或感知的种族、民族、国籍、性别、性别认同、性取向、宗教信仰、年龄、残疾状况、种姓或与系统性偏见或边缘化相关的任何其他特征等特征攻击、诋毁、恐吓、降级、针对或排斥个人或群体的内容。
- 禁止针对个人或群体进行威胁、恐吓、侮辱、贬低或贬低的语言或图像、宣扬身体伤害或其他虐待行为（如跟踪）的内容。
- 禁止故意欺骗并可能对公共利益产生不利影响的内容，包括与健康、安全、选举诚信或公民参与相关的欺骗性或不真实内容。
- 直接支持非法主动攻击或造成技术危害的恶意软件活动的内容，例如提供恶意可执行文件、组织拒绝服务攻击或管理命令和控制服务器。
docSite/content/docs/community/open-source.md → docSite/content/docs/agreement/open-source.md
......@@ -4,7 +4,7 @@ description: ' FastGPT 开源许可证'
icon: 'verified_user'
draft: false
toc: true
weight: 1120
weight: 1210
---
FastGPT 项目在 Apache License 2.0 许可下开源，同时包含以下附加条件：
......
docSite/content/docs/commercial/_index.md 0 → 100644
---
weight: 1100
title: '商业版介绍'
description: 'FastGPT 商业版介绍'
icon: 'shopping_cart'
draft: false
images: []
---
<!-- 1100 ~ 1200 -->
docSite/content/docs/commercial.md → docSite/content/docs/commercial/intro.md
......@@ -4,7 +4,7 @@ description: 'FastGPT 商业版相关说明'
icon: 'shopping_cart'
draft: false
toc: true
weight: 20
weight: 1001
---
## FastGPT 线上服务
......
docSite/content/docs/community/support.md → docSite/content/docs/community.md
......@@ -4,7 +4,7 @@ description: ' 加入 FastGPT 开发者社区和我们一起成长'
icon: 'forum'
draft: false
toc: true
weight: 1110
weight: 1300
---
FastGPT 是一个由用户和贡献者参与推动的开源项目，如果您对产品使用存在疑问和建议，可尝试以下方式寻求支持。我们的团队与社区会竭尽所能为您提供帮助。
......
docSite/content/docs/course/_index.md 0 → 100644
---
weight: 100
title: '基础教程'
description: 'FastGPT 基础教程'
icon: 'import_contacts'
draft: false
images: []
---
<!-- 100 ~ 300 -->
\ No newline at end of file
docSite/content/docs/quick-start.md → docSite/content/docs/course/quick-start.md
......@@ -4,7 +4,7 @@ description: '快速体验 FastGPT 基础功能'
icon: 'rocket_launch'
draft: false
toc: true
weight: 30
weight: 101
---
更多使用技巧，[查看视屏教程](https://www.bilibili.com/video/BV1n34y1A7Bo/?spm_id_from=333.337.search-card.all.click&vd_source=903c2b09b7412037c2eddc6a8fb9828b)
......
docSite/content/docs/development/_index.md
---
weight: 500
title: '开发指南'
weight: 700
title: '开发与部署指南'
description: '本地开发 FastGPT 必看'
icon: 'code_blocks'
draft: false
images: []
---
<!-- 700 ~ 1100 -->
docSite/content/docs/development/configuration.md
......@@ -4,10 +4,10 @@ description: 'FastGPT 配置参数介绍'
icon: 'settings'
draft: false
toc: true
weight: 520
weight: 708
---
由于环境变量不利于配置复杂的内容，新版 FastGPT 采用了 ConfigMap 的形式挂载配置文件，你可以在 `projects/app/data/config.json` 看到默认的配置文件。可以参考 [docker-compose 快速部署](/docs/installation/docker/) 来挂载配置文件。
由于环境变量不利于配置复杂的内容，新版 FastGPT 采用了 ConfigMap 的形式挂载配置文件，你可以在 `projects/app/data/config.json` 看到默认的配置文件。可以参考 [docker-compose 快速部署](/docs/development/docker/) 来挂载配置文件。
**开发环境下**，你需要将示例配置文件 `config.json` 复制成 `config.local.json` 文件才会生效。
......
docSite/content/docs/custom-models/_index.md → docSite/content/docs/development/custom-models/_index.md
......@@ -6,3 +6,4 @@ icon: 'model_training'
draft: false
images: []
---
<!-- 900~950 -->
\ No newline at end of file
docSite/content/docs/custom-models/chatglm2-m3e.md → docSite/content/docs/development/custom-models/chatglm2-m3e.md
......@@ -23,7 +23,7 @@ FastGPT 默认使用了 OpenAI 的 LLM 模型和向量模型，如果想要私
也可以通过环境变量引入：sk-key。有关docker环境变量引入的方法请自寻教程，此处不再赘述。
```
## 接入 [One API](/docs/installation/one-api/)
## 接入 [One API](/docs/development/one-api/)
为 chatglm2 和 m3e-large 各添加一个渠道，参数如下：
......
docSite/content/docs/development/design/_index.md
---
weight: 540
weight: 960
title: "设计方案"
description: "FastGPT 部分设计方案"
icon: public
draft: false
images: []
---
\ No newline at end of file
---
<!-- 960~1050 -->
\ No newline at end of file
docSite/content/docs/development/design/dataset.md
---
weight: 541
weight: 961
title: "数据集"
description: "FastGPT 数据集中文件与数据的设计方案"
icon: dataset
......
docSite/content/docs/installation/docker.md → docSite/content/docs/development/docker.md
......@@ -4,7 +4,7 @@ description: '使用 Docker Compose 快速部署 FastGPT'
icon: ''
draft: false
toc: true
weight: 720
weight: 707
---
## 准备条件
......@@ -13,13 +13,13 @@ weight: 720
### 1. 准备好代理环境（国外服务器可忽略）
确保可以访问 OpenAI，具体方案可以参考：[代理方案](/docs/installation/proxy/)。或直接在 Sealos 上 [部署 OneAPI](/docs/installation/one-api)，既解决代理问题也能实现多 Key 轮询、接入其他大模型。
确保可以访问 OpenAI，具体方案可以参考：[代理方案](/docs/development/proxy/)。或直接在 Sealos 上 [部署 OneAPI](/docs/development/one-api)，既解决代理问题也能实现多 Key 轮询、接入其他大模型。
### 2. 多模型支持
FastGPT 使用了 one-api 项目来管理模型池，其可以兼容 OpenAI 、Azure 、国内主流模型和本地模型等。
可选择 [Sealos 快速部署 OneAPI](/docs/installation/one-api)，更多部署方法可参考该项目的 [README](https://github.com/songquanpeng/one-api)，也可以直接通过以下按钮一键部署：
可选择 [Sealos 快速部署 OneAPI](/docs/development/one-api)，更多部署方法可参考该项目的 [README](https://github.com/songquanpeng/one-api)，也可以直接通过以下按钮一键部署：
[![](https://fastly.jsdelivr.net/gh/labring-actions/templates@main/Deploy-on-Sealos.svg)](https://cloud.sealos.io/?openapp=system-fastdeploy%3FtemplateName%3Done-api)
......
docSite/content/docs/development/intro.md
---
title: '快速开始'
title: '快速开始本地开发'
description: '对 FastGPT 进行开发调试'
icon: 'developer_guide'
draft: false
toc: true
weight: 510
weight: 705
---
本文档介绍了如何设置开发环境以构建和测试 [FastGPT](https://fastgpt.run)。
......@@ -46,7 +46,7 @@ git clone git@github.com:<github_username>/FastGPT.git
### 3. 安装数据库
第一次开发，需要先部署数据库，建议本地开发可以随便找一台 2C2G 的轻量小数据库实践。数据库部署教程：[Docker 快速部署](/docs/installation/docker/)。部署完了，可以本地访问其数据库。
第一次开发，需要先部署数据库，建议本地开发可以随便找一台 2C2G 的轻量小数据库实践。数据库部署教程：[Docker 快速部署](/docs/development/docker/)。部署完了，可以本地访问其数据库。
### 4. 初始配置
......
docSite/content/docs/installation/one-api.md → docSite/content/docs/development/one-api.md
......@@ -4,7 +4,7 @@ description: '通过接入 One API 来实现对各种大模型的支持'
icon: 'Api'
draft: false
toc: true
weight: 730
weight: 708
---
* 默认情况下，FastGPT 只配置了 GPT 的 3 个模型，如果你需要接入其他模型，需要进行一些额外配置。
......
docSite/content/docs/development/openapi/_index.md
---
weight: 560
weight: 850
title: "OpenAPI 接口文档"
description: "FastGPT OpenAPI 文档"
icon: api
draft: false
images: []
---
\ No newline at end of file
---
<!-- 850~900 -->
\ No newline at end of file
docSite/content/docs/development/openapi/auth.md
......@@ -4,7 +4,7 @@ description: 'FastGPT Api Key 使用与鉴权'
icon: 'key'
draft: false
toc: true
weight: 561
weight: 851
---
## 使用说明
......
docSite/content/docs/development/openapi/chat.md
......@@ -4,7 +4,7 @@ description: 'FastGPT OpenAPI 对话接口'
icon: 'chat'
draft: false
toc: true
weight: 562
weight: 852
---
## 发起对话
......
docSite/content/docs/development/openapi/dataset.md
......@@ -4,7 +4,7 @@ description: 'FastGPT OpenAPI 知识库接口'
icon: 'dataset'
draft: false
toc: true
weight: 563
weight: 853
---
| 如何获取知识库ID（datasetId） | 如何获取文件集合ID（collection_id） |
......
docSite/content/docs/development/openapi/share.md
......@@ -4,7 +4,7 @@ description: 'FastGPT 分享链接鉴权'
icon: 'share'
draft: false
toc: true
weight: 564
weight: 860
---
## 使用说明
......
docSite/content/docs/installation/proxy/_index.md → docSite/content/docs/development/proxy/_index.md
---
weight: 740
weight: 950
title: "代理方案"
description: "使用代理访问 OpenAI"
icon: public
description: "FastGPT 私有化部署代理方案"
icon: wify_proxy
draft: false
images: []
---
\ No newline at end of file
---
<!-- 950~960 -->
\ No newline at end of file
docSite/content/docs/installation/proxy/cloudflare.md → docSite/content/docs/development/proxy/cloudflare.md
......@@ -4,7 +4,7 @@ description: "使用 Cloudflare Worker 实现中转"
icon: "foggy"
draft: false
toc: true
weight: 742
weight: 953
---
[参考 "不做了睡觉" 的教程](https://gravel-twister-d32.notion.site/FastGPT-API-ba7bb261d5fd4fd9bbb2f0607dacdc9e)
......
docSite/content/docs/installation/proxy/http_proxy.md → docSite/content/docs/development/proxy/http_proxy.md
......@@ -4,7 +4,7 @@ description: "使用 HTTP 代理实现中转"
icon: "http"
draft: false
toc: true
weight: 743
weight: 952
---
如果你有代理工具（例如 [Clash](https://github.com/Dreamacro/clash) 或者 [sing-box](https://github.com/SagerNet/sing-box)），也可以使用 HTTP 代理来访问 OpenAI。只需要添加以下两个环境变量即可：
......
docSite/content/docs/installation/proxy/nginx.md → docSite/content/docs/development/proxy/nginx.md
......@@ -4,7 +4,7 @@ description: "使用 Sealos 部署 Nginx 实现中转"
icon: "cloud_sync"
draft: false
toc: true
weight: 741
weight: 951
---
## 登录 Sealos
......
docSite/content/docs/installation/sealos.md → docSite/content/docs/development/sealos.md
......@@ -4,7 +4,7 @@ description: "使用 Sealos 一键部署 FastGPT"
icon: "cloud"
draft: false
toc: true
weight: 710
weight: 706
---
Sealos 的服务器在国外，不需要额外处理网络问题，无需服务器、无需魔法、无需域名，支持高并发 & 动态伸缩。点击以下按钮即可一键部署 👇
......
docSite/content/docs/installation/upgrading/_index.md → docSite/content/docs/development/upgrading/_index.md
---
weight: 760
weight: 750
title: "版本更新/升级操作"
description: "FastGPT 版本更新介绍及升级操作"
icon: upgrade
draft: false
images: []
---
\ No newline at end of file
---
<!-- 750~850 -->
\ No newline at end of file
docSite/content/docs/development/upgrading/intro.md 0 → 100644
---
title: '升级说明'
description: 'FastGPT 升级说明'
icon: 'upgrade'
draft: false
toc: true
weight: 751
---
FastGPT 升级包括两个步骤：
1. 镜像升级
2. 执行升级初始化脚本
## 镜像名
**git版**
- FastGPT 主镜像：ghcr.io/labring/fastgpt:latest
- 商业版镜像：ghcr.io/c121914yu/fastgpt-pro:latest
- Admin 镜像：ghcr.io/c121914yu/fastgpt-admin:latest
**阿里云**
- FastGPT 主镜像: registry.cn-hangzhou.aliyuncs.com/fastgpt/fastgpt
- 商业版镜像：ghcr:registry.cn-hangzhou.aliyuncs.com/fastgpt/fastgpt-pro
- Admin 镜像: registry.cn-hangzhou.aliyuncs.com/fastgpt/fastgpt-admin
镜像由镜像名和`Tag`组成，例如: registry.cn-hangzhou.aliyuncs.com/fastgpt/fastgpt:v4.6.1 代表`4.6.3`版本镜像，具体可以看 docker hub, github 仓库。
## Sealos 修改镜像
1. 打开 [Sealos Cloud](https://cloud.sealos.io/)， 找到桌面上的应用管理
![](/imgs/updateImageSealos1.jpg)
2. 选择对应的应用 - 点击右边三个点 - 变更
![](/imgs/updateImageSealos2.jpg)
3. 修改镜像 - 确认变更
如果要修改配置文件，可以拉到下面的`配置文件`进行修改。
![](/imgs/updateImageSealos3.jpg)
## Docker-Compose 修改镜像
直接修改`yml`文件中的`image: `即可。随后执行:
```bash
docker-compose pull
docker-compose up -d
```
## 执行升级初始化脚本
镜像更新完后，可以查看文档中的`版本介绍`，通常需要执行升级脚本的版本都会标明`需要初始化`，打开对应的文档，参考说明执行初始化脚本即可，大部分时候都是需要发送一个`POST`请求。
## QA
### {{host}} 是什么
{{}} 代表变量， {{host}}代表一个名为 host 的变量。指的是你服务器的域名或 IP。
Sealos 中，你可以在下图中找到你的域名：
![](/imgs/updateImageSealos4.jpg)
### 如何获取 rootkey
从`docker-compose.yml`中的`environment`中获取，对应的是`ROOT_KEY`的值。
sealos 中可以从上图左侧的环境变量中获取。
### 如何跨版本升级！！
建议逐一版本升级，防止脏数据。例如，当前版本是4.4.7，需要升级到4.6。
1. 修改镜像到4.5，执行初始化
2. 修改镜像到4.5.1，执行初始化
3. 修改镜像到4.5.2，执行初始化
4. 修改镜像到4.6，执行初始化
5. .....
逐一升级
\ No newline at end of file
docSite/content/docs/installation/_index.md deleted 100644 → 0
---
weight: 700
title: "私有化部署"
description: "FastGPT 私有化部署文档"
icon: menu_book
draft: false
images: []
---
\ No newline at end of file
docSite/content/docs/intro.md
......@@ -52,9 +52,9 @@ FastGPT 对外的 API 接口对齐了 OpenAI 官方接口，可以直接接入
## FastGPT 特点
1. **项目完全开源**
1. **项目开源**
FastGPT 遵循 Apache License 2.0 开源协议，你可以 [Fork](https://github.com/labring/FastGPT/fork) 之后进行二次开发和发布。FastGPT 社区版将保留核心功能，商业版仅在社区版基础上使用 API 的形式进行扩展，不影响学习使用。
FastGPT 遵循附加条件 Apache License 2.0 开源协议，你可以 [Fork](https://github.com/labring/FastGPT/fork) 之后进行二次开发和发布。FastGPT 社区版将保留核心功能，商业版仅在社区版基础上使用 API 的形式进行扩展，不影响学习使用。
2. **独特的 QA 结构**
......@@ -80,34 +80,3 @@ FastGPT 对外的 API 接口对齐了 OpenAI 官方接口，可以直接接入
![](/imgs/functional-arch.webp)
## 免责声明
由于生成式 AI 的特性，其在不同国家的管控措施也会有所不同。请所有使用者务必遵守所在地的相关法律。
免责声明：以任何违反 FastGPT 可接受使用政策的方式使用，包括但不限于法律、法规、政府命令或法令禁止的任何用途，或任何侵犯他人权利的使用；由使用者自行承担。我们对由客户使用产生的问题概不负责。
下面是各国对生成式AI的管控条例的链接：
[中国生成式人工智能服务管理办法（征求意见稿）](http://www.cac.gov.cn/2023-04/11/c_1682854275475410.htm)
### 内容要求
我们禁止使用我们对接的模型服务生成可能对个人或社会造成伤害的内容。保障平台的安全性，是长期稳定运营的关键。如发现任何利用平台接入模型能力进行违规内容生成和使用，将立即封号，账号余额不退。
- 剥削和虐待
- 禁止描述、展示或宣扬儿童性剥削或性虐待的内容，无论法律是否禁止。这包括涉及儿童或使儿童色情的内容。
- 禁止描述或用于培养儿童的内容。修饰是成年人以剥削，特别是性剥削为目的与儿童建立关系的行为。这包括以性剥削、贩运或其他形式剥削为目的与儿童交流。
- 未经同意的私密内容
- 服务禁止描述、提供或宣传未经同意的亲密活动的内容。
- 禁止描述、提供特征或宣传或用于招揽商业性活动和性服务的内容。这包括鼓励和协调真正的性活动。
- 禁止描述或用于人口贩运目的的内容。这包括招募人员、便利交通、支付和助长对人的剥削，如强迫劳动、家庭奴役、役、强迫婚姻和强迫医疗程序。
- 自杀和自残，禁止描述、赞美、支持、促进、美化、鼓励和/或指导个人自残或自杀的内容。
- 暴力内容和行为
- 禁止描述、展示或宣扬血腥暴力或血腥的内容。
- 禁止描绘恐怖主义行为的内容；赞扬或支持恐怖组织、恐怖行为者或暴力恐怖意识形态；鼓励恐怖活动；向恐怖组织或恐怖事业提供援助；或协助恐怖组织招募成员。
- 禁止通过暴力威胁或煽动来鼓吹或宣扬对他人的暴力行为的内容。
- 仇恨言论和歧视
- 禁止基于实际或感知的种族、民族、国籍、性别、性别认同、性取向、宗教信仰、年龄、残疾状况、种姓或与系统性偏见或边缘化相关的任何其他特征等特征攻击、诋毁、恐吓、降级、针对或排斥个人或群体的内容。
- 禁止针对个人或群体进行威胁、恐吓、侮辱、贬低或贬低的语言或图像、宣扬身体伤害或其他虐待行为（如跟踪）的内容。
- 禁止故意欺骗并可能对公共利益产生不利影响的内容，包括与健康、安全、选举诚信或公民参与相关的欺骗性或不真实内容。
- 直接支持非法主动攻击或造成技术危害的恶意软件活动的内容，例如提供恶意可执行文件、组织拒绝服务攻击或管理命令和控制服务器。
docSite/content/docs/pricing.md
......@@ -4,7 +4,7 @@ description: 'FastGPT 线上版定价'
icon: 'currency_yen'
draft: false
toc: true
weight: 11
weight: 1200
---
## Tokens 说明
......
docSite/content/docs/use-cases/_index.md
---
weight: 300
weight: 500
title: '使用案例'
description: '有关 FastGPT 其他实践案例的更多信息'
icon: 'cases'
draft: false
images: []
---
<!-- 500 ~ 700 -->
docSite/content/docs/use-cases/ai_settings.md
......@@ -4,7 +4,7 @@ description: "FastGPT AI 高级配置说明"
icon: "sign_language"
draft: false
toc: true
weight: 310
weight: 501
---
在 FastGPT 的 AI 对话模块中，有一个 AI 高级配置，里面包含了 AI 模型的参数配置，本文详细介绍这些配置的含义。
......
docSite/content/docs/use-cases/datasetEngine.md
......@@ -4,7 +4,7 @@ description: "本节会详细介绍 FastGPT 知识库结构设计，理解其 QA
icon: "dataset"
draft: false
toc: true
weight: 311
weight: 502
---
## 理解向量
......@@ -84,7 +84,7 @@ FastGPT 采用了 `PostgresSQL` 的 `PG Vector` 插件作为向量检索器，
### 导入数据方案5 - API导入
参考[FastGPT OpenAPI使用](/docs/development/openapi/#知识库添加数据)。
参考[FastGPT OpenAPI使用](/docs/development/openapi)。
## QA的组合与引用提示词构建
......
docSite/content/docs/use-cases/feishu.md
......@@ -4,7 +4,7 @@ description: "FastGPT 接入飞书机器人 "
icon: "chat"
draft: false
toc: true
weight: 322
weight: 503
---
# FastGPT 一分钟接入飞书
......
docSite/content/docs/use-cases/kb.md
---
title: " 打造高质量 AI 知识库"
title: " 打造高质量 AI 知识库(过期)"
description: " 利用 FastGPT 打造高质量 AI 知识库"
icon: "school"
draft: false
toc: true
weight: 340
weight: 699
---
## 前言
......
docSite/content/docs/use-cases/onwechat.md
......@@ -4,7 +4,7 @@ description: "FastGPT 对接 chatgpt-on-wechat"
icon: "chat"
draft: false
toc: true
weight: 312
weight: 504
---
# 1 分钟对接 chatgpt-on-wechat
......
docSite/content/docs/use-cases/openapi.md
......@@ -4,7 +4,7 @@ description: "通过与 OpenAI 兼容的 API 对接第三方应用"
icon: "model_training"
draft: false
toc: true
weight: 312
weight: 505
---
## 获取 API 秘钥
......
docSite/content/docs/use-cases/wechat.md
......@@ -4,7 +4,7 @@ description: "FastGPT 接入微信和企业微信 "
icon: "chat"
draft: false
toc: true
weight: 322
weight: 506
---
# FastGPT 三分钟接入微信/企业微信
......
docSite/content/docs/workflow/_index.md
---
weight: 100
weight: 300
title: '高级编排'
description: 'FastGPT 高级编排文档'
icon: 'family_history'
draft: false
images: []
---
<!-- 300 ~ 500 -->
docSite/content/docs/workflow/examples/_index.md
---
weight: 140
weight: 400
title: "编排示例"
description: "介绍 FastGPT 的高级编排实践案例"
icon: "list"
......
docSite/content/docs/workflow/examples/fixingEvidence.md
......@@ -4,7 +4,7 @@ description: '利用指定回复，创建固定的开头和结尾'
icon: 'healing'
draft: false
toc: true
weight: 142
weight: 401
---
![](/imgs/demo-fix-evidence1.png)
......
docSite/content/docs/workflow/examples/google_search.md
......@@ -4,7 +4,7 @@ description: '将 FastGPT 外接搜索引擎'
icon: 'search'
draft: false
toc: true
weight: 143
weight: 402
---
![](/imgs/google_search_1.png)
......
docSite/content/docs/workflow/examples/lab_appointment.md
......@@ -4,7 +4,7 @@ description: '展示高级编排操作数据库的能力'
icon: 'database'
draft: false
toc: true
weight: 141
weight: 403
---
![](/imgs/demo-appointment1.png)
......
docSite/content/docs/workflow/examples/op_question.md
......@@ -4,7 +4,7 @@ description: '利用 GPT 优化和完善知识库æœç´¢è¯ï¼Œå®žçŽ°ä¸Šä¸‹æ–‡å…³è
icon: 'search'
draft: false
toc: true
weight: 144
weight: 404
---
![](/imgs/demo_op_question1.png)
......
docSite/content/docs/workflow/examples/versatile_assistant.md
......@@ -4,7 +4,7 @@ description: '赋予联网功能，将用户问题进行分类，细分后对接
icon: 'search'
draft: false
toc: true
weight: 143
weight: 406
---
![](/imgs/versatile_assistant_1.png)
......
docSite/content/docs/workflow/intro.md
......@@ -4,7 +4,7 @@ description: "快速了解 FastGPT 高级编排"
icon: "circle"
draft: false
toc: true
weight: 110
weight: 301
---
FastGPT 从 V4 版本开始采用新的交互方式来构建 AI 应用。使用了 Flow 节点编排的方式来实现复杂工作流，提高可玩性和扩展性。但同时也提高了上手的门槛，有一定开发背景的用户使用起来会比较容易。
......
docSite/content/docs/workflow/modules/_index.md
---
weight: 120
weight: 350
title: "模块介绍"
description: "介绍 FastGPT 的常用模块"
icon: "apps"
......
docSite/content/docs/workflow/modules/ai_chat.md
......@@ -4,7 +4,7 @@ description: "FastGPT AI 对话模块介绍"
icon: "chat"
draft: false
toc: true
weight: 123
weight: 351
---
## 特点
......@@ -21,7 +21,7 @@ weight: 123
### 对话模型
可以通过 [config.json](/docs/installation/reference/models/) 配置可选的对话模型，通过 [one-api](/docs/installation/one-api/) 来实现多模型接入。
可以通过 [config.json](/docs/development/configuration/) 配置可选的对话模型，通过 [one-api](/docs/development/one-api/) 来实现多模型接入。
### 温度 & 回复上限
......
docSite/content/docs/workflow/modules/content_extract.md
......@@ -4,7 +4,7 @@ description: "FastGPT 内容提取模块介绍"
icon: "content_paste_go"
draft: false
toc: true
weight: 125
weight: 352
---
## 特点
......
docSite/content/docs/workflow/modules/guide.md
......@@ -4,7 +4,7 @@ description: "FastGPT 用户引导模块介绍"
icon: "psychology"
draft: false
toc: true
weight: 127
weight: 353
---
## 特点
......
docSite/content/docs/workflow/modules/history.md
......@@ -4,7 +4,7 @@ description: "FastGPT 历史记录模块介绍"
icon: "history"
draft: false
toc: true
weight: 128
weight: 354
---
# 特点
......
docSite/content/docs/workflow/modules/http.md
......@@ -4,7 +4,7 @@ description: "FastGPT HTTP 模块介绍"
icon: "http"
draft: false
toc: true
weight: 129
weight: 355
---
## 特点
......@@ -101,4 +101,4 @@ export default async function (ctx: FunctionContext) {
## 作用
基于 HTTP 模块可以无限扩展，比如操作数据库、执行联网搜索、发送邮箱等等。如果你有有趣的案例，欢迎提交 PR 到 [编排案例](/docs/category/examples)
\ No newline at end of file
基于 HTTP 模块可以无限扩展，比如操作数据库、执行联网搜索、发送邮箱等等。如果你有有趣的案例，欢迎提交 PR 到 [编排案例](/docs/workflow/examples)
\ No newline at end of file
docSite/content/docs/workflow/modules/input.md
......@@ -4,7 +4,7 @@ description: "FastGPT 用户问题模块介绍"
icon: "input"
draft: false
toc: true
weight: 131
weight: 356
---
## 特点
......
docSite/content/docs/workflow/modules/kb_search.md
......@@ -4,7 +4,7 @@ description: 'FastGPT AI 知识库搜索模块介绍'
icon: 'chat'
draft: false
toc: true
weight: 124
weight: 357
---
## 特点
......
docSite/content/docs/workflow/modules/question.md
......@@ -4,7 +4,7 @@ description: "FastGPT 问题分类模块介绍"
icon: "quiz"
draft: false
toc: true
weight: 126
weight: 358
---
## 特点
......
docSite/content/docs/workflow/modules/reply.md
......@@ -4,7 +4,7 @@ description: "FastGPT 指定回复模块介绍"
icon: "reply"
draft: false
toc: true
weight: 130
weight: 359
---
## 特点
......
docSite/content/docs/workflow/modules/trigger.md
......@@ -4,7 +4,7 @@ description: "FastGPT 触发器模块介绍"
icon: "work_history"
draft: false
toc: true
weight: 121
weight: 360
---
细心的同学可以发现，在每个功能模块里都会有一个叫【触发器】的外部输入，并且是 any 类型。
......
docSite/content/docs/workflow/modules/variable.md
......@@ -4,7 +4,7 @@ description: "FastGPT 全局变量模块介绍"
icon: "variables"
draft: false
toc: true
weight: 122
weight: 361
---
## 特点
......
packages/global/common/error/code/dataset.ts
......@@ -6,7 +6,9 @@ export enum DatasetErrEnum {
unCreateCollection = 'unCreateCollection',
unAuthDatasetCollection = 'unAuthDatasetCollection',
unAuthDatasetData = 'unAuthDatasetData',
unAuthDatasetFile = 'unAuthDatasetFile'
unAuthDatasetFile = 'unAuthDatasetFile',
unLinkCollection = 'unLinkCollection'
}
const datasetErr = [
{
......@@ -28,6 +30,10 @@ const datasetErr = [
{
statusText: DatasetErrEnum.unCreateCollection,
message: '无权创建数据集'
},
{
statusText: DatasetErrEnum.unLinkCollection,
message: 'core.dataset.error.unLinkCollection'
}
];
export default datasetErr.reduce((acc, cur, index) => {
......
packages/global/core/module/template/system/http.ts
......@@ -21,7 +21,7 @@ export const HttpModule: FlowModuleTemplateType = {
valueType: ModuleDataTypeEnum.string,
label: '请求地址',
description: '请求目标地址',
placeholder: 'https://api.fastgpt.run/getInventory',
placeholder: 'https://api.ai.com/getInventory',
required: true,
showTargetInApp: false,
showTargetInPlugin: false
......
packages/service/core/dataset/collection/schema.ts
......@@ -44,7 +44,6 @@ const DatasetCollectionSchema = new Schema({
enum: Object.keys(DatasetCollectionTypeMap),
required: true
},
name: {
type: String,
required: true
......
packages/service/core/dataset/collection/utils.ts
import type { CollectionWithDatasetType } from '@fastgpt/global/core/dataset/type.d';
import { MongoDatasetCollection } from './schema';
import { ParentTreePathItemType } from '@fastgpt/global/common/parentFolder/type';
import type { ParentTreePathItemType } from '@fastgpt/global/common/parentFolder/type.d';
import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset';
import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter';
import { MongoDatasetTraining } from '../training/schema';
import { urlsFetch } from '@fastgpt/global/common/file/tools';
import { DatasetCollectionTypeEnum } from '@fastgpt/global/core/dataset/constant';
/**
* get all collection by top collectionId
......@@ -58,3 +64,64 @@ export function getCollectionUpdateTime({ name, time }: { time?: Date; name: str
if (name.startsWith('手动') || ['manual', 'mark'].includes(name)) return new Date('2999/9/9');
return new Date();
}
/* link collection start load data */
export const loadingOneChunkCollection = async ({
collectionId,
tmbId,
billId,
rawText
}: {
collectionId: string;
tmbId: string;
billId?: string;
rawText?: string;
}) => {
const collection = (await MongoDatasetCollection.findById(collectionId).populate(
'datasetId'
)) as CollectionWithDatasetType;
if (!collection) {
return Promise.reject(DatasetErrEnum.unCreateCollection);
}
const newRawText = await (async () => {
if (rawText) return rawText;
// link
if (collection.type === DatasetCollectionTypeEnum.link && collection.rawLink) {
// crawl new data
const result = await urlsFetch({
urlList: [collection.rawLink],
selector: collection.datasetId?.websiteConfig?.selector
});
return result[0].content;
}
// file
return '';
})();
// split data
const { chunks } = splitText2Chunks({
text: newRawText,
chunkLen: collection.chunkSize || 512
});
// insert to training queue
await MongoDatasetTraining.insertMany(
chunks.map((item, i) => ({
teamId: collection.teamId,
tmbId,
datasetId: collection.datasetId._id,
collectionId: collection._id,
billId,
mode: collection.trainingType,
prompt: '',
model: collection.datasetId.vectorModel,
q: item,
a: '',
chunkIndex: i
}))
);
};
packages/service/core/dataset/data/controller.ts
......@@ -41,8 +41,8 @@ export async function delCollectionRelevantData({
collectionIds: string[];
fileIds: string[];
}) {
collectionIds = collectionIds.map((item) => String(item));
const filterFileIds = fileIds.filter(Boolean);
collectionIds = collectionIds.filter(Boolean).map((item) => String(item));
const filterFileIds = fileIds.filter(Boolean).map((item) => String(item));
// delete training data
await MongoDatasetTraining.deleteMany({
......
packages/service/core/dataset/training/controller.ts 0 → 100644
import { delay } from '@fastgpt/global/common/system/utils';
import { MongoDatasetTraining } from './schema';
export const lockTrainingDataByTeamId = async (teamId: string, retry = 3): Promise<any> => {
try {
await MongoDatasetTraining.updateMany(
{
teamId
},
{
lockTime: new Date('2999/5/5')
}
);
} catch (error) {
if (retry > 0) {
await delay(1000);
return lockTrainingDataByTeamId(teamId, retry - 1);
}
return Promise.reject(error);
}
};
packages/service/support/wallet/bill/controller.ts 0 → 100644
import { BillSourceEnum } from '@fastgpt/global/support/wallet/bill/constants';
import { MongoBill } from './schema';
export const createTrainingBill = async ({
teamId,
tmbId,
appName,
billSource,
vectorModel,
agentModel
}: {
teamId: string;
tmbId: string;
appName: string;
billSource: `${BillSourceEnum}`;
vectorModel: string;
agentModel: string;
}) => {
const { _id } = await MongoBill.create({
teamId,
tmbId,
appName,
source: billSource,
list: [
{
moduleName: 'wallet.moduleName.index',
model: vectorModel,
amount: 0,
tokenLen: 0
},
{
moduleName: 'wallet.moduleName.qa',
model: agentModel,
amount: 0,
tokenLen: 0
}
],
total: 0
});
return { billId: String(_id) };
};
projects/app/public/locales/en/common.json
......@@ -131,6 +131,7 @@
"Config": "Config",
"Confirm": "Confirm",
"Confirm Create": "Create",
"Confirm Import": "Import",
"Confirm Move": "Move here",
"Confirm Update": "Update",
"Copy": "Copy",
......@@ -153,6 +154,7 @@
"Input": "Input",
"Intro": "Intro",
"Last Step": "Last",
"Load Failed": "Load Failed",
"Loading": "Loading",
"Max credit": "Credit",
"Max credit tips": "What is the maximum amount of money that can be consumed by the link? If the link is exceeded, it will be banned. -1 indicates no limit.",
......@@ -191,6 +193,10 @@
"Upload File Failed": "Upload File Failed",
"Username": "UserName",
"Website": "Website",
"avatar": {
"Select Avatar": "Select Avatar",
"Select Failed": "Select Failed"
},
"choosable": "choosable",
"confirm": {
"Common Tip": "Operational Confirm"
......@@ -203,6 +209,11 @@
},
"export": "",
"file": {
"File Content": "File Content",
"File Name": "File Name",
"File content can not be empty": "File content can not be empty",
"Filename Can not Be Empty": "Filename Can not Be Empty",
"Read File Error": "Read file error",
"Select file amount limit 100": "You can select a maximum of 100 files at a time"
},
"folder": {
......@@ -218,6 +229,9 @@
"link": {
"UnValid": "UnValid Link"
},
"price": {
"Amount": "{{amount}}{{unit}}"
},
"speech": {
"error tip": "Speech Failed"
}
......@@ -226,7 +240,11 @@
"Max Token": "MaxTokens",
"ai": {
"Model": "Model",
"Prompt": "Prompt"
"Prompt": "Prompt",
"model": {
"Dataset Agent Model": "Agent Model",
"Vector Model": "Vector Model"
}
},
"app": {
"App params config": "App Config",
......@@ -276,18 +294,23 @@
},
"dataset": {
"All Dataset": "All Dataset",
"Avatar": "Avatar",
"Choose Dataset": "Choose Dataset",
"Common Dataset": "Common Dataset",
"Common Dataset Desc": "Knowledge bases can be built by importing files, web links, or manual entry",
"Create dataset": "Create Dataset",
"Dataset": "Dataset",
"Dataset ID": "Dataset ID",
"Dataset Type": "Dataset Type",
"Delete Confirm": "Are you sure to delete the knowledge base? Data cannot be recovered after deletion, please confirm!",
"Delete Website Tips": "Confirm to delete the website",
"Empty Dataset": "",
"Empty Dataset Tips": "There is no knowledge base yet, go create one!",
"Folder Dataset": "Folder",
"Go Dataset": "To Dataset",
"Intro Placeholder": "This dataset has not yet been introduced~",
"My Dataset": "My Dataset",
"Name": "Name",
"Quote Length": "Quote Length",
"Read Dataset": "Read Dataset",
"Search Top K": "Top K",
......@@ -301,10 +324,15 @@
"collection": {
"Click top config website": "Config",
"Empty Tip": "The collection is empty",
"QA Prompt": "QA Prompt",
"Start Sync Tip": "Are you sure to start synchronizing data? The old data will be deleted and then re-acquired, please confirm!",
"Sync": "Data Sync",
"Website Create Success": "Created successfully, data is being synchronized",
"Website Empty Tip": "No associated website yet,",
"Website Empty Tip": "No associated website yet",
"Website Link": "Website Link",
"Website Sync": "Website",
"id": "Id",
"Sync Collection": "Data Sync",
"metadata": {
"Chunk Size": "Chunk Size",
"Createtime": "Create Time",
......@@ -328,19 +356,50 @@
}
},
"data": {
"Auxiliary Data": "Auxiliary Data",
"Auxiliary Data Placeholder": "This section is optional, usually in conjunction with the previous [data content], to build a structured prompt word, for special scenarios, up to {{maxToken}} words.",
"Auxiliary Data Tip": "This section is divided into optional fields \n This content is usually designed to work with the previous data content to build structured prompt words for special scenarios",
"Data Content": "Data Content",
"Data Content Placeholder": "The input field is required, and the content is usually a description of the knowledge point, or it can be a user question, at most {{maxToken}} words.",
"Data Content Tip": "The input box is a required field \n The content is usually a description of the knowledge point, but also can be a user question.",
"Default Index Tip": "Cannot be edited, the default index will use the text of [related data content] and [auxiliary data] to generate an index directly, if the default index is not needed, you can delete it. Each piece of data must have more than one index. After all indexes are deleted, a default index is automatically generated.",
"Edit": "Edit Data",
"Empty Tip": "This collection has no data yet",
"Search data placeholder": "Search relevant data",
"Too Long": "Content is too long",
"Total Amount": "{{total}} Chunks",
"data is deleted": "Data is deleted",
"id": "Data ID"
},
"error": {
"Start Sync Failed": "Start Sync Failed"
},
"file": "File",
"folder": "Folder",
"import": {
"CSV Import": "CSV QA Import",
"CSV Import Tip": "Import q and a from csv, data is required to be sorted out in advance",
"Chunk Range": "Range: 100~{{max}}",
"Chunk Split": "Chunk Split",
"Chunk Split Tip": "Select the files and split the by sentences",
"Csv format error": "The csv file format is incorrect, please ensure that the index and content columns are two",
"Estimated Price": "Estimated Price",
"Estimated Price Tips": "Index generation is billed as: {{price}}/1k tokens",
"Fetch Error": "Get link failed",
"Fetch Url": "Url",
"Fetch url placeholder": "Up to 10 links, one per line.",
"Fetch url tip": "Only static links can be read, please check the results",
"Ideal chunk length": "Ideal chunk length",
"Ideal chunk length Tips": "Segment by end symbol. We recommend that your document should be properly punctuated to ensure that each complete sentence length does not exceed this value \n Chinese document recommended 400~1000\n English document recommended 600~1200"
"Ideal chunk length Tips": "Segment by end symbol. We recommend that your document should be properly punctuated to ensure that each complete sentence length does not exceed this value \n Chinese document recommended 400~1000\n English document recommended 600~1200",
"Import Failed": "Import Failed",
"Import Success Tip": "The {{num}} group data is imported successfully. Please wait for training.",
"Import Tip": "This task cannot be terminated and takes some time to generate indexes. Please confirm the import. If the balance is insufficient, the unfinished task will be suspended and can continue after topping up.",
"Only Show First 50 Chunk": "Show only part",
"QA Import": "QA Split",
"QA Import Tip": "Select the files and let the LLM automatically generate QA",
"Re Preview": "RePreview",
"Set Chunk Error": "Split chunks error",
"Total Chunk Preview": "Chunk Preview: {{totalChunks}} "
},
"link": "Link",
"search": {
......@@ -365,6 +424,7 @@
"syncing": "Syncing"
},
"test": {
"Search Test": "Search Test",
"Test": "Start",
"Test Result": "Results",
"Test Text": "Text",
......@@ -386,7 +446,8 @@
"Confirm Create Tips": "Confirm to synchronize the site, the synchronization task will start later, please confirm!",
"Confirm Update Tips": "Are you sure to update the site configuration? The synchronization starts immediately with the new configuration. Please confirm",
"Selector": "Selector",
"Start Sync": "Start Sync"
"Start Sync": "Start Sync",
"UnValid Website Tip": "Your site may not be static and cannot be synchronized"
}
},
"module": {
......@@ -452,7 +513,6 @@
}
},
"dataset": {
"Agent Model": "Learning Model",
"Chunk Length": "Chunk Length",
"Confirm move the folder": "Confirm Move",
"Confirm to delete the data": "Confirm to delete the data?",
......@@ -474,7 +534,6 @@
"Manual Mark": "Manual Mark",
"Mark Data": "Mark Data",
"Move Failed": "Move Failed",
"My Dataset": "My Dataset",
"Queue Desc": "This data refers to the current amount of training for the entire system. FastGPT uses queued training, and if you have too much data to train, you may need to wait for a while",
"Select Dataset": "Select Dataset",
"Select Dataset Tips": "Select only knowledge bases with the same index model",
......@@ -518,7 +577,6 @@
},
"input is empty": "The data content cannot be empty"
},
"deleteDatasetTips": "Are you sure to delete the knowledge base? Data cannot be recovered after deletion, please confirm!",
"deleteFolderTips": "Are you sure to delete this folder and all the knowledge bases it contains? Data cannot be recovered after deletion, please confirm!",
"import csv tip": "Ensure that the CSV is in UTF-8 format; otherwise, garbled characters will be displayed",
"test": {
......
projects/app/public/locales/zh/common.json
......@@ -131,6 +131,7 @@
"Config": "配置",
"Confirm": "确认",
"Confirm Create": "确认创建",
"Confirm Import": "确认导入",
"Confirm Move": "移动到这",
"Confirm Update": "确认更新",
"Copy": "复制",
......@@ -153,6 +154,7 @@
"Input": "输入",
"Intro": "介绍",
"Last Step": "上一步",
"Load Failed": "加载失败",
"Loading": "加载中",
"Max credit": "最大金额",
"Max credit tips": "该链接最大可消耗多少金额，超出后链接将被禁止使用。-1 代表无限制。",
......@@ -177,7 +179,7 @@
"Select File Failed": "选择文件异常",
"Select One Folder": "选择一个目录",
"Set Avatar": "点击设置头像",
"Set Name": "取个响亮的名字",
"Set Name": "取个名字",
"Status": "状态",
"Team": "团队",
"Test": "测试",
......@@ -191,6 +193,10 @@
"Upload File Failed": "上传文件失败",
"Username": "用户名",
"Website": "网站",
"avatar": {
"Select Avatar": "点击选择头像",
"Select Failed": "选择头像异常"
},
"choosable": "可选",
"confirm": {
"Common Tip": "操作确认"
......@@ -203,6 +209,11 @@
},
"export": "",
"file": {
"File Content": "文件内容",
"File Name": "文件名",
"File content can not be empty": "文件内容不能为空",
"Filename Can not Be Empty": "文件名不能为空",
"Read File Error": "解析文件失败",
"Select file amount limit 100": "每次最多选择100个文件"
},
"folder": {
......@@ -218,6 +229,9 @@
"link": {
"UnValid": "无效的链接"
},
"price": {
"Amount": "{{amount}}{{unit}}"
},
"speech": {
"error tip": "语音转文字失败"
}
......@@ -226,7 +240,11 @@
"Max Token": "单条数据上限",
"ai": {
"Model": "AI 模型",
"Prompt": "提示词"
"Prompt": "提示词",
"model": {
"Dataset Agent Model": "文件处理模型",
"Vector Model": "索引模型"
}
},
"app": {
"App params config": "应用配置",
......@@ -276,18 +294,23 @@
},
"dataset": {
"All Dataset": "全部知识库",
"Avatar": "知识库头像",
"Choose Dataset": "关联知识库",
"Common Dataset": "通用知识库",
"Common Dataset Desc": "可通过导入文件、网页链接或手动录入形式构建知识库",
"Create dataset": "创建一个知识库",
"Dataset": "知识库",
"Dataset ID": "知识库 ID",
"Dataset Type": "知识库类型",
"Delete Confirm": "确认删除该知识库？删除后数据无法恢复，请确认！",
"Delete Website Tips": "确认删除该站点？",
"Empty Dataset": "",
"Empty Dataset Tips": "还没有知识库，快去创建一个吧！",
"Folder Dataset": "文件夹",
"Go Dataset": "前往知识库",
"Intro Placeholder": "这个知识库还没有介绍~",
"My Dataset": "我的知识库",
"Name": "知识库名称",
"Quote Length": "引用内容长度",
"Read Dataset": "查看知识库详情",
"Search Top K": "单次搜索数量",
......@@ -301,10 +324,15 @@
"collection": {
"Click top config website": "点击配置网站",
"Empty Tip": "数据集空空如也",
"QA Prompt": "QA 拆分引导词",
"Start Sync Tip": "确认开始同步数据？将会删除旧数据后重新获取，请确认！",
"Sync": "同步数据",
"Website Create Success": "创建成功，正在同步数据",
"Website Empty Tip": "还没有关联网站，",
"Website Empty Tip": "还没有关联网站",
"Website Link": "Web 站点地址",
"Website Sync": "Web 站点同步",
"id": "集合ID",
"Sync Collection": "数据同步",
"metadata": {
"Chunk Size": "分割大小",
"Createtime": "创建时间",
......@@ -328,19 +356,50 @@
}
},
"data": {
"Auxiliary Data": "辅助数据",
"Auxiliary Data Placeholder": "该部分为可选填项, 通常是为了与前面的【数据内容】配合，构建结构化提示词，用于特殊场景，最多 {{maxToken}} 字。",
"Auxiliary Data Tip": "该部分为可选填项\n该内容通常是为了与前面的数据内容配合，构建结构化提示词，用于特殊场景",
"Data Content": "相关数据内容",
"Data Content Placeholder": "该输入框是必填项，该内容通常是对于知识点的描述，也可以是用户的问题，最多 {{maxToken}} 字。",
"Data Content Tip": "该输入框是必填项\n该内容通常是对于知识点的描述，也可以是用户的问题。",
"Default Index Tip": "无法编辑，默认索引会使用【相关数据内容】与【辅助数据】的文本直接生成索引，如不需要默认索引，可删除。 每条数据必须保证有一个以上索引，所有索引被删除后，会自动生成默认索引。",
"Edit": "编辑数据",
"Empty Tip": "这个集合还没有数据~",
"Search data placeholder": "搜索相关数据",
"Too Long": "总长度超长了",
"Total Amount": "{{total}} 组",
"data is deleted": "该数据已被删除",
"id": "数据ID"
},
"error": {
"Start Sync Failed": "开始同步失败"
},
"file": "文件",
"folder": "目录",
"import": {
"CSV Import": "CSV 导入",
"CSV Import Tip": "通过批量导入问答对，要求提前整理好数据",
"Chunk Range": "范围: 100~{{max}}",
"Chunk Split": "直接分段",
"Chunk Split Tip": "选择文本文件，直接将其按分段进行处理",
"Csv format error": "csv 文件格式有误,请确保 index 和 content 两列",
"Estimated Price": "预估价格",
"Estimated Price Tips": "索引生成计费为: {{price}}/1k tokens",
"Fetch Error": "获取链接失败",
"Fetch Url": "网络链接",
"Fetch url placeholder": "最多10个链接，每行一个。",
"Fetch url tip": "仅支持读取静态链接，请注意检查结果",
"Ideal chunk length": "理想分块长度",
"Ideal chunk length Tips": "按结束符号进行分段。我们建议您的文档应合理的使用标点符号，以确保每个完整的句子长度不要超过该值\n中文文档建议400~1000\n英文文档建议600~1200"
"Ideal chunk length Tips": "按结束符号进行分段。我们建议您的文档应合理的使用标点符号，以确保每个完整的句子长度不要超过该值\n中文文档建议400~1000\n英文文档建议600~1200",
"Import Failed": "导入文件失败",
"Import Success Tip": "共成功导入 {{num}} 组数据，请耐心等待训练.",
"Import Tip": "该任务无法终止，需要一定时间生成索引，请确认导入。如果余额不足，未完成的任务会被暂停，充值后可继续进行。",
"Only Show First 50 Chunk": "仅展示部分",
"QA Import": "QA拆分",
"QA Import Tip": "选择文本文件，让大模型自动生成问答对",
"Re Preview": "重新生成预览",
"Set Chunk Error": "文本分段异常",
"Total Chunk Preview": "分段预览({{totalChunks}}组)"
},
"link": "链接",
"search": {
......@@ -365,6 +424,7 @@
"syncing": "同步中"
},
"test": {
"Search Test": "搜索测试",
"Test": "测试",
"Test Result": "测试结果",
"Test Text": "测试文本",
......@@ -386,7 +446,8 @@
"Confirm Create Tips": "确认同步该站点，同步任务将随后开启，请确认！",
"Confirm Update Tips": "确认更新站点配置？会立即按新的配置开始同步，请确认！",
"Selector": "选择器",
"Start Sync": "开始同步"
"Start Sync": "开始同步",
"UnValid Website Tip": "您的站点可能非静态站点，无法同步"
}
},
"module": {
......@@ -452,7 +513,6 @@
}
},
"dataset": {
"Agent Model": "文件处理模型",
"Chunk Length": "数据总量",
"Confirm move the folder": "确认移动到该目录",
"Confirm to delete the data": "确认删除该数据？",
......@@ -474,7 +534,6 @@
"Manual Mark": "手动标注",
"Mark Data": "标注数据",
"Move Failed": "移动出现错误~",
"My Dataset": "我的知识库",
"Queue Desc": "该数据是指整个系统当前待训练的数量。{{title}} 采用排队训练的方式，如果待训练的数据过多，可能需要等待一段时间",
"Select Dataset": "选择该知识库",
"Select Dataset Tips": "仅能选择同一个索引模型的知识库",
......@@ -518,7 +577,6 @@
},
"input is empty": "数据内容不能为空 "
},
"deleteDatasetTips": "确认删除该知识库？删除后数据无法恢复，请确认！",
"deleteFolderTips": "确认删除该文件夹及其包含的所有知识库？删除后数据无法恢复，请确认！",
"import csv tip": "请确保CSV为UTF-8格式，否则会乱码",
"test": {
......
projects/app/src/components/Layout/navbar.tsx
......@@ -12,6 +12,7 @@ import MyIcon from '../Icon';
import { useTranslation } from 'next-i18next';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import MyTooltip from '../MyTooltip';
import { getDocPath } from '@/web/common/system/doc';
export enum NavbarTypeEnum {
normal = 'normal',
......@@ -181,7 +182,7 @@ const Navbar = ({ unread }: { unread: number }) => {
mb={0}
color={'#9096a5'}
onClick={() => {
window.open(`${feConfigs.docUrl}/docs/intro`);
window.open(getDocPath('/docs/intro'));
}}
>
<MyIcon name={'common/courseLight'} width={'26px'} height={'26px'} />
......
projects/app/src/components/core/module/AIChatSettingsModal.tsx
......@@ -25,6 +25,7 @@ import { PromptTemplateItem } from '@fastgpt/global/core/ai/type.d';
import type { AIChatModuleProps } from '@fastgpt/global/core/module/node/type.d';
import type { AppSimpleEditConfigTemplateType } from '@fastgpt/global/core/app/type.d';
import { SimpleModeTemplate_FastGPT_Universal } from '@/global/core/app/constants';
import { getDocPath } from '@/web/common/system/doc';
const PromptTemplate = dynamic(() => import('@/components/PromptTemplate'));
......@@ -77,7 +78,7 @@ const AIChatSettingsModal = ({
{t('app.AI Advanced Settings')}
{feConfigs?.docUrl && (
<Link
href={`${feConfigs.docUrl}/docs/use-cases/ai_settings/`}
href={getDocPath('/docs/use-cases/ai_settings/')}
target={'_blank'}
ml={1}
textDecoration={'underline'}
......
projects/app/src/components/support/apikey/Table.tsx
......@@ -39,6 +39,7 @@ import MyModal from '@/components/MyModal';
import { useForm } from 'react-hook-form';
import { useRequest } from '@/web/common/hooks/useRequest';
import MyTooltip from '@/components/MyTooltip';
import { getDocPath } from '@/web/common/system/doc';
type EditProps = EditApiKeyProps & { _id?: string };
const defaultEditData: EditProps = {
......@@ -82,9 +83,9 @@ const ApiKeyTable = ({ tips, appId }: { tips: string; appId?: string }) => {
<Box fontSize={['md', 'xl']} fontWeight={'bold'}>
API 秘钥管理
</Box>
{feConfigs.docUrl && (
{feConfigs?.docUrl && (
<Link
href={feConfigs.openAPIDocUrl || `${feConfigs.docUrl}/docs/development/openapi`}
href={feConfigs.openAPIDocUrl || getDocPath('/docs/development/openapi')}
target={'_blank'}
ml={1}
color={'myBlue.600'}
......
projects/app/src/pages/account/components/Info.tsx
......@@ -30,6 +30,7 @@ import { useRouter } from 'next/router';
import MySelect from '@/components/Select';
import { formatPrice } from '@fastgpt/global/support/wallet/bill/tools';
import { putUpdateMemberName } from '@/web/support/user/team/api';
import { getDocPath } from '@/web/common/system/doc';
const TeamMenu = dynamic(() => import('@/components/support/user/team/TeamMenu'));
const PayModal = dynamic(() => import('./PayModal'), {
......@@ -261,7 +262,7 @@ const UserInfo = () => {
cursor={'pointer'}
userSelect={'none'}
onClick={() => {
window.open(`${feConfigs.docUrl}/docs/intro`);
window.open(getDocPath('/docs/intro'));
}}
>
<MyIcon name={'common/courseLight'} w={'18px'} />
......
projects/app/src/pages/api/core/app/create.ts
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
......
projects/app/src/pages/api/core/dataset/collection/delete.ts
......@@ -30,7 +30,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
// delete
await delCollectionRelevantData({
collectionIds: delIdList,
fileIds: collections.map((item) => String(item.metadata?.fileId)).filter(Boolean)
fileIds: collections.map((item) => item.metadata?.fileId).filter(Boolean)
});
// delete collection
......
projects/app/src/pages/api/core/dataset/collection/sync/link.ts 0 → 100644
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/dataset';
import { loadingOneChunkCollection } from '@fastgpt/service/core/dataset/collection/utils';
import { delCollectionRelevantData } from '@fastgpt/service/core/dataset/data/controller';
import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import { DatasetCollectionTypeEnum } from '@fastgpt/global/core/dataset/constant';
import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset';
import { createTrainingBill } from '@fastgpt/service/support/wallet/bill/controller';
import { BillSourceEnum } from '@fastgpt/global/support/wallet/bill/constants';
import { getQAModel, getVectorModel } from '@/service/core/ai/model';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
const { collectionId } = req.body as { collectionId: string };
if (!collectionId) {
throw new Error('CollectionIdId is required');
}
const { collection, tmbId } = await authDatasetCollection({
req,
authToken: true,
collectionId,
per: 'w'
});
if (collection.type !== DatasetCollectionTypeEnum.link || !collection.rawLink) {
return Promise.reject(DatasetErrEnum.unLinkCollection);
}
const vectorModelData = getVectorModel(collection.datasetId.vectorModel);
const agentModelData = getQAModel(collection.datasetId.agentModel);
// create training bill
const { billId } = await createTrainingBill({
teamId: collection.teamId,
tmbId,
appName: 'core.dataset.collection.Sync Collection',
billSource: BillSourceEnum.training,
vectorModel: vectorModelData.name,
agentModel: agentModelData.name
});
// create a collection and delete old
const { _id } = await MongoDatasetCollection.create({
parentId: collection.parentId,
teamId: collection.teamId,
tmbId: collection.tmbId,
datasetId: collection.datasetId._id,
type: collection.type,
name: collection.name,
createTime: collection.createTime,
trainingType: collection.trainingType,
chunkSize: collection.chunkSize,
fileId: collection.fileId,
rawLink: collection.rawLink,
metadata: collection.metadata
});
// start load
await loadingOneChunkCollection({
collectionId: _id,
tmbId,
billId
});
// delete old collection
await Promise.all([
delCollectionRelevantData({
collectionIds: [collection._id],
fileIds: collection.fileId ? [collection.fileId] : []
}),
MongoDatasetCollection.findByIdAndRemove(collection._id)
]);
jsonRes(res);
} catch (err) {
jsonRes(res, {
code: 500,
error: err
});
}
}
projects/app/src/pages/api/support/openapi/create.ts
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
......
projects/app/src/pages/api/support/openapi/delete.ts
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
......
projects/app/src/pages/api/support/wallet/bill/createTrainingBill.ts
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { MongoBill } from '@fastgpt/service/support/wallet/bill/schema';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { BillSourceEnum } from '@fastgpt/global/support/wallet/bill/constants';
import { CreateTrainingBillProps } from '@fastgpt/global/support/wallet/bill/api.d';
import { getQAModel, getVectorModel } from '@/service/core/ai/model';
import { createTrainingBill } from '@fastgpt/service/support/wallet/bill/controller';
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
......@@ -17,30 +17,17 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
const vectorModelData = getVectorModel(vectorModel);
const agentModelData = getQAModel(agentModel);
const { _id } = await MongoBill.create({
const { billId } = await createTrainingBill({
teamId,
tmbId,
appName: name,
source: BillSourceEnum.training,
list: [
{
moduleName: 'wallet.moduleName.index',
model: vectorModelData.name,
amount: 0,
tokenLen: 0
},
{
moduleName: 'wallet.moduleName.qa',
model: agentModelData.name,
amount: 0,
tokenLen: 0
}
],
total: 0
billSource: BillSourceEnum.training,
vectorModel: vectorModelData.name,
agentModel: agentModelData.name
});
jsonRes<string>(res, {
data: _id
data: billId
});
} catch (err) {
jsonRes(res, {
......
projects/app/src/pages/api/user/account/loginByPassword.ts
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { MongoUser } from '@fastgpt/service/support/user/schema';
......
projects/app/src/pages/api/user/account/loginout.ts
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { clearCookie } from '@fastgpt/service/support/permission/controller';
......
projects/app/src/pages/api/user/account/tokenLogin.ts
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
......
projects/app/src/pages/api/user/account/update.ts
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { MongoUser } from '@fastgpt/service/support/user/schema';
......
projects/app/src/pages/api/user/account/updatePasswordByOld.ts
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
......
projects/app/src/pages/api/v1/chat/getHistory.ts
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
......
projects/app/src/pages/app/detail/components/OutLink/Share.tsx
......@@ -44,6 +44,7 @@ import { feConfigs } from '@/web/common/system/staticData';
import MyTooltip from '@/components/MyTooltip';
import MyModal from '@/components/MyModal';
import dayjs from 'dayjs';
import { getDocPath } from '@/web/common/system/doc';
const Share = ({ appId }: { appId: string }) => {
const { t } = useTranslation();
......@@ -357,7 +358,7 @@ function EditLinkModal({
/>
</Flex>
<Link
href={`${feConfigs.docUrl}/docs/development/openapi/share`}
href={getDocPath('/docs/development/openapi/share')}
target={'_blank'}
fontSize={'sm'}
color={'myGray.500'}
......
projects/app/src/pages/components/Footer.tsx
......@@ -5,6 +5,7 @@ import { useTranslation } from 'next-i18next';
import Avatar from '@/components/Avatar';
import { useRouter } from 'next/router';
import CommunityModal from '@/components/CommunityModal';
import { getDocPath } from '@/web/common/system/doc';
const Footer = () => {
const { t } = useTranslation();
......@@ -47,7 +48,7 @@ const Footer = () => {
{
label: t('home.Footer Docs'),
onClick: () => {
window.open(`${feConfigs.docUrl}/docs/intro`, '_blank');
window.open(getDocPath('/docs/intro'), '_blank');
}
}
]
......
projects/app/src/pages/components/Navbar.tsx
......@@ -7,6 +7,7 @@ import Avatar from '@/components/Avatar';
import CommunityModal from '@/components/CommunityModal';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import MyIcon from '@/components/Icon';
import { getDocPath } from '@/web/common/system/doc';
const Navbar = () => {
const router = useRouter();
......@@ -47,7 +48,7 @@ const Navbar = () => {
label: t('home.Docs'),
key: 'docs',
onClick: () => {
window.open(`${feConfigs.docUrl}/docs/intro`);
window.open(getDocPath('/docs/intro'));
}
}
]
......
projects/app/src/pages/dataset/detail/components/CollectionCard.tsx
......@@ -22,7 +22,7 @@ import {
putDatasetCollectionById,
postDatasetCollection,
getDatasetCollectionPathById,
postWebsiteSync
postLinkCollectionSync
} from '@/web/core/dataset/api';
import { useQuery } from '@tanstack/react-query';
import { debounce } from 'lodash';
......@@ -61,7 +61,6 @@ import { useUserStore } from '@/web/support/user/useUserStore';
import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
import { useDatasetStore } from '@/web/core/dataset/store/dataset';
import { DatasetSchemaType } from '@fastgpt/global/core/dataset/type';
import { postCreateTrainingBill } from '@/web/support/wallet/bill/api';
const FileImportModal = dynamic(() => import('./Import/ImportModal'), {});
const WebSiteConfigModal = dynamic(() => import('./Import/WebsiteConfig'), {});
......@@ -78,11 +77,15 @@ const CollectionCard = () => {
const { isPc } = useSystemStore();
const { userInfo } = useUserStore();
const [searchText, setSearchText] = useState('');
const { datasetDetail, updateDataset, loadDatasetDetail } = useDatasetStore();
const { datasetDetail, updateDataset, startWebsiteSync, loadDatasetDetail } = useDatasetStore();
const { openConfirm, ConfirmModal } = useConfirm({
const { openConfirm: openDeleteConfirm, ConfirmModal: ConfirmDeleteModal } = useConfirm({
content: t('dataset.Confirm to delete the file')
});
const { openConfirm: openSyncConfirm, ConfirmModal: ConfirmSyncModal } = useConfirm({
content: t('core.dataset.collection.Start Sync Tip')
});
const {
isOpen: isOpenFileImportModal,
onOpen: onOpenFileImportModal,
......@@ -112,7 +115,7 @@ const CollectionCard = () => {
Pagination,
total,
getData,
isLoading,
isLoading: isGetting,
pageNum,
pageSize
} = usePagination<DatasetCollectionsListItemType>({
......@@ -231,27 +234,23 @@ const CollectionCard = () => {
const { mutate: onUpdateDatasetWebsiteConfig, isLoading: isUpdating } = useRequest({
mutationFn: async (websiteConfig: DatasetSchemaType['websiteConfig']) => {
onCloseWebsiteModal();
const [_, billId] = await Promise.all([
updateDataset({
id: datasetDetail._id,
websiteConfig,
status: DatasetStatusEnum.syncing
}),
postCreateTrainingBill({
name: 'core.dataset.training.Website Sync',
vectorModel: datasetDetail.vectorModel.model,
agentModel: datasetDetail.agentModel.model
})
]);
return billId;
},
onSuccess(billId: string) {
try {
postWebsiteSync({ datasetId: datasetDetail._id, billId });
} catch (error) {}
await updateDataset({
id: datasetDetail._id,
websiteConfig
});
return startWebsiteSync();
},
errorToast: t('common.Update Failed')
});
const { mutate: onclickStartSync, isLoading: isSyncing } = useRequest({
mutationFn: (collectionId: string) => {
return postLinkCollectionSync(collectionId);
},
onSuccess() {
getData(pageNum);
},
errorToast: t('core.dataset.error.Start Sync Failed')
});
const { data: paths = [] } = useQuery(['getDatasetCollectionPathById', parentId], () =>
getDatasetCollectionPathById(parentId)
......@@ -261,6 +260,16 @@ const CollectionCard = () => {
() => !!formatCollections.find((item) => item.trainingAmount > 0),
[formatCollections]
);
const isLoading = useMemo(
() =>
isCreating ||
isDeleting ||
isUpdating ||
isSyncing ||
(isGetting && collections.length === 0),
[collections.length, isCreating, isDeleting, isGetting, isSyncing, isUpdating]
);
useQuery(
['refreshCollection'],
() => {
......@@ -589,6 +598,23 @@ const CollectionCard = () => {
</MenuButton>
}
menuList={[
...(collection.type === DatasetCollectionTypeEnum.link
? [
{
child: (
<Flex alignItems={'center'}>
<MyIcon name={'common/refreshLight'} w={'14px'} mr={2} />
{t('core.dataset.collection.Sync')}
</Flex>
),
onClick: () =>
openSyncConfirm(() => {
console.log(collection._id);
onclickStartSync(collection._id);
})()
}
]
: []),
{
child: (
<Flex alignItems={'center'}>
......@@ -629,7 +655,7 @@ const CollectionCard = () => {
</Flex>
),
onClick: () =>
openConfirm(
openDeleteConfirm(
() => {
onDelCollection(collection._id);
},
......@@ -659,21 +685,38 @@ const CollectionCard = () => {
t('core.dataset.collection.Empty Tip')
) : (
<Flex>
{t('core.dataset.collection.Website Empty Tip')}
<Box textDecoration={'underline'} cursor={'pointer'} onClick={onOpenWebsiteModal}>
{t('core.dataset.collection.Click top config website')}
</Box>
{datasetDetail.status === DatasetStatusEnum.syncing && (
<>{t('core.dataset.status.syncing')}</>
)}
{datasetDetail.status === DatasetStatusEnum.active && (
<>
{!datasetDetail?.websiteConfig?.url ? (
<>
{t('core.dataset.collection.Website Empty Tip')}
{', '}
<Box
textDecoration={'underline'}
cursor={'pointer'}
onClick={onOpenWebsiteModal}
>
{t('core.dataset.collection.Click top config website')}
</Box>
</>
) : (
<>{t('core.dataset.website.UnValid Website Tip')}</>
)}
</>
)}
</Flex>
)
}
/>
)}
</TableContainer>
<Loading
loading={isCreating || isDeleting || isUpdating || (isLoading && collections.length === 0)}
/>
<Loading loading={isLoading} />
<ConfirmModal />
<ConfirmDeleteModal />
<ConfirmSyncModal />
<EditTitleModal />
<EditCreateVirtualFileModal />
{isOpenFileImportModal && (
......
projects/app/src/pages/dataset/detail/components/DataCard.tsx
......@@ -189,7 +189,7 @@ const DataCard = () => {
textDecoration={'none'}
/>
<Box fontSize={'sm'} color={'myGray.500'}>
文件ID:{' '}
{t('core.dataset.collection.id')}:{' '}
<Box as={'span'} userSelect={'all'}>
{collection?._id}
</Box>
......@@ -229,7 +229,7 @@ const DataCard = () => {
<Flex my={3} alignItems={'center'}>
<Box>
<Box as={'span'} fontSize={['md', 'lg']}>
{total}组
{t('core.dataset.data.Total Amount', { total })}
</Box>
</Box>
<Box flex={1} mr={1} />
......@@ -243,7 +243,7 @@ const DataCard = () => {
/>
}
w={['200px', '300px']}
placeholder="根据匹配知识，预期答案和来源进行搜索"
placeholder={t('core.dataset.data.Search data placeholder')}
value={searchText}
onChange={(e) => {
setSearchText(e.target.value);
......@@ -410,7 +410,7 @@ const DataCard = () => {
<Flex flexDirection={'column'} alignItems={'center'} pt={'10vh'}>
<MyIcon name="empty" w={'48px'} h={'48px'} color={'transparent'} />
<Box mt={2} color={'myGray.500'}>
内容空空的，快创建一个吧！
{t('core.dataset.data.Empty Tip')}
</Box>
</Flex>
)}
......
projects/app/src/pages/dataset/detail/components/Import/Chunk.tsx
......@@ -41,7 +41,7 @@ const ChunkImport = () => {
} = useImportStore();
const { openConfirm, ConfirmModal } = useConfirm({
content: `该任务无法终止，需要一定时间生成索引，请确认导入。如果余额不足，未完成的任务会被暂停，充值后可继续进行。`
content: t('core.dataset.import.Import Tip')
});
return (
......@@ -63,7 +63,11 @@ const ChunkImport = () => {
}
}}
>
<MyTooltip label={`范围: 100~${datasetDetail.vectorModel.maxToken}`}>
<MyTooltip
label={t('core.dataset.import.Chunk Range', {
max: datasetDetail.vectorModel.maxToken
})}
>
<NumberInput
ml={4}
defaultValue={chunkLen}
......@@ -87,20 +91,22 @@ const ChunkImport = () => {
{/* price */}
<Flex py={4} alignItems={'center'}>
<Box>
预估价格
{t('core.dataset.import.Estimated Price')}
<MyTooltip
label={`索引生成计费为: ${formatPrice(unitPrice, 1000)}/1k tokens`}
label={t('core.dataset.import.Estimated Price Tips', {
price: formatPrice(unitPrice, 1000)
})}
forceShow
>
<QuestionOutlineIcon ml={1} />
</MyTooltip>
</Box>
<Box ml={4}>{price}元</Box>
<Box ml={4}>{t('common.price.Amount', { amount: price, unit: '元' })}</Box>
</Flex>
<Flex mt={3}>
{showRePreview && (
<Button variant={'base'} mr={4} onClick={onReSplitChunks}>
重新生成预览
{t('core.dataset.import.Re Preview')}
</Button>
)}
<Button isDisabled={uploading} onClick={openConfirm(onclickUpload)}>
......
projects/app/src/pages/dataset/detail/components/Import/CreateFileModal.tsx
......@@ -37,20 +37,20 @@ const CreateFileModal = ({
>
<ModalBody>
<Box mb={1} fontSize={'sm'}>
文件名
{t('common.file.File Name')}
</Box>
<Input
mb={5}
{...register('filename', {
required: '文件名不能为空'
required: t('common.file.Filename Can not Be Empty')
})}
/>
<Box mb={1} fontSize={'sm'}>
文件内容
{t('common.file.File Content')}
</Box>
<Textarea
{...register('content', {
required: '文件内容不能为空'
required: t('common.file.File content can not be empty')
})}
rows={12}
whiteSpace={'nowrap'}
......@@ -59,10 +59,10 @@ const CreateFileModal = ({
</ModalBody>
<ModalFooter>
<Button variant={'base'} mr={4} onClick={onClose}>
取消
{t('common.Close')}
</Button>
<Button isLoading={isLoading} onClick={mutate}>
确认
{t('common.Confirm Create')}
</Button>
</ModalFooter>
</MyModal>
......
projects/app/src/pages/dataset/detail/components/Import/Csv.tsx
......@@ -16,7 +16,7 @@ const CsvImport = () => {
useImportStore();
const { openConfirm, ConfirmModal } = useConfirm({
content: `该任务无法终止，需要一定时间生成索引，请确认导入。如果余额不足，未完成的任务会被暂停，充值后可继续进行。`
content: t('core.dataset.import.Import Tip')
});
return (
......@@ -25,7 +25,7 @@ const CsvImport = () => {
fileExtension={fileExtension}
showUrlFetch={false}
fileTemplate={{
filename: 'csv 模板.csv',
filename: 'csv templates.csv',
value: csvTemplate,
type: 'text/csv'
}}
......@@ -33,7 +33,11 @@ const CsvImport = () => {
>
<Flex mt={3}>
<Button isDisabled={uploading} onClick={openConfirm(onclickUpload)}>
{uploading ? <Box>{Math.round((successChunks / totalChunks) * 100)}%</Box> : '确认导入'}
{uploading ? (
<Box>{Math.round((successChunks / totalChunks) * 100)}%</Box>
) : (
t('common.Confirm Import')
)}
</Button>
</Flex>
</SelectorContainer>
......
projects/app/src/pages/dataset/detail/components/Import/FileSelect.tsx
......@@ -137,7 +137,7 @@ const FileSelect = ({
if (extension === 'csv') {
const { header, data } = await readCsvContent(file);
if (header[0] !== 'index' || header[1] !== 'content') {
throw new Error('csv 文件格式有误,请确保 index 和 content 两列');
throw new Error(t('core.dataset.import.Csv format error'));
}
const filterData = data
......@@ -205,7 +205,7 @@ const FileSelect = ({
} catch (error: any) {
console.log(error);
toast({
title: getErrText(error, '解析文件失败'),
title: getErrText(error, t('common.file.Read File Error')),
status: 'error'
});
}
......
projects/app/src/pages/dataset/detail/components/Import/ImportModal.tsx
......@@ -10,7 +10,6 @@ const CsvImport = dynamic(() => import('./Csv'), {});
import MyModal from '@/components/MyModal';
import Provider from './Provider';
import { useDatasetStore } from '@/web/core/dataset/store/dataset';
import { qaModelList } from '@/web/common/system/staticData';
import {
DatasetCollectionTrainingModeEnum,
TrainingModeEnum
......@@ -96,20 +95,20 @@ const ImportData = ({
list={[
{
icon: 'indexImport',
title: '直接分段',
desc: '选择文本文件，直接将其按分段进行处理',
title: t('core.dataset.import.Chunk Split'),
desc: t('core.dataset.import.Chunk Split Tip'),
value: ImportTypeEnum.chunk
},
{
icon: 'qaImport',
title: 'QA拆分',
desc: '选择文本文件，让大模型自动生成问答对',
title: t('core.dataset.import.QA Import'),
desc: t('core.dataset.import.QA Import Tip'),
value: ImportTypeEnum.qa
},
{
icon: 'csvImport',
title: 'CSV 导入',
desc: '批量导入问答对，是最精准的数据',
title: t('core.dataset.import.CSV Import'),
desc: t('core.dataset.import.CSV Import Tip'),
value: ImportTypeEnum.csv
}
]}
......
projects/app/src/pages/dataset/detail/components/Import/Provider.tsx
......@@ -178,12 +178,12 @@ const Provider = ({
},
onSuccess(num) {
toast({
title: `共成功导入 ${num} 组数据，请耐心等待训练.`,
title: t('core.dataset.import.Import Success Tip', { num }),
status: 'success'
});
onUploadSuccess();
},
errorToast: '导入文件失败'
errorToast: t('core.dataset.import.Import Failed')
});
const onReSplitChunks = useCallback(async () => {
......@@ -212,10 +212,10 @@ const Provider = ({
} catch (error) {
toast({
status: 'warning',
title: getErrText(error, '文本分段异常')
title: getErrText(error, t('core.dataset.import.Set Chunk Error'))
});
}
}, [chunkLen, toast]);
}, [chunkLen, chunkOverlapRatio, t, toast]);
const reset = useCallback(() => {
setFiles([]);
......@@ -255,6 +255,7 @@ export default React.memo(Provider);
export const PreviewFileOrChunk = () => {
const theme = useTheme();
const { t } = useTranslation();
const { setFiles, previewFile, setPreviewFile, setReShowRePreview, totalChunks, files } =
useImportStore();
......@@ -312,11 +313,11 @@ export const PreviewFileOrChunk = () => {
<Box pt={[3, 6]}>
<Flex px={[4, 8]} alignItems={'center'}>
<Box fontSize={['lg', 'xl']} fontWeight={'bold'}>
分段预览({totalChunks}组)
{t('core.dataset.import.Total Chunk Preview', { totalChunks })}
</Box>
{totalChunks > 50 && (
<Box ml={2} fontSize={'sm'} color={'myhGray.500'}>
仅展示部分
{t('core.dataset.import.Only Show First 50 Chunk')}
</Box>
)}
</Flex>
......
projects/app/src/pages/dataset/detail/components/Import/QA.tsx
......@@ -7,10 +7,12 @@ import { QuestionOutlineIcon } from '@chakra-ui/icons';
import { Prompt_AgentQA } from '@/global/core/prompt/agent';
import { useImportStore, SelectorContainer, PreviewFileOrChunk } from './Provider';
import { useDatasetStore } from '@/web/core/dataset/store/dataset';
import { useTranslation } from 'next-i18next';
const fileExtension = '.txt, .docx, .pdf, .md';
const QAImport = () => {
const { t } = useTranslation();
const { datasetDetail } = useDatasetStore();
const agentModel = datasetDetail.agentModel;
const unitPrice = agentModel?.price || 3;
......@@ -27,7 +29,7 @@ const QAImport = () => {
} = useImportStore();
const { openConfirm, ConfirmModal } = useConfirm({
content: `该任务无法终止！导入后会自动调用大模型生成问答对，会有一些细节丢失，请确认！如果余额不足，未完成的任务会被暂停。`
content: t('core.dataset.import.Import Tip')
});
const [prompt, setPrompt] = useState(Prompt_AgentQA.description);
......@@ -38,7 +40,7 @@ const QAImport = () => {
{/* prompt */}
<Box p={3} bg={'myWhite.600'} borderRadius={'md'}>
<Box mb={1} fontWeight={'bold'}>
QA 拆分引导词
{t('core.dataset.collection.QA Prompt')}
</Box>
<Box whiteSpace={'pre-wrap'} fontSize={'sm'}>
<Textarea
......@@ -55,24 +57,30 @@ const QAImport = () => {
{/* price */}
<Flex py={5} alignItems={'center'}>
<Box>
预估价格
{t('core.dataset.import.Estimated Price')}
<MyTooltip
label={`索引生成计费为: ${formatPrice(unitPrice, 1000)}/1k tokens`}
label={t('core.dataset.import.Estimated Price Tips', {
price: formatPrice(unitPrice, 1000)
})}
forceShow
>
<QuestionOutlineIcon ml={1} />
</MyTooltip>
</Box>
<Box ml={4}>{price}元</Box>
<Box ml={4}>{t('common.price.Amount', { amount: price, unit: '元' })}</Box>
</Flex>
<Flex mt={3}>
{showRePreview && (
<Button variant={'base'} mr={4} onClick={onReSplitChunks}>
重新生成预览
{t('core.dataset.import.Re Preview')}
</Button>
)}
<Button isDisabled={uploading} onClick={openConfirm(() => onclickUpload({ prompt }))}>
{uploading ? <Box>{Math.round((successChunks / totalChunks) * 100)}%</Box> : '确认导入'}
{uploading ? (
<Box>{Math.round((successChunks / totalChunks) * 100)}%</Box>
) : (
t('common.Confirm Import')
)}
</Button>
</Flex>
</SelectorContainer>
......
projects/app/src/pages/dataset/detail/components/Info.tsx
import React, {
useCallback,
useState,
useRef,
forwardRef,
useImperativeHandle,
ForwardedRef
} from 'react';
import React, { useCallback, useState, useMemo } from 'react';
import { useRouter } from 'next/router';
import { Box, Flex, Button, FormControl, IconButton, Input, Textarea } from '@chakra-ui/react';
import { QuestionOutlineIcon, DeleteIcon } from '@chakra-ui/icons';
import { Box, Flex, Button, IconButton, Input, Textarea } from '@chakra-ui/react';
import { DeleteIcon } from '@chakra-ui/icons';
import { delDatasetById } from '@/web/core/dataset/api';
import { useSelectFile } from '@/web/common/file/hooks/useSelectFile';
import { useToast } from '@/web/common/hooks/useToast';
import { useDatasetStore } from '@/web/core/dataset/store/dataset';
import { useConfirm } from '@/web/common/hooks/useConfirm';
import { UseFormReturn } from 'react-hook-form';
import { useForm } from 'react-hook-form';
import { compressImgFileAndUpload } from '@/web/common/file/controller';
import type { DatasetItemType } from '@fastgpt/global/core/dataset/type.d';
import Avatar from '@/components/Avatar';
......@@ -23,26 +16,22 @@ import { useTranslation } from 'next-i18next';
import PermissionRadio from '@/components/support/permission/Radio';
import MySelect from '@/components/Select';
import { qaModelList } from '@/web/common/system/staticData';
import { useRequest } from '@/web/common/hooks/useRequest';
const Info = ({
datasetId,
form
}: {
datasetId: string;
form: UseFormReturn<DatasetItemType, any>;
}) => {
const Info = ({ datasetId }: { datasetId: string }) => {
const { t } = useTranslation();
const { getValues, formState, setValue, register, handleSubmit } = form;
const InputRef = useRef<HTMLInputElement>(null);
const { datasetDetail, loadDatasets, updateDataset } = useDatasetStore();
const { getValues, setValue, register, handleSubmit } = useForm<DatasetItemType>({
defaultValues: datasetDetail
});
const { toast } = useToast();
const router = useRouter();
const [btnLoading, setBtnLoading] = useState(false);
const [refresh, setRefresh] = useState(false);
const { openConfirm, ConfirmModal } = useConfirm({
content: '确认删除该知识库？数据将无法恢复，请确认！'
content: t('core.dataset.Delete Confirm'),
type: 'delete'
});
const { File, onOpen: onOpenSelectFile } = useSelectFile({
......@@ -50,107 +39,68 @@ const Info = ({
multiple: false
});
const { datasetDetail, loadDatasets, updateDataset } = useDatasetStore();
/* 点击删除 */
const onclickDelKb = useCallback(async () => {
setBtnLoading(true);
try {
await delDatasetById(datasetId);
toast({
title: '删除成功',
status: 'success'
});
const { mutate: onclickDelete, isLoading: isDeleting } = useRequest({
mutationFn: () => {
return delDatasetById(datasetId);
},
onSuccess() {
router.replace(`/dataset/list`);
await loadDatasets();
} catch (err: any) {
toast({
title: err?.message || '删除失败',
status: 'error'
});
}
setBtnLoading(false);
}, [setBtnLoading, datasetId, toast, router, loadDatasets]);
},
successToast: t('common.Delete Success'),
errorToast: t('common.Delete Failed')
});
const saveSubmitSuccess = useCallback(
async (data: DatasetItemType) => {
setBtnLoading(true);
try {
await updateDataset({
id: datasetId,
...data
});
toast({
title: '更新成功',
status: 'success'
});
loadDatasets();
} catch (err: any) {
toast({
title: err?.message || '更新失败',
status: 'error'
});
}
setBtnLoading(false);
const { mutate: onclickSave, isLoading: isSaving } = useRequest({
mutationFn: (data: DatasetItemType) => {
return updateDataset({
id: datasetId,
...data
});
},
[updateDataset, datasetId, toast, loadDatasets]
);
const saveSubmitError = useCallback(() => {
// deep search message
const deepSearch = (obj: any): string => {
if (!obj) return '提交表单错误';
if (!!obj.message) {
return obj.message;
}
return deepSearch(Object.values(obj)[0]);
};
toast({
title: deepSearch(formState.errors),
status: 'error',
duration: 4000,
isClosable: true
});
}, [formState.errors, toast]);
onSuccess() {
loadDatasets();
},
successToast: t('common.Update Success'),
errorToast: t('common.Update Failed')
});
const onSelectFile = useCallback(
async (e: File[]) => {
const { mutate: onSelectFile, isLoading: isSelecting } = useRequest({
mutationFn: (e: File[]) => {
const file = e[0];
if (!file) return;
try {
const src = await compressImgFileAndUpload({
file,
maxW: 300,
maxH: 300
});
if (!file) return Promise.resolve(null);
return compressImgFileAndUpload({
file,
maxW: 300,
maxH: 300
});
},
onSuccess(src: string | null) {
if (src) {
setValue('avatar', src);
setRefresh((state) => !state);
} catch (err: any) {
toast({
title: typeof err === 'string' ? err : '头像选择异常',
status: 'warning'
});
}
},
[setRefresh, setValue, toast]
);
errorToast: t('common.avatar.Select Failed')
});
const btnLoading = useMemo(() => isDeleting || isSaving, [isDeleting, isSaving]);
return (
<Box py={5} px={[5, 10]}>
<Flex mt={5} w={'100%'} alignItems={'center'}>
<Box flex={['0 0 90px', '0 0 160px']} w={0}>
知识库 ID
{t('core.dataset.Dataset ID')}
</Box>
<Box flex={1}>{datasetDetail._id}</Box>
</Flex>
<Flex mt={5} w={'100%'} alignItems={'center'}>
<Box flex={['0 0 90px', '0 0 160px']} w={0}>
知识库头像
{t('core.dataset.Avatar')}
</Box>
<Box flex={[1, '0 0 300px']}>
<MyTooltip label={'点击切换头像'}>
<MyTooltip label={t('common.avatar.Select Avatar')}>
<Avatar
m={'auto'}
src={getValues('avatar')}
......@@ -164,19 +114,13 @@ const Info = ({
</Flex>
<Flex mt={8} w={'100%'} alignItems={'center'}>
<Box flex={['0 0 90px', '0 0 160px']} w={0}>
知识库名称
{t('core.dataset.Name')}
</Box>
<Input
flex={[1, '0 0 300px']}
maxLength={30}
{...register('name', {
required: '知识库名称不能为空'
})}
/>
<Input flex={[1, '0 0 300px']} maxLength={30} {...register('name')} />
</Flex>
<Flex mt={8} w={'100%'} alignItems={'center'}>
<Box flex={['0 0 90px', '0 0 160px']} w={0}>
索引模型
{t('core.ai.model.Vector Model')}
</Box>
<Box flex={[1, '0 0 300px']}>{getValues('vectorModel').name}</Box>
</Flex>
......@@ -188,7 +132,7 @@ const Info = ({
</Flex>
<Flex mt={6} alignItems={'center'}>
<Box flex={['0 0 90px', '0 0 160px']} w={0}>
{t('dataset.Agent Model')}
{t('core.ai.model.Dataset Agent Model')}
</Box>
<Box flex={[1, '0 0 300px']}>
<MySelect
......@@ -234,9 +178,9 @@ const Info = ({
isLoading={btnLoading}
mr={4}
w={'100px'}
onClick={handleSubmit(saveSubmitSuccess, saveSubmitError)}
onClick={handleSubmit((data) => onclickSave(data))}
>
保存
{t('common.Save')}
</Button>
{datasetDetail.isOwner && (
<IconButton
......@@ -249,7 +193,7 @@ const Info = ({
color: 'red.600',
borderColor: 'red.600'
}}
onClick={openConfirm(onclickDelKb)}
onClick={openConfirm(onclickDelete)}
/>
)}
</Flex>
......
projects/app/src/pages/dataset/detail/components/InputDataModal.tsx
......@@ -28,6 +28,7 @@ import SideTabs from '@/components/SideTabs';
import { useLoading } from '@/web/common/hooks/useLoading';
import DeleteIcon from '@/components/Icon/delete';
import { defaultCollectionDetail } from '@/constants/dataset';
import { getDocPath } from '@/web/common/system/doc';
export type RawSourceTextProps = BoxProps & {
sourceName?: string;
......@@ -122,7 +123,7 @@ const InputDataModal = ({
}
if (countPromptTokens(e.q) >= maxToken) {
return toast({
title: '总长度超长了',
title: t('core.dataset.data.Too Long'),
status: 'warning'
});
}
......@@ -214,7 +215,7 @@ const InputDataModal = ({
return openConfirm(onDeleteData)();
}
if (e === TabEnum.doc) {
return window.open(`${feConfigs.docUrl}/docs/use-cases/datasetengine`, '_blank');
return window.open(getDocPath('/docs/use-cases/datasetengine'), '_blank');
}
setCurrentTab(e);
}}
......@@ -236,17 +237,15 @@ const InputDataModal = ({
<Box as="span" color={'red.600'}>
*
</Box>
{'相关数据内容'}
{t('core.dataset.data.Data Content')}
</Box>
<MyTooltip
label={'该输入框是必填项\n该内容通常是对于知识点的描述，也可以是用户的问题。'}
>
<MyTooltip label={t('core.dataset.data.Data Content Tip')}>
<QuestionOutlineIcon ml={1} />
</MyTooltip>
</Flex>
<Textarea
mt={1}
placeholder={`该输入框是必填项，该内容通常是对于知识点的描述，也可以是用户的问题，最多 ${maxToken} 字。`}
placeholder={t('core.dataset.data.Data Content Placeholder', { maxToken })}
maxLength={maxToken}
rows={12}
bg={'myWhite.400'}
......@@ -257,20 +256,16 @@ const InputDataModal = ({
</Box>
<Box mt={5}>
<Flex>
<Box>{'辅助数据'}</Box>
<MyTooltip
label={
'该部分为可选填项\n该内容通常是为了与前面的数据内容配合，构建结构化提示词，用于特殊场景'
}
>
<Box>{t('core.dataset.data.Auxiliary Data')}</Box>
<MyTooltip label={t('core.dataset.data.Auxiliary Data Tip')}>
<QuestionOutlineIcon ml={1} />
</MyTooltip>
</Flex>
<Textarea
mt={1}
placeholder={`该部分为可选填项, 通常是为了与前面的【数据内容】配合，构建结构化提示词，用于特殊场景，最多 ${
maxToken * 1.5
} 字。`}
placeholder={t('core.dataset.data.Auxiliary Data Placeholder', {
maxToken: maxToken * 1.5
})}
bg={'myWhite.400'}
rows={12}
maxLength={maxToken * 1.5}
......@@ -310,10 +305,7 @@ const InputDataModal = ({
/>
</Flex>
{index.defaultIndex ? (
<Box>
无法编辑，默认索引会使用【相关数据内容】与【辅助数据】的文本直接生成索引，如不需要默认索引，可删除。
每条数据必须保证有一个以上索引，所有索引被删除后，会自动生成默认索引。
</Box>
<Box>{t('core.dataset.data.Default Index Tip')}</Box>
) : (
<Textarea
maxLength={maxToken}
......@@ -369,7 +361,7 @@ const InputDataModal = ({
// @ts-ignore
onClick={handleSubmit(defaultValue.id ? onUpdateData : sureImportData)}
>
{defaultValue.id ? '确认变更' : '确认导入'}
{defaultValue.id ? t('common.Confirm Update') : t('common.Confirm Import')}
</Button>
</MyTooltip>
</Flex>
......
projects/app/src/pages/dataset/detail/components/Test.tsx
......@@ -236,7 +236,7 @@ const Test = ({ datasetId }: { datasetId: string }) => {
border={theme.borders.base}
_notLast={{ mb: 2 }}
cursor={'pointer'}
title={'编辑'}
title={t('common.Edit')}
onClick={async () => {
try {
setLoading(true);
......
projects/app/src/pages/dataset/detail/index.tsx
import React, { useCallback, useRef } from 'react';
import React, { useCallback } from 'react';
import { useRouter } from 'next/router';
import { Box, Flex, IconButton, useTheme } from '@chakra-ui/react';
import { useToast } from '@/web/common/hooks/useToast';
import { useForm } from 'react-hook-form';
import { useQuery } from '@tanstack/react-query';
import type { DatasetItemType } from '@fastgpt/global/core/dataset/type.d';
import { getErrText } from '@fastgpt/global/common/error/utils';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import Tabs from '@/components/Tabs';
......@@ -24,7 +22,13 @@ import Script from 'next/script';
import CollectionCard from './components/CollectionCard';
import { useDatasetStore } from '@/web/core/dataset/store/dataset';
import { useUserStore } from '@/web/support/user/useUserStore';
import { DatasetTypeMap } from '../../../../../../packages/global/core/dataset/constant';
import {
DatasetStatusEnum,
DatasetTypeEnum,
DatasetTypeMap
} from '@fastgpt/global/core/dataset/constant';
import { useConfirm } from '@/web/common/hooks/useConfirm';
import { useRequest } from '@/web/common/hooks/useRequest';
const DataCard = dynamic(() => import('./components/DataCard'), {
ssr: false
......@@ -46,17 +50,26 @@ const Detail = ({ datasetId, currentTab }: { datasetId: string; currentTab: `${T
const { toast } = useToast();
const router = useRouter();
const { isPc } = useSystemStore();
const { datasetDetail, loadDatasetDetail } = useDatasetStore();
const { datasetDetail, loadDatasetDetail, startWebsiteSync } = useDatasetStore();
const { userInfo } = useUserStore();
const tabList = [
{ label: '数据集', id: TabEnum.collectionCard, icon: 'overviewLight' },
{ label: '搜索测试', id: TabEnum.test, icon: 'kbTest' },
{ label: t('core.dataset.Dataset'), id: TabEnum.collectionCard, icon: 'overviewLight' },
{ label: t('core.dataset.test.Search Test'), id: TabEnum.test, icon: 'kbTest' },
...(userInfo?.team.canWrite && datasetDetail.isOwner
? [{ label: '配置', id: TabEnum.info, icon: 'settingLight' }]
? [{ label: t('common.Config'), id: TabEnum.info, icon: 'settingLight' }]
: [])
];
const { ConfirmModal: ConfirmSyncModal, openConfirm: openConfirmSync } = useConfirm({
type: 'common'
});
const { mutate: onUpdateDatasetWebsiteConfig, isLoading: isUpdating } = useRequest({
mutationFn: () => startWebsiteSync(),
errorToast: t('common.Update Failed')
});
const setCurrentTab = useCallback(
(tab: `${TabEnum}`) => {
router.replace({
......@@ -69,18 +82,11 @@ const Detail = ({ datasetId, currentTab }: { datasetId: string; currentTab: `${T
[datasetId, router]
);
const form = useForm<DatasetItemType>({
defaultValues: datasetDetail
});
useQuery([datasetId], () => loadDatasetDetail(datasetId), {
onSuccess(res) {
form.reset(res);
},
onError(err: any) {
router.replace(`/dataset/list`);
toast({
title: getErrText(err, '获取知识库异常'),
title: getErrText(err, t('common.Load Failed')),
status: 'error'
});
}
......@@ -116,7 +122,26 @@ const Detail = ({ datasetId, currentTab }: { datasetId: string; currentTab: `${T
mr={1}
w={'16px'}
/>
<Box>{t(DatasetTypeMap[datasetDetail.type]?.label)}</Box>
<Box flex={1}>{t(DatasetTypeMap[datasetDetail.type]?.label)}</Box>
{datasetDetail.type === DatasetTypeEnum.websiteDataset &&
datasetDetail.status === DatasetStatusEnum.active && (
<MyTooltip label={t('core.dataset.website.Start Sync')}>
<MyIcon
mt={1}
name={'common/refreshLight'}
w={'12px'}
color={'myGray.500'}
cursor={'pointer'}
onClick={() =>
openConfirmSync(
onUpdateDatasetWebsiteConfig,
undefined,
t('core.dataset.website.Confirm Create Tips')
)()
}
/>
</MyTooltip>
)}
</Flex>
)}
<SideTabs
......@@ -188,11 +213,12 @@ const Detail = ({ datasetId, currentTab }: { datasetId: string; currentTab: `${T
{currentTab === TabEnum.collectionCard && <CollectionCard />}
{currentTab === TabEnum.dataCard && <DataCard />}
{currentTab === TabEnum.test && <Test datasetId={datasetId} />}
{currentTab === TabEnum.info && <Info datasetId={datasetId} form={form} />}
{currentTab === TabEnum.info && <Info datasetId={datasetId} />}
</Box>
)}
</Flex>
</PageContainer>
<ConfirmSyncModal isLoading={isUpdating} />
</>
);
};
......
projects/app/src/pages/dataset/list/component/CreateModal.tsx
......@@ -57,7 +57,7 @@ const CreateModal = ({ onClose, parentId }: { onClose: () => void; parentId?: st
setRefresh((state) => !state);
} catch (err: any) {
toast({
title: getErrText(err, '头像选择异常'),
title: getErrText(err, t('common.avatar.Select Failed')),
status: 'warning'
});
}
......@@ -71,8 +71,8 @@ const CreateModal = ({ onClose, parentId }: { onClose: () => void; parentId?: st
const id = await postCreateDataset(data);
return id;
},
successToast: '创建成功',
errorToast: '创建知识库出现意外',
successToast: t('common.Create Success'),
errorToast: t('common.Create Failed'),
onSuccess(id) {
router.push(`/dataset/detail?datasetId=${id}`);
}
......@@ -122,10 +122,10 @@ const CreateModal = ({ onClose, parentId }: { onClose: () => void; parentId?: st
</>
<Box mt={5}>
<Box color={'myGray.800'} fontWeight={'bold'}>
取个名字
{t('common.Set Name')}
</Box>
<Flex mt={1} alignItems={'center'}>
<MyTooltip label={'点击设置头像'}>
<MyTooltip label={t('common.avatar.Select Avatar')}>
<Avatar
flexShrink={0}
src={getValues('avatar')}
......@@ -143,14 +143,12 @@ const CreateModal = ({ onClose, parentId }: { onClose: () => void; parentId?: st
bg={'myWhite.600'}
placeholder={t('common.Name')}
maxLength={30}
{...register('name', {
required: '知识库名称不能为空~'
})}
{...register('name')}
/>
</Flex>
</Box>
<Flex mt={6} alignItems={'center'}>
<Box flex={'0 0 100px'}>索引模型</Box>
<Box flex={'0 0 100px'}>{t('core.ai.model.Vector Model')}</Box>
<Box flex={1}>
<MySelect
w={'100%'}
......@@ -167,7 +165,7 @@ const CreateModal = ({ onClose, parentId }: { onClose: () => void; parentId?: st
</Box>
</Flex>
<Flex mt={6} alignItems={'center'}>
<Box flex={'0 0 100px'}>{t('dataset.Agent Model')}</Box>
<Box flex={'0 0 100px'}>{t('core.ai.model.Dataset Agent Model')}</Box>
<Box flex={1}>
<MySelect
w={'100%'}
......
projects/app/src/pages/dataset/list/component/MoveModal.tsx
......@@ -41,7 +41,7 @@ const MoveModal = ({
() => [
{
parentId: '',
parentName: t('dataset.My Dataset')
parentName: t('core.dataset.My Dataset')
},
...(data?.[1] || [])
],
......@@ -95,7 +95,7 @@ const MoveModal = ({
))}
</Flex>
) : (
<Box>我的知识库</Box>
<Box>{t('core.dataset.My Dataset')}</Box>
)}
</>
}
......
projects/app/src/pages/dataset/list/index.tsx
......@@ -57,8 +57,8 @@ const Kb = () => {
const DeleteTipsMap = useRef({
[DatasetTypeEnum.folder]: t('dataset.deleteFolderTips'),
[DatasetTypeEnum.dataset]: t('dataset.deleteDatasetTips'),
[DatasetTypeEnum.websiteDataset]: t('core.dataset.Delete Website Tips')
[DatasetTypeEnum.dataset]: t('core.dataset.Delete Confirm'),
[DatasetTypeEnum.websiteDataset]: t('core.dataset.Delete Confirm')
});
const { openConfirm, ConfirmModal } = useConfirm({
......@@ -143,7 +143,7 @@ const Kb = () => {
<Flex flex={1} alignItems={'center'}>
<Image src={'/imgs/module/db.png'} alt={''} mr={2} h={'24px'} />
<Box className="textlg" letterSpacing={1} fontSize={'24px'} fontWeight={'bold'}>
{t('dataset.My Dataset')}
{t('core.dataset.My Dataset')}
</Box>
</Flex>
}
......
projects/app/src/pages/login/components/LoginForm.tsx
......@@ -11,6 +11,7 @@ import { feConfigs } from '@/web/common/system/staticData';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import MyIcon from '@/components/Icon';
import { customAlphabet } from 'nanoid';
import { getDocPath } from '@/web/common/system/doc';
const nanoid = customAlphabet('abcdefghijklmnopqrstuvwxyz1234567890', 8);
interface Props {
......@@ -146,7 +147,7 @@ const LoginForm = ({ setPageType, loginSuccess }: Props) => {
<Box textAlign={'center'} mt={2} fontSize={'sm'}>
使用即代表你同意我们的{' '}
<Link
href={`${feConfigs.docUrl}/docs/intro/#%e5%85%8d%e8%b4%a3%e5%a3%b0%e6%98%8e`}
href={getDocPath('/docs/agreement/disclaimer/')}
target={'_blank'}
color={'myBlue.600'}
>
......
projects/app/src/pages/tools/index.tsx
......@@ -6,6 +6,7 @@ import { useRouter } from 'next/router';
import { feConfigs } from '@/web/common/system/staticData';
import { serviceSideProps } from '@/web/common/utils/i18n';
import { useTranslation } from 'next-i18next';
import { getDocPath } from '@/web/common/system/doc';
const Tools = () => {
const { t } = useTranslation();
......@@ -44,7 +45,7 @@ const Tools = () => {
{
icon: 'common/courseLight',
label: '使用文档',
link: `${feConfigs.docUrl}/docs/intro`
link: getDocPath('/docs/intro')
}
]
: [])
......
projects/app/src/service/events/generateQA.ts
......@@ -12,6 +12,8 @@ import { pushDataToDatasetCollection } from '@/pages/api/core/dataset/data/pushD
import { getErrText } from '@fastgpt/global/common/error/utils';
import { authTeamBalance } from '../support/permission/auth/bill';
import type { PushDatasetDataChunkProps } from '@fastgpt/global/core/dataset/api.d';
import { UserErrEnum } from '@fastgpt/global/common/error/code/user';
import { lockTrainingDataByTeamId } from '@fastgpt/service/core/dataset/training/controller';
const reduceQueue = (retry = false) => {
global.qaQueueLen = global.qaQueueLen > 0 ? global.qaQueueLen - 1 : 0;
......@@ -91,26 +93,22 @@ export async function generateQA(): Promise<any> {
// auth balance
try {
await authTeamBalance(data.teamId);
} catch (error) {
// send inform and lock data
try {
sendOneInform({
type: 'system',
title: '文本训练任务中止',
content:
'该团队账号余额不足，文本训练任务中止，重新充值后将会继续。暂停的任务将在 7 天后被删除。',
tmbId: data.tmbId
});
console.log('余额不足，暂停【QA】生成任务');
await MongoDatasetTraining.updateMany(
{
teamId: data.teamId
},
{
lockTime: new Date('2999/5/5')
}
);
} catch (error) {}
} catch (error: any) {
if (error?.statusText === UserErrEnum.balanceNotEnough) {
// send inform and lock data
try {
sendOneInform({
type: 'system',
title: '文本训练任务中止',
content:
'该团队账号余额不足，文本训练任务中止，重新充值后将会继续。暂停的任务将在 7 天后被删除。',
tmbId: data.tmbId
});
console.log('余额不足，暂停【QA】生成任务');
lockTrainingDataByTeamId(data.teamId);
} catch (error) {}
}
reduceQueue();
return generateQA();
}
......
projects/app/src/service/events/generateVector.ts
......@@ -6,6 +6,8 @@ import { addLog } from '@fastgpt/service/common/mongo/controller';
import { getErrText } from '@fastgpt/global/common/error/utils';
import { authTeamBalance } from '@/service/support/permission/auth/bill';
import { pushGenerateVectorBill } from '@/service/support/wallet/bill/push';
import { UserErrEnum } from '@fastgpt/global/common/error/code/user';
import { lockTrainingDataByTeamId } from '@fastgpt/service/core/dataset/training/controller';
const reduceQueue = (retry = false) => {
global.vectorQueueLen = global.vectorQueueLen > 0 ? global.vectorQueueLen - 1 : 0;
......@@ -93,26 +95,22 @@ export async function generateVector(): Promise<any> {
// auth balance
try {
await authTeamBalance(data.teamId);
} catch (error) {
// send inform and lock data
try {
sendOneInform({
type: 'system',
title: '文本训练任务中止',
content:
'该团队账号余额不足，文本训练任务中止，重新充值后将会继续。暂停的任务将在 7 天后被删除。',
tmbId: data.tmbId
});
console.log('余额不足，暂停【向量】生成任务');
await MongoDatasetTraining.updateMany(
{
teamId: data.teamId
},
{
lockTime: new Date('2999/5/5')
}
);
} catch (error) {}
} catch (error: any) {
if (error?.statusText === UserErrEnum.balanceNotEnough) {
// send inform and lock data
try {
sendOneInform({
type: 'system',
title: '文本训练任务中止',
content:
'该团队账号余额不足，文本训练任务中止，重新充值后将会继续。暂停的任务将在 7 天后被删除。',
tmbId: data.tmbId
});
console.log('余额不足，暂停【向量】生成任务');
lockTrainingDataByTeamId(data.teamId);
} catch (error) {}
}
reduceQueue();
return generateVector();
}
......
projects/app/src/service/moduleDispatch/chat/oneapi.ts
......@@ -76,7 +76,8 @@ export const dispatchChatCompletion = async (props: ChatProps): Promise<ChatResp
quoteTemplate
});
if (modelConstantsData.censor) {
// censor model and system key
if (modelConstantsData.censor && !user.openaiAccount?.key) {
await postTextCensor({
text: `${systemPrompt}
${quoteText}
......
projects/app/src/web/common/hooks/useConfirm.tsx
......@@ -57,7 +57,7 @@ export const useConfirm = (props?: {
[onOpen]
),
ConfirmModal: useCallback(
() => (
({ isLoading }: { isLoading?: boolean }) => (
<MyModal
isOpen={isOpen}
onClose={onClose}
......@@ -82,6 +82,7 @@ export const useConfirm = (props?: {
<Button
{...(bg && { bg: `${bg} !important` })}
ml={4}
isLoading={isLoading}
onClick={() => {
onClose();
typeof confirmCb.current === 'function' && confirmCb.current();
......
projects/app/src/web/common/system/doc.ts 0 → 100644
import { feConfigs } from './staticData';
export const getDocPath = (path: string) => {
if (!feConfigs?.docUrl) return '';
if (feConfigs.docUrl.endsWith('/')) return feConfigs.docUrl;
return feConfigs.docUrl + path;
};
projects/app/src/web/core/dataset/api.ts
......@@ -46,6 +46,11 @@ export const putDatasetById = (data: DatasetUpdateBody) => PUT<void>(`/core/data
export const delDatasetById = (id: string) => DELETE(`/core/dataset/delete?id=${id}`);
export const postWebsiteSync = (data: PostWebsiteSyncParams) =>
POST(`/plusApi/core/dataset/websiteSync`, data, {
timeout: 600000
}).catch();
export const getCheckExportLimit = (datasetId: string) =>
GET(`/core/dataset/checkExportLimit`, { datasetId });
......@@ -66,10 +71,8 @@ export const putDatasetCollectionById = (data: UpdateDatasetCollectionParams) =>
POST(`/core/dataset/collection/update`, data);
export const delDatasetCollectionById = (params: { collectionId: string }) =>
DELETE(`/core/dataset/collection/delete`, params);
export const postWebsiteSync = (data: PostWebsiteSyncParams) =>
POST(`/plusApi/core/dataset/websiteSync`, data, {
timeout: 600000
}).catch();
export const postLinkCollectionSync = (collectionId: string) =>
POST(`/core/dataset/collection/sync/link`, { collectionId });
/* =============================== data ==================================== */
/* get dataset list */
......
projects/app/src/web/core/dataset/store/dataset.ts
......@@ -2,9 +2,17 @@ import { create } from 'zustand';
import { devtools, persist } from 'zustand/middleware';
import { immer } from 'zustand/middleware/immer';
import type { DatasetItemType, DatasetListItemType } from '@fastgpt/global/core/dataset/type.d';
import { getAllDataset, getDatasets, getDatasetById, putDatasetById } from '@/web/core/dataset/api';
import {
getAllDataset,
getDatasets,
getDatasetById,
putDatasetById,
postWebsiteSync
} from '@/web/core/dataset/api';
import { defaultDatasetDetail } from '@/constants/dataset';
import type { DatasetUpdateBody } from '@fastgpt/global/core/dataset/api.d';
import { DatasetStatusEnum } from '@fastgpt/global/core/dataset/constant';
import { postCreateTrainingBill } from '@/web/support/wallet/bill/api';
type State = {
allDatasets: DatasetListItemType[];
......@@ -15,6 +23,7 @@ type State = {
datasetDetail: DatasetItemType;
loadDatasetDetail: (id: string, init?: boolean) => Promise<DatasetItemType>;
updateDataset: (data: DatasetUpdateBody) => Promise<any>;
startWebsiteSync: () => Promise<any>;
};
export const useDatasetStore = create<State>()(
......@@ -76,6 +85,22 @@ export const useDatasetStore = create<State>()(
: item
);
});
},
async startWebsiteSync() {
const [_, billId] = await Promise.all([
get().updateDataset({
id: get().datasetDetail._id,
status: DatasetStatusEnum.syncing
}),
postCreateTrainingBill({
name: 'core.dataset.training.Website Sync',
vectorModel: get().datasetDetail.vectorModel.model,
agentModel: get().datasetDetail.agentModel.model
})
]);
try {
postWebsiteSync({ datasetId: get().datasetDetail._id, billId });
} catch (error) {}
}
})),
{
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment