Commercial baseurl (#697)

84cc4baf · Archer · GitHub · dfa4c083 · 84cc4baf · dfa4c083
Commit 84cc4baf authored Jan 06, 2024 by Archer Committed by GitHub Jan 06, 2024
44 changed files
--- a/README.md
+++ b/README.md
@@ -105,7 +105,7 @@ https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409b

  [![](https://cdn.jsdelivr.us/gh/labring-actions/templates@main/Deploy-on-Sealos.svg)](https://cloud.sealos.io/?openapp=system-fastdeploy%3FtemplateName%3Dfastgpt)

-  由于需要部署数据库，部署完后需要等待 2~4 分钟才能正常访问。默认用了最低配置，首次访问时会有些慢。
+  由于需要部署数据库，部署完后需要等待 2~4 分钟才能正常访问。默认用了最低配置，首次访问时会有些慢。相关使用教程可查看：[Sealos 部署 FastGPT](https://doc.fastgpt.in/docs/development/sealos/)

 * [快开始本地开发](https://doc.fastgpt.in/docs/development/intro/)
 * [部署 FastGPT](https://doc.fastgpt.in/docs/development/sealos)

--- a/docSite/assets/imgs/KBProcess.jpg
+++ b/docSite/assets/imgs/KBProcess.jpg
--- a/docSite/assets/imgs/chatmodels1.png
+++ b/docSite/assets/imgs/chatmodels1.png
--- a/docSite/assets/imgs/data_search1.png
+++ b/docSite/assets/imgs/data_search1.png
--- a/docSite/assets/imgs/onsealos10.png
+++ b/docSite/assets/imgs/onsealos10.png
--- a/docSite/assets/imgs/onsealos12.png
+++ b/docSite/assets/imgs/onsealos12.png
--- a/docSite/assets/imgs/onsealosl11.PNG
+++ b/docSite/assets/imgs/onsealosl11.PNG
--- a/docSite/assets/imgs/sealos13.png
+++ b/docSite/assets/imgs/sealos13.png
--- a/docSite/content/docs/development/configuration.md
+++ b/docSite/content/docs/development/configuration.md
@@ -21,7 +21,6 @@ weight: 708
 ```json
 {
  "SystemParams": {
-    "pluginBaseUrl": "", // 商业版接口地址
    "vectorMaxProcess": 15, // 向量生成最大进程，结合数据库性能和 key 来设置
    "qaMaxProcess": 15,  // QA 生成最大进程，结合数据库性能和 key 来设置
    "pgHNSWEfSearch": 100  // pg vector 索引参数，越大精度高但速度慢
@@ -169,7 +168,6 @@ weight: 708
 ```json
 {
  "systemEnv": {
-    "pluginBaseUrl": "", // 商业版接口地址
    "vectorMaxProcess": 15, // 向量生成最大进程，结合数据库性能和 key 来设置
    "qaMaxProcess": 15,  // QA 生成最大进程，结合数据库性能和 key 来设置
    "pgHNSWEfSearch": 100  // pg vector 索引参数，越大精度高但速度慢
@@ -234,7 +232,8 @@ weight: 708
      "name": "GPT35-16k",
      "maxContext": 16000,
      "maxResponse": 16000,
-      "price": 0
+      "inputPrice": 0,
+      "outputPrice": 0
    }
  ],
  "cqModels": [ // 问题分类模型
@@ -254,6 +253,7 @@ weight: 708
      "maxContext": 8000,
      "maxResponse": 8000,
      "inputPrice": 0,
+      "outputPrice": 0,
      "toolChoice": true,
      "functionPrompt": ""
    }
@@ -264,7 +264,8 @@ weight: 708
      "name": "GPT35-1106",
      "maxContext": 16000,
      "maxResponse": 4000,
-            "outputPrice": 0,
+      "inputPrice": 0,
+      "outputPrice": 0,
      "toolChoice": true,
      "functionPrompt": ""
    }

--- a/docSite/content/docs/development/docker.md
+++ b/docSite/content/docs/development/docker.md
@@ -10,11 +10,11 @@ weight: 707
 ## 推荐配置

 {{< table "table-hover table-striped-columns" >}}
-| 环境 | 推荐配置（单节点） |
-| ---- | ---- |
-| 测试 | 2c2g  |
-| 100w 组向量 | 4c16g |
-| 500w 组向量 | 16c64g |
+| 环境 | 最低配置（单节点） | 推荐配置 |
+| ---- | ---- | ---- |
+| 测试 | 2c2g  | 2c4g |
+| 100w 组向量 | 4c8g 50GB | 4c16g 50GB |
+| 500w 组向量 | 8c32g | 16c64g 200GB |
 {{< /table >}}

 ### 1. 准备好代理环境（国外服务器可忽略）

--- a/docSite/content/docs/development/openapi/dataset.md
+++ b/docSite/content/docs/development/openapi/dataset.md
@@ -9,7 +9,7 @@ weight: 853

 | 如何获取知识库ID（datasetId） | 如何获取文件集合ID（collection_id） |
 | --------------------- | --------------------- |
-| ![](/imgs/getDatasetId.png) | ![](/imgs/getfile_id.png) |
+| ![](/imgs/getDatasetId.jpg) | ![](/imgs/getfile_id.png) |




--- a/docSite/content/docs/development/openapi/share.md
+++ b/docSite/content/docs/development/openapi/share.md
@@ -41,7 +41,7 @@ weight: 860
 ## 配置教程
 ### 1. 配置身份校验地址

-![](/imgs/share-setlink.jpg)
+![](/imgs/share-setlink.png)

 配置校验地址后，在每次分享链接使用时，都会向对应的地址发起校验和上报请求。

@@ -251,7 +251,7 @@ curl --location --request POST '{{host}}/shareAuth/finish' \

 ### 1. 创建3个Laf接口

-![](/imgs/share-auth1.jpg)
+![](/imgs/share-auth1.png)




--- a/docSite/content/docs/development/sealos.md
+++ b/docSite/content/docs/development/sealos.md
@@ -7,6 +7,11 @@ toc: true
 weight: 706
 ---

+## 部署架构图
+
+![](/imgs/sealos-fastgpt.webp)
+
+## 一键部署
 Sealos 的服务器在国外，不需要额外处理网络问题，无需服务器、无需魔法、无需域名，支持高并发 & 动态伸缩。点击以下按钮即可一键部署 👇

 [![](https://fastly.jsdelivr.net/gh/labring-actions/templates@main/Deploy-on-Sealos.svg)](https://cloud.sealos.io/?openapp=system-fastdeploy%3FtemplateName%3Dfastgpt)
@@ -19,11 +24,13 @@ Sealos 的服务器在国外，不需要额外处理网络问题，无需服务

 ![](/imgs/sealos2.png)

-> 用户名：`root`
-> 
-> 密码就是刚刚一键部署时设置的环境变量
+### 登录
+
+用户名：`root`

-## 修改配置文件和环境变量
+密码是刚刚一键部署时设置的`root_password`
+
+### 修改配置文件和环境变量

 在 Sealos 中，你可以打开`应用管理`（App Launchpad）看到部署的 FastGPT，可以打开`数据库`（Database）看到对应的数据库。

@@ -35,13 +42,9 @@ Sealos 的服务器在国外，不需要额外处理网络问题，无需服务
 在 Sealos 上，FastGPT 一共运行了 1 个服务和 2 个数据库，如暂停和删除请注意数据库一同操作。（你可以白天启动，晚上暂停它们，省钱大法）
 {{% /alert %}}

-## 更新
-
-点击重启会自动拉取最新镜像更新，请确保镜像`tag`正确。
+### 更新

-## 部署架构图
-
-![](/imgs/sealos-fastgpt.webp)
+点击变更或重启会自动拉取镜像更新，请确保镜像`tag`正确。建议不要使用`latest`，改成固定版本号。

 ## Sealos 使用


--- a/docSite/content/docs/development/upgrading/45.md
+++ b/docSite/content/docs/development/upgrading/45.md
@@ -40,8 +40,8 @@ CREATE INDEX CONCURRENTLY vector_index ON modeldata USING hnsw (vector vector_ip

 |                       |                       |
 | --------------------- | --------------------- |
-| ![](/imgs/v45-1.png) | ![](/imgs/v45-2.png) |
-| ![](/imgs/v45-3.png) | ![](/imgs/v45-4.png) |
+| ![](/imgs/v45-1.jpg) | ![](/imgs/v45-2.jpg) |
+| ![](/imgs/v45-3.jpg) | ![](/imgs/v45-4.jpg) |




--- a/docSite/content/docs/development/upgrading/466.md
+++ b/docSite/content/docs/development/upgrading/466.md
@@ -11,6 +11,16 @@ weight: 830

 为了减少代码重复度，我们对配置文件做了一些修改：[点击查看最新的配置文件](/docs/development/configuration/)

+## 商业版变更
+
+1. 更新商业版镜像到 4.6.6 版本。
+2. 将旧版配置文件中的 `SystemParams.pluginBaseUrl` 放置到环境变量中:
+
+    PRO_URL=商业版镜像地址（此处不再需要以 /api 结尾），例如:   
+    PRO_URL=http://fastgpt-plugin.ns-hsss5d.svc.cluster.local:3000
+
+3. 原本在配置文件中的 `FeConfig` 已被移除，可以直接打开新的商业版镜像外网地址进行配置。包括 FastGPT 的各个参数和模型都可以直接在商业版镜像中配置，无需再变更 `config.json` 文件。
+
 ## V4.6.6 更新说明

 1. 查看 [FastGPT 2024 RoadMap](https://github.com/labring/FastGPT?tab=readme-ov-file#-%E5%9C%A8%E7%BA%BF%E4%BD%BF%E7%94%A8)

--- a/docSite/content/docs/development/upgrading/intro.md
+++ b/docSite/content/docs/development/upgrading/intro.md
@@ -36,13 +36,13 @@ FastGPT 升级包括两个步骤：

 2. 选择对应的应用 - 点击右边三个点 - 变更

-![](/imgs/updateImageSealos2.jpg)
+![](/imgs/updateImageSealos2.png)

 3. 修改镜像 - 确认变更

    如果要修改配置文件，可以拉到下面的`配置文件`进行修改。

-![](/imgs/updateImageSealos3.jpg)
+![](/imgs/updateImageSealos3.png)

 ## Docker-Compose 修改镜像

@@ -66,7 +66,7 @@ docker-compose up -d

 Sealos 中，你可以在下图中找到你的域名：

-![](/imgs/updateImageSealos4.jpg)
+![](/imgs/updateImageSealos4.png)


 ### 如何获取 rootkey

--- a/docSite/content/docs/use-cases/ai_settings.md
+++ b/docSite/content/docs/use-cases/ai_settings.md
@@ -113,9 +113,9 @@ Tips: 建议根据不同的场景，每种知识库仅选择1类数据类型，

 | 通用模板配置及效果 | 问答模板配置及效果 |
 | --- | --- |
-| ![](/imgs/datasetprompt1.png) | ![](/imgs/datasetprompt2.png) |
-| ![](/imgs/datasetprompt3.png) | ![](/imgs/datasetprompt5.png) |
-| ![](/imgs/datasetprompt4.png) | ![](/imgs/datasetprompt6.png) |
+| ![](/imgs/datasetprompt1.jpg) | ![](/imgs/datasetprompt2.jpg) |
+| ![](/imgs/datasetprompt3.jpg) | ![](/imgs/datasetprompt5.jpg) |
+| ![](/imgs/datasetprompt4.jpg) | ![](/imgs/datasetprompt6.jpg) |

 #### 严格模板

@@ -123,7 +123,7 @@ Tips: 建议根据不同的场景，每种知识库仅选择1类数据类型，

 | 非严格模板效果 | 选择严格模板 | 严格模板效果 |
 | --- | --- | --- |
-| ![](/imgs/datasetprompt7.png) | ![](/imgs/datasetprompt8.png) |![](/imgs/datasetprompt9.png) |
+| ![](/imgs/datasetprompt7.jpg) | ![](/imgs/datasetprompt8.jpg) |![](/imgs/datasetprompt9.jpg) |

 #### 提示词设计思路


--- a/docSite/content/docs/use-cases/datasetEngine.md
+++ b/docSite/content/docs/use-cases/datasetEngine.md
@@ -80,7 +80,7 @@ FastGPT 采用了 `PostgresSQL` 的 `PG Vector` 插件作为向量检索器，

 有些数据较为独特，可能需要单独的进行预处理分割后再导入 FastGPT，此时可以选择 csv 导入，可批量的将处理好的数据导入。

-![](/imgs/datasetEngine11.png)
+![](/imgs/datasetEngine11.jpg)

 ### 导入数据方案5 - API导入


--- a/docSite/content/docs/workflow/examples/lab_appointment.md
+++ b/docSite/content/docs/workflow/examples/lab_appointment.md
@@ -69,9 +69,9 @@ HTTP 模块允许你调用任意 GET/POST 类型的 HTTP 接口，从而实现
 2. 可以通过内容提取模块，实现自然语言转结构化数据，从而实现复杂的逻辑操作。
 3. 内容提取 + HTTP 模块允许你无限扩展。

-**难点**
+**困难点**

-1. 模型对连续对话的分类和提取能力不足
+1. 模型对连续对话时，分类和提取能力不足


 # 附件

--- a/docSite/content/docs/workflow/examples/versatile_assistant.md
+++ b/docSite/content/docs/workflow/examples/versatile_assistant.md
@@ -40,7 +40,7 @@ weight: 406

 1. 找到查询天气的 API 接口

-![](/imgs/versatile_assistant_3.jpg)
+![](/imgs/versatile_assistant_3.png)

 2. 由于我想要的效果是用户可以随意问接下来一周内任意时间的天气（比如用户可以问“接下来一周的天气适合晾被子吗”），所以选择了上面接口的这个格式：https://api.vvhan.com/api/weather?city=徐州&type=week

@@ -109,7 +109,7 @@ if __name__ == '__main__':

 第一步就是对用户问题进行分类，如图红框部分：

-![](/imgs/versatile_assistant_4.jpg)
+![](/imgs/versatile_assistant_4.png)

 ### 接口参数获取及处理

@@ -132,7 +132,7 @@ if __name__ == '__main__':

 如图：

-![](/imgs/versatile_assistant_5.jpg)
+![](/imgs/versatile_assistant_5.png)

 ### AI 总结回复

@@ -147,7 +147,7 @@ if __name__ == '__main__':

 如图：

-![](/imgs/versatile_assistant_6.jpg)
+![](/imgs/versatile_assistant_6.png)

 ## 模块编排

@@ -1526,7 +1526,7 @@ PS2：配置中的问题分类还包含着“联网搜索”，这个是另一

 ## 效果图

-![](/imgs/versatile_assistant_7.jpg)
+![](/imgs/versatile_assistant_7.png)

 ## 后记


--- a/packages/global/common/system/types/index.d.ts
+++ b/packages/global/common/system/types/index.d.ts
@@ -54,7 +54,6 @@ export type FastGPTFeConfigsType = {
 };

 export type SystemEnvType = {
-  pluginBaseUrl?: string;
  openapiPrefix?: string;
  vectorMaxProcess: number;
  qaMaxProcess: number;

--- a/packages/global/support/user/inform/constants.ts
+++ b/packages/global/support/user/inform/constants.ts
 export enum InformTypeEnum {
-  system = 'system'
+  system = 'system',
+  admin = 'admin'
 }

 export const InformTypeMap = {
  [InformTypeEnum.system]: {
    label: '系统通知'
+  },
+  [InformTypeEnum.admin]: {
+    label: '管理员'
  }
 };
--- a/packages/global/support/user/inform/type.d.ts
+++ b/packages/global/support/user/inform/type.d.ts
-import { InformTypeEnum } from './constant';
+import { InformTypeEnum } from './constants';

 export type SendInformProps = {
  tmbId?: string;

--- a/packages/service/common/api/plusRequest.ts
+++ b/packages/service/common/api/plusRequest.ts
 import axios, { Method, InternalAxiosRequestConfig, AxiosResponse } from 'axios';
+import { FastGPTProUrl } from '../system/constants';

 interface ConfigType {
  headers?: { [key: string]: string };
@@ -70,7 +71,7 @@ instance.interceptors.request.use(requestStart, (err) => Promise.reject(err));
 instance.interceptors.response.use(responseSuccess, (err) => Promise.reject(err));

 export function request(url: string, data: any, config: ConfigType, method: Method): any {
-  if (!global.systemEnv || !global.systemEnv?.pluginBaseUrl) {
+  if (!FastGPTProUrl) {
    console.log('未部署商业版接口', url);
    return Promise.reject('The The request was denied...');
  }
@@ -84,7 +85,7 @@ export function request(url: string, data: any, config: ConfigType, method: Meth

  return instance
    .request({
-      baseURL: global.systemEnv.pluginBaseUrl,
+      baseURL: FastGPTProUrl,
      url,
      method,
      data: ['POST', 'PUT'].includes(method) ? data : null,

--- a/packages/service/common/string/markdown.ts
+++ b/packages/service/common/string/markdown.ts
@@ -21,9 +21,6 @@ export const htmlToMarkdown = (html?: string | null) =>
      worker.terminate();
      reject(err);
    });
-    worker.on('exit', (code) => {
-      console.log('html 2 md finish', code);
-    });

    worker.postMessage(html);
  });
--- a/packages/service/common/system/config/controller.ts
+++ b/packages/service/common/system/config/controller.ts
 import { SystemConfigsTypeEnum } from '@fastgpt/global/common/system/config/constants';
 import { MongoSystemConfigs } from './schema';
 import { FastGPTConfigFileType } from '@fastgpt/global/common/system/types';
+import { FastGPTProUrl } from '../constants';

 export const getFastGPTConfigFromDB = async () => {
+  if (!FastGPTProUrl) return {} as FastGPTConfigFileType;
+
  const res = await MongoSystemConfigs.findOne({
    type: SystemConfigsTypeEnum.fastgpt
  }).sort({
@@ -11,5 +14,5 @@ export const getFastGPTConfigFromDB = async () => {

  const config = res?.value || {};

-  return config as Omit<FastGPTConfigFileType, 'systemEnv'>;
+  return config as FastGPTConfigFileType;
 };
--- a/packages/service/common/system/constants.ts
+++ b/packages/service/common/system/constants.ts
+export const FastGPTProUrl = process.env.PRO_URL ? `${process.env.PRO_URL}/api` : '';
--- a/packages/service/support/outLink/tools.ts
+++ b/packages/service/support/outLink/tools.ts
 import axios from 'axios';
 import { MongoOutLink } from './schema';
+import { FastGPTProUrl } from '../../common/system/constants';

 export const updateOutLinkUsage = async ({
  shareId,
@@ -30,7 +31,7 @@ export const pushResult2Remote = async ({
  shareId?: string;
  responseData?: any[];
 }) => {
-  if (!shareId || !outLinkUid || !global.systemEnv?.pluginBaseUrl) return;
+  if (!shareId || !outLinkUid || !FastGPTProUrl) return;
  try {
    const outLink = await MongoOutLink.findOne({
      shareId

--- a/projects/app/.env.template
+++ b/projects/app/.env.template
@@ -20,6 +20,8 @@ CHAT_API_KEY=sk-xxxx
 MONGODB_URI=mongodb://username:password@0.0.0.0:27017/fastgpt?authSource=admin
 # PG 数据库连接参数
 PG_URL=postgresql://username:password@host:port/postgres
+# 商业版地址
+PRO_URL=
 # 首页路径
 HOME_URL=/
 # Loki Log Path

--- a/projects/app/data/config.json
+++ b/projects/app/data/config.json
 {
  "systemEnv": {
-    "pluginBaseUrl": "",
+    "openapiPrefix": "fastgpt",
    "vectorMaxProcess": 15,
    "qaMaxProcess": 15,
    "pgHNSWEfSearch": 100

--- a/projects/app/src/components/ChatBox/ResponseTags.tsx
+++ b/projects/app/src/components/ChatBox/ResponseTags.tsx
@@ -95,7 +95,7 @@ const ResponseTags = ({
          <ChatBoxDivider icon="core/chat/quoteFill" text={t('chat.Quote')} />
          <Flex alignItems={'center'} flexWrap={'wrap'} gap={2}>
            {sourceList.map((item) => (
-              <MyTooltip key={item.sourceName} label={t('core.chat.quote.Read Quote')}>
+              <MyTooltip key={item.collectionId} label={t('core.chat.quote.Read Quote')}>
                <Flex
                  alignItems={'center'}
                  fontSize={'sm'}

--- a/projects/app/src/pages/api/admin/initv46.ts
+++ b/projects/app/src/pages/api/admin/initv46.ts
@@ -24,6 +24,7 @@ import { MongoPlugin } from '@fastgpt/service/core/plugin/schema';
 import { POST } from '@fastgpt/service/common/api/plusRequest';
 import { authCert } from '@fastgpt/service/support/permission/auth/common';
 import { getGFSCollection } from '@fastgpt/service/common/file/gridfs/controller';
+import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';

 export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  try {
@@ -36,7 +37,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
    await initDatasetAndApp();
    await initCollectionFileTeam(limit);

-    if (global.systemEnv.pluginBaseUrl) {
+    if (FastGPTProUrl) {
      POST('/admin/init46');
    }


--- a/projects/app/src/pages/api/common/system/getInitData.ts
+++ b/projects/app/src/pages/api/common/system/getInitData.ts
@@ -14,6 +14,7 @@ import { connectToDatabase } from '@/service/mongo';
 import { PluginTemplateType } from '@fastgpt/global/core/plugin/type';
 import { readConfigData } from '@/service/common/system';
 import { exit } from 'process';
+import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';

 export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  await getInitConfig();
@@ -105,10 +106,13 @@ export async function initSystemConfig() {
  const config: FastGPTConfigFileType = {
    feConfigs: {
      ...defaultFeConfigs,
-      ...(fileRes.feConfigs || {}),
-      ...(dbConfig.feConfigs || {})
+      ...(dbConfig.feConfigs || {}),
+      isPlus: !!FastGPTProUrl
+    },
+    systemEnv: {
+      ...fileRes.systemEnv,
+      ...(dbConfig.systemEnv || {})
    },
-    systemEnv: fileRes.systemEnv,
    chatModels: dbConfig.chatModels || fileRes.chatModels || [],
    qaModels: dbConfig.qaModels || fileRes.qaModels || [],
    cqModels: dbConfig.cqModels || fileRes.cqModels || [],
@@ -121,10 +125,7 @@ export async function initSystemConfig() {
  };

  // set config
-  global.feConfigs = {
-    isPlus: !!config.systemEnv?.pluginBaseUrl,
-    ...config.feConfigs
-  };
+  global.feConfigs = config.feConfigs;
  global.systemEnv = config.systemEnv;

  global.chatModels = config.chatModels;

--- a/projects/app/src/pages/api/common/system/refreshConfig.ts
+++ b/projects/app/src/pages/api/common/system/refreshConfig.ts
@@ -12,8 +12,6 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<

    console.log(`refresh config`);
    console.log({
-      feConfigs: global.feConfigs,
-      systemEnv: global.systemEnv,
      chatModels: global.chatModels,
      qaModels: global.qaModels,
      cqModels: global.cqModels,
@@ -22,7 +20,9 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
      vectorModels: global.vectorModels,
      reRankModels: global.reRankModels,
      audioSpeechModels: global.audioSpeechModels,
-      whisperModel: global.whisperModel
+      whisperModel: global.whisperModel,
+      feConfigs: global.feConfigs,
+      systemEnv: global.systemEnv
    });
  } catch (error) {
    console.log(error);

--- a/projects/app/src/pages/api/core/dataset/data/pushData.ts
+++ b/projects/app/src/pages/api/core/dataset/data/pushData.ts
@@ -96,32 +96,31 @@ export async function pushDataToDatasetCollection({
    repeat: [],
    error: []
  };
-  await Promise.all(
-    data.map(async (item) => {
-      if (!item.q) {
-        filterResult.error.push(item);
-        return;
-      }
-
-      const text = item.q + item.a;
-
-      // count q token
-      const token = countPromptTokens(item.q);
-
-      if (token > maxToken) {
-        filterResult.overToken.push(item);
-        return;
-      }
-
-      if (set.has(text)) {
-        console.log('repeat', item);
-        filterResult.repeat.push(item);
-      } else {
-        filterResult.success.push(item);
-        set.add(text);
-      }
-    })
-  );
+
+  data.forEach((item) => {
+    if (!item.q) {
+      filterResult.error.push(item);
+      return;
+    }
+
+    const text = item.q + item.a;
+
+    // count q token
+    const token = countPromptTokens(item.q);
+
+    if (token > maxToken) {
+      filterResult.overToken.push(item);
+      return;
+    }
+
+    if (set.has(text)) {
+      console.log('repeat', item);
+      filterResult.repeat.push(item);
+    } else {
+      filterResult.success.push(item);
+      set.add(text);
+    }
+  });

  // 插入记录
  const insertRes = await MongoDatasetTraining.insertMany(

--- a/projects/app/src/pages/api/core/plugin/templates.ts
+++ b/projects/app/src/pages/api/core/plugin/templates.ts
@@ -8,6 +8,7 @@ import { FlowModuleTemplateType } from '@fastgpt/global/core/module/type';
 import { ModuleTemplateTypeEnum } from '@fastgpt/global/core/module/constants';
 import { GET } from '@fastgpt/service/common/api/plusRequest';
 import type { PluginTemplateType } from '@fastgpt/global/core/plugin/type.d';
+import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';

 export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
  try {
@@ -16,7 +17,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<

    const [userPlugins, plusPlugins] = await Promise.all([
      MongoPlugin.find({ teamId }).lean(),
-      global.systemEnv?.pluginBaseUrl ? GET<PluginTemplateType[]>('/core/plugin/getTemplates') : []
+      FastGPTProUrl ? GET<PluginTemplateType[]>('/core/plugin/getTemplates') : []
    ]);

    const data: FlowModuleTemplateType[] = [

--- a/projects/app/src/pages/api/plusApi/[...path].ts
+++ b/projects/app/src/pages/api/plusApi/[...path].ts
@@ -4,10 +4,11 @@ import { request } from '@fastgpt/service/common/api/plusRequest';
 import type { Method } from 'axios';
 import { setCookie } from '@fastgpt/service/support/permission/controller';
 import { getInitConfig } from '../common/system/getInitData';
+import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';

 export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  try {
-    if (!global.systemEnv?.pluginBaseUrl) {
+    if (!FastGPTProUrl) {
      await getInitConfig();
    }


--- a/projects/app/src/service/core/app/utils.ts
+++ b/projects/app/src/service/core/app/utils.ts
 import { AppSimpleEditConfigTemplateType } from '@fastgpt/global/core/app/type';
 import { GET } from '@fastgpt/service/common/api/plusRequest';
+import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';

 export async function getSimpleTemplatesFromPlus(): Promise<AppSimpleEditConfigTemplateType[]> {
  try {
-    if (!global.systemEnv?.pluginBaseUrl) return [];
+    if (!FastGPTProUrl) return [];

    return GET<AppSimpleEditConfigTemplateType[]>('/core/app/getSimpleTemplates');
  } catch (error) {

--- a/projects/app/src/service/core/dataset/utils.ts
+++ b/projects/app/src/service/core/dataset/utils.ts
-import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
-
-/**
- * Same value judgment
- */
-export async function hasSameValue({
-  collectionId,
-  q,
-  a = ''
-}: {
-  collectionId: string;
-  q: string;
-  a?: string;
-}) {
-  const count = await MongoDatasetData.countDocuments({
-    q,
-    a,
-    collectionId
-  });
-
-  if (count > 0) {
-    return Promise.reject('已经存在完全一致的数据');
-  }
-}
--- a/projects/app/src/service/events/generateQA.ts
+++ b/projects/app/src/service/events/generateQA.ts
@@ -140,7 +140,7 @@ ${replaceVariable(Prompt_AgentQA.fixedText, { text })}`;
    const qaArr = formatSplitText(answer, text); // 格式化后的QA对

    // get vector and insert
-    await pushDataToDatasetCollection({
+    const { insertLen } = await pushDataToDatasetCollection({
      teamId: data.teamId,
      tmbId: data.tmbId,
      collectionId: data.collectionId,
@@ -162,7 +162,7 @@ ${replaceVariable(Prompt_AgentQA.fixedText, { text })}`;
    });

    // add bill
-    if (qaArr.length > 0) {
+    if (insertLen > 0) {
      pushQABill({
        teamId: data.teamId,
        tmbId: data.tmbId,

--- a/projects/app/src/service/moduleDispatch/chat/oneapi.ts
+++ b/projects/app/src/service/moduleDispatch/chat/oneapi.ts
@@ -247,7 +247,7 @@ function filterQuote({
    maxTokens: model.quoteMaxToken,
    messages: quoteQA.map((item, index) => ({
      obj: ChatRoleEnum.System,
-      value: getValue(item, index)
+      value: getValue(item, index).trim()
    }))
  });


--- a/projects/app/src/service/support/permission/auth/bill.ts
+++ b/projects/app/src/service/support/permission/auth/bill.ts
 import { GET } from '@fastgpt/service/common/api/plusRequest';
+import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';

 export const authTeamBalance = async (teamId: string) => {
-  if (global.systemEnv?.pluginBaseUrl) {
+  if (FastGPTProUrl) {
    return GET('/support/permission/authBalance', { teamId });
  }
  return true;

--- a/projects/app/src/service/support/user/inform/api.ts
+++ b/projects/app/src/service/support/user/inform/api.ts
 import { POST } from '@fastgpt/service/common/api/plusRequest';
 import { SendInformProps } from '@fastgpt/global/support/user/inform/type';
+import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';

 export function sendOneInform(data: SendInformProps) {
-  if (!global.systemEnv?.pluginBaseUrl) return;
+  if (!FastGPTProUrl) return;
  return POST('/support/user/inform/create', data);
 }
--- a/projects/app/src/service/support/wallet/bill/controller.ts
+++ b/projects/app/src/service/support/wallet/bill/controller.ts
 import { ConcatBillProps, CreateBillProps } from '@fastgpt/global/support/wallet/bill/api';
 import { addLog } from '@fastgpt/service/common/system/log';
 import { POST } from '@fastgpt/service/common/api/plusRequest';
+import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';

 export function createBill(data: CreateBillProps) {
-  if (!global.systemEnv?.pluginBaseUrl) return;
+  if (!FastGPTProUrl) return;
  if (data.total === 0) {
    addLog.info('0 Bill', data);
  }
@@ -12,7 +13,7 @@ export function createBill(data: CreateBillProps) {
  } catch (error) {}
 }
 export function concatBill(data: ConcatBillProps) {
-  if (!global.systemEnv?.pluginBaseUrl) return;
+  if (!FastGPTProUrl) return;
  if (data.total === 0) {
    addLog.info('0 Bill', data);
  }