Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 67c52992 by Archer Committed by GitHub
Options

External dataset (#1519) 

* perf: local file create collection

* rename middleware

* perf: remove code

* feat: next14

* feat: external file dataset

* collection tags field

* external file dataset doc

* fix: ts
parent 2d1ec9b3
Showing with 851 additions and 376 deletions
.npmrc
public-hoist-pattern[]=*tiktoken* public-hoist-pattern[]=*tiktoken*
public-hoist-pattern[]=*react*
.vscode/nextapi.code-snippets
...@@ -11,7 +11,7 @@ ...@@ -11,7 +11,7 @@
"prefix": "nextapi", "prefix": "nextapi",
"body": [ "body": [
"import type { ApiRequestProps, ApiResponseType } from '@fastgpt/service/type/next';", "import type { ApiRequestProps, ApiResponseType } from '@fastgpt/service/type/next';",
"import { NextAPI } from '@/service/middle/entry';", "import { NextAPI } from '@/service/middleware/entry';",
"", "",
"export type ${TM_FILENAME_BASE}Query = {};", "export type ${TM_FILENAME_BASE}Query = {};",
"", "",
......
docSite/content/docs/course/externalFile.md 0 → 100644
---
title: '外部文件知识库'
description: 'FastGPT 外部文件知识库功能介绍和使用方式'
icon: 'language'
draft: false
toc: true
weight: 107
---
外部文件库是 FastGPT 商业版特有功能。它允许接入你现在的文件系统，无需将文件再导入一份到 FastGPT 中。
并且，阅读权限可以通过你的文件系统进行控制。
| | | |
| --------------------- | --------------------- | --------------------- |
| ![](/imgs/external_file0.png) | ![](/imgs/external_file1.png) | ![](/imgs/external_file2.png) |
## 导入参数说明
- 外部预览地址：用于跳转你的文件阅读地址，会携带“文件阅读ID”进行访问。
- 文件访问URL：文件可访问的地址。
- 文件阅读ID：通常情况下，文件访问URL是临时的。如果希望永久可以访问，你需要使用该文件阅读ID，并配合上“外部预览地址”，跳转至新的阅读地址进行原文件访问。
- 文件名：默认会自动解析文件访问URL上的文件名。如果你手动填写，将会以手动填写的值为准。
[点击查看API导入文档](/docs/development/openapi/dataset/#创建一个外部文件库集合商业版)
\ No newline at end of file
docSite/content/docs/development/openapi/dataset.md
...@@ -295,6 +295,24 @@ curl --location --request DELETE 'http://localhost:3000/api/core/dataset/delete? ...@@ -295,6 +295,24 @@ curl --location --request DELETE 'http://localhost:3000/api/core/dataset/delete?
## 集合 ## 集合
### 通用创建参数说明
**入参**
| 参数 | 说明 | 必填 |
| --- | --- | --- |
| datasetId | 知识库ID | ✅ |
| parentId： | 父级ID，不填则默认为根目录 | |
| trainingType | 训练模式。chunk: 按文本长度进行分割;qa: QA拆分;auto: 增强训练 | ✅ |
| chunkSize | 预估块大小 | |
| chunkSplitter | 自定义最高优先分割符号 | |
| qaPrompt | qa拆分提示词 | |
**出参**
- collectionId - 新建的集合ID
- insertLen：插入的块数量
### 创建一个空的集合 ### 创建一个空的集合
{{< tabs tabTotal="3" >}} {{< tabs tabTotal="3" >}}
...@@ -500,7 +518,7 @@ data 为集合的 ID。 ...@@ -500,7 +518,7 @@ data 为集合的 ID。
{{< /tab >}} {{< /tab >}}
{{< /tabs >}} {{< /tabs >}}
### 创建一个文件集合(商业版) ### 创建一个文件集合
传入一个文件，创建一个集合，会读取文件内容进行分割。目前支持：pdf, docx, md, txt, html, csv。 传入一个文件，创建一个集合，会读取文件内容进行分割。目前支持：pdf, docx, md, txt, html, csv。
...@@ -509,7 +527,7 @@ data 为集合的 ID。 ...@@ -509,7 +527,7 @@ data 为集合的 ID。
{{< markdownify >}} {{< markdownify >}}
```bash ```bash
curl --location --request POST 'http://localhost:3000/api/proApi/core/dataset/collection/create/file' \ curl --location --request POST 'http://localhost:3000/api/core/dataset/collection/create/localFile' \
--header 'Authorization: Bearer {{authorization}}' \ --header 'Authorization: Bearer {{authorization}}' \
--form 'file=@"C:\\Users\\user\\Desktop\\fastgpt测试文件\\index.html"' \ --form 'file=@"C:\\Users\\user\\Desktop\\fastgpt测试文件\\index.html"' \
--form 'data="{\"datasetId\":\"6593e137231a2be9c5603ba7\",\"parentId\":null,\"trainingType\":\"chunk\",\"chunkSize\":512,\"chunkSplitter\":\"\",\"qaPrompt\":\"\",\"metadata\":{}}"' --form 'data="{\"datasetId\":\"6593e137231a2be9c5603ba7\",\"parentId\":null,\"trainingType\":\"chunk\",\"chunkSize\":512,\"chunkSplitter\":\"\",\"qaPrompt\":\"\",\"metadata\":{}}"'
...@@ -565,6 +583,68 @@ data 为集合的 ID。 ...@@ -565,6 +583,68 @@ data 为集合的 ID。
{{< /tab >}} {{< /tab >}}
{{< /tabs >}} {{< /tabs >}}
### 创建一个外部文件库集合（商业版）
{{< tabs tabTotal="3" >}}
{{< tab tabName="请求示例" >}}
{{< markdownify >}}
```bash
curl --location --request POST 'http://localhost:3000/api/proApi/core/dataset/collection/create/externalFileUrl' \
--header 'Authorization: Bearer {{authorization}}' \
--header 'User-Agent: Apifox/1.0.0 (https://apifox.com)' \
--header 'Content-Type: application/json' \
--data-raw '{
"externalFileUrl":"https://image.xxxxx.com/fastgpt-dev/%E6%91%82.pdf",
"externalFileId":"1111",
"filename":"自定义文件名",
"datasetId":"6642d105a5e9d2b00255b27b",
"parentId": null,
"trainingType": "chunk",
"chunkSize":512,
"chunkSplitter":"",
"qaPrompt":""
}'
```
{{< /markdownify >}}
{{< /tab >}}
{{< tab tabName="参数说明" >}}
{{< markdownify >}}
| 参数 | 说明 | 必填 |
| --- | --- | --- |
| externalFileUrl | 文件访问链接（可以是临时链接） | ✅ |
| externalFileId | 外部文件ID | |
| filename | 自定义文件名 | |
{{< /markdownify >}}
{{< /tab >}}
{{< tab tabName="响应示例" >}}
{{< markdownify >}}
data 为集合的 ID。
```json
{
"code": 200,
"statusText": "",
"message": "",
"data": {
"collectionId": "6646fcedfabd823cdc6de746",
"insertLen": 3
}
}
```
{{< /markdownify >}}
{{< /tab >}}
{{< /tabs >}}
### 获取集合列表 ### 获取集合列表
{{< tabs tabTotal="3" >}} {{< tabs tabTotal="3" >}}
......
docSite/content/docs/development/upgrading/481.md
...@@ -35,8 +35,11 @@ curl --location --request POST 'https://{{host}}/api/admin/clearInvalidData' \ ...@@ -35,8 +35,11 @@ curl --location --request POST 'https://{{host}}/api/admin/clearInvalidData' \
## V4.8.1 更新说明 ## V4.8.1 更新说明
1. 新增 - 知识库重新选择向量模型重建 1. 新增 - 知识库重新选择向量模型重建
2. 新增 - 工作流节点版本变更提示，并可以同步最新版本。 2. 新增 - 对话框支持问题模糊检索提示，可自定义预设问题词库。
3. 优化 - 插件输入的 debug 模式，支持全量参数输入渲染。 3. 新增 - 工作流节点版本变更提示，并可以同步最新版本配置，避免存在隐藏脏数据。
4. 修复 - 插件输入默认值被清空问题。 4. 新增 - 开放文件导入知识库接口到开源版， [点击插件文档](/docs/development/openapi/dataset/#创建一个文件集合)
5. 修复 - 工作流删除节点的动态输入和输出时候，没有正确的删除连接线，导致可能出现逻辑异常。 5. 新增 - 外部文件源知识库, [点击查看文档](/docs/course/externalfile/)
6. 修复 - 定时器清理脏数据任务 6. 优化 - 插件输入的 debug 模式，支持全量参数输入渲染。
\ No newline at end of file 7. 修复 - 插件输入默认值被清空问题。
8. 修复 - 工作流删除节点的动态输入和输出时候，没有正确的删除连接线，导致可能出现逻辑异常。
9. 修复 - 定时器清理脏数据任务
\ No newline at end of file
packages/global/core/dataset/api.d.ts
...@@ -26,18 +26,27 @@ export type DatasetCollectionChunkMetadataType = { ...@@ -26,18 +26,27 @@ export type DatasetCollectionChunkMetadataType = {
qaPrompt?: string; qaPrompt?: string;
metadata?: Record<string, any>; metadata?: Record<string, any>;
}; };
// create collection params
export type CreateDatasetCollectionParams = DatasetCollectionChunkMetadataType & { export type CreateDatasetCollectionParams = DatasetCollectionChunkMetadataType & {
datasetId: string; datasetId: string;
name: string; name: string;
type: `${DatasetCollectionTypeEnum}`; type: DatasetCollectionTypeEnum;
tags?: string[];
fileId?: string; fileId?: string;
rawLink?: string; rawLink?: string;
externalFileId?: string;
externalFileUrl?: string;
rawTextLength?: number; rawTextLength?: number;
hashRawText?: string; hashRawText?: string;
}; };
export type ApiCreateDatasetCollectionParams = DatasetCollectionChunkMetadataType & { export type ApiCreateDatasetCollectionParams = DatasetCollectionChunkMetadataType & {
datasetId: string; datasetId: string;
tags?: string[];
}; };
export type TextCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams & { export type TextCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams & {
name: string; name: string;
...@@ -58,6 +67,11 @@ export type CsvTableCreateDatasetCollectionParams = { ...@@ -58,6 +67,11 @@ export type CsvTableCreateDatasetCollectionParams = {
parentId?: string; parentId?: string;
fileId: string; fileId: string;
}; };
export type ExternalFileCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams & {
externalFileId?: string;
externalFileUrl: string;
filename?: string;
};
/* ================= data ===================== */ /* ================= data ===================== */
export type PgSearchRawType = { export type PgSearchRawType = {
......
packages/global/core/dataset/collection/constants.ts
/* sourceId = prefix-id; id=fileId;link url;externalId */ /* sourceId = prefix-id; id=fileId;link url;externalFileId */
export enum CollectionSourcePrefixEnum { export enum CollectionSourcePrefixEnum {
local = 'local', local = 'local',
link = 'link', link = 'link',
......
packages/global/core/dataset/collection/utils.ts 0 → 100644
import { CollectionWithDatasetType, DatasetCollectionSchemaType } from '../type';
export const getCollectionSourceData = (
collection?: CollectionWithDatasetType | DatasetCollectionSchemaType
) => {
return {
sourceId:
collection?.fileId ||
collection?.rawLink ||
collection?.externalFileId ||
collection?.externalFileUrl,
sourceName: collection?.name || ''
};
};
packages/global/core/dataset/constants.ts
...@@ -22,7 +22,7 @@ export const DatasetTypeMap = { ...@@ -22,7 +22,7 @@ export const DatasetTypeMap = {
collectionLabel: 'common.Website' collectionLabel: 'common.Website'
}, },
[DatasetTypeEnum.externalFile]: { [DatasetTypeEnum.externalFile]: {
icon: 'core/dataset/commonDataset', icon: 'core/dataset/externalDataset',
label: 'External File', label: 'External File',
collectionLabel: 'common.File' collectionLabel: 'common.File'
} }
...@@ -44,9 +44,11 @@ export const DatasetStatusMap = { ...@@ -44,9 +44,11 @@ export const DatasetStatusMap = {
/* ------------ collection -------------- */ /* ------------ collection -------------- */
export enum DatasetCollectionTypeEnum { export enum DatasetCollectionTypeEnum {
folder = 'folder', folder = 'folder',
virtual = 'virtual',
file = 'file', file = 'file',
link = 'link', // one link link = 'link', // one link
virtual = 'virtual' externalFile = 'externalFile'
} }
export const DatasetCollectionTypeMap = { export const DatasetCollectionTypeMap = {
[DatasetCollectionTypeEnum.folder]: { [DatasetCollectionTypeEnum.folder]: {
...@@ -55,6 +57,9 @@ export const DatasetCollectionTypeMap = { ...@@ -55,6 +57,9 @@ export const DatasetCollectionTypeMap = {
[DatasetCollectionTypeEnum.file]: { [DatasetCollectionTypeEnum.file]: {
name: 'core.dataset.file' name: 'core.dataset.file'
}, },
[DatasetCollectionTypeEnum.externalFile]: {
name: 'core.dataset.externalFile'
},
[DatasetCollectionTypeEnum.link]: { [DatasetCollectionTypeEnum.link]: {
name: 'core.dataset.link' name: 'core.dataset.link'
}, },
......
packages/global/core/dataset/read.ts
import { DatasetSourceReadTypeEnum, ImportDataSourceEnum } from './constants'; import { DatasetSourceReadTypeEnum, ImportDataSourceEnum } from './constants';
export const rawTextBackupPrefix = 'index,content';
export const importType2ReadType = (type: ImportDataSourceEnum) => { export const importType2ReadType = (type: ImportDataSourceEnum) => {
if (type === ImportDataSourceEnum.csvTable || type === ImportDataSourceEnum.fileLocal) { if (type === ImportDataSourceEnum.csvTable || type === ImportDataSourceEnum.fileLocal) {
return DatasetSourceReadTypeEnum.fileLocal; return DatasetSourceReadTypeEnum.fileLocal;
......
packages/global/core/dataset/type.d.ts
...@@ -41,7 +41,7 @@ export type DatasetCollectionSchemaType = { ...@@ -41,7 +41,7 @@ export type DatasetCollectionSchemaType = {
datasetId: string; datasetId: string;
parentId?: string; parentId?: string;
name: string; name: string;
type: `${DatasetCollectionTypeEnum}`; type: DatasetCollectionTypeEnum;
createTime: Date; createTime: Date;
updateTime: Date; updateTime: Date;
...@@ -50,13 +50,15 @@ export type DatasetCollectionSchemaType = { ...@@ -50,13 +50,15 @@ export type DatasetCollectionSchemaType = {
chunkSplitter?: string; chunkSplitter?: string;
qaPrompt?: string; qaPrompt?: string;
sourceId?: string; // relate CollectionSourcePrefixEnum tags?: string[];
fileId?: string; // local file id fileId?: string; // local file id
rawLink?: string; // link url rawLink?: string; // link url
externalFileId?: string; //external file id
rawTextLength?: number; rawTextLength?: number;
hashRawText?: string; hashRawText?: string;
externalSourceUrl?: string; // external import url externalFileUrl?: string; // external import url
metadata?: { metadata?: {
webPageSelector?: string; webPageSelector?: string;
relatedImgId?: string; // The id of the associated image collections relatedImgId?: string; // The id of the associated image collections
......
packages/global/core/dataset/utils.ts
...@@ -3,7 +3,7 @@ import { getFileIcon } from '../../common/file/icon'; ...@@ -3,7 +3,7 @@ import { getFileIcon } from '../../common/file/icon';
import { strIsLink } from '../../common/string/tools'; import { strIsLink } from '../../common/string/tools';
export function getCollectionIcon( export function getCollectionIcon(
type: `${DatasetCollectionTypeEnum}` = DatasetCollectionTypeEnum.file, type: DatasetCollectionTypeEnum = DatasetCollectionTypeEnum.file,
name = '' name = ''
) { ) {
if (type === DatasetCollectionTypeEnum.folder) { if (type === DatasetCollectionTypeEnum.folder) {
...@@ -24,13 +24,13 @@ export function getSourceNameIcon({ ...@@ -24,13 +24,13 @@ export function getSourceNameIcon({
sourceName: string; sourceName: string;
sourceId?: string; sourceId?: string;
}) { }) {
if (strIsLink(sourceId)) { const fileIcon = getFileIcon(decodeURIComponent(sourceName), '');
return 'common/linkBlue';
}
const fileIcon = getFileIcon(sourceName, '');
if (fileIcon) { if (fileIcon) {
return fileIcon; return fileIcon;
} }
if (strIsLink(sourceId)) {
return 'common/linkBlue';
}
return 'file/fill/manual'; return 'file/fill/manual';
} }
......
packages/global/package.json
...@@ -10,7 +10,7 @@ ...@@ -10,7 +10,7 @@
"js-yaml": "^4.1.0", "js-yaml": "^4.1.0",
"jschardet": "3.1.1", "jschardet": "3.1.1",
"nanoid": "^4.0.1", "nanoid": "^4.0.1",
"next": "13.5.2", "next": "14.2.3",
"openai": "4.28.0", "openai": "4.28.0",
"openapi-types": "^12.1.3", "openapi-types": "^12.1.3",
"timezones-list": "^3.0.2" "timezones-list": "^3.0.2"
......
packages/service/common/file/gridfs/controller.ts
...@@ -7,7 +7,7 @@ import { MongoFileSchema } from './schema'; ...@@ -7,7 +7,7 @@ import { MongoFileSchema } from './schema';
import { detectFileEncoding } from '@fastgpt/global/common/file/tools'; import { detectFileEncoding } from '@fastgpt/global/common/file/tools';
import { CommonErrEnum } from '@fastgpt/global/common/error/code/common'; import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
import { MongoRawTextBuffer } from '../../buffer/rawText/schema'; import { MongoRawTextBuffer } from '../../buffer/rawText/schema';
import { readFileRawContent } from '../read/utils'; import { readRawContentByFileBuffer } from '../read/utils';
import { PassThrough } from 'stream'; import { PassThrough } from 'stream';
export function getGFSCollection(bucket: `${BucketNameEnum}`) { export function getGFSCollection(bucket: `${BucketNameEnum}`) {
...@@ -196,7 +196,7 @@ export const readFileContentFromMongo = async ({ ...@@ -196,7 +196,7 @@ export const readFileContentFromMongo = async ({
}); });
})(); })();
const { rawText } = await readFileRawContent({ const { rawText } = await readRawContentByFileBuffer({
extension, extension,
isQAImport, isQAImport,
teamId, teamId,
......
packages/service/common/file/read/utils.ts
import { markdownProcess, simpleMarkdownText } from '@fastgpt/global/common/string/markdown'; import { markdownProcess } from '@fastgpt/global/common/string/markdown';
import { uploadMongoImg } from '../image/controller'; import { uploadMongoImg } from '../image/controller';
import { MongoImageTypeEnum } from '@fastgpt/global/common/file/image/constants'; import { MongoImageTypeEnum } from '@fastgpt/global/common/file/image/constants';
import { addHours } from 'date-fns'; import { addHours } from 'date-fns';
import { WorkerNameEnum, runWorker } from '../../../worker/utils'; import { WorkerNameEnum, runWorker } from '../../../worker/utils';
import fs from 'fs';
import { detectFileEncoding } from '@fastgpt/global/common/file/tools';
import { ReadFileResponse } from '../../../worker/file/type'; import { ReadFileResponse } from '../../../worker/file/type';
import { rawTextBackupPrefix } from '@fastgpt/global/core/dataset/read';
export const initMarkdownText = ({ export const initMarkdownText = ({
teamId, teamId,
...@@ -28,7 +29,34 @@ export const initMarkdownText = ({ ...@@ -28,7 +29,34 @@ export const initMarkdownText = ({
}) })
}); });
export const readFileRawContent = async ({ export type readRawTextByLocalFileParams = {
teamId: string;
path: string;
metadata?: Record<string, any>;
};
export const readRawTextByLocalFile = async (params: readRawTextByLocalFileParams) => {
const { path } = params;
const extension = path?.split('.')?.pop()?.toLowerCase() || '';
const buffer = fs.readFileSync(path);
const encoding = detectFileEncoding(buffer);
const { rawText } = await readRawContentByFileBuffer({
extension,
isQAImport: false,
teamId: params.teamId,
encoding,
buffer,
metadata: params.metadata
});
return {
rawText
};
};
export const readRawContentByFileBuffer = async ({
extension, extension,
isQAImport, isQAImport,
teamId, teamId,
...@@ -69,9 +97,3 @@ export const readFileRawContent = async ({ ...@@ -69,9 +97,3 @@ export const readFileRawContent = async ({
return { rawText }; return { rawText };
}; };
export const htmlToMarkdown = async (html?: string | null) => {
const md = await runWorker<string>(WorkerNameEnum.htmlStr2Md, { html: html || '' });
return simpleMarkdownText(md);
};
projects/app/src/service/middle/entry.ts → packages/service/common/middle/entry.ts
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '../response';
import type { NextApiResponse } from 'next'; import type { NextApiResponse } from 'next';
import { connectToDatabase } from '../mongo'; import { withNextCors } from './cors';
import { withNextCors } from '@fastgpt/service/common/middle/cors'; import { ApiRequestProps } from '../../type/next';
import { ApiRequestProps } from '@fastgpt/service/type/next';
export type NextApiHandler<T = any> = ( export type NextApiHandler<T = any> = (
req: ApiRequestProps, req: ApiRequestProps,
res: NextApiResponse<T> res: NextApiResponse<T>
) => unknown | Promise<unknown>; ) => unknown | Promise<unknown>;
export const NextAPI = (...args: NextApiHandler[]): NextApiHandler => { export const NextEntry = ({ beforeCallback = [] }: { beforeCallback?: Promise<any>[] }) => {
return async function api(req: ApiRequestProps, res: NextApiResponse) { return (...args: NextApiHandler[]): NextApiHandler => {
try { return async function api(req: ApiRequestProps, res: NextApiResponse) {
await Promise.all([withNextCors(req, res), connectToDatabase()]); try {
await Promise.all([withNextCors(req, res), ...beforeCallback]);
let response = null; let response = null;
for (const handler of args) { for (const handler of args) {
response = await handler(req, res); response = await handler(req, res);
} }
const contentType = res.getHeader('Content-Type'); const contentType = res.getHeader('Content-Type');
if ((!contentType || contentType === 'application/json') && !res.writableFinished) { if ((!contentType || contentType === 'application/json') && !res.writableFinished) {
return jsonRes(res, {
code: 200,
data: response
});
}
} catch (error) {
return jsonRes(res, { return jsonRes(res, {
code: 200, code: 500,
data: response error,
url: req.url
}); });
} }
} catch (error) { };
return jsonRes(res, {
code: 500,
error,
url: req.url
});
}
}; };
}; };
packages/service/common/string/cheerio.ts
import { UrlFetchParams, UrlFetchResponse } from '@fastgpt/global/common/file/api'; import { UrlFetchParams, UrlFetchResponse } from '@fastgpt/global/common/file/api';
import * as cheerio from 'cheerio'; import * as cheerio from 'cheerio';
import axios from 'axios'; import axios from 'axios';
import { htmlToMarkdown } from '../file/read/utils'; import { htmlToMarkdown } from './utils';
export const cheerioToHtml = ({ export const cheerioToHtml = ({
fetchUrl, fetchUrl,
......
packages/service/common/string/utils.ts 0 → 100644
import { simpleMarkdownText } from '@fastgpt/global/common/string/markdown';
import { WorkerNameEnum, runWorker } from '../../worker/utils';
export const htmlToMarkdown = async (html?: string | null) => {
const md = await runWorker<string>(WorkerNameEnum.htmlStr2Md, { html: html || '' });
return simpleMarkdownText(md);
};
packages/service/core/dataset/collection/controller.ts
...@@ -32,6 +32,9 @@ export async function createOneCollection({ ...@@ -32,6 +32,9 @@ export async function createOneCollection({
fileId, fileId,
rawLink, rawLink,
externalFileId,
externalFileUrl,
hashRawText, hashRawText,
rawTextLength, rawTextLength,
metadata = {}, metadata = {},
...@@ -61,6 +64,8 @@ export async function createOneCollection({ ...@@ -61,6 +64,8 @@ export async function createOneCollection({
fileId, fileId,
rawLink, rawLink,
externalFileId,
externalFileUrl,
rawTextLength, rawTextLength,
hashRawText, hashRawText,
......
packages/service/core/dataset/collection/schema.ts
...@@ -66,7 +66,11 @@ const DatasetCollectionSchema = new Schema({ ...@@ -66,7 +66,11 @@ const DatasetCollectionSchema = new Schema({
type: String type: String
}, },
sourceId: String, tags: {
type: [String],
default: []
},
// local file collection // local file collection
fileId: { fileId: {
type: Schema.Types.ObjectId, type: Schema.Types.ObjectId,
...@@ -74,13 +78,13 @@ const DatasetCollectionSchema = new Schema({ ...@@ -74,13 +78,13 @@ const DatasetCollectionSchema = new Schema({
}, },
// web link collection // web link collection
rawLink: String, rawLink: String,
// external collection // external collection
externalFileId: String,
// metadata // metadata
rawTextLength: Number, rawTextLength: Number,
hashRawText: String, hashRawText: String,
externalSourceUrl: String, // external import url externalFileUrl: String, // external import url
metadata: { metadata: {
type: Object, type: Object,
default: {} default: {}
......
packages/service/core/dataset/read.ts
...@@ -2,13 +2,20 @@ import { BucketNameEnum } from '@fastgpt/global/common/file/constants'; ...@@ -2,13 +2,20 @@ import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { DatasetSourceReadTypeEnum } from '@fastgpt/global/core/dataset/constants'; import { DatasetSourceReadTypeEnum } from '@fastgpt/global/core/dataset/constants';
import { readFileContentFromMongo } from '../../common/file/gridfs/controller'; import { readFileContentFromMongo } from '../../common/file/gridfs/controller';
import { urlsFetch } from '../../common/string/cheerio'; import { urlsFetch } from '../../common/string/cheerio';
import { rawTextBackupPrefix } from '@fastgpt/global/core/dataset/read';
import { parseCsvTable2Chunks } from './training/utils'; import { parseCsvTable2Chunks } from './training/utils';
import { TextSplitProps, splitText2Chunks } from '@fastgpt/global/common/string/textSplitter'; import { TextSplitProps, splitText2Chunks } from '@fastgpt/global/common/string/textSplitter';
import axios from 'axios'; import axios from 'axios';
import { readFileRawContent } from '../../common/file/read/utils'; import { readRawContentByFileBuffer } from '../../common/file/read/utils';
export const readFileRawTextByUrl = async ({ teamId, url }: { teamId: string; url: string }) => { export const readFileRawTextByUrl = async ({
teamId,
url,
relatedId
}: {
teamId: string;
url: string;
relatedId?: string;
}) => {
const response = await axios({ const response = await axios({
method: 'get', method: 'get',
url: url, url: url,
...@@ -18,11 +25,14 @@ export const readFileRawTextByUrl = async ({ teamId, url }: { teamId: string; ur ...@@ -18,11 +25,14 @@ export const readFileRawTextByUrl = async ({ teamId, url }: { teamId: string; ur
const buffer = Buffer.from(response.data, 'binary'); const buffer = Buffer.from(response.data, 'binary');
const { rawText } = await readFileRawContent({ const { rawText } = await readRawContentByFileBuffer({
extension, extension,
teamId, teamId,
buffer, buffer,
encoding: 'utf-8' encoding: 'utf-8',
metadata: {
relatedId
}
}); });
return rawText; return rawText;
...@@ -38,13 +48,15 @@ export const readDatasetSourceRawText = async ({ ...@@ -38,13 +48,15 @@ export const readDatasetSourceRawText = async ({
type, type,
sourceId, sourceId,
isQAImport, isQAImport,
selector selector,
relatedId
}: { }: {
teamId: string; teamId: string;
type: DatasetSourceReadTypeEnum; type: DatasetSourceReadTypeEnum;
sourceId: string; sourceId: string;
isQAImport?: boolean; isQAImport?: boolean;
selector?: string; selector?: string;
relatedId?: string;
}): Promise<string> => { }): Promise<string> => {
if (type === DatasetSourceReadTypeEnum.fileLocal) { if (type === DatasetSourceReadTypeEnum.fileLocal) {
const { rawText } = await readFileContentFromMongo({ const { rawText } = await readFileContentFromMongo({
...@@ -64,7 +76,8 @@ export const readDatasetSourceRawText = async ({ ...@@ -64,7 +76,8 @@ export const readDatasetSourceRawText = async ({
} else if (type === DatasetSourceReadTypeEnum.externalFile) { } else if (type === DatasetSourceReadTypeEnum.externalFile) {
const rawText = await readFileRawTextByUrl({ const rawText = await readFileRawTextByUrl({
teamId, teamId,
url: sourceId url: sourceId,
relatedId
}); });
return rawText; return rawText;
} }
......
packages/service/core/dataset/search/controller.ts
...@@ -18,6 +18,7 @@ import { countPromptTokens } from '../../../common/string/tiktoken/index'; ...@@ -18,6 +18,7 @@ import { countPromptTokens } from '../../../common/string/tiktoken/index';
import { datasetSearchResultConcat } from '@fastgpt/global/core/dataset/search/utils'; import { datasetSearchResultConcat } from '@fastgpt/global/core/dataset/search/utils';
import { hashStr } from '@fastgpt/global/common/string/tools'; import { hashStr } from '@fastgpt/global/common/string/tools';
import { jiebaSplit } from '../../../common/string/jieba'; import { jiebaSplit } from '../../../common/string/jieba';
import { getCollectionSourceData } from '@fastgpt/global/core/dataset/collection/utils';
type SearchDatasetDataProps = { type SearchDatasetDataProps = {
teamId: string; teamId: string;
...@@ -98,7 +99,7 @@ export async function searchDatasetData(props: SearchDatasetDataProps) { ...@@ -98,7 +99,7 @@ export async function searchDatasetData(props: SearchDatasetDataProps) {
}, },
'datasetId collectionId q a chunkIndex indexes' 'datasetId collectionId q a chunkIndex indexes'
) )
.populate('collectionId', 'name fileId rawLink') .populate('collectionId', 'name fileId rawLink externalFileId externalFileUrl')
.lean()) as DatasetDataWithCollectionType[]; .lean()) as DatasetDataWithCollectionType[];
// add score to data(It's already sorted. The first one is the one with the most points) // add score to data(It's already sorted. The first one is the one with the most points)
...@@ -130,8 +131,7 @@ export async function searchDatasetData(props: SearchDatasetDataProps) { ...@@ -130,8 +131,7 @@ export async function searchDatasetData(props: SearchDatasetDataProps) {
chunkIndex: data.chunkIndex, chunkIndex: data.chunkIndex,
datasetId: String(data.datasetId), datasetId: String(data.datasetId),
collectionId: String(data.collectionId?._id), collectionId: String(data.collectionId?._id),
sourceName: data.collectionId?.name || '', ...getCollectionSourceData(data.collectionId),
sourceId: data.collectionId?.fileId || data.collectionId?.rawLink,
score: [{ type: SearchScoreTypeEnum.embedding, value: data.score, index }] score: [{ type: SearchScoreTypeEnum.embedding, value: data.score, index }]
}; };
...@@ -205,8 +205,7 @@ export async function searchDatasetData(props: SearchDatasetDataProps) { ...@@ -205,8 +205,7 @@ export async function searchDatasetData(props: SearchDatasetDataProps) {
id: String(item._id), id: String(item._id),
datasetId: String(item.datasetId), datasetId: String(item.datasetId),
collectionId: String(item.collectionId), collectionId: String(item.collectionId),
sourceName: collection?.name || '', ...getCollectionSourceData(collection),
sourceId: collection?.fileId || collection?.rawLink,
q: item.q, q: item.q,
a: item.a, a: item.a,
chunkIndex: item.chunkIndex, chunkIndex: item.chunkIndex,
......
packages/service/core/dataset/training/controller.ts
...@@ -174,7 +174,7 @@ export async function pushDataListToTrainingQueue({ ...@@ -174,7 +174,7 @@ export async function pushDataListToTrainingQueue({
} catch (error: any) { } catch (error: any) {
addLog.error(`Insert error`, error); addLog.error(`Insert error`, error);
// 如果有错误，将失败的文档添加到失败列表中 // 如果有错误，将失败的文档添加到失败列表中
error.writeErrors.forEach((writeError: any) => { error.writeErrors?.forEach((writeError: any) => {
failedDocuments.push(data[writeError.index]); failedDocuments.push(data[writeError.index]);
}); });
console.log('failed', failedDocuments); console.log('failed', failedDocuments);
......
packages/service/core/dataset/training/schema.ts
...@@ -35,7 +35,7 @@ const TrainingDataSchema = new Schema({ ...@@ -35,7 +35,7 @@ const TrainingDataSchema = new Schema({
}, },
billId: { billId: {
// concat bill // concat bill
type: Schema.Types.ObjectId type: String
}, },
mode: { mode: {
type: String, type: String,
......
packages/service/core/workflow/dispatch/tools/runLaf.ts
...@@ -53,7 +53,7 @@ export const dispatchLafRequest = async (props: LafRequestProps): Promise<LafRes ...@@ -53,7 +53,7 @@ export const dispatchLafRequest = async (props: LafRequestProps): Promise<LafRes
appId, appId,
chatId, chatId,
responseChatItemId, responseChatItemId,
histories: histories.slice(0, 10) histories: histories?.slice(0, 10)
}, },
variables, variables,
...dynamicInput, ...dynamicInput,
......
packages/service/package.json
...@@ -21,7 +21,7 @@ ...@@ -21,7 +21,7 @@
"mammoth": "^1.6.0", "mammoth": "^1.6.0",
"mongoose": "^7.0.2", "mongoose": "^7.0.2",
"multer": "1.4.5-lts.1", "multer": "1.4.5-lts.1",
"next": "13.5.2", "next": "14.2.3",
"nextjs-cors": "^2.1.2", "nextjs-cors": "^2.1.2",
"node-cron": "^3.0.3", "node-cron": "^3.0.3",
"node-xlsx": "^0.23.0", "node-xlsx": "^0.23.0",
......
packages/service/support/permission/teamLimit.ts
...@@ -19,7 +19,9 @@ export const checkDatasetLimit = async ({ ...@@ -19,7 +19,9 @@ export const checkDatasetLimit = async ({
if (!standardConstants) return; if (!standardConstants) return;
if (usedSize + insertLen >= datasetMaxSize) { if (usedSize + insertLen >= datasetMaxSize) {
return Promise.reject(TeamErrEnum.datasetSizeNotEnough); return Promise.reject(
`您的知识库容量为: ${datasetMaxSize}组，已使用: ${usedSize}组，导入当前文件需要: ${insertLen}组，请增加知识库容量后导入。`
);
} }
if (usedPoints >= totalPoints) { if (usedPoints >= totalPoints) {
......
packages/service/worker/file/read.ts
...@@ -9,7 +9,7 @@ import { readXlsxRawText } from './extension/xlsx'; ...@@ -9,7 +9,7 @@ import { readXlsxRawText } from './extension/xlsx';
import { readCsvRawText } from './extension/csv'; import { readCsvRawText } from './extension/csv';
parentPort?.on('message', async (props: ReadRawTextProps<Uint8Array>) => { parentPort?.on('message', async (props: ReadRawTextProps<Uint8Array>) => {
const readFileRawContent = async (params: ReadRawTextByBuffer) => { const readRawContentByFileBuffer = async (params: ReadRawTextByBuffer) => {
switch (params.extension) { switch (params.extension) {
case 'txt': case 'txt':
case 'md': case 'md':
...@@ -41,7 +41,7 @@ parentPort?.on('message', async (props: ReadRawTextProps<Uint8Array>) => { ...@@ -41,7 +41,7 @@ parentPort?.on('message', async (props: ReadRawTextProps<Uint8Array>) => {
try { try {
parentPort?.postMessage({ parentPort?.postMessage({
type: 'success', type: 'success',
data: await readFileRawContent(newProps) data: await readRawContentByFileBuffer(newProps)
}); });
} catch (error) { } catch (error) {
console.log(error); console.log(error);
......
packages/web/components/common/Icon/constants.ts
...@@ -101,6 +101,7 @@ export const iconPaths = { ...@@ -101,6 +101,7 @@ export const iconPaths = {
'core/dataset/commonDataset': () => import('./icons/core/dataset/commonDataset.svg'), 'core/dataset/commonDataset': () => import('./icons/core/dataset/commonDataset.svg'),
'core/dataset/datasetFill': () => import('./icons/core/dataset/datasetFill.svg'), 'core/dataset/datasetFill': () => import('./icons/core/dataset/datasetFill.svg'),
'core/dataset/datasetLight': () => import('./icons/core/dataset/datasetLight.svg'), 'core/dataset/datasetLight': () => import('./icons/core/dataset/datasetLight.svg'),
'core/dataset/externalDataset': () => import('./icons/core/dataset/externalDataset.svg'),
'core/dataset/fileCollection': () => import('./icons/core/dataset/fileCollection.svg'), 'core/dataset/fileCollection': () => import('./icons/core/dataset/fileCollection.svg'),
'core/dataset/fullTextRecall': () => import('./icons/core/dataset/fullTextRecall.svg'), 'core/dataset/fullTextRecall': () => import('./icons/core/dataset/fullTextRecall.svg'),
'core/dataset/manualCollection': () => import('./icons/core/dataset/manualCollection.svg'), 'core/dataset/manualCollection': () => import('./icons/core/dataset/manualCollection.svg'),
......
packages/web/components/common/Icon/icons/core/dataset/externalDataset.svg 0 → 100644
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 23 20">
<path fill-rule="evenodd" clip-rule="evenodd"
d="M15.7233 1.21707C14.9554 1.3079 14.4221 2.00197 14.532 2.76731L16.6975 17.8447C16.8074 18.6101 17.519 19.1569 18.2868 19.066L21.5433 18.6808C22.3111 18.59 22.8445 17.8959 22.7345 17.1306L20.5691 2.05317C20.4592 1.28782 19.7476 0.741021 18.9797 0.831852L15.7233 1.21707ZM0.830017 2.32412C0.830017 1.55092 1.45682 0.924121 2.23002 0.924121H5.51815C6.29135 0.924121 6.91815 1.55092 6.91815 2.32412V17.675C6.91815 18.4482 6.29135 19.075 5.51815 19.075H2.23002C1.45682 19.075 0.830017 18.4482 0.830017 17.675V2.32412ZM7.9198 2.32412C7.9198 1.55092 8.5466 0.924121 9.3198 0.924121H12.6079C13.3811 0.924121 14.0079 1.55092 14.0079 2.32412V17.675C14.0079 18.4482 13.3811 19.075 12.6079 19.075H9.3198C8.5466 19.075 7.9198 18.4482 7.9198 17.675V2.32412Z"
fill="#8A95A7" />
</svg>
\ No newline at end of file
packages/web/components/common/Input/NumberInput/index.tsx
...@@ -10,6 +10,7 @@ import React from 'react'; ...@@ -10,6 +10,7 @@ import React from 'react';
type Props = Omit<NumberInputProps, 'onChange'> & { type Props = Omit<NumberInputProps, 'onChange'> & {
onChange: (e: number | '') => any; onChange: (e: number | '') => any;
placeholder?: string;
}; };
const MyNumberInput = (props: Props) => { const MyNumberInput = (props: Props) => {
...@@ -24,7 +25,7 @@ const MyNumberInput = (props: Props) => { ...@@ -24,7 +25,7 @@ const MyNumberInput = (props: Props) => {
} }
}} }}
> >
<NumberInputField /> <NumberInputField placeholder={props?.placeholder} />
<NumberInputStepper> <NumberInputStepper>
<NumberIncrementStepper /> <NumberIncrementStepper />
<NumberDecrementStepper /> <NumberDecrementStepper />
......
packages/web/components/common/Textarea/JsonEditor/index.tsx
...@@ -22,6 +22,7 @@ type Props = Omit<BoxProps, 'resize' | 'onChange'> & { ...@@ -22,6 +22,7 @@ type Props = Omit<BoxProps, 'resize' | 'onChange'> & {
onChange?: (e: string) => void; onChange?: (e: string) => void;
variables?: EditorVariablePickerType[]; variables?: EditorVariablePickerType[];
defaultHeight?: number; defaultHeight?: number;
placeholder?: string;
}; };
const options = { const options = {
......
packages/web/package.json
...@@ -27,18 +27,18 @@ ...@@ -27,18 +27,18 @@
"next-i18next": "15.2.0", "next-i18next": "15.2.0",
"papaparse": "^5.4.1", "papaparse": "^5.4.1",
"pdfjs-dist": "4.0.269", "pdfjs-dist": "4.0.269",
"react": "18.2.0", "react": "18.3.1",
"use-context-selector": "^1.4.4", "use-context-selector": "^1.4.4",
"react-day-picker": "^8.7.1", "react-day-picker": "^8.7.1",
"react-dom": "18.2.0", "react-dom": "18.3.1",
"react-i18next": "13.5.0", "react-i18next": "13.5.0",
"react-beautiful-dnd": "^13.1.1" "react-beautiful-dnd": "^13.1.1"
}, },
"devDependencies": { "devDependencies": {
"@types/lodash": "^4.14.191", "@types/lodash": "^4.14.191",
"@types/papaparse": "^5.3.7", "@types/papaparse": "^5.3.7",
"@types/react": "18.2.0", "@types/react": "18.3.0",
"@types/react-dom": "18.2.0", "@types/react-dom": "18.3.0",
"@types/react-beautiful-dnd": "^13.1.8" "@types/react-beautiful-dnd": "^13.1.8"
} }
} }
pnpm-lock.yaml
This source diff could not be displayed because it is too large. You can view the blob instead.
projects/app/i18n/en/dataset.json
{ {
"Collection tags": "Tags",
"Common Dataset": "Common dataset", "Common Dataset": "Common dataset",
"Common Dataset Desc": "Can be built by importing files, web links, or manual entry", "Common Dataset Desc": "Can be built by importing files, web links, or manual entry",
"Confirm to rebuild embedding tip": "Are you sure to switch the knowledge base index?\nSwitching index is a very heavy operation that requires re-indexing all the data in your knowledge base, which may take a long time. Please ensure that the remaining points in your account are sufficient.\n\nIn addition, you need to be careful to modify the applications that select this knowledge base to avoid mixing them with other index model knowledge bases.", "Confirm to rebuild embedding tip": "Are you sure to switch the knowledge base index?\nSwitching index is a very heavy operation that requires re-indexing all the data in your knowledge base, which may take a long time. Please ensure that the remaining points in your account are sufficient.\n\nIn addition, you need to be careful to modify the applications that select this knowledge base to avoid mixing them with other index model knowledge bases.",
...@@ -6,6 +7,7 @@ ...@@ -6,6 +7,7 @@
"External file Dataset Desc": "You can import files from an external file library to build a knowledge base. Files are not stored twice", "External file Dataset Desc": "You can import files from an external file library to build a knowledge base. Files are not stored twice",
"External id": "File id", "External id": "File id",
"External read url": "External read url", "External read url": "External read url",
"External read url tip": "You can configure the reading address of your file library. This allows users to read and authenticate. You can currently use the {{fileId}} variable to refer to the external file ID.",
"External url": "File read url", "External url": "File read url",
"Folder Dataset": "Folder", "Folder Dataset": "Folder",
"Rebuild embedding start tip": "The task of switching index models has begun", "Rebuild embedding start tip": "The task of switching index models has begun",
......
projects/app/i18n/en/file.json
{ {
"Click to view raw source": "View source",
"Click to view file": "Click to view the original file", "Click to view file": "Click to view the original file",
"Release the mouse to upload the file": "Release the mouse to upload the file", "Release the mouse to upload the file": "Release the mouse to upload the file",
"upload error description": "Only supports uploading multiple files or one folder at a time", "upload error description": "Only supports uploading multiple files or one folder at a time",
......
projects/app/i18n/zh/dataset.json
{ {
"Collection tags": "集合标签",
"Common Dataset": "通用知识库", "Common Dataset": "通用知识库",
"Common Dataset Desc": "可通过导入文件、网页链接或手动录入形式构建知识库", "Common Dataset Desc": "可通过导入文件、网页链接或手动录入形式构建知识库",
"Confirm to rebuild embedding tip": "确认为知识库切换索引？\n切换索引是一个非常重量的操作，需要对您知识库内所有数据进行重新索引，时间可能较长，请确保账号内剩余积分充足。\n\n此外，你还需要注意修改选择该知识库的应用，避免它们与其他索引模型知识库混用。", "Confirm to rebuild embedding tip": "确认为知识库切换索引？\n切换索引是一个非常重量的操作，需要对您知识库内所有数据进行重新索引，时间可能较长，请确保账号内剩余积分充足。\n\n此外，你还需要注意修改选择该知识库的应用，避免它们与其他索引模型知识库混用。",
...@@ -6,6 +7,7 @@ ...@@ -6,6 +7,7 @@
"External file Dataset Desc": "可以从外部文件库导入文件构建知识库，文件不会进行二次存储", "External file Dataset Desc": "可以从外部文件库导入文件构建知识库，文件不会进行二次存储",
"External id": "文件阅读ID", "External id": "文件阅读ID",
"External read url": "外部预览地址", "External read url": "外部预览地址",
"External read url tip": "可以配置你文件库的阅读地址。便于对用户进行阅读鉴权操作。目前可以使用 {{fileId}} 变量来指代外部文件ID。",
"External url": "文件访问URL", "External url": "文件访问URL",
"Folder Dataset": "文件夹", "Folder Dataset": "文件夹",
"Rebuild embedding start tip": "切换索引模型任务已开始", "Rebuild embedding start tip": "切换索引模型任务已开始",
......
projects/app/i18n/zh/file.json
{ {
"Click to view raw source": "点击查看来源",
"Click to view file": "点击查看原始文件", "Click to view file": "点击查看原始文件",
"Release the mouse to upload the file": "松开鼠标上传文件", "Release the mouse to upload the file": "松开鼠标上传文件",
"upload error description": "单次只支持上传多个文件或者一个文件夹", "upload error description": "单次只支持上传多个文件或者一个文件夹",
......
projects/app/next.config.js
...@@ -88,8 +88,6 @@ const nextConfig = { ...@@ -88,8 +88,6 @@ const nextConfig = {
}, },
transpilePackages: ['@fastgpt/*', 'ahooks', '@chakra-ui/*', 'react'], transpilePackages: ['@fastgpt/*', 'ahooks', '@chakra-ui/*', 'react'],
experimental: { experimental: {
// 外部包独立打包
serverComponentsExternalPackages: ['mongoose', 'pg'],
// 指定导出包优化，按需引入包模块 // 指定导出包优化，按需引入包模块
optimizePackageImports: ['mongoose', 'pg'], optimizePackageImports: ['mongoose', 'pg'],
outputFileTracingRoot: path.join(__dirname, '../../') outputFileTracingRoot: path.join(__dirname, '../../')
......
projects/app/package.json
...@@ -42,13 +42,13 @@ ...@@ -42,13 +42,13 @@
"lodash": "^4.17.21", "lodash": "^4.17.21",
"mermaid": "^10.2.3", "mermaid": "^10.2.3",
"nanoid": "^4.0.1", "nanoid": "^4.0.1",
"next": "13.5.2", "next": "14.2.3",
"next-i18next": "15.2.0", "next-i18next": "15.2.0",
"nextjs-node-loader": "^1.1.5", "nextjs-node-loader": "^1.1.5",
"nprogress": "^0.2.0", "nprogress": "^0.2.0",
"react": "18.2.0", "react": "18.3.1",
"react-day-picker": "^8.7.1", "react-day-picker": "^8.7.1",
"react-dom": "18.2.0", "react-dom": "18.3.1",
"react-hook-form": "7.43.1", "react-hook-form": "7.43.1",
"react-i18next": "13.5.0", "react-i18next": "13.5.0",
"react-markdown": "^8.0.7", "react-markdown": "^8.0.7",
...@@ -71,12 +71,12 @@ ...@@ -71,12 +71,12 @@
"@types/jsonwebtoken": "^9.0.3", "@types/jsonwebtoken": "^9.0.3",
"@types/lodash": "^4.14.191", "@types/lodash": "^4.14.191",
"@types/node": "^20.8.5", "@types/node": "^20.8.5",
"@types/react": "18.2.0", "@types/react": "18.3.0",
"@types/react-dom": "18.2.0", "@types/react-dom": "18.3.0",
"@types/react-syntax-highlighter": "^15.5.6", "@types/react-syntax-highlighter": "^15.5.6",
"@types/request-ip": "^0.0.37", "@types/request-ip": "^0.0.37",
"eslint": "8.34.0", "eslint": "8.34.0",
"eslint-config-next": "13.1.6", "eslint-config-next": "14.2.3",
"nextjs-node-loader": "^1.1.5", "nextjs-node-loader": "^1.1.5",
"typescript": "4.9.5" "typescript": "4.9.5"
} }
......
projects/app/src/components/ChatBox/QuoteModal.tsx
...@@ -46,7 +46,7 @@ const QuoteModal = ({ ...@@ -46,7 +46,7 @@ const QuoteModal = ({
title={ title={
<Box> <Box>
{metadata ? ( {metadata ? (
<RawSourceBox {...metadata} canView={false} /> <RawSourceBox {...metadata} canView={showDetail} />
) : ( ) : (
<>{t('core.chat.Quote Amount', { amount: rawSearch.length })}</> <>{t('core.chat.Quote Amount', { amount: rawSearch.length })}</>
)} )}
......
projects/app/src/components/Markdown/index.tsx
...@@ -14,7 +14,6 @@ import MyTooltip from '../MyTooltip'; ...@@ -14,7 +14,6 @@ import MyTooltip from '../MyTooltip';
import { useTranslation } from 'next-i18next'; import { useTranslation } from 'next-i18next';
import { EventNameEnum, eventBus } from '@/web/common/utils/eventbus'; import { EventNameEnum, eventBus } from '@/web/common/utils/eventbus';
import MyIcon from '@fastgpt/web/components/common/Icon'; import MyIcon from '@fastgpt/web/components/common/Icon';
import { getFileAndOpen } from '@/web/core/dataset/utils';
import { MARKDOWN_QUOTE_SIGN } from '@fastgpt/global/core/chat/constants'; import { MARKDOWN_QUOTE_SIGN } from '@fastgpt/global/core/chat/constants';
const CodeLight = dynamic(() => import('./CodeLight'), { ssr: false }); const CodeLight = dynamic(() => import('./CodeLight'), { ssr: false });
...@@ -132,7 +131,7 @@ const A = React.memo(function A({ children, ...props }: any) { ...@@ -132,7 +131,7 @@ const A = React.memo(function A({ children, ...props }: any) {
); );
} }
// quote link // quote link(未使用)
if (children?.length === 1 && typeof children?.[0] === 'string') { if (children?.length === 1 && typeof children?.[0] === 'string') {
const text = String(children); const text = String(children);
if (text === MARKDOWN_QUOTE_SIGN && props.href) { if (text === MARKDOWN_QUOTE_SIGN && props.href) {
...@@ -147,7 +146,7 @@ const A = React.memo(function A({ children, ...props }: any) { ...@@ -147,7 +146,7 @@ const A = React.memo(function A({ children, ...props }: any) {
_hover={{ _hover={{
color: 'primary.700' color: 'primary.700'
}} }}
onClick={() => getFileAndOpen(props.href)} // onClick={() => getCollectionSourceAndOpen(props.href)}
/> />
</MyTooltip> </MyTooltip>
); );
......
projects/app/src/components/core/dataset/QuoteItem.tsx
...@@ -218,6 +218,7 @@ const QuoteItem = ({ ...@@ -218,6 +218,7 @@ const QuoteItem = ({
<RawSourceBox <RawSourceBox
fontWeight={'bold'} fontWeight={'bold'}
color={'black'} color={'black'}
collectionId={quoteItem.collectionId}
sourceName={quoteItem.sourceName} sourceName={quoteItem.sourceName}
sourceId={quoteItem.sourceId} sourceId={quoteItem.sourceId}
canView={canViewSource} canView={canViewSource}
......
projects/app/src/components/core/dataset/RawSourceBox.tsx
import React, { useMemo } from 'react'; import React, { useMemo } from 'react';
import { Box, BoxProps } from '@chakra-ui/react'; import { Box, BoxProps } from '@chakra-ui/react';
import { useToast } from '@fastgpt/web/hooks/useToast';
import { getErrText } from '@fastgpt/global/common/error/utils';
import MyTooltip from '@/components/MyTooltip'; import MyTooltip from '@/components/MyTooltip';
import { useTranslation } from 'next-i18next'; import { useTranslation } from 'next-i18next';
import { getFileAndOpen } from '@/web/core/dataset/utils'; import { getCollectionSourceAndOpen } from '@/web/core/dataset/hooks/readCollectionSource';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import { getSourceNameIcon } from '@fastgpt/global/core/dataset/utils'; import { getSourceNameIcon } from '@fastgpt/global/core/dataset/utils';
import MyIcon from '@fastgpt/web/components/common/Icon'; import MyIcon from '@fastgpt/web/components/common/Icon';
import { useI18n } from '@/web/context/I18n'; import { useI18n } from '@/web/context/I18n';
type Props = BoxProps & { type Props = BoxProps & {
sourceName?: string; sourceName?: string;
collectionId: string;
sourceId?: string; sourceId?: string;
canView?: boolean; canView?: boolean;
}; };
const RawSourceBox = ({ sourceId, sourceName = '', canView = true, ...props }: Props) => { const RawSourceBox = ({
sourceId,
collectionId,
sourceName = '',
canView = true,
...props
}: Props) => {
const { t } = useTranslation(); const { t } = useTranslation();
const { fileT } = useI18n(); const { fileT } = useI18n();
const { toast } = useToast();
const { setLoading } = useSystemStore();
const canPreview = useMemo(() => !!sourceId && canView, [canView, sourceId]); const canPreview = !!sourceId && canView;
const icon = useMemo(() => getSourceNameIcon({ sourceId, sourceName }), [sourceId, sourceName]); const icon = useMemo(() => getSourceNameIcon({ sourceId, sourceName }), [sourceId, sourceName]);
const read = getCollectionSourceAndOpen(collectionId);
return ( return (
<MyTooltip label={canPreview ? fileT('Click to view file') : ''} shouldWrapChildren={false}> <MyTooltip
label={canPreview ? fileT('Click to view raw source') : ''}
shouldWrapChildren={false}
>
<Box <Box
color={'myGray.900'} color={'myGray.900'}
fontWeight={'medium'} fontWeight={'medium'}
...@@ -37,18 +43,7 @@ const RawSourceBox = ({ sourceId, sourceName = '', canView = true, ...props }: P ...@@ -37,18 +43,7 @@ const RawSourceBox = ({ sourceId, sourceName = '', canView = true, ...props }: P
? { ? {
cursor: 'pointer', cursor: 'pointer',
textDecoration: 'underline', textDecoration: 'underline',
onClick: async () => { onClick: read
setLoading(true);
try {
await getFileAndOpen(sourceId as string);
} catch (error) {
toast({
title: getErrText(error, t('error.fileNotFound')),
status: 'error'
});
}
setLoading(false);
}
} }
: {})} : {})}
{...props} {...props}
......
projects/app/src/components/core/workflow/Flow/nodes/NodeIfElse/ListItem.tsx
...@@ -4,11 +4,11 @@ import { ...@@ -4,11 +4,11 @@ import {
DraggableStateSnapshot DraggableStateSnapshot
} from '@fastgpt/web/components/common/DndDrag/index'; } from '@fastgpt/web/components/common/DndDrag/index';
import Container from '../../components/Container'; import Container from '../../components/Container';
import { DragHandleIcon, MinusIcon, SmallAddIcon } from '@chakra-ui/icons'; import { MinusIcon, SmallAddIcon } from '@chakra-ui/icons';
import { IfElseListItemType } from '@fastgpt/global/core/workflow/template/system/ifElse/type'; import { IfElseListItemType } from '@fastgpt/global/core/workflow/template/system/ifElse/type';
import MyIcon from '@fastgpt/web/components/common/Icon'; import MyIcon from '@fastgpt/web/components/common/Icon';
import { ReferenceValueProps } from '@fastgpt/global/core/workflow/type/io'; import { ReferenceValueProps } from '@fastgpt/global/core/workflow/type/io';
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'next-i18next';
import { ReferSelector, useReference } from '../render/RenderInput/templates/Reference'; import { ReferSelector, useReference } from '../render/RenderInput/templates/Reference';
import { WorkflowIOValueTypeEnum } from '@fastgpt/global/core/workflow/constants'; import { WorkflowIOValueTypeEnum } from '@fastgpt/global/core/workflow/constants';
import { import {
......
projects/app/src/components/core/workflow/Flow/nodes/NodeVariableUpdate.tsx
...@@ -2,7 +2,7 @@ import React, { useCallback, useMemo } from 'react'; ...@@ -2,7 +2,7 @@ import React, { useCallback, useMemo } from 'react';
import NodeCard from './render/NodeCard'; import NodeCard from './render/NodeCard';
import { NodeProps } from 'reactflow'; import { NodeProps } from 'reactflow';
import { FlowNodeItemType } from '@fastgpt/global/core/workflow/type'; import { FlowNodeItemType } from '@fastgpt/global/core/workflow/type';
import { useTranslation } from 'react-i18next'; import { useTranslation } from 'next-i18next';
import { import {
Box, Box,
Button, Button,
......
projects/app/src/pages/api/admin/initv481.ts
...@@ -3,7 +3,7 @@ import { jsonRes } from '@fastgpt/service/common/response'; ...@@ -3,7 +3,7 @@ import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo'; import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common'; import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { PgClient } from '@fastgpt/service/common/vectorStore/pg'; import { PgClient } from '@fastgpt/service/common/vectorStore/pg';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
import { PgDatasetTableName } from '@fastgpt/global/common/vectorStore/constants'; import { PgDatasetTableName } from '@fastgpt/global/common/vectorStore/constants';
import { connectionMongo } from '@fastgpt/service/common/mongo'; import { connectionMongo } from '@fastgpt/service/common/mongo';
import { addLog } from '@fastgpt/service/common/system/log'; import { addLog } from '@fastgpt/service/common/system/log';
......
projects/app/src/pages/api/common/file/previewContent.ts
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
import type { NextApiResponse } from 'next'; import type { NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '@fastgpt/service/common/response';
import { authFile } from '@fastgpt/service/support/permission/auth/file'; import { authFile } from '@fastgpt/service/support/permission/auth/file';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
import { DatasetSourceReadTypeEnum } from '@fastgpt/global/core/dataset/constants'; import { DatasetSourceReadTypeEnum } from '@fastgpt/global/core/dataset/constants';
import { readDatasetSourceRawText } from '@fastgpt/service/core/dataset/read'; import { readDatasetSourceRawText } from '@fastgpt/service/core/dataset/read';
import { ApiRequestProps } from '@fastgpt/service/type/next'; import { ApiRequestProps } from '@fastgpt/service/type/next';
......
projects/app/src/pages/api/core/ai/token.ts
import type { ApiRequestProps, ApiResponseType } from '@fastgpt/service/type/next'; import type { ApiRequestProps, ApiResponseType } from '@fastgpt/service/type/next';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
import { authCert } from '@fastgpt/service/support/permission/auth/common'; import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { ChatCompletionMessageParam } from '@fastgpt/global/core/ai/type'; import { ChatCompletionMessageParam } from '@fastgpt/global/core/ai/type';
import { countGptMessagesTokens } from '@fastgpt/service/common/string/tiktoken'; import { countGptMessagesTokens } from '@fastgpt/service/common/string/tiktoken';
......
projects/app/src/pages/api/core/app/create.ts
...@@ -7,7 +7,7 @@ import { authUserNotVisitor } from '@fastgpt/service/support/permission/auth/use ...@@ -7,7 +7,7 @@ import { authUserNotVisitor } from '@fastgpt/service/support/permission/auth/use
import { checkTeamAppLimit } from '@fastgpt/service/support/permission/teamLimit'; import { checkTeamAppLimit } from '@fastgpt/service/support/permission/teamLimit';
import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun'; import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun';
import { MongoAppVersion } from '@fastgpt/service/core/app/versionSchema'; import { MongoAppVersion } from '@fastgpt/service/core/app/versionSchema';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
const { const {
......
projects/app/src/pages/api/core/app/del.ts
...@@ -6,7 +6,7 @@ import { authApp } from '@fastgpt/service/support/permission/auth/app'; ...@@ -6,7 +6,7 @@ import { authApp } from '@fastgpt/service/support/permission/auth/app';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema'; import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun'; import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun';
import { MongoAppVersion } from '@fastgpt/service/core/app/versionSchema'; import { MongoAppVersion } from '@fastgpt/service/core/app/versionSchema';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
const { appId } = req.query as { appId: string }; const { appId } = req.query as { appId: string };
......
projects/app/src/pages/api/core/app/detail.tsx
...@@ -2,7 +2,7 @@ import type { NextApiRequest, NextApiResponse } from 'next'; ...@@ -2,7 +2,7 @@ import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo'; import { connectToDatabase } from '@/service/mongo';
import { authApp } from '@fastgpt/service/support/permission/auth/app'; import { authApp } from '@fastgpt/service/support/permission/auth/app';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
/* 获取我的模型 */ /* 获取我的模型 */
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
......
projects/app/src/pages/api/core/app/getChatLogs.ts
...@@ -7,7 +7,7 @@ import { addDays } from 'date-fns'; ...@@ -7,7 +7,7 @@ import { addDays } from 'date-fns';
import type { GetAppChatLogsParams } from '@/global/core/api/appReq.d'; import type { GetAppChatLogsParams } from '@/global/core/api/appReq.d';
import { authApp } from '@fastgpt/service/support/permission/auth/app'; import { authApp } from '@fastgpt/service/support/permission/auth/app';
import { ChatItemCollectionName } from '@fastgpt/service/core/chat/chatItemSchema'; import { ChatItemCollectionName } from '@fastgpt/service/core/chat/chatItemSchema';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler( async function handler(
req: NextApiRequest, req: NextApiRequest,
......
projects/app/src/pages/api/core/app/list.ts
...@@ -3,7 +3,7 @@ import { MongoApp } from '@fastgpt/service/core/app/schema'; ...@@ -3,7 +3,7 @@ import { MongoApp } from '@fastgpt/service/core/app/schema';
import { mongoRPermission } from '@fastgpt/global/support/permission/utils'; import { mongoRPermission } from '@fastgpt/global/support/permission/utils';
import { AppListItemType } from '@fastgpt/global/core/app/type'; import { AppListItemType } from '@fastgpt/global/core/app/type';
import { authUserRole } from '@fastgpt/service/support/permission/auth/user'; import { authUserRole } from '@fastgpt/service/support/permission/auth/user';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>): Promise<AppListItemType[]> { async function handler(req: NextApiRequest, res: NextApiResponse<any>): Promise<AppListItemType[]> {
// 凭证校验 // 凭证校验
......
projects/app/src/pages/api/core/app/update.ts
...@@ -3,7 +3,7 @@ import { MongoApp } from '@fastgpt/service/core/app/schema'; ...@@ -3,7 +3,7 @@ import { MongoApp } from '@fastgpt/service/core/app/schema';
import type { AppUpdateParams } from '@/global/core/app/api'; import type { AppUpdateParams } from '@/global/core/app/api';
import { authApp } from '@fastgpt/service/support/permission/auth/app'; import { authApp } from '@fastgpt/service/support/permission/auth/app';
import { beforeUpdateAppFormat } from '@fastgpt/service/core/app/controller'; import { beforeUpdateAppFormat } from '@fastgpt/service/core/app/controller';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
/* 获取我的模型 */ /* 获取我的模型 */
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
......
projects/app/src/pages/api/core/app/version/list.ts
import type { NextApiRequest, NextApiResponse } from 'next'; import type { NextApiRequest, NextApiResponse } from 'next';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
import { MongoAppVersion } from '@fastgpt/service/core/app/versionSchema'; import { MongoAppVersion } from '@fastgpt/service/core/app/versionSchema';
import { PaginationProps, PaginationResponse } from '@fastgpt/web/common/fetch/type'; import { PaginationProps, PaginationResponse } from '@fastgpt/web/common/fetch/type';
import { AppVersionSchemaType } from '@fastgpt/global/core/app/version'; import { AppVersionSchemaType } from '@fastgpt/global/core/app/version';
......
projects/app/src/pages/api/core/app/version/publish.ts
import type { NextApiRequest, NextApiResponse } from 'next'; import type { NextApiRequest, NextApiResponse } from 'next';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
import { authApp } from '@fastgpt/service/support/permission/auth/app'; import { authApp } from '@fastgpt/service/support/permission/auth/app';
import { MongoAppVersion } from '@fastgpt/service/core/app/versionSchema'; import { MongoAppVersion } from '@fastgpt/service/core/app/versionSchema';
import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun'; import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun';
......
projects/app/src/pages/api/core/app/version/revert.ts
import type { NextApiRequest, NextApiResponse } from 'next'; import type { NextApiRequest, NextApiResponse } from 'next';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
import { authApp } from '@fastgpt/service/support/permission/auth/app'; import { authApp } from '@fastgpt/service/support/permission/auth/app';
import { MongoAppVersion } from '@fastgpt/service/core/app/versionSchema'; import { MongoAppVersion } from '@fastgpt/service/core/app/versionSchema';
import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun'; import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun';
......
projects/app/src/pages/api/core/chat/init.ts
...@@ -9,7 +9,7 @@ import { getChatItems } from '@fastgpt/service/core/chat/controller'; ...@@ -9,7 +9,7 @@ import { getChatItems } from '@fastgpt/service/core/chat/controller';
import { ChatErrEnum } from '@fastgpt/global/common/error/code/chat'; import { ChatErrEnum } from '@fastgpt/global/common/error/code/chat';
import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/workflow/runtime/constants'; import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/workflow/runtime/constants';
import { getAppLatestVersion } from '@fastgpt/service/core/app/controller'; import { getAppLatestVersion } from '@fastgpt/service/core/app/controller';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler( async function handler(
req: NextApiRequest, req: NextApiRequest,
......
projects/app/src/pages/api/core/dataset/allDataset.ts
...@@ -5,7 +5,7 @@ import type { DatasetSimpleItemType } from '@fastgpt/global/core/dataset/type.d' ...@@ -5,7 +5,7 @@ import type { DatasetSimpleItemType } from '@fastgpt/global/core/dataset/type.d'
import { mongoRPermission } from '@fastgpt/global/support/permission/utils'; import { mongoRPermission } from '@fastgpt/global/support/permission/utils';
import { authUserRole } from '@fastgpt/service/support/permission/auth/user'; import { authUserRole } from '@fastgpt/service/support/permission/auth/user';
import { DatasetTypeEnum } from '@fastgpt/global/core/dataset/constants'; import { DatasetTypeEnum } from '@fastgpt/global/core/dataset/constants';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
/* get all dataset by teamId or tmbId */ /* get all dataset by teamId or tmbId */
async function handler( async function handler(
......
projects/app/src/pages/api/core/dataset/collection/create/csvTable.ts
...@@ -17,8 +17,6 @@ import { pushDataListToTrainingQueue } from '@fastgpt/service/core/dataset/train ...@@ -17,8 +17,6 @@ import { pushDataListToTrainingQueue } from '@fastgpt/service/core/dataset/train
import { createTrainingUsage } from '@fastgpt/service/support/wallet/usage/controller'; import { createTrainingUsage } from '@fastgpt/service/support/wallet/usage/controller';
import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants'; import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants';
import { getLLMModel, getVectorModel } from '@fastgpt/service/core/ai/model'; import { getLLMModel, getVectorModel } from '@fastgpt/service/core/ai/model';
import { parseCsvTable2Chunks } from '@fastgpt/service/core/dataset/training/utils';
import { startTrainingQueue } from '@/service/core/dataset/training/utils';
import { rawText2Chunks } from '@fastgpt/service/core/dataset/read'; import { rawText2Chunks } from '@fastgpt/service/core/dataset/read';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) { export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
...@@ -106,8 +104,6 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse< ...@@ -106,8 +104,6 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
return collectionId; return collectionId;
}); });
startTrainingQueue(true);
jsonRes(res); jsonRes(res);
} catch (error) { } catch (error) {
jsonRes(res, { jsonRes(res, {
......
projects/app/src/pages/api/core/dataset/collection/create/file.ts → projects/app/src/pages/api/core/dataset/collection/create/fileId.ts
import type { NextApiRequest, NextApiResponse } from 'next'; import type { NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo'; import { connectToDatabase } from '@/service/mongo';
import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller'; import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller';
...@@ -12,7 +12,6 @@ import { ...@@ -12,7 +12,6 @@ import {
import { BucketNameEnum } from '@fastgpt/global/common/file/constants'; import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun'; import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun';
import { MongoImage } from '@fastgpt/service/common/file/image/schema'; import { MongoImage } from '@fastgpt/service/common/file/image/schema';
import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter';
import { checkDatasetLimit } from '@fastgpt/service/support/permission/teamLimit'; import { checkDatasetLimit } from '@fastgpt/service/support/permission/teamLimit';
import { predictDataLimitLength } from '@fastgpt/global/core/dataset/utils'; import { predictDataLimitLength } from '@fastgpt/global/core/dataset/utils';
import { pushDataListToTrainingQueue } from '@fastgpt/service/core/dataset/training/controller'; import { pushDataListToTrainingQueue } from '@fastgpt/service/core/dataset/training/controller';
...@@ -20,11 +19,15 @@ import { createTrainingUsage } from '@fastgpt/service/support/wallet/usage/contr ...@@ -20,11 +19,15 @@ import { createTrainingUsage } from '@fastgpt/service/support/wallet/usage/contr
import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants'; import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants';
import { getLLMModel, getVectorModel } from '@fastgpt/service/core/ai/model'; import { getLLMModel, getVectorModel } from '@fastgpt/service/core/ai/model';
import { hashStr } from '@fastgpt/global/common/string/tools'; import { hashStr } from '@fastgpt/global/common/string/tools';
import { startTrainingQueue } from '@/service/core/dataset/training/utils';
import { MongoRawTextBuffer } from '@fastgpt/service/common/buffer/rawText/schema'; import { MongoRawTextBuffer } from '@fastgpt/service/common/buffer/rawText/schema';
import { rawText2Chunks } from '@fastgpt/service/core/dataset/read'; import { rawText2Chunks } from '@fastgpt/service/core/dataset/read';
import { NextAPI } from '@/service/middleware/entry';
import { ApiRequestProps } from '@fastgpt/service/type/next';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(
req: ApiRequestProps<FileIdCreateDatasetCollectionParams>,
res: NextApiResponse<any>
) {
const { const {
fileId, fileId,
trainingType = TrainingModeEnum.chunk, trainingType = TrainingModeEnum.chunk,
...@@ -32,122 +35,115 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse< ...@@ -32,122 +35,115 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
chunkSplitter, chunkSplitter,
qaPrompt, qaPrompt,
...body ...body
} = req.body as FileIdCreateDatasetCollectionParams; } = req.body;
try { await connectToDatabase();
await connectToDatabase();
const { teamId, tmbId, dataset } = await authDataset({ const { teamId, tmbId, dataset } = await authDataset({
req, req,
authToken: true, authToken: true,
authApiKey: true, authApiKey: true,
per: 'w', per: 'w',
datasetId: body.datasetId datasetId: body.datasetId
}); });
// 1. read file // 1. read file
const { rawText, filename } = await readFileContentFromMongo({ const { rawText, filename } = await readFileContentFromMongo({
teamId, teamId,
bucketName: BucketNameEnum.dataset, bucketName: BucketNameEnum.dataset,
fileId fileId
}); });
// 2. split chunks // 2. split chunks
const chunks = rawText2Chunks({ const chunks = rawText2Chunks({
rawText, rawText,
chunkLen: chunkSize, chunkLen: chunkSize,
overlapRatio: trainingType === TrainingModeEnum.chunk ? 0.2 : 0, overlapRatio: trainingType === TrainingModeEnum.chunk ? 0.2 : 0,
customReg: chunkSplitter ? [chunkSplitter] : [] customReg: chunkSplitter ? [chunkSplitter] : []
}); });
// 3. auth limit
await checkDatasetLimit({
teamId,
insertLen: predictDataLimitLength(trainingType, chunks)
});
// 3. auth limit await mongoSessionRun(async (session) => {
await checkDatasetLimit({ // 4. create collection
const { _id: collectionId } = await createOneCollection({
...body,
teamId, teamId,
insertLen: predictDataLimitLength(trainingType, chunks) tmbId,
}); type: DatasetCollectionTypeEnum.file,
name: filename,
fileId,
metadata: {
relatedImgId: fileId
},
await mongoSessionRun(async (session) => { // special metadata
// 4. create collection trainingType,
const { _id: collectionId } = await createOneCollection({ chunkSize,
...body, chunkSplitter,
teamId, qaPrompt,
tmbId,
type: DatasetCollectionTypeEnum.file,
name: filename,
fileId,
metadata: {
relatedImgId: fileId
},
// special metadata hashRawText: hashStr(rawText),
trainingType, rawTextLength: rawText.length,
chunkSize, session
chunkSplitter, });
qaPrompt,
hashRawText: hashStr(rawText), // 5. create training bill
rawTextLength: rawText.length, const { billId } = await createTrainingUsage({
session teamId,
}); tmbId,
appName: filename,
billSource: UsageSourceEnum.training,
vectorModel: getVectorModel(dataset.vectorModel)?.name,
agentModel: getLLMModel(dataset.agentModel)?.name,
session
});
// 5. create training bill // 6. insert to training queue
const { billId } = await createTrainingUsage({ await pushDataListToTrainingQueue({
teamId, teamId,
tmbId, tmbId,
appName: filename, datasetId: dataset._id,
billSource: UsageSourceEnum.training, collectionId,
vectorModel: getVectorModel(dataset.vectorModel)?.name, agentModel: dataset.agentModel,
agentModel: getLLMModel(dataset.agentModel)?.name, vectorModel: dataset.vectorModel,
session trainingMode: trainingType,
}); prompt: qaPrompt,
billId,
data: chunks.map((item, index) => ({
...item,
chunkIndex: index
})),
session
});
// 6. insert to training queue // 7. remove related image ttl
await pushDataListToTrainingQueue({ await MongoImage.updateMany(
{
teamId, teamId,
tmbId, 'metadata.relatedId': fileId
datasetId: dataset._id, },
collectionId, {
agentModel: dataset.agentModel, // Remove expiredTime to avoid ttl expiration
vectorModel: dataset.vectorModel, $unset: {
trainingMode: trainingType, expiredTime: 1
prompt: qaPrompt,
billId,
data: chunks.map((item, index) => ({
...item,
chunkIndex: index
})),
session
});
// 7. remove related image ttl
await MongoImage.updateMany(
{
teamId,
'metadata.relatedId': fileId
},
{
// Remove expiredTime to avoid ttl expiration
$unset: {
expiredTime: 1
}
},
{
session
} }
); },
{
return collectionId; session
}); }
);
// remove buffer return collectionId;
await MongoRawTextBuffer.deleteOne({ sourceId: fileId }); });
startTrainingQueue(true); // remove buffer
await MongoRawTextBuffer.deleteOne({ sourceId: fileId });
jsonRes(res); jsonRes(res);
} catch (error) {
jsonRes(res, {
code: 500,
error
});
}
} }
export default NextAPI(handler);
projects/app/src/pages/api/core/dataset/collection/create/localFile.ts 0 → 100644
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { uploadFile } from '@fastgpt/service/common/file/gridfs/controller';
import { getUploadModel } from '@fastgpt/service/common/file/multer';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { FileCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api';
import { removeFilesByPaths } from '@fastgpt/service/common/file/utils';
import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller';
import {
DatasetCollectionTypeEnum,
TrainingModeEnum
} from '@fastgpt/global/core/dataset/constants';
import { getNanoid, hashStr } from '@fastgpt/global/common/string/tools';
import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter';
import { checkDatasetLimit } from '@fastgpt/service/support/permission/teamLimit';
import { predictDataLimitLength } from '@fastgpt/global/core/dataset/utils';
import { pushDataListToTrainingQueue } from '@fastgpt/service/core/dataset/training/controller';
import { createTrainingUsage } from '@fastgpt/service/support/wallet/usage/controller';
import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants';
import { getDatasetModel, getVectorModel } from '@fastgpt/service/core/ai/model';
import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun';
import { MongoImage } from '@fastgpt/service/common/file/image/schema';
import { readRawTextByLocalFile } from '@fastgpt/service/common/file/read/utils';
import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
/**
* Creates the multer uploader
*/
const upload = getUploadModel({
maxSize: (global.feConfigs?.uploadFileMaxSize || 500) * 1024 * 1024
});
let filePaths: string[] = [];
try {
const { file, data, bucketName } = await upload.doUpload<FileCreateDatasetCollectionParams>(
req,
res,
BucketNameEnum.dataset
);
filePaths = [file.path];
if (!file || !bucketName) {
throw new Error('file is empty');
}
const { teamId, tmbId, dataset } = await authDataset({
req,
authApiKey: true,
per: 'w',
datasetId: data.datasetId
});
const {
trainingType = TrainingModeEnum.chunk,
chunkSize = 512,
chunkSplitter,
qaPrompt
} = data;
const { fileMetadata, collectionMetadata, ...collectionData } = data;
const collectionName = file.originalname;
const relatedImgId = getNanoid();
// 1. read file
const { rawText } = await readRawTextByLocalFile({
teamId,
path: file.path,
metadata: {
...fileMetadata,
relatedId: relatedImgId
}
});
// 2. upload file
const fileId = await uploadFile({
teamId,
tmbId,
bucketName,
path: file.path,
filename: file.originalname,
contentType: file.mimetype,
metadata: fileMetadata
});
// 3. delete tmp file
removeFilesByPaths(filePaths);
// 4. split raw text to chunks
const { chunks } = splitText2Chunks({
text: rawText,
chunkLen: chunkSize,
overlapRatio: trainingType === TrainingModeEnum.chunk ? 0.2 : 0,
customReg: chunkSplitter ? [chunkSplitter] : []
});
// 5. check dataset limit
await checkDatasetLimit({
teamId,
insertLen: predictDataLimitLength(trainingType, chunks)
});
// 6. create collection and training bill
const { collectionId, insertResults } = await mongoSessionRun(async (session) => {
const { _id: collectionId } = await createOneCollection({
...collectionData,
name: collectionName,
teamId,
tmbId,
type: DatasetCollectionTypeEnum.file,
fileId,
rawTextLength: rawText.length,
hashRawText: hashStr(rawText),
metadata: {
...collectionMetadata,
relatedImgId
},
session
});
const { billId } = await createTrainingUsage({
teamId,
tmbId,
appName: collectionName,
billSource: UsageSourceEnum.training,
vectorModel: getVectorModel(dataset.vectorModel)?.name,
agentModel: getDatasetModel(dataset.agentModel)?.name
});
// 7. push chunks to training queue
const insertResults = await pushDataListToTrainingQueue({
teamId,
tmbId,
datasetId: dataset._id,
collectionId,
agentModel: dataset.agentModel,
vectorModel: dataset.vectorModel,
trainingMode: trainingType,
prompt: qaPrompt,
billId,
data: chunks.map((text, index) => ({
q: text,
chunkIndex: index
}))
});
// 8. remove image expired time
await MongoImage.updateMany(
{
teamId,
'metadata.relatedId': relatedImgId
},
{
// Remove expiredTime to avoid ttl expiration
$unset: {
expiredTime: 1
}
},
{
session
}
);
return {
collectionId,
insertResults
};
});
jsonRes(res, {
data: { collectionId, results: insertResults }
});
} catch (error) {
removeFilesByPaths(filePaths);
return Promise.reject(error);
}
}
export const config = {
api: {
bodyParser: false
}
};
export default NextAPI(handler);
projects/app/src/pages/api/core/dataset/collection/detail.ts
...@@ -8,6 +8,7 @@ import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/ ...@@ -8,6 +8,7 @@ import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/
import { DatasetCollectionItemType } from '@fastgpt/global/core/dataset/type'; import { DatasetCollectionItemType } from '@fastgpt/global/core/dataset/type';
import { BucketNameEnum } from '@fastgpt/global/common/file/constants'; import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { getFileById } from '@fastgpt/service/common/file/gridfs/controller'; import { getFileById } from '@fastgpt/service/common/file/gridfs/controller';
import { getCollectionSourceData } from '@fastgpt/global/core/dataset/collection/utils';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) { export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try { try {
...@@ -36,8 +37,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse< ...@@ -36,8 +37,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
data: { data: {
...collection, ...collection,
canWrite, canWrite,
sourceName: collection?.name, ...getCollectionSourceData(collection),
sourceId: collection?.fileId || collection?.rawLink,
file file
} }
}); });
......
projects/app/src/pages/api/core/dataset/collection/read.ts 0 → 100644
import type { ApiRequestProps, ApiResponseType } from '@fastgpt/service/type/next';
import { NextAPI } from '@/service/middleware/entry';
import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/dataset';
import { DatasetCollectionTypeEnum } from '@fastgpt/global/core/dataset/constants';
import { createFileToken } from '@fastgpt/service/support/permission/controller';
import { BucketNameEnum, ReadFileBaseUrl } from '@fastgpt/global/common/file/constants';
export type readCollectionSourceQuery = {
collectionId: string;
};
export type readCollectionSourceBody = {};
export type readCollectionSourceResponse = {
type: 'url';
value: string;
};
async function handler(
req: ApiRequestProps<readCollectionSourceBody, readCollectionSourceQuery>,
res: ApiResponseType<any>
): Promise<readCollectionSourceResponse> {
const { collection, teamId, tmbId } = await authDatasetCollection({
req,
authToken: true,
authApiKey: true,
collectionId: req.query.collectionId,
per: 'r'
});
const sourceUrl = await (async () => {
if (collection.type === DatasetCollectionTypeEnum.file && collection.fileId) {
const token = await createFileToken({
bucketName: BucketNameEnum.dataset,
teamId,
tmbId,
fileId: collection.fileId
});
return `${ReadFileBaseUrl}?token=${token}`;
}
if (collection.type === DatasetCollectionTypeEnum.link && collection.rawLink) {
return collection.rawLink;
}
if (collection.type === DatasetCollectionTypeEnum.externalFile) {
if (collection.externalFileId && collection.datasetId.externalReadUrl) {
return collection.datasetId.externalReadUrl.replace(
'{{fileId}}',
collection.externalFileId
);
}
if (collection.externalFileUrl) {
return collection.externalFileUrl;
}
}
return '';
})();
return {
type: 'url',
value: sourceUrl
};
}
export default NextAPI(handler);
projects/app/src/pages/api/core/dataset/data/delete.ts
...@@ -2,7 +2,7 @@ import type { NextApiRequest, NextApiResponse } from 'next'; ...@@ -2,7 +2,7 @@ import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '@fastgpt/service/common/response';
import { authDatasetData } from '@/service/support/permission/auth/dataset'; import { authDatasetData } from '@/service/support/permission/auth/dataset';
import { deleteDatasetData } from '@/service/core/dataset/data/controller'; import { deleteDatasetData } from '@/service/core/dataset/data/controller';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
const { id: dataId } = req.query as { const { id: dataId } = req.query as {
......
projects/app/src/pages/api/core/dataset/data/detail.ts
...@@ -2,7 +2,7 @@ import type { NextApiRequest, NextApiResponse } from 'next'; ...@@ -2,7 +2,7 @@ import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo'; import { connectToDatabase } from '@/service/mongo';
import { authDatasetData } from '@/service/support/permission/auth/dataset'; import { authDatasetData } from '@/service/support/permission/auth/dataset';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
export type Response = { export type Response = {
id: string; id: string;
......
projects/app/src/pages/api/core/dataset/data/insertData.ts
...@@ -15,7 +15,7 @@ import { pushGenerateVectorUsage } from '@/service/support/wallet/usage/push'; ...@@ -15,7 +15,7 @@ import { pushGenerateVectorUsage } from '@/service/support/wallet/usage/push';
import { InsertOneDatasetDataProps } from '@/global/core/dataset/api'; import { InsertOneDatasetDataProps } from '@/global/core/dataset/api';
import { simpleText } from '@fastgpt/global/common/string/tools'; import { simpleText } from '@fastgpt/global/common/string/tools';
import { checkDatasetLimit } from '@fastgpt/service/support/permission/teamLimit'; import { checkDatasetLimit } from '@fastgpt/service/support/permission/teamLimit';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
const { collectionId, q, a, indexes } = req.body as InsertOneDatasetDataProps; const { collectionId, q, a, indexes } = req.body as InsertOneDatasetDataProps;
......
projects/app/src/pages/api/core/dataset/data/list.ts
...@@ -7,7 +7,7 @@ import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/ ...@@ -7,7 +7,7 @@ import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema'; import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
import { PagingData } from '@/types'; import { PagingData } from '@/types';
import { replaceRegChars } from '@fastgpt/global/common/string/tools'; import { replaceRegChars } from '@fastgpt/global/common/string/tools';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
let { let {
......
projects/app/src/pages/api/core/dataset/data/pushData.ts
/* push data to training queue */ /* push data to training queue */
import type { NextApiRequest, NextApiResponse } from 'next'; import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import type { import type {
PushDatasetDataProps, PushDatasetDataProps,
PushDatasetDataResponse PushDatasetDataResponse
...@@ -10,7 +9,7 @@ import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/ ...@@ -10,7 +9,7 @@ import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/
import { checkDatasetLimit } from '@fastgpt/service/support/permission/teamLimit'; import { checkDatasetLimit } from '@fastgpt/service/support/permission/teamLimit';
import { predictDataLimitLength } from '@fastgpt/global/core/dataset/utils'; import { predictDataLimitLength } from '@fastgpt/global/core/dataset/utils';
import { pushDataListToTrainingQueue } from '@fastgpt/service/core/dataset/training/controller'; import { pushDataListToTrainingQueue } from '@fastgpt/service/core/dataset/training/controller';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
const body = req.body as PushDatasetDataProps; const body = req.body as PushDatasetDataProps;
......
projects/app/src/pages/api/core/dataset/data/update.ts
...@@ -6,7 +6,7 @@ import { authDatasetData } from '@/service/support/permission/auth/dataset'; ...@@ -6,7 +6,7 @@ import { authDatasetData } from '@/service/support/permission/auth/dataset';
import { pushGenerateVectorUsage } from '@/service/support/wallet/usage/push'; import { pushGenerateVectorUsage } from '@/service/support/wallet/usage/push';
import { UpdateDatasetDataProps } from '@/global/core/dataset/api'; import { UpdateDatasetDataProps } from '@/global/core/dataset/api';
import { checkDatasetLimit } from '@fastgpt/service/support/permission/teamLimit'; import { checkDatasetLimit } from '@fastgpt/service/support/permission/teamLimit';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
const { id, q = '', a, indexes = [] } = req.body as UpdateDatasetDataProps; const { id, q = '', a, indexes = [] } = req.body as UpdateDatasetDataProps;
......
projects/app/src/pages/api/core/dataset/exportAll.ts
...@@ -8,7 +8,7 @@ import { ...@@ -8,7 +8,7 @@ import {
checkExportDatasetLimit, checkExportDatasetLimit,
updateExportDatasetLimit updateExportDatasetLimit
} from '@fastgpt/service/support/user/utils'; } from '@fastgpt/service/support/user/utils';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
let { datasetId } = req.query as { let { datasetId } = req.query as {
......
projects/app/src/pages/api/core/dataset/file/getPreviewChunks.ts
...@@ -3,7 +3,7 @@ import { authFile } from '@fastgpt/service/support/permission/auth/file'; ...@@ -3,7 +3,7 @@ import { authFile } from '@fastgpt/service/support/permission/auth/file';
import { DatasetSourceReadTypeEnum } from '@fastgpt/global/core/dataset/constants'; import { DatasetSourceReadTypeEnum } from '@fastgpt/global/core/dataset/constants';
import { rawText2Chunks, readDatasetSourceRawText } from '@fastgpt/service/core/dataset/read'; import { rawText2Chunks, readDatasetSourceRawText } from '@fastgpt/service/core/dataset/read';
import { authCert } from '@fastgpt/service/support/permission/auth/common'; import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
import { ApiRequestProps } from '@fastgpt/service/type/next'; import { ApiRequestProps } from '@fastgpt/service/type/next';
export type PostPreviewFilesChunksProps = { export type PostPreviewFilesChunksProps = {
......
projects/app/src/pages/api/core/dataset/file/getPreviewUrl.ts deleted 100644 → 0
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authDatasetFile } from '@fastgpt/service/support/permission/auth/dataset';
import { createFileToken } from '@fastgpt/service/support/permission/controller';
import { BucketNameEnum, ReadFileBaseUrl } from '@fastgpt/global/common/file/constants';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
const { fileId } = req.query as { fileId: string };
if (!fileId) {
throw new Error('fileId is empty');
}
const { teamId, tmbId } = await authDatasetFile({ req, authToken: true, fileId, per: 'r' });
const token = await createFileToken({
bucketName: BucketNameEnum.dataset,
teamId,
tmbId,
fileId
});
jsonRes(res, {
data: `${ReadFileBaseUrl}?token=${token}`
});
} catch (error) {
jsonRes(res, {
code: 500,
error
});
}
}
projects/app/src/pages/api/core/dataset/list.ts
...@@ -6,7 +6,7 @@ import { MongoDataset } from '@fastgpt/service/core/dataset/schema'; ...@@ -6,7 +6,7 @@ import { MongoDataset } from '@fastgpt/service/core/dataset/schema';
import { mongoRPermission } from '@fastgpt/global/support/permission/utils'; import { mongoRPermission } from '@fastgpt/global/support/permission/utils';
import { authUserRole } from '@fastgpt/service/support/permission/auth/user'; import { authUserRole } from '@fastgpt/service/support/permission/auth/user';
import { getVectorModel } from '@fastgpt/service/core/ai/model'; import { getVectorModel } from '@fastgpt/service/core/ai/model';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
const { parentId, type } = req.query as { parentId?: string; type?: DatasetTypeEnum }; const { parentId, type } = req.query as { parentId?: string; type?: DatasetTypeEnum };
......
projects/app/src/pages/api/core/dataset/searchTest.ts
...@@ -12,7 +12,7 @@ import { ...@@ -12,7 +12,7 @@ import {
checkTeamAIPoints, checkTeamAIPoints,
checkTeamReRankPermission checkTeamReRankPermission
} from '@fastgpt/service/support/permission/teamLimit'; } from '@fastgpt/service/support/permission/teamLimit';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
const { const {
......
projects/app/src/pages/api/core/dataset/training/getDatasetTrainingQueue.ts
import type { ApiRequestProps, ApiResponseType } from '@fastgpt/service/type/next'; import type { ApiRequestProps, ApiResponseType } from '@fastgpt/service/type/next';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset'; import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema'; import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
import { MongoDatasetTraining } from '@fastgpt/service/core/dataset/training/schema'; import { MongoDatasetTraining } from '@fastgpt/service/core/dataset/training/schema';
......
projects/app/src/pages/api/core/dataset/training/rebuildEmbedding.ts
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset'; import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun'; import { mongoSessionRun } from '@fastgpt/service/common/mongo/sessionRun';
import { MongoDataset } from '@fastgpt/service/core/dataset/schema'; import { MongoDataset } from '@fastgpt/service/core/dataset/schema';
......
projects/app/src/pages/api/core/workflow/debug.ts
...@@ -7,7 +7,7 @@ import { authCert } from '@fastgpt/service/support/permission/auth/common'; ...@@ -7,7 +7,7 @@ import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { getUserChatInfoAndAuthTeamPoints } from '@/service/support/permission/auth/team'; import { getUserChatInfoAndAuthTeamPoints } from '@/service/support/permission/auth/team';
import { PostWorkflowDebugProps, PostWorkflowDebugResponse } from '@/global/core/workflow/api'; import { PostWorkflowDebugProps, PostWorkflowDebugResponse } from '@/global/core/workflow/api';
import { authPluginCrud } from '@fastgpt/service/support/permission/auth/plugin'; import { authPluginCrud } from '@fastgpt/service/support/permission/auth/plugin';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler( async function handler(
req: NextApiRequest, req: NextApiRequest,
......
projects/app/src/pages/api/support/user/team/limit/exportDatasetLimit.ts
import type { NextApiRequest, NextApiResponse } from 'next'; import type { NextApiRequest, NextApiResponse } from 'next';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset'; import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { checkExportDatasetLimit } from '@fastgpt/service/support/user/utils'; import { checkExportDatasetLimit } from '@fastgpt/service/support/user/utils';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
async function handler(req: NextApiRequest, res: NextApiResponse<any>) { async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
const { datasetId } = req.query as { const { datasetId } = req.query as {
......
projects/app/src/pages/api/v1/audio/transcriptions.ts
...@@ -9,7 +9,7 @@ import { authChatCert } from '@/service/support/permission/auth/chat'; ...@@ -9,7 +9,7 @@ import { authChatCert } from '@/service/support/permission/auth/chat';
import { MongoApp } from '@fastgpt/service/core/app/schema'; import { MongoApp } from '@fastgpt/service/core/app/schema';
import { getGuideModule, splitGuideModule } from '@fastgpt/global/core/workflow/utils'; import { getGuideModule, splitGuideModule } from '@fastgpt/global/core/workflow/utils';
import { OutLinkChatAuthProps } from '@fastgpt/global/support/permission/chat'; import { OutLinkChatAuthProps } from '@fastgpt/global/support/permission/chat';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
const upload = getUploadModel({ const upload = getUploadModel({
maxSize: 2 maxSize: 2
......
projects/app/src/pages/api/v1/chat/completions.ts
...@@ -44,7 +44,7 @@ import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/workflow/runti ...@@ -44,7 +44,7 @@ import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/workflow/runti
import { dispatchWorkFlowV1 } from '@fastgpt/service/core/workflow/dispatchV1'; import { dispatchWorkFlowV1 } from '@fastgpt/service/core/workflow/dispatchV1';
import { setEntryEntries } from '@fastgpt/service/core/workflow/dispatchV1/utils'; import { setEntryEntries } from '@fastgpt/service/core/workflow/dispatchV1/utils';
import { NextAPI } from '@/service/middle/entry'; import { NextAPI } from '@/service/middleware/entry';
import { getAppLatestVersion } from '@fastgpt/service/core/app/controller'; import { getAppLatestVersion } from '@fastgpt/service/core/app/controller';
type FastGptWebChatProps = { type FastGptWebChatProps = {
......
projects/app/src/pages/dataset/detail/components/CollectionCard/Header.tsx
...@@ -84,7 +84,7 @@ const Header = ({}: {}) => { ...@@ -84,7 +84,7 @@ const Header = ({}: {}) => {
...props ...props
}: { }: {
name: string; name: string;
type: `${DatasetCollectionTypeEnum}`; type: DatasetCollectionTypeEnum;
callback?: (id: string) => void; callback?: (id: string) => void;
trainingType?: TrainingModeEnum; trainingType?: TrainingModeEnum;
rawLink?: string; rawLink?: string;
......
projects/app/src/pages/dataset/detail/components/DataCard.tsx
...@@ -38,16 +38,14 @@ import { TabEnum } from '..'; ...@@ -38,16 +38,14 @@ import { TabEnum } from '..';
import { useUserStore } from '@/web/support/user/useUserStore'; import { useUserStore } from '@/web/support/user/useUserStore';
import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant'; import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
import { useSystemStore } from '@/web/common/system/useSystemStore'; import { useSystemStore } from '@/web/common/system/useSystemStore';
import { import { DatasetCollectionTypeMap, TrainingTypeMap } from '@fastgpt/global/core/dataset/constants';
DatasetCollectionTypeMap,
TrainingModeEnum,
TrainingTypeMap
} from '@fastgpt/global/core/dataset/constants';
import { formatTime2YMDHM } from '@fastgpt/global/common/string/time'; import { formatTime2YMDHM } from '@fastgpt/global/common/string/time';
import { formatFileSize } from '@fastgpt/global/common/file/tools'; import { formatFileSize } from '@fastgpt/global/common/file/tools';
import { getFileAndOpen } from '@/web/core/dataset/utils'; import { getCollectionSourceAndOpen } from '@/web/core/dataset/hooks/readCollectionSource';
import MyTooltip from '@/components/MyTooltip'; import MyTooltip from '@/components/MyTooltip';
import { usePagination } from '@fastgpt/web/hooks/usePagination'; import { usePagination } from '@fastgpt/web/hooks/usePagination';
import { getCollectionSourceData } from '@fastgpt/global/core/dataset/collection/utils';
import { useI18n } from '@/web/context/I18n';
const DataCard = () => { const DataCard = () => {
const BoxRef = useRef<HTMLDivElement>(null); const BoxRef = useRef<HTMLDivElement>(null);
...@@ -62,6 +60,7 @@ const DataCard = () => { ...@@ -62,6 +60,7 @@ const DataCard = () => {
}; };
const { Loading, setIsLoading } = useLoading({ defaultLoading: true }); const { Loading, setIsLoading } = useLoading({ defaultLoading: true });
const { t } = useTranslation(); const { t } = useTranslation();
const { datasetT } = useI18n();
const [searchText, setSearchText] = useState(''); const [searchText, setSearchText] = useState('');
const { toast } = useToast(); const { toast } = useToast();
const { openConfirm, ConfirmModal } = useConfirm({ const { openConfirm, ConfirmModal } = useConfirm({
...@@ -69,6 +68,7 @@ const DataCard = () => { ...@@ -69,6 +68,7 @@ const DataCard = () => {
type: 'delete' type: 'delete'
}); });
const { isOpen, onOpen, onClose } = useDisclosure(); const { isOpen, onOpen, onClose } = useDisclosure();
const readSource = getCollectionSourceAndOpen(collectionId);
const { const {
data: datasetDataList, data: datasetDataList,
...@@ -169,7 +169,17 @@ const DataCard = () => { ...@@ -169,7 +169,17 @@ const DataCard = () => {
value: webSelector value: webSelector
} }
] ]
: []) : []),
{
...(collection.tags
? [
{
label: datasetT('Collection tags'),
value: collection.tags?.join(', ') || '-'
}
]
: [])
}
]; ];
}, [collection, t]); }, [collection, t]);
...@@ -196,13 +206,15 @@ const DataCard = () => { ...@@ -196,13 +206,15 @@ const DataCard = () => {
/> />
<Flex className="textEllipsis" flex={'1 0 0'} mr={[3, 5]} alignItems={'center'}> <Flex className="textEllipsis" flex={'1 0 0'} mr={[3, 5]} alignItems={'center'}>
<Box lineHeight={1.2}> <Box lineHeight={1.2}>
<RawSourceBox {collection?._id && (
sourceName={collection?.name} <RawSourceBox
sourceId={collection?.fileId || collection?.rawLink} collectionId={collection._id}
fontSize={['md', 'lg']} {...getCollectionSourceData(collection)}
color={'black'} fontSize={['md', 'lg']}
textDecoration={'none'} color={'black'}
/> textDecoration={'none'}
/>
)}
<Box fontSize={'sm'} color={'myGray.500'}> <Box fontSize={'sm'} color={'myGray.500'}>
{t('core.dataset.collection.id')}:{' '} {t('core.dataset.collection.id')}:{' '}
<Box as={'span'} userSelect={'all'}> <Box as={'span'} userSelect={'all'}>
...@@ -412,10 +424,7 @@ const DataCard = () => { ...@@ -412,10 +424,7 @@ const DataCard = () => {
</Flex> </Flex>
))} ))}
{collection?.sourceId && ( {collection?.sourceId && (
<Button <Button variant={'whitePrimary'} onClick={readSource}>
variant={'whitePrimary'}
onClick={() => collection.sourceId && getFileAndOpen(collection.sourceId)}
>
{t('core.dataset.collection.metadata.read source')} {t('core.dataset.collection.metadata.read source')}
</Button> </Button>
)} )}
......
projects/app/src/pages/dataset/detail/components/Import/commonProgress/Upload.tsx
...@@ -15,12 +15,12 @@ import { ImportDataSourceEnum } from '@fastgpt/global/core/dataset/constants'; ...@@ -15,12 +15,12 @@ import { ImportDataSourceEnum } from '@fastgpt/global/core/dataset/constants';
import { useTranslation } from 'next-i18next'; import { useTranslation } from 'next-i18next';
import MyIcon from '@fastgpt/web/components/common/Icon'; import MyIcon from '@fastgpt/web/components/common/Icon';
import { useRequest } from '@fastgpt/web/hooks/useRequest'; import { useRequest } from '@fastgpt/web/hooks/useRequest';
import { useDatasetStore } from '@/web/core/dataset/store/dataset';
import { useToast } from '@fastgpt/web/hooks/useToast'; import { useToast } from '@fastgpt/web/hooks/useToast';
import { useRouter } from 'next/router'; import { useRouter } from 'next/router';
import { TabEnum } from '../../../index'; import { TabEnum } from '../../../index';
import { import {
postCreateDatasetCsvTableCollection, postCreateDatasetCsvTableCollection,
postCreateDatasetExternalFileCollection,
postCreateDatasetFileCollection, postCreateDatasetFileCollection,
postCreateDatasetLinkCollection, postCreateDatasetLinkCollection,
postCreateDatasetTextCollection postCreateDatasetTextCollection
...@@ -95,6 +95,13 @@ const Upload = () => { ...@@ -95,6 +95,13 @@ const Upload = () => {
...commonParams, ...commonParams,
fileId: item.dbFileId fileId: item.dbFileId
}); });
} else if (importSource === ImportDataSourceEnum.externalFile && item.externalFileUrl) {
await postCreateDatasetExternalFileCollection({
...commonParams,
externalFileUrl: item.externalFileUrl,
externalFileId: item.externalFileId,
filename: item.sourceName
});
} }
setSources((state) => setSources((state) =>
......
projects/app/src/pages/dataset/detail/components/Import/components/PreviewChunks.tsx
...@@ -44,7 +44,8 @@ const PreviewChunks = ({ ...@@ -44,7 +44,8 @@ const PreviewChunks = ({
if (importSource === ImportDataSourceEnum.csvTable) { if (importSource === ImportDataSourceEnum.csvTable) {
return getPreviewChunks({ return getPreviewChunks({
type: importType2ReadType(importSource), type: importType2ReadType(importSource),
sourceId: previewSource.dbFileId || previewSource.link || previewSource.sourceUrl || '', sourceId:
previewSource.dbFileId || previewSource.link || previewSource.externalFileUrl || '',
chunkSize, chunkSize,
overlapRatio: chunkOverlapRatio, overlapRatio: chunkOverlapRatio,
customSplitChar: processParamsForm.getValues('customSplitChar'), customSplitChar: processParamsForm.getValues('customSplitChar'),
...@@ -55,7 +56,8 @@ const PreviewChunks = ({ ...@@ -55,7 +56,8 @@ const PreviewChunks = ({
return getPreviewChunks({ return getPreviewChunks({
type: importType2ReadType(importSource), type: importType2ReadType(importSource),
sourceId: previewSource.dbFileId || previewSource.link || previewSource.sourceUrl || '', sourceId:
previewSource.dbFileId || previewSource.link || previewSource.externalFileUrl || '',
chunkSize, chunkSize,
overlapRatio: chunkOverlapRatio, overlapRatio: chunkOverlapRatio,
customSplitChar: processParamsForm.getValues('customSplitChar'), customSplitChar: processParamsForm.getValues('customSplitChar'),
......
projects/app/src/pages/dataset/detail/components/Import/components/PreviewRawText.tsx
...@@ -22,7 +22,7 @@ const PreviewRawText = ({ ...@@ -22,7 +22,7 @@ const PreviewRawText = ({
const { importSource, processParamsForm } = useContextSelector(DatasetImportContext, (v) => v); const { importSource, processParamsForm } = useContextSelector(DatasetImportContext, (v) => v);
const { data, isLoading } = useQuery( const { data, isLoading } = useQuery(
['previewSource', previewSource.dbFileId, previewSource.link, previewSource.sourceUrl], ['previewSource', previewSource.dbFileId, previewSource.link, previewSource.externalFileUrl],
() => { () => {
if (importSource === ImportDataSourceEnum.fileCustom && previewSource.rawText) { if (importSource === ImportDataSourceEnum.fileCustom && previewSource.rawText) {
return { return {
...@@ -39,7 +39,8 @@ const PreviewRawText = ({ ...@@ -39,7 +39,8 @@ const PreviewRawText = ({
return getPreviewFileContent({ return getPreviewFileContent({
type: importType2ReadType(importSource), type: importType2ReadType(importSource),
sourceId: previewSource.dbFileId || previewSource.link || previewSource.sourceUrl || '', sourceId:
previewSource.dbFileId || previewSource.link || previewSource.externalFileUrl || '',
isQAImport: false, isQAImport: false,
selector: processParamsForm.getValues('webSelector') selector: processParamsForm.getValues('webSelector')
}); });
......
projects/app/src/pages/dataset/detail/components/Import/diffSource/ExternalFile.tsx
...@@ -50,16 +50,16 @@ const CustomLinkInput = () => { ...@@ -50,16 +50,16 @@ const CustomLinkInput = () => {
const { register, reset, handleSubmit, control } = useForm<{ const { register, reset, handleSubmit, control } = useForm<{
list: { list: {
sourceName: string; sourceName: string;
sourceUrl: string; externalFileUrl: string;
externalId: string; externalFileId: string;
}[]; }[];
}>({ }>({
defaultValues: { defaultValues: {
list: [ list: [
{ {
sourceName: '', sourceName: '',
sourceUrl: '', externalFileUrl: '',
externalId: '' externalFileId: ''
} }
] ]
} }
...@@ -80,8 +80,8 @@ const CustomLinkInput = () => { ...@@ -80,8 +80,8 @@ const CustomLinkInput = () => {
reset({ reset({
list: sources.map((item) => ({ list: sources.map((item) => ({
sourceName: item.sourceName, sourceName: item.sourceName,
sourceUrl: item.sourceUrl || '', externalFileUrl: item.externalFileUrl || '',
externalId: item.externalId || '' externalFileId: item.externalFileId || ''
})) }))
}); });
} }
...@@ -104,7 +104,7 @@ const CustomLinkInput = () => { ...@@ -104,7 +104,7 @@ const CustomLinkInput = () => {
<Tr key={item.id}> <Tr key={item.id}>
<Td> <Td>
<Input <Input
{...register(`list.${index}.sourceUrl`, { {...register(`list.${index}.externalFileUrl`, {
required: index !== list.length - 1, required: index !== list.length - 1,
onBlur(e) { onBlur(e) {
const val = (e.target.value || '') as string; const val = (e.target.value || '') as string;
...@@ -112,15 +112,15 @@ const CustomLinkInput = () => { ...@@ -112,15 +112,15 @@ const CustomLinkInput = () => {
const sourceName = val.split('/').pop() || ''; const sourceName = val.split('/').pop() || '';
update(index, { update(index, {
...list[index], ...list[index],
sourceUrl: val, externalFileUrl: val,
sourceName: decodeURIComponent(sourceName) sourceName: decodeURIComponent(sourceName)
}); });
} }
if (val && index === list.length - 1) { if (val && index === list.length - 1) {
append({ append({
sourceName: '', sourceName: '',
sourceUrl: '', externalFileUrl: '',
externalId: '' externalFileId: ''
}); });
} }
} }
...@@ -128,7 +128,7 @@ const CustomLinkInput = () => { ...@@ -128,7 +128,7 @@ const CustomLinkInput = () => {
/> />
</Td> </Td>
<Td> <Td>
<Input {...register(`list.${index}.externalId`)} /> <Input {...register(`list.${index}.externalFileId`)} />
</Td> </Td>
<Td> <Td>
<Input {...register(`list.${index}.sourceName`)} /> <Input {...register(`list.${index}.sourceName`)} />
...@@ -154,26 +154,26 @@ const CustomLinkInput = () => { ...@@ -154,26 +154,26 @@ const CustomLinkInput = () => {
onClick={() => { onClick={() => {
append({ append({
sourceName: '', sourceName: '',
sourceUrl: '', externalFileUrl: '',
externalId: '' externalFileId: ''
}); });
}} }}
> >
{commonT('Add new')} {commonT('Add new')}
</Button> </Button>
<Button <Button
isDisabled={list.filter((item) => !!item.sourceUrl).length === 0} isDisabled={list.filter((item) => !!item.externalFileUrl).length === 0}
onClick={handleSubmit((data) => { onClick={handleSubmit((data) => {
setSources( setSources(
data.list data.list
.filter((item) => !!item.sourceUrl) .filter((item) => !!item.externalFileUrl)
.map((item) => ({ .map((item) => ({
id: getNanoid(32), id: getNanoid(32),
createStatus: 'waiting', createStatus: 'waiting',
sourceName: item.sourceName || item.sourceUrl, sourceName: item.sourceName || item.externalFileUrl,
icon: getFileIcon(item.sourceUrl), icon: getFileIcon(item.externalFileUrl),
externalId: item.externalId, externalFileId: item.externalFileId,
sourceUrl: item.sourceUrl externalFileUrl: item.externalFileUrl
})) }))
); );
......
projects/app/src/pages/dataset/detail/components/Info.tsx
import React, { useState, useMemo } from 'react'; import React, { useState, useMemo } from 'react';
import { useRouter } from 'next/router'; import { useRouter } from 'next/router';
import { Box, Flex, Button, IconButton, Input, Textarea } from '@chakra-ui/react'; import { Box, Flex, Button, IconButton, Input, Textarea, HStack } from '@chakra-ui/react';
import { DeleteIcon } from '@chakra-ui/icons'; import { DeleteIcon } from '@chakra-ui/icons';
import { delDatasetById } from '@/web/core/dataset/api'; import { delDatasetById } from '@/web/core/dataset/api';
import { useSelectFile } from '@/web/common/file/hooks/useSelectFile'; import { useSelectFile } from '@/web/common/file/hooks/useSelectFile';
import { useDatasetStore } from '@/web/core/dataset/store/dataset';
import { useConfirm } from '@fastgpt/web/hooks/useConfirm'; import { useConfirm } from '@fastgpt/web/hooks/useConfirm';
import { useForm } from 'react-hook-form'; import { useForm } from 'react-hook-form';
import { compressImgFileAndUpload } from '@/web/common/file/controller'; import { compressImgFileAndUpload } from '@/web/common/file/controller';
...@@ -24,6 +23,7 @@ import { useContextSelector } from 'use-context-selector'; ...@@ -24,6 +23,7 @@ import { useContextSelector } from 'use-context-selector';
import { DatasetPageContext } from '@/web/core/dataset/context/datasetPageContext'; import { DatasetPageContext } from '@/web/core/dataset/context/datasetPageContext';
import MyDivider from '@fastgpt/web/components/common/MyDivider/index'; import MyDivider from '@fastgpt/web/components/common/MyDivider/index';
import { DatasetTypeEnum } from '@fastgpt/global/core/dataset/constants'; import { DatasetTypeEnum } from '@fastgpt/global/core/dataset/constants';
import QuestionTip from '@fastgpt/web/components/common/MyTooltip/QuestionTip';
const Info = ({ datasetId }: { datasetId: string }) => { const Info = ({ datasetId }: { datasetId: string }) => {
const { t } = useTranslation(); const { t } = useTranslation();
...@@ -191,9 +191,10 @@ const Info = ({ datasetId }: { datasetId: string }) => { ...@@ -191,9 +191,10 @@ const Info = ({ datasetId }: { datasetId: string }) => {
{datasetDetail.type === DatasetTypeEnum.externalFile && ( {datasetDetail.type === DatasetTypeEnum.externalFile && (
<> <>
<Flex w={'100%'} alignItems={'center'}> <Flex w={'100%'} alignItems={'center'}>
<Box fontSize={['sm', 'md']} flex={['0 0 90px', '0 0 160px']} w={0}> <HStack fontSize={['sm', 'md']} flex={['0 0 90px', '0 0 160px']} w={0}>
{datasetT('External read url')} <Box>{datasetT('External read url')}</Box>
</Box> <QuestionTip label={datasetT('External read url tip')} />
</HStack>
<Input <Input
flex={[1, '0 0 320px']} flex={[1, '0 0 320px']}
placeholder="https://test.com/read?fileId={{fileId}}" placeholder="https://test.com/read?fileId={{fileId}}"
......
projects/app/src/pages/dataset/detail/components/InputDataModal.tsx
...@@ -237,6 +237,7 @@ const InputDataModal = ({ ...@@ -237,6 +237,7 @@ const InputDataModal = ({
w={'210px'} w={'210px'}
className="textEllipsis3" className="textEllipsis3"
whiteSpace={'pre-wrap'} whiteSpace={'pre-wrap'}
collectionId={collection._id}
sourceName={collection.sourceName} sourceName={collection.sourceName}
sourceId={collection.sourceId} sourceId={collection.sourceId}
mb={6} mb={6}
......
projects/app/src/pages/dataset/list/component/CreateModal.tsx
...@@ -116,13 +116,13 @@ const CreateModal = ({ onClose, parentId }: { onClose: () => void; parentId?: st ...@@ -116,13 +116,13 @@ const CreateModal = ({ onClose, parentId }: { onClose: () => void; parentId?: st
value: DatasetTypeEnum.websiteDataset, value: DatasetTypeEnum.websiteDataset,
icon: 'core/dataset/websiteDataset', icon: 'core/dataset/websiteDataset',
desc: datasetT('Website Dataset Desc') desc: datasetT('Website Dataset Desc')
},
{
title: datasetT('External File'),
value: DatasetTypeEnum.externalFile,
icon: 'core/dataset/externalDataset',
desc: datasetT('External file Dataset Desc')
} }
// {
// title: datasetT('External File'),
// value: DatasetTypeEnum.externalFile,
// icon: 'core/dataset/websiteDataset',
// desc: datasetT('External file Dataset Desc')
// }
] ]
: []) : [])
]} ]}
......
projects/app/src/service/middleware/entry.ts 0 → 100644
import { connectToDatabase } from '../mongo';
import { NextEntry } from '@fastgpt/service/common/middle/entry';
export const NextAPI = NextEntry({
beforeCallback: [connectToDatabase()]
});
projects/app/src/web/core/dataset/api.ts
...@@ -14,6 +14,7 @@ import type { ...@@ -14,6 +14,7 @@ import type {
CreateDatasetCollectionParams, CreateDatasetCollectionParams,
CsvTableCreateDatasetCollectionParams, CsvTableCreateDatasetCollectionParams,
DatasetUpdateBody, DatasetUpdateBody,
ExternalFileCreateDatasetCollectionParams,
FileIdCreateDatasetCollectionParams, FileIdCreateDatasetCollectionParams,
LinkCreateDatasetCollectionParams, LinkCreateDatasetCollectionParams,
PostWebsiteSyncParams, PostWebsiteSyncParams,
...@@ -44,6 +45,7 @@ import type { ...@@ -44,6 +45,7 @@ import type {
PostPreviewFilesChunksProps, PostPreviewFilesChunksProps,
PreviewChunksResponse PreviewChunksResponse
} from '@/pages/api/core/dataset/file/getPreviewChunks'; } from '@/pages/api/core/dataset/file/getPreviewChunks';
import type { readCollectionSourceResponse } from '@/pages/api/core/dataset/collection/read';
/* ======================== dataset ======================= */ /* ======================== dataset ======================= */
export const getDatasets = (data: { parentId?: string; type?: DatasetTypeEnum }) => export const getDatasets = (data: { parentId?: string; type?: DatasetTypeEnum }) =>
...@@ -85,7 +87,9 @@ export const getDatasetCollectionById = (id: string) => ...@@ -85,7 +87,9 @@ export const getDatasetCollectionById = (id: string) =>
export const postDatasetCollection = (data: CreateDatasetCollectionParams) => export const postDatasetCollection = (data: CreateDatasetCollectionParams) =>
POST<string>(`/core/dataset/collection/create`, data); POST<string>(`/core/dataset/collection/create`, data);
export const postCreateDatasetFileCollection = (data: FileIdCreateDatasetCollectionParams) => export const postCreateDatasetFileCollection = (data: FileIdCreateDatasetCollectionParams) =>
POST<{ collectionId: string }>(`/core/dataset/collection/create/file`, data, { timeout: 120000 }); POST<{ collectionId: string }>(`/core/dataset/collection/create/fileId`, data, {
timeout: 120000
});
export const postCreateDatasetLinkCollection = (data: LinkCreateDatasetCollectionParams) => export const postCreateDatasetLinkCollection = (data: LinkCreateDatasetCollectionParams) =>
POST<{ collectionId: string }>(`/core/dataset/collection/create/link`, data); POST<{ collectionId: string }>(`/core/dataset/collection/create/link`, data);
export const postCreateDatasetTextCollection = (data: TextCreateDatasetCollectionParams) => export const postCreateDatasetTextCollection = (data: TextCreateDatasetCollectionParams) =>
...@@ -94,6 +98,12 @@ export const postCreateDatasetCsvTableCollection = (data: CsvTableCreateDatasetC ...@@ -94,6 +98,12 @@ export const postCreateDatasetCsvTableCollection = (data: CsvTableCreateDatasetC
POST<{ collectionId: string }>(`/core/dataset/collection/create/csvTable`, data, { POST<{ collectionId: string }>(`/core/dataset/collection/create/csvTable`, data, {
timeout: 120000 timeout: 120000
}); });
export const postCreateDatasetExternalFileCollection = (
data: ExternalFileCreateDatasetCollectionParams
) =>
POST<{ collectionId: string }>(`/proApi/core/dataset/collection/create/externalFileUrl`, data, {
timeout: 120000
});
export const putDatasetCollectionById = (data: UpdateDatasetCollectionParams) => export const putDatasetCollectionById = (data: UpdateDatasetCollectionParams) =>
POST(`/core/dataset/collection/update`, data); POST(`/core/dataset/collection/update`, data);
...@@ -144,6 +154,6 @@ export const getDatasetTrainingQueue = (datasetId: string) => ...@@ -144,6 +154,6 @@ export const getDatasetTrainingQueue = (datasetId: string) =>
export const getPreviewChunks = (data: PostPreviewFilesChunksProps) => export const getPreviewChunks = (data: PostPreviewFilesChunksProps) =>
POST<PreviewChunksResponse>('/core/dataset/file/getPreviewChunks', data); POST<PreviewChunksResponse>('/core/dataset/file/getPreviewChunks', data);
/* ================== file ======================== */ /* ================== read source ======================== */
export const getFileViewUrl = (fileId: string) => export const getCollectionSource = (collectionId: string) =>
GET<string>('/core/dataset/file/getPreviewUrl', { fileId }); GET<readCollectionSourceResponse>('/core/dataset/collection/read', { collectionId });
projects/app/src/web/core/dataset/constants.ts
import { defaultQAModels, defaultVectorModels } from '@fastgpt/global/core/ai/model'; import { defaultQAModels, defaultVectorModels } from '@fastgpt/global/core/ai/model';
import { DatasetTypeEnum, TrainingModeEnum } from '@fastgpt/global/core/dataset/constants'; import {
DatasetCollectionTypeEnum,
DatasetTypeEnum,
TrainingModeEnum
} from '@fastgpt/global/core/dataset/constants';
import type { import type {
DatasetCollectionItemType, DatasetCollectionItemType,
DatasetItemType DatasetItemType
...@@ -46,7 +50,7 @@ export const defaultCollectionDetail: DatasetCollectionItemType = { ...@@ -46,7 +50,7 @@ export const defaultCollectionDetail: DatasetCollectionItemType = {
}, },
parentId: '', parentId: '',
name: '', name: '',
type: 'file', type: DatasetCollectionTypeEnum.file,
updateTime: new Date(), updateTime: new Date(),
canWrite: false, canWrite: false,
sourceName: '', sourceName: '',
......
projects/app/src/web/core/dataset/hooks/readCollectionSource.ts 0 → 100644
import { useSystemStore } from '@/web/common/system/useSystemStore';
import { getCollectionSource } from '@/web/core/dataset/api';
import { getErrText } from '@fastgpt/global/common/error/utils';
import { useToast } from '@fastgpt/web/hooks/useToast';
import { useTranslation } from 'next-i18next';
export function getCollectionSourceAndOpen(collectionId: string) {
const { toast } = useToast();
const { t } = useTranslation();
const { setLoading } = useSystemStore();
return async () => {
try {
setLoading(true);
const { value: url } = await getCollectionSource(collectionId);
if (!url) {
throw new Error('No file found');
}
if (url.startsWith('/')) {
window.open(`${location.origin}${url}`, '_blank');
} else {
window.open(url, '_blank');
}
} catch (error) {
toast({
title: getErrText(error, t('error.fileNotFound')),
status: 'error'
});
}
setLoading(false);
};
}
projects/app/src/web/core/dataset/type.d.ts
...@@ -28,8 +28,8 @@ export type ImportSourceItemType = { ...@@ -28,8 +28,8 @@ export type ImportSourceItemType = {
rawText?: string; rawText?: string;
// external file // external file
sourceUrl?: string; externalFileUrl?: string;
externalId?: string; externalFileId?: string;
}; };
export type ImportSourceParamsType = UseFormReturn< export type ImportSourceParamsType = UseFormReturn<
......
projects/app/src/web/core/dataset/utils.ts deleted 100644 → 0
import { getFileViewUrl } from '@/web/core/dataset/api';
import { strIsLink } from '@fastgpt/global/common/string/tools';
export async function getFileAndOpen(fileId: string) {
if (strIsLink(fileId)) {
return window.open(fileId, '_blank');
}
const url = await getFileViewUrl(fileId);
const asPath = `${location.origin}${url}`;
window.open(asPath, '_blank');
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment