Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit f88b51b5 by Archer Committed by GitHub
Options

feat(dataset): support CSV and Excel template imports (#7449) 

* docs: update v4.16 release notes

* fix(dataset): support legacy CSV imports

* chore(i18n): refine dataset import prompts

* chore(i18n): show current dataset CSV format

* feat(dataset): support Excel template imports

* fix(dataset): reuse readFile worker for imports

* docs(dataset): document template imports
parent ded69214
Showing with 764 additions and 308 deletions
document/content/guide/dataset/meta.en.json
{
"title": "Knowledge Base",
"root": false,
"pages": [
"rag",
"dataset_engine",
"collection_tags",
"websync",
"third-party",
"faq"
]
"pages": ["rag", "dataset_engine", "collection_tags", "websync", "third-party", "template", "faq"]
}
document/content/guide/dataset/meta.json
{
"title": "知识库",
"root": false,
"pages": [
"rag",
"dataset_engine",
"collection_tags",
"websync",
"third-party",
"faq"
]
"pages": ["rag", "dataset_engine", "collection_tags", "websync", "third-party", "template", "faq"]
}
document/content/guide/dataset/template.en.mdx 0 → 100644
---
title: Template Import
description: Batch-import knowledge base data from a CSV or Excel template
---
Template import lets you add prepared content or question-answer pairs to a knowledge base in batches. FastGPT accepts `.csv` and `.xlsx` files and creates knowledge base entries from the questions, answers, indexes, and metadata in the template.
## File Structure
The first row must contain the headers. The following columns are supported:
| Header | Required | Count | Description |
| ---------- | -------- | ---------- | --------------------------------------------------------------- |
| `q` | Yes | Exactly 1 | Content or a question |
| `a` | Yes | Exactly 1 | The answer; it can be empty when importing standalone content |
| `index` | No | Repeatable | A custom index. A row can contain multiple indexes |
| `metadata` | No | At most 1 | A JSON object for custom information such as source or category |
Each row represents one knowledge base entry. `q` and `a` should not both be empty. Headers can appear in any order, but do not add unsupported headers.
### CSV Example
```csv
q,a,index,index,metadata
"What is FastGPT?","FastGPT is an AI agent development platform.","FastGPT overview","AI agent platform","{""source"":""product-doc"",""category"":""overview""}"
"How do I import knowledge base data?","Use a CSV or Excel template.","knowledge base import","template import","{""source"":""help-center""}"
```
Use UTF-8 encoding for CSV files. Cells that contain commas, line breaks, or double quotes must be escaped according to CSV rules.
### Excel Example
Excel files use the same headers and data structure as CSV files:
| q | a | index | index | metadata |
| ------------------------------------ | -------------------------------------------- | --------------------- | ----------------- | ------------------------------------------------ |
| What is FastGPT? | FastGPT is an AI agent development platform. | FastGPT overview | AI agent platform | `{"source":"product-doc","category":"overview"}` |
| How do I import knowledge base data? | Use a CSV or Excel template. | knowledge base import | template import | `{"source":"help-center"}` |
Excel files must meet these requirements:
- Use the `.xlsx` extension. `.xls` files are not supported.
- Include exactly one worksheet.
- Do not contain merged cells.
- Use the first row for the template headers.
## Import a Template
1. Open the target knowledge base and select **Template Import** from the import menu.
![Template Import dialog](/imgs/template-import-dialog.png)
2. Select **Download CSV Template** for an example, or prepare an `.xlsx` file with the same structure.
3. Add your data and verify the headers, cell contents, and file format.
4. Select the file and confirm the import. You can import one file at a time.
5. After the import finishes, review the data and training status in the knowledge base collection.
![Knowledge base collection after a template import](/imgs/template-import-result.png)
## Metadata
Use `metadata` to attach structured information to each entry. The cell must contain a valid JSON object, for example:
```json
{ "source": "product-doc", "category": "overview", "version": 2 }
```
Do not use an array, plain text, or invalid JSON. In CSV files, escape the JSON according to CSV rules. In Excel files, enter the JSON string directly in the cell.
## Invalid File Format
If FastGPT reports an invalid file format, check the following:
- The file uses the `.csv` or `.xlsx` extension.
- The header row contains only supported columns.
- There is exactly one `q` column and one `a` column, with no more than one `metadata` column.
- Quotes, commas, and line breaks are correctly escaped in CSV files.
- The Excel file contains exactly one worksheet and no merged cells.
Start with a small test file. After confirming the format, import larger datasets in batches.
document/content/guide/dataset/template.mdx 0 → 100644
---
title: 模板导入
description: 使用 CSV 或 Excel 模板批量导入知识库数据
---
模板导入适合将已经整理好的内容或问答对批量写入知识库。FastGPT 支持导入 `.csv` 和 `.xlsx` 文件，并根据模板中的问题、答案、索引和元数据创建知识库数据。
## 文件结构
文件的第一行必须是表头。支持以下列：
| 表头 | 是否必需 | 数量 | 说明 |
| ---------- | -------- | --------- | ----------------------------------------- |
| `q` | 是 | 1 列 | 内容或问题 |
| `a` | 是 | 1 列 | 答案；导入普通内容时可以留空 |
| `index` | 否 | 可重复 | 自定义索引，同一行可以填写多个索引 |
| `metadata` | 否 | 最多 1 列 | JSON 对象，用于保存来源、分类等自定义信息 |
每一行代表一条知识库数据，`q` 和 `a` 不应同时为空。表头顺序不受限制，但不要使用模板之外的表头。
### CSV 示例
```csv
q,a,index,index,metadata
"FastGPT 是什么？","FastGPT 是一个 AI Agent 构建平台。","FastGPT 简介","AI Agent 平台","{""source"":""product-doc"",""category"":""overview""}"
"如何导入知识库数据？","可以使用 CSV 或 Excel 模板导入。","知识库导入","模板导入","{""source"":""help-center""}"
```
CSV 文件建议使用 UTF-8 编码。如果单元格中包含逗号、换行或双引号，需要按照 CSV 规则正确转义。
### Excel 示例
Excel 文件使用与 CSV 相同的表头和数据结构：
| q | a | index | index | metadata |
| -------------------- | ---------------------------------- | ------------ | ------------- | ------------------------------------------------ |
| FastGPT 是什么？ | FastGPT 是一个 AI Agent 构建平台。 | FastGPT 简介 | AI Agent 平台 | `{"source":"product-doc","category":"overview"}` |
| 如何导入知识库数据？ | 可以使用 CSV 或 Excel 模板导入。 | 知识库导入 | 模板导入 | `{"source":"help-center"}` |
Excel 文件需要满足以下要求：
- 文件扩展名为 `.xlsx`，不支持 `.xls`
- 只能包含一个工作表
- 不能包含合并单元格
- 第一行必须是模板表头
## 导入步骤
1. 打开目标知识库，在导入菜单中选择「模板导入」。
![模板导入弹窗](/imgs/template-import-dialog.png)
2. 点击「下载 CSV 模板」获取示例文件，或者按照相同结构准备 `.xlsx` 文件。
3. 填写数据并检查表头、单元格内容和文件格式。
4. 选择文件并确认导入。每次只能导入一个文件。
5. 导入完成后，在知识库集合中检查数据及训练状态。
![模板导入完成后的知识库集合](/imgs/template-import-result.png)
## 元数据
`metadata` 用于为每条数据附加结构化信息。单元格内容应为有效的 JSON 对象，例如：
```json
{ "source": "product-doc", "category": "overview", "version": 2 }
```
不要填写数组、纯文本或包含语法错误的 JSON。CSV 中的 JSON 需要按照 CSV 规则转义；Excel 单元格中可以直接填写 JSON 字符串。
## 文件格式异常
出现「文件格式异常」提示时，请依次检查：
- 文件是否为 `.csv` 或 `.xlsx`
- 表头是否包含且仅包含支持的列
- `q`、`a` 是否各有一列，`metadata` 是否不超过一列
- CSV 的引号、逗号和换行是否正确转义
- Excel 是否只有一个工作表且没有合并单元格
建议先使用少量数据测试，确认格式正确后再分批导入大量数据。
document/content/self-host/upgrading/4-16/41601.en.mdx
......@@ -115,8 +115,9 @@ The script first filters Apps by HTTP tool type, then migrates historical versio
3. App Workflow automatically archives and restores Workspaces when the Sandbox Provider or runtime image changes. The upgrade completes silently during the current run.
4. Workflow tool nodes can delegate selected input parameters to the Agent for generation while preserving fixed values, references, and user inputs.
5. ChatAgent tool selection supports explicitly choosing whether parameters should be generated by AI.
6. Knowledge Base data supports custom `metadata`, which can be imported as JSON through the API or CSV template. Search results and backup exports preserve this field. CSV templates only allow `q`, `a`, `index`, and `metadata` headers; `q`, `a`, and `metadata` each use one column, while `index` may use multiple columns in any order.
6. Knowledge Base data supports custom `metadata`, which can be imported as JSON through the API, CSV templates, or Excel templates. Search results and backup exports preserve this field. Template and backup imports accept both `.csv` and `.xlsx` files with `q`, `a`, `index`, and `metadata` headers. `q`, `a`, and `metadata` each use one column, while `index` may use multiple columns in any order. Excel files must contain a single worksheet with no merged cells. FastGPT reports an invalid file format when it cannot parse a CSV or Excel file correctly.
7. Large-file chunked uploads.
8. System tool keys configured by administrators are now encrypted, with backward compatibility for existing keys.
## ⚙️ Improvements
......@@ -126,6 +127,7 @@ The script first filters Apps by HTTP tool type, then migrates historical versio
4. App and Skill now share runtime image upgrade status, and the Skill editor can continuously poll for upgrade results.
5. Sandbox file writes now create parent directories automatically, preventing failures when writing to nested paths.
6. Improved compatibility handling for legacy Workflow data and tool parameters.
7. Updated the Agent Ask UI.
## 🐛 Fixes
......@@ -139,6 +141,8 @@ The script first filters Apps by HTTP tool type, then migrates historical versio
8. Fixed uninstalled tools still appearing in the system tool list.
9. Fixed the default Agent/Agent V2 version selection so it chooses the latest version by default.
10. Fixed images embedded in S3-hosted files with spaces failing to parse because of malformed keys and returning 404 errors.
11. Fixed duplicate headers in MCP SSE mode.
12. Fixed unencrypted Agent V2 system tool keys.
## 🛠️ Code Improvements
......@@ -147,3 +151,4 @@ The script first filters Apps by HTTP tool type, then migrates historical versio
3. Optimized Workflow schemas and unified tool calls with form rendering.
4. Extended tool JSON Schema support for additional data types.
5. Unified service file-read timeouts.
6. Hardened system tool permissions in multi-process deployments.
document/content/self-host/upgrading/4-16/41601.mdx
......@@ -115,8 +115,9 @@ curl -X POST 'https://你的域名/api/admin/4160/initHttpToolSchema' \
3. App Workflow 在 Sandbox Provider 或运行时镜像变化时自动归档并恢复 Workspace，升级过程在当前运行中静默完成。
4. 工作流工具节点支持将指定输入参数交由 Agent 自动生成，并保留固定值、引用和用户输入等既有配置。
5. ChatAgent 选择工具时，支持手动指定是否为 AI 生成参数。
6. 知识库数据支持自定义 `metadata`，可通过 API 或 CSV 模板导入 JSON 元数据；检索结果和备份导出会保留该字段。CSV 模板仅允许 `q`、`a`、`index`、`metadata` 表头，`q`、`a`、`metadata` 各一列，`index` 可多列且顺序任意。
6. 知识库数据支持自定义 `metadata`，可通过 API、CSV 或 Excel 模板导入 JSON 元数据；检索结果和备份导出会保留该字段。模板导入和备份导入均支持 `.csv` 和 `.xlsx` 文件，使用 `q`、`a`、`index`、`metadata` 表头；`q`、`a`、`metadata` 各一列，`index` 可多列且顺序任意。Excel 文件仅支持单个工作表且不能包含合并单元格，无法正确解析的 CSV 或 Excel 文件会提示文件格式异常。
7. 大文件分块上传。
8. 管理员配置系统工具密钥时，加密（兼容已配置的密钥）。
## ⚙️ 优化
......@@ -126,6 +127,7 @@ curl -X POST 'https://你的域名/api/admin/4160/initHttpToolSchema' \
4. App 与 Skill 统一运行时镜像升级状态，Skill 编辑页可持续轮询升级结果。
5. Sandbox 文件写入前自动创建父目录，避免写入嵌套路径失败。
6. 优化工作流旧数据、工具参数等兼容问题。
7. Agent Ask UI。
## 🐛 修复
......@@ -139,6 +141,8 @@ curl -X POST 'https://你的域名/api/admin/4160/initHttpToolSchema' \
8. 修复系统工具列表中已卸载工具的展示问题。
9. 修复 Agent/Agent V2 默认版本选择逻辑，使其默认选择最新版本。
10. S3 文件如果有空格时，解析其文件内的图片，会因 key 异常 404。
11. MCP SSE 模式，header 重复。
12. Agent V2 系统工具密钥未加密。
## 🛠️ 代码优化
......@@ -147,3 +151,4 @@ curl -X POST 'https://你的域名/api/admin/4160/initHttpToolSchema' \
3. 工作流 schema 优化，统一工具调用和表单渲染。
4. 扩展工具 JSON Schema，支持更多数据类型。
5. 统一服务文件读取超时时间。
6. 优化系统工具多进程权限安全问题。
document/content/toc.en.mdx
......@@ -58,6 +58,7 @@ description: FastGPT Toc
- [/en/guide/dataset/dataset_engine](/en/guide/dataset/dataset_engine)
- [/en/guide/dataset/faq](/en/guide/dataset/faq)
- [/en/guide/dataset/rag](/en/guide/dataset/rag)
- [/en/guide/dataset/template](/en/guide/dataset/template)
- [/en/guide/dataset/third-party/api_dataset](/en/guide/dataset/third-party/api_dataset)
- [/en/guide/dataset/third-party/dingtalk_dataset](/en/guide/dataset/third-party/dingtalk_dataset)
- [/en/guide/dataset/third-party/lark_dataset](/en/guide/dataset/third-party/lark_dataset)
......
document/content/toc.mdx
......@@ -58,6 +58,7 @@ description: FastGPT 文档目录
- [/guide/dataset/dataset_engine](/guide/dataset/dataset_engine)
- [/guide/dataset/faq](/guide/dataset/faq)
- [/guide/dataset/rag](/guide/dataset/rag)
- [/guide/dataset/template](/guide/dataset/template)
- [/guide/dataset/third-party/api_dataset](/guide/dataset/third-party/api_dataset)
- [/guide/dataset/third-party/dingtalk_dataset](/guide/dataset/third-party/dingtalk_dataset)
- [/guide/dataset/third-party/lark_dataset](/guide/dataset/third-party/lark_dataset)
......
document/data/doc-last-modified.json
......@@ -111,6 +111,8 @@
"content/guide/dataset/faq.mdx": "2026-06-04T16:10:15+08:00",
"content/guide/dataset/rag.en.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/dataset/rag.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/dataset/template.en.mdx": "2026-06-04T16:10:15+08:00",
"content/guide/dataset/template.mdx": "2026-06-04T16:10:15+08:00",
"content/guide/dataset/third-party/api_dataset.en.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/dataset/third-party/api_dataset.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/dataset/third-party/dingtalk_dataset.en.mdx": "2026-05-07T15:06:40+08:00",
......@@ -331,8 +333,8 @@
"content/self-host/upgrading/4-15/4155.mdx": "2026-07-30T11:22:58+08:00",
"content/self-host/upgrading/4-15/4156.en.mdx": "2026-07-31T17:46:31+08:00",
"content/self-host/upgrading/4-15/4156.mdx": "2026-07-31T17:46:31+08:00",
"content/self-host/upgrading/4-16/41601.en.mdx": "2026-08-03T17:31:52+08:00",
"content/self-host/upgrading/4-16/41601.mdx": "2026-08-03T17:31:52+08:00",
"content/self-host/upgrading/4-16/41601.en.mdx": "2026-08-04T20:18:35+08:00",
"content/self-host/upgrading/4-16/41601.mdx": "2026-08-04T20:18:35+08:00",
"content/self-host/upgrading/outdated/40.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/40.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/41.en.mdx": "2026-07-25T00:27:20+08:00",
......
packages/global/openapi/core/dataset/collection/createApi.ts
......@@ -172,7 +172,7 @@ export const CreateImageCollectionMultipartSchema = z.object({
});
/* ============================================================================
* API: 导入备份 CSV 文件创建集合
* API: 导入备份 CSV 或 Excel 文件创建集合
* Route: POST /core/dataset/collection/create/backup
* Content-Type: multipart/form-data
* ============================================================================ */
......@@ -188,13 +188,13 @@ export const CreateBackupCollectionMultipartSchema = z.object({
file: z.any().meta({
format: 'binary',
description:
'备份 CSV 文件（表头由 q、a、index、metadata 组成，q/a/metadata 各一列，index 可多列且顺序任意，metadata 单元格为 JSON object）'
'备份 CSV 或 Excel 文件（表头由 q、a、index、metadata 组成，q/a/metadata 各一列，index 可多列且顺序任意，metadata 单元格为 JSON object；Excel 仅支持单工作表且不能包含合并单元格；兼容旧版 q、a、indexes 表头）'
}),
data: CreateBackupCollectionFormSchema.meta({ description: '集合参数（JSON 序列化后传入）' })
});
/* ============================================================================
* API: 导入模板 CSV 文件创建集合
* API: 导入模板 CSV 或 Excel 文件创建集合
* Route: POST /core/dataset/collection/create/template
* Content-Type: multipart/form-data
* ============================================================================ */
......@@ -210,7 +210,7 @@ export const CreateTemplateCollectionMultipartSchema = z.object({
file: z.any().meta({
format: 'binary',
description:
'模板 CSV 文件（表头由 q、a、index、metadata 组成，q/a/metadata 各一列，index 可多列且顺序任意，metadata 单元格为 JSON object）'
'模板 CSV 或 Excel 文件（表头由 q、a、index、metadata 组成，q/a/metadata 各一列，index 可多列且顺序任意，metadata 单元格为 JSON object；Excel 仅支持单工作表且不能包含合并单元格；兼容旧版 q、a、indexes 表头）'
}),
data: CreateTemplateCollectionFormSchema.meta({ description: '集合参数（JSON 序列化后传入）' })
});
......
packages/global/openapi/core/dataset/collection/createPath.ts
......@@ -233,13 +233,13 @@ export const DatasetCollectionCreatePath: OpenAPIPath = {
},
/* ============================================================
* 导入备份 CSV 文件创建集合（multipart/form-data）
* 导入备份 CSV 或 Excel 文件创建集合（multipart/form-data）
* ============================================================ */
'/core/dataset/collection/create/backup': {
post: {
summary: '导入备份 CSV 创建集合',
summary: '导入备份文件创建集合',
description:
'上传表头由 q、a、index、metadata 组成的 CSV 备份文件，q/a/metadata 各一列，index 可多列且顺序任意，恢复数据到知识库集合。metadata 单元格为 JSON object。`file` 为 CSV 文件，`data` 为 JSON 序列化的集合参数对象',
'上传 CSV 或 Excel 备份文件，恢复数据到知识库集合。表头由 q、a、index、metadata 组成，q/a/metadata 各一列，index 可多列且顺序任意，metadata 单元格为 JSON object；Excel 仅支持单工作表且不能包含合并单元格；同时兼容旧版 q、a、indexes 表头。`file` 为上传文件，`data` 为 JSON 序列化的集合参数对象',
tags: [DevApiTagsMap.datasetCollectionCrteate],
requestBody: {
content: {
......@@ -260,13 +260,13 @@ export const DatasetCollectionCreatePath: OpenAPIPath = {
},
/* ============================================================
* 导入模板 CSV 文件创建集合（multipart/form-data）
* 导入模板 CSV 或 Excel 文件创建集合（multipart/form-data）
* ============================================================ */
'/core/dataset/collection/create/template': {
post: {
summary: '导入模板 CSV 创建集合',
summary: '导入模板文件创建集合',
description:
'上传表头由 q、a、index、metadata 组成的 CSV 模板文件，q/a/metadata 各一列，index 可多列且顺序任意，批量导入数据到知识库集合。metadata 单元格为 JSON object。`file` 为 CSV 文件，`data` 为 JSON 序列化的集合参数对象',
'上传 CSV 或 Excel 模板文件，批量导入数据到知识库集合。表头由 q、a、index、metadata 组成，q/a/metadata 各一列，index 可多列且顺序任意，metadata 单元格为 JSON object；Excel 仅支持单工作表且不能包含合并单元格；同时兼容旧版 q、a、indexes 表头。`file` 为上传文件，`data` 为 JSON 序列化的集合参数对象',
tags: [DevApiTagsMap.datasetCollectionCrteate],
requestBody: {
content: {
......
packages/service/common/file/read/utils.ts
......@@ -73,9 +73,7 @@ export const readFileContentByBuffer = async ({
prefix: string;
expiredTime?: Date;
};
}): Promise<{
rawText: string;
}> => {
}): Promise<Pick<ReadFileResponse, 'rawText' | 'tableInfo'>> => {
// 归一化扩展名为小写，避免大写/混合大小写后缀（如 .PDF）无法匹配解析器（#6996）
const extension = rawExtension.toLowerCase();
......@@ -274,7 +272,7 @@ export const readFileContentByBuffer = async ({
const start = Date.now();
logger.debug('Start parsing file', { extension });
const { rawText, formatText } = await (async () => {
const { rawText, formatText, tableInfo } = await (async () => {
if (extension === 'pdf') {
return await pdfParseFn();
}
......@@ -284,6 +282,7 @@ export const readFileContentByBuffer = async ({
logger.debug('File parsing completed', { extension, durationMs: Date.now() - start });
return {
rawText: getFormatText ? formatText || rawText : rawText
rawText: getFormatText ? formatText || rawText : rawText,
tableInfo
};
};
packages/service/core/dataset/importFile.ts 0 → 100644
import path from 'path';
import Papa from 'papaparse';
import { readRawTextByLocalFile } from '../../common/file/read/utils';
import { parseDatasetCsvHeaders } from './read';
const supportedDatasetImportExtensions = new Set(['.csv', '.xlsx']);
/**
* 将知识库模板或备份文件解析为标准 CSV 文本。
* CSV 与 Excel 共用表头校验；Excel 仅允许单工作表且不能包含合并单元格，避免结构被静默改写。
*/
export const parseDatasetImportFile = async ({
teamId,
tmbId,
filePath,
filename,
encoding
}: {
teamId: string;
tmbId: string;
filePath: string;
filename: string;
encoding: string;
}) => {
const extension = path.extname(filename).toLowerCase();
if (!supportedDatasetImportExtensions.has(extension)) {
throw new Error('Unsupported dataset import file extension');
}
const { rawText, tableInfo } = await readRawTextByLocalFile({
teamId,
tmbId,
path: filePath,
encoding,
getFormatText: false
});
if (extension === '.xlsx') {
if (tableInfo?.sheetCount !== 1) {
throw new Error('Excel dataset import requires exactly one worksheet');
}
if (tableInfo.mergedCellCount > 0) {
throw new Error('Excel dataset import does not support merged cells');
}
}
const result = Papa.parse<string[]>(rawText);
if (result.errors.length > 0) {
throw new Error('Invalid dataset import content');
}
const rows = result.data;
const normalizedRows = rows.map((row) => row.map((cell) => String(cell ?? '')));
const [headers = []] = normalizedRows;
if (!parseDatasetCsvHeaders(headers).validTypedHeader) {
throw new Error('Invalid dataset import headers');
}
return Papa.unparse(normalizedRows);
};
packages/service/core/dataset/read.ts
......@@ -24,11 +24,11 @@ import { getBackendFileOperationTimeoutMs } from '../../common/file/parseTimeout
const logger = getLogger(LogCategories.MODULE.DATASET.FILE);
const datasetCsvColumnTypes = new Set(['q', 'a', 'index', 'metadata']);
const datasetCsvColumnTypes = new Set(['q', 'a', 'index', 'indexes', 'metadata']);
/**
* 解析 CSV 模板表头，严格限制为 q/a/index/metadata 四类固定列名，并保留原始列顺序。
* q、a 必须各出现一次，metadata 最多一列，index 可以重复。
* 解析知识库 CSV 表头，支持新版 q/a/index/metadata 和旧版 q/a/indexes 结构。
* q、a 必须各出现一次，metadata 最多一列，index/indexes 可以重复。
*/
export const parseDatasetCsvHeaders = (headers: string[]) => {
const normalized = headers.map((header) => header.trim().toLowerCase());
......
packages/service/test/common/file/read/utils.test.ts
......@@ -21,6 +21,16 @@ const {
formatText: buffer.toString(encoding || 'utf-8')
};
}
if (extension === 'xlsx') {
return {
rawText: 'q,a\nquestion,answer',
formatText: '| q | a |',
tableInfo: {
sheetCount: 1,
mergedCellCount: 0
}
};
}
return {
rawText: `parsed-${extension}-content`,
formatText: `parsed-${extension}-content`
......@@ -173,6 +183,25 @@ describe('readFileContentByBuffer', () => {
);
});
it('should preserve table information returned by the readFile worker', async () => {
const result = await readFileContentByBuffer({
teamId,
tmbId,
extension: 'xlsx',
buffer: Buffer.from('xlsx-content'),
encoding: 'utf-8',
getFormatText: false
});
expect(result).toEqual({
rawText: 'q,a\nquestion,answer',
tableInfo: {
sheetCount: 1,
mergedCellCount: 0
}
});
});
it('should use system parse for non-pdf files', async () => {
const buffer = Buffer.from('markdown content');
......
packages/service/test/core/dataset/importFile.test.ts 0 → 100644
import { beforeEach, describe, expect, it, vi } from 'vitest';
import Papa from 'papaparse';
const mockReadRawTextByLocalFile = vi.hoisted(() => vi.fn());
vi.mock('@fastgpt/service/common/file/read/utils', () => ({
readRawTextByLocalFile: mockReadRawTextByLocalFile
}));
import { parseDatasetImportFile } from '@fastgpt/service/core/dataset/importFile';
const defaultParams = {
teamId: 'team-id',
tmbId: 'tmb-id',
filePath: '/tmp/template.csv',
filename: 'template.csv',
encoding: 'utf-8'
};
describe('parseDatasetImportFile', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('parses valid CSV content from the system readFile worker', async () => {
mockReadRawTextByLocalFile.mockResolvedValue({
rawText: 'q,a,index,metadata\n"question, one","line 1\nline 2",tag,"{""rank"":1}"'
});
const rawText = await parseDatasetImportFile({
...defaultParams,
filename: 'template.CSV'
});
expect(mockReadRawTextByLocalFile).toHaveBeenCalledWith({
teamId: defaultParams.teamId,
tmbId: defaultParams.tmbId,
path: defaultParams.filePath,
encoding: defaultParams.encoding,
getFormatText: false
});
expect(Papa.parse(rawText).data).toEqual([
['q', 'a', 'index', 'metadata'],
['question, one', 'line 1\nline 2', 'tag', '{"rank":1}']
]);
});
it('accepts a single-sheet Excel result without merged cells', async () => {
mockReadRawTextByLocalFile.mockResolvedValue({
rawText: 'q,a,index,metadata\n"question, one","line 1\nline 2",tag,"{""source"":""excel""}"',
tableInfo: {
sheetCount: 1,
mergedCellCount: 0
}
});
const rawText = await parseDatasetImportFile({
...defaultParams,
filePath: '/tmp/template.xlsx',
filename: 'template.xlsx'
});
expect(Papa.parse(rawText).data).toEqual([
['q', 'a', 'index', 'metadata'],
['question, one', 'line 1\nline 2', 'tag', '{"source":"excel"}']
]);
});
it('accepts legacy headers returned by the Excel worker', async () => {
mockReadRawTextByLocalFile.mockResolvedValue({
rawText: 'q,a,indexes\nquestion,answer,tag',
tableInfo: {
sheetCount: 1,
mergedCellCount: 0
}
});
await expect(
parseDatasetImportFile({
...defaultParams,
filePath: '/tmp/backup.xlsx',
filename: 'backup.xlsx'
})
).resolves.toContain('indexes');
});
it('rejects unsupported extensions before reading the file', async () => {
await expect(
parseDatasetImportFile({
...defaultParams,
filePath: '/tmp/template.xls',
filename: 'template.xls'
})
).rejects.toThrow('extension');
expect(mockReadRawTextByLocalFile).not.toHaveBeenCalled();
});
it.each([
{
name: 'malformed content',
result: { rawText: 'q,a\n"question,answer' },
error: 'content'
},
{
name: 'invalid headers',
result: { rawText: 'question,answer\nquestion,answer' },
error: 'headers'
},
{
name: 'empty content',
result: { rawText: '' },
error: 'content'
}
])('rejects $name returned by the worker', async ({ result, error }) => {
mockReadRawTextByLocalFile.mockResolvedValue(result);
await expect(parseDatasetImportFile(defaultParams)).rejects.toThrow(error);
});
it.each([
{
name: 'missing table information',
tableInfo: undefined,
error: 'exactly one worksheet'
},
{
name: 'multiple worksheets',
tableInfo: { sheetCount: 2, mergedCellCount: 0 },
error: 'exactly one worksheet'
},
{
name: 'merged cells',
tableInfo: { sheetCount: 1, mergedCellCount: 1 },
error: 'merged cells'
}
])('rejects Excel with $name', async ({ tableInfo, error }) => {
mockReadRawTextByLocalFile.mockResolvedValue({
rawText: 'q,a\nquestion,answer',
tableInfo
});
await expect(
parseDatasetImportFile({
...defaultParams,
filePath: '/tmp/template.xlsx',
filename: 'template.xlsx'
})
).rejects.toThrow(error);
});
});
packages/service/test/core/dataset/textSplitter.test.ts
......@@ -445,16 +445,39 @@ function buildCsv(rows: string[][]): string {
return rows.map((row) => row.join(',')).join('\n');
}
describe('parseDatasetCsvHeaders', () => {
it('accepts current and legacy dataset CSV headers', () => {
expect(parseDatasetCsvHeaders(['metadata', 'index', 'a', 'q', 'index'])).toEqual({
normalized: ['metadata', 'index', 'a', 'q', 'index'],
typedHeader: true,
validTypedHeader: true
});
expect(parseDatasetCsvHeaders([' q ', ' A ', 'indexes'])).toEqual({
normalized: ['q', 'a', 'indexes'],
typedHeader: true,
validTypedHeader: true
});
});
it('rejects empty, duplicate required, duplicate metadata, and unknown headers', () => {
expect(parseDatasetCsvHeaders([]).validTypedHeader).toBe(false);
expect(parseDatasetCsvHeaders(['q', 'q', 'a']).validTypedHeader).toBe(false);
expect(parseDatasetCsvHeaders(['q', 'a', 'a']).validTypedHeader).toBe(false);
expect(parseDatasetCsvHeaders(['q', 'a', 'metadata', 'metadata']).validTypedHeader).toBe(false);
expect(parseDatasetCsvHeaders(['q', 'a', 'source'])).toEqual({
normalized: ['q', 'a', 'source'],
typedHeader: false,
validTypedHeader: false
});
});
});
describe('rawText2Chunks backupParse', () => {
it('accepts typed CSV headers in any order and parses one JSON metadata column', async () => {
const csv =
'metadata,index,a,q,index\n"{""source"":""crm"",""rank"":3}",tag1,answer,question,tag2';
const result = await rawText2Chunks({ rawText: csv, backupParse: true });
expect(parseDatasetCsvHeaders(['metadata', 'index', 'a', 'q']).validTypedHeader).toBe(true);
expect(parseDatasetCsvHeaders(['q', 'a', 'indexes']).validTypedHeader).toBe(false);
expect(parseDatasetCsvHeaders(['q', 'a', 'metadata', 'metadata']).validTypedHeader).toBe(false);
expect(parseDatasetCsvHeaders(['q', 'a', 'source']).validTypedHeader).toBe(false);
expect(result).toEqual([
{
q: 'question',
......
packages/service/test/worker/readFile/extension/xlsx.test.ts
import { describe, expect, it } from 'vitest';
import Papa from 'papaparse';
import XLSX from 'xlsx';
import { readXlsxRawText } from '@fastgpt/service/worker/readFile/extension/xlsx';
......@@ -22,9 +23,18 @@ describe('readXlsxRawText', () => {
encoding: 'utf-8'
});
expect(result.rawText).toBe(
',name|alias,,age,city,\n,Alice|A,,30,Bei\njing,\n,,,,,\n,,,,,\n,,,,,\n,Bob,,25,Shanghai,'
);
expect(Papa.parse(result.rawText).data).toEqual([
['', 'name|alias', '', 'age', 'city', ''],
['', 'Alice|A', '', '30', 'Bei\njing', ''],
['', '', '', '', '', ''],
['', '', '', '', '', ''],
['', '', '', '', '', ''],
['', 'Bob', '', '25', 'Shanghai', '']
]);
expect(result.tableInfo).toEqual({
sheetCount: 1,
mergedCellCount: 0
});
expect(result.formatText).toContain('| name\\|alias | age | city |');
expect(result.formatText).toContain('| Alice\\|A | 30 | Bei\\njing |');
expect(result.formatText).toContain('| Bob | 25 | Shanghai |');
......@@ -60,6 +70,10 @@ describe('readXlsxRawText', () => {
expect(result.formatText).toContain('| 销售 | 张三 | 华东 | 华东 | 华东 |');
expect(result.formatText).toContain('| 销售 | 李四 | 华东 | 华东 | 华东 |');
expect(result.formatText).toContain('| 技术 | 王五 | 华南 | | |');
expect(result.tableInfo).toEqual({
sheetCount: 1,
mergedCellCount: 3
});
});
it('should fill merged cells when sheet data starts from a non-A1 range', async () => {
......@@ -86,5 +100,43 @@ describe('readXlsxRawText', () => {
expect(result.formatText).toContain('| 部门 | 姓名 |');
expect(result.formatText).toContain('| 销售 | 张三 |');
expect(result.formatText).toContain('| 销售 | 李四 |');
expect(result.tableInfo).toEqual({
sheetCount: 1,
mergedCellCount: 1
});
});
it('should report multiple worksheets and preserve CSV cell boundaries', async () => {
const workbook = XLSX.utils.book_new();
XLSX.utils.book_append_sheet(
workbook,
XLSX.utils.aoa_to_sheet([
['q', 'a', 'metadata'],
['question, one', 'line 1\nline 2', '{"source":"excel"}']
]),
'Sheet1'
);
XLSX.utils.book_append_sheet(
workbook,
XLSX.utils.aoa_to_sheet([
['q', 'a'],
['question two', 'answer two']
]),
'Sheet2'
);
const buffer = XLSX.write(workbook, { type: 'buffer', bookType: 'xlsx' });
const result = await readXlsxRawText({
extension: 'xlsx',
buffer,
encoding: 'utf-8'
});
expect(result.rawText).toContain('"question, one"');
expect(result.rawText).toContain('"line 1\nline 2"');
expect(result.tableInfo).toEqual({
sheetCount: 2,
mergedCellCount: 0
});
});
});
packages/service/worker/readFile/extension/xlsx.ts
import { CUSTOM_SPLIT_SIGN } from '../../../common/string/textSplitter';
import { type ReadRawTextByBuffer, type ReadFileResponse } from '../type';
import Papa from 'papaparse';
import XLSX from 'xlsx';
import { filterEmptyTableData, formatMarkdownTableRow } from './utils';
......@@ -52,7 +53,8 @@ export const readXlsxRawText = async ({
return {
name,
data
data,
mergedCellCount: merges.length
};
});
......@@ -64,7 +66,7 @@ export const readXlsxRawText = async ({
const format2Csv = result.map(({ name, data }) => {
return {
title: `#${name}`,
csvText: data.map((item) => item.join(',')).join('\n')
csvText: Papa.unparse(data)
};
});
......@@ -86,6 +88,10 @@ ${data.slice(1).map(formatMarkdownTableRow).join('\n')}`;
return {
rawText: rawText,
formatText
formatText,
tableInfo: {
sheetCount: result.length,
mergedCellCount: result.reduce((count, item) => count + item.mergedCellCount, 0)
}
};
};
packages/service/worker/readFile/type.ts
......@@ -39,4 +39,8 @@ export type ParsedPage = {
export type ReadFileResponse = {
rawText: string;
formatText?: string;
tableInfo?: {
sheetCount: number;
mergedCellCount: number;
};
};
packages/web/components/common/Icon/constants.ts
......@@ -458,8 +458,6 @@ export const iconPaths = {
'modal/changePer': () => import('./icons/modal/changePer.svg'),
'modal/concat': () => import('./icons/modal/concat.svg'),
'modal/edit': () => import('./icons/modal/edit.svg'),
'modal/manualDataset': () => import('./icons/modal/manualDataset.svg'),
'modal/selectSource': () => import('./icons/modal/selectSource.svg'),
'modal/setting': () => import('./icons/modal/setting.svg'),
'modal/teamPlans': () => import('./icons/modal/teamPlans.svg'),
'model/huggingface': () => import('./icons/model/huggingface.svg'),
......
packages/web/components/common/Icon/icons/modal/manualDataset.svg deleted 100644 → 0
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 23 20" >
<path fill-rule="evenodd" clip-rule="evenodd"
d="M5.71816 2.87964H17.3957C18.3404 2.87964 18.9267 2.88125 19.367 2.91781C19.7805 2.95215 19.8857 3.00685 19.9005 3.01459L19.9008 3.01474C20.12 3.12859 20.2987 3.3073 20.4126 3.52647L20.4127 3.52677C20.4205 3.54165 20.4752 3.64681 20.5095 4.0603C20.5461 4.50061 20.5477 5.08688 20.5477 6.03164V13.9685C20.5477 14.9133 20.5461 15.4995 20.5095 15.9399C20.4752 16.3533 20.4205 16.4585 20.4127 16.4734L20.4126 16.4737C20.2987 16.6929 20.12 16.8716 19.9008 16.9854L19.9005 16.9856C19.8857 16.9933 19.7805 17.048 19.367 17.0823C18.9267 17.1189 18.3404 17.1205 17.3957 17.1205H5.71816C4.77341 17.1205 4.18714 17.1189 3.74682 17.0823C3.33333 17.048 3.22817 16.9933 3.21329 16.9856L3.21299 16.9854C2.99382 16.8716 2.81512 16.6929 2.70127 16.4737L2.70111 16.4734C2.69337 16.4585 2.63867 16.3533 2.60434 15.9399C2.56777 15.4995 2.56616 14.9133 2.56616 13.9685V6.03164C2.56616 5.08688 2.56777 4.50061 2.60434 4.0603C2.63867 3.64681 2.69337 3.54165 2.70111 3.52677L2.70127 3.52647C2.81512 3.3073 2.99382 3.12859 3.21299 3.01474L3.21329 3.01459C3.22817 3.00685 3.33333 2.95215 3.74682 2.91781C4.18714 2.88125 4.77341 2.87964 5.71816 2.87964ZM0.566162 6.03164C0.566162 4.20927 0.566162 3.29808 0.926439 2.60452C1.23004 2.02006 1.70659 1.54352 2.29104 1.23992C2.98461 0.879639 3.89579 0.879639 5.71816 0.879639H17.3957C19.218 0.879639 20.1292 0.879639 20.8228 1.23992C21.4073 1.54352 21.8838 2.02006 22.1874 2.60452C22.5477 3.29808 22.5477 4.20927 22.5477 6.03164V13.9685C22.5477 15.7909 22.5477 16.7021 22.1874 17.3956C21.8838 17.9801 21.4073 18.4566 20.8228 18.7602C20.1292 19.1205 19.218 19.1205 17.3957 19.1205H5.71816C3.89579 19.1205 2.98461 19.1205 2.29104 18.7602C1.70659 18.4566 1.23004 17.9801 0.926439 17.3956C0.566162 16.7021 0.566162 15.7909 0.566162 13.9685V6.03164ZM5.58889 5.09675C5.03661 5.09675 4.58889 5.54446 4.58889 6.09675C4.58889 6.64903 5.03661 7.09675 5.58889 7.09675H6.18889V12.9353H5.58889C5.03661 12.9353 4.58889 13.383 4.58889 13.9353C4.58889 14.4876 5.03661 14.9353 5.58889 14.9353H8.58889C9.14118 14.9353 9.58889 14.4876 9.58889 13.9353C9.58889 13.383 9.14118 12.9353 8.58889 12.9353H7.98889L7.98889 7.09675H8.58889C9.14118 7.09675 9.58889 6.64903 9.58889 6.09675C9.58889 5.54446 9.14118 5.09675 8.58889 5.09675L5.58889 5.09675ZM13.1274 13.9353C13.1274 14.4876 12.6797 14.9353 12.1274 14.9353C11.5751 14.9353 11.1274 14.4876 11.1274 13.9353C11.1274 13.383 11.5751 12.9353 12.1274 12.9353C12.6797 12.9353 13.1274 13.383 13.1274 13.9353ZM15.6678 14.9353C16.2201 14.9353 16.6678 14.4876 16.6678 13.9353C16.6678 13.383 16.2201 12.9353 15.6678 12.9353C15.1155 12.9353 14.6678 13.383 14.6678 13.9353C14.6678 14.4876 15.1155 14.9353 15.6678 14.9353Z"
fill="#3370FF" />
</svg>
\ No newline at end of file
packages/web/components/common/Icon/icons/modal/selectSource.svg deleted 100644 → 0
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 25 24" >
<path fill-rule="evenodd" clip-rule="evenodd"
d="M15.8432 4.64576C14.2573 3.93912 12.4854 3.76406 10.7919 4.14669C9.09836 4.52932 7.57389 5.44914 6.44584 6.76896C5.31778 8.08878 4.64658 9.7379 4.53235 11.4703C4.41811 13.2028 4.86695 14.9258 5.81194 16.3823C6.75692 17.8388 8.14741 18.9508 9.77603 19.5525C11.4047 20.1542 13.1841 20.2133 14.8491 19.7211C16.5141 19.2288 17.9753 18.2115 19.0149 16.8209C20.0544 15.4303 20.6166 13.7409 20.6176 12.0047V11.1726C20.6176 10.6203 21.0653 10.1726 21.6176 10.1726C22.1699 10.1726 22.6176 10.6203 22.6176 11.1726V12.0053C22.6164 14.1728 21.9145 16.2824 20.6167 18.0184C19.3189 19.7544 17.4947 21.0244 15.4162 21.639C13.3376 22.2535 11.1161 22.1797 9.08293 21.4286C7.04975 20.6774 5.31385 19.2892 4.13413 17.4708C2.9544 15.6525 2.39407 13.5016 2.53668 11.3388C2.6793 9.17595 3.51722 7.11719 4.92549 5.46952C6.33376 3.82185 8.23692 2.67354 10.3511 2.19586C12.4653 1.71819 14.6773 1.93673 16.6572 2.8189C17.1616 3.04368 17.3884 3.63486 17.1636 4.13933C16.9388 4.6438 16.3476 4.87054 15.8432 4.64576ZM22.3244 4.05154C22.7151 4.44187 22.7154 5.07504 22.3251 5.46576L13.2737 14.5262C13.0862 14.7139 12.8318 14.8194 12.5665 14.8194C12.3012 14.8195 12.0467 14.7141 11.8591 14.5265L9.14374 11.8111C8.75322 11.4206 8.75322 10.7874 9.14374 10.3969C9.53427 10.0064 10.1674 10.0064 10.558 10.3969L12.5659 12.4049L20.9102 4.05225C21.3005 3.66153 21.9337 3.66121 22.3244 4.05154Z"
fill="#3370FF" />
</svg>
\ No newline at end of file
packages/web/i18n/en/dataset.json
......@@ -11,7 +11,7 @@
"backup_dataset_success": "Backup created successfully",
"backup_dataset_tip": "Re-import the CSV file downloaded when exporting a knowledge base.",
"backup_mode": "Backup import",
"backup_template_invalid": "Invalid backup file format. Expected a CSV file with columns: q, a, indexes",
"backup_template_invalid": "Invalid file format. Check the headers and content. Headers must be q, a, index, metadata (index may repeat; metadata is optional). Excel files must contain a single worksheet with no merged cells",
"batch_delete": "Batch delete",
"chunk_max_tokens": "Max chunk tokens",
"chunk_process_params": "Chunk processing parameters",
......@@ -178,7 +178,7 @@
"tag.tags": "Tags",
"tag.total_tags": "Total {{total}} tags",
"template_dataset": "Template import",
"template_file_invalid": "The template file format is incorrect, it should be the csv file with the first column as q,a,indexes",
"template_file_invalid": "Invalid file format. Check the headers and content. Headers must be q, a, index, metadata (index may repeat; metadata is optional). Excel files must contain a single worksheet with no merged cells",
"template_mode": "Template import",
"the_knowledge_base_has_indexes_that_are_being_trained_or_being_rebuilt": "The Dataset has indexes that are being trained or rebuilt",
"total_num_files": "Total {{total}} files",
......
packages/web/i18n/zh-CN/dataset.json
......@@ -11,7 +11,7 @@
"backup_dataset_success": "备份创建成功",
"backup_dataset_tip": "可以将导出知识库时，下载的 csv 文件重新导入。",
"backup_mode": "备份导入",
"backup_template_invalid": "备份文件格式不正确，应该是首列为 q,a,indexes 的 csv 文件",
"backup_template_invalid": "文件格式异常，请检查表头和内容是否符合要求。表头应为 q、a、index、metadata（index 可重复，metadata 可选）；Excel 文件仅支持单个工作表，且不能包含合并单元格",
"batch_delete": "批量删除",
"chunk_max_tokens": "分块上限",
"chunk_process_params": "分块处理参数",
......@@ -178,7 +178,7 @@
"tag.tags": "标签",
"tag.total_tags": "共{{total}}个标签",
"template_dataset": "模版导入",
"template_file_invalid": "模板文件格式不正确，应该是首列为 q,a,indexes 的 csv 文件",
"template_file_invalid": "文件格式异常，请检查表头和内容是否符合要求。表头应为 q、a、index、metadata（index 可重复，metadata 可选）；Excel 文件仅支持单个工作表，且不能包含合并单元格",
"template_mode": "模板导入",
"the_knowledge_base_has_indexes_that_are_being_trained_or_being_rebuilt": "知识库有训练中或正在重建的索引",
"total_num_files": "共 {{total}} 个文件",
......
packages/web/i18n/zh-Hant/dataset.json
......@@ -11,7 +11,7 @@
"backup_dataset_success": "備份創建成功",
"backup_dataset_tip": "可以將導出知識庫時，下載的 csv 文件重新導入。",
"backup_mode": "備份導入",
"backup_template_invalid": "備份文件格式不正確，應該是首列為 q,a,indexes 的 csv 文件",
"backup_template_invalid": "文件格式異常，請檢查表頭和內容是否符合要求。表頭應為 q、a、index、metadata（index 可重複，metadata 可選）；Excel 文件僅支持單個工作表，且不能包含合併單元格",
"batch_delete": "批量刪除",
"chunk_max_tokens": "分塊上限",
"chunk_process_params": "分塊處理參數",
......@@ -178,7 +178,7 @@
"tag.tags": "標籤",
"tag.total_tags": "共 {{total}} 個標籤",
"template_dataset": "模版導入",
"template_file_invalid": "模板文件格式不正確，應該是首列為 q,a,indexes 的 csv 文件",
"template_file_invalid": "文件格式異常，請檢查表頭和內容是否符合要求。表頭應為 q、a、index、metadata（index 可重複，metadata 可選）；Excel 文件僅支持單個工作表，且不能包含合併單元格",
"template_mode": "模板導入",
"the_knowledge_base_has_indexes_that_are_being_trained_or_being_rebuilt": "資料集有索引正在訓練或重建中",
"total_num_files": "共 {{total}} 個文件",
......
pro @ a9977faf
Subproject commit 2bc391e398ec4da405442a47416291fa440129c2
Subproject commit a9977faf4ea2af995e1ac399515684f79162d05d
projects/app/src/pageComponents/dataset/EditFolderModal.tsx
import React, { useMemo, useRef, useState } from 'react';
import { ModalFooter, ModalBody, Input, Button } from '@chakra-ui/react';
import MyModal from '@fastgpt/web/components/common/MyModal';
import { Input, Button } from '@chakra-ui/react';
import MyModal from '@fastgpt/web/components/v2/common/MyModal';
import { useTranslation } from 'next-i18next';
import { useRequest } from '@fastgpt/web/hooks/useRequest';
......@@ -44,21 +44,24 @@ const EditFolderModal = ({
);
return (
<MyModal isOpen onClose={onClose} iconSrc="common/folderFill" title={typeMap.title}>
<ModalBody>
<Input
ref={inputRef}
defaultValue={name}
placeholder={t('common:dataset.Folder Name') || ''}
autoFocus
maxLength={100}
/>
</ModalBody>
<ModalFooter>
<MyModal
isOpen
onClose={onClose}
title={typeMap.title}
size="sm"
footer={
<Button isLoading={isLoading} onClick={onSave}>
{t('common:Confirm')}
</Button>
</ModalFooter>
}
>
<Input
ref={inputRef}
defaultValue={name}
placeholder={t('common:dataset.Folder Name') || ''}
autoFocus
maxLength={100}
/>
</MyModal>
);
};
......
projects/app/src/pageComponents/dataset/detail/CollectionCard/BackupImportModal.tsx
import React, { useState } from 'react';
import MyModal from '@fastgpt/web/components/common/MyModal';
import MyModal from '@fastgpt/web/components/v2/common/MyModal';
import { useTranslation } from 'next-i18next';
import { Box, Button, HStack, ModalBody, ModalFooter, VStack } from '@chakra-ui/react';
import { Box, Button, HStack, VStack } from '@chakra-ui/react';
import MyIcon from '@fastgpt/web/components/common/Icon';
import MyIconButton from '@fastgpt/web/components/common/Icon/button';
import { postBackupDatasetCollection } from '@/web/core/dataset/api/collection';
......@@ -42,51 +42,55 @@ const BackupImportModal = ({
);
return (
<MyModal iconSrc="backup" iconColor={'primary.600'} isOpen title={t('dataset:backup_dataset')}>
<ModalBody>
<LightTip mb={3} icon="common/info" text={t('dataset:backup_dataset_tip')} />
<MyModal
isOpen
title={t('dataset:backup_dataset')}
size="md"
footer={
<>
<Button isLoading={isBackupLoading} variant="whiteBase" onClick={onClose}>
{t('common:Close')}
</Button>
<Button onClick={onBackupImport} isDisabled={selectFiles.length === 0 || isBackupLoading}>
{isBackupLoading
? percent === 100
? t('dataset:data_parsing')
: t('dataset:data_uploading', { num: percent })
: t('common:comfirm_import')}
</Button>
</>
}
>
<LightTip mb={3} icon="common/info" text={t('dataset:backup_dataset_tip')} />
<FileSelectorBox
maxCount={1}
fileType=".csv"
selectFiles={selectFiles}
setSelectFiles={(e) => setSelectFiles(e)}
/>
{/* File render */}
{selectFiles.length > 0 && (
<VStack mt={4} gap={2}>
{selectFiles.map((item, index) => (
<HStack key={index} w={'100%'}>
<MyIcon name={item.icon as any} w={'1rem'} />
<Box color={'myGray.900'}>{item.name}</Box>
<Box fontSize={'xs'} color={'myGray.500'} flex={1}>
{item.size}
</Box>
<MyIconButton
icon="delete"
hoverColor="red.500"
hoverBg="red.50"
onClick={() => {
setSelectFiles(selectFiles.filter((_, i) => i !== index));
}}
/>
</HStack>
))}
</VStack>
)}
</ModalBody>
<ModalFooter>
<Button isLoading={isBackupLoading} variant="whiteBase" mr={2} onClick={onClose}>
{t('common:Close')}
</Button>
<Button onClick={onBackupImport} isDisabled={selectFiles.length === 0 || isBackupLoading}>
{isBackupLoading
? percent === 100
? t('dataset:data_parsing')
: t('dataset:data_uploading', { num: percent })
: t('common:Import')}
</Button>
</ModalFooter>
<FileSelectorBox
maxCount={1}
fileType=".csv,.xlsx"
selectFiles={selectFiles}
setSelectFiles={(e) => setSelectFiles(e)}
/>
{/* File render */}
{selectFiles.length > 0 && (
<VStack mt={4} gap={2}>
{selectFiles.map((item, index) => (
<HStack key={index} w={'100%'}>
<MyIcon name={item.icon as any} w={'1rem'} />
<Box color={'myGray.900'}>{item.name}</Box>
<Box fontSize={'xs'} color={'myGray.500'} flex={1}>
{item.size}
</Box>
<MyIconButton
icon="delete"
hoverColor="red.500"
hoverBg="red.50"
onClick={() => {
setSelectFiles(selectFiles.filter((_, i) => i !== index));
}}
/>
</HStack>
))}
</VStack>
)}
</MyModal>
);
};
......
projects/app/src/pageComponents/dataset/detail/CollectionCard/Header.tsx
......@@ -600,10 +600,7 @@ const Header = ({
name={editFolderData.name}
/>
)}
<EditCreateVirtualFileModal
iconSrc={'modal/manualDataset'}
closeBtnText={t('common:Cancel')}
/>
<EditCreateVirtualFileModal closeBtnText={t('common:Cancel')} size="sm" />
{isOpenFileSourceSelector && <FileSourceSelector onClose={onCloseFileSourceSelector} />}
{isOpenBackupImportModal && (
<BackupImportModal
......
projects/app/src/pageComponents/dataset/detail/CollectionCard/TemplateImportModal.tsx
import React, { useState } from 'react';
import MyModal from '@fastgpt/web/components/common/MyModal';
import MyModal from '@fastgpt/web/components/v2/common/MyModal';
import { useTranslation } from 'next-i18next';
import { Box, Button, HStack, ModalBody, ModalFooter, VStack, Flex, Link } from '@chakra-ui/react';
import { Box, Button, HStack, VStack, Flex, Link } from '@chakra-ui/react';
import FileSelector, { type SelectFileItemType } from '@/components/Select/FileSelectorBox';
import MyIcon from '@fastgpt/web/components/common/Icon';
import MyIconButton from '@fastgpt/web/components/common/Icon/button';
......@@ -63,96 +63,93 @@ const TemplateImportModal = ({
return (
<MyModal
iconSrc="common/layer"
iconColor={'primary.600'}
title={t('dataset:template_dataset')}
isOpen
w={'500px'}
h={'auto'}
size="md"
footer={
<>
<Button isLoading={isImporting} variant="whiteBase" onClick={onClose}>
{t('common:Close')}
</Button>
<Button onClick={onImport} isDisabled={selectFiles.length === 0 || isImporting}>
{isImporting
? percent === 100
? t('dataset:data_parsing')
: t('dataset:data_uploading', { num: percent })
: t('common:comfirm_import')}
</Button>
</>
}
>
<ModalBody py={6} px={8}>
<VStack spacing={3} alignItems="stretch">
<Flex justify={'space-between'} align={'center'} fontSize={'sm'} fontWeight={500}>
<Box color={'myGray.900'}>{t('dataset:upload_by_template_format')}</Box>
<Link
display={'flex'}
alignItems={'center'}
gap={0.5}
href={getDocPath('/guide/dataset/template')}
color="primary.600"
target="_blank"
>
<MyIcon name={'book'} w={'18px'} />
{t('common:Instructions')}
</Link>
</Flex>
<Button
variant="whiteBase"
w={'100%'}
h={'48px'}
leftIcon={<MyIcon name={'common/download'} w={'18px'} />}
onClick={handleDownloadTemplate}
<VStack spacing={3} alignItems="stretch">
<Flex justify={'space-between'} align={'center'} fontSize={'sm'} fontWeight={500}>
<Box color={'myGray.900'}>{t('dataset:upload_by_template_format')}</Box>
<Link
display={'flex'}
alignItems={'center'}
gap={0.5}
href={getDocPath('/guide/dataset/template')}
color="primary.600"
target="_blank"
>
{t('dataset:download_csv_template')}
</Button>
<MyIcon name={'book'} w={'18px'} />
{t('common:Instructions')}
</Link>
</Flex>
<FileSelector
maxCount={1}
fileType=".csv"
selectFiles={selectFiles}
setSelectFiles={setSelectFiles}
FileTypeNode={
<Box fontSize={'xs'}>
<Trans
i18nKey={i18nT('file:template_csv_file_select_tip')}
values={{
fileType: '.csv'
}}
components={{
highlight: <Box as="span" color="primary.600" fontWeight="medium" />
<Button
variant="whiteBase"
w={'100%'}
h={'48px'}
leftIcon={<MyIcon name={'common/download'} w={'18px'} />}
onClick={handleDownloadTemplate}
>
{t('dataset:download_csv_template')}
</Button>
<FileSelector
maxCount={1}
fileType=".csv,.xlsx"
selectFiles={selectFiles}
setSelectFiles={setSelectFiles}
FileTypeNode={
<Box fontSize={'xs'}>
<Trans
i18nKey={i18nT('file:template_csv_file_select_tip')}
values={{
fileType: '.csv, .xlsx'
}}
components={{
highlight: <Box as="span" color="primary.600" fontWeight="medium" />
}}
/>
</Box>
}
/>
{/* File render */}
{selectFiles.length > 0 && (
<VStack gap={2}>
{selectFiles.map((item, index) => (
<HStack key={index} w={'100%'}>
<MyIcon name={item.icon as any} w={'1rem'} />
<Box color={'myGray.900'}>{item.name}</Box>
<Box fontSize={'xs'} color={'myGray.500'} flex={1}>
{item.size}
</Box>
<MyIconButton
icon="delete"
hoverColor="red.500"
hoverBg="red.50"
onClick={() => {
setSelectFiles(selectFiles.filter((_, i) => i !== index));
}}
/>
</Box>
}
/>
{/* File render */}
{selectFiles.length > 0 && (
<VStack gap={2}>
{selectFiles.map((item, index) => (
<HStack key={index} w={'100%'}>
<MyIcon name={item.icon as any} w={'1rem'} />
<Box color={'myGray.900'}>{item.name}</Box>
<Box fontSize={'xs'} color={'myGray.500'} flex={1}>
{item.size}
</Box>
<MyIconButton
icon="delete"
hoverColor="red.500"
hoverBg="red.50"
onClick={() => {
setSelectFiles(selectFiles.filter((_, i) => i !== index));
}}
/>
</HStack>
))}
</VStack>
)}
</VStack>
</ModalBody>
<ModalFooter>
<Button isLoading={isImporting} variant="whiteBase" mr={2} onClick={onClose}>
{t('common:Close')}
</Button>
<Button onClick={onImport} isDisabled={selectFiles.length === 0 || isImporting}>
{isImporting
? percent === 100
? t('dataset:data_parsing')
: t('dataset:data_uploading', { num: percent })
: t('common:comfirm_import')}
</Button>
</ModalFooter>
</HStack>
))}
</VStack>
)}
</VStack>
</MyModal>
);
};
......
projects/app/src/pageComponents/dataset/detail/Import/components/FileSourceSelector.tsx
import React, { useState } from 'react';
import MyModal from '@fastgpt/web/components/common/MyModal';
import { ModalBody, ModalFooter, Button } from '@chakra-ui/react';
import MyModal from '@fastgpt/web/components/v2/common/MyModal';
import { Button } from '@chakra-ui/react';
import { useTranslation } from 'next-i18next';
import LeftRadio from '@fastgpt/web/components/common/Radio/LeftRadio';
import { useRouter } from 'next/router';
......@@ -16,34 +16,9 @@ const FileModeSelector = ({ onClose }: { onClose: () => void }) => {
<MyModal
isOpen
onClose={onClose}
iconSrc="modal/selectSource"
title={t('common:core.dataset.import.Select source')}
w={'600px'}
>
<ModalBody px={6} py={4}>
<LeftRadio
list={[
{
title: t('common:core.dataset.import.Local file'),
desc: t('common:core.dataset.import.Local file desc'),
value: ImportDataSourceEnum.fileLocal
},
{
title: t('common:core.dataset.import.Web link'),
desc: t('common:core.dataset.import.Web link desc'),
value: ImportDataSourceEnum.fileLink
},
{
title: t('common:core.dataset.import.Custom text'),
desc: t('common:core.dataset.import.Custom text desc'),
value: ImportDataSourceEnum.fileCustom
}
]}
value={value}
onChange={setValue}
/>
</ModalBody>
<ModalFooter>
size="md"
footer={
<Button
onClick={() =>
router.replace({
......@@ -57,7 +32,29 @@ const FileModeSelector = ({ onClose }: { onClose: () => void }) => {
>
{t('common:Confirm')}
</Button>
</ModalFooter>
}
>
<LeftRadio
list={[
{
title: t('common:core.dataset.import.Local file'),
desc: t('common:core.dataset.import.Local file desc'),
value: ImportDataSourceEnum.fileLocal
},
{
title: t('common:core.dataset.import.Web link'),
desc: t('common:core.dataset.import.Web link desc'),
value: ImportDataSourceEnum.fileLink
},
{
title: t('common:core.dataset.import.Custom text'),
desc: t('common:core.dataset.import.Custom text desc'),
value: ImportDataSourceEnum.fileCustom
}
]}
value={value}
onChange={setValue}
/>
</MyModal>
);
};
......
projects/app/src/pages/api/core/dataset/collection/create/backup.ts
import type { ApiRequestProps } from '@fastgpt/next/type';
import { NextAPI } from '@/service/middleware/entry';
import { getLogger, LogCategories } from '@fastgpt/service/common/logger';
import { readRawTextByLocalFile } from '@fastgpt/service/common/file/read/utils';
import { authDataset } from '@fastgpt/service/support/permission/dataset/auth';
import { WritePermissionVal } from '@fastgpt/global/support/permission/constant';
import { createCollectionAndInsertData } from '@fastgpt/service/core/dataset/collection/controller';
......@@ -10,12 +9,11 @@ import {
DatasetCollectionTypeEnum
} from '@fastgpt/global/core/dataset/constants';
import { i18nT } from '@fastgpt/global/common/i18n/utils';
import { isCSVFile } from '@fastgpt/global/common/file/utils';
import { multer } from '@fastgpt/service/common/file/multer';
import { getS3DatasetSource } from '@fastgpt/service/common/s3/sources/dataset';
import { CreateBackupCollectionFormSchema } from '@fastgpt/global/openapi/core/dataset/collection/createApi';
import { checkDatasetIndexLimit } from '@fastgpt/service/support/permission/teamLimit';
import { getDatasetCsvHeaders, parseDatasetCsvHeaders } from '@fastgpt/service/core/dataset/read';
import { parseDatasetImportFile } from '@fastgpt/service/core/dataset/importFile';
const logger = getLogger(LogCategories.MODULE.DATASET.COLLECTION);
async function handler(req: ApiRequestProps) {
......@@ -30,10 +28,6 @@ async function handler(req: ApiRequestProps) {
const filename = decodeURIComponent(result.fileMetadata.originalname);
const { datasetId, parentId } = CreateBackupCollectionFormSchema.parse(result.data);
if (!isCSVFile(filename)) {
return Promise.reject('File must be a CSV file');
}
const { teamId, tmbId, dataset } = await authDataset({
req,
authToken: true,
......@@ -48,18 +42,16 @@ async function handler(req: ApiRequestProps) {
insertLen: 1
});
const { rawText } = await readRawTextByLocalFile({
const rawText = await parseDatasetImportFile({
teamId,
tmbId,
path: result.fileMetadata.path,
encoding: result.fileMetadata.encoding,
getFormatText: false
});
const { validTypedHeader } = parseDatasetCsvHeaders(getDatasetCsvHeaders(rawText));
if (!validTypedHeader) {
filePath: result.fileMetadata.path,
filename,
encoding: result.fileMetadata.encoding
}).catch((error) => {
logger.warn('Backup dataset import file parse failed', { filename, error });
return Promise.reject(i18nT('dataset:backup_template_invalid'));
}
});
const fileId = await getS3DatasetSource().upload({
datasetId: dataset._id,
......
projects/app/src/pages/api/core/dataset/collection/create/template.ts
import type { ApiRequestProps } from '@fastgpt/next/type';
import { NextAPI } from '@/service/middleware/entry';
import { getLogger, LogCategories } from '@fastgpt/service/common/logger';
import { readRawTextByLocalFile } from '@fastgpt/service/common/file/read/utils';
import { authDataset } from '@fastgpt/service/support/permission/dataset/auth';
import { WritePermissionVal } from '@fastgpt/global/support/permission/constant';
import { createCollectionAndInsertData } from '@fastgpt/service/core/dataset/collection/controller';
......@@ -10,12 +9,11 @@ import {
DatasetCollectionTypeEnum
} from '@fastgpt/global/core/dataset/constants';
import { i18nT } from '@fastgpt/global/common/i18n/utils';
import { isCSVFile } from '@fastgpt/global/common/file/utils';
import { multer } from '@fastgpt/service/common/file/multer';
import { getS3DatasetSource } from '@fastgpt/service/common/s3/sources/dataset';
import { CreateTemplateCollectionFormSchema } from '@fastgpt/global/openapi/core/dataset/collection/createApi';
import { checkDatasetIndexLimit } from '@fastgpt/service/support/permission/teamLimit';
import { getDatasetCsvHeaders, parseDatasetCsvHeaders } from '@fastgpt/service/core/dataset/read';
import { parseDatasetImportFile } from '@fastgpt/service/core/dataset/importFile';
const logger = getLogger(LogCategories.MODULE.DATASET.COLLECTION);
async function handler(req: ApiRequestProps) {
......@@ -30,10 +28,6 @@ async function handler(req: ApiRequestProps) {
const filename = decodeURIComponent(result.fileMetadata.originalname);
const { datasetId, parentId } = CreateTemplateCollectionFormSchema.parse(result.data);
if (!isCSVFile(filename)) {
return Promise.reject('File must be a CSV file');
}
const { teamId, tmbId, dataset } = await authDataset({
req,
authToken: true,
......@@ -48,18 +42,16 @@ async function handler(req: ApiRequestProps) {
insertLen: 1
});
const { rawText } = await readRawTextByLocalFile({
const rawText = await parseDatasetImportFile({
teamId,
tmbId,
path: result.fileMetadata.path,
encoding: result.fileMetadata.encoding,
getFormatText: false
});
const { validTypedHeader } = parseDatasetCsvHeaders(getDatasetCsvHeaders(rawText));
if (!validTypedHeader) {
filePath: result.fileMetadata.path,
filename,
encoding: result.fileMetadata.encoding
}).catch((error) => {
logger.warn('Template dataset import file parse failed', { filename, error });
return Promise.reject(i18nT('dataset:template_file_invalid'));
}
});
const fileId = await getS3DatasetSource().upload({
datasetId: dataset._id,
......@@ -86,7 +78,7 @@ async function handler(req: ApiRequestProps) {
return {};
} catch (error) {
logger.error(`Backup dataset collection create error: ${error}`);
logger.error(`Template dataset collection create error: ${error}`);
return Promise.reject(error);
} finally {
multer.clearDiskTempFiles(filepaths);
......
projects/app/src/web/common/hooks/useEditTitle.tsx
import React, { useCallback, useRef } from 'react';
import { ModalFooter, ModalBody, Input, useDisclosure, Button, Box } from '@chakra-ui/react';
import MyModal from '@fastgpt/web/components/common/MyModal';
import { Input, useDisclosure, Button } from '@chakra-ui/react';
import MyModal from '@fastgpt/web/components/v2/common/MyModal';
import { useToast } from '@fastgpt/web/hooks/useToast';
import { useTranslation } from 'next-i18next';
import { useRequest } from '@fastgpt/web/hooks/useRequest';
......@@ -74,42 +74,46 @@ export const useEditTitle = ({
}
}, [canEmpty, onClose, toast, valueRule]);
// eslint-disable-next-line react/display-name
const EditModal = useCallback(
({
maxLength = 50,
iconSrc = 'modal/edit',
closeBtnText = t('common:Close')
closeBtnText = t('common:Close'),
size = 'md'
}: {
maxLength?: number;
iconSrc?: string;
closeBtnText?: string;
size?: 'sm' | 'md' | 'lg' | 'xl';
}) => {
const { runAsync, loading } = useRequest(onclickConfirm);
return (
<MyModal isOpen={isOpen} onClose={onClose} iconSrc={iconSrc} title={title} maxW={'500px'}>
<ModalBody>
{!!tip && <FormLabel mb={2}>{tip}</FormLabel>}
<Input
ref={inputRef}
defaultValue={defaultValue.current}
placeholder={placeholder}
autoFocus
maxLength={maxLength}
/>
</ModalBody>
<ModalFooter>
{!!closeBtnText && (
<Button mr={3} variant={'whiteBase'} onClick={onClose}>
{closeBtnText}
<MyModal
isOpen={isOpen}
onClose={onClose}
title={title}
size={size}
footer={
<>
{!!closeBtnText && (
<Button variant={'whiteBase'} onClick={onClose}>
{closeBtnText}
</Button>
)}
<Button onClick={runAsync} isLoading={loading}>
{t('common:Confirm')}
</Button>
)}
<Button onClick={runAsync} isLoading={loading}>
{t('common:Confirm')}
</Button>
</ModalFooter>
</>
}
>
{!!tip && <FormLabel mb={2}>{tip}</FormLabel>}
<Input
ref={inputRef}
defaultValue={defaultValue.current}
placeholder={placeholder}
autoFocus
maxLength={maxLength}
/>
</MyModal>
);
},
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment