feat: add metadata field to dataset_data for custom metadata support\… (#7400)

* feat: add metadata field to dataset_data for custom metadata support\n\n- Zod schemas: add metadata to DatasetDataSchema, DatasetDataItemSchema, CreateDatasetDataPropsSchema, UpdateDatasetDataPropsSchema, PushDataChunkSchema, GetDataListItemSchema\n- MongoDB schemas: add metadata to dataset_datas and dataMetadata to dataset_trainings\n- CSV import: parse header row to identify non-q/a/indexes columns as metadata keys\n- Training pipeline: pass metadata through training queue (dataMetadata) to dataset_data (metadata)\n- Search output: include metadata in search result items via buildSearchResultItem\n- API: include metadata in v2/list responses\n- Template: update download template header to demo metadata column, relax header validation to check only first two columns are q,a\n\nCo-Authored-By: Claude <noreply@anthropic.com> * fix: preserve dataset metadata across import and export * chore: remove metadata design document * fix: allow a single metadata template column * fix: reject legacy dataset template headers * docs: add dataset metadata release note --------- Co-authored-by: hexiaoyan30010 <30010@sangfor.com> Co-authored-by: Archer <545436317@qq.com>

feat: add metadata field to dataset_data for custom metadata support\… (#7400)
* feat: add metadata field to dataset_data for custom metadata support\n\n- Zod schemas: add metadata to DatasetDataSchema, DatasetDataItemSchema, CreateDatasetDataPropsSchema, UpdateDatasetDataPropsSchema, PushDataChunkSchema, GetDataListItemSchema\n- MongoDB schemas: add metadata to dataset_datas and dataMetadata to dataset_trainings\n- CSV import: parse header row to identify non-q/a/indexes columns as metadata keys\n- Training pipeline: pass metadata through training queue (dataMetadata) to dataset_data (metadata)\n- Search output: include metadata in search result items via buildSearchResultItem\n- API: include metadata in v2/list responses\n- Template: update download template header to demo metadata column, relax header validation to check only first two columns are q,a\n\nCo-Authored-By: Claude <noreply@anthropic.com> * fix: preserve dataset metadata across import and export * chore: remove metadata design document * fix: allow a single metadata template column * fix: reject legacy dataset template headers * docs: add dataset metadata release note --------- Co-authored-by: hexiaoyan30010 <30010@sangfor.com> Co-authored-by: Archer <545436317@qq.com>
5f7d76f4 · Hxy · GitHub · 934776f4 · 5f7d76f4 · 5f7d76f4
Commit 5f7d76f4 authored Aug 03, 2026 by Hxy Committed by GitHub Aug 03, 2026
23 changed files
--- a/document/content/self-host/upgrading/4-16/41601.en.mdx
+++ b/document/content/self-host/upgrading/4-16/41601.en.mdx
@@ -93,6 +93,7 @@ The script first selects HTTP tool Apps by type, then migrates historical versio
 3. App Workflows now archive and restore their Workspace automatically when the Sandbox Provider or runtime image changes, completing the upgrade within the current run.
 4. Workflow tool nodes can now have selected input parameters generated by the Agent while preserving existing fixed-value, reference, and user-input settings.
 5. When ChatAgent selects a tool, each parameter can be explicitly marked as AI-generated.
+6. Knowledge Base data now supports custom `metadata`. JSON metadata can be imported through the API or CSV templates and is preserved in search results and backup exports. CSV templates only allow `q`, `a`, `index`, and `metadata` headers; `q`, `a`, and `metadata` must each appear once, while `index` may appear multiple times in any order.
 ## ⚙️ Improvements

--- a/document/content/self-host/upgrading/4-16/41601.mdx
+++ b/document/content/self-host/upgrading/4-16/41601.mdx
@@ -93,6 +93,7 @@ curl -X POST 'https://你的域名/api/admin/4160/initHttpToolSchema' \
 3. App Workflow 在 Sandbox Provider 或运行时镜像变化时自动归档并恢复 Workspace，升级过程在当前运行中静默完成。
 4. 工作流工具节点支持将指定输入参数交由 Agent 自动生成，并保留固定值、引用和用户输入等既有配置。
 5. ChatAgent 选择工具时，支持手动指定是否为 AI 生成参数。
+6. 知识库数据支持自定义 `metadata`，可通过 API 或 CSV 模板导入 JSON 元数据；检索结果和备份导出会保留该字段。CSV 模板仅允许 `q`、`a`、`index`、`metadata` 表头，`q`、`a`、`metadata` 各一列，`index` 可多列且顺序任意。
 ## ⚙️ 优化

--- a/document/data/doc-last-modified.json
+++ b/document/data/doc-last-modified.json
@@ -329,8 +329,8 @@
  "content/self-host/upgrading/4-15/4154.mdx": "2026-07-30T11:22:58+08:00",
  "content/self-host/upgrading/4-15/4155.en.mdx": "2026-07-30T11:22:58+08:00",
  "content/self-host/upgrading/4-15/4155.mdx": "2026-07-30T11:22:58+08:00",
-  "content/self-host/upgrading/4-16/41601.en.mdx": "2026-07-30T15:22:52+08:00",
+  "content/self-host/upgrading/4-16/41601.en.mdx": "2026-08-03T10:30:00+08:00",
-  "content/self-host/upgrading/4-16/41601.mdx": "2026-07-30T15:22:52+08:00",
+  "content/self-host/upgrading/4-16/41601.mdx": "2026-08-03T10:30:00+08:00",
  "content/self-host/upgrading/outdated/40.en.mdx": "2026-07-25T00:27:20+08:00",
  "content/self-host/upgrading/outdated/40.mdx": "2026-07-25T00:27:20+08:00",
  "content/self-host/upgrading/outdated/41.en.mdx": "2026-07-25T00:27:20+08:00",

--- a/packages/global/core/dataset/type.ts
+++ b/packages/global/core/dataset/type.ts
@@ -215,7 +215,8 @@ export const DatasetDataSchema = DatasetDataFieldSchema.extend({
  fullTextToken: z.string().meta({ description: '全文 token' }),
  indexes: z.array(DatasetDataIndexItemSchema).meta({ description: '向量索引' }),
  rebuilding: z.boolean().optional().meta({ description: '重建中' }),
-  imageDescMap: z.record(z.string(), z.string()).optional().meta({ description: '图片描述映射' })
+  imageDescMap: z.record(z.string(), z.string()).optional().meta({ description: '图片描述映射' }),
+  metadata: z.record(z.string(), z.any()).optional().meta({ description: '自定义元数据' })
 });
 export type DatasetDataSchemaType = z.infer<typeof DatasetDataSchema>;
@@ -245,6 +246,10 @@ export const DatasetTrainingSchema = z.object({
  a: z.string().meta({ description: '回答/补充文本' }),
  imageId: z.string().optional().meta({ description: '图片 ID' }),
  imageDescMap: z.record(z.string(), z.string()).optional().meta({ description: '图片描述映射' }),
+  dataMetadata: z
+    .record(z.string(), z.any())
+    .optional()
+    .meta({ description: '自定义元数据（训练时透传）' }),
  chunkIndex: z.number().meta({ description: '块索引' }),
  indexSize: z.number().optional().meta({ description: '索引大小' }),
  weight: z.number().meta({ description: '权重' }),
@@ -364,7 +369,8 @@ export const DatasetDataItemSchema = DatasetDataFieldSchema.extend({
  chunkIndex: z.number().meta({ description: '块索引' }),
  indexes: z.array(DatasetDataIndexItemSchema).meta({ description: '向量索引' }),
  imageDescMap: z.record(z.string(), z.string()).optional().meta({ description: '图片描述映射' }),
-  isOwner: z.boolean().meta({ description: '是否为 owner' })
+  isOwner: z.boolean().meta({ description: '是否为 owner' }),
+  metadata: z.record(z.string(), z.any()).optional().meta({ description: '自定义元数据' })
 });
 export type DatasetDataItemType = z.infer<typeof DatasetDataItemSchema>;
@@ -390,7 +396,8 @@ export const UpdateDatasetDataPropsSchema = z.object({
  }),
  indexPrefix: z.string().optional().meta({
    description: '索引前缀标题'
-  })
+  }),
+  metadata: z.record(z.string(), z.any()).optional().meta({ description: '自定义元数据' })
 });
 export type UpdateDatasetDataPropsType = z.infer<typeof UpdateDatasetDataPropsSchema>;
@@ -408,7 +415,8 @@ export const CreateDatasetDataPropsSchema = z.object({
    .array(DatasetDataIndexItemSchema.omit({ dataId: true }))
    .optional()
    .meta({ description: '向量索引列表' }),
-  indexPrefix: z.string().optional().meta({ description: '索引前缀标题' })
+  indexPrefix: z.string().optional().meta({ description: '索引前缀标题' }),
+  metadata: z.record(z.string(), z.any()).optional().meta({ description: '自定义元数据' })
 });
 export type CreateDatasetDataPropsType = z.infer<typeof CreateDatasetDataPropsSchema>;
@@ -435,7 +443,8 @@ export type DatasetFileSchemaType = z.infer<typeof DatasetFileSchema>;
 export const SearchDataResponseItemSchema = DatasetDataItemSchema.omit({
  teamId: true,
  indexes: true,
-  isOwner: true
+  isOwner: true,
+  metadata: true
 })
  .extend({
    score: z
@@ -446,7 +455,8 @@ export const SearchDataResponseItemSchema = DatasetDataItemSchema.omit({
          index: z.number().meta({ description: '索引' })
        })
      )
-      .meta({ description: '评分列表' })
+      .meta({ description: '评分列表' }),
+    metadata: z.record(z.string(), z.any()).optional().meta({ description: '自定义元数据' })
  })
  .meta({ description: '搜索数据响应项' });
 export type SearchDataResponseItemType = z.infer<typeof SearchDataResponseItemSchema>;

--- a/packages/global/openapi/core/dataset/collection/createApi.ts
+++ b/packages/global/openapi/core/dataset/collection/createApi.ts
@@ -185,7 +185,11 @@ export type CreateBackupCollectionFormType = z.infer<typeof CreateBackupCollecti
 // OpenAPI 文档专用
 export const CreateBackupCollectionMultipartSchema = z.object({
-  file: z.any().meta({ format: 'binary', description: '备份 CSV 文件（格式：q,a,indexes）' }),
+  file: z.any().meta({
+    format: 'binary',
+    description:
+      '备份 CSV 文件（表头由 q、a、index、metadata 组成，q/a/metadata 各一列，index 可多列且顺序任意，metadata 单元格为 JSON object）'
+  }),
  data: CreateBackupCollectionFormSchema.meta({ description: '集合参数（JSON 序列化后传入）' })
 });
@@ -203,7 +207,11 @@ export type CreateTemplateCollectionFormType = z.infer<typeof CreateTemplateColl
 // OpenAPI 文档专用
 export const CreateTemplateCollectionMultipartSchema = z.object({
-  file: z.any().meta({ format: 'binary', description: '模板 CSV 文件（格式：q,a,indexes）' }),
+  file: z.any().meta({
+    format: 'binary',
+    description:
+      '模板 CSV 文件（表头由 q、a、index、metadata 组成，q/a/metadata 各一列，index 可多列且顺序任意，metadata 单元格为 JSON object）'
+  }),
  data: CreateTemplateCollectionFormSchema.meta({ description: '集合参数（JSON 序列化后传入）' })
 });

--- a/packages/global/openapi/core/dataset/collection/createPath.ts
+++ b/packages/global/openapi/core/dataset/collection/createPath.ts
@@ -239,7 +239,7 @@ export const DatasetCollectionCreatePath: OpenAPIPath = {
    post: {
      summary: '导入备份 CSV 创建集合',
      description:
-        '上传格式为 q,a,indexes 的 CSV 备份文件，恢复数据到知识库集合。`file` 为 CSV 文件，`data` 为 JSON 序列化的集合参数对象',
+        '上传表头由 q、a、index、metadata 组成的 CSV 备份文件，q/a/metadata 各一列，index 可多列且顺序任意，恢复数据到知识库集合。metadata 单元格为 JSON object。`file` 为 CSV 文件，`data` 为 JSON 序列化的集合参数对象',
      tags: [DevApiTagsMap.datasetCollectionCrteate],
      requestBody: {
        content: {
@@ -266,7 +266,7 @@ export const DatasetCollectionCreatePath: OpenAPIPath = {
    post: {
      summary: '导入模板 CSV 创建集合',
      description:
-        '上传格式为 q,a,indexes 的 CSV 模板文件，批量导入数据到知识库集合。`file` 为 CSV 文件，`data` 为 JSON 序列化的集合参数对象',
+        '上传表头由 q、a、index、metadata 组成的 CSV 模板文件，q/a/metadata 各一列，index 可多列且顺序任意，批量导入数据到知识库集合。metadata 单元格为 JSON object。`file` 为 CSV 文件，`data` 为 JSON 序列化的集合参数对象',
      tags: [DevApiTagsMap.datasetCollectionCrteate],
      requestBody: {
        content: {

--- a/packages/global/openapi/core/dataset/data/api.ts
+++ b/packages/global/openapi/core/dataset/data/api.ts
@@ -32,7 +32,10 @@ const PushDataChunkSchema = z.object({
  indexes: z
    .array(DatasetDataIndexItemSchema.omit({ dataId: true }))
    .optional()
-    .meta({ description: '额外向量索引' })
+    .meta({ description: '额外向量索引' }),
+  metadata: z.record(z.string(), z.any()).optional().meta({
+    description: '自定义元数据'
+  })
 });
 export type PushDataChunkType = z.infer<typeof PushDataChunkSchema>;
@@ -55,7 +58,7 @@ export type GetDatasetDataDetailResponse = z.infer<typeof GetDatasetDataDetailRe
 * API: 更新数据集数据
 * Route: PUT /api/core/dataset/data/update
 * ============================================================================ */
-export const UpdateDatasetDataBodySchema = UpdateDatasetDataPropsSchema;
+export const UpdateDatasetDataBodySchema = UpdateDatasetDataPropsSchema.omit({ metadata: true });
 export type UpdateDatasetDataBody = z.infer<typeof UpdateDatasetDataBodySchema>;
 export const UpdateDatasetDataResponseSchema = z.object({
  q: z.string().optional().meta({

--- a/packages/global/test/openapi/core/dataset/data/api.test.ts
+++ b/packages/global/test/openapi/core/dataset/data/api.test.ts
+import { describe, expect, it } from 'vitest';
+import { SearchDataResponseItemSchema } from '@fastgpt/global/core/dataset/type';
+import { InsertDataBodySchema } from '@fastgpt/global/openapi/core/dataset/data/api';
+const objectId = '507f1f77bcf86cd799439011';
+describe('dataset data metadata API schemas', () => {
+  it('keeps metadata in the immediate insert request schema', () => {
+    const result = InsertDataBodySchema.parse({
+      collectionId: objectId,
+      q: 'question',
+      metadata: {
+        rank: 3,
+        nested: { source: 'crm' }
+      }
+    });
+    expect(result.metadata).toEqual({
+      rank: 3,
+      nested: { source: 'crm' }
+    });
+  });
+  it('accepts arbitrary JSON metadata in search responses', () => {
+    const result = SearchDataResponseItemSchema.parse({
+      id: objectId,
+      datasetId: objectId,
+      collectionId: objectId,
+      q: 'question',
+      a: 'answer',
+      updateTime: new Date(),
+      sourceName: 'collection',
+      chunkIndex: 0,
+      score: [],
+      metadata: {
+        rank: 3,
+        enabled: true,
+        nested: { source: 'crm' }
+      }
+    });
+    expect(result.metadata).toEqual({
+      rank: 3,
+      enabled: true,
+      nested: { source: 'crm' }
+    });
+  });
+});
--- a/packages/service/core/dataset/data/schema.ts
+++ b/packages/service/core/dataset/data/schema.ts
@@ -38,6 +38,9 @@ const DatasetDataSchema = new Schema({
  },
  imageId: String,
  imageDescMap: Object,
+  metadata: {
+    type: Object
+  },
  history: {
    type: [
      {

--- a/packages/service/core/dataset/read.ts
+++ b/packages/service/core/dataset/read.ts
@@ -23,6 +23,36 @@ import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset';
 const logger = getLogger(LogCategories.MODULE.DATASET.FILE);
+const datasetCsvColumnTypes = new Set(['q', 'a', 'index', 'metadata']);
+/**
+ * 解析 CSV 模板表头，严格限制为 q/a/index/metadata 四类固定列名，并保留原始列顺序。
+ * q、a 必须各出现一次，metadata 最多一列，index 可以重复。
+ */
+export const parseDatasetCsvHeaders = (headers: string[]) => {
+  const normalized = headers.map((header) => header.trim().toLowerCase());
+  const typedHeader =
+    normalized.length > 0 && normalized.every((header) => datasetCsvColumnTypes.has(header));
+  return {
+    normalized,
+    typedHeader,
+    validTypedHeader:
+      typedHeader &&
+      normalized.filter((header) => header === 'q').length === 1 &&
+      normalized.filter((header) => header === 'a').length === 1 &&
+      normalized.filter((header) => header === 'metadata').length <= 1
+  };
+};
+/**
+ * 从 CSV 原文读取第一行表头，统一复用 PapaParse，避免 API 层用字符串 split 误判带引号的表头。
+ */
+export const getDatasetCsvHeaders = (rawText: string) => {
+  const [headers = []] = Papa.parse(rawText).data as string[][];
+  return headers;
+};
 export const readFileRawTextByUrl = async ({
  teamId,
  tmbId,
@@ -310,24 +340,94 @@ export const rawText2Chunks = async ({
    q: string;
    a: string;
    indexes?: string[];
+    metadata?: Record<string, any>;
    imageIdList?: string[];
  }[]
 > => {
  const parseDatasetBackup2Chunks = (rawText: string) => {
    const csvArr = Papa.parse(rawText).data as string[][];
+    if (csvArr.length < 2) return { chunks: [] };
+    const rawHeaders = csvArr[0];
+    const { normalized: headers, typedHeader } = parseDatasetCsvHeaders(rawHeaders);
+    // Build column index mapping
+    let qIdx = -1,
+      aIdx = -1;
+    const indexesIdxs: number[] = [];
+    const metadataKeys: { idx: number; key: string }[] = [];
+    const metadataIdxs: number[] = [];
+    headers.forEach((header, idx) => {
+      if (header === 'q') {
+        qIdx = idx;
+      } else if (header === 'a') {
+        aIdx = idx;
+      } else if (header === 'index' || header === 'indexes') {
+        indexesIdxs.push(idx);
+      } else if (typedHeader && header === 'metadata') {
+        metadataIdxs.push(idx);
+      } else {
+        metadataKeys.push({ idx, key: rawHeaders[idx].trim() });
+      }
+    });
+    // 旧导出格式只有一个 indexes 表头，但数据行会把多个索引展开到后续单元格。
+    const legacyIndexesStart =
+      metadataKeys.length === 0 && metadataIdxs.length === 0 && indexesIdxs.length === 1
+        ? indexesIdxs[0]
+        : undefined;
    const chunks = csvArr
      .slice(1)
-      .map((item) => ({
+      .map((item) => {
-        q: item[0] || '',
+        const q = qIdx >= 0 ? item[qIdx] || '' : '';
-        a: item[1] || '',
+        const a = aIdx >= 0 ? item[aIdx] || '' : '';
-        indexes: item.slice(2).filter((item) => item.trim()),
-        imageIdList
+        const indexes = (
-      }))
+          legacyIndexesStart !== undefined
+            ? item.slice(legacyIndexesStart)
+            : indexesIdxs.map((idx) => item[idx])
+        )
+          .map((value) => (value || '').trim())
+          .filter(Boolean);
+        // Build metadata: only include non-empty values
+        let metadata: Record<string, any> | undefined;
+        for (const { idx, key } of metadataKeys) {
+          const val = (item[idx] || '').trim();
+          if (val) {
+            metadata = metadata || {};
+            metadata[key] = val;
+          }
+        }
+        for (const idx of metadataIdxs) {
+          const val = (item[idx] || '').trim();
+          if (!val) continue;
+          let parsedValue: Record<string, any> | undefined;
+          try {
+            const parsed = JSON.parse(val);
+            if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
+              parsedValue = parsed;
+            }
+          } catch {}
+          metadata = metadata || {};
+          if (parsedValue) {
+            Object.assign(metadata, parsedValue);
+          } else {
+            // 固定 metadata 表头没有字段名，非法 JSON 仍按列序保留，避免静默丢值。
+            metadata[`metadata_${idx}`] = val;
+          }
+        }
+        return { q, a, indexes, metadata, imageIdList };
+      })
      .filter((item) => item.q || item.a);
-    return {
+    return { chunks };
-      chunks
-    };
  };
  if (backupParse) {

--- a/packages/service/core/dataset/search/defaultRecall/constant.ts
+++ b/packages/service/core/dataset/search/defaultRecall/constant.ts
@@ -12,7 +12,8 @@ export const datasetDataSelectField = {
  imageId: 1,
  imageDescMap: 1,
  chunkIndex: 1,
-  indexes: 1
+  indexes: 1,
+  metadata: 1
 };
 /**

--- a/packages/service/core/dataset/search/defaultRecall/result.ts
+++ b/packages/service/core/dataset/search/defaultRecall/result.ts
@@ -37,6 +37,7 @@ export const buildSearchResultItem = ({
    imageId: data.imageId,
    chunkIndex: data.chunkIndex,
    ...(includeIndexes ? { indexes: data.indexes } : {}),
+    ...(data.metadata ? { metadata: data.metadata } : {}),
    datasetId: String(data.datasetId),
    collectionId: String(data.collectionId),
    ...getCollectionSourceData(collection),

--- a/packages/service/core/dataset/training/controller.ts
+++ b/packages/service/core/dataset/training/controller.ts
@@ -193,6 +193,7 @@ export const pushDataListToTrainingQueue = async ({
          ...(item.q && { q: item.q }),
          ...(item.a && { a: item.a }),
          ...(item.imageId && { imageId: item.imageId }),
+          ...(item.metadata && { dataMetadata: item.metadata }),
          chunkIndex: item.chunkIndex ?? 0,
          indexSize,
          weight: weight ?? 0,

--- a/packages/service/core/dataset/training/schema.ts
+++ b/packages/service/core/dataset/training/schema.ts
@@ -68,6 +68,9 @@ const TrainingDataSchema = new Schema({
  },
  imageId: String,
  imageDescMap: Object,
+  dataMetadata: {
+    type: Object
+  },
  chunkIndex: {
    type: Number,
    default: 0

--- a/packages/service/support/permission/dataset/auth.ts
+++ b/packages/service/support/permission/dataset/auth.ts
@@ -218,6 +218,7 @@ export async function authDatasetData({
    indexes: datasetData.indexes,
    datasetId: String(datasetData.datasetId),
    collectionId: String(datasetData.collectionId),
+    metadata: datasetData.metadata,
    sourceName: result.collection.name || '',
    sourceId: result.collection?.fileId || result.collection?.rawLink,
    isOwner: String(datasetData.tmbId) === String(result.tmbId)

--- a/packages/service/test/core/dataset/textSplitter.test.ts
+++ b/packages/service/test/core/dataset/textSplitter.test.ts
--- a/projects/app/src/pageComponents/dataset/detail/CollectionCard/TemplateImportModal.tsx
+++ b/projects/app/src/pageComponents/dataset/detail/CollectionCard/TemplateImportModal.tsx
@@ -44,9 +44,9 @@ const TemplateImportModal = ({
  );
  const handleDownloadTemplate = () => {
-    const templateContent = `q,a,indexes
+    const templateContent = `q,a,index,index,metadata
-"Who are you?","I am an AI assistant, here to help with your questions and provide support. I can assist with learning, daily life queries, and creative ideas.","1. What are you?\n2. What can you do?\n3. What topics can you help with?\n4. How do you assist users?\n5. What's your goal?","Who are you? I am an AI assistant..."
+"Who are you?","I am an AI assistant, here to help with your questions and provide support.","1. What are you?","2. What can you do?","{""source"":""template"",""category"":""faq""}"
-"What are you?","I am an AI assistant designed to help users with their questions and provide support across various topics.","What are you?","I am an AI assistant..."`;
+"What are you?","I am an AI assistant designed to help users with their questions and provide support across various topics.","What are you?","How can you help?","{""source"":""template"",""category"":""general""}"`;
    const blob = new Blob([templateContent], { type: 'text/csv;charset=utf-8;' });
    const link = document.createElement('a');

--- a/projects/app/src/pages/api/core/dataset/collection/create/backup.ts
+++ b/projects/app/src/pages/api/core/dataset/collection/create/backup.ts
@@ -15,6 +15,7 @@ import { multer } from '@fastgpt/service/common/file/multer';
 import { getS3DatasetSource } from '@fastgpt/service/common/s3/sources/dataset';
 import { CreateBackupCollectionFormSchema } from '@fastgpt/global/openapi/core/dataset/collection/createApi';
 import { checkDatasetIndexLimit } from '@fastgpt/service/support/permission/teamLimit';
+import { getDatasetCsvHeaders, parseDatasetCsvHeaders } from '@fastgpt/service/core/dataset/read';
 const logger = getLogger(LogCategories.MODULE.DATASET.COLLECTION);
 async function handler(req: ApiRequestProps) {
@@ -55,7 +56,8 @@ async function handler(req: ApiRequestProps) {
      getFormatText: false
    });
-    if (!rawText.trim().startsWith('q,a,indexes')) {
+    const { validTypedHeader } = parseDatasetCsvHeaders(getDatasetCsvHeaders(rawText));
+    if (!validTypedHeader) {
      return Promise.reject(i18nT('dataset:backup_template_invalid'));
    }

--- a/projects/app/src/pages/api/core/dataset/collection/create/template.ts
+++ b/projects/app/src/pages/api/core/dataset/collection/create/template.ts
@@ -15,6 +15,7 @@ import { multer } from '@fastgpt/service/common/file/multer';
 import { getS3DatasetSource } from '@fastgpt/service/common/s3/sources/dataset';
 import { CreateTemplateCollectionFormSchema } from '@fastgpt/global/openapi/core/dataset/collection/createApi';
 import { checkDatasetIndexLimit } from '@fastgpt/service/support/permission/teamLimit';
+import { getDatasetCsvHeaders, parseDatasetCsvHeaders } from '@fastgpt/service/core/dataset/read';
 const logger = getLogger(LogCategories.MODULE.DATASET.COLLECTION);
 async function handler(req: ApiRequestProps) {
@@ -55,7 +56,8 @@ async function handler(req: ApiRequestProps) {
      getFormatText: false
    });
-    if (!rawText.trim().startsWith('q,a,indexes')) {
+    const { validTypedHeader } = parseDatasetCsvHeaders(getDatasetCsvHeaders(rawText));
+    if (!validTypedHeader) {
      return Promise.reject(i18nT('dataset:template_file_invalid'));
    }

--- a/projects/app/src/pages/api/core/dataset/data/insertData.ts
+++ b/projects/app/src/pages/api/core/dataset/data/insertData.ts
@@ -24,7 +24,7 @@ import {
 } from '@fastgpt/global/openapi/core/dataset/data/api';
 async function handler(req: ApiRequestProps): Promise<InsertDataResponse> {
-  const { collectionId, q, a, indexes } = parseApiInput({
+  const { collectionId, q, a, indexes, metadata } = parseApiInput({
    req,
    bodySchema: InsertDataBodySchema
  }).body;
@@ -82,7 +82,8 @@ async function handler(req: ApiRequestProps): Promise<InsertDataResponse> {
    indexPrefix: indexPrefixTitle ? `# ${name}` : undefined,
    embeddingModel: vectorModelData.model,
    imageIndex: !!imageIndex,
-    indexes: formatIndexes
+    indexes: formatIndexes,
+    metadata
  });
  pushGenerateVectorUsage({

--- a/projects/app/src/pages/api/core/dataset/exportAll.ts
+++ b/projects/app/src/pages/api/core/dataset/exportAll.ts
@@ -22,6 +22,12 @@ type DataItemType = {
  q: string;
  a: string;
  indexes: DatasetDataSchemaType['indexes'];
+  metadata?: Record<string, any>;
+};
+type ExportSchemaType = {
+  maxIndexCount?: number;
+  hasMetadata?: number;
 };
 async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
@@ -46,22 +52,52 @@ async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
    fields: '_id'
  });
+  const match = {
+    teamId,
+    datasetId: { $in: datasets.map((d) => d._id) }
+  };
+  const [exportSchema] = await MongoDatasetData.aggregate<ExportSchemaType>(
+    [
+      { $match: match },
+      { $limit: 50000 },
+      {
+        $project: {
+          indexCount: { $size: { $ifNull: ['$indexes', []] } },
+          hasMetadata: {
+            $cond: [{ $eq: [{ $type: '$metadata' }, 'object'] }, 1, 0]
+          }
+        }
+      },
+      {
+        $group: {
+          _id: null,
+          maxIndexCount: { $max: '$indexCount' },
+          hasMetadata: { $max: '$hasMetadata' }
+        }
+      }
+    ],
+    { ...readFromSecondary }
+  );
+  const indexCount = exportSchema?.maxIndexCount || 0;
+  const hasMetadata = exportSchema?.hasMetadata === 1;
+  const headers = [
+    'q',
+    'a',
+    ...Array.from({ length: indexCount }, () => 'index'),
+    ...(hasMetadata ? ['metadata'] : [])
+  ];
  res.setHeader('Content-Type', 'text/csv; charset=utf-8;');
  res.setHeader(
    'Content-Disposition',
    `attachment; filename=${encodeURIComponent(dataset.name)}-backup.csv;`
  );
-  const cursor = MongoDatasetData.find<DataItemType>(
+  const cursor = MongoDatasetData.find<DataItemType>(match, 'q a indexes metadata', {
-    {
-      teamId,
-      datasetId: { $in: datasets.map((d) => d._id) }
-    },
-    'q a indexes',
-    {
    ...readFromSecondary
-    }
+  })
-  )
    .limit(50000)
    .cursor();
@@ -70,14 +106,19 @@ async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
    readStream: cursor
  });
-  write(`\uFEFFq,a,indexes`);
+  write(`\uFEFF${headers.join(',')}`);
  cursor.on('data', (doc: DataItemType) => {
    const sanitizedQ = sanitizeCsvField(doc.q || '');
    const sanitizedA = sanitizeCsvField(doc.a || '');
-    const sanitizedIndexes = doc.indexes.map((i) => sanitizeCsvField(i.text || '')).join(',');
+    const sanitizedIndexes = Array.from({ length: indexCount }, (_, index) =>
+      sanitizeCsvField(doc.indexes?.[index]?.text || '')
+    );
+    const sanitizedMetadata = hasMetadata
+      ? [sanitizeCsvField(doc.metadata ? JSON.stringify(doc.metadata) : '')]
+      : [];
-    write(`\n${sanitizedQ},${sanitizedA},${sanitizedIndexes}`);
+    write(`\n${[sanitizedQ, sanitizedA, ...sanitizedIndexes, ...sanitizedMetadata].join(',')}`);
  });
  cursor.on('end', () => {

--- a/projects/app/src/service/core/dataset/data/data.ts
+++ b/projects/app/src/service/core/dataset/data/data.ts
@@ -117,6 +117,7 @@ export class DatasetDataOperation {
    embeddingModel,
    imageIndex,
    imageDescMap,
+    metadata,
    session
  }: CreateDatasetDataPropsType & {
    embeddingModel: string;
@@ -167,6 +168,7 @@ export class DatasetDataOperation {
          a,
          imageId,
          imageDescMap,
+          ...(metadata && { metadata }),
          chunkIndex,
          indexes: results
        }
@@ -221,6 +223,7 @@ export class DatasetDataOperation {
    indexSize = 512,
    indexPrefix,
    imageIndex,
+    metadata,
    forceRebuild = false
  }: UpdateDatasetDataByIndexesProps) {
    const embModel = getEmbeddingModel(model);
@@ -294,6 +297,9 @@ export class DatasetDataOperation {
          : mongoData.history;
      mongoData.q = nextQ;
      mongoData.a = nextA;
+      if (metadata !== undefined) {
+        mongoData.metadata = metadata;
+      }
      mongoData.indexes = newIndexes;
      await mongoData.save({ session });

--- a/projects/app/src/service/core/dataset/queues/generateVector.ts
+++ b/projects/app/src/service/core/dataset/queues/generateVector.ts
@@ -357,6 +357,7 @@ const insertData = async ({ trainingData }: { trainingData: TrainingDataType }) 
      a: trainingData.a,
      imageId: trainingData.imageId,
      imageDescMap: trainingData.imageDescMap,
+      ...(trainingData.dataMetadata && { metadata: trainingData.dataMetadata }),
      chunkIndex: trainingData.chunkIndex,
      indexSize: trainingData.indexSize || getMaxIndexSize(embModel),
      indexes: trainingData.indexes || [],