Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit e5f8805b by Archer Committed by GitHub
Options

feat: add SoMark PDF provider and clear missing-file errors (#7398) 

* feat: add SoMark PDF provider

* docs: document SoMark PDF provider

* docs: update environment and v4.15.5 notes
parent fb6be9fc
Showing with 711 additions and 18 deletions
deploy/version/main/docker-compose.template.yml
...@@ -221,6 +221,8 @@ services: ...@@ -221,6 +221,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/content/self-host/config/env.en.mdx
...@@ -206,17 +206,16 @@ These variables are mainly validated by `packages/service/env.ts` and apply to ` ...@@ -206,17 +206,16 @@ These variables are mainly validated by `packages/service/env.ts` and apply to `
| `LLM_REQUEST_TRACKING_RETENTION_HOURS` | `6` | LLM request tracking retention, in hours. | | `LLM_REQUEST_TRACKING_RETENTION_HOURS` | `6` | LLM request tracking retention, in hours. |
| `MAX_HTML_TRANSFORM_CHARS` | `1000000` | Maximum number of characters for HTML-to-Markdown conversion. Larger content is not converted. | | `MAX_HTML_TRANSFORM_CHARS` | `1000000` | Maximum number of characters for HTML-to-Markdown conversion. Larger content is not converted. |
## App-Specific Variables ## Additional App Variables
These variables are mainly read by `projects/app`. Some are currently defined in `packages/service/env.ts` for shared validation, but their actual consumers are still App-side code. These variables are mainly read by `projects/app`. Some are currently defined in `packages/service/env.ts` for shared validation, but their actual consumers are still App-side code.
| Variable | Default | Description | | Variable | Default | Description |
| ----------------------------- | -------- | --------------------------------------------------------------------------- | | ------------------------------- | -------- | --------------------------------------------------------------------------------------------------- |
| `DEFAULT_ROOT_PSW` | `123456` | Default password for initializing the root user. | | `DEFAULT_ROOT_PSW` | `123456` | Default password for initializing the root user. |
| `SYSTEM_NAME` | `AI` | Default system name for the page title. | | `SYSTEM_NAME` | `AI` | Default system name for the page title. |
| `SYSTEM_DESCRIPTION` | Empty | Page meta description. If unset, the default i18n text is used. | | `SYSTEM_DESCRIPTION` | Empty | Page meta description. If unset, the default i18n text is used. |
| `SYSTEM_FAVICON` | Empty | Page favicon URL. If unset, the favicon from system config is used. | | `SYSTEM_FAVICON` | Empty | Page favicon URL. If unset, the favicon from system config is used. |
| `CONFIG_JSON_PATH` | Empty | Directory for reading `config.json` in production. Defaults to `/app/data`. |
| `CHINESE_IP_REDIRECT_URL` | Empty | China IP redirect URL in frontend config. | | `CHINESE_IP_REDIRECT_URL` | Empty | China IP redirect URL in frontend config. |
| `PAY_FORM_URL` | Empty | Payment form URL in frontend config. | | `PAY_FORM_URL` | Empty | Payment form URL in frontend config. |
| `SHOW_COUPON` | `false` | Whether redemption codes are shown. | | `SHOW_COUPON` | `false` | Whether redemption codes are shown. |
...@@ -226,6 +225,81 @@ These variables are mainly read by `projects/app`. Some are currently defined in ...@@ -226,6 +225,81 @@ These variables are mainly read by `projects/app`. Some are currently defined in
| `APP_REGISTRATION_URL` | Empty | App registration application URL. Currently kept mostly for compatibility. | | `APP_REGISTRATION_URL` | Empty | App registration application URL. Currently kept mostly for compatibility. |
| `PASSWORD_EXPIRED_MONTH` | Empty | Password expiration period in months. Empty means passwords do not expire. | | `PASSWORD_EXPIRED_MONTH` | Empty | Password expiration period in months. Empty means passwords do not expire. |
| `OPENAPI_KEY_MAX_COUNT` | `100` | Maximum number of system API Keys one team member can create. Minimum is 1. | | `OPENAPI_KEY_MAX_COUNT` | `100` | Maximum number of system API Keys one team member can create. Minimum is 1. |
| `SSE_MCP_SERVER_PROXY_ENDPOINT` | Empty | MCP SSE server proxy URL. Do not include a trailing slash. Required when publishing an SSE MCP App. |
### Open-Source Only
Starting with v4.15.0, the open-source edition no longer reads `config.json`. When upgrading from an earlier version, remove the file's volume mount and migrate the old settings to environment variables using the table below. If you did not use these optional settings, you do not need to add them.
| Former `config.json` field | Current environment variable | Default | Description |
| ------------------------------------------- | ---------------------------- | -------- | ------------------------------------------------------------------------------ |
| `systemEnv.customPdfParse.url` | `CUSTOM_PDF_PARSE_URL` | Empty | Custom PDF parsing service URL. |
| `systemEnv.customPdfParse.key` | `CUSTOM_PDF_PARSE_KEY` | Empty | Custom PDF parsing service key. |
| `systemEnv.customPdfParse.doc2xKey` | `DOC2X_KEY` | Empty | Doc2x PDF parsing service key. |
| `systemEnv.customPdfParse.textinAppId` | `TEXTIN_APP_ID` | Empty | TextIn service App ID. |
| `systemEnv.customPdfParse.textinSecretCode` | `TEXTIN_SECRET_CODE` | Empty | TextIn service Secret Code. |
| `systemEnv.hnswEfSearch` | `HNSW_EF_SEARCH` | `100` | The `hnsw.ef_search` vector search parameter for PG, OceanBase, and openGauss. |
| `systemEnv.hnswMaxScanTuples` | `HNSW_MAX_SCAN_TUPLES` | `100000` | Maximum number of tuples scanned during vector search. Applies only to PG. |
| `systemEnv.datasetParseMaxProcess` | `DATASET_PARSE_MAX_PROCESS` | `10` | Maximum concurrency for the Knowledge Base file parsing queue. |
| `systemEnv.vectorMaxProcess` | `VECTOR_MAX_PROCESS` | `10` | Maximum concurrency for the vector training queue. |
| `systemEnv.qaMaxProcess` | `QA_MAX_PROCESS` | `10` | Maximum concurrency for the Q&A splitting queue. |
| `systemEnv.vlmMaxProcess` | `VLM_MAX_PROCESS` | `10` | Maximum concurrency for the image understanding model queue. |
#### Enhanced PDF Parsing
The open-source edition supports custom PDF parsing services, SoMark, TextIn, and Doc2x. Configure only one service. If you configure more than one, FastGPT uses this priority order: custom PDF parsing service, SoMark, TextIn, then Doc2x.
##### Use the Sealos PDF Parsing Service
1. Open [Sealos AI Proxy](https://hzh.sealos.run/?uid=fnWRt09fZP&openapp=system-aiproxy) and create an API key.
2. Add the API key to FastGPT:
```dotenv
CUSTOM_PDF_PARSE_URL=https://aiproxy.hzh.sealos.run/v1/parse/pdf?model=parse-pdf
CUSTOM_PDF_PARSE_KEY=your-sealos-api-key
```
##### Use SoMark
1. Open [SoMark Studio](https://somark.ai/Studio/apikey) and create an API key.
2. Add the API key to FastGPT:
```dotenv
SOMARK_API_KEY=sk-your-api-key
```
The SoMark synchronous parsing endpoint accepts files up to 200 MB and 300 pages. See the [SoMark API documentation](https://docs.somark.ai/en/api-reference) for the complete limits and error codes.
##### Use Another Custom PDF Parsing Service
```dotenv
CUSTOM_PDF_PARSE_URL=https://your-pdf-parser.example.com/v2/parse/file
CUSTOM_PDF_PARSE_KEY=your-service-key
```
`CUSTOM_PDF_PARSE_KEY` is optional. When set, FastGPT sends it to the parsing service as `Authorization: Bearer <CUSTOM_PDF_PARSE_KEY>`. The service must accept a `multipart/form-data` POST request with a `file` field and return JSON in this format:
```json
{
"pages": 10,
"markdown": "Parsed Markdown content"
}
```
##### Use TextIn
```dotenv
TEXTIN_APP_ID=your-app-id
TEXTIN_SECRET_CODE=your-secret-code
```
##### Use Doc2x
```dotenv
DOC2X_KEY=your-api-key
```
Restart FastGPT after changing the environment variables. Then enable **Enhanced PDF Parsing** when importing files into a Knowledge Base or configuring App file uploads. PDFs use the configured enhanced parsing service only when this option is enabled; otherwise, FastGPT uses its built-in parser.
## Admin-Specific Variables ## Admin-Specific Variables
......
document/content/self-host/config/env.mdx
...@@ -211,12 +211,11 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说 ...@@ -211,12 +211,11 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说
以下变量主要由 `projects/app` 读取。其中部分变量当前定义在 `packages/service/env.ts` 中做统一校验，但实际消费点仍在 App 层。 以下变量主要由 `projects/app` 读取。其中部分变量当前定义在 `packages/service/env.ts` 中做统一校验，但实际消费点仍在 App 层。
| 变量 | 默认值 | 说明 | | 变量 | 默认值 | 说明 |
| ----------------------------- | -------- | ------------------------------------------------------- | | ------------------------------- | -------- | ---------------------------------------------------------------------- |
| `DEFAULT_ROOT_PSW` | `123456` | 初始化 root 用户默认密码。 | | `DEFAULT_ROOT_PSW` | `123456` | 初始化 root 用户默认密码。 |
| `SYSTEM_NAME` | `AI` | 页面标题默认系统名。 | | `SYSTEM_NAME` | `AI` | 页面标题默认系统名。 |
| `SYSTEM_DESCRIPTION` | 空 | 页面 Meta 描述，不配置时使用默认国际化文案。 | | `SYSTEM_DESCRIPTION` | 空 | 页面 Meta 描述，不配置时使用默认国际化文案。 |
| `SYSTEM_FAVICON` | 空 | 页面 favicon 地址，不配置时使用系统配置中的 favicon。 | | `SYSTEM_FAVICON` | 空 | 页面 favicon 地址，不配置时使用系统配置中的 favicon。 |
| `CONFIG_JSON_PATH` | 空 | 生产环境读取 `config.json` 的目录，默认 `/app/data`。 |
| `CHINESE_IP_REDIRECT_URL` | 空 | 前端配置中的中国 IP 跳转地址。 | | `CHINESE_IP_REDIRECT_URL` | 空 | 前端配置中的中国 IP 跳转地址。 |
| `PAY_FORM_URL` | 空 | 前端配置中的付费表单地址。 | | `PAY_FORM_URL` | 空 | 前端配置中的付费表单地址。 |
| `SHOW_COUPON` | `false` | 是否展示兑换码功能。 | | `SHOW_COUPON` | `false` | 是否展示兑换码功能。 |
...@@ -226,6 +225,81 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说 ...@@ -226,6 +225,81 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说
| `APP_REGISTRATION_URL` | 空 | 应用备案申请地址；当前主要作为兼容配置保留。 | | `APP_REGISTRATION_URL` | 空 | 应用备案申请地址；当前主要作为兼容配置保留。 |
| `PASSWORD_EXPIRED_MONTH` | 空 | 密码过期月份数；为空表示不过期。 | | `PASSWORD_EXPIRED_MONTH` | 空 | 密码过期月份数；为空表示不过期。 |
| `OPENAPI_KEY_MAX_COUNT` | `100` | 单个团队成员最多可创建的系统 API Key 数量，最小值为 1。 | | `OPENAPI_KEY_MAX_COUNT` | `100` | 单个团队成员最多可创建的系统 API Key 数量，最小值为 1。 |
| `SSE_MCP_SERVER_PROXY_ENDPOINT` | 空 | MCP SSE Server 代理地址，末尾不要带 `/`。发布 SSE MCP 应用时需要配置。 |
### 开源版特有
从 4.15.0 起，开源版不再读取 `config.json`。如果从旧版本升级，请删除该文件的 volume 挂载，并按下表将原配置改为环境变量。没有使用过这些可选配置时，无需额外添加。
| 原 `config.json` 字段 | 当前环境变量 | 默认值 | 说明 |
| ------------------------------------------- | --------------------------- | -------- | ---------------------------------------------------------------------- |
| `systemEnv.customPdfParse.url` | `CUSTOM_PDF_PARSE_URL` | 空 | 自定义 PDF 解析服务地址。 |
| `systemEnv.customPdfParse.key` | `CUSTOM_PDF_PARSE_KEY` | 空 | 自定义 PDF 解析服务密钥。 |
| `systemEnv.customPdfParse.doc2xKey` | `DOC2X_KEY` | 空 | Doc2x PDF 解析服务密钥。 |
| `systemEnv.customPdfParse.textinAppId` | `TEXTIN_APP_ID` | 空 | 合合信息 TextIn 服务 App ID。 |
| `systemEnv.customPdfParse.textinSecretCode` | `TEXTIN_SECRET_CODE` | 空 | 合合信息 TextIn 服务 Secret Code。 |
| `systemEnv.hnswEfSearch` | `HNSW_EF_SEARCH` | `100` | 向量检索的 `hnsw.ef_search` 参数，仅对 PG、OceanBase、openGauss 生效。 |
| `systemEnv.hnswMaxScanTuples` | `HNSW_MAX_SCAN_TUPLES` | `100000` | 向量检索最大扫描数据量，仅对 PG 生效。 |
| `systemEnv.datasetParseMaxProcess` | `DATASET_PARSE_MAX_PROCESS` | `10` | 知识库文件解析队列最大并发数。 |
| `systemEnv.vectorMaxProcess` | `VECTOR_MAX_PROCESS` | `10` | 向量训练队列最大并发数。 |
| `systemEnv.qaMaxProcess` | `QA_MAX_PROCESS` | `10` | 问答拆分队列最大并发数。 |
| `systemEnv.vlmMaxProcess` | `VLM_MAX_PROCESS` | `10` | 图片理解模型处理队列最大并发数。 |
#### PDF 增强解析配置
开源版支持接入自定义 PDF 解析服务、SoMark、TextIn 或 Doc2x。选择一种服务配置即可；如果同时配置多种服务，调用优先级为：自定义 PDF 解析服务、SoMark、TextIn、Doc2x。
##### 使用 Sealos PDF 解析服务
1. 打开 [Sealos AI Proxy](https://hzh.sealos.run/?uid=fnWRt09fZP&openapp=system-aiproxy)，申请 API Key。
2. 将 API Key 配置到 FastGPT：
```dotenv
CUSTOM_PDF_PARSE_URL=https://aiproxy.hzh.sealos.run/v1/parse/pdf?model=parse-pdf
CUSTOM_PDF_PARSE_KEY=your-sealos-api-key
```
##### 使用 SoMark
1. 打开 [SoMark Studio](https://somark.ai/Studio/apikey)，创建 API Key。
2. 将 API Key 配置到 FastGPT：
```dotenv
SOMARK_API_KEY=sk-your-api-key
```
SoMark 同步解析接口单文件最大支持 200 MB、300 页。完整限制和错误码见 [SoMark API 文档](https://docs.somark.ai/en/api-reference)。
##### 使用其他自定义 PDF 解析服务
```dotenv
CUSTOM_PDF_PARSE_URL=https://your-pdf-parser.example.com/v2/parse/file
CUSTOM_PDF_PARSE_KEY=your-service-key
```
`CUSTOM_PDF_PARSE_KEY` 可选。配置后，FastGPT 会通过 `Authorization: Bearer <CUSTOM_PDF_PARSE_KEY>` 请求解析服务。解析服务需接收包含 `file` 字段的 `multipart/form-data` POST 请求，并返回以下 JSON：
```json
{
"pages": 10,
"markdown": "Parsed Markdown content"
}
```
##### 使用 TextIn
```dotenv
TEXTIN_APP_ID=your-app-id
TEXTIN_SECRET_CODE=your-secret-code
```
##### 使用 Doc2x
```dotenv
DOC2X_KEY=your-api-key
```
修改环境变量后需要重启 FastGPT。然后在知识库导入文件或应用文件上传配置中勾选“PDF 增强解析”，上传的 PDF 才会使用已配置的增强解析服务；未勾选时仍使用 FastGPT 内置解析器。
## Admin 额外变量 ## Admin 额外变量
......
document/content/self-host/upgrading/4-15/4155.en.mdx
...@@ -8,6 +8,7 @@ description: 'FastGPT V4.15.5 Release Notes' ...@@ -8,6 +8,7 @@ description: 'FastGPT V4.15.5 Release Notes'
## 🚀 New Features ## 🚀 New Features
1. Added Cloudflare R2 object storage support, including the R2 S3 API, presigned access, and custom domains for public buckets. 1. Added Cloudflare R2 object storage support, including the R2 S3 API, presigned access, and custom domains for public buckets.
2. Added the SoMark PDF enhanced parsing provider. Configure it with `SOMARK_API_KEY`; when multiple PDF providers are configured, FastGPT uses this priority order: custom PDF parsing service, SoMark, TextIn, then Doc2x. See the [environment variable configuration](../../config/env) for details.
## ⚙️ Improvements ## ⚙️ Improvements
...@@ -19,6 +20,9 @@ description: 'FastGPT V4.15.5 Release Notes' ...@@ -19,6 +20,9 @@ description: 'FastGPT V4.15.5 Release Notes'
## 🐛 Fixes ## 🐛 Fixes
1. Fixed Alibaba Cloud OSS metadata reads that looked for ETags in the wrong response field, causing missing ETags and downstream metadata validation failures. 1. Fixed Alibaba Cloud OSS metadata reads that looked for ETags in the wrong response field, causing missing ETags and downstream metadata validation failures.
2. Fixed missing S3/MinIO source files being surfaced as `Unknown`; they now return a translated file-not-found error with HTTP 404.
3. Fixed input fields disappearing from older plugin nodes after an upgrade.
4. Fixed avatar URLs being encoded twice.
## 🛠️ Code Improvements ## 🛠️ Code Improvements
......
document/content/self-host/upgrading/4-15/4155.mdx
...@@ -8,6 +8,7 @@ description: 'FastGPT V4.15.5 更新说明' ...@@ -8,6 +8,7 @@ description: 'FastGPT V4.15.5 更新说明'
## 🚀 新增内容 ## 🚀 新增内容
1. 新增 Cloudflare R2 对象存储支持，兼容 R2 S3 API、预签名访问和公开 bucket 自定义域名。 1. 新增 Cloudflare R2 对象存储支持，兼容 R2 S3 API、预签名访问和公开 bucket 自定义域名。
2. 新增 SoMark PDF 增强解析提供商，支持通过 `SOMARK_API_KEY` 配置；多个 PDF 服务同时配置时，调用优先级为自定义 PDF 解析服务、SoMark、TextIn、Doc2x。具体配置见[环境变量配置](../../config/env)。
## ⚙️ 优化 ## ⚙️ 优化
...@@ -19,6 +20,9 @@ description: 'FastGPT V4.15.5 更新说明' ...@@ -19,6 +20,9 @@ description: 'FastGPT V4.15.5 更新说明'
## 🐛 修复 ## 🐛 修复
1. 修复阿里云 OSS `getObjectMetadata` 从错误字段读取 ETag，导致 ETag 缺失并触发下游元数据校验失败的问题。 1. 修复阿里云 OSS `getObjectMetadata` 从错误字段读取 ETag，导致 ETag 缺失并触发下游元数据校验失败的问题。
2. 修复 S3/MinIO 源文件不存在时 API 返回 `Unknown` 的问题，改为返回文件找不到并使用 HTTP 404。
3. 修复旧插件节点升级后输入框消失的问题。
4. 修复头像 URL 被重复编码的问题。
## 🛠️ 代码优化 ## 🛠️ 代码优化
......
document/data/doc-last-modified.json
...@@ -167,16 +167,16 @@ ...@@ -167,16 +167,16 @@
"content/plugin/model-presets.mdx": "2026-06-04T16:10:15+08:00", "content/plugin/model-presets.mdx": "2026-06-04T16:10:15+08:00",
"content/plugin/system-tool-development.en.mdx": "2026-07-02T11:54:55+08:00", "content/plugin/system-tool-development.en.mdx": "2026-07-02T11:54:55+08:00",
"content/plugin/system-tool-development.mdx": "2026-07-02T11:54:55+08:00", "content/plugin/system-tool-development.mdx": "2026-07-02T11:54:55+08:00",
"content/self-host/config/env.en.mdx": "2026-07-26T21:10:56+08:00", "content/self-host/config/env.en.mdx": "2026-07-29T14:21:55+08:00",
"content/self-host/config/env.mdx": "2026-07-26T21:10:56+08:00", "content/self-host/config/env.mdx": "2026-07-29T14:21:55+08:00",
"content/self-host/config/model/intro.en.mdx": "2026-06-04T16:10:15+08:00", "content/self-host/config/model/intro.en.mdx": "2026-06-04T16:10:15+08:00",
"content/self-host/config/model/intro.mdx": "2026-06-04T16:10:15+08:00", "content/self-host/config/model/intro.mdx": "2026-06-04T16:10:15+08:00",
"content/self-host/config/model/minimax.en.mdx": "2026-06-03T10:40:17+08:00", "content/self-host/config/model/minimax.en.mdx": "2026-06-03T10:40:17+08:00",
"content/self-host/config/model/minimax.mdx": "2026-06-03T10:40:17+08:00", "content/self-host/config/model/minimax.mdx": "2026-06-03T10:40:17+08:00",
"content/self-host/config/model/siliconCloud.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/config/model/siliconCloud.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/config/model/siliconCloud.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/config/model/siliconCloud.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/config/object-storage.en.mdx": "2026-07-26T21:10:56+08:00", "content/self-host/config/object-storage.en.mdx": "2026-07-26T21:56:28+08:00",
"content/self-host/config/object-storage.mdx": "2026-07-26T21:10:56+08:00", "content/self-host/config/object-storage.mdx": "2026-07-26T21:56:28+08:00",
"content/self-host/config/remote-debug-suite.en.mdx": "2026-06-27T22:05:51+08:00", "content/self-host/config/remote-debug-suite.en.mdx": "2026-06-27T22:05:51+08:00",
"content/self-host/config/remote-debug-suite.mdx": "2026-06-27T22:05:51+08:00", "content/self-host/config/remote-debug-suite.mdx": "2026-06-27T22:05:51+08:00",
"content/self-host/config/sandbox/common.en.mdx": "2026-07-02T15:38:53+08:00", "content/self-host/config/sandbox/common.en.mdx": "2026-07-02T15:38:53+08:00",
...@@ -327,8 +327,8 @@ ...@@ -327,8 +327,8 @@
"content/self-host/upgrading/4-15/4153.mdx": "2026-07-18T12:51:59+08:00", "content/self-host/upgrading/4-15/4153.mdx": "2026-07-18T12:51:59+08:00",
"content/self-host/upgrading/4-15/4154.en.mdx": "2026-07-25T00:27:20+08:00", "content/self-host/upgrading/4-15/4154.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/4154.mdx": "2026-07-25T00:27:20+08:00", "content/self-host/upgrading/4-15/4154.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/4155.en.mdx": "2026-07-26T21:10:56+08:00", "content/self-host/upgrading/4-15/4155.en.mdx": "2026-07-29T14:37:16+08:00",
"content/self-host/upgrading/4-15/4155.mdx": "2026-07-26T21:10:56+08:00", "content/self-host/upgrading/4-15/4155.mdx": "2026-07-29T14:37:16+08:00",
"content/self-host/upgrading/outdated/40.en.mdx": "2026-07-25T00:27:20+08:00", "content/self-host/upgrading/outdated/40.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/40.mdx": "2026-07-25T00:27:20+08:00", "content/self-host/upgrading/outdated/40.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/41.en.mdx": "2026-07-25T00:27:20+08:00", "content/self-host/upgrading/outdated/41.en.mdx": "2026-07-25T00:27:20+08:00",
...@@ -469,6 +469,6 @@ ...@@ -469,6 +469,6 @@
"content/self-host/upgrading/outdated/499.mdx": "2026-05-07T15:06:40+08:00", "content/self-host/upgrading/outdated/499.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/upgrade-intruction.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/upgrade-intruction.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/upgrade-intruction.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/upgrade-intruction.mdx": "2026-04-26T21:08:47+08:00",
"content/toc.en.mdx": "2026-07-26T18:37:23+08:00", "content/toc.en.mdx": "2026-07-26T21:56:28+08:00",
"content/toc.mdx": "2026-07-26T18:37:23+08:00" "content/toc.mdx": "2026-07-26T21:56:28+08:00"
} }
\ No newline at end of file
document/public/deploy/docker/main/cn/docker-compose.milvus.yml
...@@ -279,6 +279,8 @@ services: ...@@ -279,6 +279,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/cn/docker-compose.oceanbase.yml
...@@ -257,6 +257,8 @@ services: ...@@ -257,6 +257,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/cn/docker-compose.opengauss.yml
...@@ -241,6 +241,8 @@ services: ...@@ -241,6 +241,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/cn/docker-compose.pg.yml
...@@ -239,6 +239,8 @@ services: ...@@ -239,6 +239,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/cn/docker-compose.seekdb.yml
...@@ -244,6 +244,8 @@ services: ...@@ -244,6 +244,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/cn/docker-compose.zilliz.yml
...@@ -220,6 +220,8 @@ services: ...@@ -220,6 +220,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/global/docker-compose.milvus.yml
...@@ -279,6 +279,8 @@ services: ...@@ -279,6 +279,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/global/docker-compose.oceanbase.yml
...@@ -257,6 +257,8 @@ services: ...@@ -257,6 +257,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/global/docker-compose.opengauss.yml
...@@ -241,6 +241,8 @@ services: ...@@ -241,6 +241,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/global/docker-compose.pg.yml
...@@ -239,6 +239,8 @@ services: ...@@ -239,6 +239,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/global/docker-compose.seekdb.yml
...@@ -244,6 +244,8 @@ services: ...@@ -244,6 +244,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
document/public/deploy/docker/main/global/docker-compose.zilliz.yml
...@@ -220,6 +220,8 @@ services: ...@@ -220,6 +220,8 @@ services:
# 自定义 PDF 解析服务配置 # 自定义 PDF 解析服务配置
CUSTOM_PDF_PARSE_URL: CUSTOM_PDF_PARSE_URL:
CUSTOM_PDF_PARSE_KEY: CUSTOM_PDF_PARSE_KEY:
# SoMark
SOMARK_API_KEY:
# Doc2x # Doc2x
DOC2X_KEY: DOC2X_KEY:
# 合合信息 # 合合信息
......
packages/global/common/error/code/common.ts
...@@ -24,7 +24,8 @@ const datasetErr = [ ...@@ -24,7 +24,8 @@ const datasetErr = [
}, },
{ {
statusText: CommonErrEnum.fileNotFound, statusText: CommonErrEnum.fileNotFound,
message: i18nT('common:error.fileNotFound') message: i18nT('common:error.fileNotFound'),
httpStatus: 404
}, },
{ {
statusText: CommonErrEnum.unAuthFile, statusText: CommonErrEnum.unAuthFile,
...@@ -54,7 +55,8 @@ export default datasetErr.reduce((acc, cur, index) => { ...@@ -54,7 +55,8 @@ export default datasetErr.reduce((acc, cur, index) => {
code: startCode + index, code: startCode + index,
statusText: cur.statusText, statusText: cur.statusText,
message: cur.message, message: cur.message,
data: null data: null,
...(cur.httpStatus !== undefined ? { httpStatus: cur.httpStatus } : {})
} }
}; };
}, {} as ErrType<`${CommonErrEnum}`>); }, {} as ErrType<`${CommonErrEnum}`>);
packages/global/common/system/types/index.ts
...@@ -216,6 +216,7 @@ export type customDomainType = { ...@@ -216,6 +216,7 @@ export type customDomainType = {
export type customPdfParseType = { export type customPdfParseType = {
url?: string; url?: string;
key?: string; key?: string;
somarkApiKey?: string;
doc2xKey?: string; doc2xKey?: string;
textinAppId?: string; textinAppId?: string;
textinSecretCode?: string; textinSecretCode?: string;
......
packages/service/common/file/read/utils.ts
...@@ -6,6 +6,7 @@ import { parseMarkdownBase64Images } from '@fastgpt/global/common/string/markdow ...@@ -6,6 +6,7 @@ import { parseMarkdownBase64Images } from '@fastgpt/global/common/string/markdow
import { createPdfParseUsage } from '../../../support/wallet/usage/controller'; import { createPdfParseUsage } from '../../../support/wallet/usage/controller';
import { useDoc2xServer } from '../../../thirdProvider/doc2x'; import { useDoc2xServer } from '../../../thirdProvider/doc2x';
import { useTextinServer } from '../../../thirdProvider/textin'; import { useTextinServer } from '../../../thirdProvider/textin';
import { useSomarkServer } from '../../../thirdProvider/somark';
import { readRawContentFromBuffer } from '../../../worker/function'; import { readRawContentFromBuffer } from '../../../worker/function';
import { getLogger, LogCategories } from '../../logger'; import { getLogger, LogCategories } from '../../logger';
import { getImageBuffer } from '../image/utils'; import { getImageBuffer } from '../image/utils';
...@@ -161,6 +162,25 @@ export const readFileContentByBuffer = async ({ ...@@ -161,6 +162,25 @@ export const readFileContentByBuffer = async ({
formatText: text formatText: text
}; };
}; };
const parsePdfFromSomark = async (): Promise<ReadFileResponse> => {
const apiKey = global.systemEnv.customPdfParse?.somarkApiKey;
if (!apiKey) return systemParse();
const { pages, text: rawText } = await useSomarkServer({ apiKey }).parsePDF(buffer);
const text = await parseMarkdownImages(rawText);
createPdfParseUsage({
teamId,
tmbId,
pages,
usageId
});
return {
rawText: text,
formatText: text
};
};
// Textin api // Textin api
const parsePdfFromTextin = async (): Promise<ReadFileResponse> => { const parsePdfFromTextin = async (): Promise<ReadFileResponse> => {
const appId = global.systemEnv.customPdfParse?.textinAppId; const appId = global.systemEnv.customPdfParse?.textinAppId;
...@@ -243,6 +263,7 @@ export const readFileContentByBuffer = async ({ ...@@ -243,6 +263,7 @@ export const readFileContentByBuffer = async ({
const pdfParseFn = async (): Promise<ReadFileResponse> => { const pdfParseFn = async (): Promise<ReadFileResponse> => {
if (!customPdfParse) return systemParse(); if (!customPdfParse) return systemParse();
if (global.systemEnv.customPdfParse?.url) return parsePdfFromCustomService(); if (global.systemEnv.customPdfParse?.url) return parsePdfFromCustomService();
if (global.systemEnv.customPdfParse?.somarkApiKey) return parsePdfFromSomark();
if (global.systemEnv.customPdfParse?.textinAppId) return parsePdfFromTextin(); if (global.systemEnv.customPdfParse?.textinAppId) return parsePdfFromTextin();
if (global.systemEnv.customPdfParse?.doc2xKey) return parsePdfFromDoc2x(); if (global.systemEnv.customPdfParse?.doc2xKey) return parsePdfFromDoc2x();
......
packages/service/common/response/index.ts
...@@ -43,6 +43,14 @@ function resolveHttpStatusForApiError( ...@@ -43,6 +43,14 @@ function resolveHttpStatusForApiError(
return bc; return bc;
} }
if (
typeof processedError.httpStatus === 'number' &&
processedError.httpStatus >= 400 &&
processedError.httpStatus <= 599
) {
return processedError.httpStatus;
}
// packages/global/common/error/code/s3.ts：510000 段为上传校验类客户端错误 // packages/global/common/error/code/s3.ts：510000 段为上传校验类客户端错误
if (typeof bc === 'number' && bc >= 510000 && bc < 511000) { if (typeof bc === 'number' && bc >= 510000 && bc < 511000) {
return 400; return 400;
......
packages/service/common/s3/buckets/base.ts
...@@ -10,6 +10,7 @@ import { ...@@ -10,6 +10,7 @@ import {
} from '../contracts/type'; } from '../contracts/type';
import { getSystemMaxFileSize } from '../config/constants'; import { getSystemMaxFileSize } from '../config/constants';
import { S3ErrEnum } from '@fastgpt/global/common/error/code/s3'; import { S3ErrEnum } from '@fastgpt/global/common/error/code/s3';
import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
import { createUploadConstraints } from '../utils/uploadConstraints'; import { createUploadConstraints } from '../utils/uploadConstraints';
import path from 'node:path'; import path from 'node:path';
import { MongoS3TTL } from '../models/ttl'; import { MongoS3TTL } from '../models/ttl';
...@@ -30,8 +31,18 @@ const logger = getLogger(LogCategories.INFRA.S3); ...@@ -30,8 +31,18 @@ const logger = getLogger(LogCategories.INFRA.S3);
type IStorage = ReturnType<typeof createStorage>; type IStorage = ReturnType<typeof createStorage>;
// Check if the error is a "file not found" type error, which should be treated as success // Check whether a storage error means that the requested file does not exist.
export const isFileNotFoundError = (error: any): boolean => { export const isFileNotFoundError = (error: any): boolean => {
if (
error?.statusCode === 404 ||
error?.status === 404 ||
error?.$metadata?.httpStatusCode === 404 ||
error?.code === 'NotFound' ||
error?.name === 'NotFound'
) {
return true;
}
if (error instanceof S3Error) { if (error instanceof S3Error) {
// Handle various "not found" error codes // Handle various "not found" error codes
return ( return (
...@@ -339,7 +350,12 @@ export class S3BaseBucket { ...@@ -339,7 +350,12 @@ export class S3BaseBucket {
} }
async getFileMetadata(key: string) { async getFileMetadata(key: string) {
const metadataResponse = await this.client.getObjectMetadata({ key }); const metadataResponse = await this.client.getObjectMetadata({ key }).catch((error) => {
if (isFileNotFoundError(error)) {
throw CommonErrEnum.fileNotFound;
}
throw error;
});
if (!metadataResponse) return; if (!metadataResponse) return;
const contentLength = metadataResponse.contentLength; const contentLength = metadataResponse.contentLength;
......
packages/service/common/system/tools.ts
...@@ -22,6 +22,7 @@ export const initFastGPTConfig = (config?: FastGPTConfigFileType) => { ...@@ -22,6 +22,7 @@ export const initFastGPTConfig = (config?: FastGPTConfigFileType) => {
// Special config computed // Special config computed
config.feConfigs.showCustomPdfParse = config.feConfigs.showCustomPdfParse =
!!config.systemEnv.customPdfParse?.url || !!config.systemEnv.customPdfParse?.url ||
!!config.systemEnv.customPdfParse?.somarkApiKey ||
!!config.systemEnv.customPdfParse?.textinAppId || !!config.systemEnv.customPdfParse?.textinAppId ||
!!config.systemEnv.customPdfParse?.doc2xKey; !!config.systemEnv.customPdfParse?.doc2xKey;
config.feConfigs.customPdfParsePrice = config.systemEnv.customPdfParse?.price || 0; config.feConfigs.customPdfParsePrice = config.systemEnv.customPdfParse?.price || 0;
......
packages/service/env.ts
...@@ -118,6 +118,9 @@ export const serviceEnv = createEnv({ ...@@ -118,6 +118,9 @@ export const serviceEnv = createEnv({
CUSTOM_PDF_PARSE_KEY: z.string().optional().meta({ CUSTOM_PDF_PARSE_KEY: z.string().optional().meta({
description: '自定义 PDF 解析服务密钥' description: '自定义 PDF 解析服务密钥'
}), }),
SOMARK_API_KEY: z.string().optional().meta({
description: 'SoMark PDF 解析服务密钥'
}),
DOC2X_KEY: z.string().optional().meta({ DOC2X_KEY: z.string().optional().meta({
description: 'Doc2x PDF 解析服务密钥' description: 'Doc2x PDF 解析服务密钥'
}), }),
......
packages/service/test/common/file/read/utils.test.ts
...@@ -7,6 +7,7 @@ import path from 'path'; ...@@ -7,6 +7,7 @@ import path from 'path';
const { const {
mockReadRawContentFromBuffer, mockReadRawContentFromBuffer,
mockAxiosPost, mockAxiosPost,
mockSomarkParsePDF,
mockDoc2xParsePDF, mockDoc2xParsePDF,
mockTextinParsePDF, mockTextinParsePDF,
mockUploadImage2S3Bucket, mockUploadImage2S3Bucket,
...@@ -25,6 +26,10 @@ const { ...@@ -25,6 +26,10 @@ const {
}; };
}), }),
mockAxiosPost: vi.fn(), mockAxiosPost: vi.fn(),
mockSomarkParsePDF: vi.fn().mockResolvedValue({
pages: 2,
text: 'somark-parsed-text'
}),
mockDoc2xParsePDF: vi.fn().mockResolvedValue({ mockDoc2xParsePDF: vi.fn().mockResolvedValue({
pages: 1, pages: 1,
text: 'doc2x-parsed-text' text: 'doc2x-parsed-text'
...@@ -57,6 +62,12 @@ vi.mock('@fastgpt/service/thirdProvider/doc2x', () => ({ ...@@ -57,6 +62,12 @@ vi.mock('@fastgpt/service/thirdProvider/doc2x', () => ({
})) }))
})); }));
vi.mock('@fastgpt/service/thirdProvider/somark', () => ({
useSomarkServer: vi.fn(() => ({
parsePDF: mockSomarkParsePDF
}))
}));
vi.mock('@fastgpt/service/thirdProvider/textin', () => ({ vi.mock('@fastgpt/service/thirdProvider/textin', () => ({
useTextinServer: vi.fn(() => ({ useTextinServer: vi.fn(() => ({
parsePDF: mockTextinParsePDF parsePDF: mockTextinParsePDF
...@@ -130,6 +141,7 @@ describe('readRawTextByLocalFile', () => { ...@@ -130,6 +141,7 @@ describe('readRawTextByLocalFile', () => {
describe('readFileContentByBuffer', () => { describe('readFileContentByBuffer', () => {
beforeEach(() => { beforeEach(() => {
vi.clearAllMocks();
global.systemEnv = {} as any; global.systemEnv = {} as any;
}); });
...@@ -337,6 +349,115 @@ describe('readFileContentByBuffer', () => { ...@@ -337,6 +349,115 @@ describe('readFileContentByBuffer', () => {
expect(result.rawText).toBe('textin-parsed-text'); expect(result.rawText).toBe('textin-parsed-text');
}); });
it('should use SoMark service for pdf when somarkApiKey is configured', async () => {
global.systemEnv = {
customPdfParse: { somarkApiKey: 'sk-test' }
} as any;
const result = await readFileContentByBuffer({
teamId,
tmbId,
extension: 'pdf',
buffer: Buffer.from('pdf content'),
encoding: 'utf-8',
customPdfParse: true
});
expect(mockSomarkParsePDF).toHaveBeenCalledWith(Buffer.from('pdf content'));
expect(result.rawText).toBe('somark-parsed-text');
});
it('should upload SoMark markdown images with the shared image handler', async () => {
global.systemEnv = {
customPdfParse: { somarkApiKey: 'sk-test' }
} as any;
mockSomarkParsePDF.mockResolvedValueOnce({
pages: 2,
text: 'image ![img](https://somark.ai/image.png)'
});
const result = await readFileContentByBuffer({
teamId,
tmbId,
extension: 'pdf',
buffer: Buffer.from('pdf content'),
encoding: 'utf-8',
customPdfParse: true,
imageKeyOptions: {
prefix: 'dataset/ds1/file-parsed'
}
});
expect(mockGetImageBuffer).toHaveBeenCalledWith('https://somark.ai/image.png');
expect(result.rawText).toContain('https://s3.example.com/uploaded-image.png');
});
it('should prefer custom URL, SoMark, Textin, and Doc2x in that order', async () => {
global.systemEnv = {
customPdfParse: {
url: 'http://custom-pdf-service.com/parse',
key: 'custom-key',
somarkApiKey: 'sk-test',
textinAppId: 'app-id',
textinSecretCode: 'secret',
doc2xKey: 'doc2x-key'
}
} as any;
mockAxiosPost.mockResolvedValueOnce({
data: {
pages: 1,
markdown: 'custom-service-result'
}
});
const customResult = await readFileContentByBuffer({
teamId,
tmbId,
extension: 'pdf',
buffer: Buffer.from('pdf content'),
encoding: 'utf-8',
customPdfParse: true
});
expect(customResult.rawText).toBe('custom-service-result');
expect(mockSomarkParsePDF).not.toHaveBeenCalled();
const providerConfig = global.systemEnv.customPdfParse!;
providerConfig.url = undefined;
const somarkResult = await readFileContentByBuffer({
teamId,
tmbId,
extension: 'pdf',
buffer: Buffer.from('pdf content'),
encoding: 'utf-8',
customPdfParse: true
});
expect(somarkResult.rawText).toBe('somark-parsed-text');
expect(mockTextinParsePDF).not.toHaveBeenCalled();
providerConfig.somarkApiKey = undefined;
const textinResult = await readFileContentByBuffer({
teamId,
tmbId,
extension: 'pdf',
buffer: Buffer.from('pdf content'),
encoding: 'utf-8',
customPdfParse: true
});
expect(textinResult.rawText).toBe('textin-parsed-text');
expect(mockDoc2xParsePDF).not.toHaveBeenCalled();
providerConfig.textinAppId = undefined;
const doc2xResult = await readFileContentByBuffer({
teamId,
tmbId,
extension: 'pdf',
buffer: Buffer.from('pdf content'),
encoding: 'utf-8',
customPdfParse: true
});
expect(doc2xResult.rawText).toBe('doc2x-parsed-text');
});
it('should pass Textin image upload handler when imageKeyOptions is provided', async () => { it('should pass Textin image upload handler when imageKeyOptions is provided', async () => {
global.systemEnv = { global.systemEnv = {
customPdfParse: { textinAppId: 'app-id', textinSecretCode: 'secret' } customPdfParse: { textinAppId: 'app-id', textinSecretCode: 'secret' }
......
packages/service/test/common/response/index.test.ts
...@@ -3,6 +3,7 @@ import { z } from 'zod'; ...@@ -3,6 +3,7 @@ import { z } from 'zod';
import { ApiRequestInputParseError } from '../../../common/zod/requestParseError'; import { ApiRequestInputParseError } from '../../../common/zod/requestParseError';
import { UserError } from '@fastgpt/global/common/error/utils'; import { UserError } from '@fastgpt/global/common/error/utils';
import { ERROR_ENUM, ERROR_RESPONSE } from '@fastgpt/global/common/error/errorCode'; import { ERROR_ENUM, ERROR_RESPONSE } from '@fastgpt/global/common/error/errorCode';
import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
vi.unmock('@fastgpt/service/common/response'); vi.unmock('@fastgpt/service/common/response');
...@@ -85,6 +86,17 @@ describe('processError zod logging', () => { ...@@ -85,6 +86,17 @@ describe('processError zod logging', () => {
}); });
}); });
describe('processError HTTP status mapping', () => {
it('maps a missing file to HTTP 404', () => {
const processed = processError({
error: CommonErrEnum.fileNotFound
});
expect(processed.code).toBe(507002);
expect(processed.httpStatus).toBe(404);
});
});
describe('getSseErrorResponse logging', () => { describe('getSseErrorResponse logging', () => {
beforeEach(() => { beforeEach(() => {
logger.info.mockClear(); logger.info.mockClear();
......
packages/service/test/common/s3/buckets/base.test.ts 0 → 100644
import { describe, expect, it, vi } from 'vitest';
import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
vi.unmock('@fastgpt/service/common/s3/buckets/base');
const { S3BaseBucket, isFileNotFoundError } =
await import('@fastgpt/service/common/s3/buckets/base');
describe('S3BaseBucket file-not-found handling', () => {
it.each([
{ name: 'AWS metadata status', error: { $metadata: { httpStatusCode: 404 } } },
{ name: 'AWS error code', error: { code: 'NotFound' } },
{ name: 'HTTP status', error: { statusCode: 404 } }
])('recognizes $name as a missing file', ({ error }) => {
expect(isFileNotFoundError(error)).toBe(true);
});
it('converts a missing object metadata error to fileNotFound', async () => {
const getObjectMetadata = vi.fn().mockRejectedValue({
$metadata: { httpStatusCode: 404 }
});
const bucket = new S3BaseBucket({ getObjectMetadata } as any, undefined);
await expect(bucket.getFileMetadata('dataset/file.pdf')).rejects.toBe(
CommonErrEnum.fileNotFound
);
});
it('preserves non-not-found storage errors', async () => {
const error = new Error('storage unavailable');
const getObjectMetadata = vi.fn().mockRejectedValue(error);
const bucket = new S3BaseBucket({ getObjectMetadata } as any, undefined);
await expect(bucket.getFileMetadata('dataset/file.pdf')).rejects.toBe(error);
});
});
packages/service/test/common/system/tools.test.ts 0 → 100644
import { beforeEach, describe, expect, it, vi } from 'vitest';
vi.mock('@fastgpt/service/env', () => ({
hasAgentSandboxConfig: vi.fn(() => false),
serviceEnv: {
UPLOAD_FILE_MAX_SIZE: 1000,
UPLOAD_FILE_MAX_AMOUNT: 1000,
MAX_FOLDER_DEPTH: 4
}
}));
vi.mock('@fastgpt/service/core/ai/sandbox/interface/config', () => ({
getAgentSandboxArchiveMaxBytes: vi.fn(() => 1024),
getAgentSandboxMaxFileBytes: vi.fn(() => 512),
getAgentSandboxSkillMaxBytes: vi.fn(() => 512)
}));
const { initFastGPTConfig } = await import('@fastgpt/service/common/system/tools');
describe('initFastGPTConfig', () => {
beforeEach(() => {
global.feConfigs = {} as any;
global.systemEnv = {} as any;
});
it('只配置 SoMark 时开启 PDF 增强解析入口', () => {
initFastGPTConfig({
feConfigs: {},
systemEnv: {
customPdfParse: {
somarkApiKey: 'sk-test'
}
}
} as any);
expect(global.feConfigs.showCustomPdfParse).toBe(true);
expect(global.systemEnv.customPdfParse?.somarkApiKey).toBe('sk-test');
});
});
packages/service/test/env.test.ts
...@@ -12,6 +12,7 @@ const originalEnv = { ...@@ -12,6 +12,7 @@ const originalEnv = {
SYNC_INDEX: process.env.SYNC_INDEX, SYNC_INDEX: process.env.SYNC_INDEX,
AES256_SECRET_KEY: process.env.AES256_SECRET_KEY, AES256_SECRET_KEY: process.env.AES256_SECRET_KEY,
INVOKE_TOKEN_SECRET: process.env.INVOKE_TOKEN_SECRET, INVOKE_TOKEN_SECRET: process.env.INVOKE_TOKEN_SECRET,
SOMARK_API_KEY: process.env.SOMARK_API_KEY,
PRO_URL: process.env.PRO_URL, PRO_URL: process.env.PRO_URL,
PRO_TOKEN: process.env.PRO_TOKEN, PRO_TOKEN: process.env.PRO_TOKEN,
VITEST: process.env.VITEST, VITEST: process.env.VITEST,
...@@ -42,6 +43,7 @@ describe('serviceEnv', () => { ...@@ -42,6 +43,7 @@ describe('serviceEnv', () => {
vi.stubEnv('SYNC_INDEX', originalEnv.SYNC_INDEX); vi.stubEnv('SYNC_INDEX', originalEnv.SYNC_INDEX);
vi.stubEnv('AES256_SECRET_KEY', originalEnv.AES256_SECRET_KEY); vi.stubEnv('AES256_SECRET_KEY', originalEnv.AES256_SECRET_KEY);
vi.stubEnv('INVOKE_TOKEN_SECRET', originalEnv.INVOKE_TOKEN_SECRET); vi.stubEnv('INVOKE_TOKEN_SECRET', originalEnv.INVOKE_TOKEN_SECRET);
vi.stubEnv('SOMARK_API_KEY', originalEnv.SOMARK_API_KEY);
vi.stubEnv('PRO_URL', originalEnv.PRO_URL); vi.stubEnv('PRO_URL', originalEnv.PRO_URL);
vi.stubEnv('PRO_TOKEN', originalEnv.PRO_TOKEN); vi.stubEnv('PRO_TOKEN', originalEnv.PRO_TOKEN);
vi.stubEnv('VITEST', originalEnv.VITEST); vi.stubEnv('VITEST', originalEnv.VITEST);
...@@ -71,6 +73,19 @@ describe('serviceEnv', () => { ...@@ -71,6 +73,19 @@ describe('serviceEnv', () => {
}); });
}); });
it('reads the optional SoMark API key', async () => {
vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey');
vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret');
vi.stubEnv('INVOKE_TOKEN_SECRET', validInvokeTokenSecret);
vi.stubEnv('SOMARK_API_KEY', 'sk-somark-test');
await expect(importServiceEnv()).resolves.toMatchObject({
serviceEnv: {
SOMARK_API_KEY: 'sk-somark-test'
}
});
});
it('validates SYSTEM_MAX_STRING_LENGTH_M during service env init', async () => { it('validates SYSTEM_MAX_STRING_LENGTH_M during service env init', async () => {
vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey'); vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey');
vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret'); vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret');
......
packages/service/test/thirdProvider/somark.test.ts 0 → 100644
import FormData from 'form-data';
import { beforeEach, describe, expect, it, vi } from 'vitest';
const { postMock } = vi.hoisted(() => ({
postMock: vi.fn()
}));
vi.mock('@fastgpt/service/common/api/axios', () => ({
createProxyAxios: vi.fn(() => ({
post: postMock
}))
}));
const { useSomarkServer } = await import('@fastgpt/service/thirdProvider/somark');
const mockSomarkSuccess = () => {
postMock.mockResolvedValueOnce({
data: {
code: 0,
message: 'success',
data: {
metadata: {
page_num: 3
},
result: {
outputs: {
markdown: '# Parsed document'
}
}
}
}
});
};
describe('useSomarkServer', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('通过同步接口解析 PDF 并返回 Markdown 与页数', async () => {
mockSomarkSuccess();
const result = await useSomarkServer({ apiKey: 'sk-test' }).parsePDF(
Buffer.from('pdf-content')
);
expect(result).toEqual({
pages: 3,
text: '# Parsed document'
});
expect(postMock).toHaveBeenCalledWith(
'/parse/sync',
expect.any(FormData),
expect.objectContaining({
headers: expect.objectContaining({
'content-type': expect.stringContaining('multipart/form-data')
})
})
);
const form = postMock.mock.calls[0][1] as FormData;
const body = form.getBuffer().toString();
expect(body).toContain('name="file"; filename="file.pdf"');
expect(body).toContain('Content-Type: application/pdf');
expect(body).toContain('name="api_key"');
expect(body).toContain('sk-test');
expect(body).toContain('name="output_formats"');
expect(body).toContain('markdown');
expect(body).toContain('"image":"url"');
expect(body).toContain('"formula":"latex"');
expect(body).toContain('"table":"markdown"');
});
it('业务状态码非零时返回带 SoMark 前缀的错误', async () => {
postMock.mockResolvedValueOnce({
data: {
code: 1107,
message: 'Invalid API Key'
}
});
await expect(
useSomarkServer({ apiKey: 'invalid-key' }).parsePDF(Buffer.from('pdf-content'))
).rejects.toThrow('[SoMark] Invalid API Key');
});
it('成功响应缺少 Markdown 时拒绝结果', async () => {
postMock.mockResolvedValueOnce({
data: {
code: 0,
data: {
metadata: {
page_num: 1
},
result: {
outputs: {}
}
}
}
});
await expect(
useSomarkServer({ apiKey: 'sk-test' }).parsePDF(Buffer.from('pdf-content'))
).rejects.toThrow('[SoMark] No markdown content in response');
});
it('成功响应页数无效时拒绝结果', async () => {
postMock.mockResolvedValueOnce({
data: {
code: 0,
data: {
metadata: {
page_num: 0
},
result: {
outputs: {
markdown: '# Parsed document'
}
}
}
}
});
await expect(
useSomarkServer({ apiKey: 'sk-test' }).parsePDF(Buffer.from('pdf-content'))
).rejects.toThrow('[SoMark] Invalid page count in response');
});
it('HTTP 请求失败时规范化错误信息', async () => {
postMock.mockRejectedValueOnce(new Error('network unavailable'));
await expect(
useSomarkServer({ apiKey: 'sk-test' }).parsePDF(Buffer.from('pdf-content'))
).rejects.toThrow('[SoMark] network unavailable');
});
});
packages/service/thirdProvider/somark/index.ts 0 → 100644
import FormData from 'form-data';
import { getErrText } from '@fastgpt/global/common/error/utils';
import { createProxyAxios } from '../../common/api/axios';
import { getLogger, LogCategories } from '../../common/logger';
type SomarkResponse = {
code?: number;
message?: string;
data?: {
metadata?: {
page_num?: number;
};
result?: {
outputs?: {
markdown?: string;
};
};
};
};
/**
* 创建 SoMark PDF 解析客户端。
*
* SoMark 的 API Key 通过 multipart body 传递，不使用 Authorization header。
* 当前只请求 Markdown，图片保留为 URL，交由上层统一转存。
*/
export const useSomarkServer = ({ apiKey }: { apiKey: string }) => {
const logger = getLogger(LogCategories.MODULE.DATASET.FILE);
const instance = createProxyAxios({
baseURL: 'https://somark.ai/api/v1',
timeout: 600000
});
const parsePDF = async (fileBuffer: Buffer) => {
const startTime = Date.now();
const form = new FormData();
form.append('file', fileBuffer, {
filename: 'file.pdf',
contentType: 'application/pdf'
});
form.append('api_key', apiKey);
form.append('output_formats', 'markdown');
form.append(
'element_formats',
JSON.stringify({
image: 'url',
formula: 'latex',
table: 'markdown',
cs: 'image'
})
);
try {
const { data } = await instance.post<SomarkResponse>('/parse/sync', form, {
headers: form.getHeaders()
});
if (data?.code !== 0) {
return Promise.reject(
new Error(`[SoMark] ${data?.message || `API error: ${data?.code ?? 'Unknown error'}`}`)
);
}
const text = data.data?.result?.outputs?.markdown;
if (!text) {
return Promise.reject(new Error('[SoMark] No markdown content in response'));
}
const pages = data.data?.metadata?.page_num;
if (typeof pages !== 'number' || !Number.isInteger(pages) || pages < 1) {
return Promise.reject(new Error('[SoMark] Invalid page count in response'));
}
logger.debug('SoMark PDF parse finished', {
durationMs: Date.now() - startTime,
pages
});
return {
pages,
text
};
} catch (error) {
if (error instanceof Error && error.message.startsWith('[SoMark]')) {
throw error;
}
return Promise.reject(new Error(`[SoMark] ${getErrText(error)}`));
}
};
return {
parsePDF
};
};
pro @ d3dfe1da
Subproject commit 81b75623c6d6accdfc42f3b3df4c6b00fa023fc9 Subproject commit d3dfe1daa4fd4c290bdb29e90a83b472f2769c03
projects/app/.env.template
...@@ -251,6 +251,8 @@ MAX_HTML_TRANSFORM_CHARS=1000000 ...@@ -251,6 +251,8 @@ MAX_HTML_TRANSFORM_CHARS=1000000
# CUSTOM_PDF_PARSE_URL= # CUSTOM_PDF_PARSE_URL=
# 自定义 PDF 解析服务密钥 # 自定义 PDF 解析服务密钥
# CUSTOM_PDF_PARSE_KEY= # CUSTOM_PDF_PARSE_KEY=
# SoMark PDF 解析服务密钥
# SOMARK_API_KEY=
# Doc2x PDF 解析服务密钥 # Doc2x PDF 解析服务密钥
# DOC2X_KEY= # DOC2X_KEY=
# 合合信息 Textin 服务 App ID # 合合信息 Textin 服务 App ID
......
projects/app/src/service/common/system/index.ts
...@@ -177,6 +177,7 @@ export async function initSystemConfig() { ...@@ -177,6 +177,7 @@ export async function initSystemConfig() {
customPdfParse: { customPdfParse: {
url: serviceEnv.CUSTOM_PDF_PARSE_URL, url: serviceEnv.CUSTOM_PDF_PARSE_URL,
key: serviceEnv.CUSTOM_PDF_PARSE_KEY, key: serviceEnv.CUSTOM_PDF_PARSE_KEY,
somarkApiKey: serviceEnv.SOMARK_API_KEY,
doc2xKey: serviceEnv.DOC2X_KEY, doc2xKey: serviceEnv.DOC2X_KEY,
textinAppId: serviceEnv.TEXTIN_APP_ID, textinAppId: serviceEnv.TEXTIN_APP_ID,
textinSecretCode: serviceEnv.TEXTIN_SECRET_CODE textinSecretCode: serviceEnv.TEXTIN_SECRET_CODE
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment