Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 94134b04 by Xianquan Committed by GitHub
Options

refactor(s3): remove presigned download mode (#7321) 

* refactor(s3): remove presigned download mode

* fix: resolve app and pro build checks
parent 43ee4ec9
Showing with 125 additions and 156 deletions
.agents/design/common/s3/s3-storage-refactor.md
...@@ -7,7 +7,7 @@ ...@@ -7,7 +7,7 @@
1. 用短链替代模型可见的长 JWT 上传、下载链接。 1. 用短链替代模型可见的长 JWT 上传、下载链接。
2. 将上传校验拆成 `policy + hint + evidence`，支持无后缀文件和显式自定义扩展名。 2. 将上传校验拆成 `policy + hint + evidence`，支持无后缀文件和显式自定义扩展名。
3. ChatBox 删除上传占位时真正中止预签和 PUT 请求，并阻止异步结果回写。 3. ChatBox 删除上传占位时真正中止预签和 PUT 请求，并阻止异步结果回写。
4. 支持 `short-proxy`、`short-redirect`、`presigned` 三种下载模式。 4. 支持 `short-proxy`、`short-redirect` 两种下载模式，默认使用 `short-proxy`。
5. 抽取 `@fastgpt-sdk/storage/access-link`，让协议状态机与 FastGPT/Mongo 适配解耦。 5. 抽取 `@fastgpt-sdk/storage/access-link`，让协议状态机与 FastGPT/Mongo 适配解耦。
6. 保持文件对象、临时 TTL、聊天删除和短链记录的原有清理语义。 6. 保持文件对象、临时 TTL、聊天删除和短链记录的原有清理语义。
...@@ -98,10 +98,11 @@ Mongo 是跨实例一致性的必要状态，不使用进程内缓存。性能 ...@@ -98,10 +98,11 @@ Mongo 是跨实例一致性的必要状态，不使用进程内缓存。性能
| --- | --- | --- | --- | | --- | --- | --- | --- |
| `short-proxy` | FastGPT 短链 | App 代理对象流 | 默认；私有 S3、最稳定 | | `short-proxy` | FastGPT 短链 | App 代理对象流 | 默认；私有 S3、最稳定 |
| `short-redirect` | FastGPT 短链 | App 校验后 302，客户端直连 S3/CDN | 有外部 endpoint，降低 App 带宽 | | `short-redirect` | FastGPT 短链 | App 校验后 302，客户端直连 S3/CDN | 有外部 endpoint，降低 App 带宽 |
| `presigned` | S3 长链接 | 客户端直连 S3/CDN | 仅显式兼容旧行为 |
`short-redirect` 必须先完成短链校验，再生成不超过短链剩余寿命的临时 presigned URL。已经 302 给客户端的 URL 在其短 TTL 内无法被 alias revoke 立即收回，这是该模式的明确代价。 `short-redirect` 必须先完成短链校验，再生成不超过短链剩余寿命的临时 presigned URL。已经 302 给客户端的 URL 在其短 TTL 内无法被 alias revoke 立即收回，这是该模式的明确代价。
业务接口不再接受单次下载模式覆盖，`createExternalUrl()` 始终返回短链；传输方式只由全局 `STORAGE_DOWNLOAD_URL_MODE` 决定。底层 presigned 能力仅用于 `short-redirect` 的临时 `Location` 和服务端内部预览，不再作为对外下载模式。
环境变量： 环境变量：
| 变量 | 说明 | | 变量 | 说明 |
...@@ -111,7 +112,7 @@ Mongo 是跨实例一致性的必要状态，不使用进程内缓存。性能 ...@@ -111,7 +112,7 @@ Mongo 是跨实例一致性的必要状态，不使用进程内缓存。性能
| `STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS` | 302 临时 S3 URL TTL，默认 300 秒 | | `STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS` | 302 临时 S3 URL TTL，默认 300 秒 |
| `FILE_DOMAIN` | 完整 FastGPT 文件 API 域名，同时影响上传和下载 URL | | `FILE_DOMAIN` | 完整 FastGPT 文件 API 域名，同时影响上传和下载 URL |
| `FILE_DOWNLOAD_PUBLIC_URL_PREFIX` | 可选下载短链公开前缀，只影响下载 | | `FILE_DOWNLOAD_PUBLIC_URL_PREFIX` | 可选下载短链公开前缀，只影响下载 |
| `STORAGE_EXTERNAL_ENDPOINT` / `STORAGE_S3_CDN_ENDPOINT` | redirect/presigned 使用的客户端可访问地址 | | `STORAGE_EXTERNAL_ENDPOINT` / `STORAGE_S3_CDN_ENDPOINT` | `short-redirect` 使用的客户端可访问地址 |
使用 nginx 缩短公开下载路径时，可配置： 使用 nginx 缩短公开下载路径时，可配置：
...@@ -206,7 +207,7 @@ Route 必须捕获可预期文件代理错误并映射状态码，不能全部 ...@@ -206,7 +207,7 @@ Route 必须捕获可预期文件代理错误并映射状态码，不能全部
| `sdk/storage/src/access-link/` | 短链协议、crypto、日期分桶、store ports、状态机 | | `sdk/storage/src/access-link/` | 短链协议、crypto、日期分桶、store ports、状态机 |
| `packages/service/common/s3/accessLink/` | Mongo adapter、schema、URL、日志和 service 实例 | | `packages/service/common/s3/accessLink/` | Mongo adapter、schema、URL、日志和 service 实例 |
| `packages/service/common/s3/uploadPolicy/` | policy/hint/evidence 类型与最终裁决 | | `packages/service/common/s3/uploadPolicy/` | policy/hint/evidence 类型与最终裁决 |
| `packages/service/common/s3/buckets/base.ts` | 统一签发上传短链和三种下载 URL | | `packages/service/common/s3/buckets/base.ts` | 统一签发上传和下载短链 |
| `projects/app/src/pages/api/system/file/u/` | 新上传短链入口 | | `projects/app/src/pages/api/system/file/u/` | 新上传短链入口 |
| `projects/app/src/pages/api/system/file/d/` | 新下载短链入口和 proxy/redirect 分流 | | `projects/app/src/pages/api/system/file/d/` | 新下载短链入口和 proxy/redirect 分流 |
| `projects/app/src/service/common/s3/proxy.ts` | 上传/下载代理、流释放和错误响应 | | `projects/app/src/service/common/s3/proxy.ts` | 上传/下载代理、流释放和错误响应 |
...@@ -220,7 +221,7 @@ Route 必须捕获可预期文件代理错误并映射状态码，不能全部 ...@@ -220,7 +221,7 @@ Route 必须捕获可预期文件代理错误并映射状态码，不能全部
- [ ] 下载短链正常、过期、篡改、撤销、对象不存在的状态码正确。 - [ ] 下载短链正常、过期、篡改、撤销、对象不存在的状态码正确。
- [ ] `short-proxy` 支持 GET/HEAD、range、客户端断开且不泄漏流。 - [ ] `short-proxy` 支持 GET/HEAD、range、客户端断开且不泄漏流。
- [ ] `short-redirect` 先验签再 302，Location TTL 不超过短链剩余寿命。 - [ ] `short-redirect` 先验签再 302，Location TTL 不超过短链剩余寿命。
- [ ] `presigned` 仅在显式模式下返回长 S3 URL。 - [ ] 配置旧值 `presigned` 时启动失败，业务 API 不再提供直出长 S3 URL 的参数。
- [ ] 上传 token 过期、撤销、重复使用、大小限制和 metadata 行为正确。 - [ ] 上传 token 过期、撤销、重复使用、大小限制和 metadata 行为正确。
- [ ] 无后缀 magic/text 文件按 policy 处理；unknown binary 无安全 fallback 时拒绝。 - [ ] 无后缀 magic/text 文件按 policy 处理；unknown binary 无安全 fallback 时拒绝。
- [ ] `.exe` 改名 `.png` 被拒绝；显式允许的 opaque 自定义扩展名可以上传。 - [ ] `.exe` 改名 `.png` 被拒绝；显式允许的 opaque 自定义扩展名可以上传。
...@@ -234,7 +235,7 @@ Route 必须捕获可预期文件代理错误并映射状态码，不能全部 ...@@ -234,7 +235,7 @@ Route 必须捕获可预期文件代理错误并映射状态码，不能全部
## 11. 升级与兼容 ## 11. 升级与兼容
1. 上线前固定 `FILE_TOKEN_KEY`，确认 Mongo 用户具备创建 TTL/unique 索引权限。 1. 上线前固定 `FILE_TOKEN_KEY`，确认 Mongo 用户具备创建 TTL/unique 索引权限。
2. 默认使用 `short-proxy`；外部 S3/CDN 部署可切换 `short-redirect`；仅兼容旧消费者时使用 `presigned`。 2. 默认使用 `short-proxy`；外部 S3/CDN 部署可切换 `short-redirect`。旧配置 `presigned` 必须在升级前改为其中一种模式。
3. nginx 根路径短链是可选展示层，App 的 `/api/system/file/d/*` 必须始终可达。 3. nginx 根路径短链是可选展示层，App 的 `/api/system/file/d/*` 必须始终可达。
4. 历史 JWT 路由和历史 Chat key 在兼容期继续工作；新写入统一使用短链和 source-aware key。 4. 历史 JWT 路由和历史 Chat key 在兼容期继续工作；新写入统一使用短链和 source-aware key。
5. 回滚前必须评估已写入消息、dataset 或工具结果的新 `/d` 链接；直接移除新路由会使历史内容不可访问。 5. 回滚前必须评估已写入消息、dataset 或工具结果的新 `/d` 链接；直接移除新路由会使历史内容不可访问。
.agents/issue/s3-refactor-analysis.md
...@@ -173,6 +173,6 @@ if (allowedExtensions.length > 0 && (!extension || !allowedExtensions.includes(e ...@@ -173,6 +173,6 @@ if (allowedExtensions.length > 0 && (!extension || !allowedExtensions.includes(e
## 8. 待用户确认的问题 ## 8. 待用户确认的问题
1. 短链是否只用于 `storageDownloadMode=proxy` 的代理链接，还是所有模型可见的文件预览链接都强制走短代理，即使环境配置了外部 S3 presigned URL？ 1. 已决策：所有模型可见的文件预览链接都使用短链；无外部 S3 地址时走 `short-proxy`，配置外部地址后可显式切换为 `short-redirect`。
2. 无后缀纯文本文件在 allowedExtensions 包含多个文本类型时，是否允许按 `text/plain` 接受，还是必须要求前端提供可信 `contentType` hint？ 2. 无后缀纯文本文件在 allowedExtensions 包含多个文本类型时，是否允许按 `text/plain` 接受，还是必须要求前端提供可信 `contentType` hint？
3. ChatBox 用户取消上传后，如果 S3 实际已经完成写入，是否需要立即投递 S3 删除任务，还是只保证不会进入本轮 chat 文件列表并依赖 TTL 清理？ 3. ChatBox 用户取消上传后，如果 S3 实际已经完成写入，是否需要立即投递 S3 删除任务，还是只保证不会进入本轮 chat 文件列表并依赖 TTL 清理？
deploy/version/main/docker-compose.template.yml
...@@ -25,7 +25,7 @@ x-share-db-config: &x-share-db-config ...@@ -25,7 +25,7 @@ x-share-db-config: &x-share-db-config
STORAGE_SECRET_ACCESS_KEY: minioadmin STORAGE_SECRET_ACCESS_KEY: minioadmin
STORAGE_PUBLIC_BUCKET: fastgpt-public STORAGE_PUBLIC_BUCKET: fastgpt-public
STORAGE_PRIVATE_BUCKET: fastgpt-private STORAGE_PRIVATE_BUCKET: fastgpt-private
STORAGE_EXTERNAL_ENDPOINT: http://192.168.0.2:9000 # 一个服务器和客户端均可访问到存储桶的地址，可以是固定的宿主机 IP 或者域名，注意不要填写成 127.0.0.1 或者 localhost 等本地回环地址(因为容器里无法使用) STORAGE_DOWNLOAD_URL_MODE: short-proxy # short-proxy | short-redirect
STORAGE_S3_ENDPOINT: http://fastgpt-minio:9000 # 协议://域名(IP):端口 STORAGE_S3_ENDPOINT: http://fastgpt-minio:9000 # 协议://域名(IP):端口
STORAGE_S3_FORCE_PATH_STYLE: true STORAGE_S3_FORCE_PATH_STYLE: true
STORAGE_S3_MAX_RETRIES: 3 STORAGE_S3_MAX_RETRIES: 3
......
deploy/version/v4.15/docker-compose.template.yml
...@@ -33,8 +33,7 @@ x-share-db-config: &x-share-db-config ...@@ -33,8 +33,7 @@ x-share-db-config: &x-share-db-config
STORAGE_SECRET_ACCESS_KEY: minioadmin STORAGE_SECRET_ACCESS_KEY: minioadmin
STORAGE_PUBLIC_BUCKET: fastgpt-public STORAGE_PUBLIC_BUCKET: fastgpt-public
STORAGE_PRIVATE_BUCKET: fastgpt-private STORAGE_PRIVATE_BUCKET: fastgpt-private
STORAGE_EXTERNAL_ENDPOINT: http://192.168.0.2:9000 # 一个服务器和客户端均可访问到存储桶的地址，可以是固定的宿主机 IP 或者域名，注意不要填写成 127.0.0.1 或者 localhost 等本地回环地址(因为容器里无法使用) STORAGE_DOWNLOAD_URL_MODE: short-proxy # short-proxy | short-redirect
STORAGE_S3_CDN_ENDPOINT:
STORAGE_S3_ENDPOINT: http://fastgpt-minio:9000 # 协议://域名(IP):端口 STORAGE_S3_ENDPOINT: http://fastgpt-minio:9000 # 协议://域名(IP):端口
STORAGE_S3_FORCE_PATH_STYLE: true STORAGE_S3_FORCE_PATH_STYLE: true
STORAGE_S3_MAX_RETRIES: 3 STORAGE_S3_MAX_RETRIES: 3
......
document/content/self-host/config/env.mdx
...@@ -96,29 +96,31 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说 ...@@ -96,29 +96,31 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说
### 对象存储 ### 对象存储
| 变量 | 默认值 | 说明 | | 变量 | 默认值 | 说明 |
| -------------------------------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------- | | --------------------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------- |
| `STORAGE_VENDOR` | `minio` | 对象存储类型，可选 `minio`、`aws-s3`、`cos`、`oss`。 | | `STORAGE_VENDOR` | `minio` | 对象存储类型，可选 `minio`、`aws-s3`、`cos`、`oss`。 |
| `STORAGE_PUBLIC_BUCKET` | `fastgpt-public` | 公开文件 Bucket。 | | `STORAGE_PUBLIC_BUCKET` | `fastgpt-public` | 公开文件 Bucket。 |
| `STORAGE_PRIVATE_BUCKET` | `fastgpt-private` | 私有文件 Bucket。 | | `STORAGE_PRIVATE_BUCKET` | `fastgpt-private` | 私有文件 Bucket。 |
| `STORAGE_REGION` | `us-east-1` | 对象存储 Region。 | | `STORAGE_REGION` | `us-east-1` | 对象存储 Region。 |
| `STORAGE_EXTERNAL_ENDPOINT` | 空 | 外部可访问的对象存储地址，用于浏览器或外部服务访问。 | | `STORAGE_EXTERNAL_ENDPOINT` | 空 | 外部可访问的对象存储地址，用于浏览器或外部服务访问。 |
| `STORAGE_S3_CDN_ENDPOINT` | 空 | S3/MinIO 预签名下载与预览访问的 CDN 地址；仅在当前请求走 `presigned` 下载模式时替换访问 URL，不改变默认下载模式。 | | `STORAGE_S3_CDN_ENDPOINT` | 空 | `short-redirect` 临时下载地址使用的 CDN 地址；配置时必须同时配置 `STORAGE_EXTERNAL_ENDPOINT`。 |
| `STORAGE_S3_ENDPOINT` | `http://localhost:9000` | S3/MinIO 兼容 API 地址。 | | `STORAGE_DOWNLOAD_URL_MODE` | `short-proxy` | 下载模式，可选 `short-proxy` 或 `short-redirect`；对外始终返回 FastGPT 短链。 |
| `STORAGE_PUBLIC_ACCESS_EXTRA_SUB_PATH` | 空 | 公开文件访问路径的额外子路径。 | | `STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS` | `300` | `short-redirect` 模式下临时对象存储/CDN 下载地址的有效时间，单位秒。 |
| `STORAGE_ACCESS_KEY_ID` | `minioadmin` | 对象存储 Access Key。 | | `STORAGE_S3_ENDPOINT` | `http://localhost:9000` | S3/MinIO 兼容 API 地址。 |
| `STORAGE_SECRET_ACCESS_KEY` | `minioadmin` | 对象存储 Secret Key。 | | `STORAGE_PUBLIC_ACCESS_EXTRA_SUB_PATH` | 空 | 公开文件访问路径的额外子路径。 |
| `STORAGE_S3_FORCE_PATH_STYLE` | `false` | S3 是否强制 path-style 访问，MinIO 通常需要开启。 | | `STORAGE_ACCESS_KEY_ID` | `minioadmin` | 对象存储 Access Key。 |
| `STORAGE_S3_MAX_RETRIES` | `3` | S3 客户端最大重试次数。 | | `STORAGE_SECRET_ACCESS_KEY` | `minioadmin` | 对象存储 Secret Key。 |
| `STORAGE_COS_PROTOCOL` | `https:` | 腾讯云 COS 访问协议，可选 `https:` 或 `http:`。 | | `STORAGE_S3_FORCE_PATH_STYLE` | `false` | S3 是否强制 path-style 访问，MinIO 通常需要开启。 |
| `STORAGE_COS_USE_ACCELERATE` | `false` | 腾讯云 COS 是否使用全球加速域名。 | | `STORAGE_S3_MAX_RETRIES` | `3` | S3 客户端最大重试次数。 |
| `STORAGE_COS_CNAME_DOMAIN` | 空 | 腾讯云 COS 自定义 CNAME 域名。 | | `STORAGE_COS_PROTOCOL` | `https:` | 腾讯云 COS 访问协议，可选 `https:` 或 `http:`。 |
| `STORAGE_COS_PROXY` | 空 | 腾讯云 COS 代理地址。 | | `STORAGE_COS_USE_ACCELERATE` | `false` | 腾讯云 COS 是否使用全球加速域名。 |
| `STORAGE_OSS_ENDPOINT` | 空 | 阿里云 OSS Endpoint。 | | `STORAGE_COS_CNAME_DOMAIN` | 空 | 腾讯云 COS 自定义 CNAME 域名。 |
| `STORAGE_OSS_CNAME` | `false` | 阿里云 OSS 是否使用 CNAME。 | | `STORAGE_COS_PROXY` | 空 | 腾讯云 COS 代理地址。 |
| `STORAGE_OSS_INTERNAL` | `false` | 阿里云 OSS 是否使用内网 Endpoint。 | | `STORAGE_OSS_ENDPOINT` | 空 | 阿里云 OSS Endpoint。 |
| `STORAGE_OSS_SECURE` | `false` | 阿里云 OSS 是否使用 HTTPS。 | | `STORAGE_OSS_CNAME` | `false` | 阿里云 OSS 是否使用 CNAME。 |
| `STORAGE_OSS_ENABLE_PROXY` | `true` | 阿里云 OSS 是否启用代理访问。 | | `STORAGE_OSS_INTERNAL` | `false` | 阿里云 OSS 是否使用内网 Endpoint。 |
| `STORAGE_OSS_SECURE` | `false` | 阿里云 OSS 是否使用 HTTPS。 |
| `STORAGE_OSS_ENABLE_PROXY` | `true` | 阿里云 OSS 是否启用代理访问。 |
### 日志、指标与追踪 ### 日志、指标与追踪
......
document/content/self-host/config/object-storage.en.mdx
...@@ -25,10 +25,11 @@ This guide covers environment variable configuration for object storage provider ...@@ -25,10 +25,11 @@ This guide covers environment variable configuration for object storage provider
### Transfer Behavior ### Transfer Behavior
- Uploads always go through the FastGPT backend proxy. - Uploads always go through the FastGPT backend proxy.
- Downloads support both `proxy` and `presigned` modes. - External download URLs are always FastGPT short links. FastGPT no longer returns object storage presigned URLs directly.
- The default download mode is inferred from `STORAGE_EXTERNAL_ENDPOINT`: - `STORAGE_DOWNLOAD_URL_MODE` supports two modes and defaults to `short-proxy`:
- not configured: default to `proxy` - `short-proxy`: FastGPT validates the short link and proxies the file stream. No public object storage endpoint is required.
- configured: default to `presigned` - `short-redirect`: FastGPT validates the short link, then redirects to a short-lived object storage or CDN URL. File traffic bypasses FastGPT.
- Self-hosted MinIO requires `STORAGE_EXTERNAL_ENDPOINT` when using `short-redirect`.
### Self-Hosted MinIO and AWS S3 ### Self-Hosted MinIO and AWS S3
...@@ -36,8 +37,8 @@ This guide covers environment variable configuration for object storage provider ...@@ -36,8 +37,8 @@ This guide covers environment variable configuration for object storage provider
> In theory, any object storage with S3 protocol support comparable to MinIO will work, such as SeaweedFS, RustFS, etc. > In theory, any object storage with S3 protocol support comparable to MinIO will work, such as SeaweedFS, RustFS, etc.
- `STORAGE_S3_ENDPOINT` Internal connection address. Can be a container ID, e.g., `http://fastgpt-minio:9000` - `STORAGE_S3_ENDPOINT` Internal connection address. Can be a container ID, e.g., `http://fastgpt-minio:9000`
- `STORAGE_EXTERNAL_ENDPOINT` An address accessible by both **server** and **client** to reach the bucket. Use a fixed host IP or domain name — don't use `127.0.0.1` or `localhost` (containers can't access loopback addresses). Once configured, the default download mode automatically becomes `presigned`. - `STORAGE_EXTERNAL_ENDPOINT` An address accessible by both **server** and **client** to reach the bucket. Use a fixed host IP or domain name — don't use `127.0.0.1` or `localhost` (containers can't access loopback addresses). This variable does not change the download mode automatically.
- `STORAGE_S3_CDN_ENDPOINT` [Optional] CDN endpoint for S3/MinIO presigned download and preview access. FastGPT rewrites generated file access URLs to this CDN endpoint only when the current download uses `presigned` mode. This variable does not change the default download mode. Uploads still go through the FastGPT backend proxy and do not use the CDN. - `STORAGE_S3_CDN_ENDPOINT` [Optional] CDN endpoint used for temporary `short-redirect` download URLs. This variable does not change the default download mode and requires `STORAGE_EXTERNAL_ENDPOINT`. Uploads still go through the FastGPT backend proxy and do not use the CDN.
- `STORAGE_S3_FORCE_PATH_STYLE` [Optional] Virtual-hosted-style or path-style routing. If vendor is `minio`, this is fixed to `true`. - `STORAGE_S3_FORCE_PATH_STYLE` [Optional] Virtual-hosted-style or path-style routing. If vendor is `minio`, this is fixed to `true`.
- `STORAGE_S3_MAX_RETRIES` [Optional] Maximum request retry attempts. Default: 3 - `STORAGE_S3_MAX_RETRIES` [Optional] Maximum request retry attempts. Default: 3
...@@ -52,7 +53,9 @@ STORAGE_ACCESS_KEY_ID=your_access_key ...@@ -52,7 +53,9 @@ STORAGE_ACCESS_KEY_ID=your_access_key
STORAGE_SECRET_ACCESS_KEY=your_secret_key STORAGE_SECRET_ACCESS_KEY=your_secret_key
STORAGE_PUBLIC_BUCKET=fastgpt-public STORAGE_PUBLIC_BUCKET=fastgpt-public
STORAGE_PRIVATE_BUCKET=fastgpt-private STORAGE_PRIVATE_BUCKET=fastgpt-private
STORAGE_EXTERNAL_ENDPOINT=http://127.0.0.1:9000 STORAGE_DOWNLOAD_URL_MODE=short-redirect
STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS=300
STORAGE_EXTERNAL_ENDPOINT=https://s3.example.com
STORAGE_S3_CDN_ENDPOINT=https://cdn.example.com STORAGE_S3_CDN_ENDPOINT=https://cdn.example.com
STORAGE_S3_ENDPOINT=http://127.0.0.1:9000 STORAGE_S3_ENDPOINT=http://127.0.0.1:9000
STORAGE_S3_FORCE_PATH_STYLE=true STORAGE_S3_FORCE_PATH_STYLE=true
......
document/content/self-host/config/object-storage.mdx
...@@ -25,18 +25,19 @@ import FastGPTLink from '@/components/docs/linkFastGPT'; ...@@ -25,18 +25,19 @@ import FastGPTLink from '@/components/docs/linkFastGPT';
### 传输模式说明 ### 传输模式说明
- 上传固定走 FastGPT 后端代理。 - 上传固定走 FastGPT 后端代理。
- 下载支持 `proxy` 和 `presigned` 两种模式。 - 对外下载地址固定为 FastGPT 短链，不再直接返回对象存储预签名长链接。
- 默认下载模式会根据 `STORAGE_EXTERNAL_ENDPOINT` 自动判断： - `STORAGE_DOWNLOAD_URL_MODE` 支持两种模式，默认值为 `short-proxy`：
- 未配置：默认 `proxy` - `short-proxy`：FastGPT 校验短链并代理文件流，无需配置公网对象存储地址。
- 已配置：默认 `presigned` - `short-redirect`：FastGPT 校验短链后 302 到短时效对象存储/CDN 地址，文件流量不经过 FastGPT。
- 自部署 MinIO 使用 `short-redirect` 时必须配置 `STORAGE_EXTERNAL_ENDPOINT`。
### 自部署的 MinIO 和 AWS S3 ### 自部署的 MinIO 和 AWS S3
> MinIO 这类产品对 AWS S3 协议支持比较完整，因此使用 Minio 和 AWS S3 配置几乎可以是相同的，只是因为服务商提供和自部署的区别，会有额外的配置。> 因此理论上任何对 AWS S3 协议的支持程度至少和 MinIO 相当的对象存储服务都可以使用，比如 SeaweedFS、RustFS 等。 > MinIO 这类产品对 AWS S3 协议支持比较完整，因此使用 Minio 和 AWS S3 配置几乎可以是相同的，只是因为服务商提供和自部署的区别，会有额外的配置。> 因此理论上任何对 AWS S3 协议的支持程度至少和 MinIO 相当的对象存储服务都可以使用，比如 SeaweedFS、RustFS 等。
- `STORAGE_S3_ENDPOINT` 内网连接地址，可以是容器 ID 连接，比如 `http://fastgpt-minio:9000` - `STORAGE_S3_ENDPOINT` 内网连接地址，可以是容器 ID 连接，比如 `http://fastgpt-minio:9000`
- `STORAGE_EXTERNAL_ENDPOINT` 一个**服务器**和**客户端**均可访问到存储桶的地址，可以是固定的宿主机 IP 或者域名，注意不要填写成 127.0.0.1 或者 localhost 等本地回环地址(因为容器里无法使用)。配置后，默认下载模式会自动切换为 `presigned`。 - `STORAGE_EXTERNAL_ENDPOINT` 一个**服务器**和**客户端**均可访问到存储桶的地址，可以是固定的宿主机 IP 或者域名，注意不要填写成 127.0.0.1 或者 localhost 等本地回环地址(因为容器里无法使用)。该变量不会自动改变下载模式。
- `STORAGE_S3_CDN_ENDPOINT`【可选】S3/MinIO 预签名下载与预览访问的 CDN 地址。仅在当前下载走 `presigned` 模式时，FastGPT 会在生成文件访问 URL 后把地址替换成该 CDN 地址；该变量不会改变默认下载模式。上传仍走 FastGPT 后端代理，不会使用 CDN。 - `STORAGE_S3_CDN_ENDPOINT`【可选】`short-redirect` 临时下载地址使用的 CDN 地址。该变量不会改变默认下载模式，且配置时必须同时配置 `STORAGE_EXTERNAL_ENDPOINT`。上传仍走 FastGPT 后端代理，不会使用 CDN。
- `STORAGE_S3_FORCE_PATH_STYLE`【可选】虚拟主机风格路由或路径路由风格，其中如果厂商填写了 `minio` 的话，该值被固定为 `true` - `STORAGE_S3_FORCE_PATH_STYLE`【可选】虚拟主机风格路由或路径路由风格，其中如果厂商填写了 `minio` 的话，该值被固定为 `true`
- `STORAGE_S3_MAX_RETRIES`【可选】请求最大尝试次数，默认为 3 次 - `STORAGE_S3_MAX_RETRIES`【可选】请求最大尝试次数，默认为 3 次
...@@ -51,7 +52,9 @@ STORAGE_ACCESS_KEY_ID=your_access_key ...@@ -51,7 +52,9 @@ STORAGE_ACCESS_KEY_ID=your_access_key
STORAGE_SECRET_ACCESS_KEY=your_secret_key STORAGE_SECRET_ACCESS_KEY=your_secret_key
STORAGE_PUBLIC_BUCKET=fastgpt-public STORAGE_PUBLIC_BUCKET=fastgpt-public
STORAGE_PRIVATE_BUCKET=fastgpt-private STORAGE_PRIVATE_BUCKET=fastgpt-private
STORAGE_EXTERNAL_ENDPOINT=http://127.0.0.1:9000 STORAGE_DOWNLOAD_URL_MODE=short-redirect
STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS=300
STORAGE_EXTERNAL_ENDPOINT=https://s3.example.com
STORAGE_S3_CDN_ENDPOINT=https://cdn.example.com STORAGE_S3_CDN_ENDPOINT=https://cdn.example.com
STORAGE_S3_ENDPOINT=http://127.0.0.1:9000 STORAGE_S3_ENDPOINT=http://127.0.0.1:9000
STORAGE_S3_FORCE_PATH_STYLE=true STORAGE_S3_FORCE_PATH_STYLE=true
......
document/data/doc-last-modified.json
...@@ -137,8 +137,8 @@ ...@@ -137,8 +137,8 @@
"content/guide/version/cloud/privacy.mdx": "2026-05-07T15:06:40+08:00", "content/guide/version/cloud/privacy.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/version/cloud/terms.en.mdx": "2026-05-07T15:06:40+08:00", "content/guide/version/cloud/terms.en.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/version/cloud/terms.mdx": "2026-05-28T13:54:38+08:00", "content/guide/version/cloud/terms.mdx": "2026-05-28T13:54:38+08:00",
"content/guide/version/commercial.en.mdx": "2026-07-16T21:14:34+08:00", "content/guide/version/commercial.en.mdx": "2026-07-16T23:04:39+08:00",
"content/guide/version/commercial.mdx": "2026-07-16T21:14:34+08:00", "content/guide/version/commercial.mdx": "2026-07-16T23:04:39+08:00",
"content/guide/version/opensource/intro.en.mdx": "2026-05-07T15:06:40+08:00", "content/guide/version/opensource/intro.en.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/version/opensource/intro.mdx": "2026-05-07T15:06:40+08:00", "content/guide/version/opensource/intro.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/version/opensource/license.en.mdx": "2026-05-07T15:06:40+08:00", "content/guide/version/opensource/license.en.mdx": "2026-05-07T15:06:40+08:00",
...@@ -167,8 +167,8 @@ ...@@ -167,8 +167,8 @@
"content/plugin/model-presets.mdx": "2026-06-04T16:10:15+08:00", "content/plugin/model-presets.mdx": "2026-06-04T16:10:15+08:00",
"content/plugin/system-tool-development.en.mdx": "2026-07-02T11:54:55+08:00", "content/plugin/system-tool-development.en.mdx": "2026-07-02T11:54:55+08:00",
"content/plugin/system-tool-development.mdx": "2026-07-02T11:54:55+08:00", "content/plugin/system-tool-development.mdx": "2026-07-02T11:54:55+08:00",
"content/self-host/config/env.en.mdx": "2026-07-16T19:20:17+08:00", "content/self-host/config/env.en.mdx": "2026-07-16T23:04:39+08:00",
"content/self-host/config/env.mdx": "2026-07-16T19:20:17+08:00", "content/self-host/config/env.mdx": "2026-07-16T23:04:39+08:00",
"content/self-host/config/model/intro.en.mdx": "2026-06-04T16:10:15+08:00", "content/self-host/config/model/intro.en.mdx": "2026-06-04T16:10:15+08:00",
"content/self-host/config/model/intro.mdx": "2026-06-04T16:10:15+08:00", "content/self-host/config/model/intro.mdx": "2026-06-04T16:10:15+08:00",
"content/self-host/config/model/minimax.en.mdx": "2026-06-03T10:40:17+08:00", "content/self-host/config/model/minimax.en.mdx": "2026-06-03T10:40:17+08:00",
...@@ -320,8 +320,8 @@ ...@@ -320,8 +320,8 @@
"content/self-host/upgrading/4-15/41507.mdx": "2026-06-30T17:31:43+08:00", "content/self-host/upgrading/4-15/41507.mdx": "2026-06-30T17:31:43+08:00",
"content/self-host/upgrading/4-15/4151.en.mdx": "2026-07-07T21:14:28+08:00", "content/self-host/upgrading/4-15/4151.en.mdx": "2026-07-07T21:14:28+08:00",
"content/self-host/upgrading/4-15/4151.mdx": "2026-07-07T21:14:28+08:00", "content/self-host/upgrading/4-15/4151.mdx": "2026-07-07T21:14:28+08:00",
"content/self-host/upgrading/4-15/4152.en.mdx": "2026-07-16T20:55:59+08:00", "content/self-host/upgrading/4-15/4152.en.mdx": "2026-07-16T23:04:39+08:00",
"content/self-host/upgrading/4-15/4152.mdx": "2026-07-16T20:55:59+08:00", "content/self-host/upgrading/4-15/4152.mdx": "2026-07-16T23:04:39+08:00",
"content/self-host/upgrading/outdated/40.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/outdated/40.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/40.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/outdated/40.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/41.en.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/outdated/41.en.mdx": "2026-04-26T21:08:47+08:00",
......
packages/global/openapi/core/chat/file/api.ts
...@@ -14,15 +14,10 @@ const withChatFileTarget = <T extends z.ZodRawShape>(shape: T) => ...@@ -14,15 +14,10 @@ const withChatFileTarget = <T extends z.ZodRawShape>(shape: T) =>
createOutLinkChatTargetInputSchema(shape).transform(transformChatAuthTargetInput); createOutLinkChatTargetInputSchema(shape).transform(transformChatAuthTargetInput);
const withInternalChatFileTarget = <T extends z.ZodRawShape>(shape: T) => const withInternalChatFileTarget = <T extends z.ZodRawShape>(shape: T) =>
createChatTargetInputSchema(shape).transform(transformChatTargetInput); createChatTargetInputSchema(shape).transform(transformChatTargetInput);
const ChatFileDownloadModeSchema = z
.enum(['short-proxy', 'short-redirect', 'presigned'])
.optional()
.describe('下载链接模式');
export const PresignChatFileGetUrlRawSchema = createOutLinkChatTargetInputSchema({ export const PresignChatFileGetUrlRawSchema = createOutLinkChatTargetInputSchema({
key: z.string().min(1).describe('文件key'), key: z.string().min(1).describe('文件key'),
chatId: z.string().min(1).describe('对话ID'), chatId: z.string().min(1).describe('对话ID'),
mode: ChatFileDownloadModeSchema,
outLinkAuthData: OutLinkChatAuthSchema.optional().describe('外链鉴权数据') outLinkAuthData: OutLinkChatAuthSchema.optional().describe('外链鉴权数据')
}).meta({ }).meta({
example: { example: {
...@@ -37,7 +32,6 @@ export const PresignChatFileGetUrlRawSchema = createOutLinkChatTargetInputSchema ...@@ -37,7 +32,6 @@ export const PresignChatFileGetUrlRawSchema = createOutLinkChatTargetInputSchema
export const PresignChatFileGetUrlSchema = withChatFileTarget({ export const PresignChatFileGetUrlSchema = withChatFileTarget({
key: z.string().min(1).describe('文件key'), key: z.string().min(1).describe('文件key'),
chatId: z.string().min(1).describe('对话ID'), chatId: z.string().min(1).describe('对话ID'),
mode: ChatFileDownloadModeSchema,
outLinkAuthData: OutLinkChatAuthSchema.optional().describe('外链鉴权数据') outLinkAuthData: OutLinkChatAuthSchema.optional().describe('外链鉴权数据')
}); });
export type PresignChatFileGetUrlParams = z.input<typeof PresignChatFileGetUrlSchema>; export type PresignChatFileGetUrlParams = z.input<typeof PresignChatFileGetUrlSchema>;
......
packages/global/openapi/core/chat/file/index.ts
...@@ -71,7 +71,7 @@ export const ChatFilePath: OpenAPIPath = { ...@@ -71,7 +71,7 @@ export const ChatFilePath: OpenAPIPath = {
}, },
responses: { responses: {
200: { 200: {
description: '成功获取对话文件预签名 URL', description: '成功获取对话文件短链接',
content: { content: {
'application/json': { 'application/json': {
schema: z.string() schema: z.string()
......
packages/service/common/s3/buckets/base.ts
...@@ -8,11 +8,7 @@ import { ...@@ -8,11 +8,7 @@ import {
CreatePostPresignedUrlParamsSchema, CreatePostPresignedUrlParamsSchema,
type CreatePostPresignedUrlResult type CreatePostPresignedUrlResult
} from '../contracts/type'; } from '../contracts/type';
import { import { getSystemMaxFileSize } from '../config/constants';
storageDownloadUrlMode,
getSystemMaxFileSize,
replaceS3UrlWithCdnEndpoint
} from '../config/constants';
import { S3ErrEnum } from '@fastgpt/global/common/error/code/s3'; import { S3ErrEnum } from '@fastgpt/global/common/error/code/s3';
import { createUploadConstraints } from '../utils/uploadConstraints'; import { createUploadConstraints } from '../utils/uploadConstraints';
import path from 'node:path'; import path from 'node:path';
...@@ -278,32 +274,19 @@ export class S3BaseBucket { ...@@ -278,32 +274,19 @@ export class S3BaseBucket {
async createExternalUrl(params: createPreviewUrlParams) { async createExternalUrl(params: createPreviewUrlParams) {
const parsed = CreateGetPresignedUrlParamsSchema.parse(params); const parsed = CreateGetPresignedUrlParamsSchema.parse(params);
const { key, expiredHours, mode, responseContentType } = parsed; const { key, expiredHours, responseContentType } = parsed;
const expires = expiredHours ? expiredHours * 60 * 60 : 30 * 60; // expires 的单位是秒 默认 30 分钟 const expires = expiredHours ? expiredHours * 60 * 60 : 30 * 60; // expires 的单位是秒 默认 30 分钟
if ((mode ?? storageDownloadUrlMode) !== 'presigned') {
return {
bucket: this.bucketName,
key,
url: await createS3DownloadAccessUrl({
objectKey: key,
bucketName: this.bucketName,
expiredTime: addMinutes(new Date(), Math.ceil(expires / 60)),
filename: path.basename(key),
responseContentType
})
};
}
const result = await this.externalClient.generatePresignedGetUrl({
key,
expiredSeconds: expires,
...(responseContentType ? { responseContentType } : {})
});
return { return {
...result, bucket: this.bucketName,
url: replaceS3UrlWithCdnEndpoint(result.url) key,
url: await createS3DownloadAccessUrl({
objectKey: key,
bucketName: this.bucketName,
expiredTime: addMinutes(new Date(), Math.ceil(expires / 60)),
filename: path.basename(key),
responseContentType
})
}; };
} }
......
packages/service/common/s3/contracts/type.ts
...@@ -33,12 +33,9 @@ export const S3SourcesSchema = z.enum(['avatar', 'chat', 'dataset', 'temp', 'raw ...@@ -33,12 +33,9 @@ export const S3SourcesSchema = z.enum(['avatar', 'chat', 'dataset', 'temp', 'raw
export const S3Sources = S3SourcesSchema.enum; export const S3Sources = S3SourcesSchema.enum;
export type S3SourceType = z.infer<typeof S3SourcesSchema>; export type S3SourceType = z.infer<typeof S3SourcesSchema>;
export const StorageDownloadUrlModeSchema = z.enum(['short-proxy', 'short-redirect', 'presigned']); export const StorageDownloadUrlModeSchema = z.enum(['short-proxy', 'short-redirect']);
export type StorageDownloadUrlMode = z.infer<typeof StorageDownloadUrlModeSchema>; export type StorageDownloadUrlMode = z.infer<typeof StorageDownloadUrlModeSchema>;
export const DownloadModeSchema = StorageDownloadUrlModeSchema;
export type DownloadMode = z.infer<typeof DownloadModeSchema>;
export const CreatePostPresignedUrlParamsSchema = z.object({ export const CreatePostPresignedUrlParamsSchema = z.object({
filename: z.string().min(1), filename: z.string().min(1),
rawKey: z.string().min(1), rawKey: z.string().min(1),
...@@ -69,7 +66,6 @@ export type CreatePostPresignedUrlResult = z.infer<typeof CreatePostPresignedUrl ...@@ -69,7 +66,6 @@ export type CreatePostPresignedUrlResult = z.infer<typeof CreatePostPresignedUrl
export const CreateGetPresignedUrlParamsSchema = z.object({ export const CreateGetPresignedUrlParamsSchema = z.object({
key: z.string().nonempty(), key: z.string().nonempty(),
expiredHours: z.number().positive().optional(), expiredHours: z.number().positive().optional(),
mode: DownloadModeSchema.optional(),
responseContentType: z.string().nonempty().optional() responseContentType: z.string().nonempty().optional()
}); });
export type createPreviewUrlParams = z.infer<typeof CreateGetPresignedUrlParamsSchema>; export type createPreviewUrlParams = z.infer<typeof CreateGetPresignedUrlParamsSchema>;
......
packages/service/common/s3/sources/chat/index.ts
import { S3PrivateBucket } from '../../buckets/private'; import { S3PrivateBucket } from '../../buckets/private';
import { type DownloadMode, S3Sources } from '../../contracts/type'; import { S3Sources } from '../../contracts/type';
import { import {
type CheckChatFileKeys, type CheckChatFileKeys,
type DelChatFileByPrefixParams, type DelChatFileByPrefixParams,
...@@ -72,16 +72,11 @@ export class S3ChatSource extends S3PrivateBucket { ...@@ -72,16 +72,11 @@ export class S3ChatSource extends S3PrivateBucket {
} }
} }
async createGetChatFileURL(params: { async createGetChatFileURL(params: { key: string; expiredHours?: number; external: boolean }) {
key: string; const { key, expiredHours = 1, external = false } = params; // 默认一个小时
expiredHours?: number;
external: boolean;
mode?: DownloadMode;
}) {
const { key, expiredHours = 1, external = false, mode } = params; // 默认一个小时
if (external) { if (external) {
return await this.createExternalUrl({ key, expiredHours, mode }); return await this.createExternalUrl({ key, expiredHours });
} }
return await this.createPreviewUrl({ key, expiredHours }); return await this.createPreviewUrl({ key, expiredHours });
} }
...@@ -185,10 +180,7 @@ export function getS3ChatSource() { ...@@ -185,10 +180,7 @@ export function getS3ChatSource() {
return global.chatBucket; return global.chatBucket;
} }
export const createChatFilePreviewUrlGetter = (options?: { export const createChatFilePreviewUrlGetter = (options?: { expiredHours?: number }) => {
expiredHours?: number;
mode?: DownloadMode;
}) => {
const s3ChatSource = getS3ChatSource(); const s3ChatSource = getS3ChatSource();
return async (key: string) => { return async (key: string) => {
......
packages/service/env.ts
...@@ -174,11 +174,11 @@ export const serviceEnv = createEnv({ ...@@ -174,11 +174,11 @@ export const serviceEnv = createEnv({
STORAGE_EXTERNAL_ENDPOINT: UrlSchema.optional(), STORAGE_EXTERNAL_ENDPOINT: UrlSchema.optional(),
STORAGE_S3_CDN_ENDPOINT: UrlSchema.optional(), STORAGE_S3_CDN_ENDPOINT: UrlSchema.optional(),
STORAGE_DOWNLOAD_URL_MODE: z STORAGE_DOWNLOAD_URL_MODE: z
.enum(['short-proxy', 'short-redirect', 'presigned']) .enum(['short-proxy', 'short-redirect'])
.default('short-proxy') .default('short-proxy')
.meta({ .meta({
description: description:
'下载链接模式：short-proxy 返回 FastGPT 短链并由 app 代理；short-redirect 返回 FastGPT 短链并 302 到短 TTL S3 链接；presigned 直接返回 S3 预签名长链' '下载链接模式：short-proxy 返回 FastGPT 短链并由 app 代理；short-redirect 返回 FastGPT 短链并 302 到短 TTL S3 链接'
}), }),
STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS: IntSchema.min(1).default(300).meta({ STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS: IntSchema.min(1).default(300).meta({
description: 'short-redirect 模式下临时 S3 预签名下载链接 TTL（秒）' description: 'short-redirect 模式下临时 S3 预签名下载链接 TTL（秒）'
......
packages/service/env.util.ts
...@@ -34,7 +34,7 @@ type S3Env = { ...@@ -34,7 +34,7 @@ type S3Env = {
/** /**
* 校验对象存储下载模式依赖的公网访问地址。 * 校验对象存储下载模式依赖的公网访问地址。
* CDN 仅用于替换已生成的外部 URL，因此必须同时配置外部地址。 * CDN 仅用于替换已生成的外部 URL，因此必须同时配置外部地址。
* 除此之外，仅 MinIO 的直连模式需要显式提供客户端可访问的外部地址。 * 除此之外，仅 MinIO 的 short-redirect 模式需要显式提供客户端可访问的外部地址。
*/ */
export const validateS3Env = (env: S3Env): void => { export const validateS3Env = (env: S3Env): void => {
if (env.STORAGE_S3_CDN_ENDPOINT && !env.STORAGE_EXTERNAL_ENDPOINT) { if (env.STORAGE_S3_CDN_ENDPOINT && !env.STORAGE_EXTERNAL_ENDPOINT) {
...@@ -44,9 +44,7 @@ export const validateS3Env = (env: S3Env): void => { ...@@ -44,9 +44,7 @@ export const validateS3Env = (env: S3Env): void => {
} }
const requiresExternalEndpoint = const requiresExternalEndpoint =
env.STORAGE_VENDOR === 'minio' && env.STORAGE_VENDOR === 'minio' && env.STORAGE_DOWNLOAD_URL_MODE === 'short-redirect';
(env.STORAGE_DOWNLOAD_URL_MODE === 'short-redirect' ||
env.STORAGE_DOWNLOAD_URL_MODE === 'presigned');
if (!requiresExternalEndpoint || env.STORAGE_EXTERNAL_ENDPOINT) { if (!requiresExternalEndpoint || env.STORAGE_EXTERNAL_ENDPOINT) {
return; return;
......
packages/service/test/common/s3/config/constants.test.ts
...@@ -103,9 +103,9 @@ describe('s3 storage constants', () => { ...@@ -103,9 +103,9 @@ describe('s3 storage constants', () => {
expect(canUseStorageDownloadRedirect).toBe(true); expect(canUseStorageDownloadRedirect).toBe(true);
}); });
it('rewrites external presigned URLs from S3BaseBucket', async () => { it('returns a short link in short redirect mode', async () => {
vi.stubEnv('STORAGE_EXTERNAL_ENDPOINT', 'https://s3.example.com'); vi.stubEnv('STORAGE_EXTERNAL_ENDPOINT', 'https://s3.example.com');
vi.stubEnv('STORAGE_S3_CDN_ENDPOINT', 'https://cdn.example.com'); vi.stubEnv('STORAGE_DOWNLOAD_URL_MODE', 'short-redirect');
const { S3BaseBucket } = await vi.importActual< const { S3BaseBucket } = await vi.importActual<
typeof import('@fastgpt/service/common/s3/buckets/base') typeof import('@fastgpt/service/common/s3/buckets/base')
...@@ -118,16 +118,16 @@ describe('s3 storage constants', () => { ...@@ -118,16 +118,16 @@ describe('s3 storage constants', () => {
const bucket = new S3BaseBucket(storage, undefined); const bucket = new S3BaseBucket(storage, undefined);
const result = await bucket.createExternalUrl({ const result = await bucket.createExternalUrl({
key: 'chat/app/user/chat/file.png', key: 'chat/app/user/chat/file.png'
mode: 'presigned'
}); });
expect(storage.generatePresignedGetUrl).toHaveBeenCalledWith({ expect(storage.generatePresignedGetUrl).not.toHaveBeenCalled();
key: 'chat/app/user/chat/file.png', expect(result).toMatchObject({
expiredSeconds: 1800 bucket: 'fastgpt-private',
key: 'chat/app/user/chat/file.png'
}); });
expect(result.url).toBe( expect(result.url).toMatch(
'https://cdn.example.com/get/fastgpt-private/chat%2Fapp%2Fuser%2Fchat%2Ffile.png' /\/api\/system\/file\/d\/[A-Za-z0-9_-]{16}\.[0-9a-z]+\.[A-Za-z0-9_-]{22}$/
); );
}); });
...@@ -158,7 +158,7 @@ describe('s3 storage constants', () => { ...@@ -158,7 +158,7 @@ describe('s3 storage constants', () => {
); );
}); });
it('passes response content type overrides into external presigned URLs', async () => { it('keeps internal presigned previews for server-side storage access', async () => {
const { S3BaseBucket } = await vi.importActual< const { S3BaseBucket } = await vi.importActual<
typeof import('@fastgpt/service/common/s3/buckets/base') typeof import('@fastgpt/service/common/s3/buckets/base')
>('@fastgpt/service/common/s3/buckets/base'); >('@fastgpt/service/common/s3/buckets/base');
...@@ -169,9 +169,8 @@ describe('s3 storage constants', () => { ...@@ -169,9 +169,8 @@ describe('s3 storage constants', () => {
}); });
const bucket = new S3BaseBucket(storage, undefined); const bucket = new S3BaseBucket(storage, undefined);
const result = await bucket.createExternalUrl({ const result = await bucket.createPreviewUrl({
key: 'dataset/team/aaa.md', key: 'dataset/team/aaa.md',
mode: 'presigned',
responseContentType: 'text/markdown; charset=utf-8' responseContentType: 'text/markdown; charset=utf-8'
}); });
......
packages/service/test/core/chat/utils.test.ts
...@@ -16,7 +16,6 @@ const { mockCreateGetChatFileURL } = vi.hoisted(() => ({ ...@@ -16,7 +16,6 @@ const { mockCreateGetChatFileURL } = vi.hoisted(() => ({
type MockCreatePreviewOptions = { type MockCreatePreviewOptions = {
expiredHours?: number; expiredHours?: number;
mode?: 'short-proxy' | 'short-redirect' | 'presigned';
}; };
vi.mock('@fastgpt/service/common/s3/sources/chat', () => ({ vi.mock('@fastgpt/service/common/s3/sources/chat', () => ({
......
packages/service/test/core/workflow/dispatch/interactive/formInput.test.ts
...@@ -14,7 +14,6 @@ const { mockCreateGetChatFileURL } = vi.hoisted(() => ({ ...@@ -14,7 +14,6 @@ const { mockCreateGetChatFileURL } = vi.hoisted(() => ({
type MockCreatePreviewOptions = { type MockCreatePreviewOptions = {
expiredHours?: number; expiredHours?: number;
mode?: 'short-proxy' | 'short-redirect' | 'presigned';
}; };
vi.mock('@fastgpt/service/common/s3/sources/chat', () => ({ vi.mock('@fastgpt/service/common/s3/sources/chat', () => ({
......
packages/service/test/core/workflow/dispatch/plugin/runInput.test.ts
...@@ -10,7 +10,6 @@ const { mockCreateGetChatFileURL } = vi.hoisted(() => ({ ...@@ -10,7 +10,6 @@ const { mockCreateGetChatFileURL } = vi.hoisted(() => ({
type MockCreatePreviewOptions = { type MockCreatePreviewOptions = {
expiredHours?: number; expiredHours?: number;
mode?: 'short-proxy' | 'short-redirect' | 'presigned';
}; };
vi.mock('@fastgpt/service/common/s3/sources/chat', () => ({ vi.mock('@fastgpt/service/common/s3/sources/chat', () => ({
......
packages/service/test/env.test.ts
...@@ -7,6 +7,7 @@ const originalEnv = { ...@@ -7,6 +7,7 @@ const originalEnv = {
AGENT_SANDBOX_DISK_MB: process.env.AGENT_SANDBOX_DISK_MB, AGENT_SANDBOX_DISK_MB: process.env.AGENT_SANDBOX_DISK_MB,
FILE_TOKEN_KEY: process.env.FILE_TOKEN_KEY, FILE_TOKEN_KEY: process.env.FILE_TOKEN_KEY,
FILE_DOWNLOAD_PUBLIC_URL_PREFIX: process.env.FILE_DOWNLOAD_PUBLIC_URL_PREFIX, FILE_DOWNLOAD_PUBLIC_URL_PREFIX: process.env.FILE_DOWNLOAD_PUBLIC_URL_PREFIX,
STORAGE_DOWNLOAD_URL_MODE: process.env.STORAGE_DOWNLOAD_URL_MODE,
AES256_SECRET_KEY: process.env.AES256_SECRET_KEY, AES256_SECRET_KEY: process.env.AES256_SECRET_KEY,
INVOKE_TOKEN_SECRET: process.env.INVOKE_TOKEN_SECRET, INVOKE_TOKEN_SECRET: process.env.INVOKE_TOKEN_SECRET,
PRO_URL: process.env.PRO_URL, PRO_URL: process.env.PRO_URL,
...@@ -34,6 +35,7 @@ describe('serviceEnv', () => { ...@@ -34,6 +35,7 @@ describe('serviceEnv', () => {
vi.stubEnv('AGENT_SANDBOX_DISK_MB', originalEnv.AGENT_SANDBOX_DISK_MB); vi.stubEnv('AGENT_SANDBOX_DISK_MB', originalEnv.AGENT_SANDBOX_DISK_MB);
vi.stubEnv('FILE_TOKEN_KEY', originalEnv.FILE_TOKEN_KEY); vi.stubEnv('FILE_TOKEN_KEY', originalEnv.FILE_TOKEN_KEY);
vi.stubEnv('FILE_DOWNLOAD_PUBLIC_URL_PREFIX', originalEnv.FILE_DOWNLOAD_PUBLIC_URL_PREFIX); vi.stubEnv('FILE_DOWNLOAD_PUBLIC_URL_PREFIX', originalEnv.FILE_DOWNLOAD_PUBLIC_URL_PREFIX);
vi.stubEnv('STORAGE_DOWNLOAD_URL_MODE', originalEnv.STORAGE_DOWNLOAD_URL_MODE);
vi.stubEnv('AES256_SECRET_KEY', originalEnv.AES256_SECRET_KEY); vi.stubEnv('AES256_SECRET_KEY', originalEnv.AES256_SECRET_KEY);
vi.stubEnv('INVOKE_TOKEN_SECRET', originalEnv.INVOKE_TOKEN_SECRET); vi.stubEnv('INVOKE_TOKEN_SECRET', originalEnv.INVOKE_TOKEN_SECRET);
vi.stubEnv('PRO_URL', originalEnv.PRO_URL); vi.stubEnv('PRO_URL', originalEnv.PRO_URL);
...@@ -132,6 +134,15 @@ describe('serviceEnv', () => { ...@@ -132,6 +134,15 @@ describe('serviceEnv', () => {
}); });
}); });
it('rejects the removed presigned download mode during service env init', async () => {
vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey');
vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret');
vi.stubEnv('INVOKE_TOKEN_SECRET', validInvokeTokenSecret);
vi.stubEnv('STORAGE_DOWNLOAD_URL_MODE', 'presigned');
await expect(importServiceEnv()).rejects.toThrow('Invalid environment variables');
});
it('uses PRO_TOKEN only when configured or running tests', async () => { it('uses PRO_TOKEN only when configured or running tests', async () => {
vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey'); vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey');
vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret'); vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret');
......
packages/service/test/env.util.test.ts
...@@ -21,26 +21,23 @@ describe('validateS3Env', () => { ...@@ -21,26 +21,23 @@ describe('validateS3Env', () => {
).toThrow('STORAGE_EXTERNAL_ENDPOINT is required when STORAGE_S3_CDN_ENDPOINT is configured'); ).toThrow('STORAGE_EXTERNAL_ENDPOINT is required when STORAGE_S3_CDN_ENDPOINT is configured');
}); });
it.each(['short-redirect', 'presigned'] as const)( it('requires an external endpoint for MinIO short redirect mode', () => {
'requires an external endpoint for MinIO %s mode', expect(() =>
(mode) => { validateS3Env({
expect(() => ...baseEnv,
validateS3Env({ STORAGE_DOWNLOAD_URL_MODE: 'short-redirect'
...baseEnv, })
STORAGE_DOWNLOAD_URL_MODE: mode ).toThrow(
}) 'STORAGE_EXTERNAL_ENDPOINT is required when STORAGE_VENDOR is minio and STORAGE_DOWNLOAD_URL_MODE is short-redirect'
).toThrow( );
`STORAGE_EXTERNAL_ENDPOINT is required when STORAGE_VENDOR is minio and STORAGE_DOWNLOAD_URL_MODE is ${mode}` });
);
}
);
it('allows AWS S3 to use its vendor-managed public endpoint', () => { it('allows AWS S3 to use its vendor-managed public endpoint', () => {
expect(() => expect(() =>
validateS3Env({ validateS3Env({
...baseEnv, ...baseEnv,
STORAGE_VENDOR: 'aws-s3', STORAGE_VENDOR: 'aws-s3',
STORAGE_DOWNLOAD_URL_MODE: 'presigned' STORAGE_DOWNLOAD_URL_MODE: 'short-redirect'
}) })
).not.toThrow(); ).not.toThrow();
}); });
...@@ -69,7 +66,7 @@ describe('validateS3Env', () => { ...@@ -69,7 +66,7 @@ describe('validateS3Env', () => {
expect(() => expect(() =>
validateS3Env({ validateS3Env({
...baseEnv, ...baseEnv,
STORAGE_DOWNLOAD_URL_MODE: 'presigned', STORAGE_DOWNLOAD_URL_MODE: 'short-redirect',
STORAGE_EXTERNAL_ENDPOINT: 'https://s3.example.com', STORAGE_EXTERNAL_ENDPOINT: 'https://s3.example.com',
STORAGE_S3_CDN_ENDPOINT: 'https://cdn.example.com' STORAGE_S3_CDN_ENDPOINT: 'https://cdn.example.com'
}) })
...@@ -83,7 +80,7 @@ describe('validateS3Env', () => { ...@@ -83,7 +80,7 @@ describe('validateS3Env', () => {
validateS3Env({ validateS3Env({
...baseEnv, ...baseEnv,
STORAGE_VENDOR: vendor, STORAGE_VENDOR: vendor,
STORAGE_DOWNLOAD_URL_MODE: 'presigned' STORAGE_DOWNLOAD_URL_MODE: 'short-redirect'
}) })
).not.toThrow(); ).not.toThrow();
} }
......
pnpm-lock.yaml
...@@ -921,9 +921,6 @@ importers: ...@@ -921,9 +921,6 @@ importers:
'@larksuiteoapi/node-sdk': '@larksuiteoapi/node-sdk':
specifier: ^1.59.0 specifier: ^1.59.0
version: 1.67.0(bufferutil@4.1.0)(utf-8-validate@5.0.10) version: 1.67.0(bufferutil@4.1.0)(utf-8-validate@5.0.10)
'@llamaindex/liteparse-wasm':
specifier: 'catalog:'
version: 2.0.8
'@node-rs/jieba': '@node-rs/jieba':
specifier: 'catalog:' specifier: 'catalog:'
version: 2.0.1 version: 2.0.1
pro @ a21a182d
Subproject commit 9216a4fc1430b2576cab6a1828ffe85fcad08590 Subproject commit a21a182ded89dded02e51b56ff45194b38686d15
projects/app/src/pages/api/core/chat/file/presignChatFileGetUrl.ts
...@@ -8,7 +8,7 @@ import { isAuthorizedChatFileS3Key } from '@fastgpt/service/common/s3/sources/ch ...@@ -8,7 +8,7 @@ import { isAuthorizedChatFileS3Key } from '@fastgpt/service/common/s3/sources/ch
import { ChatErrEnum } from '@fastgpt/global/common/error/code/chat'; import { ChatErrEnum } from '@fastgpt/global/common/error/code/chat';
async function handler(req: ApiRequestProps): Promise<string> { async function handler(req: ApiRequestProps): Promise<string> {
const { key, chatId, sourceType, sourceId, mode, outLinkAuthData } = parseApiInput({ const { key, chatId, sourceType, sourceId, outLinkAuthData } = parseApiInput({
req, req,
bodySchema: PresignChatFileGetUrlSchema bodySchema: PresignChatFileGetUrlSchema
}).body; }).body;
...@@ -36,7 +36,7 @@ async function handler(req: ApiRequestProps): Promise<string> { ...@@ -36,7 +36,7 @@ async function handler(req: ApiRequestProps): Promise<string> {
return Promise.reject(ChatErrEnum.unAuthChat); return Promise.reject(ChatErrEnum.unAuthChat);
} }
const { url } = await getS3ChatSource().createGetChatFileURL({ key, external: true, mode }); const { url } = await getS3ChatSource().createGetChatFileURL({ key, external: true });
return url; return url;
} }
......
projects/app/src/web/core/workflow/workflowStartAutoFill.ts
...@@ -9,7 +9,7 @@ import type { ...@@ -9,7 +9,7 @@ import type {
ReferenceItemValueType, ReferenceItemValueType,
ReferenceValueType ReferenceValueType
} from '@fastgpt/global/core/workflow/type/io'; } from '@fastgpt/global/core/workflow/type/io';
import { isEqual } from 'lodash'; import { isEqual } from 'lodash-es';
/** 引用输入是否尚未选择（空占位 / 未选变量），区别于曾经选中但已失效的引用。 */ /** 引用输入是否尚未选择（空占位 / 未选变量），区别于曾经选中但已失效的引用。 */
const isUnsetReferenceValue = (value: unknown) => { const isUnsetReferenceValue = (value: unknown) => {
......
projects/app/test/pages/api/core/chat/file/presignChatFileGetUrl.test.ts
...@@ -69,8 +69,7 @@ describe('presignChatFileGetUrl', () => { ...@@ -69,8 +69,7 @@ describe('presignChatFileGetUrl', () => {
); );
expect(mocks.createGetChatFileURL).toHaveBeenCalledWith({ expect(mocks.createGetChatFileURL).toHaveBeenCalledWith({
key: `chat/${appId}/${uid}/${chatId}/demo.pdf`, key: `chat/${appId}/${uid}/${chatId}/demo.pdf`,
external: true, external: true
mode: undefined
}); });
}); });
...@@ -102,8 +101,7 @@ describe('presignChatFileGetUrl', () => { ...@@ -102,8 +101,7 @@ describe('presignChatFileGetUrl', () => {
); );
expect(mocks.createGetChatFileURL).toHaveBeenCalledWith({ expect(mocks.createGetChatFileURL).toHaveBeenCalledWith({
key: `chat/${ChatSourceTypeEnum.skillEdit}/${skillId}/${uid}/${chatId}/demo.pdf`, key: `chat/${ChatSourceTypeEnum.skillEdit}/${skillId}/${uid}/${chatId}/demo.pdf`,
external: true, external: true
mode: undefined
}); });
}); });
...@@ -142,8 +140,7 @@ describe('presignChatFileGetUrl', () => { ...@@ -142,8 +140,7 @@ describe('presignChatFileGetUrl', () => {
); );
expect(mocks.createGetChatFileURL).toHaveBeenCalledWith({ expect(mocks.createGetChatFileURL).toHaveBeenCalledWith({
key: `chat/${resolvedAppId}/${uid}/${chatId}/demo.pdf`, key: `chat/${resolvedAppId}/${uid}/${chatId}/demo.pdf`,
external: true, external: true
mode: undefined
}); });
}); });
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment