Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 672e5b4a by Archer Committed by GitHub
Options

chore: refresh dependencies and complete object storage compatibility (#7379) 

* chore: refresh workspace dependencies

* submodule

* fix: complete OSS storage compatibility for v4.15.5

* fix: complete COS storage integration compatibility

* fix: align portable storage key limit

* test: expand cross-provider storage integration coverage

* feat: add Cloudflare R2 storage support

* fix: use supported docs code fence language
parent f200b392
Showing with 1343 additions and 331 deletions
document/content/self-host/config/env.en.mdx
......@@ -100,11 +100,12 @@ These variables are mainly validated by `packages/service/env.ts` and apply to `
| Variable | Default | Description |
| --------------------------------------- | ----------------------- | ----------------------------------------------------------------------------------------------------- |
| `STORAGE_VENDOR` | `minio` | Object storage vendor. Supported values are `minio`, `aws-s3`, `cos`, and `oss`. |
| `STORAGE_VENDOR` | `minio` | Object storage vendor. Supported values are `minio`, `aws-s3`, `r2`, `cos`, and `oss`. |
| `STORAGE_PUBLIC_BUCKET` | `fastgpt-public` | Public file bucket. |
| `STORAGE_PRIVATE_BUCKET` | `fastgpt-private` | Private file bucket. |
| `STORAGE_REGION` | `us-east-1` | Object storage region. |
| `STORAGE_EXTERNAL_ENDPOINT` | Empty | Externally reachable object storage endpoint for browsers or external services. |
| `STORAGE_R2_PUBLIC_ENDPOINT` | Empty | Public HTTPS domain for a Cloudflare R2 bucket; required when `STORAGE_VENDOR=r2`. |
| `STORAGE_S3_CDN_ENDPOINT` | Empty | CDN endpoint used for temporary `short-redirect` download URLs. Requires `STORAGE_EXTERNAL_ENDPOINT`. |
| `STORAGE_DOWNLOAD_URL_MODE` | `short-proxy` | Download mode: `short-proxy` or `short-redirect`. External URLs are always FastGPT short links. |
| `STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS` | `300` | Lifetime of the temporary object storage or CDN URL used by `short-redirect`, in seconds. |
......
document/content/self-host/config/env.mdx
......@@ -100,11 +100,12 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说
| 变量 | 默认值 | 说明 |
| --------------------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------- |
| `STORAGE_VENDOR` | `minio` | 对象存储类型，可选 `minio`、`aws-s3`、`cos`、`oss`。 |
| `STORAGE_VENDOR` | `minio` | 对象存储类型，可选 `minio`、`aws-s3`、`r2`、`cos`、`oss`。 |
| `STORAGE_PUBLIC_BUCKET` | `fastgpt-public` | 公开文件 Bucket。 |
| `STORAGE_PRIVATE_BUCKET` | `fastgpt-private` | 私有文件 Bucket。 |
| `STORAGE_REGION` | `us-east-1` | 对象存储 Region。 |
| `STORAGE_EXTERNAL_ENDPOINT` | 空 | 外部可访问的对象存储地址，用于浏览器或外部服务访问。 |
| `STORAGE_R2_PUBLIC_ENDPOINT` | 空 | Cloudflare R2 公开 bucket 的 HTTPS 公网域名；`STORAGE_VENDOR=r2` 时必填。 |
| `STORAGE_S3_CDN_ENDPOINT` | 空 | `short-redirect` 临时下载地址使用的 CDN 地址；配置时必须同时配置 `STORAGE_EXTERNAL_ENDPOINT`。 |
| `STORAGE_DOWNLOAD_URL_MODE` | `short-proxy` | 下载模式，可选 `short-proxy` 或 `short-redirect`；对外始终返回 FastGPT 短链。 |
| `STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS` | `300` | `short-redirect` 模式下临时对象存储/CDN 下载地址的有效时间，单位秒。 |
......
document/content/self-host/config/object-storage.en.mdx
......@@ -8,21 +8,11 @@ import FastGPTLink from '@/components/docs/linkFastGPT';
## Object Storage Configuration
This guide covers environment variable configuration for object storage providers supported by FastGPT, including self-hosted MinIO, AWS S3, Alibaba Cloud OSS, and Tencent Cloud COS.
This guide covers environment variable configuration for object storage providers supported by FastGPT, including self-hosted MinIO, AWS S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS.
### Common Required Environment Variables
FastGPT supports MinIO, AWS S3, Alibaba Cloud OSS, Tencent Cloud COS, and Cloudflare R2. Except for local MinIO development, create `STORAGE_PUBLIC_BUCKET` and `STORAGE_PRIVATE_BUCKET` ahead of time and grant the FastGPT access key read/write permission on both buckets.
> - Temporary credential authentication (e.g., STS) is not supported. Ensure service security on your own.
> - Private bucket reuse is not supported. If you set the private and public bucket names to the same value, ensure the bucket policy is at least **public read, private write**.
- `STORAGE_VENDOR` Enum value. Options: `minio`, `aws-s3`, `oss`, `cos`.
- `STORAGE_REGION` Region where the object storage service is located, e.g., `us-east-1`. Refer to your provider's region list. For self-hosted MinIO, any value works.
- `STORAGE_ACCESS_KEY_ID` Access Key ID for the service credentials
- `STORAGE_SECRET_ACCESS_KEY` Secret Access Key for the service credentials
- `STORAGE_PUBLIC_BUCKET` FastGPT public resource bucket name
- `STORAGE_PRIVATE_BUCKET` FastGPT private resource bucket name
### Transfer Behavior
## Access Modes
- Uploads always go through the FastGPT backend proxy.
- External download URLs are always FastGPT short links. FastGPT no longer returns object storage presigned URLs directly.
......@@ -31,10 +21,11 @@ This guide covers environment variable configuration for object storage provider
- `short-redirect`: FastGPT validates the short link, then redirects to a short-lived object storage or CDN URL. File traffic bypasses FastGPT.
- Self-hosted MinIO requires `STORAGE_EXTERNAL_ENDPOINT` when using `short-redirect`.
### Self-Hosted MinIO and AWS S3
## Provider Configuration
### MinIO
> MinIO has strong AWS S3 protocol support, so MinIO and AWS S3 configurations are nearly identical — differences come from provider-specific or self-hosted requirements.
> In theory, any object storage with S3 protocol support comparable to MinIO will work, such as SeaweedFS, RustFS, etc.
> MinIO has strong AWS S3 protocol support and is suitable for local development and self-hosted deployments. In theory, any object storage with S3 protocol support comparable to MinIO will work, such as SeaweedFS or RustFS.
- `STORAGE_S3_ENDPOINT` Internal connection address. Can be a container ID, e.g., `http://fastgpt-minio:9000`
- `STORAGE_EXTERNAL_ENDPOINT` An address accessible by both **server** and **client** to reach the bucket. Use a fixed host IP or domain name — don't use `127.0.0.1` or `localhost` (containers can't access loopback addresses). This variable does not change the download mode automatically.
......@@ -53,15 +44,27 @@ STORAGE_ACCESS_KEY_ID=your_access_key
STORAGE_SECRET_ACCESS_KEY=your_secret_key
STORAGE_PUBLIC_BUCKET=fastgpt-public
STORAGE_PRIVATE_BUCKET=fastgpt-private
STORAGE_DOWNLOAD_URL_MODE=short-redirect
STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS=300
STORAGE_EXTERNAL_ENDPOINT=https://s3.example.com
STORAGE_S3_CDN_ENDPOINT=https://cdn.example.com
STORAGE_S3_ENDPOINT=http://127.0.0.1:9000
STORAGE_S3_FORCE_PATH_STYLE=true
STORAGE_S3_MAX_RETRIES=3
```
### AWS S3
AWS S3 uses the same S3-compatible variables as MinIO. For production, create separate public and private buckets in advance and configure public-read or CloudFront/custom-domain access only for the public bucket.
```dotenv
STORAGE_VENDOR=aws-s3
STORAGE_REGION=ap-southeast-1
STORAGE_ACCESS_KEY_ID=your_access_key
STORAGE_SECRET_ACCESS_KEY=your_secret_key
STORAGE_PUBLIC_BUCKET=fastgpt-public
STORAGE_PRIVATE_BUCKET=fastgpt-private
STORAGE_S3_ENDPOINT=https://s3.ap-southeast-1.amazonaws.com
STORAGE_S3_FORCE_PATH_STYLE=false
STORAGE_S3_MAX_RETRIES=3
```
### Alibaba Cloud OSS
> - [CORS Configuration](https://help.aliyun.com/zh/oss/user-guide/configure-cross-origin-resource-sharing/?spm=5176.8466032.console-base_help.dexternal.1bcd1450Wau6J6#b58400ec36rqf)
......@@ -71,6 +74,8 @@ STORAGE_S3_MAX_RETRIES=3
- `STORAGE_OSS_SECURE` Whether TLS is enabled. Disable if your domain doesn't have a certificate.
- `STORAGE_OSS_INTERNAL` [Optional] Whether to use internal network access. Enable if your service is also on Alibaba Cloud to save bandwidth. Default: disabled
Set the OSS public bucket to public-read and keep the private bucket private. The same Access Key can be used for both buckets, but the bucket names must remain distinct.
**Complete Example**
```dotenv
......@@ -95,6 +100,8 @@ STORAGE_OSS_INTERNAL=false
- `STORAGE_COS_CNAME_DOMAIN` [Optional] Custom domain, e.g., `your-domain.com`
- `STORAGE_COS_PROXY` [Optional] Proxy server, e.g., `http://localhost:7897`
COS bucket names must include the account App ID suffix, for example `fastgpt-public-1250000000`. Configure anonymous read only for the public bucket and keep the private bucket private.
**Complete Example**
```dotenv
......@@ -109,3 +116,23 @@ STORAGE_COS_USE_ACCELERATE=false
STORAGE_COS_CNAME_DOMAIN=
STORAGE_COS_PROXY=
```
### Cloudflare R2
R2 uses the S3-compatible API. Set `STORAGE_REGION` to `auto` and use the account-level S3 endpoint from Cloudflare as `STORAGE_S3_ENDPOINT`. FastGPT does not rewrite R2 presigned URLs through `STORAGE_S3_CDN_ENDPOINT`; private objects should normally use the default `short-proxy` download mode.
`STORAGE_R2_PUBLIC_ENDPOINT` is required for public objects. Set it to the HTTPS custom domain (or another public HTTPS domain bound to the bucket). This is separate from the R2 S3 API endpoint and must not contain query parameters.
For production, use a custom domain instead of the rate-limited `r2.dev` development URL. Create both R2 buckets in advance; FastGPT checks that production buckets exist at startup and does not create them automatically.
```dotenv
STORAGE_VENDOR=r2
STORAGE_REGION=auto
STORAGE_S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
STORAGE_R2_PUBLIC_ENDPOINT=https://assets.example.com
STORAGE_ACCESS_KEY_ID=<r2-access-key-id>
STORAGE_SECRET_ACCESS_KEY=<r2-secret-access-key>
STORAGE_PUBLIC_BUCKET=<r2-public-bucket>
STORAGE_PRIVATE_BUCKET=<r2-private-bucket>
STORAGE_S3_FORCE_PATH_STYLE=false
```
document/content/self-host/config/object-storage.mdx
......@@ -8,21 +8,11 @@ import FastGPTLink from '@/components/docs/linkFastGPT';
## 对象存储服务配置介绍
这里提供了 FastGPT 目前支持的对象存储厂商，包括自部署的 MinIO、AWS S3、阿里云 OSS 和腾讯云 COS 的环境变量配置说明
这里提供了 FastGPT 目前支持的对象存储厂商，包括自部署的 MinIO、AWS S3、Cloudflare R2、阿里云 OSS 和腾讯云 COS 的环境变量配置说明
### 通用且必需的环境变量
FastGPT 支持 MinIO、AWS S3、Alibaba Cloud OSS、Tencent Cloud COS 和 Cloudflare R2。除 MinIO 本地开发外，建议提前创建 `STORAGE_PUBLIC_BUCKET` 和 `STORAGE_PRIVATE_BUCKET`，并确保 FastGPT 使用的 Access Key 对两个桶都有读写权限。
> - 不支持对临时身份校验密钥的支持，比如 STS 等，请自行保证服务的安全性。
> - 不支持对私有桶的复用，如果把私有桶名和公开桶名设置为同一个的话，请保证存储桶的策略至少为【公开读私有写】
- `STORAGE_VENDOR` 该变量为一个枚举值，可选值为：`minio`、`aws-s3`、`oss` 和 `cos`。
- `STORAGE_REGION` 提供的对象存储服务所在的地区如 `us-east-1` 等，具体请根据服务厂商所提供的地区来填写；如果是自部署的 MinIO 等服务这个值可随意填写。
- `STORAGE_ACCESS_KEY_ID` 服务访问密钥的 Access Key ID
- `STORAGE_SECRET_ACCESS_KEY` 服务访问密钥的 Secret Access Key
- `STORAGE_PUBLIC_BUCKET` FastGPT 公开资源存储桶桶名
- `STORAGE_PRIVATE_BUCKET` FastGPT 私有资源存储桶桶名
### 传输模式说明
## 访问模式说明
- 上传固定走 FastGPT 后端代理。
- 对外下载地址固定为 FastGPT 短链，不再直接返回对象存储预签名长链接。
......@@ -31,9 +21,11 @@ import FastGPTLink from '@/components/docs/linkFastGPT';
- `short-redirect`：FastGPT 校验短链后 302 到短时效对象存储/CDN 地址，文件流量不经过 FastGPT。
- 自部署 MinIO 使用 `short-redirect` 时必须配置 `STORAGE_EXTERNAL_ENDPOINT`。
### 自部署的 MinIO 和 AWS S3
## 提供商配置
### MinIO
> MinIO 这类产品对 AWS S3 协议支持比较完整，因此使用 Minio 和 AWS S3 配置几乎可以是相同的，只是因为服务商提供和自部署的区别，会有额外的配置。> 因此理论上任何对 AWS S3 协议的支持程度至少和 MinIO 相当的对象存储服务都可以使用，比如 SeaweedFS、RustFS 等。
> MinIO 对 AWS S3 协议支持比较完整，适合本地开发和自部署场景。理论上任何对 AWS S3 协议的支持程度至少和 MinIO 相当的对象存储服务也可以使用，比如 SeaweedFS、RustFS。
- `STORAGE_S3_ENDPOINT` 内网连接地址，可以是容器 ID 连接，比如 `http://fastgpt-minio:9000`
- `STORAGE_EXTERNAL_ENDPOINT` 一个**服务器**和**客户端**均可访问到存储桶的地址，可以是固定的宿主机 IP 或者域名，注意不要填写成 127.0.0.1 或者 localhost 等本地回环地址(因为容器里无法使用)。该变量不会自动改变下载模式。
......@@ -52,15 +44,27 @@ STORAGE_ACCESS_KEY_ID=your_access_key
STORAGE_SECRET_ACCESS_KEY=your_secret_key
STORAGE_PUBLIC_BUCKET=fastgpt-public
STORAGE_PRIVATE_BUCKET=fastgpt-private
STORAGE_DOWNLOAD_URL_MODE=short-redirect
STORAGE_DOWNLOAD_REDIRECT_TTL_SECONDS=300
STORAGE_EXTERNAL_ENDPOINT=https://s3.example.com
STORAGE_S3_CDN_ENDPOINT=https://cdn.example.com
STORAGE_S3_ENDPOINT=http://127.0.0.1:9000
STORAGE_S3_FORCE_PATH_STYLE=true
STORAGE_S3_MAX_RETRIES=3
```
### AWS S3
AWS S3 与 MinIO 使用同一套 S3 兼容变量。生产环境建议提前创建 public/private 两个 bucket，并为 public bucket 配置公开读取策略或 CloudFront/自定义域名。
```dotenv
STORAGE_VENDOR=aws-s3
STORAGE_REGION=ap-southeast-1
STORAGE_ACCESS_KEY_ID=your_access_key
STORAGE_SECRET_ACCESS_KEY=your_secret_key
STORAGE_PUBLIC_BUCKET=fastgpt-public
STORAGE_PRIVATE_BUCKET=fastgpt-private
STORAGE_S3_ENDPOINT=https://s3.ap-southeast-1.amazonaws.com
STORAGE_S3_FORCE_PATH_STYLE=false
STORAGE_S3_MAX_RETRIES=3
```
### 阿里云 OSS
> - [跨域配置](https://help.aliyun.com/zh/oss/user-guide/configure-cross-origin-resource-sharing/?spm=5176.8466032.console-base_help.dexternal.1bcd1450Wau6J6#b58400ec36rqf)
......@@ -70,6 +74,8 @@ STORAGE_S3_MAX_RETRIES=3
- `STORAGE_OSS_SECURE` 是否开启了 TLS，如果域名没有认证证书的话，请关闭该选项
- `STORAGE_OSS_INTERNAL`【可选】是否开启内网访问，如果你的服务也在阿里云的话可以开启并节省流量，默认关闭
OSS 的 public bucket 需要设置为公开读，private bucket 保持私有。两个 bucket 可以使用同一组 Access Key，但不要把两个 bucket 配成同名。
**完整示例**
```dotenv
......@@ -94,6 +100,8 @@ STORAGE_OSS_INTERNAL=false
- `STORAGE_COS_CNAME_DOMAIN`【可选】自定义域名，如 `your-domain.com`
- `STORAGE_COS_PROXY`【可选】代理服务器，如 `http://localhost:7897`
COS bucket 名称必须包含账号 App ID 后缀，例如 `fastgpt-public-1250000000`。public bucket 需要配置匿名读，private bucket 保持私有。
**完整示例**
```dotenv
......@@ -108,3 +116,23 @@ STORAGE_COS_USE_ACCELERATE=false
STORAGE_COS_CNAME_DOMAIN=
STORAGE_COS_PROXY=
```
### Cloudflare R2
R2 使用 S3 兼容 API。`STORAGE_REGION` 固定填写 `auto`，`STORAGE_S3_ENDPOINT` 填写 Cloudflare 控制台提供的账户级 S3 endpoint。R2 不支持通过 FastGPT 的 `STORAGE_S3_CDN_ENDPOINT` 重写预签名 URL；私有对象仍建议使用默认的 `short-proxy` 下载模式。
`STORAGE_R2_PUBLIC_ENDPOINT` 必须配置为公开 bucket 的自定义域名（或其他已绑定到该 bucket 的公开 HTTPS 域名），用于生成公开文件 URL。该地址不是 R2 S3 API endpoint，也不应包含查询参数。
R2 生产环境建议使用自定义域名，不建议使用受速率限制的 `r2.dev` 公共开发 URL。R2 public/private bucket 都应提前创建；FastGPT 启动时只检查 bucket 是否存在，不会自动创建生产 bucket。
```dotenv
STORAGE_VENDOR=r2
STORAGE_REGION=auto
STORAGE_S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
STORAGE_R2_PUBLIC_ENDPOINT=https://assets.example.com
STORAGE_ACCESS_KEY_ID=<r2-access-key-id>
STORAGE_SECRET_ACCESS_KEY=<r2-secret-access-key>
STORAGE_PUBLIC_BUCKET=<r2-public-bucket>
STORAGE_PRIVATE_BUCKET=<r2-private-bucket>
STORAGE_S3_FORCE_PATH_STYLE=false
```
document/content/self-host/upgrading/4-15/4155.en.mdx 0 → 100644
---
title: 'V4.15.5 (In Progress)'
description: 'FastGPT V4.15.5 Release Notes'
---
> This document is in progress. The final release may contain additional changes.
## 🚀 New Features
1. Added Cloudflare R2 object storage support, including the R2 S3 API, presigned access, and custom domains for public buckets.
## ⚙️ Improvements
1. Centralized more workspace dependency versions in the pnpm catalog and refreshed the lockfile.
2. Added Chinese and English runtime fonts to the Agent Sandbox image to improve font availability for text and image-related tasks.
3. Updated the OSS adapter to support string uploads required by the `IStorage` contract and to preserve the stored `Content-Type` when OSS does not support response-header overrides.
4. Added a missing-object preflight to the COS adapter so downloads follow the shared error contract.
## 🐛 Fixes
1. Fixed Alibaba Cloud OSS metadata reads that looked for ETags in the wrong response field, causing missing ETags and downstream metadata validation failures.
## 🛠️ Code Improvements
1. Added cross-provider S3 SDK integration tests for MinIO, AWS S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS, covering private/public buckets, public URLs, and real presigned URL access.
document/content/self-host/upgrading/4-15/4155.mdx 0 → 100644
---
title: 'V4.15.5（进行中）'
description: 'FastGPT V4.15.5 更新说明'
---
> 本文档正在整理中，内容以最终发布版本为准。
## 🚀 新增内容
1. 新增 Cloudflare R2 对象存储支持，兼容 R2 S3 API、预签名访问和公开 bucket 自定义域名。
## ⚙️ 优化
1. 统一工作区依赖版本管理，将更多子项目依赖迁移到 pnpm catalog，并刷新锁定文件。
2. Agent Sandbox 镜像补充中英文运行时字体，改善文本和图像相关任务的字体可用性。
3. OSS 适配器支持 `IStorage` 契约中的字符串上传，并在无法覆盖响应 `Content-Type` 的 OSS 场景下沿用对象原始类型。
4. COS 适配器对缺失对象下载进行预检，确保符合统一下载错误契约。
## 🐛 修复
1. 修复阿里云 OSS `getObjectMetadata` 从错误字段读取 ETag，导致 ETag 缺失并触发下游元数据校验失败的问题。
## 🛠️ 代码优化
1. S3 SDK 增加跨 MinIO、AWS S3、Cloudflare R2、OSS 和 COS 的通用集成测试，覆盖 private/public bucket、公开 URL 和预签名 URL 的真实访问。
document/content/self-host/upgrading/4-15/meta.en.json
{
"title": "4.15.x",
"description": "",
"pages": ["4154", "4153", "4152", "4151", "41500", "41507", "41506", "41505", "41504", "41503", "41502", "41501"]
"pages": ["4155", "4154", "4153", "4152", "4151", "41500", "41507", "41506", "41505", "41504", "41503", "41502", "41501"]
}
document/content/self-host/upgrading/4-15/meta.json
{
"title": "4.15.x",
"description": "",
"pages": ["4154", "4153", "4152", "4151", "41500", "41507", "41506", "41505", "41504", "41503", "41502", "41501"]
"pages": ["4155", "4154", "4153", "4152", "4151", "41500", "41507", "41506", "41505", "41504", "41503", "41502", "41501"]
}
document/content/toc.en.mdx
......@@ -154,6 +154,7 @@ description: FastGPT Toc
- [/en/self-host/upgrading/4-14/41481](/en/self-host/upgrading/4-14/41481)
- [/en/self-host/upgrading/4-14/4149](/en/self-host/upgrading/4-14/4149)
- [/en/self-host/upgrading/4-15/41500](/en/self-host/upgrading/4-15/41500)
- [/en/self-host/upgrading/4-15/41501](/en/self-host/upgrading/4-15/41501)
- [/en/self-host/upgrading/4-15/41502](/en/self-host/upgrading/4-15/41502)
- [/en/self-host/upgrading/4-15/41503](/en/self-host/upgrading/4-15/41503)
- [/en/self-host/upgrading/4-15/41504](/en/self-host/upgrading/4-15/41504)
......@@ -164,6 +165,7 @@ description: FastGPT Toc
- [/en/self-host/upgrading/4-15/4152](/en/self-host/upgrading/4-15/4152)
- [/en/self-host/upgrading/4-15/4153](/en/self-host/upgrading/4-15/4153)
- [/en/self-host/upgrading/4-15/4154](/en/self-host/upgrading/4-15/4154)
- [/en/self-host/upgrading/4-15/4155](/en/self-host/upgrading/4-15/4155)
- [/en/self-host/upgrading/outdated/40](/en/self-host/upgrading/outdated/40)
- [/en/self-host/upgrading/outdated/41](/en/self-host/upgrading/outdated/41)
- [/en/self-host/upgrading/outdated/4100](/en/self-host/upgrading/outdated/4100)
......
document/content/toc.mdx
......@@ -167,6 +167,7 @@ description: FastGPT 文档目录
- [/self-host/upgrading/4-15/4152](/self-host/upgrading/4-15/4152)
- [/self-host/upgrading/4-15/4153](/self-host/upgrading/4-15/4153)
- [/self-host/upgrading/4-15/4154](/self-host/upgrading/4-15/4154)
- [/self-host/upgrading/4-15/4155](/self-host/upgrading/4-15/4155)
- [/self-host/upgrading/outdated/40](/self-host/upgrading/outdated/40)
- [/self-host/upgrading/outdated/41](/self-host/upgrading/outdated/41)
- [/self-host/upgrading/outdated/4100](/self-host/upgrading/outdated/4100)
......
document/data/doc-last-modified.json
......@@ -167,16 +167,16 @@
"content/plugin/model-presets.mdx": "2026-06-04T16:10:15+08:00",
"content/plugin/system-tool-development.en.mdx": "2026-07-02T11:54:55+08:00",
"content/plugin/system-tool-development.mdx": "2026-07-02T11:54:55+08:00",
"content/self-host/config/env.en.mdx": "2026-07-22T11:47:48+08:00",
"content/self-host/config/env.mdx": "2026-07-22T11:47:48+08:00",
"content/self-host/config/env.en.mdx": "2026-07-26T21:10:56+08:00",
"content/self-host/config/env.mdx": "2026-07-26T21:10:56+08:00",
"content/self-host/config/model/intro.en.mdx": "2026-06-04T16:10:15+08:00",
"content/self-host/config/model/intro.mdx": "2026-06-04T16:10:15+08:00",
"content/self-host/config/model/minimax.en.mdx": "2026-06-03T10:40:17+08:00",
"content/self-host/config/model/minimax.mdx": "2026-06-03T10:40:17+08:00",
"content/self-host/config/model/siliconCloud.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/config/model/siliconCloud.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/config/object-storage.en.mdx": "2026-07-17T14:00:26+08:00",
"content/self-host/config/object-storage.mdx": "2026-07-17T14:00:26+08:00",
"content/self-host/config/object-storage.en.mdx": "2026-07-26T21:10:56+08:00",
"content/self-host/config/object-storage.mdx": "2026-07-26T21:10:56+08:00",
"content/self-host/config/remote-debug-suite.en.mdx": "2026-06-27T22:05:51+08:00",
"content/self-host/config/remote-debug-suite.mdx": "2026-06-27T22:05:51+08:00",
"content/self-host/config/sandbox/common.en.mdx": "2026-07-02T15:38:53+08:00",
......@@ -227,30 +227,30 @@
"content/self-host/troubleshooting/model-errors.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/troubleshooting/s3-issues.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/troubleshooting/s3-issues.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4120.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4120.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4121.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4121.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4120.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-12/4120.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-12/4121.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-12/4121.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-12/4122.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4122.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4123.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4123.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4124.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4124.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-13/4130.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-13/4130.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-12/4124.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-12/4124.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-13/4130.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-13/4130.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-13/4131.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-13/4131.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-13/4132.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-13/4132.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4140.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4140.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4141.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4141.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/41410.en.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/4-14/41410.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/4-14/41411.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/41411.mdx": "2026-04-26T21:28:27+08:00",
"content/self-host/upgrading/4-13/4132.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-13/4132.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4140.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4140.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4141.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4141.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/41410.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/41410.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/41411.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/41411.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/41412.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/41412.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/41413.en.mdx": "2026-04-26T21:08:47+08:00",
......@@ -285,179 +285,182 @@
"content/self-host/upgrading/4-14/41428.mdx": "2026-07-06T22:53:13+08:00",
"content/self-host/upgrading/4-14/41429.en.mdx": "2026-07-06T22:53:13+08:00",
"content/self-host/upgrading/4-14/41429.mdx": "2026-07-06T22:53:13+08:00",
"content/self-host/upgrading/4-14/4143.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4143.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4144.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4144.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4145.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4145.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/41451.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/41451.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4143.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4143.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4144.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4144.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4145.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4145.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/41451.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/41451.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4146.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4146.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4147.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4147.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4148.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4148.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4147.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4147.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4148.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4148.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/41481.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/41481.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4149.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-14/4149.mdx": "2026-07-01T12:13:58+08:00",
"content/self-host/upgrading/4-15/41500.en.mdx": "2026-07-02T11:56:02+08:00",
"content/self-host/upgrading/4-15/41500.mdx": "2026-07-02T11:56:02+08:00",
"content/self-host/upgrading/4-15/41501.mdx": "2026-07-01T12:13:58+08:00",
"content/self-host/upgrading/4-15/41502.en.mdx": "2026-05-25T11:21:30+08:00",
"content/self-host/upgrading/4-15/41502.mdx": "2026-06-23T13:54:06+08:00",
"content/self-host/upgrading/4-15/41503.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/4-15/41503.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/4-15/41504.en.mdx": "2026-06-10T19:02:59+08:00",
"content/self-host/upgrading/4-15/41504.mdx": "2026-06-15T23:34:43+08:00",
"content/self-host/upgrading/4-15/41505.en.mdx": "2026-07-13T14:52:51+08:00",
"content/self-host/upgrading/4-15/41505.mdx": "2026-07-13T14:52:51+08:00",
"content/self-host/upgrading/4-15/41506.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/4-15/41506.mdx": "2026-07-01T12:13:58+08:00",
"content/self-host/upgrading/4-15/41507.en.mdx": "2026-06-30T17:31:43+08:00",
"content/self-host/upgrading/4-15/41507.mdx": "2026-06-30T17:31:43+08:00",
"content/self-host/upgrading/4-15/4151.en.mdx": "2026-07-21T12:01:28+08:00",
"content/self-host/upgrading/4-15/4151.mdx": "2026-07-21T12:01:28+08:00",
"content/self-host/upgrading/4-15/4152.en.mdx": "2026-07-20T15:11:00+08:00",
"content/self-host/upgrading/4-15/4152.mdx": "2026-07-20T15:11:00+08:00",
"content/self-host/upgrading/4-14/4149.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-14/4149.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41500.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41500.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41501.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41501.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41502.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41502.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41503.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41503.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41504.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41504.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41505.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41505.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41506.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41506.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41507.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/41507.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/4151.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/4151.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/4152.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/4152.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/4153.en.mdx": "2026-07-18T12:51:59+08:00",
"content/self-host/upgrading/4-15/4153.mdx": "2026-07-18T12:51:59+08:00",
"content/self-host/upgrading/4-15/4154.en.mdx": "2026-07-22T11:49:20+08:00",
"content/self-host/upgrading/4-15/4154.mdx": "2026-07-22T11:49:20+08:00",
"content/self-host/upgrading/outdated/40.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/40.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/41.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/41.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4100.en.mdx": "2026-06-09T16:03:58+08:00",
"content/self-host/upgrading/outdated/4100.mdx": "2026-06-09T16:03:58+08:00",
"content/self-host/upgrading/outdated/4101.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4101.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4110.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4110.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/4-15/4154.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/4154.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/4-15/4155.en.mdx": "2026-07-26T21:10:56+08:00",
"content/self-host/upgrading/4-15/4155.mdx": "2026-07-26T21:10:56+08:00",
"content/self-host/upgrading/outdated/40.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/40.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/41.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/41.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4100.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4100.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4101.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4101.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4110.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4110.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4111.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4111.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/42.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/42.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/421.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/421.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/43.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/43.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/44.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/44.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/441.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/441.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/442.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/442.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/445.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/445.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/43.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/43.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/44.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/44.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/441.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/441.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/442.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/442.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/445.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/445.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/446.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/446.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/447.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/447.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/447.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/447.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/45.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/45.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/451.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/451.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/451.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/451.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/452.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/452.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/46.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/46.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/46.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/46.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/461.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/461.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/462.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/462.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/463.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/463.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/464.en.mdx": "2026-06-22T11:01:59+08:00",
"content/self-host/upgrading/outdated/464.mdx": "2026-06-22T11:01:59+08:00",
"content/self-host/upgrading/outdated/465.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/465.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/466.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/466.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/467.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/467.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/468.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/468.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/469.en.mdx": "2026-06-22T11:01:59+08:00",
"content/self-host/upgrading/outdated/469.mdx": "2026-06-22T11:01:59+08:00",
"content/self-host/upgrading/outdated/47.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/47.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/471.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/471.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/462.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/462.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/463.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/463.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/464.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/464.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/465.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/465.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/466.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/466.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/467.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/467.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/468.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/468.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/469.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/469.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/47.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/47.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/471.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/471.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/48.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/48.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/481.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/481.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4810.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4810.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4811.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4811.mdx": "2026-06-23T13:54:06+08:00",
"content/self-host/upgrading/outdated/4812.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4812.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4813.en.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/outdated/4813.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/outdated/481.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/481.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4810.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4810.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4811.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4811.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4812.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4812.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4813.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4813.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4814.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4814.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4815.en.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/outdated/4815.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/outdated/4816.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/4816.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/4817.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4817.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4818.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4818.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4819.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4819.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/482.en.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/482.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/4820.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4820.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4815.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4815.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4816.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4816.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4817.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4817.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4818.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4818.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4819.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4819.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/482.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/482.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4820.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4820.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4821.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4821.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4822.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4822.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4823.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4823.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4822.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4822.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4823.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4823.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/483.en.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/483.mdx": "2026-05-24T00:53:50+08:00",
"content/self-host/upgrading/outdated/484.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/484.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/485.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/485.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/486.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/486.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/484.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/484.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/485.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/485.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/486.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/486.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/487.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/487.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/488.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/488.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/489.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/489.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/490.en.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/490.mdx": "2026-06-30T22:10:03+08:00",
"content/self-host/upgrading/outdated/491.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/491.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/488.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/488.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/489.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/489.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/490.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/490.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/491.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/491.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4910.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4910.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4911.en.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/outdated/4911.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/outdated/4912.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4912.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4911.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4911.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4912.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4912.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/4913.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4913.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4914.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/4914.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/492.en.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/outdated/492.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/outdated/492.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/492.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/493.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/493.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/494.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/494.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/494.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/494.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/495.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/495.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/496.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/496.mdx": "2026-06-30T11:25:38+08:00",
"content/self-host/upgrading/outdated/496.en.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/496.mdx": "2026-07-25T00:27:20+08:00",
"content/self-host/upgrading/outdated/497.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/497.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/outdated/498.en.mdx": "2026-04-26T21:08:47+08:00",
......@@ -466,6 +469,6 @@
"content/self-host/upgrading/outdated/499.mdx": "2026-05-07T15:06:40+08:00",
"content/self-host/upgrading/upgrade-intruction.en.mdx": "2026-04-26T21:08:47+08:00",
"content/self-host/upgrading/upgrade-intruction.mdx": "2026-04-26T21:08:47+08:00",
"content/toc.en.mdx": "2026-07-21T13:14:50+08:00",
"content/toc.mdx": "2026-07-21T13:14:50+08:00"
}
"content/toc.en.mdx": "2026-07-26T18:37:23+08:00",
"content/toc.mdx": "2026-07-26T18:37:23+08:00"
}
\ No newline at end of file
document/package.json
......@@ -47,7 +47,7 @@
"@types/node": "24.0.13",
"@types/react": "^19.1.8",
"@types/react-dom": "^19.1.6",
"postcss": "^8.5.10",
"postcss": "catalog:",
"tailwindcss": "^4.1.11",
"textlint": "catalog:lint",
"textlint-plugin-mdx": "catalog:lint",
......
packages/service/common/s3/buckets/private.ts
......@@ -2,6 +2,7 @@ import { S3BaseBucket } from './base';
import { createDefaultStorageOptions } from '../config/constants';
import {
type IAwsS3CompatibleStorageOptions,
type IR2StorageOptions,
createStorage,
type ICosStorageOptions,
type IOssStorageOptions,
......@@ -51,6 +52,23 @@ export class S3PrivateBucket extends S3BaseBucket {
endpoint: externalEndpoint
}
};
} else if (vendor === 'r2') {
const config = {
region,
vendor,
credentials,
endpoint: storageOptions.endpoint,
maxRetries: storageOptions.maxRetries,
forcePathStyle: false,
publicAccessExtraSubPath: storageOptions.publicAccessExtraSubPath
} as Omit<IR2StorageOptions, 'bucket'>;
return {
config,
externalConfig: {
...config,
endpoint: externalEndpoint || storageOptions.endpoint
}
};
} else if (vendor === 'cos') {
return {
config: {
......
packages/service/common/s3/buckets/public.ts
......@@ -2,6 +2,7 @@ import { S3BaseBucket } from './base';
import { createDefaultStorageOptions } from '../config/constants';
import {
type IAwsS3CompatibleStorageOptions,
type IR2StorageOptions,
type ICosStorageOptions,
type IOssStorageOptions,
createStorage,
......@@ -52,6 +53,24 @@ export class S3PublicBucket extends S3BaseBucket {
endpoint: externalEndpoint
}
};
} else if (vendor === 'r2') {
const config = {
region,
vendor,
credentials,
endpoint: storageOptions.endpoint,
maxRetries: storageOptions.maxRetries,
forcePathStyle: false,
publicEndpoint: storageOptions.publicEndpoint,
publicAccessExtraSubPath: storageOptions.publicAccessExtraSubPath
} as Omit<IR2StorageOptions, 'bucket'>;
return {
config,
externalConfig: {
...config,
endpoint: externalEndpoint || storageOptions.endpoint
}
};
} else if (vendor === 'cos') {
return {
config: {
......@@ -86,7 +105,7 @@ export class S3PublicBucket extends S3BaseBucket {
const client = createStorage({ bucket: publicBucket, ...config });
let externalClient: ReturnType<typeof createStorage> | undefined = undefined;
if (externalEndpoint) {
if (externalEndpoint || vendor === 'r2') {
externalClient = createStorage({
bucket: publicBucket,
...externalConfig
......
packages/service/common/s3/config/constants.ts
......@@ -2,6 +2,7 @@ import type {
IAwsS3CompatibleStorageOptions,
ICosStorageOptions,
IOssStorageOptions,
IR2StorageOptions,
IStorageOptions
} from '@fastgpt-sdk/storage';
import { serviceEnv } from '../../../env';
......@@ -20,6 +21,7 @@ type BucketStorageOptions = {
publicBucket: string;
privateBucket: string;
externalEndpoint?: string;
publicEndpoint?: string;
};
const storageRegion = serviceEnv.STORAGE_REGION;
......@@ -39,7 +41,8 @@ const storagePublicAccessExtraSubPath = serviceEnv.STORAGE_PUBLIC_ACCESS_EXTRA_S
const bucketStorageOptions = {
publicBucket: S3Buckets.public,
privateBucket: S3Buckets.private,
externalEndpoint: storageExternalEndpoint
externalEndpoint: storageExternalEndpoint,
publicEndpoint: serviceEnv.STORAGE_R2_PUBLIC_ENDPOINT
} satisfies BucketStorageOptions;
const awsCompatibleSharedOptions = {
......@@ -80,6 +83,22 @@ export function createDefaultStorageOptions() {
} satisfies Omit<IAwsS3CompatibleStorageOptions, 'bucket'> & BucketStorageOptions;
}
case 'r2': {
return {
vendor: 'r2',
endpoint: storageS3Endpoint,
region: storageRegion,
credentials: {
accessKeyId: serviceEnv.STORAGE_ACCESS_KEY_ID,
secretAccessKey: serviceEnv.STORAGE_SECRET_ACCESS_KEY
},
forcePathStyle: false,
...bucketStorageOptions,
maxRetries: serviceEnv.STORAGE_S3_MAX_RETRIES,
publicAccessExtraSubPath: storagePublicAccessExtraSubPath
} satisfies Omit<IR2StorageOptions, 'bucket'> & BucketStorageOptions;
}
case 'cos': {
return {
vendor: 'cos',
......@@ -120,7 +139,7 @@ export function createDefaultStorageOptions() {
}
export function replaceS3UrlWithCdnEndpoint(url: string) {
if (!storageS3CdnEndpoint) {
if (!storageS3CdnEndpoint || storageVendor === 'r2') {
return url;
}
......
packages/service/common/security/fileUrlValidator.ts
......@@ -24,6 +24,7 @@ const systemWhiteList = (() => {
const list: string[] = [];
appendHost({ list, value: serviceEnv.STORAGE_S3_ENDPOINT, allowRawHost: true });
appendHost({ list, value: serviceEnv.STORAGE_EXTERNAL_ENDPOINT });
appendHost({ list, value: serviceEnv.STORAGE_R2_PUBLIC_ENDPOINT });
appendHost({ list, value: serviceEnv.STORAGE_S3_CDN_ENDPOINT });
appendHost({ list, value: serviceEnv.FE_DOMAIN });
appendHost({ list, value: serviceEnv.PRO_URL });
......
packages/service/env.const.ts
......@@ -7,5 +7,5 @@ export const SYSTEM_STRING_LENGTH_UNIT = 1_000_000;
export const LogLevelSchema = z.enum(['trace', 'debug', 'info', 'warning', 'error', 'fatal']);
// S3
export const StorageVendorSchema = z.enum(['minio', 'aws-s3', 'cos', 'oss']);
export const StorageVendorSchema = z.enum(['minio', 'aws-s3', 'r2', 'cos', 'oss']);
export const StorageCosProtocolSchema = z.enum(['https:', 'http:']);
packages/service/env.ts
......@@ -178,6 +178,7 @@ export const serviceEnv = createEnv({
STORAGE_PRIVATE_BUCKET: z.string().default('fastgpt-private'),
STORAGE_REGION: z.string().default('us-east-1'),
STORAGE_EXTERNAL_ENDPOINT: UrlSchema.optional(),
STORAGE_R2_PUBLIC_ENDPOINT: UrlSchema.optional(),
STORAGE_S3_CDN_ENDPOINT: UrlSchema.optional(),
STORAGE_DOWNLOAD_URL_MODE: z
.enum(['short-proxy', 'short-redirect'])
......
packages/service/env.util.ts
......@@ -28,6 +28,7 @@ type S3Env = {
STORAGE_VENDOR: z.infer<typeof StorageVendorSchema>;
STORAGE_DOWNLOAD_URL_MODE: StorageDownloadUrlMode;
STORAGE_EXTERNAL_ENDPOINT?: string;
STORAGE_R2_PUBLIC_ENDPOINT?: string;
STORAGE_S3_CDN_ENDPOINT?: string;
};
......@@ -43,6 +44,18 @@ export const validateS3Env = (env: S3Env): void => {
);
}
if (env.STORAGE_VENDOR === 'r2' && env.STORAGE_S3_CDN_ENDPOINT) {
throw new Error(
'Invalid S3 environment variables: STORAGE_S3_CDN_ENDPOINT is not supported when STORAGE_VENDOR is r2.'
);
}
if (env.STORAGE_VENDOR === 'r2' && !env.STORAGE_R2_PUBLIC_ENDPOINT) {
throw new Error(
'Invalid S3 environment variables: STORAGE_R2_PUBLIC_ENDPOINT is required when STORAGE_VENDOR is r2.'
);
}
const requiresExternalEndpoint =
env.STORAGE_VENDOR === 'minio' && env.STORAGE_DOWNLOAD_URL_MODE === 'short-redirect';
......
packages/service/test/env.util.test.ts
......@@ -85,6 +85,27 @@ describe('validateS3Env', () => {
).not.toThrow();
}
);
it('requires a public endpoint for R2', () => {
expect(() =>
validateS3Env({
...baseEnv,
STORAGE_VENDOR: 'r2'
})
).toThrow('STORAGE_R2_PUBLIC_ENDPOINT');
});
it('rejects CDN URL rewriting for R2', () => {
expect(() =>
validateS3Env({
...baseEnv,
STORAGE_VENDOR: 'r2',
STORAGE_R2_PUBLIC_ENDPOINT: 'https://assets.example.com',
STORAGE_EXTERNAL_ENDPOINT: 'https://account.r2.cloudflarestorage.com',
STORAGE_S3_CDN_ENDPOINT: 'https://cdn.example.com'
})
).toThrow('STORAGE_S3_CDN_ENDPOINT is not supported');
});
});
describe('env util', () => {
......
pnpm-lock.yaml
This source diff could not be displayed because it is too large. You can view the blob instead.
pnpm-workspace.yaml
......@@ -44,7 +44,7 @@ catalog:
'@xterm/addon-fit': ^0.10.0
'@xterm/xterm': ^5.5.0
ahooks: ^3.9.5
axios: ^1.16.0
axios: ^1.18.0
chalk: ^5.6.2
date-fns: ^3.6.0
dayjs: 1.11.19
......@@ -52,7 +52,8 @@ catalog:
express: ^4
file-type: 21.3.0
gpt-tokenizer: 3.4.0
hono: 4.12.25
hono: 4.12.27
'@hono/node-server': ^2.0.10
i18next: 23.16.8
ipaddr.js: ^2.4.0
js-yaml: ^4.1.1
......@@ -86,6 +87,7 @@ catalog:
vaul: ^1.1.2
vitest: ^4.1.5
zod: ^4
postcss: ^8.5.12
catalogMode: prefer
......
pro @ 7027ac73
Subproject commit d0cdce9e0de80af2ef697526da4958b6882a791f
Subproject commit 7027ac732193b6997eac387f556f23eb2e3bab22
projects/agent-sandbox/Dockerfile
......@@ -48,11 +48,14 @@ ENV DEBIAN_FRONTEND=noninteractive
ENV BUN_INSTALL=/usr/local
ENV PATH="${BUN_INSTALL}/bin:${PATH}"
# 安装 sandbox 运行期基础工具链 (Node.js/Python/Bun/git 等)
# 安装 sandbox 运行期基础工具链和中英文字体
RUN apt-get update && apt-get install -y --no-install-recommends \
bash \
ca-certificates \
curl \
fontconfig \
fonts-dejavu-core \
fonts-noto-cjk \
git \
jq \
python3 \
......
projects/code-sandbox/package.json
......@@ -20,7 +20,7 @@
},
"dependencies": {
"@fastgpt-sdk/otel": "workspace:*",
"@hono/node-server": "^1.13.7",
"@hono/node-server": "catalog:",
"@fastgpt/global": "workspace:*",
"@t3-oss/env-core": "catalog:",
"acorn": "^8.15.0",
......
projects/marketplace/package.json
......@@ -46,7 +46,7 @@
"@types/react-dom": "catalog:",
"eslint": "catalog:lint",
"eslint-config-next": "catalog:lint",
"postcss": "^8",
"postcss": "catalog:",
"tailwindcss": "^3.4.1",
"typescript": "catalog:"
}
......
projects/marketplace/src/env.ts
......@@ -3,7 +3,7 @@ import z from 'zod';
import { BoolSchema, IntSchema } from '@fastgpt/global/common/zod';
const LogLevelSchema = z.enum(['trace', 'debug', 'info', 'warning', 'error', 'fatal']);
const StorageVendorSchema = z.enum(['minio', 'aws-s3', 'cos', 'oss']);
const StorageVendorSchema = z.enum(['minio', 'aws-s3', 'r2', 'cos', 'oss']);
const StorageCosProtocolSchema = z.enum(['https:', 'http:']);
export const marketplaceEnv = createEnv({
......@@ -22,6 +22,7 @@ export const marketplaceEnv = createEnv({
STORAGE_PUBLIC_BUCKET: z.string().optional(),
STORAGE_REGION: z.string().default('us-east-1'),
STORAGE_EXTERNAL_ENDPOINT: z.string().optional(),
STORAGE_R2_PUBLIC_ENDPOINT: z.string().optional(),
STORAGE_S3_ENDPOINT: z.string().optional(),
STORAGE_PUBLIC_ACCESS_EXTRA_SUB_PATH: z.string().optional(),
STORAGE_ACCESS_KEY_ID: z.string().optional(),
......@@ -73,6 +74,7 @@ export const marketplaceStorageEnv = {
normalizeStorageEndpoint(marketplaceEnv.STORAGE_S3_ENDPOINT || marketplaceEnv.S3_ENDPOINT) ||
'http://localhost:9000',
externalEndpoint: normalizeStorageEndpoint(marketplaceEnv.STORAGE_EXTERNAL_ENDPOINT),
publicEndpoint: normalizeStorageEndpoint(marketplaceEnv.STORAGE_R2_PUBLIC_ENDPOINT),
accessKeyId: marketplaceEnv.STORAGE_ACCESS_KEY_ID || marketplaceEnv.S3_ACCESS_KEY || 'minioadmin',
secretAccessKey:
marketplaceEnv.STORAGE_SECRET_ACCESS_KEY || marketplaceEnv.S3_SECRET_KEY || 'minioadmin'
......
projects/marketplace/src/service/s3/index.ts
......@@ -5,12 +5,27 @@ import type { Readable } from 'node:stream';
type StorageClient = ReturnType<typeof createStorage>;
const getStorageConfig = (endpoint = marketplaceStorageEnv.endpoint): IStorageOptions => {
const { vendor, publicBucket, region, accessKeyId, secretAccessKey } = marketplaceStorageEnv;
const { vendor, publicBucket, region, accessKeyId, secretAccessKey, publicEndpoint } =
marketplaceStorageEnv;
const credentials = {
accessKeyId,
secretAccessKey
};
if (vendor === 'r2') {
return {
vendor,
bucket: publicBucket,
region,
credentials,
endpoint,
forcePathStyle: false,
maxRetries: marketplaceEnv.STORAGE_S3_MAX_RETRIES,
publicAccessExtraSubPath: marketplaceEnv.STORAGE_PUBLIC_ACCESS_EXTRA_SUB_PATH,
publicEndpoint
};
}
if (vendor === 'minio' || vendor === 'aws-s3') {
return {
vendor,
......
projects/volume-manager/package.json
......@@ -14,7 +14,7 @@
"pnpm": "10.x"
},
"dependencies": {
"@hono/node-server": "^1.13.7",
"@hono/node-server": "catalog:",
"hono": "catalog:",
"undici": "catalog:",
"zod": "catalog:"
......
sdk/storage/.env.example
......@@ -8,6 +8,9 @@ STORAGE_TEST_MINIO_ENDPOINT=http://127.0.0.1:9000
STORAGE_TEST_MINIO_REGION=us-east-1
STORAGE_TEST_MINIO_ACCESS_KEY_ID=minioadmin
STORAGE_TEST_MINIO_SECRET_ACCESS_KEY=minioadmin
STORAGE_TEST_MINIO_PUBLIC_BUCKET=
STORAGE_TEST_MINIO_PRIVATE_BUCKET=
STORAGE_TEST_MINIO_PUBLIC_ENDPOINT=
# AWS S3 or another S3-compatible endpoint
STORAGE_TEST_AWS_S3_ENABLED=false
......@@ -18,6 +21,16 @@ STORAGE_TEST_AWS_S3_ACCESS_KEY_ID=
STORAGE_TEST_AWS_S3_SECRET_ACCESS_KEY=
STORAGE_TEST_AWS_S3_FORCE_PATH_STYLE=false
# Cloudflare R2
STORAGE_TEST_R2_ENABLED=false
STORAGE_TEST_R2_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
STORAGE_TEST_R2_REGION=auto
STORAGE_TEST_R2_ACCESS_KEY_ID=
STORAGE_TEST_R2_SECRET_ACCESS_KEY=
STORAGE_TEST_R2_PUBLIC_BUCKET=
STORAGE_TEST_R2_PRIVATE_BUCKET=
STORAGE_TEST_R2_PUBLIC_ENDPOINT=
# Alibaba Cloud OSS
STORAGE_TEST_OSS_ENABLED=false
STORAGE_TEST_OSS_BUCKET=
......@@ -25,6 +38,9 @@ STORAGE_TEST_OSS_ENDPOINT=
STORAGE_TEST_OSS_REGION=
STORAGE_TEST_OSS_ACCESS_KEY_ID=
STORAGE_TEST_OSS_SECRET_ACCESS_KEY=
STORAGE_TEST_OSS_PUBLIC_BUCKET=
STORAGE_TEST_OSS_PRIVATE_BUCKET=
STORAGE_TEST_OSS_PUBLIC_ENDPOINT=
# Tencent Cloud COS
STORAGE_TEST_COS_ENABLED=false
......@@ -34,3 +50,6 @@ STORAGE_TEST_COS_REGION=
STORAGE_TEST_COS_APP_ID=
STORAGE_TEST_COS_ACCESS_KEY_ID=
STORAGE_TEST_COS_SECRET_ACCESS_KEY=
STORAGE_TEST_COS_PUBLIC_BUCKET=
STORAGE_TEST_COS_PRIVATE_BUCKET=
STORAGE_TEST_COS_PUBLIC_ENDPOINT=
sdk/storage/README.md
......@@ -185,7 +185,7 @@ await fetch(putUrl, {
## 注意事项
- **key 使用统一规范**：所有 adapter 都在远端请求前要求 1 - 850 UTF-8 bytes，拒绝前导 `/`、反斜线、连续 `//`、控制字符和 `.`/`..` 路径段；空格及 `+ # & % ?`、中文、emoji 可正常使用。
- **key 使用统一规范**：所有 adapter 都在远端请求前要求 1 - 800 UTF-8 bytes，拒绝前导 `/`、反斜线、连续 `//`、控制字符和 `.`/`..` 路径段；空格及 `+ # & % ?`、中文、emoji 可正常使用。
- **按前缀删除是高危操作**：`prefix` 必须是非空字符串；强烈建议使用业务隔离前缀（例如 `team/{teamId}/`），避免误删整桶。
- **metadata 厂商差异**：不同厂商对元数据 key 前缀/大小写/可用字符/大小限制不同，建议使用简单 ASCII key，并控制总体大小。
- **流式下载/上传**：大文件建议使用 `Readable`，减少内存峰值。
......@@ -212,10 +212,10 @@ pnpm --filter @fastgpt-sdk/storage test:integration:minio
集成测试分为两层：
- `test/integration/common`：18 个 `IStorage` 通用契约，每个启用的 provider 都运行完全相同的用例。
- `test/integration/common`：26 个 `IStorage` 通用契约，每个启用的 provider 都运行完全相同的用例，包含 ETag、800 字节 Unicode key、预签名 headers、流式取消，以及 1000 条分页/批量删除边界。
- `test/integration/minio`：11 个 MinIO 专项用例，覆盖中断运行后的桶重建、400/1000 条分页边界、URL 编码、公共策略、真实 HTTP socket 超时，以及等待响应头和读取响应体时的下载取消。
- `test/integration/transport`：2 个无需云凭证的 OSS/COS 真实 socket 取消用例。
每个 provider 使用配置中的固定专用测试桶。每次 suite 启动时，harness 会先清空并删除可能由上次失败运行遗留的同名桶，再重新创建；结束时也会清理。不要对同一组测试配置并发运行集成测试。未启用的 provider 会被跳过。
每个 provider 使用配置中的固定专用测试桶。云端 provider 每次 suite 启动和结束时只清理该桶中的对象并保留空桶，避免全局 bucket 名称删除后的最终一致性窗口；MinIO 专项测试仍会删除并重建桶以验证自动创建行为。不要对同一组测试配置并发运行集成测试。未启用的 provider 会被跳过。
发布前会执行 `prepublishOnly` 自动构建产物到 `dist/`。
sdk/storage/src/adapters/aws-s3.adapter.ts
......@@ -10,7 +10,7 @@ import {
PutObjectCommand,
S3Client
} from '@aws-sdk/client-s3';
import type { IAwsS3CompatibleStorageOptions, IStorage } from '../interface';
import type { IAwsS3CompatibleStorageOptions, IR2StorageOptions, IStorage } from '../interface';
import type {
UploadObjectParams,
UploadObjectResult,
......@@ -64,8 +64,8 @@ export class AwsS3StorageAdapter implements IStorage {
return this.options.bucket;
}
constructor(protected readonly options: IAwsS3CompatibleStorageOptions) {
if (options.vendor !== 'aws-s3' && options.vendor !== 'minio') {
constructor(protected readonly options: IAwsS3CompatibleStorageOptions | IR2StorageOptions) {
if (options.vendor !== 'aws-s3' && options.vendor !== 'minio' && options.vendor !== 'r2') {
throw new Error('Invalid storage vendor');
}
......@@ -74,7 +74,13 @@ export class AwsS3StorageAdapter implements IStorage {
credentials: options.credentials,
endpoint: options.endpoint,
forcePathStyle: options.forcePathStyle,
maxAttempts: options.maxRetries
maxAttempts: options.maxRetries,
...(options.vendor === 'r2'
? {
requestChecksumCalculation: 'WHEN_REQUIRED' as const,
responseChecksumValidation: 'WHEN_REQUIRED' as const
}
: {})
});
}
......@@ -389,7 +395,13 @@ export class AwsS3StorageAdapter implements IStorage {
const encodedKey = encodeObjectKeyPath(key);
let url: string;
if (this.options.forcePathStyle) {
if (this.options.publicEndpoint) {
const endpoint = new URL(this.options.publicEndpoint);
if (endpoint.search || endpoint.hash) {
throw new Error('publicEndpoint must not contain query or hash');
}
url = `${endpoint.toString().replace(/\/+$/, '')}/${encodedKey}`;
} else if (this.options.forcePathStyle) {
if (this.options.publicAccessExtraSubPath) {
url = `${this.options.endpoint}/${trim(this.options.publicAccessExtraSubPath, '/')}/${this.options.bucket}/${encodedKey}`;
} else {
......
sdk/storage/src/adapters/cos.adapter.ts
......@@ -29,7 +29,7 @@ import type {
GeneratePublicGetUrlResult
} from '../types';
import { PassThrough } from 'node:stream';
import { camelCase, isError, isNotNil, kebabCase } from 'es-toolkit';
import { camelCase, chunk, isError, isNotNil, kebabCase } from 'es-toolkit';
import { DEFAULT_PRESIGNED_URL_EXPIRED_SECONDS } from '../constants';
import {
bindAbortSignalToReadable,
......@@ -216,6 +216,22 @@ export class CosStorageAdapter implements IStorage {
assertStorageObjectKey(params.key);
throwIfStorageDownloadAborted(params.abortSignal);
// COS returns the output stream before reporting a missing-object error.
// Preflight with HEAD so IStorage rejects missing downloads consistently.
await new Promise<void>((resolve, reject) => {
this.client.headObject(
{
Bucket: this.options.bucket,
Region: this.options.region,
Key: params.key
},
(err) => {
if (err) return reject(this.handleCosError(err));
resolve();
}
);
});
const passThrough = new PassThrough();
bindAbortSignalToReadable({ readable: passThrough, abortSignal: params.abortSignal });
......@@ -277,24 +293,30 @@ export class CosStorageAdapter implements IStorage {
};
}
const result = await new Promise<COS.DeleteMultipleObjectResult>((resolve, reject) => {
this.client.deleteMultipleObject(
{
Bucket: this.options.bucket,
Region: this.options.region,
Objects: keys.map((key) => ({ Key: key }))
},
(err, data) => {
if (err) {
return reject(this.handleCosError(err));
// COS 单次 DeleteMultipleObject 最多接受 1000 个对象。
const failedKeys: StorageObjectKey[] = [];
for (const keyChunk of chunk(keys, 1000)) {
const result = await new Promise<COS.DeleteMultipleObjectResult>((resolve, reject) => {
this.client.deleteMultipleObject(
{
Bucket: this.options.bucket,
Region: this.options.region,
Objects: keyChunk.map((key) => ({ Key: key }))
},
(err, data) => {
if (err) {
return reject(this.handleCosError(err));
}
resolve(data);
}
resolve(data);
}
);
});
);
});
failedKeys.push(...(result.Error?.map((e) => e.Key).filter(isNotNil) ?? []));
}
return {
keys: result.Error.map((e) => e.Key).filter(isNotNil),
keys: failedKeys,
bucket: this.options.bucket
};
}
......
sdk/storage/src/adapters/oss.adapter.ts
......@@ -29,7 +29,7 @@ import type {
GeneratePublicGetUrlResult
} from '../types';
import type { Readable } from 'node:stream';
import { camelCase, difference, kebabCase } from 'es-toolkit';
import { camelCase, chunk, difference, kebabCase } from 'es-toolkit';
import { DEFAULT_PRESIGNED_URL_EXPIRED_SECONDS } from '../constants';
import {
bindAbortSignalToReadable,
......@@ -108,11 +108,15 @@ export class OssStorageAdapter implements IStorage {
}
const headers = result.res.headers as Record<string, string>;
const etag =
headers.etag ?? Object.entries(headers).find(([key]) => key.toLowerCase() === 'etag')?.[1];
const normalizedEtag = etag?.replace(/"/g, '');
return {
key,
metadata,
etag: headers['etag'] ? headers['etag'].replace(/"/g, '') : undefined,
etag: normalizedEtag,
bucket: this.options.bucket,
contentType: headers['content-type'],
contentLength: headers['content-length'] ? Number(headers['content-length']) : undefined
......@@ -136,6 +140,8 @@ export class OssStorageAdapter implements IStorage {
async uploadObject(params: UploadObjectParams): Promise<UploadObjectResult> {
const { key, body, contentType, contentLength, contentDisposition, metadata } = params;
assertStorageObjectKey(key);
// ali-oss 会把字符串 body 当成本地文件路径，因此先转成字节以满足 IStorage 契约。
const uploadBody = typeof body === 'string' ? Buffer.from(body) : body;
const headers: Record<string, any> = {
'x-oss-storage-class': 'Standard',
......@@ -153,7 +159,7 @@ export class OssStorageAdapter implements IStorage {
}
}
await this.client.put(key, body, {
await this.client.put(key, uploadBody, {
headers,
mime: contentType,
meta
......@@ -204,31 +210,36 @@ export class OssStorageAdapter implements IStorage {
};
}
// verbose 模式会返回成功删除的 key；quiet 全成功时响应为空，无法与失败区分。
const result = await this.client.deleteMulti(keys, { quiet: false });
const deletedKeys = (() => {
const deletedItems: unknown = result.deleted;
if (!Array.isArray(deletedItems)) return [];
const normalizedKeys: string[] = [];
for (const item of deletedItems) {
if (typeof item === 'string') {
normalizedKeys.push(item);
continue;
}
// ali-oss 的类型声明是 string[]，但标准 OSS XML 在运行时解析为 { Key }[]。
if (item && typeof item === 'object' && 'Key' in item && typeof item.Key === 'string') {
normalizedKeys.push(item.Key);
continue;
// OSS 单次 DeleteMultipleObjects 最多接受 1000 个 key；verbose 模式会返回成功删除的 key。
const failedKeys: StorageObjectKey[] = [];
for (const keyChunk of chunk(keys, 1000)) {
const result = await this.client.deleteMulti(keyChunk, { quiet: false });
const deletedKeys = (() => {
const deletedItems: unknown = result.deleted;
if (!Array.isArray(deletedItems)) return [];
const normalizedKeys: string[] = [];
for (const item of deletedItems) {
if (typeof item === 'string') {
normalizedKeys.push(item);
continue;
}
// ali-oss 的类型声明是 string[]，但标准 OSS XML 在运行时解析为 { Key }[]。
if (item && typeof item === 'object' && 'Key' in item && typeof item.Key === 'string') {
normalizedKeys.push(item.Key);
continue;
}
return [];
}
return [];
}
return normalizedKeys;
})();
return normalizedKeys;
})();
failedKeys.push(...difference(keyChunk, deletedKeys));
}
return {
bucket: this.options.bucket,
keys: difference(keys, deletedKeys)
keys: failedKeys
};
}
......@@ -314,20 +325,15 @@ export class OssStorageAdapter implements IStorage {
}
async generatePresignedGetUrl(params: PresignedGetUrlParams): Promise<PresignedGetUrlResult> {
const { key, expiredSeconds, responseContentType } = params;
const { key, expiredSeconds } = params;
assertStorageObjectKey(key);
const expiresIn = expiredSeconds ? expiredSeconds : DEFAULT_PRESIGNED_URL_EXPIRED_SECONDS;
// OSS 不支持 response-content-type 覆盖，会返回 InvalidRequest；
// 保持签名 URL 有效并沿用对象保存时的 Content-Type。
const url = this.client.signatureUrl(key, {
method: 'GET',
expires: expiresIn,
...(responseContentType
? {
response: {
'content-type': responseContentType
}
}
: {})
expires: expiresIn
});
return {
......
sdk/storage/src/adapters/r2.adapter.ts 0 → 100644
import type { IR2StorageOptions, IStorage } from '../interface';
import { AwsS3StorageAdapter } from './aws-s3.adapter';
export class R2StorageAdapter extends AwsS3StorageAdapter implements IStorage {
constructor(options: IR2StorageOptions) {
super(options);
}
}
sdk/storage/src/assert.ts
import { InvalidStorageObjectKeyError, type InvalidStorageObjectKeyReason } from './errors';
/** 四个 adapter 都可移植的对象 key 最大 UTF-8 字节数。 */
export const MAX_STORAGE_OBJECT_KEY_UTF8_BYTES = 850;
export const MAX_STORAGE_OBJECT_KEY_UTF8_BYTES = 800;
function throwInvalidStorageObjectKey({
field,
......
sdk/storage/src/factory.ts
......@@ -2,11 +2,17 @@ import { AwsS3StorageAdapter } from './adapters/aws-s3.adapter';
import { CosStorageAdapter } from './adapters/cos.adapter';
import { MinioStorageAdapter } from './adapters/minio.adapter';
import { OssStorageAdapter } from './adapters/oss.adapter';
import { R2StorageAdapter } from './adapters/r2.adapter';
import type { IStorageOptions } from './interface';
export function createStorage(
options: IStorageOptions
): AwsS3StorageAdapter | OssStorageAdapter | CosStorageAdapter | MinioStorageAdapter {
):
| AwsS3StorageAdapter
| OssStorageAdapter
| CosStorageAdapter
| MinioStorageAdapter
| R2StorageAdapter {
switch (options.vendor) {
case 'aws-s3': {
return new AwsS3StorageAdapter(options);
......@@ -24,6 +30,10 @@ export function createStorage(
return new MinioStorageAdapter(options);
}
case 'r2': {
return new R2StorageAdapter(options);
}
default: {
throw new Error(`Unsupported storage vendor: ${String((options as any)?.vendor)}`);
}
......
sdk/storage/src/index.ts
......@@ -5,6 +5,7 @@ export type {
IStorage,
IStorageOptions,
IAwsS3CompatibleStorageOptions,
IR2StorageOptions,
IOssStorageOptions,
ICosStorageOptions,
ICommonStorageOptions
......@@ -49,6 +50,7 @@ export {
assertStorageObjectPrefix
} from './assert';
export { AwsS3StorageAdapter } from './adapters/aws-s3.adapter';
export { R2StorageAdapter } from './adapters/r2.adapter';
export { CosStorageAdapter } from './adapters/cos.adapter';
export { MinioStorageAdapter } from './adapters/minio.adapter';
export { OssStorageAdapter } from './adapters/oss.adapter';
......
sdk/storage/src/interface.ts
......@@ -76,6 +76,11 @@ export interface ICommonStorageOptions {
* - 用于在公共访问时添加额外的前缀，例如 `/sub-path`。
*/
publicAccessExtraSubPath?: string;
/**
* 公共对象访问域名，可选。适用于 R2 自定义域名等不应暴露 bucket 名称的场景。
*/
publicEndpoint?: string;
}
/**
......@@ -127,6 +132,13 @@ export interface IAwsS3CompatibleStorageOptions extends ICommonStorageOptions {
maxRetries?: number;
}
export type IR2StorageOptions = ICommonStorageOptions & {
vendor: 'r2';
endpoint: string;
forcePathStyle?: false;
maxRetries?: number;
};
/**
* 阿里云 OSS 存储配置。
*/
......@@ -241,6 +253,7 @@ export interface IBosStorageOptions extends ICommonStorageOptions {
*/
export type IStorageOptions =
| IAwsS3CompatibleStorageOptions
| IR2StorageOptions
| IOssStorageOptions
| ICosStorageOptions;
......
sdk/storage/src/types.ts
......@@ -24,7 +24,7 @@ export type StorageBucketName = string;
* 说明：
* - 在同一个 bucket 内唯一标识一个对象。
* - 通常形如：`a/b/c.txt`（用 `/` 形成“目录”层级，但对象存储并不是真正的目录结构）。
* - SDK 统一限制为 1 - 850 UTF-8 bytes，并拒绝不可移植的控制字符、反斜线、空路径段和 `.`/`..` 路径段。
* - SDK 统一限制为 1 - 800 UTF-8 bytes，并拒绝不可移植的控制字符、反斜线、空路径段和 `.`/`..` 路径段。
* - 空格、`+`、`#`、`&`、`%`、`?`、中文和 emoji 均为合法字符，由 adapter 在 URL 层编码。
*/
export type StorageObjectKey = string;
......@@ -244,7 +244,10 @@ export type PresignedGetUrlParams = {
key: StorageObjectKey;
/** 过期时间（秒），可选，默认 1800 秒。 */
expiredSeconds?: number;
/** 覆盖下载响应的 Content-Type。 */
/**
* 请求覆盖下载响应的 Content-Type。
* 若底层 provider 不支持响应头覆盖，则沿用对象保存时的 Content-Type。
*/
responseContentType?: string;
};
......
sdk/storage/test/global-setup.ts
......@@ -5,12 +5,16 @@ import fs from 'node:fs';
*/
function getEnvFilePath(): URL | undefined {
const files = ['.env.test.local', '.env.test', '.env.local', '.env'];
const packageRoot = new URL('../', import.meta.url);
return files.map((f) => new URL(f, import.meta.url)).find((p) => fs.existsSync(p));
return files.map((f) => new URL(f, packageRoot)).find((p) => fs.existsSync(p));
}
export function setup() {
process.loadEnvFile(getEnvFilePath());
const envFilePath = getEnvFilePath();
if (envFilePath) {
process.loadEnvFile(envFilePath);
}
}
export function teardown() {
......
sdk/storage/test/integration/common/storage.contract.ts
import { Readable } from 'node:stream';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { InvalidStorageObjectKeyError } from '../../../src/errors';
import {
ValidTestBucketNamePrefixPattern,
type StorageIntegrationContext,
type StorageIntegrationProvider
} from '../providers';
import { createAsciiKeyAtLength } from '../helpers';
import { MAX_STORAGE_OBJECT_KEY_UTF8_BYTES } from '../../../src/assert';
const readBody = async (body: Readable): Promise<Buffer> => {
const chunks: Buffer[] = [];
......@@ -15,6 +17,42 @@ const readBody = async (body: Readable): Promise<Buffer> => {
return Buffer.concat(chunks);
};
const waitForStreamClose = (stream: Readable): Promise<void> =>
new Promise((resolve, reject) => {
const timer = setTimeout(() => reject(new Error('Download stream did not close')), 5000);
stream.once('close', () => {
clearTimeout(timer);
resolve();
});
stream.once('error', () => {
// Abort is expected to surface as a stream error before close.
});
});
/** 以有限并发写入大量对象，避免集成测试自身制造无限请求并发。 */
const uploadInBatches = async ({
context,
keys,
batchSize = 25
}: {
context: StorageIntegrationContext;
keys: string[];
batchSize?: number;
}) => {
for (let index = 0; index < keys.length; index += batchSize) {
await Promise.all(
keys.slice(index, index + batchSize).map((key) =>
context.storage.uploadObject({
key,
body: 'x',
contentType: 'text/plain',
contentLength: 1
})
)
);
}
};
/**
* 对任意 IStorage 实现执行相同的外部行为契约。
* Provider 只负责环境和 bucket 生命周期，断言不依赖厂商 SDK。
......@@ -59,7 +97,7 @@ export const runStorageAdapterContract = (provider: StorageIntegrationProvider)
contentType: 'text/plain',
contentLength: content.length,
contentDisposition: 'attachment; filename="basic.txt"',
metadata: { traceId: 'contract-basic' }
metadata: { traceId: 'contract-basic', emptyValue: '' }
})
).resolves.toEqual({ bucket: context.bucket, key });
......@@ -75,13 +113,33 @@ export const runStorageAdapterContract = (provider: StorageIntegrationProvider)
key,
contentType: 'text/plain',
contentLength: content.length,
metadata: { traceId: 'contract-basic' }
metadata: { traceId: 'contract-basic', emptyValue: '' }
});
expect(metadata.etag).toBeTruthy();
const download = await context.storage.downloadObject({ key });
expect(download).toMatchObject({ bucket: context.bucket, key });
await expect(readBody(download.body)).resolves.toEqual(content);
const signedGet = await context.storage.generatePresignedGetUrl({ key });
const signedGetResponse = await fetch(signedGet.url);
expect(signedGetResponse.ok).toBe(true);
expect(signedGetResponse.headers.get('content-disposition')).toContain('basic.txt');
});
it('returns a stable etag when metadata is read repeatedly', async () => {
const key = `${context.rootPrefix}object/etag.txt`;
await context.storage.uploadObject({
key,
body: 'etag-contract-content'
});
const firstMetadata = await context.storage.getObjectMetadata({ key });
const secondMetadata = await context.storage.getObjectMetadata({ key });
expect(firstMetadata.etag).toEqual(expect.any(String));
expect(firstMetadata.etag).toBe(secondMetadata.etag);
});
it('accepts Readable and string upload bodies', async () => {
......@@ -175,10 +233,46 @@ export const runStorageAdapterContract = (provider: StorageIntegrationProvider)
expect(contents).toEqual(entries.map(({ content }) => content));
});
it('round-trips and deletes an object key at the portable 850-byte limit', async () => {
it('lists and deletes a prefix across more than 1000 objects', async () => {
const prefix = `${context.rootPrefix}large-prefix/`;
const keys = Array.from({ length: 1001 }, (_, index) => `${prefix}${index}.txt`);
await uploadInBatches({ context, keys });
const listed = await context.storage.listObjects({ prefix });
expect(new Set(listed.keys)).toEqual(new Set(keys));
await expect(context.storage.deleteObjectsByPrefix({ prefix })).resolves.toEqual({
bucket: context.bucket,
keys: []
});
await expect(context.storage.listObjects({ prefix })).resolves.toEqual({
bucket: context.bucket,
keys: []
});
});
it('accepts a multi-delete request larger than the provider batch limit', async () => {
const prefix = `${context.rootPrefix}large-delete-request/`;
const keys = Array.from({ length: 1001 }, (_, index) => `${prefix}${index}.txt`);
await context.storage.uploadObject({ key: keys[0], body: 'existing' });
await expect(context.storage.deleteObjectsByMultiKeys({ keys })).resolves.toEqual({
bucket: context.bucket,
keys: []
});
await expect(context.storage.checkObjectExists({ key: keys[0] })).resolves.toMatchObject({
exists: false
});
});
it(`round-trips and deletes an object key at the portable ${MAX_STORAGE_OBJECT_KEY_UTF8_BYTES}-byte limit`, async () => {
const keyPrefix = `${context.rootPrefix}long-key/`;
const key = createAsciiKeyAtLength({ prefix: keyPrefix, byteLength: 850 });
expect(Buffer.byteLength(key)).toBe(850);
const maxObjectKeyBytes = MAX_STORAGE_OBJECT_KEY_UTF8_BYTES;
const key = createAsciiKeyAtLength({
prefix: keyPrefix,
byteLength: maxObjectKeyBytes
});
expect(Buffer.byteLength(key)).toBe(maxObjectKeyBytes);
await context.storage.uploadObject({ key, body: 'long-key-content' });
await expect(context.storage.checkObjectExists({ key })).resolves.toMatchObject({
......@@ -195,10 +289,57 @@ export const runStorageAdapterContract = (provider: StorageIntegrationProvider)
});
});
it('round-trips a multibyte object key at the portable byte limit', async () => {
const keyPrefix = `${context.rootPrefix}unicode-key/`;
const asciiKey = createAsciiKeyAtLength({
prefix: keyPrefix,
byteLength: MAX_STORAGE_OBJECT_KEY_UTF8_BYTES
});
// Replace three ASCII bytes with one three-byte CJK character without changing total size.
const key = `${asciiKey.slice(0, -3)}\u4e2d`;
expect(Buffer.byteLength(key)).toBe(MAX_STORAGE_OBJECT_KEY_UTF8_BYTES);
await context.storage.uploadObject({ key, body: 'unicode-boundary' });
const download = await context.storage.downloadObject({ key });
await expect(readBody(download.body)).resolves.toEqual(Buffer.from('unicode-boundary'));
await expect(context.storage.listObjects({ prefix: keyPrefix })).resolves.toMatchObject({
keys: expect.arrayContaining([key])
});
await expect(context.storage.deleteObject({ key })).resolves.toEqual({
bucket: context.bucket,
key
});
});
it('rejects an object key above the portable byte limit before creating an object', async () => {
const prefix = `${context.rootPrefix}too-long/`;
const key = createAsciiKeyAtLength({
prefix,
byteLength: MAX_STORAGE_OBJECT_KEY_UTF8_BYTES + 1
});
await expect(context.storage.uploadObject({ key, body: 'too-long' })).rejects.toMatchObject(
{
name: InvalidStorageObjectKeyError.name,
reason: 'too_long',
actualBytes: MAX_STORAGE_OBJECT_KEY_UTF8_BYTES + 1,
maxBytes: MAX_STORAGE_OBJECT_KEY_UTF8_BYTES
}
);
await expect(context.storage.listObjects({ prefix })).resolves.toEqual({
bucket: context.bucket,
keys: []
});
});
it('lists and copies keys containing path and URL-sensitive characters', async () => {
const sourceKey = `${context.rootPrefix}special/team # & + % ?/\u6587\u4ef6-\ud83d\ude00.txt`;
const targetKey = `${context.rootPrefix}special/copied file.txt`;
await context.storage.uploadObject({ key: sourceKey, body: 'special-content' });
await context.storage.uploadObject({
key: sourceKey,
body: 'special-content',
metadata: { copySource: 'special-contract' }
});
const listed = await context.storage.listObjects({
prefix: `${context.rootPrefix}special/`
......@@ -211,10 +352,16 @@ export const runStorageAdapterContract = (provider: StorageIntegrationProvider)
).resolves.toEqual({ bucket: context.bucket, sourceKey, targetKey });
const copied = await context.storage.downloadObject({ key: targetKey });
await expect(readBody(copied.body)).resolves.toEqual(Buffer.from('special-content'));
await expect(context.storage.getObjectMetadata({ key: targetKey })).resolves.toMatchObject({
metadata: { copySource: 'special-contract' }
});
await expect(context.storage.checkObjectExists({ key: sourceKey })).resolves.toMatchObject({
exists: true
});
});
it('uploads and downloads through presigned URLs', async () => {
const key = `${context.rootPrefix}presigned/file.txt`;
const key = `${context.rootPrefix}presigned/folder name/file #+&%?.txt`;
const content = 'presigned-content';
const put = await context.storage.generatePresignedPutUrl({
key,
......@@ -242,6 +389,10 @@ export const runStorageAdapterContract = (provider: StorageIntegrationProvider)
expect(getResponse.ok).toBe(true);
expect(getResponse.headers.get('content-type')).toContain('text/plain');
await expect(getResponse.text()).resolves.toBe(content);
await expect(context.storage.getObjectMetadata({ key })).resolves.toMatchObject({
contentType: 'text/plain',
metadata: { uploadSource: 'contract' }
});
});
it('generates a public URL that preserves reserved characters inside the key path', () => {
......@@ -255,6 +406,28 @@ export const runStorageAdapterContract = (provider: StorageIntegrationProvider)
expect(url.search).toBe('');
});
it('uploads and reads a public object through the generated access URL', async () => {
const publicStorage = context.publicStorage;
if (!publicStorage) return;
const key = `${context.rootPrefix}public-access/file.txt`;
try {
await publicStorage.uploadObject({
key,
body: 'public-access-content',
contentType: 'text/plain',
contentLength: 21
});
const publicUrl = publicStorage.generatePublicGetUrl({ key }).url;
const response = await fetch(publicUrl);
expect(response.ok).toBe(true);
await expect(response.text()).resolves.toBe('public-access-content');
} finally {
await publicStorage.deleteObject({ key }).catch(() => undefined);
}
});
it('rejects a download that was aborted before dispatch', async () => {
const key = `${context.rootPrefix}abort/file.txt`;
await context.storage.uploadObject({ key, body: 'abort-content' });
......@@ -266,6 +439,35 @@ export const runStorageAdapterContract = (provider: StorageIntegrationProvider)
).rejects.toMatchObject({ name: 'AbortError' });
});
it('closes an in-flight download when aborted after response begins', async () => {
const key = `${context.rootPrefix}abort/in-flight.bin`;
await context.storage.uploadObject({
key,
body: Buffer.alloc(4 * 1024 * 1024, 0x61),
contentType: 'application/octet-stream'
});
const controller = new AbortController();
const { body } = await context.storage.downloadObject({
key,
abortSignal: controller.signal
});
body.on('error', () => {});
const firstChunk = new Promise<void>((resolve, reject) => {
body.once('data', () => {
body.pause();
resolve();
});
body.once('error', reject);
});
await firstChunk;
const closePromise = waitForStreamClose(body);
controller.abort(new Error('integration test aborted'));
await closePromise;
expect(body.destroyed).toBe(true);
});
it('deletes a single object idempotently', async () => {
const key = `${context.rootPrefix}delete/single.txt`;
await context.storage.uploadObject({ key, body: 'delete-me' });
......@@ -369,6 +571,35 @@ export const runStorageAdapterContract = (provider: StorageIntegrationProvider)
).resolves.toEqual({ bucket: context.bucket, keys: [] });
});
it('accepts omitted and empty list prefixes', async () => {
const markerKey = `${context.rootPrefix}list-prefix/marker.txt`;
await context.storage.uploadObject({ key: markerKey, body: 'marker' });
const omittedPrefix = await context.storage.listObjects({});
const emptyPrefix = await context.storage.listObjects({ prefix: '' });
expect(omittedPrefix).toEqual(emptyPrefix);
expect(omittedPrefix.keys).toContain(markerKey);
});
it('round-trips valid path-edge keys without treating them as dot segments', async () => {
const keys = [
`${context.rootPrefix}path-edge/folder/`,
`${context.rootPrefix}path-edge/.hidden`,
`${context.rootPrefix}path-edge/..backup`,
`${context.rootPrefix}path-edge/trailing-space `
];
await uploadInBatches({ context, keys, batchSize: 4 });
const listed = await context.storage.listObjects({
prefix: `${context.rootPrefix}path-edge/`
});
expect(new Set(listed.keys)).toEqual(new Set(keys));
await expect(context.storage.deleteObjectsByMultiKeys({ keys })).resolves.toEqual({
bucket: context.bucket,
keys: []
});
});
it('allows independently created adapters to be destroyed repeatedly', async () => {
const isolatedStorage = context.createStorage();
await expect(isolatedStorage.ensureBucket()).resolves.toMatchObject({
......
sdk/storage/test/integration/helpers.ts
......@@ -29,19 +29,19 @@ export const createAsciiKeyAtLength = ({
};
/**
* 删除已存在的固定集成测试桶，供下次运行重新创建干净环境。
* `DeleteObjectsResult.keys` 是失败项；只要存在失败 key，就保留桶并让测试失败。
* 清空已存在的集成测试 bucket，但保留 bucket 本身。
*
* 云厂商的 bucket 名称通常是全局命名空间，删除后重新创建可能经历较长的最终一致性窗口；
* 真实 provider 测试应复用专用空 bucket，避免测试结果受 bucket 回收延迟影响。
*/
export const removeIntegrationBucketIfExists = async ({
export const clearIntegrationBucketObjects = async ({
storage,
bucketExists,
deleteBucket
bucketExists
}: {
storage: IStorage;
bucketExists: () => Promise<boolean>;
deleteBucket: () => Promise<void>;
}): Promise<void> => {
if (!(await bucketExists())) return;
}): Promise<boolean> => {
if (!(await bucketExists())) return false;
const { keys } = await storage.listObjects({});
if (keys.length > 0) {
......@@ -51,5 +51,19 @@ export const removeIntegrationBucketIfExists = async ({
}
}
return true;
};
/** 清空并删除已存在的集成测试 bucket；MinIO 专项测试用此验证自动建桶。 */
export const removeIntegrationBucketIfExists = async ({
storage,
bucketExists,
deleteBucket
}: {
storage: IStorage;
bucketExists: () => Promise<boolean>;
deleteBucket: () => Promise<void>;
}): Promise<void> => {
if (!(await clearIntegrationBucketObjects({ storage, bucketExists }))) return;
await deleteBucket();
};
sdk/storage/test/integration/minio/minio.integration.test.ts
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import type { MinioStorageAdapter } from '../../../src/adapters/minio.adapter';
import { InvalidStorageObjectKeyError } from '../../../src/errors';
import { MAX_STORAGE_OBJECT_KEY_UTF8_BYTES } from '../../../src/assert';
import { minioIntegrationProvider, type StorageIntegrationContext } from '../providers';
import { createAsciiKeyAtLength } from '../helpers';
......@@ -95,15 +96,18 @@ describe.skipIf(!minioIntegrationProvider.enabled).sequential('MinIO-specific in
});
});
it('rejects an object key beyond the portable 850-byte limit without creating an object', async () => {
it(`rejects an object key beyond the portable ${MAX_STORAGE_OBJECT_KEY_UTF8_BYTES}-byte limit without creating an object`, async () => {
const prefix = `${context.rootPrefix}too-long/`;
const key = createAsciiKeyAtLength({ prefix, byteLength: 851 });
const key = createAsciiKeyAtLength({
prefix,
byteLength: MAX_STORAGE_OBJECT_KEY_UTF8_BYTES + 1
});
await expect(context.storage.uploadObject({ key, body: 'too-long' })).rejects.toMatchObject({
name: InvalidStorageObjectKeyError.name,
reason: 'too_long',
actualBytes: 851,
maxBytes: 850
actualBytes: MAX_STORAGE_OBJECT_KEY_UTF8_BYTES + 1,
maxBytes: MAX_STORAGE_OBJECT_KEY_UTF8_BYTES
});
await expect(context.storage.listObjects({ prefix })).resolves.toEqual({
bucket: context.bucket,
......
sdk/storage/test/integration/providers.ts
......@@ -12,13 +12,14 @@ import * as Minio from 'minio';
import { createStorage } from '../../src/factory';
import type { IStorage } from '../../src/interface';
import type { EnsureBucketResult } from '../../src/types';
import { removeIntegrationBucketIfExists } from './helpers';
import { clearIntegrationBucketObjects, removeIntegrationBucketIfExists } from './helpers';
export type StorageIntegrationProviderName = 'aws-s3' | 'minio' | 'oss' | 'cos';
export type StorageIntegrationProviderName = 'aws-s3' | 'r2' | 'minio' | 'oss' | 'cos';
export type StorageIntegrationContext = {
provider: StorageIntegrationProviderName;
storage: IStorage;
publicStorage?: IStorage;
bucket: string;
rootPrefix: string;
initialEnsureResult: EnsureBucketResult;
......@@ -40,7 +41,13 @@ const getRequiredEnv = (name: string): string => {
return value;
};
export const ValidTestBucketNamePrefixPattern = /^fastgpt-sdk\.integration-/;
export const ValidTestBucketNamePrefixPattern =
/^(?:fastgpt-sdk(?:\.|-)integration-|s3-test-|fastgpt-test-)/;
const getR2TestBucket = (envName: string): string => {
const bucket = getRequiredEnv(envName);
if (bucket.startsWith('s3-test-')) return bucket;
throw new Error(`Met invalid R2 test bucket name to protect non-test buckets`);
};
const getTestBucket = (envName: string): string => {
const bucket = getRequiredEnv(envName);
if (ValidTestBucketNamePrefixPattern.test(bucket)) {
......@@ -67,23 +74,66 @@ const isBucketNotFoundError = (error: unknown): boolean => {
);
};
const isBucketAlreadyExistsError = (error: unknown): boolean => {
if (!error || typeof error !== 'object') return false;
const value = error as { name?: unknown; code?: unknown };
const identifiers = [value.name, value.code].map(String);
return identifiers.some((identifier) =>
['BucketAlreadyExists', 'BucketAlreadyOwnedByYou', 'BucketAlreadyExistsError'].includes(
identifier
)
);
};
/**
* 创建云端测试 bucket，并容忍删除后的全局命名空间最终一致性窗口。
* 若 bucket 已被其他账号占用，最终会保留原始错误而不会误用该 bucket。
*/
const ensureTestBucket = async ({
bucketExists,
createBucket
}: {
bucketExists: () => Promise<boolean>;
createBucket: () => Promise<void>;
}): Promise<void> => {
let lastError: unknown;
for (let attempt = 0; attempt < 6; attempt += 1) {
if (await bucketExists()) return;
try {
await createBucket();
return;
} catch (error) {
if (!isBucketAlreadyExistsError(error)) throw error;
lastError = error;
await new Promise((resolve) => setTimeout(resolve, 500 * (attempt + 1)));
}
}
throw lastError ?? new Error('Unable to create integration test bucket');
};
const createContextResult = async ({
provider,
storage,
publicStorage,
bucket,
initialEnsureResult,
createStorage,
bucketExists,
deleteBucket,
preserveBucket,
destroyProvider
}: {
provider: StorageIntegrationProviderName;
storage: IStorage;
publicStorage?: IStorage;
bucket: string;
initialEnsureResult: EnsureBucketResult;
createStorage: () => IStorage;
bucketExists: () => Promise<boolean>;
deleteBucket: () => Promise<void>;
preserveBucket?: boolean;
destroyProvider?: () => Promise<void> | void;
}): Promise<StorageIntegrationContext> => ({
provider,
......@@ -94,11 +144,16 @@ const createContextResult = async ({
createStorage,
cleanup: async () => {
try {
await removeIntegrationBucketIfExists({ storage, bucketExists, deleteBucket });
if (preserveBucket) {
await clearIntegrationBucketObjects({ storage, bucketExists });
} else {
await removeIntegrationBucketIfExists({ storage, bucketExists, deleteBucket });
}
} finally {
try {
await storage.destroy();
} finally {
await publicStorage?.destroy();
await destroyProvider?.();
}
}
......@@ -115,7 +170,9 @@ const createMinioProvider = (): StorageIntegrationProvider => ({
const secretAccessKey = getRequiredEnv('STORAGE_TEST_MINIO_SECRET_ACCESS_KEY');
const endpointUrl = new URL(endpoint);
const useSSL = endpointUrl.protocol === 'https:';
const bucket = getTestBucket('STORAGE_TEST_MINIO_BUCKET');
const bucket = getTestBucket('STORAGE_TEST_MINIO_PRIVATE_BUCKET');
const publicBucket = getTestBucket('STORAGE_TEST_MINIO_PUBLIC_BUCKET');
const publicEndpoint = getRequiredEnv('STORAGE_TEST_MINIO_PUBLIC_ENDPOINT');
const adminClient = new Minio.Client({
endPoint: endpointUrl.hostname,
port: endpointUrl.port ? Number(endpointUrl.port) : useSSL ? 443 : 80,
......@@ -135,6 +192,17 @@ const createMinioProvider = (): StorageIntegrationProvider => ({
credentials: { accessKeyId, secretAccessKey }
});
const storage = createMinioStorage();
const publicStorage = createStorage({
vendor: 'minio',
bucket: publicBucket,
endpoint,
region,
forcePathStyle: true,
maxRetries: 1,
publicEndpoint,
credentials: { accessKeyId, secretAccessKey }
});
await publicStorage.ensureBucket();
const bucketExists = () => adminClient.bucketExists(bucket);
const deleteBucket = () => adminClient.removeBucket(bucket);
await removeIntegrationBucketIfExists({ storage, bucketExists, deleteBucket });
......@@ -143,6 +211,7 @@ const createMinioProvider = (): StorageIntegrationProvider => ({
return createContextResult({
provider: 'minio',
storage,
publicStorage,
bucket,
initialEnsureResult,
createStorage: createMinioStorage,
......@@ -161,7 +230,9 @@ const createAwsS3Provider = (): StorageIntegrationProvider => ({
const accessKeyId = getRequiredEnv('STORAGE_TEST_AWS_S3_ACCESS_KEY_ID');
const secretAccessKey = getRequiredEnv('STORAGE_TEST_AWS_S3_SECRET_ACCESS_KEY');
const forcePathStyle = isEnabled('STORAGE_TEST_AWS_S3_FORCE_PATH_STYLE');
const bucket = getTestBucket('STORAGE_TEST_AWS_S3_BUCKET');
const bucket = getTestBucket('STORAGE_TEST_AWS_S3_PRIVATE_BUCKET');
const publicBucket = getTestBucket('STORAGE_TEST_AWS_S3_PUBLIC_BUCKET');
const publicEndpoint = getRequiredEnv('STORAGE_TEST_AWS_S3_PUBLIC_ENDPOINT');
const adminClient = new S3Client({
endpoint,
region,
......@@ -190,26 +261,114 @@ const createAwsS3Provider = (): StorageIntegrationProvider => ({
credentials: { accessKeyId, secretAccessKey }
});
const storage = createAwsStorage();
await removeIntegrationBucketIfExists({ storage, bucketExists, deleteBucket });
await adminClient.send(
new CreateBucketCommand({
Bucket: bucket,
CreateBucketConfiguration:
region === 'us-east-1'
? undefined
: { LocationConstraint: region as BucketLocationConstraint }
})
);
const publicStorage = createStorage({
vendor: 'aws-s3',
bucket: publicBucket,
endpoint,
region,
forcePathStyle,
maxRetries: 1,
publicEndpoint,
credentials: { accessKeyId, secretAccessKey }
});
await publicStorage.ensureBucket();
await clearIntegrationBucketObjects({ storage, bucketExists });
await ensureTestBucket({
bucketExists,
createBucket: () =>
adminClient
.send(
new CreateBucketCommand({
Bucket: bucket,
CreateBucketConfiguration:
region === 'us-east-1'
? undefined
: { LocationConstraint: region as BucketLocationConstraint }
})
)
.then(() => undefined)
});
const initialEnsureResult = await storage.ensureBucket();
return createContextResult({
provider: 'aws-s3',
storage,
publicStorage,
bucket,
initialEnsureResult,
createStorage: createAwsStorage,
bucketExists,
deleteBucket,
preserveBucket: true,
destroyProvider: () => adminClient.destroy()
});
}
});
const createR2Provider = (): StorageIntegrationProvider => ({
name: 'r2',
enabled: isEnabled('STORAGE_TEST_R2_ENABLED'),
createContext: async () => {
const endpoint = getRequiredEnv('STORAGE_TEST_R2_ENDPOINT');
const region = getRequiredEnv('STORAGE_TEST_R2_REGION');
const accessKeyId = getRequiredEnv('STORAGE_TEST_R2_ACCESS_KEY_ID');
const secretAccessKey = getRequiredEnv('STORAGE_TEST_R2_SECRET_ACCESS_KEY');
const bucket = getR2TestBucket('STORAGE_TEST_R2_PRIVATE_BUCKET');
getR2TestBucket('STORAGE_TEST_R2_PUBLIC_BUCKET');
const publicEndpoint = getRequiredEnv('STORAGE_TEST_R2_PUBLIC_ENDPOINT');
const credentials = { accessKeyId, secretAccessKey };
const adminClient = new S3Client({
endpoint,
region,
credentials,
requestChecksumCalculation: 'WHEN_REQUIRED',
responseChecksumValidation: 'WHEN_REQUIRED'
});
const bucketExists = async () => {
try {
await adminClient.send(new HeadBucketCommand({ Bucket: bucket }));
return true;
} catch (error) {
if (isBucketNotFoundError(error)) return false;
throw error;
}
};
const deleteBucket = () =>
adminClient.send(new DeleteBucketCommand({ Bucket: bucket })).then(() => undefined);
const createR2Storage = () =>
createStorage({
vendor: 'r2',
bucket,
endpoint,
region,
forcePathStyle: false,
maxRetries: 1,
publicEndpoint,
credentials
});
const storage = createR2Storage();
const publicStorage = createStorage({
vendor: 'r2',
bucket: getR2TestBucket('STORAGE_TEST_R2_PUBLIC_BUCKET'),
endpoint,
region,
forcePathStyle: false,
maxRetries: 1,
publicEndpoint,
credentials
});
const initialEnsureResult = await storage.ensureBucket();
return createContextResult({
provider: 'r2',
storage,
publicStorage,
bucket,
initialEnsureResult,
createStorage: createR2Storage,
bucketExists,
deleteBucket,
preserveBucket: true,
destroyProvider: () => adminClient.destroy()
});
}
......@@ -223,7 +382,9 @@ const createOssProvider = (): StorageIntegrationProvider => ({
const region = getRequiredEnv('STORAGE_TEST_OSS_REGION');
const accessKeyId = getRequiredEnv('STORAGE_TEST_OSS_ACCESS_KEY_ID');
const secretAccessKey = getRequiredEnv('STORAGE_TEST_OSS_SECRET_ACCESS_KEY');
const bucket = getTestBucket('STORAGE_TEST_OSS_BUCKET');
const bucket = getTestBucket('STORAGE_TEST_OSS_PRIVATE_BUCKET');
const publicBucket = getTestBucket('STORAGE_TEST_OSS_PUBLIC_BUCKET');
const publicEndpoint = getRequiredEnv('STORAGE_TEST_OSS_PUBLIC_ENDPOINT');
const adminClient = new OSS({
endpoint,
region,
......@@ -240,6 +401,16 @@ const createOssProvider = (): StorageIntegrationProvider => ({
credentials: { accessKeyId, secretAccessKey }
});
const storage = createOssStorage();
const publicStorage = createStorage({
vendor: 'oss',
bucket: publicBucket,
region,
endpoint: new URL(publicEndpoint).host,
cname: true,
secure: true,
credentials: { accessKeyId, secretAccessKey }
});
await publicStorage.ensureBucket();
const bucketExists = async () => {
try {
await adminClient.getBucketInfo(bucket);
......@@ -250,18 +421,23 @@ const createOssProvider = (): StorageIntegrationProvider => ({
}
};
const deleteBucket = () => adminClient.deleteBucket(bucket).then(() => undefined);
await removeIntegrationBucketIfExists({ storage, bucketExists, deleteBucket });
await adminClient.putBucket(bucket);
await clearIntegrationBucketObjects({ storage, bucketExists });
await ensureTestBucket({
bucketExists,
createBucket: () => adminClient.putBucket(bucket).then(() => undefined)
});
const initialEnsureResult = await storage.ensureBucket();
return createContextResult({
provider: 'oss',
storage,
publicStorage,
bucket,
initialEnsureResult,
createStorage: createOssStorage,
bucketExists,
deleteBucket
deleteBucket,
preserveBucket: true
});
}
});
......@@ -274,7 +450,9 @@ const createCosProvider = (): StorageIntegrationProvider => ({
const appId = getRequiredEnv('STORAGE_TEST_COS_APP_ID');
const accessKeyId = getRequiredEnv('STORAGE_TEST_COS_ACCESS_KEY_ID');
const secretAccessKey = getRequiredEnv('STORAGE_TEST_COS_SECRET_ACCESS_KEY');
const bucket = getTestBucket('STORAGE_TEST_COS_BUCKET');
const bucket = getTestBucket('STORAGE_TEST_COS_PRIVATE_BUCKET');
const publicBucket = getTestBucket('STORAGE_TEST_COS_PUBLIC_BUCKET');
const publicEndpoint = getRequiredEnv('STORAGE_TEST_COS_PUBLIC_ENDPOINT');
if (!bucket.endsWith(`-${appId}`)) {
throw new Error('STORAGE_TEST_COS_BUCKET must end with the configured COS app ID');
}
......@@ -288,6 +466,15 @@ const createCosProvider = (): StorageIntegrationProvider => ({
credentials: { accessKeyId, secretAccessKey }
});
const storage = createCosStorage();
const publicStorage = createStorage({
vendor: 'cos',
bucket: publicBucket,
region,
protocol: 'https:',
domain: new URL(publicEndpoint).host,
credentials: { accessKeyId, secretAccessKey }
});
await publicStorage.ensureBucket();
const bucketExists = async () => {
try {
await adminClient.headBucket({ Bucket: bucket, Region: region });
......@@ -299,18 +486,24 @@ const createCosProvider = (): StorageIntegrationProvider => ({
};
const deleteBucket = () =>
adminClient.deleteBucket({ Bucket: bucket, Region: region }).then(() => undefined);
await removeIntegrationBucketIfExists({ storage, bucketExists, deleteBucket });
await adminClient.putBucket({ Bucket: bucket, Region: region });
await clearIntegrationBucketObjects({ storage, bucketExists });
await ensureTestBucket({
bucketExists,
createBucket: () =>
adminClient.putBucket({ Bucket: bucket, Region: region }).then(() => undefined)
});
const initialEnsureResult = await storage.ensureBucket();
return createContextResult({
provider: 'cos',
storage,
publicStorage,
bucket,
initialEnsureResult,
createStorage: createCosStorage,
bucketExists,
deleteBucket
deleteBucket,
preserveBucket: true
});
}
});
......@@ -319,6 +512,7 @@ export const minioIntegrationProvider = createMinioProvider();
export const storageIntegrationProviders: StorageIntegrationProvider[] = [
createAwsS3Provider(),
createR2Provider(),
minioIntegrationProvider,
createOssProvider(),
createCosProvider()
......
sdk/storage/test/integration/r2/r2.integration.test.ts 0 → 100644
import { randomUUID } from 'node:crypto';
import { afterAll, describe, expect, it } from 'vitest';
import { createStorage } from '../../../src/factory';
import type { IStorage } from '../../../src/interface';
const getRequiredEnv = (name: string): string => {
const value = process.env[name]?.trim();
if (!value) throw new Error(`Missing integration test environment variable: ${name}`);
return value;
};
const r2Enabled = process.env.STORAGE_TEST_R2_ENABLED?.toLowerCase() === 'true';
describe.skipIf(!r2Enabled).sequential('R2-specific integration', () => {
let storage: IStorage | undefined;
const key = `contract/${randomUUID()}/public/file.txt`;
afterAll(async () => {
if (!storage) return;
await storage.deleteObject({ key }).catch(() => undefined);
await storage.destroy();
});
it('reads a public object through the configured public endpoint', async () => {
const endpoint = getRequiredEnv('STORAGE_TEST_R2_ENDPOINT');
const region = getRequiredEnv('STORAGE_TEST_R2_REGION');
const accessKeyId = getRequiredEnv('STORAGE_TEST_R2_ACCESS_KEY_ID');
const secretAccessKey = getRequiredEnv('STORAGE_TEST_R2_SECRET_ACCESS_KEY');
const bucket = getRequiredEnv('STORAGE_TEST_R2_PUBLIC_BUCKET');
const publicEndpoint = getRequiredEnv('STORAGE_TEST_R2_PUBLIC_ENDPOINT');
storage = createStorage({
vendor: 'r2',
bucket,
endpoint,
region,
credentials: { accessKeyId, secretAccessKey },
forcePathStyle: false,
publicEndpoint
});
await storage.uploadObject({
key,
body: 'r2-public-integration',
contentType: 'text/plain',
contentLength: 21
});
const publicUrl = storage.generatePublicGetUrl({ key }).url;
const response = await fetch(publicUrl);
expect(response.status).toBe(200);
await expect(response.text()).resolves.toBe('r2-public-integration');
});
});
sdk/storage/test/integration/transport/abort.integration.test.ts
......@@ -29,7 +29,16 @@ const createStalledObjectServer = () => {
const requestReceived = new Promise<void>((resolve) => {
notifyRequest = resolve;
});
const server = http.createServer((_request, response) => {
const server = http.createServer((request, response) => {
if (request.method === 'HEAD') {
response.writeHead(200, {
'Content-Length': '0',
'Content-Type': 'application/octet-stream'
});
response.end();
return;
}
response.writeHead(200, {
'Content-Length': '100',
'Content-Type': 'application/octet-stream'
......
sdk/storage/test/unit/adapters/aws-s3.adapter.test.ts
......@@ -141,4 +141,72 @@ describe('AwsS3StorageAdapter.generatePublicGetUrl', () => {
expect(adapter.generatePublicGetUrl({ key: 'folder name/file #+.txt' }).url).toBe(expectedUrl);
});
it('uses the R2 public endpoint without exposing the bucket name', () => {
const adapter = new AwsS3StorageAdapter({
vendor: 'r2',
bucket: 'fastgpt-public',
endpoint: 'https://account.r2.cloudflarestorage.com',
publicEndpoint: 'https://assets.example.com/files/',
region: 'auto',
credentials: {
accessKeyId: 'access-key',
secretAccessKey: 'secret-key'
}
});
expect(adapter.generatePublicGetUrl({ key: 'folder name/file.txt' }).url).toBe(
'https://assets.example.com/files/folder%20name/file.txt'
);
});
it('does not add AWS checksum query parameters to R2 upload signatures', async () => {
const adapter = new AwsS3StorageAdapter({
vendor: 'r2',
bucket: 'fastgpt-private',
endpoint: 'https://account.r2.cloudflarestorage.com',
region: 'auto',
credentials: {
accessKeyId: 'access-key',
secretAccessKey: 'secret-key'
}
});
const result = await adapter.generatePresignedPutUrl({ key: 'file.txt' });
expect(result.url).not.toContain('checksum');
});
it('does not add checksum validation parameters to R2 download signatures', async () => {
const adapter = new AwsS3StorageAdapter({
vendor: 'r2',
bucket: 'fastgpt-private',
endpoint: 'https://account.r2.cloudflarestorage.com',
region: 'auto',
credentials: {
accessKeyId: 'access-key',
secretAccessKey: 'secret-key'
}
});
const result = await adapter.generatePresignedGetUrl({ key: 'file.txt' });
expect(result.url).not.toContain('checksum');
});
it('rejects a public endpoint containing query or hash components', () => {
const adapter = new AwsS3StorageAdapter({
vendor: 'r2',
bucket: 'fastgpt-public',
endpoint: 'https://account.r2.cloudflarestorage.com',
publicEndpoint: 'https://assets.example.com/files?download=1',
region: 'auto',
credentials: {
accessKeyId: 'access-key',
secretAccessKey: 'secret-key'
}
});
expect(() => adapter.generatePublicGetUrl({ key: 'file.txt' })).toThrow(
'publicEndpoint must not contain query or hash'
);
});
});
sdk/storage/test/unit/adapters/cos.adapter.test.ts
......@@ -70,6 +70,9 @@ describe('CosStorageAdapter.downloadObject', () => {
it('destroys the output stream when the caller aborts the download', async () => {
const adapter = createAdapter();
(adapter as any).client.headObject = vi.fn((_params: unknown, callback: Function) => {
callback(null, {});
});
(adapter as any).client.getObject = vi.fn();
const controller = new AbortController();
......@@ -99,6 +102,34 @@ describe('CosStorageAdapter deletion boundaries', () => {
expect(deleteMultipleObject).not.toHaveBeenCalled();
});
it('splits multi-delete requests at the COS 1000-object limit', async () => {
const adapter = createAdapter();
const keys = Array.from({ length: 1001 }, (_, index) => `dataset/file-${index}.txt`);
const deleteMultipleObject = vi.fn().mockImplementation((params, callback) => {
callback(null, {
Error: [],
Deleted: params.Objects
});
});
(adapter as any).client.deleteMultipleObject = deleteMultipleObject;
await expect(adapter.deleteObjectsByMultiKeys({ keys })).resolves.toEqual({
bucket: 'fastgpt-private',
keys: []
});
expect(deleteMultipleObject).toHaveBeenCalledTimes(2);
expect(deleteMultipleObject).toHaveBeenNthCalledWith(
1,
expect.objectContaining({ Objects: keys.slice(0, 1000).map((Key) => ({ Key })) }),
expect.any(Function)
);
expect(deleteMultipleObject).toHaveBeenNthCalledWith(
2,
expect.objectContaining({ Objects: keys.slice(1000).map((Key) => ({ Key })) }),
expect.any(Function)
);
});
it('rejects a whitespace-only prefix without listing objects', async () => {
const adapter = createAdapter();
const getBucket = vi.fn();
......
sdk/storage/test/unit/adapters/oss.adapter.test.ts
......@@ -48,6 +48,28 @@ describe('OssStorageAdapter.downloadObject', () => {
});
});
describe('OssStorageAdapter.getObjectMetadata', () => {
it('reads ETag from response headers returned by ali-oss', async () => {
const adapter = createAdapter();
(adapter as any).client.head = vi.fn().mockResolvedValue({
meta: {},
res: {
headers: {
etag: '"oss-etag"',
'content-type': 'text/plain',
'content-length': '4'
}
}
});
await expect(adapter.getObjectMetadata({ key: 'dataset/file.txt' })).resolves.toMatchObject({
etag: 'oss-etag',
contentType: 'text/plain',
contentLength: 4
});
});
});
describe('OssStorageAdapter deletion boundaries', () => {
it('treats an empty key list as a no-op', async () => {
const adapter = createAdapter();
......@@ -75,6 +97,23 @@ describe('OssStorageAdapter deletion boundaries', () => {
expect(deleteMulti).toHaveBeenCalledWith(['first.txt', 'second.txt'], { quiet: false });
});
it('splits multi-delete requests at the OSS 1000-key limit', async () => {
const adapter = createAdapter();
const keys = Array.from({ length: 1001 }, (_, index) => `dataset/file-${index}.txt`);
const deleteMulti = vi.fn().mockImplementation(async (keyChunk: string[]) => ({
deleted: keyChunk
}));
(adapter as any).client.deleteMulti = deleteMulti;
await expect(adapter.deleteObjectsByMultiKeys({ keys })).resolves.toEqual({
bucket: 'fastgpt-private',
keys: []
});
expect(deleteMulti).toHaveBeenCalledTimes(2);
expect(deleteMulti).toHaveBeenNthCalledWith(1, keys.slice(0, 1000), { quiet: false });
expect(deleteMulti).toHaveBeenNthCalledWith(2, keys.slice(1000), { quiet: false });
});
it('normalizes the object-shaped Deleted entries produced by ali-oss XML parsing', async () => {
const adapter = createAdapter();
(adapter as any).client.deleteMulti = vi.fn().mockResolvedValue({
......
sdk/storage/test/unit/assert.test.ts
......@@ -36,21 +36,21 @@ describe('storage object key validation', () => {
expect(() => assertStorageObjectKey(key)).not.toThrow();
});
it('accepts exactly 850 UTF-8 bytes, including multibyte characters', () => {
it('accepts exactly 800 UTF-8 bytes, including multibyte characters', () => {
const asciiKey = 'a'.repeat(MAX_STORAGE_OBJECT_KEY_UTF8_BYTES);
const unicodeKey = `${'\u4e2d'.repeat(282)}abcd`;
const unicodeKey = `${'\u4e2d'.repeat(265)}abcde`;
expect(Buffer.byteLength(asciiKey)).toBe(850);
expect(Buffer.byteLength(unicodeKey)).toBe(850);
expect(Buffer.byteLength(asciiKey)).toBe(800);
expect(Buffer.byteLength(unicodeKey)).toBe(800);
expect(() => assertStorageObjectKey(asciiKey)).not.toThrow();
expect(() => assertStorageObjectKey(unicodeKey)).not.toThrow();
});
it('rejects 851 UTF-8 bytes even when the JavaScript string is shorter', () => {
const key = `${'\u4e2d'.repeat(283)}ab`;
it('rejects 801 UTF-8 bytes even when the JavaScript string is shorter', () => {
const key = `${'\u4e2d'.repeat(266)}abc`;
expect(key.length).toBeLessThan(850);
expect(Buffer.byteLength(key)).toBe(851);
expect(key.length).toBeLessThan(801);
expect(Buffer.byteLength(key)).toBe(801);
expectInvalidKey({ value: key, reason: 'too_long' });
});
......
sdk/storage/test/unit/factory.test.ts
......@@ -3,6 +3,7 @@ import { AwsS3StorageAdapter } from '../../src/adapters/aws-s3.adapter';
import { CosStorageAdapter } from '../../src/adapters/cos.adapter';
import { MinioStorageAdapter } from '../../src/adapters/minio.adapter';
import { OssStorageAdapter } from '../../src/adapters/oss.adapter';
import { R2StorageAdapter } from '../../src/adapters/r2.adapter';
import { createStorage } from '../../src/factory';
const credentials = {
......@@ -32,6 +33,15 @@ describe('createStorage', () => {
).toBeInstanceOf(MinioStorageAdapter);
expect(
createStorage({
vendor: 'r2',
bucket: 'bucket',
endpoint: 'https://account.r2.cloudflarestorage.com',
region: 'auto',
credentials
})
).toBeInstanceOf(R2StorageAdapter);
expect(
createStorage({
vendor: 'oss',
bucket: 'bucket',
region: 'oss-cn-hangzhou',
......
sdk/storage/test/unit/integration-helpers.test.ts
import { describe, expect, it, vi } from 'vitest';
import { createVitestStorageMock } from '../../src/helper/mock';
import { removeIntegrationBucketIfExists } from '../integration/helpers';
import {
clearIntegrationBucketObjects,
removeIntegrationBucketIfExists
} from '../integration/helpers';
describe('clearIntegrationBucketObjects', () => {
it('clears objects while preserving an existing bucket', async () => {
const storage = createVitestStorageMock({ vi });
storage.__putObject('nested/file.txt', { body: Buffer.from('content') });
await expect(
clearIntegrationBucketObjects({
storage,
bucketExists: vi.fn().mockResolvedValue(true)
})
).resolves.toBe(true);
expect(storage.__objects.size).toBe(0);
});
it('reports when the bucket is absent without making list calls', async () => {
const storage = createVitestStorageMock({ vi });
await expect(
clearIntegrationBucketObjects({
storage,
bucketExists: vi.fn().mockResolvedValue(false)
})
).resolves.toBe(false);
expect(storage.listObjects).not.toHaveBeenCalled();
});
});
describe('removeIntegrationBucketIfExists', () => {
it('does nothing when the stable test bucket does not exist', async () => {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment