Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 5b49b4fb by Archer Committed by GitHub
Options

support cdn in s3 (#6896)

parent af0db988
Showing with 141 additions and 4 deletions
document/content/self-host/config/env.en.mdx
......@@ -107,6 +107,7 @@ These variables are mainly validated by `packages/service/env.ts` and apply to `
| `STORAGE_PRIVATE_BUCKET` | `fastgpt-private` | Private file bucket. |
| `STORAGE_REGION` | `us-east-1` | Object storage region. |
| `STORAGE_EXTERNAL_ENDPOINT` | Empty | Externally reachable object storage endpoint for browsers or external services. |
| `STORAGE_S3_CDN_ENDPOINT` | Empty | CDN endpoint for S3/MinIO presigned download and preview URLs. It rewrites access URLs only when the current request uses `presigned` download mode, and does not change the default download mode. |
| `STORAGE_S3_ENDPOINT` | `http://localhost:9000` | S3/MinIO-compatible API endpoint. |
| `STORAGE_PUBLIC_ACCESS_EXTRA_SUB_PATH` | Empty | Extra sub-path for public file access URLs. |
| `STORAGE_ACCESS_KEY_ID` | `minioadmin` | Object storage access key. |
......
document/content/self-host/config/env.mdx
......@@ -107,6 +107,7 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说
| `STORAGE_PRIVATE_BUCKET` | `fastgpt-private` | 私有文件 Bucket。 |
| `STORAGE_REGION` | `us-east-1` | 对象存储 Region。 |
| `STORAGE_EXTERNAL_ENDPOINT` | 空 | 外部可访问的对象存储地址，用于浏览器或外部服务访问。 |
| `STORAGE_S3_CDN_ENDPOINT` | 空 | S3/MinIO 预签名下载与预览访问的 CDN 地址；仅在当前请求走 `presigned` 下载模式时替换访问 URL，不改变默认下载模式。 |
| `STORAGE_S3_ENDPOINT` | `http://localhost:9000` | S3/MinIO 兼容 API 地址。 |
| `STORAGE_PUBLIC_ACCESS_EXTRA_SUB_PATH` | 空 | 公开文件访问路径的额外子路径。 |
| `STORAGE_ACCESS_KEY_ID` | `minioadmin` | 对象存储 Access Key。 |
......
document/content/self-host/config/object-storage.en.mdx
......@@ -37,6 +37,7 @@ This guide covers environment variable configuration for object storage provider
- `STORAGE_S3_ENDPOINT` Internal connection address. Can be a container ID, e.g., `http://fastgpt-minio:9000`
- `STORAGE_EXTERNAL_ENDPOINT` An address accessible by both **server** and **client** to reach the bucket. Use a fixed host IP or domain name — don't use `127.0.0.1` or `localhost` (containers can't access loopback addresses). Once configured, the default download mode automatically becomes `presigned`.
- `STORAGE_S3_CDN_ENDPOINT` [Optional] CDN endpoint for S3/MinIO presigned download and preview access. FastGPT rewrites generated file access URLs to this CDN endpoint only when the current download uses `presigned` mode. This variable does not change the default download mode. Uploads still go through the FastGPT backend proxy and do not use the CDN.
- `STORAGE_S3_FORCE_PATH_STYLE` [Optional] Virtual-hosted-style or path-style routing. If vendor is `minio`, this is fixed to `true`.
- `STORAGE_S3_MAX_RETRIES` [Optional] Maximum request retry attempts. Default: 3
......@@ -52,6 +53,7 @@ STORAGE_SECRET_ACCESS_KEY=your_secret_key
STORAGE_PUBLIC_BUCKET=fastgpt-public
STORAGE_PRIVATE_BUCKET=fastgpt-private
STORAGE_EXTERNAL_ENDPOINT=http://127.0.0.1:9000
STORAGE_S3_CDN_ENDPOINT=https://cdn.example.com
STORAGE_S3_ENDPOINT=http://127.0.0.1:9000
STORAGE_S3_FORCE_PATH_STYLE=true
STORAGE_S3_MAX_RETRIES=3
......
document/content/self-host/config/object-storage.mdx
......@@ -37,6 +37,7 @@ import FastGPTLink from '@/components/docs/linkFastGPT';
- `STORAGE_S3_ENDPOINT` 内网连接地址，可以是容器 id 连接，比如 `http://fastgpt-minio:9000`
- `STORAGE_EXTERNAL_ENDPOINT` 一个**服务器**和**客户端**均可访问到存储桶的地址，可以是固定的宿主机 IP 或者域名，注意不要填写成 127.0.0.1 或者 localhost 等本地回环地址(因为容器里无法使用)。配置后，默认下载模式会自动切换为 `presigned`。
- `STORAGE_S3_CDN_ENDPOINT` 【可选】S3/MinIO 预签名下载与预览访问的 CDN 地址。仅在当前下载走 `presigned` 模式时，FastGPT 会在生成文件访问 URL 后把地址替换成该 CDN 地址；该变量不会改变默认下载模式。上传仍走 FastGPT 后端代理，不会使用 CDN。
- `STORAGE_S3_FORCE_PATH_STYLE` 【可选】虚拟主机风格路由或路径路由风格，其中如果厂商填写了 `minio` 的话，该值被固定为 `true`
- `STORAGE_S3_MAX_RETRIES` 【可选】请求最大尝试次数，默认为 3 次
......@@ -52,6 +53,7 @@ STORAGE_SECRET_ACCESS_KEY=your_secret_key
STORAGE_PUBLIC_BUCKET=fastgpt-public
STORAGE_PRIVATE_BUCKET=fastgpt-private
STORAGE_EXTERNAL_ENDPOINT=http://127.0.0.1:9000
STORAGE_S3_CDN_ENDPOINT=https://cdn.example.com
STORAGE_S3_ENDPOINT=http://127.0.0.1:9000
STORAGE_S3_FORCE_PATH_STYLE=true
STORAGE_S3_MAX_RETRIES=3
......
document/data/doc-last-modified.json
......@@ -19,8 +19,8 @@
"content/guide/admin/teamMode.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/build/evaluation.en.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/build/evaluation.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/build/general/ai_settings.en.mdx": "2026-05-08T15:58:10+08:00",
"content/guide/build/general/ai_settings.mdx": "2026-05-08T15:58:10+08:00",
"content/guide/build/general/ai_settings.en.mdx": "2026-05-08T18:08:04+08:00",
"content/guide/build/general/ai_settings.mdx": "2026-05-08T18:08:04+08:00",
"content/guide/build/general/chat_input_guide.en.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/build/general/chat_input_guide.mdx": "2026-05-07T15:06:40+08:00",
"content/guide/build/general/fileInput.en.mdx": "2026-05-07T15:06:40+08:00",
......
packages/service/common/s3/buckets/base.ts
......@@ -7,7 +7,11 @@ import {
CreatePostPresignedUrlOptionsSchema,
type CreatePostPresignedUrlResult
} from '../contracts/type';
import { storageDownloadMode, getSystemMaxFileSize } from '../config/constants';
import {
storageDownloadMode,
getSystemMaxFileSize,
replaceS3UrlWithCdnEndpoint
} from '../config/constants';
import { S3ErrEnum } from '@fastgpt/global/common/error/code/s3';
import { createUploadConstraints } from '../utils/uploadConstraints';
import path from 'node:path';
......@@ -244,7 +248,15 @@ export class S3BaseBucket {
};
}
return await this.externalClient.generatePresignedGetUrl({ key, expiredSeconds: expires });
const result = await this.externalClient.generatePresignedGetUrl({
key,
expiredSeconds: expires
});
return {
...result,
url: replaceS3UrlWithCdnEndpoint(result.url)
};
}
async createPreviewUrl(params: createPreviewUrlParams) {
......
packages/service/common/s3/config/constants.ts
......@@ -23,6 +23,7 @@ type BucketStorageOptions = {
const storageRegion = serviceEnv.STORAGE_REGION;
const storageExternalEndpoint = serviceEnv.STORAGE_EXTERNAL_ENDPOINT;
export const storageS3CdnEndpoint = serviceEnv.STORAGE_S3_CDN_ENDPOINT;
const storageS3Endpoint = serviceEnv.STORAGE_S3_ENDPOINT;
export const storageDownloadMode = serviceEnv.STORAGE_EXTERNAL_ENDPOINT ? 'presigned' : 'proxy';
const storagePublicAccessExtraSubPath = serviceEnv.STORAGE_PUBLIC_ACCESS_EXTRA_SUB_PATH;
......@@ -109,3 +110,29 @@ export function createDefaultStorageOptions() {
}
}
}
export function replaceS3UrlWithCdnEndpoint(url: string) {
if (!storageS3CdnEndpoint) {
return url;
}
try {
const parsedUrl = new URL(url);
const cdnUrl = new URL(storageS3CdnEndpoint);
const cdnPath = cdnUrl.pathname.replace(/\/$/, '');
const sourcePath = parsedUrl.pathname.replace(/^\//, '');
parsedUrl.protocol = cdnUrl.protocol;
parsedUrl.host = cdnUrl.host;
parsedUrl.username = '';
parsedUrl.password = '';
if (cdnPath && cdnPath !== '/') {
parsedUrl.pathname = `${cdnPath}/${sourcePath}`;
}
return parsedUrl.toString();
} catch {
return url;
}
}
packages/service/common/security/fileUrlValidator.ts
......@@ -24,6 +24,7 @@ const systemWhiteList = (() => {
const list: string[] = [];
appendHost({ list, value: serviceEnv.STORAGE_S3_ENDPOINT, allowRawHost: true });
appendHost({ list, value: serviceEnv.STORAGE_EXTERNAL_ENDPOINT });
appendHost({ list, value: serviceEnv.STORAGE_S3_CDN_ENDPOINT });
appendHost({ list, value: serviceEnv.FE_DOMAIN });
appendHost({ list, value: serviceEnv.PRO_URL });
return list;
......
packages/service/env.ts
......@@ -122,6 +122,7 @@ export const serviceEnv = createEnv({
STORAGE_PRIVATE_BUCKET: z.string().default('fastgpt-private'),
STORAGE_REGION: z.string().default('us-east-1'),
STORAGE_EXTERNAL_ENDPOINT: UrlSchema.optional(),
STORAGE_S3_CDN_ENDPOINT: UrlSchema.optional(),
STORAGE_S3_ENDPOINT: UrlSchema.default('http://localhost:9000'),
STORAGE_PUBLIC_ACCESS_EXTRA_SUB_PATH: z.string().optional(),
STORAGE_ACCESS_KEY_ID: z.string().default('minioadmin'),
......
packages/service/test/common/s3/config/constants.test.ts 0 → 100644
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { createVitestStorageMock } from '@fastgpt-sdk/storage';
const originalEnv = {
STORAGE_EXTERNAL_ENDPOINT: process.env.STORAGE_EXTERNAL_ENDPOINT,
STORAGE_S3_CDN_ENDPOINT: process.env.STORAGE_S3_CDN_ENDPOINT
};
const loadConstants = async () => {
vi.resetModules();
return import('@fastgpt/service/common/s3/config/constants');
};
describe('s3 storage constants', () => {
beforeEach(() => {
vi.resetModules();
vi.stubEnv('STORAGE_EXTERNAL_ENDPOINT', undefined);
vi.stubEnv('STORAGE_S3_CDN_ENDPOINT', undefined);
});
afterEach(() => {
vi.stubEnv('STORAGE_EXTERNAL_ENDPOINT', originalEnv.STORAGE_EXTERNAL_ENDPOINT);
vi.stubEnv('STORAGE_S3_CDN_ENDPOINT', originalEnv.STORAGE_S3_CDN_ENDPOINT);
vi.restoreAllMocks();
});
it('keeps proxy download mode when no external or CDN endpoint is configured', async () => {
const { storageDownloadMode, replaceS3UrlWithCdnEndpoint } = await loadConstants();
expect(storageDownloadMode).toBe('proxy');
expect(replaceS3UrlWithCdnEndpoint('https://s3.example.com/bucket/file.png')).toBe(
'https://s3.example.com/bucket/file.png'
);
});
it('keeps proxy download mode but rewrites S3 URLs when CDN endpoint is configured', async () => {
vi.stubEnv('STORAGE_S3_CDN_ENDPOINT', 'https://cdn.example.com/files');
const { storageDownloadMode, replaceS3UrlWithCdnEndpoint } = await loadConstants();
expect(storageDownloadMode).toBe('proxy');
expect(
replaceS3UrlWithCdnEndpoint(
'https://fastgpt-private.s3.example.com/chat/app/file.png?X-Amz-Signature=abc#preview'
)
).toBe('https://cdn.example.com/files/chat/app/file.png?X-Amz-Signature=abc#preview');
});
it('rewrites external presigned URLs from S3BaseBucket', async () => {
vi.stubEnv('STORAGE_S3_CDN_ENDPOINT', 'https://cdn.example.com');
const { S3BaseBucket } = await vi.importActual<
typeof import('@fastgpt/service/common/s3/buckets/base')
>('@fastgpt/service/common/s3/buckets/base');
const storage = createVitestStorageMock({
vi,
bucketName: 'fastgpt-private',
baseUrl: 'https://s3.example.com'
});
const bucket = new S3BaseBucket(storage, undefined);
const result = await bucket.createExternalUrl({
key: 'chat/app/user/chat/file.png',
mode: 'presigned'
});
expect(storage.generatePresignedGetUrl).toHaveBeenCalledWith({
key: 'chat/app/user/chat/file.png',
expiredSeconds: 1800
});
expect(result.url).toBe(
'https://cdn.example.com/get/fastgpt-private/chat%2Fapp%2Fuser%2Fchat%2Ffile.png'
);
});
});
packages/service/test/common/security/fileUrlValidator.test.ts
......@@ -3,6 +3,7 @@ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
const originalEnv = {
STORAGE_S3_ENDPOINT: process.env.STORAGE_S3_ENDPOINT,
STORAGE_EXTERNAL_ENDPOINT: process.env.STORAGE_EXTERNAL_ENDPOINT,
STORAGE_S3_CDN_ENDPOINT: process.env.STORAGE_S3_CDN_ENDPOINT,
FE_DOMAIN: process.env.FE_DOMAIN,
PRO_URL: process.env.PRO_URL
};
......@@ -15,6 +16,7 @@ describe('fileUrlValidator', () => {
vi.resetModules();
vi.stubEnv('STORAGE_S3_ENDPOINT', undefined);
vi.stubEnv('STORAGE_EXTERNAL_ENDPOINT', undefined);
vi.stubEnv('STORAGE_S3_CDN_ENDPOINT', undefined);
vi.stubEnv('FE_DOMAIN', undefined);
vi.stubEnv('PRO_URL', undefined);
});
......@@ -22,6 +24,7 @@ describe('fileUrlValidator', () => {
afterEach(() => {
vi.stubEnv('STORAGE_S3_ENDPOINT', originalEnv.STORAGE_S3_ENDPOINT);
vi.stubEnv('STORAGE_EXTERNAL_ENDPOINT', originalEnv.STORAGE_EXTERNAL_ENDPOINT);
vi.stubEnv('STORAGE_S3_CDN_ENDPOINT', originalEnv.STORAGE_S3_CDN_ENDPOINT);
vi.stubEnv('FE_DOMAIN', originalEnv.FE_DOMAIN);
vi.stubEnv('PRO_URL', originalEnv.PRO_URL);
global.systemEnv = originalSystemEnv;
......@@ -46,6 +49,15 @@ describe('fileUrlValidator', () => {
expect(validateFileUrlDomain('http://external.example.com/file.png')).toBe(true);
});
it('should extract hostname from STORAGE_S3_CDN_ENDPOINT', async () => {
vi.stubEnv('STORAGE_S3_CDN_ENDPOINT', 'https://cdn.example.com/files');
global.systemEnv = { fileUrlWhitelist: ['other.com'] } as any;
const { validateFileUrlDomain } = await import(
'@fastgpt/service/common/security/fileUrlValidator'
);
expect(validateFileUrlDomain('http://cdn.example.com/file.png')).toBe(true);
});
it('should reject invalid STORAGE_EXTERNAL_ENDPOINT at env validation stage', async () => {
vi.stubEnv('STORAGE_EXTERNAL_ENDPOINT', 'not-a-valid-url');
global.systemEnv = { fileUrlWhitelist: ['other.com'] } as any;
......@@ -94,6 +106,7 @@ describe('fileUrlValidator', () => {
it('should combine all env vars into systemWhiteList', async () => {
vi.stubEnv('STORAGE_S3_ENDPOINT', 'http://s3.example.com');
vi.stubEnv('STORAGE_EXTERNAL_ENDPOINT', 'https://external.example.com');
vi.stubEnv('STORAGE_S3_CDN_ENDPOINT', 'https://cdn.example.com');
vi.stubEnv('FE_DOMAIN', 'https://fe.example.com');
vi.stubEnv('PRO_URL', 'https://pro.example.com');
global.systemEnv = { fileUrlWhitelist: ['user.com'] } as any;
......@@ -102,6 +115,7 @@ describe('fileUrlValidator', () => {
);
expect(validateFileUrlDomain('http://s3.example.com/f')).toBe(true);
expect(validateFileUrlDomain('http://external.example.com/f')).toBe(true);
expect(validateFileUrlDomain('http://cdn.example.com/f')).toBe(true);
expect(validateFileUrlDomain('http://fe.example.com/f')).toBe(true);
expect(validateFileUrlDomain('http://pro.example.com/f')).toBe(true);
expect(validateFileUrlDomain('http://user.com/f')).toBe(true);
......
projects/app/.env.template
......@@ -72,6 +72,7 @@ STORAGE_SECRET_ACCESS_KEY=minioadmin
STORAGE_PUBLIC_BUCKET=fastgpt-public
STORAGE_PRIVATE_BUCKET=fastgpt-private
STORAGE_EXTERNAL_ENDPOINT=
STORAGE_S3_CDN_ENDPOINT=
STORAGE_S3_ENDPOINT=http://localhost:9000
STORAGE_S3_FORCE_PATH_STYLE=true
STORAGE_S3_MAX_RETRIES=3
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment