Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 2ccb5b50 by Archer Committed by GitHub
Options

V4.14.4 features (#6036) 

* feat: add query optimize and bill (#6021)

* add query optimize and bill

* perf: query extension

* fix: embe model

* remove log

* remove log

* fix: test

---------

Co-authored-by: xxyyh <2289112474@qq>
Co-authored-by: archer <545436317@qq.com>

* feat: notice (#6013)

* feat: record user's language

* feat: notice points/dataset indexes; support count limit; update docker-compose.yml

* fix: ts error

* feat: send auth code i18n

* chore: dataset notice limit

* chore: adjust

* fix: ts

* fix: countLimit race condition; i18n en-prefix locale fallback to en

---------

Co-authored-by: archer <545436317@qq.com>

* perf: comment

* perf: send inform code

* fix: type error (#6029)

* feat: add ip region for chat logs (#6010)

* feat: add ip region for chat logs

* refactor: use Geolite2.mmdb

* fix: export chat logs

* fix: return location directly

* test: add unit test

* perf: log show ip data

* adjust commercial plans (#6008)

* plan frontend

* plan limit

* coupon

* discount coupon

* fix

* type

* fix audit

* type

* plan name

* legacy plan

* track

* feat: add discount coupon

* fix

* fix discount coupon

* openapi

* type

* type

* env

* api type

* fix

* fix: simple agent plugin input & agent dashboard card (#6034)

* refactor: remove gridfs (#6031)

* fix: replace gridfs multer operations with s3 compatible ops

* wip: s3 features

* refactor: remove gridfs

* fix

* perf: mock test

* doc

* doc

* doc

* fix: test

* fix: s3

* fix: mock s3

* remove invalid config

* fix: init query extension

* initv4144 (#6037)

* chore: initv4144

* fix

* version

* fix: new plans (#6039)

* fix: new plans

* qr modal tip

* fix: buffer raw text filename (#6040)

* fix: initv4144 (#6041)

* fix: pay refresh (#6042)

* fix: migration shell

* rename collection

* clear timerlock

* clear timerlock

* perf: faq

* perf: bill schema

* fix: openapi

* doc

* fix: share var render

* feat: delete dataset queue

* plan usage display (#6043)

* plan usage display

* text

* fix

* fix: ts

* perf: remove invalid code

* perf: init shell

* doc

* perf: rename field

* perf: avatar presign

* init

* custom plan text (#6045)

* fix plans

* fix

* fixed

* computed

---------

Co-authored-by: archer <545436317@qq.com>

* init shell

* plan text & price page back button (#6046)

* init

* index

* delete dataset

* delete dataset

* perf: delete dataset

* init

---------

Co-authored-by: YeYuheng <57035043+YYH211@users.noreply.github.com>
Co-authored-by: xxyyh <2289112474@qq>
Co-authored-by: Finley Ge <32237950+FinleyGe@users.noreply.github.com>
Co-authored-by: Roy <whoeverimf5@gmail.com>
Co-authored-by: heheer <heheer@sealos.io>
parent 9d72f238
Showing with 2011 additions and 1874 deletions
.claude/design/lexical-consistency-analysis.md deleted 100644 → 0
# Lexical Editor 文本解析一致性分析报告
## 执行摘要
通过对 `textToEditorState` 和 `editorStateToText` 函数的全面分析,发现了 **3 个确认的不一致性问题**,会导致用户保存后重新加载时看到与编辑器显示不同的内容。
### 严重问题总览
| 问题 | 严重性 | 影响 | 位置 |
|------|--------|------|------|
| 列表项尾部空格丢失 | 🔴 高 | 用户有意添加的空格被删除 | utils.ts:255 |
| 有序列表序号重置 | 🔴 高 | 自定义序号变成连续序号 | utils.ts:257 |
| 列表项内换行不对称 | 🟡 中 | 编辑器支持但无法往返 | processListItem |
---
## 问题 1: 列表项尾部空格丢失 🔴(已处理)
### 问题描述
在 `processListItem` 函数中使用了 `trim()` 处理列表项文本:
```typescript
// utils.ts:255
const itemTextString = itemText.join('').trim();
```
### 不一致性演示
**用户输入:**
```
- hello world
```
(注意 "world" 后面有 2 个空格)
**EditorState:**
```json
{
"type": "listitem",
"children": [
{ "type": "text", "text": "hello world " }
]
}
```
**输出文本:**
```
- hello world
```
(尾部空格被 trim 删除)
**重新加载:**
```
- hello world
```
(用户的空格永久丢失)
### 影响分析
- **用户体验**: 用户有意添加的尾部空格(可能用于格式对齐)会丢失
- **数据完整性**: 每次保存/加载循环都会丢失尾部空格
- **严重程度**: 高 - 直接影响用户输入的完整性
### 解决方案
**方案 1: 移除 trim()**
```typescript
const itemTextString = itemText.join(''); // 不使用 trim
```
**方案 2: 只移除前导空格**
```typescript
const itemTextString = itemText.join('').trimStart(); // 只移除开头空格
```
**推荐**: 方案 1,完全保留用户输入的空格
---
## 问题 2: 有序列表序号重置 🔴
### 问题描述
在输出有序列表时,使用 `index + 1` 而不是列表项自身的 `value`:
```typescript
// utils.ts:257
const prefix = listType === 'bullet' ? '- ' : `${index + 1}. `;
```
但在解析时,`numberValue` 被正确提取并存储到 `listItem.value`。
### 不一致性演示
**用户输入:**
```
1. first
2. second
5. fifth
10. tenth
```
**解析 (textToEditorState):**
```javascript
items = [
{ numberValue: 1, text: "first" },
{ numberValue: 2, text: "second" },
{ numberValue: 5, text: "fifth" },
{ numberValue: 10, text: "tenth" }
]
```
**EditorState:**
```json
[
{ "value": 1, "text": "first" },
{ "value": 2, "text": "second" },
{ "value": 5, "text": "fifth" },
{ "value": 10, "text": "tenth" }
]
```
**输出文本 (editorStateToText):**
```
1. first (index=0, 0+1=1) ✓
2. second (index=1, 1+1=2) ✓
3. fifth (index=2, 2+1=3) ✗ 应该是 5
4. tenth (index=3, 3+1=4) ✗ 应该是 10
```
**重新加载:**
用户的自定义序号 5 和 10 永久丢失,变成连续的 3 和 4。
### 影响分析
- **用户体验**: 用户有意设置的序号被强制改为连续序号
- **数据完整性**: 有序列表的语义丢失(如章节编号 1.1, 1.2, 2.1)
- **严重程度**: 高 - 改变了用户的语义表达
### 解决方案
```typescript
// utils.ts:257
const prefix = listType === 'bullet'
? '- '
: `${listItem.value || index + 1}. `;
```
使用 `listItem.value` 而不是 `index + 1`,保留原始序号。
---
## 问题 3: 列表项内换行不对称 🟡
### 问题描述
Lexical 编辑器允许在列表项内插入换行符 (`linebreak` 节点),但 `textToEditorState` 无法将包含换行的文本重新解析为列表项内换行。
### 不一致性演示
**用户在编辑器中操作:**
```
1. 输入: "- item1"
2. 按 Shift+Enter (插入软换行)
3. 继续输入: "continued content"
```
**EditorState:**
```json
{
"type": "listitem",
"children": [
{ "type": "text", "text": "item1" },
{ "type": "linebreak" },
{ "type": "text", "text": "continued content" }
]
}
```
**输出文本 (editorStateToText):**
```
- item1
continued content
```
**重新加载 (textToEditorState):**
```
行1: "- item1" → 列表项
行2: "continued content" → 段落 (不再是列表项的一部分!)
```
**最终结构变化:**
```
原来: 1个列表项(包含换行)
现在: 1个列表项 + 1个段落
```
### 影响分析
- **结构完整性**: 列表项的内部结构在保存/加载后改变
- **语义丢失**: 原本属于列表项的内容变成了独立段落
- **严重程度**: 中 - 影响文档结构,但可能符合 Markdown 语义
### 解决方案
**方案 1: 在输出时将换行转为空格**
```typescript
if (child.type === 'linebreak') {
itemText.push(' '); // 使用空格而不是 \n
}
```
**方案 2: 在编辑器中禁止列表项内换行**
- 配置 Lexical 不允许在列表项内插入 linebreak
- 用户只能通过创建新列表项来换行
**方案 3: 支持 Markdown 风格的列表项多行**
```typescript
// 识别缩进的行为列表项的继续内容
parseTextLine:
if (line.startsWith(' ') && prevLine.wasListItem) {
// 作为列表项的继续内容
}
```
**推荐**: 方案 1 (最简单) 或方案 2 (最明确)
---
## 其他潜在问题
### 问题 4: 变量节点保存后变成普通文本 🟡
**现象**:
```
EditorState: { type: 'variableLabel', variableKey: '{{var1}}' }
↓
输出文本: "{{var1}}"
↓
重新加载: { type: 'text', text: "{{var1}}" }
```
**影响**: 变量节点的功能性丢失
**分析**: 这可能是设计决策 - 变量只在编辑会话中有效,保存到文本后变成普通占位符。如果需要保持变量功能,应该使用其他存储格式(如 JSON)而不是纯文本。
### 问题 5: 非连续缩进级别可能导致结构错误 🟡
**现象**:
```
输入:
- level 0
- level 2 (跳过 level 1)
- level 1
```
**问题**: `buildListStructure` 可能无法正确处理非连续的缩进级别
**影响**: 列表嵌套结构可能不符合预期
**建议**: 规范化缩进级别,或在文档中说明只支持连续缩进
---
## 正常运作的部分 ✅
经过分析,以下功能**正常运作**,不存在一致性问题:
1. **空行处理** - 空行被正确保留和还原
2. **段落前导空格** - 修复后完全保留
3. **列表和段落边界** - 正确识别和分离
4. **特殊字符在段落中** - 只有行首的 `- ` 和 `\d+. ` 被识别为列表
5. **混合列表类型** - bullet 和 number 列表正确分离
6. **列表缩进** - 使用 TabStr 统一为 2 个空格
---
## 测试用例建议
### 测试用例 1: 列表项尾部空格
```typescript
const input = "- hello world "; // 2个尾部空格
const state = textToEditorState(input, true);
const editor = createEditorWithState(state);
const output = editorStateToText(editor);
expect(output).toBe("- hello world "); // 应保留空格
```
### 测试用例 2: 自定义列表序号
```typescript
const input = "1. first\n5. fifth\n10. tenth";
const state = textToEditorState(input, true);
const editor = createEditorWithState(state);
const output = editorStateToText(editor);
expect(output).toBe("1. first\n5. fifth\n10. tenth"); // 应保留序号
```
### 测试用例 3: 列表项换行
```typescript
// 在编辑器中创建列表项并插入 linebreak
const editor = createEditor();
// ... 创建列表项
// ... 插入 linebreak
const output = editorStateToText(editor);
const reloadedState = textToEditorState(output, true);
const reloadedEditor = createEditorWithState(reloadedState);
// 验证结构是否一致
expect(getStructure(editor)).toEqual(getStructure(reloadedEditor));
```
### 测试用例 4: 往返对称性
```typescript
const testCases = [
"simple text",
" indented text",
"- bullet list\n - nested",
"1. first\n2. second\n5. fifth",
"text\n\n\nwith\n\nempty\n\nlines",
"- item ", // 尾部空格
];
testCases.forEach(input => {
const state = textToEditorState(input, true);
const editor = createEditorWithState(state);
const output = editorStateToText(editor);
expect(output).toBe(input); // 应完全一致
});
```
---
## 修复优先级建议
### P0 - 立即修复
1. ✅ **列表项尾部空格丢失** - 影响数据完整性
2. ✅ **有序列表序号重置** - 影响语义表达
### P1 - 高优先级
3. ⚠️ **列表项内换行不对称** - 影响结构一致性
### P2 - 按需修复
4. 📝 **变量节点** - 根据产品需求决定
5. 📝 **非连续缩进** - 文档说明或规范化处理
---
## 代码修改建议
### 修改 1: 保留列表项空格
```diff
// utils.ts:255
- const itemTextString = itemText.join('').trim();
+ const itemTextString = itemText.join('');
```
### 修改 2: 使用原始列表序号
```diff
// utils.ts:257
- const prefix = listType === 'bullet' ? '- ' : `${index + 1}. `;
+ const prefix = listType === 'bullet' ? '- ' : `${listItem.value || index + 1}. `;
```
### 修改 3: 处理列表项换行(方案1)
```diff
// utils.ts:242
if (child.type === 'linebreak') {
- itemText.push('\n');
+ itemText.push(' '); // 转为空格而不是换行
}
```
---
## 总结
通过全面分析,确认了 **3 个会导致编辑器显示与解析文本不一致的问题**:
1. 🔴 列表项尾部空格丢失 → 修复: 移除 trim()
2. 🔴 有序列表序号重置 → 修复: 使用 listItem.value
3. 🟡 列表项内换行不对称 → 修复: 转换为空格或禁止
其他方面(空行、前导空格、边界处理)都运作正常。
建议优先修复前两个 P0 问题,确保用户数据的完整性和语义准确性。
.claude/design/projects_app_performance_stability_analysis.md
...@@ -608,136 +608,6 @@ function App({ Component, pageProps }: AppPropsWithLayout) { ...@@ -608,136 +608,6 @@ function App({ Component, pageProps }: AppPropsWithLayout) {
--- ---
### 🔴 H9. instrumentation.ts 初始化失败未处理,导致静默失败
**位置**: `projects/app/src/instrumentation.ts:81-84`
**问题描述**:
```typescript
} catch (error) {
console.log('Init system error', error);
exit(1);
}
```
- 初始化失败直接退出进程
- 部分初始化错误被 `.catch()` 吞没
- 缺少初始化状态检查
**风险等级**: 🔴 高危
**影响**:
- 应用启动失败但无明确错误信息
- 部分服务未初始化导致运行时错误
- 调试困难
**建议方案**:
```typescript
// 1. 详细的初始化错误处理
export async function register() {
const initSteps: Array<{
name: string;
fn: () => Promise<void>;
required: boolean;
}> = [];
try {
if (process.env.NEXT_RUNTIME !== 'nodejs') {
return;
}
const results = {
success: [] as string[],
failed: [] as Array<{ name: string; error: any }>
};
// 阶段 1: 基础连接 (必需)
try {
console.log('Connecting to MongoDB...');
await connectMongo({ db: connectionMongo, url: MONGO_URL });
results.success.push('MongoDB Main');
} catch (error) {
console.error('Fatal: MongoDB connection failed', error);
throw error;
}
try {
await connectMongo({ db: connectionLogMongo, url: MONGO_LOG_URL });
results.success.push('MongoDB Log');
} catch (error) {
console.warn('Non-fatal: MongoDB Log connection failed', error);
results.failed.push({ name: 'MongoDB Log', error });
}
// 阶段 2: 系统初始化 (必需)
try {
console.log('Initializing system config...');
await Promise.all([
getInitConfig(),
initVectorStore(),
initRootUser(),
loadSystemModels()
]);
results.success.push('System Config');
} catch (error) {
console.error('Fatal: System initialization failed', error);
throw error;
}
// 阶段 3: 可选服务
await Promise.allSettled([
preLoadWorker().catch(e => {
console.warn('Worker preload failed (non-fatal)', e);
results.failed.push({ name: 'Worker Preload', error: e });
}),
getSystemTools().catch(e => {
console.warn('System tools init failed (non-fatal)', e);
results.failed.push({ name: 'System Tools', error: e });
}),
initSystemPluginGroups().catch(e => {
console.warn('Plugin groups init failed (non-fatal)', e);
results.failed.push({ name: 'Plugin Groups', error: e });
})
]);
// 阶段 4: 后台任务
startCron();
startTrainingQueue(true);
trackTimerProcess();
console.log('Init system success', {
success: results.success,
failed: results.failed.map(f => f.name)
});
} catch (error) {
console.error('Init system critical error', error);
console.error('Stack:', error.stack);
// 发送告警通知
if (process.env.ERROR_WEBHOOK_URL) {
try {
await fetch(process.env.ERROR_WEBHOOK_URL, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
type: 'INIT_ERROR',
error: error.message,
stack: error.stack,
timestamp: new Date().toISOString()
})
});
} catch (webhookError) {
console.error('Failed to send error webhook', webhookError);
}
}
exit(1);
}
}
```
---
## 二、中危问题 (Medium Priority) ## 二、中危问题 (Medium Priority)
### 🟡 M1. Next.js 未启用 SWC 编译优化完整特性 ### 🟡 M1. Next.js 未启用 SWC 编译优化完整特性
......
.claude/design/projects_app_performance_stability_deep_analysis.md
...@@ -934,28 +934,6 @@ export const authCertWithCSRF = async (props: AuthModeType) => { ...@@ -934,28 +934,6 @@ export const authCertWithCSRF = async (props: AuthModeType) => {
## 新增中危问题 (Additional Medium Priority) ## 新增中危问题 (Additional Medium Priority)
### 🟡 M20. 向量查询缓存策略过于激进
**位置**: `packages/service/common/vectorDB/controller.ts:29-35`
**问题描述**:
```typescript
const onDelCache = throttle((teamId: string) => delRedisCache(getChcheKey(teamId)), 30000, {
leading: true,
trailing: true
});
```
- 删除操作使用 throttle,30 秒内只执行一次
- 可能导致缓存计数不准确
- 未考虑高频删除场景
**建议**:
- 删除操作直接更新缓存
- 定期全量同步缓存和数据库
- 添加缓存一致性校验
---
### 🟡 M21. 训练队列缺少优先级机制 ### 🟡 M21. 训练队列缺少优先级机制
**位置**: `packages/service/common/bullmq/index.ts:20-26` **位置**: `packages/service/common/bullmq/index.ts:20-26`
......
document/content/docs/upgrading/4-14/4144.mdx
...@@ -4,18 +4,47 @@ description: 'FastGPT V4.14.4 更新说明' ...@@ -4,18 +4,47 @@ description: 'FastGPT V4.14.4 更新说明'
--- ---
## 更新指南
### 1. 更新镜像：
### 2. 执行升级脚本
从任意终端，发起 1 个 HTTP 请求。其中 `{{rootkey}}` 替换成环境变量里的 `rootkey`；`{{host}}` 替换成**FastGPT 域名**。
```bash
curl --location --request POST 'https://{{host}}/api/admin/initv4144' \
--header 'rootkey: {{rootkey}}' \
--header 'Content-Type: application/json'
```
将 4.14.3 中，遗留的 Dataset/local 接口上传的文件，也迁移到 S3 中。
## 🚀 新增内容 ## 🚀 新增内容
1. 工具调用支持配置流输出 1. 工具调用支持配置流输出
2. AI 积分告警通知。
3. 对话日志支持展示 IP 地址归属地。
4. 通过 API 上传本地文件至知识库，保存至 S3。同时将旧版 Gridfs 代码全部移除。
5. 新版订阅套餐逻辑。
## ⚙️ 优化 ## ⚙️ 优化
1. 增加 S3 上传文件超时时长为 5 分钟。 1. 增加 S3 上传文件超时时长为 5 分钟。
2. 问题优化采用 JinaAI 的边际收益公式，获取最大边际收益的检索词。
3. 用户通知，支持中英文，以及优化模板。
4. 删除知识库采用队列异步删除模式。
## 🐛 修复 ## 🐛 修复
1. 循环节点数组，取消过滤空内容。 1. 循环节点数组，取消过滤空内容。
2. 工作流工具，未传递自定义 DataId，导致测试运行时，查看知识库提示无权限。 2. 工作流工具，未传递自定义 DataId，导致测试运行时，查看知识库提示无权限。
3. 对话 Agent 工具配置中，非必填的布尔和数字类型无法直接确认。
4. 工作台卡片在名字过长时错位。
5. 分享链接中url query 中携带全局变量时，前端 UI 不会加载该值。
6. window 下判断 CSV 文件异常。
## 插件 ## 插件
document/data/doc-last-modified.json
...@@ -118,7 +118,7 @@ ...@@ -118,7 +118,7 @@
"document/content/docs/upgrading/4-14/4141.mdx": "2025-11-19T10:15:27+08:00", "document/content/docs/upgrading/4-14/4141.mdx": "2025-11-19T10:15:27+08:00",
"document/content/docs/upgrading/4-14/4142.mdx": "2025-11-18T19:27:14+08:00", "document/content/docs/upgrading/4-14/4142.mdx": "2025-11-18T19:27:14+08:00",
"document/content/docs/upgrading/4-14/4143.mdx": "2025-11-26T20:52:05+08:00", "document/content/docs/upgrading/4-14/4143.mdx": "2025-11-26T20:52:05+08:00",
"document/content/docs/upgrading/4-14/4144.mdx": "2025-12-01T21:46:30+08:00", "document/content/docs/upgrading/4-14/4144.mdx": "2025-12-07T14:24:15+08:00",
"document/content/docs/upgrading/4-8/40.mdx": "2025-08-02T19:38:37+08:00", "document/content/docs/upgrading/4-8/40.mdx": "2025-08-02T19:38:37+08:00",
"document/content/docs/upgrading/4-8/41.mdx": "2025-08-02T19:38:37+08:00", "document/content/docs/upgrading/4-8/41.mdx": "2025-08-02T19:38:37+08:00",
"document/content/docs/upgrading/4-8/42.mdx": "2025-08-02T19:38:37+08:00", "document/content/docs/upgrading/4-8/42.mdx": "2025-08-02T19:38:37+08:00",
......
packages/global/common/error/code/team.ts
...@@ -11,6 +11,7 @@ export enum TeamErrEnum { ...@@ -11,6 +11,7 @@ export enum TeamErrEnum {
datasetAmountNotEnough = 'datasetAmountNotEnough', datasetAmountNotEnough = 'datasetAmountNotEnough',
appAmountNotEnough = 'appAmountNotEnough', appAmountNotEnough = 'appAmountNotEnough',
pluginAmountNotEnough = 'pluginAmountNotEnough', pluginAmountNotEnough = 'pluginAmountNotEnough',
appFolderAmountNotEnough = 'appFolderAmountNotEnough',
websiteSyncNotEnough = 'websiteSyncNotEnough', websiteSyncNotEnough = 'websiteSyncNotEnough',
reRankNotEnough = 'reRankNotEnough', reRankNotEnough = 'reRankNotEnough',
groupNameEmpty = 'groupNameEmpty', groupNameEmpty = 'groupNameEmpty',
...@@ -66,6 +67,10 @@ const teamErr = [ ...@@ -66,6 +67,10 @@ const teamErr = [
message: i18nT('common:code_error.team_error.plugin_amount_not_enough') message: i18nT('common:code_error.team_error.plugin_amount_not_enough')
}, },
{ {
statusText: TeamErrEnum.appFolderAmountNotEnough,
message: i18nT('common:code_error.team_error.app_folder_amount_not_enough')
},
{
statusText: TeamErrEnum.websiteSyncNotEnough, statusText: TeamErrEnum.websiteSyncNotEnough,
message: i18nT('common:code_error.team_error.website_sync_not_enough') message: i18nT('common:code_error.team_error.website_sync_not_enough')
}, },
......
packages/global/common/file/constants.ts
import { i18nT } from '../../../web/i18n/utils';
/* mongo fs bucket */ /* mongo fs bucket */
export enum BucketNameEnum { export enum BucketNameEnum {
dataset = 'dataset', dataset = 'dataset',
chat = 'chat' chat = 'chat'
} }
export const bucketNameMap = {
[BucketNameEnum.dataset]: {
label: i18nT('file:bucket_file'),
previewExpireMinutes: 30 // 30 minutes
},
[BucketNameEnum.chat]: {
label: i18nT('file:bucket_chat'),
previewExpireMinutes: 7 * 24 * 60 // 7 days
}
};
export const EndpointUrl = `${process.env.FILE_DOMAIN || process.env.FE_DOMAIN || ''}${process.env.NEXT_PUBLIC_BASE_URL || ''}`; export const EndpointUrl = `${process.env.FILE_DOMAIN || process.env.FE_DOMAIN || ''}${process.env.NEXT_PUBLIC_BASE_URL || ''}`;
export const ReadFileBaseUrl = `${EndpointUrl}/api/common/file/read`; export const ReadFileBaseUrl = `${EndpointUrl}/api/common/file/read`;
......
packages/global/common/middle/tracks/constants.ts
...@@ -9,5 +9,6 @@ export enum TrackEnum { ...@@ -9,5 +9,6 @@ export enum TrackEnum {
datasetSearch = 'datasetSearch', datasetSearch = 'datasetSearch',
readSystemAnnouncement = 'readSystemAnnouncement', readSystemAnnouncement = 'readSystemAnnouncement',
clickOperationalAd = 'clickOperationalAd', clickOperationalAd = 'clickOperationalAd',
closeOperationalAd = 'closeOperationalAd' closeOperationalAd = 'closeOperationalAd',
teamChatQPM = 'teamChatQPM'
} }
packages/global/common/string/tools.ts
...@@ -184,7 +184,7 @@ export const sliceStrStartEnd = (str: string, start: number, end: number) => { ...@@ -184,7 +184,7 @@ export const sliceStrStartEnd = (str: string, start: number, end: number) => {
return `${startContent}${overSize ? `\n\n...[hide ${str.length - start - end} chars]...\n\n` : ''}${endContent}`; return `${startContent}${overSize ? `\n\n...[hide ${str.length - start - end} chars]...\n\n` : ''}${endContent}`;
}; };
/* /*
Parse file extension from url Parse file extension from url
Test： Test：
1. https://xxx.com/file.pdf?token=123 1. https://xxx.com/file.pdf?token=123
...@@ -201,3 +201,32 @@ export const parseFileExtensionFromUrl = (url = '') => { ...@@ -201,3 +201,32 @@ export const parseFileExtensionFromUrl = (url = '') => {
const extension = fileName.split('.').pop(); const extension = fileName.split('.').pop();
return (extension || '').toLowerCase(); return (extension || '').toLowerCase();
}; };
export const formatNumberWithUnit = (num: number, locale: string = 'zh-CN'): string => {
if (num === 0) return '0';
if (!num || isNaN(num)) return '-';
const absNum = Math.abs(num);
const isNegative = num < 0;
const prefix = isNegative ? '-' : '';
if (locale === 'zh-CN') {
if (absNum >= 10000) {
const value = absNum / 10000;
const formatted = Number(value.toFixed(2)).toString();
return `${prefix}${formatted}万`;
}
return num.toLocaleString(locale);
} else {
if (absNum >= 1000000) {
const value = absNum / 1000000;
const formatted = Number(value.toFixed(2)).toString();
return `${prefix}${formatted}M`;
}
if (absNum >= 1000) {
const value = absNum / 1000;
const formatted = Number(value.toFixed(2)).toString();
return `${prefix}${formatted}K`;
}
return num.toLocaleString(locale);
}
};
packages/global/common/system/types/index.d.ts
...@@ -65,6 +65,7 @@ export type FastGPTFeConfigsType = { ...@@ -65,6 +65,7 @@ export type FastGPTFeConfigsType = {
show_compliance_copywriting?: boolean; show_compliance_copywriting?: boolean;
show_aiproxy?: boolean; show_aiproxy?: boolean;
show_coupon?: boolean; show_coupon?: boolean;
show_discount_coupon?: boolean;
concatMd?: string; concatMd?: string;
show_dataset_feishu?: boolean; show_dataset_feishu?: boolean;
......
packages/global/core/app/logs/constants.ts
...@@ -13,7 +13,8 @@ export enum AppLogKeysEnum { ...@@ -13,7 +13,8 @@ export enum AppLogKeysEnum {
ANNOTATED_COUNT = 'annotatedCount', ANNOTATED_COUNT = 'annotatedCount',
POINTS = 'points', POINTS = 'points',
RESPONSE_TIME = 'responseTime', RESPONSE_TIME = 'responseTime',
ERROR_COUNT = 'errorCount' ERROR_COUNT = 'errorCount',
REGION = 'region'
} }
export const AppLogKeysEnumMap = { export const AppLogKeysEnumMap = {
...@@ -29,7 +30,8 @@ export const AppLogKeysEnumMap = { ...@@ -29,7 +30,8 @@ export const AppLogKeysEnumMap = {
[AppLogKeysEnum.ANNOTATED_COUNT]: i18nT('app:logs_keys_annotatedCount'), [AppLogKeysEnum.ANNOTATED_COUNT]: i18nT('app:logs_keys_annotatedCount'),
[AppLogKeysEnum.POINTS]: i18nT('app:logs_keys_points'), [AppLogKeysEnum.POINTS]: i18nT('app:logs_keys_points'),
[AppLogKeysEnum.RESPONSE_TIME]: i18nT('app:logs_keys_responseTime'), [AppLogKeysEnum.RESPONSE_TIME]: i18nT('app:logs_keys_responseTime'),
[AppLogKeysEnum.ERROR_COUNT]: i18nT('app:logs_keys_errorCount') [AppLogKeysEnum.ERROR_COUNT]: i18nT('app:logs_keys_errorCount'),
[AppLogKeysEnum.REGION]: i18nT('app:logs_keys_region')
}; };
export const DefaultAppLogKeys = [ export const DefaultAppLogKeys = [
...@@ -45,7 +47,8 @@ export const DefaultAppLogKeys = [ ...@@ -45,7 +47,8 @@ export const DefaultAppLogKeys = [
{ key: AppLogKeysEnum.ANNOTATED_COUNT, enable: false }, { key: AppLogKeysEnum.ANNOTATED_COUNT, enable: false },
{ key: AppLogKeysEnum.POINTS, enable: false }, { key: AppLogKeysEnum.POINTS, enable: false },
{ key: AppLogKeysEnum.RESPONSE_TIME, enable: false }, { key: AppLogKeysEnum.RESPONSE_TIME, enable: false },
{ key: AppLogKeysEnum.ERROR_COUNT, enable: false } { key: AppLogKeysEnum.ERROR_COUNT, enable: false },
{ key: AppLogKeysEnum.REGION, enable: true }
]; ];
export enum AppLogTimespanEnum { export enum AppLogTimespanEnum {
......
packages/global/core/chat/utils.ts
...@@ -59,7 +59,6 @@ export const getHistoryPreview = ( ...@@ -59,7 +59,6 @@ export const getHistoryPreview = (
return `![Input an image](${item.file.url.slice(0, 100)}...)`; return `![Input an image](${item.file.url.slice(0, 100)}...)`;
return ''; return '';
}) })
.filter(Boolean)
.join('\n') || '' .join('\n') || ''
); );
} else if (item.obj === ChatRoleEnum.AI) { } else if (item.obj === ChatRoleEnum.AI) {
......
packages/global/core/dataset/type.d.ts
...@@ -83,6 +83,9 @@ export type DatasetSchemaType = { ...@@ -83,6 +83,9 @@ export type DatasetSchemaType = {
apiDatasetServer?: ApiDatasetServerType; apiDatasetServer?: ApiDatasetServerType;
// 软删除字段
deleteTime?: Date | null;
// abandon // abandon
autoSync?: boolean; autoSync?: boolean;
externalReadUrl?: string; externalReadUrl?: string;
......
packages/global/openapi/index.ts
...@@ -3,6 +3,7 @@ import { ChatPath } from './core/chat'; ...@@ -3,6 +3,7 @@ import { ChatPath } from './core/chat';
import { ApiKeyPath } from './support/openapi'; import { ApiKeyPath } from './support/openapi';
import { TagsMap } from './tag'; import { TagsMap } from './tag';
import { PluginPath } from './core/plugin'; import { PluginPath } from './core/plugin';
import { WalletPath } from './support/wallet';
export const openAPIDocument = createDocument({ export const openAPIDocument = createDocument({
openapi: '3.1.0', openapi: '3.1.0',
...@@ -14,7 +15,8 @@ export const openAPIDocument = createDocument({ ...@@ -14,7 +15,8 @@ export const openAPIDocument = createDocument({
paths: { paths: {
...ChatPath, ...ChatPath,
...ApiKeyPath, ...ApiKeyPath,
...PluginPath ...PluginPath,
...WalletPath
}, },
servers: [{ url: '/api' }], servers: [{ url: '/api' }],
'x-tagGroups': [ 'x-tagGroups': [
...@@ -33,6 +35,10 @@ export const openAPIDocument = createDocument({ ...@@ -33,6 +35,10 @@ export const openAPIDocument = createDocument({
{ {
name: 'ApiKey', name: 'ApiKey',
tags: [TagsMap.apiKey] tags: [TagsMap.apiKey]
},
{
name: '支付',
tags: [TagsMap.walletBill, TagsMap.walletDiscountCoupon]
} }
] ]
}); });
packages/global/openapi/support/wallet/bill/api.ts 0 → 100644
import { z } from 'zod';
import { ObjectIdSchema } from '../../../../common/type/mongo';
import {
BillTypeEnum,
BillStatusEnum,
BillPayWayEnum
} from '../../../../support/wallet/bill/constants';
import { StandardSubLevelEnum, SubModeEnum } from '../../../../support/wallet/sub/constants';
import { PaginationSchema } from '../../../api';
import { BillSchema } from '../../../../support/wallet/bill/type';
// Bill list
export const BillListQuerySchema = PaginationSchema.safeExtend({
type: z.enum(BillTypeEnum).optional().meta({ description: '订单类型筛选' })
});
export type GetBillListQueryType = z.infer<typeof BillListQuerySchema>;
export const BillListResponseSchema = z.object({
total: z.number(),
list: z.array(BillSchema)
});
export type GetBillListResponseType = z.infer<typeof BillListResponseSchema>;
// Create
export const CreateStandPlanBillSchema = z
.object({
type: z.literal(BillTypeEnum.standSubPlan).meta({ description: '订单类型：标准订阅套餐' }),
level: z.enum(StandardSubLevelEnum).meta({ description: '标准订阅等级' }),
subMode: z.enum(SubModeEnum).meta({ description: '订阅周期' }),
discountCouponId: z.string().optional().meta({ description: '优惠券 ID' })
})
.meta({ description: '标准订阅套餐订单创建参数' });
export const CreateExtractPointsBillSchema = z
.object({
type: z.literal(BillTypeEnum.extraPoints).meta({ description: '订单类型：额外积分' }),
extraPoints: z.int().min(0).meta({ description: '额外积分数量' }),
month: z.int().min(1).max(12).meta({ description: '订阅月数' }),
discountCouponId: z.string().optional().meta({ description: '优惠券 ID（未使用）' })
})
.meta({ description: '额外积分订单创建参数' });
export const CreateExtractDatasetBillSchema = z
.object({
type: z.literal(BillTypeEnum.extraDatasetSub).meta({ description: '订单类型：额外数据集存储' }),
extraDatasetSize: z.int().min(0).meta({ description: '额外数据集大小' }),
month: z.int().min(1).max(12).meta({ description: '订阅月数' }),
discountCouponId: z.string().optional().meta({ description: '优惠券 ID（未使用）' })
})
.meta({ description: '额外数据集存储订单创建参数' });
export const CreateBillPropsSchema = z.discriminatedUnion('type', [
CreateStandPlanBillSchema,
CreateExtractPointsBillSchema,
CreateExtractDatasetBillSchema
]);
export type CreateBillPropsType = z.infer<typeof CreateBillPropsSchema>;
export const UpdatePaymentPropsSchema = z.object({
billId: ObjectIdSchema,
payWay: z.enum(BillPayWayEnum)
});
export type UpdatePaymentPropsType = z.infer<typeof UpdatePaymentPropsSchema>;
export const UpdateBillResponseSchema = z
.object({
qrCode: z.string().optional().meta({ description: '支付二维码 URL' }),
iframeCode: z.string().optional().meta({ description: '支付 iframe 代码' }),
markdown: z.string().optional().meta({ description: 'Markdown 格式的支付信息' })
})
.refine((data) => data.qrCode || data.iframeCode || data.markdown, {
message: 'At least one of qrCode, iframeCode, or markdown must be provided'
});
export type UpdateBillResponseType = z.infer<typeof UpdateBillResponseSchema>;
export const CreateBillResponseSchema = UpdateBillResponseSchema.safeExtend({
billId: z.string().meta({ description: '订单 ID' }),
readPrice: z.number().min(0).meta({ description: '实际支付价格' }),
payment: z.enum(BillPayWayEnum).meta({ description: '支付方式' })
});
export type CreateBillResponseType = z.infer<typeof CreateBillResponseSchema>;
// Check pay result
export const CheckPayResultResponseSchema = z.object({
status: z.enum(BillStatusEnum),
description: z.string().optional()
});
export type CheckPayResultResponseType = z.infer<typeof CheckPayResultResponseSchema>;
// Bill detail
export const BillDetailResponseSchema = BillSchema.safeExtend({
couponName: z.string().optional()
});
export type BillDetailResponseType = z.infer<typeof BillDetailResponseSchema>;
// Cancel bill
export const CancelBillPropsSchema = z.object({
billId: ObjectIdSchema.meta({ description: '订单 ID' })
});
export type CancelBillPropsType = z.infer<typeof CancelBillPropsSchema>;
packages/global/openapi/support/wallet/bill/index.ts 0 → 100644
import { z } from 'zod';
import type { OpenAPIPath } from '../../../type';
import {
CreateBillPropsSchema,
CreateBillResponseSchema,
UpdatePaymentPropsSchema,
UpdateBillResponseSchema,
CheckPayResultResponseSchema,
BillDetailResponseSchema,
BillListQuerySchema,
CancelBillPropsSchema
} from './api';
import { TagsMap } from '../../../tag';
import { ObjectIdSchema } from '../../../../common/type/mongo';
export const BillPath: OpenAPIPath = {
'/support/wallet/bill/create': {
post: {
summary: '创建订单',
description: '创建订单订单，支持标准订阅套餐、额外积分、额外数据集存储三种类型',
tags: [TagsMap.walletBill],
requestBody: {
content: {
'application/json': {
schema: CreateBillPropsSchema
}
}
},
responses: {
200: {
description: '成功创建订单',
content: {
'application/json': {
schema: CreateBillResponseSchema
}
}
}
}
}
},
'/support/wallet/bill/pay/updatePayment': {
post: {
summary: '更新支付方式',
description: '为未支付的订单更新支付方式，返回新的支付二维码或链接',
tags: [TagsMap.walletBill],
requestBody: {
content: {
'application/json': {
schema: UpdatePaymentPropsSchema
}
}
},
responses: {
200: {
description: '成功更新支付方式',
content: {
'application/json': {
schema: UpdateBillResponseSchema
}
}
}
}
}
},
'/support/wallet/bill/pay/checkPayResult': {
get: {
summary: '检查支付结果',
description: '检查订单的支付状态，用于轮询支付结果',
tags: [TagsMap.walletBill],
requestParams: {
query: z.object({
payId: ObjectIdSchema.meta({
description: '订单 ID'
})
})
},
responses: {
200: {
description: '成功获取支付结果',
content: {
'application/json': {
schema: CheckPayResultResponseSchema
}
}
}
}
}
},
'/support/wallet/bill/detail': {
get: {
summary: '获取订单详情',
description: '根据订单 ID 获取订单详细信息，包括优惠券名称等',
tags: [TagsMap.walletBill],
requestParams: {
query: z.object({
billId: ObjectIdSchema.meta({
description: '订单 ID'
})
})
},
responses: {
200: {
description: '成功获取订单详情',
content: {
'application/json': {
schema: BillDetailResponseSchema.nullable()
}
}
}
}
}
},
'/support/wallet/bill/list': {
post: {
summary: '获取订单列表',
description: '分页获取团队的订单列表，支持按类型筛选',
tags: [TagsMap.walletBill],
requestBody: {
content: {
'application/json': {
schema: BillListQuerySchema
}
}
},
responses: {
200: {
description: '成功获取订单列表',
content: {
'application/json': {
schema: z.object({
list: z.array(BillDetailResponseSchema),
total: z.number().meta({ description: '总数' })
})
}
}
}
}
}
},
'/support/wallet/bill/cancel': {
post: {
summary: '取消订单',
description: '取消未支付的订单，如果使用了优惠券会自动返还',
tags: [TagsMap.walletBill],
requestBody: {
content: {
'application/json': {
schema: CancelBillPropsSchema
}
}
},
responses: {
200: {
description: '成功取消订单',
content: {
'application/json': {
schema: z.null()
}
}
}
}
}
}
};
packages/global/openapi/support/wallet/discountCoupon/api.ts 0 → 100644
import { z } from 'zod';
import { ObjectIdSchema } from '../../../../common/type/mongo';
import {
DiscountCouponStatusEnum,
DiscountCouponTypeEnum
} from '../../../../support/wallet/sub/discountCoupon/constants';
export const DiscountCouponSchema = z.object({
_id: ObjectIdSchema.meta({ description: '优惠券 ID' }),
teamId: ObjectIdSchema.meta({ description: '团队 ID' }),
type: z.enum(Object.values(DiscountCouponTypeEnum)).meta({ description: '优惠券类型' }),
startTime: z.coerce.date().optional().meta({ description: '生效时间' }),
expiredTime: z.coerce.date().meta({ description: '过期时间' }),
usedAt: z.coerce.date().optional().meta({ description: '使用时间' }),
createTime: z.coerce.date().meta({ description: '创建时间' })
});
export type DiscountCouponSchemaType = z.infer<typeof DiscountCouponSchema>;
export const DiscountCouponItemSchema = DiscountCouponSchema.extend({
name: z.string().meta({ description: '优惠券名称' }),
description: z.string().meta({ description: '优惠券描述' }),
discount: z.number().min(0).max(1).meta({ description: '折扣率' }),
iconZh: z.string().meta({ description: '中文图标路径' }),
iconEn: z.string().meta({ description: '英文图标路径' }),
status: z.enum(DiscountCouponStatusEnum).meta({ description: '优惠券状态' }),
billId: ObjectIdSchema.optional().meta({
description: '关联的订单 ID, 被使用后该值存在'
})
});
export const DiscountCouponListResponseSchema = z.array(DiscountCouponItemSchema);
export type DiscountCouponListResponseType = z.infer<typeof DiscountCouponListResponseSchema>;
packages/global/openapi/support/wallet/discountCoupon/index.ts 0 → 100644
import type { OpenAPIPath } from '../../../type';
import { DiscountCouponListResponseSchema } from './api';
import { TagsMap } from '../../../tag';
export const DiscountCouponPath: OpenAPIPath = {
'/support/wallet/discountCoupon/list': {
get: {
summary: '获取优惠券列表',
description: '获取团队的优惠券列表，包括优惠券状态、使用情况等信息',
tags: [TagsMap.walletDiscountCoupon],
responses: {
200: {
description: '成功获取优惠券列表',
content: {
'application/json': {
schema: DiscountCouponListResponseSchema
}
}
}
}
}
}
};
packages/global/openapi/support/wallet/index.ts 0 → 100644
import type { OpenAPIPath } from '../../type';
import { BillPath } from './bill';
import { DiscountCouponPath } from './discountCoupon';
export const WalletPath: OpenAPIPath = {
...BillPath,
...DiscountCouponPath
};
packages/global/openapi/tag.ts
...@@ -6,5 +6,7 @@ export const TagsMap = { ...@@ -6,5 +6,7 @@ export const TagsMap = {
pluginAdmin: '管理员插件管理', pluginAdmin: '管理员插件管理',
pluginToolAdmin: '管理员系统工具管理', pluginToolAdmin: '管理员系统工具管理',
pluginTeam: '团队插件管理', pluginTeam: '团队插件管理',
apiKey: 'APIKey' apiKey: 'APIKey',
walletBill: '订单',
walletDiscountCoupon: '优惠券'
}; };
packages/global/support/user/api.d.ts
import type { MemberGroupSchemaType } from 'support/permission/memberGroup/type'; import type { MemberGroupSchemaType } from '../permission/memberGroup/type';
import { MemberGroupListItemType } from 'support/permission/memberGroup/type'; import { MemberGroupListItemType } from '../permission/memberGroup/type';
import type { OAuthEnum } from './constant'; import type { OAuthEnum } from './constant';
import type { TrackRegisterParams } from './login/api';
import { TeamMemberStatusEnum } from './team/constant'; import { TeamMemberStatusEnum } from './team/constant';
import type { OrgType } from './team/org/type'; import type { OrgType } from './team/org/type';
import type { TeamMemberItemType } from './team/type'; import type { TeamMemberItemType } from './team/type';
import type { LangEnum } from '../../common/i18n/type';
import type { TrackRegisterParams } from '../marketing/type';
export type PostLoginProps = { export type PostLoginProps = {
username: string; username: string;
password: string; password: string;
code: string; code: string;
language?: `${LangEnum}`;
}; };
export type OauthLoginProps = { export type OauthLoginProps = {
type: `${OAuthEnum}`; type: `${OAuthEnum}`;
callbackUrl: string; callbackUrl: string;
props: Record<string, string>; props: Record<string, string>;
language?: `${LangEnum}`;
} & TrackRegisterParams; } & TrackRegisterParams;
export type WxLoginProps = { export type WxLoginProps = {
......
packages/global/support/user/inform/constants.ts
...@@ -17,13 +17,22 @@ export const InformLevelMap = { ...@@ -17,13 +17,22 @@ export const InformLevelMap = {
}; };
export enum SendInformTemplateCodeEnum { export enum SendInformTemplateCodeEnum {
EXPIRE_SOON = 'EXPIRE_SOON', REGISTER = 'REGISTER', // 注册
EXPIRED = 'EXPIRED', RESET_PASSWORD = 'RESET_PASSWORD', // 重置密码
FREE_CLEAN = 'FREE_CLEAN', BIND_NOTIFICATION = 'BIND_NOTIFICATION', // 绑定通知
REGISTER = 'REGISTER',
RESET_PASSWORD = 'RESET_PASSWORD', EXPIRE_SOON = 'EXPIRE_SOON', // 即将过期
BIND_NOTIFICATION = 'BIND_NOTIFICATION', EXPIRED = 'EXPIRED', // 已过期
LACK_OF_POINTS = 'LACK_OF_POINTS', FREE_CLEAN = 'FREE_CLEAN', // 免费版清理
CUSTOM = 'CUSTOM',
MANAGE_RENAME = 'MANAGE_RENAME' POINTS_THIRTY_PERCENT_REMAIN = 'POINTS_THIRTY_PERCENT_REMAIN', // 积分30%剩余
POINTS_TEN_PERCENT_REMAIN = 'POINTS_TEN_PERCENT_REMAIN', // 积分10%剩余
LACK_OF_POINTS = 'LACK_OF_POINTS', // 积分不足
DATASET_INDEX_NO_REMAIN = 'DATASET_INDEX_NO_REMAIN', // 数据集索引0剩余
DATASET_INDEX_TEN_PERCENT_REMAIN = 'DATASET_INDEX_TEN_PERCENT_REMAIN', // 数据集索引10%剩余
DATASET_INDEX_THIRTY_PERCENT_REMAIN = 'DATASET_INDEX_THIRTY_PERCENT_REMAIN', // 数据集索引30%剩余
MANAGE_RENAME = 'MANAGE_RENAME', // 管理员重命名
CUSTOM = 'CUSTOM' // 自定义
} }
packages/global/support/user/login/api.d.ts
import type { LangEnum } from '../../../common/i18n/type';
import type { TrackRegisterParams } from '../../marketing/type'; import type { TrackRegisterParams } from '../../marketing/type';
export type GetWXLoginQRResponse = { export type GetWXLoginQRResponse = {
...@@ -9,4 +10,5 @@ export type AccountRegisterBody = { ...@@ -9,4 +10,5 @@ export type AccountRegisterBody = {
username: string; username: string;
code: string; code: string;
password: string; password: string;
language?: `${LangEnum}`;
} & TrackRegisterParams; } & TrackRegisterParams;
packages/global/support/user/type.d.ts
import type { LangEnum } from '../common/i18n/type';
import type { TeamPermission } from '../permission/user/controller'; import type { TeamPermission } from '../permission/user/controller';
import type { UserStatusEnum } from './constant'; import type { UserStatusEnum } from './constant';
import type { TeamMemberStatusEnum } from './team/constant'; import type { TeamMemberStatusEnum } from './team/constant';
...@@ -12,6 +13,7 @@ export type UserModelSchema = { ...@@ -12,6 +13,7 @@ export type UserModelSchema = {
openaiKey: string; openaiKey: string;
createTime: number; createTime: number;
timezone: string; timezone: string;
language: `${LangEnum}`;
status: `${UserStatusEnum}`; status: `${UserStatusEnum}`;
lastLoginTmbId?: string; lastLoginTmbId?: string;
passwordUpdateTime?: Date; passwordUpdateTime?: Date;
...@@ -26,9 +28,9 @@ export type UserType = { ...@@ -26,9 +28,9 @@ export type UserType = {
username: string; username: string;
avatar: string; // it should be team member's avatar after 4.8.18 avatar: string; // it should be team member's avatar after 4.8.18
timezone: string; timezone: string;
language?: `${LangEnum}`;
promotionRate: UserModelSchema['promotionRate']; promotionRate: UserModelSchema['promotionRate'];
team: TeamTmbItemType; team: TeamTmbItemType;
notificationAccount?: string;
permission: TeamPermission; permission: TeamPermission;
contact?: string; contact?: string;
}; };
......
packages/global/support/wallet/bill/api.d.ts deleted 100644 → 0
import type { StandardSubLevelEnum, SubModeEnum } from '../sub/constants';
import type { BillTypeEnum, BillPayWayEnum } from './constants';
import { DrawBillQRItem } from './constants';
export type CreateOrderResponse = {
qrCode?: string;
iframeCode?: string;
markdown?: string;
};
export type CreateStandPlanBill = {
type: BillTypeEnum.standSubPlan;
level: `${StandardSubLevelEnum}`;
subMode: `${SubModeEnum}`;
};
type CreateExtractPointsBill = {
type: BillTypeEnum.extraPoints;
extraPoints: number;
};
type CreateExtractDatasetBill = {
type: BillTypeEnum.extraDatasetSub;
extraDatasetSize: number;
month: number;
};
export type CreateBillProps =
| CreateStandPlanBill
| CreateExtractPointsBill
| CreateExtractDatasetBill;
export type CreateBillResponse = {
billId: string;
readPrice: number;
payment: BillPayWayEnum;
} & CreateOrderResponse;
export type UpdatePaymentProps = {
billId: string;
payWay: BillPayWayEnum;
};
export type CheckPayResultResponse = {
status: BillStatusEnum;
description?: string;
};
packages/global/support/wallet/bill/tools.ts
import type { PriceOption } from './type'; import type { PriceOption } from './type';
// 根据积分获取月份
export const getMonthByPoints = (points: number) => {
if (points >= 200) return 12;
if (points >= 100) return 6;
if (points >= 50) return 3;
return 1;
};
// 根据月份获取积分下限 // 根据月份获取积分下限
export const getMinPointsByMonth = (month: number): number => { export const getMinPointsByMonth = (month: number): number => {
switch (month) { switch (month) {
......
packages/global/support/wallet/bill/type.d.ts deleted 100644 → 0
import type { StandardSubLevelEnum, SubModeEnum } from '../sub/constants';
import { SubTypeEnum } from '../sub/constants';
import type { BillPayWayEnum, BillStatusEnum, BillTypeEnum } from './constants';
import type { TeamInvoiceHeaderType } from '../../user/team/type';
export type BillSchemaType = {
_id: string;
userId: string;
teamId: string;
tmbId: string;
createTime: Date;
orderId: string;
status: `${BillStatusEnum}`;
type: BillTypeEnum;
price: number;
hasInvoice: boolean;
metadata: {
payWay: `${BillPayWayEnum}`;
subMode?: `${SubModeEnum}`;
standSubLevel?: `${StandardSubLevelEnum}`;
month?: number;
datasetSize?: number;
extraPoints?: number;
};
refundData?: {
amount: number;
refundId: string;
refundTime: Date;
};
};
export type ChatNodeUsageType = {
inputTokens?: number;
outputTokens?: number;
totalPoints: number;
moduleName: string;
model?: string;
};
export type InvoiceType = {
amount: number;
billIdList: string[];
} & TeamInvoiceHeaderType;
export type InvoiceSchemaType = {
_id: string;
teamId: string;
status: 1 | 2;
createTime: Date;
finishTime?: Date;
file?: Buffer;
} & InvoiceType;
export type AIPointsPriceOption = {
type: 'points';
points: number;
};
export type DatasetPriceOption = {
type: 'dataset';
size: number;
month: number;
};
export type PriceOption = AIPointsPriceOption | DatasetPriceOption;
packages/global/support/wallet/bill/type.ts 0 → 100644
import { StandardSubLevelEnum, SubModeEnum } from '../sub/constants';
import { SubTypeEnum } from '../sub/constants';
import { BillPayWayEnum, BillStatusEnum, BillTypeEnum } from './constants';
import type { TeamInvoiceHeaderType } from '../../user/team/type';
import { z } from 'zod';
import { ObjectIdSchema } from '../../../common/type/mongo';
export const BillSchema = z.object({
_id: ObjectIdSchema.meta({ description: '订单 ID' }),
userId: ObjectIdSchema.meta({ description: '用户 ID' }),
teamId: ObjectIdSchema.meta({ description: '团队 ID' }),
tmbId: ObjectIdSchema.meta({ description: '团队成员 ID' }),
createTime: z.coerce.date().meta({ description: '创建时间' }),
orderId: z.string().meta({ description: '订单 ID' }),
status: z.enum(BillStatusEnum).meta({ description: '订单状态' }),
type: z.enum(BillTypeEnum).meta({ description: '订单类型' }),
price: z.number().meta({ description: '价格' }),
couponId: ObjectIdSchema.optional().meta({
description: '优惠券 ID'
}),
hasInvoice: z.boolean().meta({ description: '是否已开发票' }),
metadata: z
.object({
payWay: z.enum(BillPayWayEnum).meta({ description: '支付方式' }),
subMode: z.enum(SubModeEnum).optional().meta({ description: '订阅周期' }),
standSubLevel: z.enum(StandardSubLevelEnum).optional().meta({ description: '订阅等级' }),
month: z.number().optional().meta({ description: '月数' }),
datasetSize: z.number().optional().meta({ description: '数据集大小' }),
extraPoints: z.number().optional().meta({ description: '额外积分' })
})
.meta({ description: '元数据' }),
refundData: z
.object({
amount: z.number().meta({ description: '退款金额' }),
refundId: z.string().meta({ description: '退款 ID' }),
refundTime: z.date().meta({ description: '退款时间' })
})
.optional()
.meta({ description: '退款数据' })
});
export type BillSchemaType = z.infer<typeof BillSchema>;
export type ChatNodeUsageType = {
inputTokens?: number;
outputTokens?: number;
totalPoints: number;
moduleName: string;
model?: string;
};
export type InvoiceType = {
amount: number;
billIdList: string[];
} & TeamInvoiceHeaderType;
export type InvoiceSchemaType = {
_id: string;
teamId: string;
status: 1 | 2;
createTime: Date;
finishTime?: Date;
file?: Buffer;
} & InvoiceType;
export type AIPointsPriceOption = {
type: 'points';
points: number;
};
export type DatasetPriceOption = {
type: 'dataset';
size: number;
month: number;
};
export type PriceOption = AIPointsPriceOption | DatasetPriceOption;
packages/global/support/wallet/sub/constants.ts
...@@ -9,17 +9,17 @@ export enum SubTypeEnum { ...@@ -9,17 +9,17 @@ export enum SubTypeEnum {
export const subTypeMap = { export const subTypeMap = {
[SubTypeEnum.standard]: { [SubTypeEnum.standard]: {
label: 'support.wallet.subscription.type.standard', label: i18nT('common:support.wallet.subscription.type.standard'),
icon: 'support/account/plans', icon: 'support/account/plans',
orderType: BillTypeEnum.standSubPlan orderType: BillTypeEnum.standSubPlan
}, },
[SubTypeEnum.extraDatasetSize]: { [SubTypeEnum.extraDatasetSize]: {
label: 'support.wallet.subscription.type.extraDatasetSize', label: i18nT('common:support.wallet.subscription.type.extraDatasetSize'),
icon: 'core/dataset/datasetLight', icon: 'core/dataset/datasetLight',
orderType: BillTypeEnum.extraDatasetSub orderType: BillTypeEnum.extraDatasetSub
}, },
[SubTypeEnum.extraPoints]: { [SubTypeEnum.extraPoints]: {
label: 'support.wallet.subscription.type.extraPoints', label: i18nT('common:support.wallet.subscription.type.extraPoints'),
icon: 'core/chat/chatLight', icon: 'core/chat/chatLight',
orderType: BillTypeEnum.extraPoints orderType: BillTypeEnum.extraPoints
} }
...@@ -31,12 +31,12 @@ export enum SubModeEnum { ...@@ -31,12 +31,12 @@ export enum SubModeEnum {
} }
export const subModeMap = { export const subModeMap = {
[SubModeEnum.month]: { [SubModeEnum.month]: {
label: 'support.wallet.subscription.mode.Month', label: i18nT('common:support.wallet.subscription.mode.Month'),
durationMonth: 1, durationMonth: 1,
payMonth: 1 payMonth: 1
}, },
[SubModeEnum.year]: { [SubModeEnum.year]: {
label: 'support.wallet.subscription.mode.Year', label: i18nT('common:support.wallet.subscription.mode.Year'),
durationMonth: 12, durationMonth: 12,
payMonth: 10 payMonth: 10
} }
...@@ -44,17 +44,39 @@ export const subModeMap = { ...@@ -44,17 +44,39 @@ export const subModeMap = {
export enum StandardSubLevelEnum { export enum StandardSubLevelEnum {
free = 'free', free = 'free',
basic = 'basic',
advanced = 'advanced',
custom = 'custom',
// @deprecated
experience = 'experience', experience = 'experience',
team = 'team', team = 'team',
enterprise = 'enterprise', enterprise = 'enterprise'
custom = 'custom'
} }
export const standardSubLevelMap = { export const standardSubLevelMap = {
[StandardSubLevelEnum.free]: { [StandardSubLevelEnum.free]: {
label: i18nT('common:support.wallet.subscription.standardSubLevel.free'), label: i18nT('common:support.wallet.subscription.standardSubLevel.free'),
desc: i18nT('common:support.wallet.subscription.standardSubLevel.free desc'), desc: i18nT('common:support.wallet.subscription.standardSubLevel.free desc'),
weight: 1 weight: 1
}, },
[StandardSubLevelEnum.basic]: {
label: i18nT('common:support.wallet.subscription.standardSubLevel.basic'),
desc: i18nT('common:support.wallet.subscription.standardSubLevel.basic_desc'),
weight: 4
},
[StandardSubLevelEnum.advanced]: {
label: i18nT('common:support.wallet.subscription.standardSubLevel.advanced'),
desc: i18nT('common:support.wallet.subscription.standardSubLevel.advanced_desc'),
weight: 5
},
[StandardSubLevelEnum.custom]: {
label: i18nT('common:support.wallet.subscription.standardSubLevel.custom'),
desc: i18nT('common:support.wallet.subscription.standardSubLevel.custom_desc'),
weight: 7
},
// deprecated
[StandardSubLevelEnum.experience]: { [StandardSubLevelEnum.experience]: {
label: i18nT('common:support.wallet.subscription.standardSubLevel.experience'), label: i18nT('common:support.wallet.subscription.standardSubLevel.experience'),
desc: i18nT('common:support.wallet.subscription.standardSubLevel.experience_desc'), desc: i18nT('common:support.wallet.subscription.standardSubLevel.experience_desc'),
...@@ -68,11 +90,6 @@ export const standardSubLevelMap = { ...@@ -68,11 +90,6 @@ export const standardSubLevelMap = {
[StandardSubLevelEnum.enterprise]: { [StandardSubLevelEnum.enterprise]: {
label: i18nT('common:support.wallet.subscription.standardSubLevel.enterprise'), label: i18nT('common:support.wallet.subscription.standardSubLevel.enterprise'),
desc: i18nT('common:support.wallet.subscription.standardSubLevel.enterprise_desc'), desc: i18nT('common:support.wallet.subscription.standardSubLevel.enterprise_desc'),
weight: 4 weight: 6
},
[StandardSubLevelEnum.custom]: {
label: i18nT('common:support.wallet.subscription.standardSubLevel.custom'),
desc: '',
weight: 5
} }
}; };
packages/global/support/wallet/sub/coupon/type.d.ts
import type { SubTypeEnum, StandardSubLevelEnum } from '../constants'; import type { SubTypeEnum, StandardSubLevelEnum } from '../constants';
import type { CouponTypeEnum } from './constants'; import type { CouponTypeEnum } from './constants';
export type CustomSubConfig = {
requestsPerMinute: number;
maxTeamMember: number;
maxAppAmount: number;
maxDatasetAmount: number;
chatHistoryStoreDuration: number;
maxDatasetSize: number;
websiteSyncPerDataset: number;
appRegistrationCount: number;
auditLogStoreDuration: number;
ticketResponseTime: number;
};
export type TeamCouponSub = { export type TeamCouponSub = {
type: `${SubTypeEnum}`; // Sub type type: `${SubTypeEnum}`; // Sub type
durationDay: number; // Duration day durationDay: number; // Duration day
level?: `${StandardSubLevelEnum}`; // Standard sub level level?: `${StandardSubLevelEnum}`; // Standard sub level
extraDatasetSize?: number; // Extra dataset size extraDatasetSize?: number; // Extra dataset size
totalPoints?: number; // Total points(Extrapoints or Standard sub) totalPoints?: number; // Total points(Extrapoints or Standard sub)
customConfig?: CustomSubConfig; // Custom config for custom level (only required when level=custom)
}; };
export type TeamCouponSchema = { export type TeamCouponSchema = {
......
packages/global/support/wallet/sub/discountCoupon/constants.ts 0 → 100644
import { i18nT } from '../../../../../web/i18n/utils';
export enum DiscountCouponTypeEnum {
monthStandardDiscount70 = 'monthStandardDiscount70',
yearStandardDiscount90 = 'yearStandardDiscount90'
}
export enum DiscountCouponStatusEnum {
active = 'active',
used = 'used',
expired = 'expired',
notStart = 'notStart'
}
// Discount coupon type config table, modify to add or update types.
export const discountCouponTypeMap = {
[DiscountCouponTypeEnum.monthStandardDiscount70]: {
type: DiscountCouponTypeEnum.monthStandardDiscount70,
name: i18nT('common:old_user_month_discount_70'),
description: i18nT('common:old_user_month_discount_70_description'),
discount: 0.7,
iconZh: '/imgs/system/discount70CN.svg',
iconEn: '/imgs/system/discount70EN.svg'
},
[DiscountCouponTypeEnum.yearStandardDiscount90]: {
type: DiscountCouponTypeEnum.yearStandardDiscount90,
name: i18nT('common:old_user_year_discount_90'),
description: i18nT('common:old_user_year_discount_90_description'),
discount: 0.9,
iconZh: '/imgs/system/discount90CN.svg',
iconEn: '/imgs/system/discount90EN.svg'
}
};
packages/global/support/wallet/sub/type.d.ts
...@@ -3,19 +3,28 @@ import type { StandardSubLevelEnum, SubModeEnum, SubTypeEnum } from './constants ...@@ -3,19 +3,28 @@ import type { StandardSubLevelEnum, SubModeEnum, SubTypeEnum } from './constants
// Content of plan // Content of plan
export type TeamStandardSubPlanItemType = { export type TeamStandardSubPlanItemType = {
name?: string; name?: string;
desc?: string; // Plan description
price: number; // read price / month price: number; // read price / month
pointPrice: number; // read price/ one thousand pointPrice: number; // read price/ one thousand
totalPoints: number; // n totalPoints: number; // n
maxTeamMember: number; maxTeamMember: number;
maxAppAmount: number; // max app or plugin amount maxAppAmount: number; // max app or plugin amount
maxDatasetAmount: number; maxDatasetAmount: number;
chatHistoryStoreDuration: number; // n day
maxDatasetSize: number; maxDatasetSize: number;
trainingWeight: number; // 1~4
permissionCustomApiKey: boolean; requestsPerMinute?: number;
permissionCustomCopyright: boolean; // feature appRegistrationCount?: number;
permissionWebsiteSync: boolean; chatHistoryStoreDuration: number; // n day
permissionTeamOperationLog: boolean; websiteSyncPerDataset?: number;
auditLogStoreDuration?: number;
ticketResponseTime?: number;
// Custom plan specific fields
priceDescription?: string;
customFormUrl?: string;
customDescriptions?: string[];
}; };
export type StandSubPlanLevelMapType = Record< export type StandSubPlanLevelMapType = Record<
...@@ -23,14 +32,21 @@ export type StandSubPlanLevelMapType = Record< ...@@ -23,14 +32,21 @@ export type StandSubPlanLevelMapType = Record<
TeamStandardSubPlanItemType TeamStandardSubPlanItemType
>; >;
export type PointsPackageItem = {
points: number;
month: number;
price: number;
};
export type SubPlanType = { export type SubPlanType = {
[SubTypeEnum.standard]: StandSubPlanLevelMapType; [SubTypeEnum.standard]?: StandSubPlanLevelMapType;
planDescriptionUrl?: string; planDescriptionUrl?: string;
appRegistrationUrl?: string;
[SubTypeEnum.extraDatasetSize]: { [SubTypeEnum.extraDatasetSize]: {
price: number; price: number;
}; };
[SubTypeEnum.extraPoints]: { [SubTypeEnum.extraPoints]: {
price: number; packages: PointsPackageItem[];
}; };
}; };
...@@ -49,6 +65,15 @@ export type TeamSubSchema = { ...@@ -49,6 +65,15 @@ export type TeamSubSchema = {
maxApp?: number; maxApp?: number;
maxDataset?: number; maxDataset?: number;
// custom level configurations
requestsPerMinute?: number;
chatHistoryStoreDuration?: number;
maxDatasetSize?: number;
websiteSyncPerDataset?: number;
appRegistrationCount?: number;
auditLogStoreDuration?: number;
ticketResponseTime?: number;
totalPoints: number; totalPoints: number;
surplusPoints: number; surplusPoints: number;
...@@ -71,4 +96,5 @@ export type ClientTeamPlanStatusType = TeamPlanStatusType & { ...@@ -71,4 +96,5 @@ export type ClientTeamPlanStatusType = TeamPlanStatusType & {
usedAppAmount: number; usedAppAmount: number;
usedDatasetSize: number; usedDatasetSize: number;
usedDatasetIndexSize: number; usedDatasetIndexSize: number;
usedRegistrationCount: number;
}; };
packages/service/common/buffer/rawText/controller.ts deleted 100644 → 0
import { retryFn } from '@fastgpt/global/common/system/utils';
import { connectionMongo, Types } from '../../mongo';
import { MongoRawTextBufferSchema, bucketName } from './schema';
import { addLog } from '../../system/log';
import { setCron } from '../../system/cron';
import { checkTimerLock } from '../../system/timerLock/utils';
import { TimerIdEnum } from '../../system/timerLock/constants';
import { gridFsStream2Buffer } from '../../file/gridfs/utils';
import { readRawContentFromBuffer } from '../../../worker/function';
const getGridBucket = () => {
return new connectionMongo.mongo.GridFSBucket(connectionMongo.connection.db!, {
bucketName: bucketName
});
};
export const addRawTextBuffer = async ({
sourceId,
sourceName,
text,
expiredTime
}: {
sourceId: string;
sourceName: string;
text: string;
expiredTime: Date;
}) => {
const gridBucket = getGridBucket();
const metadata = {
sourceId,
sourceName,
expiredTime
};
const buffer = Buffer.from(text);
const fileSize = buffer.length;
// 单块大小：尽可能大，但不超过 14MB，不小于128KB
const chunkSizeBytes = (() => {
// 计算理想块大小：文件大小 ÷ 目标块数(10)。 并且每个块需要小于 14MB
const idealChunkSize = Math.min(Math.ceil(fileSize / 10), 14 * 1024 * 1024);
// 确保块大小至少为128KB
const minChunkSize = 128 * 1024; // 128KB
// 取理想块大小和最小块大小中的较大值
let chunkSize = Math.max(idealChunkSize, minChunkSize);
// 将块大小向上取整到最接近的64KB的倍数，使其更整齐
chunkSize = Math.ceil(chunkSize / (64 * 1024)) * (64 * 1024);
return chunkSize;
})();
const uploadStream = gridBucket.openUploadStream(sourceId, {
metadata,
chunkSizeBytes
});
return retryFn(async () => {
return new Promise((resolve, reject) => {
uploadStream.end(buffer);
uploadStream.on('finish', () => {
resolve(uploadStream.id);
});
uploadStream.on('error', (error) => {
addLog.error('addRawTextBuffer error', error);
resolve('');
});
});
});
};
export const getRawTextBuffer = async (sourceId: string) => {
const gridBucket = getGridBucket();
return retryFn(async () => {
const bufferData = await MongoRawTextBufferSchema.findOne(
{
'metadata.sourceId': sourceId
},
'_id metadata'
).lean();
if (!bufferData) {
return null;
}
// Read file content
const downloadStream = gridBucket.openDownloadStream(new Types.ObjectId(bufferData._id));
const fileBuffers = await gridFsStream2Buffer(downloadStream);
const rawText = await (async () => {
if (fileBuffers.length < 10000000) {
return fileBuffers.toString('utf8');
} else {
return (
await readRawContentFromBuffer({
extension: 'txt',
encoding: 'utf8',
buffer: fileBuffers
})
).rawText;
}
})();
return {
text: rawText,
sourceName: bufferData.metadata?.sourceName || ''
};
});
};
export const deleteRawTextBuffer = async (sourceId: string): Promise<boolean> => {
const gridBucket = getGridBucket();
return retryFn(async () => {
const buffer = await MongoRawTextBufferSchema.findOne({ 'metadata.sourceId': sourceId });
if (!buffer) {
return false;
}
await gridBucket.delete(new Types.ObjectId(buffer._id));
return true;
});
};
export const updateRawTextBufferExpiredTime = async ({
sourceId,
expiredTime
}: {
sourceId: string;
expiredTime: Date;
}) => {
return retryFn(async () => {
return MongoRawTextBufferSchema.updateOne(
{ 'metadata.sourceId': sourceId },
{ $set: { 'metadata.expiredTime': expiredTime } }
);
});
};
export const clearExpiredRawTextBufferCron = async () => {
const gridBucket = getGridBucket();
const clearExpiredRawTextBuffer = async () => {
addLog.debug('Clear expired raw text buffer start');
const data = await MongoRawTextBufferSchema.find(
{
'metadata.expiredTime': { $lt: new Date() }
},
'_id'
).lean();
for (const item of data) {
try {
await gridBucket.delete(new Types.ObjectId(item._id));
} catch (error) {
addLog.error('Delete expired raw text buffer error', error);
}
}
addLog.debug('Clear expired raw text buffer end');
};
setCron('*/10 * * * *', async () => {
if (
await checkTimerLock({
timerId: TimerIdEnum.clearExpiredRawTextBuffer,
lockMinuted: 9
})
) {
try {
await clearExpiredRawTextBuffer();
} catch (error) {
addLog.error('clearExpiredRawTextBufferCron error', error);
}
}
});
};
packages/service/common/buffer/rawText/schema.ts deleted 100644 → 0
import { getMongoModel, type Types, Schema } from '../../mongo';
export const bucketName = 'buffer_rawtext';
const RawTextBufferSchema = new Schema({
metadata: {
sourceId: { type: String, required: true },
sourceName: { type: String, required: true },
expiredTime: { type: Date, required: true }
}
});
RawTextBufferSchema.index({ 'metadata.sourceId': 'hashed' });
RawTextBufferSchema.index({ 'metadata.expiredTime': -1 });
export const MongoRawTextBufferSchema = getMongoModel<{
_id: Types.ObjectId;
metadata: {
sourceId: string;
sourceName: string;
expiredTime: Date;
};
}>(`${bucketName}.files`, RawTextBufferSchema);
packages/service/common/bullmq/index.ts
...@@ -22,7 +22,10 @@ export enum QueueNames { ...@@ -22,7 +22,10 @@ export enum QueueNames {
datasetSync = 'datasetSync', datasetSync = 'datasetSync',
evaluation = 'evaluation', evaluation = 'evaluation',
s3FileDelete = 's3FileDelete', s3FileDelete = 's3FileDelete',
// abondoned
// Delete Queue
datasetDelete = 'datasetDelete',
// @deprecated
websiteSync = 'websiteSync' websiteSync = 'websiteSync'
} }
......
packages/service/common/file/gridfs/controller.ts
import { Types, connectionMongo, ReadPreference } from '../../mongo'; import { Types, connectionMongo, ReadPreference } from '../../mongo';
import type { BucketNameEnum } from '@fastgpt/global/common/file/constants'; import type { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import fsp from 'fs/promises';
import fs from 'fs';
import { type DatasetFileSchema } from '@fastgpt/global/core/dataset/type';
import { MongoChatFileSchema, MongoDatasetFileSchema } from './schema'; import { MongoChatFileSchema, MongoDatasetFileSchema } from './schema';
import { detectFileEncodingByPath } from '@fastgpt/global/common/file/tools';
import { computeGridFsChunSize, stream2Encoding } from './utils';
import { addLog } from '../../system/log';
import { Readable } from 'stream';
import { retryFn } from '@fastgpt/global/common/system/utils';
import { getS3DatasetSource } from '../../s3/sources/dataset';
import { isS3ObjectKey } from '../../s3/utils';
export function getGFSCollection(bucket: `${BucketNameEnum}`) { export function getGFSCollection(bucket: `${BucketNameEnum}`) {
MongoDatasetFileSchema; MongoDatasetFileSchema;
...@@ -18,6 +8,7 @@ export function getGFSCollection(bucket: `${BucketNameEnum}`) { ...@@ -18,6 +8,7 @@ export function getGFSCollection(bucket: `${BucketNameEnum}`) {
return connectionMongo.connection.db!.collection(`${bucket}.files`); return connectionMongo.connection.db!.collection(`${bucket}.files`);
} }
export function getGridBucket(bucket: `${BucketNameEnum}`) { export function getGridBucket(bucket: `${BucketNameEnum}`) {
return new connectionMongo.mongo.GridFSBucket(connectionMongo.connection.db!, { return new connectionMongo.mongo.GridFSBucket(connectionMongo.connection.db!, {
bucketName: bucket, bucketName: bucket,
...@@ -26,106 +17,6 @@ export function getGridBucket(bucket: `${BucketNameEnum}`) { ...@@ -26,106 +17,6 @@ export function getGridBucket(bucket: `${BucketNameEnum}`) {
}); });
} }
/* crud file */
export async function uploadFile({
bucketName,
teamId,
uid,
path,
filename,
contentType,
metadata = {}
}: {
bucketName: `${BucketNameEnum}`;
teamId: string;
uid: string; // tmbId / outLinkUId
path: string;
filename: string;
contentType?: string;
metadata?: Record<string, any>;
}) {
if (!path) return Promise.reject(`filePath is empty`);
if (!filename) return Promise.reject(`filename is empty`);
const stats = await fsp.stat(path);
if (!stats.isFile()) return Promise.reject(`${path} is not a file`);
const readStream = fs.createReadStream(path, {
highWaterMark: 256 * 1024
});
// Add default metadata
metadata.teamId = teamId;
metadata.uid = uid;
metadata.encoding = await detectFileEncodingByPath(path);
// create a gridfs bucket
const bucket = getGridBucket(bucketName);
const chunkSizeBytes = computeGridFsChunSize(stats.size);
const stream = bucket.openUploadStream(filename, {
metadata,
contentType,
chunkSizeBytes
});
// save to gridfs
await new Promise((resolve, reject) => {
readStream
.pipe(stream as any)
.on('finish', resolve)
.on('error', reject);
}).finally(() => {
readStream.destroy();
});
return String(stream.id);
}
export async function getFileById({
bucketName,
fileId
}: {
bucketName: `${BucketNameEnum}`;
fileId: string;
}) {
const db = getGFSCollection(bucketName);
const file = await db.findOne<DatasetFileSchema>({
_id: new Types.ObjectId(fileId)
});
return file || undefined;
}
export async function delFileByFileIdList({
bucketName,
fileIdList
}: {
bucketName: `${BucketNameEnum}`;
fileIdList: string[];
}): Promise<any> {
return retryFn(async () => {
const bucket = getGridBucket(bucketName);
for await (const fileId of fileIdList) {
try {
if (isS3ObjectKey(fileId, 'dataset')) {
await getS3DatasetSource().deleteDatasetFileByKey(fileId);
} else {
await bucket.delete(new Types.ObjectId(String(fileId)));
}
} catch (error: any) {
if (typeof error?.message === 'string' && error.message.includes('File not found')) {
addLog.warn('File not found', { fileId });
return;
}
return Promise.reject(error);
}
}
});
}
export async function getDownloadStream({ export async function getDownloadStream({
bucketName, bucketName,
fileId fileId
......
packages/service/common/file/gridfs/utils.ts
import { detectFileEncoding } from '@fastgpt/global/common/file/tools'; import { detectFileEncoding } from '@fastgpt/global/common/file/tools';
import { PassThrough } from 'stream'; import { PassThrough } from 'stream';
import { getGridBucket } from './controller';
import { type BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { retryFn } from '@fastgpt/global/common/system/utils';
export const createFileFromText = async ({
bucket,
filename,
text,
metadata
}: {
bucket: `${BucketNameEnum}`;
filename: string;
text: string;
metadata: Record<string, any>;
}) => {
const gridBucket = getGridBucket(bucket);
const buffer = Buffer.from(text);
const fileSize = buffer.length;
// 单块大小：尽可能大，但不超过 14MB，不小于128KB
const chunkSizeBytes = (() => {
// 计算理想块大小：文件大小 ÷ 目标块数(10)。 并且每个块需要小于 14MB
const idealChunkSize = Math.min(Math.ceil(fileSize / 10), 14 * 1024 * 1024);
// 确保块大小至少为128KB
const minChunkSize = 128 * 1024; // 128KB
// 取理想块大小和最小块大小中的较大值
let chunkSize = Math.max(idealChunkSize, minChunkSize);
// 将块大小向上取整到最接近的64KB的倍数，使其更整齐
chunkSize = Math.ceil(chunkSize / (64 * 1024)) * (64 * 1024);
return chunkSize;
})();
const uploadStream = gridBucket.openUploadStream(filename, {
metadata,
chunkSizeBytes
});
return retryFn(async () => {
return new Promise<{ fileId: string }>((resolve, reject) => {
uploadStream.end(buffer);
uploadStream.on('finish', () => {
resolve({ fileId: String(uploadStream.id) });
});
uploadStream.on('error', reject);
});
});
};
export const gridFsStream2Buffer = (stream: NodeJS.ReadableStream) => {
return new Promise<Buffer>((resolve, reject) => {
if (!stream.readable) {
return resolve(Buffer.from([]));
}
const chunks: Uint8Array[] = [];
stream.on('data', (chunk) => {
chunks.push(chunk);
});
stream.on('end', () => {
const resultBuffer = Buffer.concat(chunks); // One-time splicing
resolve(resultBuffer);
});
stream.on('error', (err) => {
reject(err);
});
});
};
export const stream2Encoding = async (stream: NodeJS.ReadableStream) => { export const stream2Encoding = async (stream: NodeJS.ReadableStream) => {
const copyStream = stream.pipe(new PassThrough()); const copyStream = stream.pipe(new PassThrough());
...@@ -109,20 +36,3 @@ export const stream2Encoding = async (stream: NodeJS.ReadableStream) => { ...@@ -109,20 +36,3 @@ export const stream2Encoding = async (stream: NodeJS.ReadableStream) => {
stream: copyStream stream: copyStream
}; };
}; };
// 单块大小：尽可能大，但不超过 14MB，不小于512KB
export const computeGridFsChunSize = (fileSize: number) => {
// 计算理想块大小：文件大小 ÷ 目标块数(10)。 并且每个块需要小于 14MB
const idealChunkSize = Math.min(Math.ceil(fileSize / 10), 14 * 1024 * 1024);
// 确保块大小至少为512KB
const minChunkSize = 512 * 1024; // 512KB
// 取理想块大小和最小块大小中的较大值
let chunkSize = Math.max(idealChunkSize, minChunkSize);
// 将块大小向上取整到最接近的64KB的倍数，使其更整齐
chunkSize = Math.ceil(chunkSize / (64 * 1024)) * (64 * 1024);
return chunkSize;
};
packages/service/common/file/image/controller.ts
...@@ -78,9 +78,10 @@ export const copyAvatarImage = async ({ ...@@ -78,9 +78,10 @@ export const copyAvatarImage = async ({
const avatarSource = getS3AvatarSource(); const avatarSource = getS3AvatarSource();
if (isS3ObjectKey(imageUrl?.slice(avatarSource.prefix.length), 'avatar')) { if (isS3ObjectKey(imageUrl?.slice(avatarSource.prefix.length), 'avatar')) {
const filename = (() => { const filename = (() => {
const last = imageUrl.split('/').pop()?.split('-')[1]; const last = imageUrl.split('/').pop();
if (!last) return getNanoid(6).concat(path.extname(imageUrl)); if (!last) return getNanoid(6).concat(path.extname(imageUrl));
return `${getNanoid(6)}-${last}`; const firstDashIndex = last.indexOf('-');
return `${getNanoid(6)}-${firstDashIndex === -1 ? last : last.slice(firstDashIndex + 1)}`;
})(); })();
const key = await getS3AvatarSource().copyAvatar({ const key = await getS3AvatarSource().copyAvatar({
key: imageUrl, key: imageUrl,
......
packages/service/common/file/multer.ts
import type { NextApiRequest, NextApiResponse } from 'next';
import multer from 'multer';
import path from 'path';
import type { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { bucketNameMap } from '@fastgpt/global/common/file/constants';
import { getNanoid } from '@fastgpt/global/common/string/tools'; import { getNanoid } from '@fastgpt/global/common/string/tools';
import { UserError } from '@fastgpt/global/common/error/utils'; import m from 'multer';
import type { NextApiRequest } from 'next';
export type FileType = { import path from 'path';
fieldname: string; import fs from 'node:fs';
originalname: string;
encoding: string; export const multer = {
mimetype: string; _storage: m.diskStorage({
filename: string; filename: (_, file, cb) => {
path: string; if (!file?.originalname) {
size: number; cb(new Error('File not found'), '');
}; } else {
const ext = path.extname(decodeURIComponent(file.originalname));
cb(null, `${getNanoid()}${ext}`);
}
}
}),
/* singleStore(maxFileSize: number = 500) {
maxSize: File max size (MB) const fileSize = maxFileSize * 1024 * 1024;
*/
export const getUploadModel = ({ maxSize = 500 }: { maxSize?: number }) => {
maxSize *= 1024 * 1024;
class UploadModel { return m({
uploaderSingle = multer({
limits: { limits: {
fieldSize: maxSize fileSize
}, },
preservePath: true, preservePath: true,
storage: multer.diskStorage({ storage: this._storage
// destination: (_req, _file, cb) => {
// cb(null, tmpFileDirPath);
// },
filename: (req, file, cb) => {
if (!file?.originalname) {
cb(new Error('File not found'), '');
} else {
const { ext } = path.parse(decodeURIComponent(file.originalname));
cb(null, `${getNanoid()}${ext}`);
}
}
})
}).single('file'); }).single('file');
async getUploadFile<T = any>( },
req: NextApiRequest,
res: NextApiResponse,
originBucketName?: `${BucketNameEnum}`
) {
return new Promise<{
file: FileType;
metadata: Record<string, any>;
data: T;
bucketName?: `${BucketNameEnum}`;
}>((resolve, reject) => {
// @ts-ignore
this.uploaderSingle(req, res, (error) => {
if (error) {
return reject(error);
}
// check bucket name multipleStore(maxFileSize: number = 500) {
const bucketName = (req.body?.bucketName || originBucketName) as `${BucketNameEnum}`; const fileSize = maxFileSize * 1024 * 1024;
if (bucketName && !bucketNameMap[bucketName]) {
return reject(new UserError('BucketName is invalid'));
}
// @ts-ignore return m({
const file = req.file as FileType;
resolve({
file: {
...file,
originalname: decodeURIComponent(file.originalname)
},
bucketName,
metadata: (() => {
if (!req.body?.metadata) return {};
try {
return JSON.parse(req.body.metadata);
} catch (error) {
return {};
}
})(),
data: (() => {
if (!req.body?.data) return {};
try {
return JSON.parse(req.body.data);
} catch (error) {
return {};
}
})()
});
});
});
}
uploaderMultiple = multer({
limits: { limits: {
fieldSize: maxSize fileSize
}, },
preservePath: true, preservePath: true,
storage: multer.diskStorage({ storage: this._storage
// destination: (_req, _file, cb) => { }).array('file', global.feConfigs?.uploadFileMaxSize);
// cb(null, tmpFileDirPath); },
// },
filename: (req, file, cb) => { resolveFormData<T extends Record<string, any>>({
if (!file?.originalname) { request,
cb(new Error('File not found'), ''); maxFileSize
} else { }: {
const { ext } = path.parse(decodeURIComponent(file.originalname)); request: NextApiRequest;
cb(null, `${getNanoid()}${ext}`); maxFileSize?: number;
}) {
return new Promise<{
data: T;
fileMetadata: Express.Multer.File;
getBuffer: () => Buffer;
getReadStream: () => fs.ReadStream;
}>((resolve, reject) => {
const handler = this.singleStore(maxFileSize);
// @ts-expect-error it can accept a NextApiRequest
handler(request, null, (error) => {
if (error) {
return reject(error);
}
// @ts-expect-error `file` will be injected by multer
const file = request.file as Express.Multer.File;
if (!file) {
return reject(new Error('File not found'));
}
const data = (() => {
if (!request.body?.data) return {};
try {
return JSON.parse(request.body.data);
} catch {
return {};
} }
})();
resolve({
data,
fileMetadata: file,
getBuffer: () => fs.readFileSync(file.path),
getReadStream: () => fs.createReadStream(file.path)
});
});
});
},
resolveMultipleFormData<T extends Record<string, any>>({
request,
maxFileSize
}: {
request: NextApiRequest;
maxFileSize?: number;
}) {
return new Promise<{
data: T;
fileMetadata: Array<Express.Multer.File>;
}>((resolve, reject) => {
const handler = this.multipleStore(maxFileSize);
// @ts-expect-error it can accept a NextApiRequest
handler(request, null, (error) => {
if (error) {
return reject(error);
} }
})
}).array('file', global.feConfigs?.uploadFileMaxSize); // @ts-expect-error `files` will be injected by multer
async getUploadFiles<T = any>(req: NextApiRequest, res: NextApiResponse) { const files = request.files as Array<Express.Multer.File>;
return new Promise<{
files: FileType[]; if (!files || files.length === 0) {
data: T; return reject(new Error('File not found'));
}>((resolve, reject) => { }
// @ts-ignore
this.uploaderMultiple(req, res, (error) => { const data = (() => {
if (error) { if (!request.body?.data) return {};
console.log(error); try {
return reject(error); return JSON.parse(request.body.data);
} catch {
return {};
} }
})();
// @ts-ignore resolve({
const files = req.files as FileType[]; data,
fileMetadata: files
resolve({
files: files.map((file) => ({
...file,
originalname: decodeURIComponent(file.originalname)
})),
data: (() => {
if (!req.body?.data) return {};
try {
return JSON.parse(req.body.data);
} catch (error) {
return {};
}
})()
});
}); });
}); });
});
},
clearDiskTempFiles(filepaths: string[]) {
for (const filepath of filepaths) {
fs.rm(filepath, { force: true }, (_) => {});
} }
} }
return new UploadModel();
}; };
packages/service/common/geo/constants.ts 0 → 100644
import path from 'node:path';
import type { LocationName } from './type';
export const dbPath = path.join(process.cwd(), 'data/GeoLite2-City.mmdb');
export const privateOrOtherLocationName: LocationName = {
city: undefined,
country: {
en: 'Other',
zh: '其他'
},
province: undefined
};
export const cleanupIntervalMs = 6 * 60 * 60 * 1000; // Run cleanup every 6 hours
packages/service/common/geo/index.ts 0 → 100644
import fs from 'node:fs';
import type { ReaderModel } from '@maxmind/geoip2-node';
import { Reader } from '@maxmind/geoip2-node';
import { cleanupIntervalMs, dbPath, privateOrOtherLocationName } from './constants';
import type { I18nName, LocationName } from './type';
import { extractLocationData } from './utils';
import type { NextApiRequest } from 'next';
import { getClientIp } from 'request-ip';
import { addLog } from '../system/log';
let reader: ReaderModel | null = null;
const locationIpMap = new Map<string, LocationName>();
function loadGeoDB() {
const dbBuffer = fs.readFileSync(dbPath);
reader = Reader.openBuffer(dbBuffer);
return reader;
}
export function getGeoReader() {
if (!reader) {
return loadGeoDB();
}
return reader;
}
export function getLocationFromIp(ip?: string, locale: keyof I18nName = 'zh') {
if (!ip) {
return privateOrOtherLocationName.country?.[locale];
}
const reader = getGeoReader();
let locationName = locationIpMap.get(ip);
if (locationName) {
return [
locationName.country?.[locale],
locationName.province?.[locale],
locationName.city?.[locale]
]
.filter(Boolean)
.join(locale === 'zh' ? '，' : ',');
}
try {
const response = reader.city(ip);
const data = extractLocationData(response);
locationName = {
city: {
en: data.city.en,
zh: data.city.zh
},
country: {
en: data.country.en,
zh: data.country.zh
},
province: {
en: data.province.en,
zh: data.province.zh
}
};
locationIpMap.set(ip, locationName);
return [
locationName.country?.[locale],
locationName.province?.[locale],
locationName.city?.[locale]
]
.filter(Boolean)
.join(locale === 'zh' ? '，' : ', ');
} catch (error) {
locationIpMap.set(ip, privateOrOtherLocationName);
return privateOrOtherLocationName.country?.[locale];
}
}
let cleanupInterval: NodeJS.Timeout | null = null;
function cleanupIpMap() {
locationIpMap.clear();
}
export function clearCleanupInterval() {
if (cleanupInterval) {
clearInterval(cleanupInterval);
cleanupInterval = null;
}
}
export function initGeo() {
cleanupInterval = setInterval(cleanupIpMap, cleanupIntervalMs);
try {
loadGeoDB();
} catch (error) {
clearCleanupInterval();
addLog.error(`Failed to load geo db`, error);
throw error;
}
}
export function getIpFromRequest(request: NextApiRequest): string {
const ip = getClientIp(request);
if (!ip || ip === '::1') {
return '127.0.0.1';
}
return ip;
}
packages/service/common/geo/type.ts 0 → 100644
export type I18nName = {
zh?: string;
en?: string;
};
export type LocationName = {
country?: I18nName;
province?: I18nName;
city?: I18nName;
};
packages/service/common/geo/utils.ts 0 → 100644
import type { City } from '@maxmind/geoip2-node';
export function extractLocationData(response: City) {
return {
city: {
id: response.city?.geonameId,
en: response.city?.names.en,
zh: response.city?.names['zh-CN']
},
country: {
id: response.country?.geonameId,
en: response.country?.names.en,
zh: response.country?.names['zh-CN']
},
province: {
id: response.subdivisions?.[0]?.geonameId,
en: response.subdivisions?.[0]?.names.en,
zh: response.subdivisions?.[0]?.names['zh-CN']
}
};
}
packages/service/common/middle/tracks/processor.ts
...@@ -15,6 +15,13 @@ const getCurrentTenMinuteBoundary = () => { ...@@ -15,6 +15,13 @@ const getCurrentTenMinuteBoundary = () => {
return boundary; return boundary;
}; };
const getCurrentMinuteBoundary = () => {
const now = new Date();
const boundary = new Date(now);
boundary.setSeconds(0, 0);
return boundary;
};
export const trackTimerProcess = async () => { export const trackTimerProcess = async () => {
while (true) { while (true) {
await countTrackTimer(); await countTrackTimer();
...@@ -31,7 +38,8 @@ export const countTrackTimer = async () => { ...@@ -31,7 +38,8 @@ export const countTrackTimer = async () => {
global.countTrackQueue = new Map(); global.countTrackQueue = new Map();
try { try {
const currentBoundary = getCurrentTenMinuteBoundary(); const currentTenMinuteBoundary = getCurrentTenMinuteBoundary();
const currentMinuteBoundary = getCurrentMinuteBoundary();
const bulkOps = queuedItems const bulkOps = queuedItems
.map(({ event, count, data }) => { .map(({ event, count, data }) => {
...@@ -44,7 +52,7 @@ export const countTrackTimer = async () => { ...@@ -44,7 +52,7 @@ export const countTrackTimer = async () => {
filter: { filter: {
event, event,
teamId, teamId,
createTime: currentBoundary, createTime: currentTenMinuteBoundary,
'data.datasetId': datasetId 'data.datasetId': datasetId
}, },
update: [ update: [
...@@ -52,7 +60,7 @@ export const countTrackTimer = async () => { ...@@ -52,7 +60,7 @@ export const countTrackTimer = async () => {
$set: { $set: {
event, event,
teamId, teamId,
createTime: { $ifNull: ['$createTime', currentBoundary] }, createTime: { $ifNull: ['$createTime', currentTenMinuteBoundary] },
data: { data: {
datasetId, datasetId,
count: { $add: [{ $ifNull: ['$data.count', 0] }, count] } count: { $add: [{ $ifNull: ['$data.count', 0] }, count] }
...@@ -65,6 +73,36 @@ export const countTrackTimer = async () => { ...@@ -65,6 +73,36 @@ export const countTrackTimer = async () => {
} }
]; ];
} }
if (event === TrackEnum.teamChatQPM) {
const { teamId } = data;
return [
{
updateOne: {
filter: {
event,
teamId,
createTime: currentMinuteBoundary
},
update: [
{
$set: {
event,
teamId,
createTime: { $ifNull: ['$createTime', currentMinuteBoundary] },
data: {
requestCount: { $add: [{ $ifNull: ['$data.requestCount', 0] }, count] }
}
}
}
],
upsert: true
}
}
];
}
return []; return [];
}) })
.flat(); .flat();
......
packages/service/common/middle/tracks/utils.ts
...@@ -146,5 +146,15 @@ export const pushTrack = { ...@@ -146,5 +146,15 @@ export const pushTrack = {
} }
}); });
}); });
},
teamChatQPM: (data: { teamId: string }) => {
if (!data.teamId) return;
pushCountTrack({
event: TrackEnum.teamChatQPM,
key: `${TrackEnum.teamChatQPM}_${data.teamId}`,
data: {
teamId: data.teamId
}
});
} }
}; };
packages/service/common/mongo/init.ts
...@@ -64,8 +64,7 @@ export async function connectMongo(props: { ...@@ -64,8 +64,7 @@ export async function connectMongo(props: {
maxIdleTimeMS: 300000, // 空闲连接超时: 5分钟,防止空闲连接长时间占用资源 maxIdleTimeMS: 300000, // 空闲连接超时: 5分钟,防止空闲连接长时间占用资源
retryWrites: true, // 重试写入: 重试写入失败的操作 retryWrites: true, // 重试写入: 重试写入失败的操作
retryReads: true, // 重试读取: 重试读取失败的操作 retryReads: true, // 重试读取: 重试读取失败的操作
serverSelectionTimeoutMS: 10000, // 服务器选择超时: 10秒,防止副本集故障时长时间阻塞 serverSelectionTimeoutMS: 10000 // 服务器选择超时: 10秒,防止副本集故障时长时间阻塞
w: 'majority' // 写入确认策略: 多数节点确认后返回,保证数据安全性
}); });
console.log('mongo connected'); console.log('mongo connected');
......
packages/service/common/s3/buckets/base.ts
...@@ -160,6 +160,18 @@ export class S3BaseBucket { ...@@ -160,6 +160,18 @@ export class S3BaseBucket {
return this.client.statObject(this.name, ...params); return this.client.statObject(this.name, ...params);
} }
async isObjectExists(key: string): Promise<boolean> {
try {
await this.client.statObject(this.name, key);
return true;
} catch (err) {
if (err instanceof S3Error && err.message === 'Not Found') {
return false;
}
return Promise.reject(err);
}
}
async fileStreamToBuffer(stream: Readable): Promise<Buffer> { async fileStreamToBuffer(stream: Readable): Promise<Buffer> {
const chunks: Buffer[] = []; const chunks: Buffer[] = [];
for await (const chunk of stream) { for await (const chunk of stream) {
...@@ -183,10 +195,7 @@ export class S3BaseBucket { ...@@ -183,10 +195,7 @@ export class S3BaseBucket {
const ext = path.extname(filename).toLowerCase(); const ext = path.extname(filename).toLowerCase();
const contentType = Mimes[ext as keyof typeof Mimes] ?? 'application/octet-stream'; const contentType = Mimes[ext as keyof typeof Mimes] ?? 'application/octet-stream';
const key = (() => { const key = params.rawKey;
if ('rawKey' in params) return params.rawKey;
return [params.source, params.teamId, `${getNanoid(6)}-${filename}`].join('/');
})();
const policy = this.externalClient.newPostPolicy(); const policy = this.externalClient.newPostPolicy();
policy.setKey(key); policy.setKey(key);
...@@ -230,9 +239,7 @@ export class S3BaseBucket { ...@@ -230,9 +239,7 @@ export class S3BaseBucket {
const { key, expiredHours } = parsed; const { key, expiredHours } = parsed;
const expires = expiredHours ? expiredHours * 60 * 60 : 30 * 60; // expires 的单位是秒 默认 30 分钟 const expires = expiredHours ? expiredHours * 60 * 60 : 30 * 60; // expires 的单位是秒 默认 30 分钟
return await this.externalClient.presignedGetObject(this.name, key, expires, { return await this.externalClient.presignedGetObject(this.name, key, expires);
'Content-Disposition': `attachment; filename="${path.basename(key)}"`
});
} }
async createPreviewUrl(params: createPreviewUrlParams) { async createPreviewUrl(params: createPreviewUrlParams) {
...@@ -241,8 +248,6 @@ export class S3BaseBucket { ...@@ -241,8 +248,6 @@ export class S3BaseBucket {
const { key, expiredHours } = parsed; const { key, expiredHours } = parsed;
const expires = expiredHours ? expiredHours * 60 * 60 : 30 * 60; // expires 的单位是秒 默认 30 分钟 const expires = expiredHours ? expiredHours * 60 * 60 : 30 * 60; // expires 的单位是秒 默认 30 分钟
return await this.client.presignedGetObject(this.name, key, expires, { return await this.client.presignedGetObject(this.name, key, expires);
'Content-Disposition': `attachment; filename="${path.basename(key)}"`
});
} }
} }
packages/service/common/s3/constants.ts
...@@ -35,6 +35,7 @@ export const defaultS3Options: { ...@@ -35,6 +35,7 @@ export const defaultS3Options: {
accessKey: process.env.S3_ACCESS_KEY || 'minioadmin', accessKey: process.env.S3_ACCESS_KEY || 'minioadmin',
secretKey: process.env.S3_SECRET_KEY || 'minioadmin', secretKey: process.env.S3_SECRET_KEY || 'minioadmin',
port: process.env.S3_PORT ? parseInt(process.env.S3_PORT) : 9000, port: process.env.S3_PORT ? parseInt(process.env.S3_PORT) : 9000,
pathStyle: process.env.S3_PATH_STYLE === 'false' ? false : true,
transportAgent: process.env.HTTP_PROXY transportAgent: process.env.HTTP_PROXY
? new HttpProxyAgent(process.env.HTTP_PROXY) ? new HttpProxyAgent(process.env.HTTP_PROXY)
: process.env.HTTPS_PROXY : process.env.HTTPS_PROXY
......
packages/service/common/s3/sources/avatar.ts
...@@ -3,6 +3,7 @@ import { MongoS3TTL } from '../schema'; ...@@ -3,6 +3,7 @@ import { MongoS3TTL } from '../schema';
import { S3PublicBucket } from '../buckets/public'; import { S3PublicBucket } from '../buckets/public';
import { imageBaseUrl } from '@fastgpt/global/common/file/image/constants'; import { imageBaseUrl } from '@fastgpt/global/common/file/image/constants';
import type { ClientSession } from 'mongoose'; import type { ClientSession } from 'mongoose';
import { getFileS3Key } from '../utils';
class S3AvatarSource { class S3AvatarSource {
private bucket: S3PublicBucket; private bucket: S3PublicBucket;
...@@ -29,8 +30,10 @@ class S3AvatarSource { ...@@ -29,8 +30,10 @@ class S3AvatarSource {
teamId: string; teamId: string;
autoExpired?: boolean; autoExpired?: boolean;
}) { }) {
const { fileKey } = getFileS3Key.avatar({ teamId, filename });
return this.bucket.createPostPresignedUrl( return this.bucket.createPostPresignedUrl(
{ filename, teamId, source: S3Sources.avatar }, { filename, rawKey: fileKey },
{ {
expiredHours: autoExpired ? 1 : undefined, // 1 Hours expiredHours: autoExpired ? 1 : undefined, // 1 Hours
maxFileSize: 5 // 5MB maxFileSize: 5 // 5MB
......
packages/service/common/s3/sources/dataset/index.ts
...@@ -2,6 +2,8 @@ import { S3Sources } from '../../type'; ...@@ -2,6 +2,8 @@ import { S3Sources } from '../../type';
import { S3PrivateBucket } from '../../buckets/private'; import { S3PrivateBucket } from '../../buckets/private';
import { parseFileExtensionFromUrl } from '@fastgpt/global/common/string/tools'; import { parseFileExtensionFromUrl } from '@fastgpt/global/common/string/tools';
import { import {
type AddRawTextBufferParams,
AddRawTextBufferParamsSchema,
type CreateGetDatasetFileURLParams, type CreateGetDatasetFileURLParams,
CreateGetDatasetFileURLParamsSchema, CreateGetDatasetFileURLParamsSchema,
type CreateUploadDatasetFileParams, type CreateUploadDatasetFileParams,
...@@ -10,18 +12,20 @@ import { ...@@ -10,18 +12,20 @@ import {
DeleteDatasetFilesByPrefixParamsSchema, DeleteDatasetFilesByPrefixParamsSchema,
type GetDatasetFileContentParams, type GetDatasetFileContentParams,
GetDatasetFileContentParamsSchema, GetDatasetFileContentParamsSchema,
type UploadDatasetFileByBufferParams, type GetRawTextBufferParams,
UploadDatasetFileByBufferParamsSchema type UploadParams,
UploadParamsSchema
} from './type'; } from './type';
import { MongoS3TTL } from '../../schema'; import { MongoS3TTL } from '../../schema';
import { addHours, addMinutes } from 'date-fns'; import { addHours, addMinutes } from 'date-fns';
import { addLog } from '../../../system/log'; import { addLog } from '../../../system/log';
import { detectFileEncoding } from '@fastgpt/global/common/file/tools'; import { detectFileEncoding } from '@fastgpt/global/common/file/tools';
import { readS3FileContentByBuffer } from '../../../file/read/utils'; import { readS3FileContentByBuffer } from '../../../file/read/utils';
import { addRawTextBuffer, getRawTextBuffer } from '../../../buffer/rawText/controller';
import path from 'node:path'; import path from 'node:path';
import { Mimes } from '../../constants'; import { Mimes } from '../../constants';
import { getFileS3Key, truncateFilename } from '../../utils'; import { getFileS3Key, truncateFilename } from '../../utils';
import { createHash } from 'node:crypto';
import { S3Error } from 'minio';
export class S3DatasetSource { export class S3DatasetSource {
public bucket: S3PrivateBucket; public bucket: S3PrivateBucket;
...@@ -61,8 +65,8 @@ export class S3DatasetSource { ...@@ -61,8 +65,8 @@ export class S3DatasetSource {
* 比如根据被解析的文档前缀去删除解析出来的图片 * 比如根据被解析的文档前缀去删除解析出来的图片
**/ **/
deleteDatasetFilesByPrefix(params: DeleteDatasetFilesByPrefixParams) { deleteDatasetFilesByPrefix(params: DeleteDatasetFilesByPrefixParams) {
const { datasetId, rawPrefix } = DeleteDatasetFilesByPrefixParamsSchema.parse(params); const { datasetId } = DeleteDatasetFilesByPrefixParamsSchema.parse(params);
const prefix = rawPrefix || [S3Sources.dataset, datasetId].filter(Boolean).join('/'); const prefix = [S3Sources.dataset, datasetId].filter(Boolean).join('/');
return this.bucket.addDeleteJob({ prefix }); return this.bucket.addDeleteJob({ prefix });
} }
...@@ -83,7 +87,14 @@ export class S3DatasetSource { ...@@ -83,7 +87,14 @@ export class S3DatasetSource {
// 获取文件状态 // 获取文件状态
getDatasetFileStat(key: string) { getDatasetFileStat(key: string) {
return this.bucket.statObject(key); try {
return this.bucket.statObject(key);
} catch (error) {
if (error instanceof S3Error && error.message === 'Not Found') {
return null;
}
return Promise.reject(error);
}
} }
// 获取文件元数据 // 获取文件元数据
...@@ -117,12 +128,11 @@ export class S3DatasetSource { ...@@ -117,12 +128,11 @@ export class S3DatasetSource {
const { fileId, teamId, tmbId, customPdfParse, getFormatText, usageId } = const { fileId, teamId, tmbId, customPdfParse, getFormatText, usageId } =
GetDatasetFileContentParamsSchema.parse(params); GetDatasetFileContentParamsSchema.parse(params);
const bufferId = `${fileId}-${customPdfParse}`; const rawTextBuffer = await this.getRawTextBuffer({ customPdfParse, sourceId: fileId });
const fileBuffer = await getRawTextBuffer(bufferId); if (rawTextBuffer) {
if (fileBuffer) {
return { return {
rawText: fileBuffer.text, rawText: rawTextBuffer.text,
filename: fileBuffer.sourceName filename: rawTextBuffer.filename
}; };
} }
...@@ -154,11 +164,11 @@ export class S3DatasetSource { ...@@ -154,11 +164,11 @@ export class S3DatasetSource {
} }
}); });
addRawTextBuffer({ this.addRawTextBuffer({
sourceId: bufferId, sourceId: fileId,
sourceName: filename, sourceName: filename,
text: rawText, text: rawText,
expiredTime: addMinutes(new Date(), 20) customPdfParse
}); });
return { return {
...@@ -168,25 +178,85 @@ export class S3DatasetSource { ...@@ -168,25 +178,85 @@ export class S3DatasetSource {
} }
// 根据文件 Buffer 上传文件 // 根据文件 Buffer 上传文件
async uploadDatasetFileByBuffer(params: UploadDatasetFileByBufferParams): Promise<string> { async upload(params: UploadParams): Promise<string> {
const { datasetId, buffer, filename } = UploadDatasetFileByBufferParamsSchema.parse(params); const { datasetId, filename, ...file } = UploadParamsSchema.parse(params);
// 截断文件名以避免S3 key过长的问题 // 截断文件名以避免 S3 key 过长的问题
const truncatedFilename = truncateFilename(filename); const truncatedFilename = truncateFilename(filename);
const { fileKey: key } = getFileS3Key.dataset({ datasetId, filename: truncatedFilename }); const { fileKey: key } = getFileS3Key.dataset({ datasetId, filename: truncatedFilename });
await this.bucket.putObject(key, buffer, buffer.length, {
const { stream, size } = (() => {
if ('buffer' in file) {
return {
stream: file.buffer,
size: file.buffer.length
};
}
return {
stream: file.stream,
size: file.size
};
})();
await MongoS3TTL.create({
minioKey: key,
bucketName: this.bucket.name,
expiredTime: addHours(new Date(), 3)
});
await this.bucket.putObject(key, stream, size, {
'content-type': Mimes[path.extname(truncatedFilename) as keyof typeof Mimes], 'content-type': Mimes[path.extname(truncatedFilename) as keyof typeof Mimes],
'upload-time': new Date().toISOString(), 'upload-time': new Date().toISOString(),
'origin-filename': encodeURIComponent(truncatedFilename) 'origin-filename': encodeURIComponent(truncatedFilename)
}); });
return key;
}
async addRawTextBuffer(params: AddRawTextBufferParams) {
const { sourceId, sourceName, text, customPdfParse } =
AddRawTextBufferParamsSchema.parse(params);
// 因为 Key 唯一对应一个 Object 所以不需要根据文件内容计算 Hash 直接用 Key 计算 Hash 就行了
const hash = createHash('md5').update(sourceId).digest('hex');
const key = getFileS3Key.rawText({ hash, customPdfParse });
await MongoS3TTL.create({ await MongoS3TTL.create({
minioKey: key, minioKey: key,
bucketName: this.bucket.name, bucketName: this.bucket.name,
expiredTime: addHours(new Date(), 3) expiredTime: addMinutes(new Date(), 20)
});
const buffer = Buffer.from(text);
await this.bucket.putObject(key, buffer, buffer.length, {
'content-type': 'text/plain',
'origin-filename': encodeURIComponent(sourceName),
'upload-time': new Date().toISOString()
}); });
return key; return key;
} }
async getRawTextBuffer(params: GetRawTextBufferParams) {
const { customPdfParse, sourceId } = params;
const hash = createHash('md5').update(sourceId).digest('hex');
const key = getFileS3Key.rawText({ hash, customPdfParse });
if (!(await this.bucket.isObjectExists(key))) return null;
const [stream, metadata] = await Promise.all([
this.bucket.getObject(key),
this.getFileMetadata(key)
]);
const buffer = await this.bucket.fileStreamToBuffer(stream);
return {
text: buffer.toString('utf-8'),
filename: metadata.filename
};
}
} }
export function getS3DatasetSource() { export function getS3DatasetSource() {
......
packages/service/common/s3/sources/dataset/type.ts
import { ObjectIdSchema } from '@fastgpt/global/common/type/mongo'; import { ObjectIdSchema } from '@fastgpt/global/common/type/mongo';
import { ReadStream } from 'fs';
import { z } from 'zod'; import { z } from 'zod';
export const CreateUploadDatasetFileParamsSchema = z.object({ export const CreateUploadDatasetFileParamsSchema = z.object({
...@@ -15,8 +16,7 @@ export const CreateGetDatasetFileURLParamsSchema = z.object({ ...@@ -15,8 +16,7 @@ export const CreateGetDatasetFileURLParamsSchema = z.object({
export type CreateGetDatasetFileURLParams = z.infer<typeof CreateGetDatasetFileURLParamsSchema>; export type CreateGetDatasetFileURLParams = z.infer<typeof CreateGetDatasetFileURLParamsSchema>;
export const DeleteDatasetFilesByPrefixParamsSchema = z.object({ export const DeleteDatasetFilesByPrefixParamsSchema = z.object({
datasetId: ObjectIdSchema.optional(), datasetId: ObjectIdSchema.optional()
rawPrefix: z.string().nonempty().optional()
}); });
export type DeleteDatasetFilesByPrefixParams = z.infer< export type DeleteDatasetFilesByPrefixParams = z.infer<
typeof DeleteDatasetFilesByPrefixParamsSchema typeof DeleteDatasetFilesByPrefixParamsSchema
...@@ -44,9 +44,27 @@ export const ParsedFileContentS3KeyParamsSchema = z.object({ ...@@ -44,9 +44,27 @@ export const ParsedFileContentS3KeyParamsSchema = z.object({
}); });
export type ParsedFileContentS3KeyParams = z.infer<typeof ParsedFileContentS3KeyParamsSchema>; export type ParsedFileContentS3KeyParams = z.infer<typeof ParsedFileContentS3KeyParamsSchema>;
export const UploadDatasetFileByBufferParamsSchema = z.object({ export const UploadParamsSchema = z.union([
datasetId: ObjectIdSchema, z.object({
buffer: z.instanceof(Buffer), datasetId: ObjectIdSchema,
filename: z.string().nonempty() filename: z.string().nonempty(),
buffer: z.instanceof(Buffer)
}),
z.object({
datasetId: ObjectIdSchema,
filename: z.string().nonempty(),
stream: z.instanceof(ReadStream),
size: z.int().positive().optional()
})
]);
export type UploadParams = z.input<typeof UploadParamsSchema>;
export const AddRawTextBufferParamsSchema = z.object({
customPdfParse: z.boolean().optional(),
sourceId: z.string().nonempty(),
sourceName: z.string().nonempty(),
text: z.string()
}); });
export type UploadDatasetFileByBufferParams = z.infer<typeof UploadDatasetFileByBufferParamsSchema>; export type AddRawTextBufferParams = z.input<typeof AddRawTextBufferParamsSchema>;
export type GetRawTextBufferParams = Pick<AddRawTextBufferParams, 'customPdfParse' | 'sourceId'>;
packages/service/common/s3/type.ts
...@@ -17,25 +17,15 @@ export type ExtensionType = keyof typeof Mimes; ...@@ -17,25 +17,15 @@ export type ExtensionType = keyof typeof Mimes;
export type S3OptionsType = typeof defaultS3Options; export type S3OptionsType = typeof defaultS3Options;
export const S3SourcesSchema = z.enum(['avatar', 'chat', 'dataset', 'temp']); export const S3SourcesSchema = z.enum(['avatar', 'chat', 'dataset', 'temp', 'rawText']);
export const S3Sources = S3SourcesSchema.enum; export const S3Sources = S3SourcesSchema.enum;
export type S3SourceType = z.infer<typeof S3SourcesSchema>; export type S3SourceType = z.infer<typeof S3SourcesSchema>;
export const CreatePostPresignedUrlParamsSchema = z.union([ export const CreatePostPresignedUrlParamsSchema = z.object({
// Option 1: Only rawKey filename: z.string().min(1),
z.object({ rawKey: z.string().min(1),
filename: z.string().min(1), metadata: z.record(z.string(), z.string()).optional()
rawKey: z.string().min(1), });
metadata: z.record(z.string(), z.string()).optional()
}),
// Option 2: filename with optional source and teamId
z.object({
filename: z.string().min(1),
source: S3SourcesSchema.optional(),
teamId: z.string().length(16).optional(),
metadata: z.record(z.string(), z.string()).optional()
})
]);
export type CreatePostPresignedUrlParams = z.infer<typeof CreatePostPresignedUrlParamsSchema>; export type CreatePostPresignedUrlParams = z.infer<typeof CreatePostPresignedUrlParamsSchema>;
export const CreatePostPresignedUrlOptionsSchema = z.object({ export const CreatePostPresignedUrlOptionsSchema = z.object({
......
packages/service/common/s3/utils.ts
...@@ -11,6 +11,7 @@ import { getNanoid } from '@fastgpt/global/common/string/tools'; ...@@ -11,6 +11,7 @@ import { getNanoid } from '@fastgpt/global/common/string/tools';
import path from 'node:path'; import path from 'node:path';
import type { ParsedFileContentS3KeyParams } from './sources/dataset/type'; import type { ParsedFileContentS3KeyParams } from './sources/dataset/type';
import { EndpointUrl } from '@fastgpt/global/common/file/constants'; import { EndpointUrl } from '@fastgpt/global/common/file/constants';
import type { NextApiRequest } from 'next';
// S3文件名最大长度配置 // S3文件名最大长度配置
export const S3_FILENAME_MAX_LENGTH = 50; export const S3_FILENAME_MAX_LENGTH = 50;
...@@ -173,6 +174,17 @@ export const getFileS3Key = { ...@@ -173,6 +174,17 @@ export const getFileS3Key = {
}; };
}, },
avatar: ({ teamId, filename }: { teamId: string; filename?: string }) => {
const { formatedFilename, extension } = getFormatedFilename(filename);
return {
fileKey: [
S3Sources.avatar,
teamId,
`${formatedFilename}${extension ? `.${extension}` : ''}`
].join('/')
};
},
// 对话中上传的文件的解析结果的图片的 Key // 对话中上传的文件的解析结果的图片的 Key
chat: ({ chat: ({
appId, appId,
...@@ -215,6 +227,10 @@ export const getFileS3Key = { ...@@ -215,6 +227,10 @@ export const getFileS3Key = {
fileKey: key, fileKey: key,
fileParsedPrefix: prefix fileParsedPrefix: prefix
}; };
},
rawText: ({ hash, customPdfParse }: { hash: string; customPdfParse?: boolean }) => {
return [S3Sources.rawText, `${hash}${customPdfParse ? '-true' : ''}`].join('/');
} }
}; };
...@@ -230,3 +246,130 @@ export function isS3ObjectKey<T extends keyof typeof S3Sources>( ...@@ -230,3 +246,130 @@ export function isS3ObjectKey<T extends keyof typeof S3Sources>(
): key is `${T}/${string}` { ): key is `${T}/${string}` {
return typeof key === 'string' && key.startsWith(`${S3Sources[source]}/`); return typeof key === 'string' && key.startsWith(`${S3Sources[source]}/`);
} }
// export const multer = {
// _storage: multer.diskStorage({
// filename: (_, file, cb) => {
// if (!file?.originalname) {
// cb(new Error('File not found'), '');
// } else {
// const ext = path.extname(decodeURIComponent(file.originalname));
// cb(null, `${getNanoid()}${ext}`);
// }
// }
// }),
// singleStore(maxFileSize: number = 500) {
// const fileSize = maxFileSize * 1024 * 1024;
// return multer({
// limits: {
// fileSize
// },
// preservePath: true,
// storage: this._storage
// }).single('file');
// },
// multipleStore(maxFileSize: number = 500) {
// const fileSize = maxFileSize * 1024 * 1024;
// return multer({
// limits: {
// fileSize
// },
// preservePath: true,
// storage: this._storage
// }).array('file', global.feConfigs?.uploadFileMaxSize);
// },
// resolveFormData({ request, maxFileSize }: { request: NextApiRequest; maxFileSize?: number }) {
// return new Promise<{
// data: Record<string, any>;
// fileMetadata: Express.Multer.File;
// getBuffer: () => Buffer;
// getReadStream: () => fs.ReadStream;
// }>((resolve, reject) => {
// const handler = this.singleStore(maxFileSize);
// // @ts-expect-error it can accept a NextApiRequest
// handler(request, null, (error) => {
// if (error) {
// return reject(error);
// }
// // @ts-expect-error `file` will be injected by multer
// const file = request.file as Express.Multer.File;
// if (!file) {
// return reject(new Error('File not found'));
// }
// const data = (() => {
// if (!request.body?.data) return {};
// try {
// return JSON.parse(request.body.data);
// } catch {
// return {};
// }
// })();
// resolve({
// data,
// fileMetadata: file,
// getBuffer: () => fs.readFileSync(file.path),
// getReadStream: () => fs.createReadStream(file.path)
// });
// });
// });
// },
// resolveMultipleFormData({
// request,
// maxFileSize
// }: {
// request: NextApiRequest;
// maxFileSize?: number;
// }) {
// return new Promise<{
// data: Record<string, any>;
// fileMetadata: Array<Express.Multer.File>;
// }>((resolve, reject) => {
// const handler = this.multipleStore(maxFileSize);
// // @ts-expect-error it can accept a NextApiRequest
// handler(request, null, (error) => {
// if (error) {
// return reject(error);
// }
// // @ts-expect-error `files` will be injected by multer
// const files = request.files as Array<Express.Multer.File>;
// if (!files || files.length === 0) {
// return reject(new Error('File not found'));
// }
// const data = (() => {
// if (!request.body?.data) return {};
// try {
// return JSON.parse(request.body.data);
// } catch {
// return {};
// }
// })();
// resolve({
// data,
// fileMetadata: files
// });
// });
// });
// },
// clearDiskTempFiles(filepaths: string[]) {
// for (const filepath of filepaths) {
// fs.unlink(filepath, (_) => {});
// }
// }
// };
packages/service/common/system/countLimit/const.ts 0 → 100644
import z from 'zod';
import { CountLimitTypeEnum } from './type';
export const CountLimitConfigType = z.record(
CountLimitTypeEnum,
z.object({ maxCount: z.number() })
);
// 只会发送 n 次通知，如需自动发送，需要主动清除记录
export const CountLimitConfig = {
[CountLimitTypeEnum.enum['notice:30PercentPoints']]: {
maxCount: 3
},
[CountLimitTypeEnum.enum['notice:10PercentPoints']]: {
maxCount: 5
},
[CountLimitTypeEnum.enum['notice:LackOfPoints']]: {
maxCount: 5
},
[CountLimitTypeEnum.enum['notice:30PercentDatasetIndexes']]: {
maxCount: 3
},
[CountLimitTypeEnum.enum['notice:10PercentDatasetIndexes']]: {
maxCount: 5
},
[CountLimitTypeEnum.enum['notice:NoDatasetIndexes']]: {
maxCount: 5
}
} satisfies z.infer<typeof CountLimitConfigType>;
packages/service/common/system/countLimit/controller.ts 0 → 100644
import type z from 'zod';
import type { CountLimitTypeEnum } from './type';
import { CountLimitConfig } from './const';
import { MongoCountLimit } from './schema';
import { mongoSessionRun } from '../../mongo/sessionRun';
/**
* Update the count limit for a specific type and key.
* @param param0 - The type, key, and update value.
* @returns The updated count limit information.
*/
export const updateCountLimit = async ({
type,
key,
update
}: {
type: z.infer<typeof CountLimitTypeEnum>;
key: string;
update: number;
}) =>
mongoSessionRun(async (session) => {
const maxCount = CountLimitConfig[type].maxCount;
const countLimit = await MongoCountLimit.findOne(
{
type,
key
},
undefined,
{
session
}
).lean();
if (!countLimit) {
// do not exist, create a new one
await MongoCountLimit.create(
[
{
type,
key,
count: update // 0 + update
}
],
{
session
}
);
return {
maxCount,
nowCount: update,
remain: maxCount - update
};
}
if (countLimit.count >= maxCount) {
return Promise.reject(`Max Count Reached, type: ${type}, key: ${key}`);
}
await MongoCountLimit.updateOne(
{
type,
key
},
{
$inc: { count: update }
},
{ session }
);
return {
maxCount,
nowCount: countLimit.count + update,
remain: maxCount - (countLimit.count + update)
};
});
/** Clean the Count limit, if no key provided, clean all the type */
export const cleanCountLimit = async ({ teamId }: { teamId: string }) =>
MongoCountLimit.deleteMany({
key: teamId
});
packages/service/common/system/countLimit/schema.ts 0 → 100644
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import type { CountLimitType } from './type';
const { Schema } = connectionMongo;
const collectionName = 'system_count_limits';
const CountLimitSchema = new Schema({
key: {
type: String,
required: true
},
type: {
type: String,
required: true
},
count: {
type: Number,
required: true,
default: 0
},
createTime: {
type: Date,
default: () => new Date()
}
});
try {
CountLimitSchema.index({ type: 1, key: 1 }, { unique: true });
CountLimitSchema.index({ createTime: 1 }, { expireAfterSeconds: 60 * 60 * 24 * 30 }); // ttl 30天
} catch (error) {
console.log(error);
}
export const MongoCountLimit = getMongoModel<CountLimitType>(collectionName, CountLimitSchema);
packages/service/common/system/countLimit/type.ts 0 → 100644
import z from 'zod';
export const CountLimitTypeEnum = z.enum([
'notice:30PercentPoints',
'notice:10PercentPoints',
'notice:LackOfPoints',
'notice:30PercentDatasetIndexes',
'notice:10PercentDatasetIndexes',
'notice:NoDatasetIndexes'
]);
export const CountLimitType = z.object({
type: CountLimitTypeEnum,
key: z.string(),
count: z.number()
});
export type CountLimitType = z.infer<typeof CountLimitType>;
export type CountLimitTypeEnum = z.infer<typeof CountLimitTypeEnum>;
packages/service/common/system/timerLock/constants.ts
...@@ -9,7 +9,8 @@ export enum TimerIdEnum { ...@@ -9,7 +9,8 @@ export enum TimerIdEnum {
clearExpiredRawTextBuffer = 'clearExpiredRawTextBuffer', clearExpiredRawTextBuffer = 'clearExpiredRawTextBuffer',
clearExpiredDatasetImage = 'clearExpiredDatasetImage', clearExpiredDatasetImage = 'clearExpiredDatasetImage',
clearExpiredMinioFiles = 'clearExpiredMinioFiles' clearExpiredMinioFiles = 'clearExpiredMinioFiles',
recordTeamQPM = 'recordTeamQPM'
} }
export enum LockNotificationEnum { export enum LockNotificationEnum {
......
packages/service/common/system/timerLock/schema.ts
...@@ -2,7 +2,7 @@ import { connectionMongo, getMongoModel } from '../../mongo'; ...@@ -2,7 +2,7 @@ import { connectionMongo, getMongoModel } from '../../mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type TimerLockSchemaType } from './type.d'; import { type TimerLockSchemaType } from './type.d';
export const collectionName = 'systemtimerlocks'; export const collectionName = 'system_timer_locks';
const TimerLockSchema = new Schema({ const TimerLockSchema = new Schema({
timerId: { timerId: {
......
packages/service/common/system/timerLock/utils.ts
...@@ -2,7 +2,7 @@ import { type ClientSession } from '../../mongo'; ...@@ -2,7 +2,7 @@ import { type ClientSession } from '../../mongo';
import { MongoTimerLock } from './schema'; import { MongoTimerLock } from './schema';
import { addMinutes } from 'date-fns'; import { addMinutes } from 'date-fns';
/* /*
利用唯一健，使得同一时间只有一个任务在执行，后创建的锁，会因唯一健创建失败，从而无法继续执行任务 利用唯一健，使得同一时间只有一个任务在执行，后创建的锁，会因唯一健创建失败，从而无法继续执行任务
*/ */
export const checkTimerLock = async ({ export const checkTimerLock = async ({
...@@ -30,3 +30,19 @@ export const checkTimerLock = async ({ ...@@ -30,3 +30,19 @@ export const checkTimerLock = async ({
return false; return false;
} }
}; };
export const cleanTimerLock = async ({
teamId,
session
}: {
teamId: string;
session?: ClientSession;
}) => {
// Match timerId pattern where lockId (last segment) equals teamId
await MongoTimerLock.deleteMany(
{
timerId: new RegExp(`--${teamId}$`)
},
{ session }
);
};
packages/service/core/ai/functions/queryExtension.ts
...@@ -6,16 +6,23 @@ import { addLog } from '../../../common/system/log'; ...@@ -6,16 +6,23 @@ import { addLog } from '../../../common/system/log';
import { filterGPTMessageByMaxContext } from '../llm/utils'; import { filterGPTMessageByMaxContext } from '../llm/utils';
import json5 from 'json5'; import json5 from 'json5';
import { createLLMResponse } from '../llm/request'; import { createLLMResponse } from '../llm/request';
import { useTextCosine } from '../hooks/useTextCosine';
/* /*
query extension - 问题扩展 Query Extension - Semantic Search Enhancement
可以根据上下文，消除指代性问题以及扩展问题，利于检索。 This module can eliminate referential ambiguity and expand queries based on context to improve retrieval.
Submodular Optimization Mode: Generate multiple candidate queries, then use submodular algorithm to select the optimal query combination
*/ */
const title = global.feConfigs?.systemTitle || 'FastAI'; const title = global.feConfigs?.systemTitle || 'FastAI';
const defaultPrompt = `## 你的任务 const defaultPrompt = `## 你的任务
你作为一个向量检索助手，你的任务是结合历史记录，从不同角度，为“原问题”生成个不同版本的“检索词”，从而提高向量检索的语义丰富度，提高向量检索的精度。 你作为一个向量检索助手，你的任务是结合历史记录，为"原问题"生成{{count}}个不同版本的"检索词"。这些检索词应该从不同角度探索主题，以提高向量检索的语义丰富度和精度。
生成的问题要求指向对象清晰明确，并与“原问题语言相同”。
## 要求
1. 每个检索词必须与原问题相关
2. 检索词应该探索不同方面（例如：原因、影响、解决方案、示例、对比等）
3. 避免检索词之间的冗余
4. 保持检索词简洁且可搜索
5. 生成的问题要求指向对象清晰明确，并与"原问题语言相同"
## 参考示例 ## 参考示例
...@@ -24,7 +31,7 @@ const defaultPrompt = `## 你的任务 ...@@ -24,7 +31,7 @@ const defaultPrompt = `## 你的任务
null null
""" """
原问题: 介绍下剧情。 原问题: 介绍下剧情。
检索词: ["介绍下故事的背景。","故事的主题是什么？","介绍下故事的主要人物。"] 检索词: ["介绍下故事的背景。","故事的主题是什么？","介绍下故事的主要人物。","故事的转折点在哪里？","故事的结局如何？"]
---------------- ----------------
历史记录: 历史记录:
""" """
...@@ -32,7 +39,7 @@ user: 对话背景。 ...@@ -32,7 +39,7 @@ user: 对话背景。
assistant: 当前对话是关于 Nginx 的介绍和使用等。 assistant: 当前对话是关于 Nginx 的介绍和使用等。
""" """
原问题: 怎么下载 原问题: 怎么下载
检索词: ["Nginx 如何下载？","下载 Nginx 需要什么条件？","有哪些渠道可以下载 Nginx？"] 检索词: ["Nginx 如何下载？","下载 Nginx 需要什么条件？","有哪些渠道可以下载 Nginx？","Nginx 各版本的下载方式有什么区别？","如何选择合适的 Nginx 版本下载？"]
---------------- ----------------
历史记录: 历史记录:
""" """
...@@ -42,7 +49,7 @@ user: 报错 "no connection" ...@@ -42,7 +49,7 @@ user: 报错 "no connection"
assistant: 报错"no connection"可能是因为…… assistant: 报错"no connection"可能是因为……
""" """
原问题: 怎么解决 原问题: 怎么解决
检索词: ["Nginx报错"no connection"如何解决？","造成'no connection'报错的原因。","Nginx提示'no connection'，要怎么办？"] 检索词: ["Nginx报错'no connection'如何解决？","造成'no connection'报错的原因。","Nginx提示'no connection'，要怎么办？","'no connection'错误的常见解决步骤。","如何预防 Nginx 'no connection' 错误？"]
---------------- ----------------
历史记录: 历史记录:
""" """
...@@ -50,7 +57,7 @@ user: How long is the maternity leave? ...@@ -50,7 +57,7 @@ user: How long is the maternity leave?
assistant: The number of days of maternity leave depends on the city in which the employee is located. Please provide your city so that I can answer your questions. assistant: The number of days of maternity leave depends on the city in which the employee is located. Please provide your city so that I can answer your questions.
""" """
原问题: ShenYang 原问题: ShenYang
检索词: ["How many days is maternity leave in Shenyang?","Shenyang's maternity leave policy.","The standard of maternity leave in Shenyang."] 检索词: ["How many days is maternity leave in Shenyang?","Shenyang's maternity leave policy.","The standard of maternity leave in Shenyang.","What benefits are included in Shenyang's maternity leave?","How to apply for maternity leave in Shenyang?"]
---------------- ----------------
历史记录: 历史记录:
""" """
...@@ -58,7 +65,7 @@ user: 作者是谁？ ...@@ -58,7 +65,7 @@ user: 作者是谁？
assistant: ${title} 的作者是 labring。 assistant: ${title} 的作者是 labring。
""" """
原问题: Tell me about him 原问题: Tell me about him
检索词: ["Introduce labring, the author of ${title}." ," Background information on author labring." "," Why does labring do ${title}?"] 检索词: ["Introduce labring, the author of ${title}." ,"Background information on author labring.","Why does labring do ${title}?","What other projects has labring worked on?","How did labring start ${title}?"]
---------------- ----------------
历史记录: 历史记录:
""" """
...@@ -74,7 +81,7 @@ user: ${title} 如何收费？ ...@@ -74,7 +81,7 @@ user: ${title} 如何收费？
assistant: ${title} 收费可以参考…… assistant: ${title} 收费可以参考……
""" """
原问题: 你知道 laf 么？ 原问题: 你知道 laf 么？
检索词: ["laf 的官网地址是多少？","laf 的使用教程。","laf 有什么特点和优势。"] 检索词: ["laf 的官网地址是多少？","laf 的使用教程。","laf 有什么特点和优势。","laf 的主要功能是什么？","laf 与其他类似产品的对比。"]
---------------- ----------------
历史记录: 历史记录:
""" """
...@@ -100,6 +107,7 @@ assistant: Laf 是一个云函数开发平台。 ...@@ -100,6 +107,7 @@ assistant: Laf 是一个云函数开发平台。
1. 输出格式为 JSON 数组，数组中每个元素为字符串。无需对输出进行任何解释。 1. 输出格式为 JSON 数组，数组中每个元素为字符串。无需对输出进行任何解释。
2. 输出语言与原问题相同。原问题为中文则输出中文；原问题为英文则输出英文。 2. 输出语言与原问题相同。原问题为中文则输出中文；原问题为英文则输出英文。
3. 确保生成恰好 {{count}} 个检索词。
## 开始任务 ## 开始任务
...@@ -114,18 +122,24 @@ export const queryExtension = async ({ ...@@ -114,18 +122,24 @@ export const queryExtension = async ({
chatBg, chatBg,
query, query,
histories = [], histories = [],
model llmModel,
embeddingModel,
generateCount = 10 // 生成优化问题集的数量，默认为10个
}: { }: {
chatBg?: string; chatBg?: string;
query: string; query: string;
histories: ChatItemType[]; histories: ChatItemType[];
model: string; llmModel: string;
embeddingModel: string;
generateCount?: number;
}): Promise<{ }): Promise<{
rawQuery: string; rawQuery: string;
extensionQueries: string[]; extensionQueries: string[];
model: string; llmModel: string;
embeddingModel: string;
inputTokens: number; inputTokens: number;
outputTokens: number; outputTokens: number;
embeddingTokens: number;
}> => { }> => {
const systemFewShot = chatBg const systemFewShot = chatBg
? `user: 对话背景。 ? `user: 对话背景。
...@@ -133,7 +147,8 @@ assistant: ${chatBg} ...@@ -133,7 +147,8 @@ assistant: ${chatBg}
` `
: ''; : '';
const modelData = getLLMModel(model); // 1. Request model
const modelData = getLLMModel(llmModel);
const filterHistories = await filterGPTMessageByMaxContext({ const filterHistories = await filterGPTMessageByMaxContext({
messages: chats2GPTMessages({ messages: histories, reserveId: false }), messages: chats2GPTMessages({ messages: histories, reserveId: false }),
maxContext: modelData.maxContext - 1000 maxContext: modelData.maxContext - 1000
...@@ -160,7 +175,8 @@ assistant: ${chatBg} ...@@ -160,7 +175,8 @@ assistant: ${chatBg}
role: 'user', role: 'user',
content: replaceVariable(defaultPrompt, { content: replaceVariable(defaultPrompt, {
query: `${query}`, query: `${query}`,
histories: concatFewShot || 'null' histories: concatFewShot || 'null',
count: generateCount.toString()
}) })
} }
] as any; ] as any;
...@@ -181,12 +197,15 @@ assistant: ${chatBg} ...@@ -181,12 +197,15 @@ assistant: ${chatBg}
return { return {
rawQuery: query, rawQuery: query,
extensionQueries: [], extensionQueries: [],
model, llmModel: modelData.model,
embeddingModel,
inputTokens: inputTokens, inputTokens: inputTokens,
outputTokens: outputTokens outputTokens: outputTokens,
embeddingTokens: 0
}; };
} }
// 2. Parse answer
const start = answer.indexOf('['); const start = answer.indexOf('[');
const end = answer.lastIndexOf(']'); const end = answer.lastIndexOf(']');
if (start === -1 || end === -1) { if (start === -1 || end === -1) {
...@@ -196,9 +215,11 @@ assistant: ${chatBg} ...@@ -196,9 +215,11 @@ assistant: ${chatBg}
return { return {
rawQuery: query, rawQuery: query,
extensionQueries: [], extensionQueries: [],
model, llmModel: modelData.model,
embeddingModel,
inputTokens: inputTokens, inputTokens: inputTokens,
outputTokens: outputTokens outputTokens: outputTokens,
embeddingTokens: 0
}; };
} }
...@@ -211,23 +232,51 @@ assistant: ${chatBg} ...@@ -211,23 +232,51 @@ assistant: ${chatBg}
try { try {
const queries = json5.parse(jsonStr) as string[]; const queries = json5.parse(jsonStr) as string[];
if (!Array.isArray(queries) || queries.length === 0) {
return {
rawQuery: query,
extensionQueries: [],
llmModel: modelData.model,
embeddingModel,
inputTokens,
outputTokens,
embeddingTokens: 0
};
}
// 3. 通过计算获取到最优的检索词
const { lazyGreedyQuerySelection, embeddingModel: useEmbeddingModel } = useTextCosine({
embeddingModel
});
const { selectedData: selectedQueries, embeddingTokens } = await lazyGreedyQuerySelection({
originalText: query,
candidates: queries,
k: Math.min(3, queries.length), // 至多 3 个
alpha: 0.3
});
return { return {
rawQuery: query, rawQuery: query,
extensionQueries: (Array.isArray(queries) ? queries : []).slice(0, 5), extensionQueries: selectedQueries,
model, llmModel: modelData.model,
embeddingModel: useEmbeddingModel,
inputTokens, inputTokens,
outputTokens outputTokens,
embeddingTokens
}; };
} catch (error) { } catch (error) {
addLog.warn('Query extension failed, not a valid JSON', { addLog.warn('Query extension failed', {
error,
answer answer
}); });
return { return {
rawQuery: query, rawQuery: query,
extensionQueries: [], extensionQueries: [],
model, llmModel: modelData.model,
embeddingModel,
inputTokens, inputTokens,
outputTokens outputTokens,
embeddingTokens: 0
}; };
} }
}; };
packages/service/core/ai/hooks/useTextCosine.ts 0 → 100644
/*
根据文本的余弦相似度，获取最大边际收益的检索词。
Reference: https://github.com/jina-ai/submodular-optimization
*/
import { getVectorsByText } from '../embedding';
import { getEmbeddingModel } from '../model';
class PriorityQueue<T> {
private heap: Array<{ item: T; priority: number }> = [];
enqueue(item: T, priority: number): void {
this.heap.push({ item, priority });
this.heap.sort((a, b) => b.priority - a.priority);
}
dequeue(): T | undefined {
return this.heap.shift()?.item;
}
isEmpty(): boolean {
return this.heap.length === 0;
}
size(): number {
return this.heap.length;
}
}
export const useTextCosine = ({ embeddingModel }: { embeddingModel: string }) => {
const vectorModel = getEmbeddingModel(embeddingModel);
// Calculate marginal gain
const computeMarginalGain = (
candidateEmbedding: number[],
selectedEmbeddings: number[][],
originalEmbedding: number[],
alpha: number = 0.3
): number => {
// Calculate cosine similarity
const cosineSimilarity = (a: number[], b: number[]): number => {
if (a.length !== b.length) {
throw new Error('Vectors must have the same length');
}
let dotProduct = 0;
let normA = 0;
let normB = 0;
for (let i = 0; i < a.length; i++) {
dotProduct += a[i] * b[i];
normA += a[i] * a[i];
normB += b[i] * b[i];
}
if (normA === 0 || normB === 0) return 0;
return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
};
if (selectedEmbeddings.length === 0) {
return alpha * cosineSimilarity(originalEmbedding, candidateEmbedding);
}
let maxSimilarity = 0;
for (const selectedEmbedding of selectedEmbeddings) {
const similarity = cosineSimilarity(candidateEmbedding, selectedEmbedding);
maxSimilarity = Math.max(maxSimilarity, similarity);
}
const relevance = alpha * cosineSimilarity(originalEmbedding, candidateEmbedding);
const diversity = 1 - maxSimilarity;
return relevance + diversity;
};
// Lazy greedy query selection algorithm
const lazyGreedyQuerySelection = async ({
originalText,
candidates,
k,
alpha = 0.3
}: {
originalText: string;
candidates: string[]; // 候选文本
k: number;
alpha?: number;
}) => {
const { tokens: embeddingTokens, vectors: embeddingVectors } = await getVectorsByText({
model: vectorModel,
input: [originalText, ...candidates],
type: 'query'
});
const originalEmbedding = embeddingVectors[0];
const candidateEmbeddings = embeddingVectors.slice(1);
const n = candidates.length;
const selected: string[] = [];
const selectedEmbeddings: number[][] = [];
// Initialize priority queue
const pq = new PriorityQueue<{ index: number; gain: number }>();
// Calculate initial marginal gain for all candidates
for (let i = 0; i < n; i++) {
const gain = computeMarginalGain(
candidateEmbeddings[i],
selectedEmbeddings,
originalEmbedding,
alpha
);
pq.enqueue({ index: i, gain }, gain);
}
// Greedy selection
for (let iteration = 0; iteration < k; iteration++) {
if (pq.isEmpty()) break;
let bestCandidate: { index: number; gain: number } | undefined;
// Find candidate with maximum marginal gain
while (!pq.isEmpty()) {
const candidate = pq.dequeue()!;
const currentGain = computeMarginalGain(
candidateEmbeddings[candidate.index],
selectedEmbeddings,
originalEmbedding,
alpha
);
if (currentGain >= candidate.gain) {
bestCandidate = { index: candidate.index, gain: currentGain };
break;
} else {
// Create new object with updated gain to avoid infinite loop
pq.enqueue({ index: candidate.index, gain: currentGain }, currentGain);
}
}
if (bestCandidate) {
selected.push(candidates[bestCandidate.index]);
selectedEmbeddings.push(candidateEmbeddings[bestCandidate.index]);
}
}
return {
selectedData: selected,
embeddingTokens
};
};
return {
lazyGreedyQuerySelection,
embeddingModel: vectorModel.model
};
};
packages/service/core/app/controller.ts
...@@ -12,7 +12,6 @@ import { SystemToolSecretInputTypeEnum } from '@fastgpt/global/core/app/tool/sys ...@@ -12,7 +12,6 @@ import { SystemToolSecretInputTypeEnum } from '@fastgpt/global/core/app/tool/sys
import { type ClientSession } from '../../common/mongo'; import { type ClientSession } from '../../common/mongo';
import { MongoEvaluation } from './evaluation/evalSchema'; import { MongoEvaluation } from './evaluation/evalSchema';
import { removeEvaluationJob } from './evaluation/mq'; import { removeEvaluationJob } from './evaluation/mq';
import { deleteChatFiles } from '../chat/controller';
import { MongoChatItem } from '../chat/chatItemSchema'; import { MongoChatItem } from '../chat/chatItemSchema';
import { MongoChat } from '../chat/chatSchema'; import { MongoChat } from '../chat/chatSchema';
import { MongoOutLink } from '../../support/outLink/schema'; import { MongoOutLink } from '../../support/outLink/schema';
...@@ -224,7 +223,7 @@ export const onDelOneApp = async ({ ...@@ -224,7 +223,7 @@ export const onDelOneApp = async ({
// Delete chats // Delete chats
for await (const app of apps) { for await (const app of apps) {
const appId = String(app._id); const appId = String(app._id);
await deleteChatFiles({ appId }); await getS3ChatSource().deleteChatFilesByPrefix({ appId });
await MongoChatItemResponse.deleteMany({ await MongoChatItemResponse.deleteMany({
appId appId
}); });
......
packages/service/core/chat/controller.ts
import type { ChatHistoryItemResType, ChatItemType } from '@fastgpt/global/core/chat/type'; import type { ChatHistoryItemResType, ChatItemType } from '@fastgpt/global/core/chat/type';
import { MongoChatItem } from './chatItemSchema'; import { MongoChatItem } from './chatItemSchema';
import { addLog } from '../../common/system/log'; import { addLog } from '../../common/system/log';
import { delFileByFileIdList, getGFSCollection } from '../../common/file/gridfs/controller';
import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { MongoChat } from './chatSchema';
import { UserError } from '@fastgpt/global/common/error/utils';
import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/workflow/runtime/constants'; import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/workflow/runtime/constants';
import { ChatRoleEnum } from '@fastgpt/global/core/chat/constants'; import { ChatRoleEnum } from '@fastgpt/global/core/chat/constants';
import { MongoChatItemResponse } from './chatItemResponseSchema'; import { MongoChatItemResponse } from './chatItemResponseSchema';
...@@ -95,41 +91,3 @@ export const addCustomFeedbacks = async ({ ...@@ -95,41 +91,3 @@ export const addCustomFeedbacks = async ({
addLog.error('addCustomFeedbacks error', error); addLog.error('addCustomFeedbacks error', error);
} }
}; };
/*
Delete chat files
1. ChatId: Delete one chat files
2. AppId: Delete all the app's chat files
*/
export const deleteChatFiles = async ({
chatIdList,
appId
}: {
chatIdList?: string[];
appId?: string;
}) => {
if (!appId && !chatIdList)
return Promise.reject(new UserError('appId or chatIdList is required'));
const appChatIdList = await (async () => {
if (appId) {
const appChatIdList = await MongoChat.find({ appId }, { chatId: 1 });
return appChatIdList.map((item) => String(item.chatId));
} else if (chatIdList) {
return chatIdList;
}
return [];
})();
const collection = getGFSCollection(BucketNameEnum.chat);
const where = {
'metadata.chatId': { $in: appChatIdList }
};
const files = await collection.find(where, { projection: { _id: 1 } }).toArray();
await delFileByFileIdList({
bucketName: BucketNameEnum.chat,
fileIdList: files.map((item) => String(item._id))
});
};
packages/service/core/chat/saveChat.ts
...@@ -245,6 +245,7 @@ export async function saveChat(props: Props) { ...@@ -245,6 +245,7 @@ export async function saveChat(props: Props) {
...chat?.metadata, ...chat?.metadata,
...metadata ...metadata
}; };
const { welcomeText, variables: variableList } = getAppChatConfig({ const { welcomeText, variables: variableList } = getAppChatConfig({
chatConfig: appChatConfig, chatConfig: appChatConfig,
systemConfigNode: getGuideModule(nodes), systemConfigNode: getGuideModule(nodes),
......
packages/service/core/dataset/apiDataset/custom/api.ts
...@@ -9,8 +9,7 @@ import { addLog } from '../../../../common/system/log'; ...@@ -9,8 +9,7 @@ import { addLog } from '../../../../common/system/log';
import { readFileRawTextByUrl } from '../../read'; import { readFileRawTextByUrl } from '../../read';
import { type ParentIdType } from '@fastgpt/global/common/parentFolder/type'; import { type ParentIdType } from '@fastgpt/global/common/parentFolder/type';
import { type RequireOnlyOne } from '@fastgpt/global/common/type/utils'; import { type RequireOnlyOne } from '@fastgpt/global/common/type/utils';
import { addRawTextBuffer, getRawTextBuffer } from '../../../../common/buffer/rawText/controller'; import { getS3DatasetSource } from '../../../../common/s3/sources/dataset';
import { addMinutes } from 'date-fns';
type ResponseDataType = { type ResponseDataType = {
success: boolean; success: boolean;
...@@ -155,11 +154,14 @@ export const useApiDatasetRequest = ({ apiServer }: { apiServer: APIFileServer } ...@@ -155,11 +154,14 @@ export const useApiDatasetRequest = ({ apiServer }: { apiServer: APIFileServer }
} }
if (previewUrl) { if (previewUrl) {
// Get from buffer // Get from buffer
const buffer = await getRawTextBuffer(previewUrl); const rawTextBuffer = await getS3DatasetSource().getRawTextBuffer({
if (buffer) { sourceId: previewUrl,
customPdfParse
});
if (rawTextBuffer) {
return { return {
title, title,
rawText: buffer.text rawText: rawTextBuffer.text
}; };
} }
...@@ -173,11 +175,11 @@ export const useApiDatasetRequest = ({ apiServer }: { apiServer: APIFileServer } ...@@ -173,11 +175,11 @@ export const useApiDatasetRequest = ({ apiServer }: { apiServer: APIFileServer }
getFormatText: true getFormatText: true
}); });
await addRawTextBuffer({ getS3DatasetSource().addRawTextBuffer({
sourceId: previewUrl, sourceId: previewUrl,
sourceName: title || '', sourceName: title || '',
text: rawText, text: rawText,
expiredTime: addMinutes(new Date(), 30) customPdfParse
}); });
return { return {
......
packages/service/core/dataset/collection/controller.ts
...@@ -12,8 +12,6 @@ import { MongoDatasetTraining } from '../training/schema'; ...@@ -12,8 +12,6 @@ import { MongoDatasetTraining } from '../training/schema';
import { MongoDatasetData } from '../data/schema'; import { MongoDatasetData } from '../data/schema';
import { delImgByRelatedId } from '../../../common/file/image/controller'; import { delImgByRelatedId } from '../../../common/file/image/controller';
import { deleteDatasetDataVector } from '../../../common/vectorDB/controller'; import { deleteDatasetDataVector } from '../../../common/vectorDB/controller';
import { delFileByFileIdList } from '../../../common/file/gridfs/controller';
import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import type { ClientSession } from '../../../common/mongo'; import type { ClientSession } from '../../../common/mongo';
import { createOrGetCollectionTags } from './utils'; import { createOrGetCollectionTags } from './utils';
import { rawText2Chunks } from '../read'; import { rawText2Chunks } from '../read';
...@@ -33,9 +31,7 @@ import { ...@@ -33,9 +31,7 @@ import {
getLLMMaxChunkSize getLLMMaxChunkSize
} from '@fastgpt/global/core/dataset/training/utils'; } from '@fastgpt/global/core/dataset/training/utils';
import { DatasetDataIndexTypeEnum } from '@fastgpt/global/core/dataset/data/constants'; import { DatasetDataIndexTypeEnum } from '@fastgpt/global/core/dataset/data/constants';
import { clearCollectionImages } from '../image/utils'; import { getS3DatasetSource } from '../../../common/s3/sources/dataset';
import { getS3DatasetSource, S3DatasetSource } from '../../../common/s3/sources/dataset';
import path from 'node:path';
import { removeS3TTL, isS3ObjectKey } from '../../../common/s3/utils'; import { removeS3TTL, isS3ObjectKey } from '../../../common/s3/utils';
export const createCollectionAndInsertData = async ({ export const createCollectionAndInsertData = async ({
...@@ -326,18 +322,13 @@ export const delCollectionRelatedSource = async ({ ...@@ -326,18 +322,13 @@ export const delCollectionRelatedSource = async ({
if (!teamId) return Promise.reject('teamId is not exist'); if (!teamId) return Promise.reject('teamId is not exist');
const fileIdList = collections.map((item) => item?.fileId || '').filter(Boolean); // FIXME: 兼容旧解析图像删除
const relatedImageIds = collections const relatedImageIds = collections
.map((item) => item?.metadata?.relatedImgId || '') .map((item) => item?.metadata?.relatedImgId || '')
.filter(Boolean); .filter(Boolean);
// Delete files and images in parallel // Delete files and images in parallel
await Promise.all([ await Promise.all([
// Delete files
delFileByFileIdList({
bucketName: BucketNameEnum.dataset,
fileIdList
}),
// Delete images // Delete images
delImgByRelatedId({ delImgByRelatedId({
teamId, teamId,
...@@ -405,10 +396,8 @@ export async function delCollection({ ...@@ -405,10 +396,8 @@ export async function delCollection({
datasetId: { $in: datasetIds }, datasetId: { $in: datasetIds },
collectionId: { $in: collectionIds } collectionId: { $in: collectionIds }
}), }),
// Delete dataset_images
clearCollectionImages(collectionIds),
// Delete images if needed // Delete images if needed
...(delImg ...(delImg // 兼容旧图像删除
? [ ? [
delImgByRelatedId({ delImgByRelatedId({
teamId, teamId,
...@@ -421,10 +410,9 @@ export async function delCollection({ ...@@ -421,10 +410,9 @@ export async function delCollection({
// Delete files if needed // Delete files if needed
...(delFile ...(delFile
? [ ? [
delFileByFileIdList({ getS3DatasetSource().deleteDatasetFilesByKeys(
bucketName: BucketNameEnum.dataset, collections.map((item) => item?.fileId || '').filter(Boolean)
fileIdList: collections.map((item) => item?.fileId || '').filter(Boolean) )
})
] ]
: []), : []),
// Delete vector data // Delete vector data
......
packages/service/core/dataset/controller.ts
...@@ -9,11 +9,6 @@ import { deleteDatasetDataVector } from '../../common/vectorDB/controller'; ...@@ -9,11 +9,6 @@ import { deleteDatasetDataVector } from '../../common/vectorDB/controller';
import { MongoDatasetDataText } from './data/dataTextSchema'; import { MongoDatasetDataText } from './data/dataTextSchema';
import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset'; import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset';
import { retryFn } from '@fastgpt/global/common/system/utils'; import { retryFn } from '@fastgpt/global/common/system/utils';
import { clearDatasetImages } from './image/utils';
import { MongoDatasetCollectionTags } from './tag/schema';
import { removeDatasetSyncJobScheduler } from './datasetSync';
import { mongoSessionRun } from '../../common/mongo/sessionRun';
import { removeImageByPath } from '../../common/file/image/controller';
import { UserError } from '@fastgpt/global/common/error/utils'; import { UserError } from '@fastgpt/global/common/error/utils';
import { getS3DatasetSource } from '../../common/s3/sources/dataset'; import { getS3DatasetSource } from '../../common/s3/sources/dataset';
...@@ -95,77 +90,39 @@ export async function delDatasetRelevantData({ ...@@ -95,77 +90,39 @@ export async function delDatasetRelevantData({
'_id teamId datasetId fileId metadata' '_id teamId datasetId fileId metadata'
).lean(); ).lean();
await retryFn(async () => { // delete training data
await Promise.all([ await MongoDatasetTraining.deleteMany({
// delete training data teamId,
MongoDatasetTraining.deleteMany({ datasetId: { $in: datasetIds }
teamId,
datasetId: { $in: datasetIds }
}),
//Delete dataset_data_texts
MongoDatasetDataText.deleteMany({
teamId,
datasetId: { $in: datasetIds }
}),
//delete dataset_datas
MongoDatasetData.deleteMany({ teamId, datasetId: { $in: datasetIds } }),
// Delete collection image and file
delCollectionRelatedSource({ collections }),
// Delete dataset Image
clearDatasetImages(datasetIds),
// Delete vector data
deleteDatasetDataVector({ teamId, datasetIds })
]);
}); });
// Delete dataset_data_texts in batches by datasetId
for (const datasetId of datasetIds) {
await MongoDatasetDataText.deleteMany({
teamId,
datasetId
}).maxTimeMS(300000); // Reduce timeout for single batch
}
// Delete dataset_datas in batches by datasetId
for (const datasetId of datasetIds) {
await MongoDatasetData.deleteMany({
teamId,
datasetId
}).maxTimeMS(300000);
}
await delCollectionRelatedSource({ collections });
// Delete vector data
await deleteDatasetDataVector({ teamId, datasetIds });
// delete collections // delete collections
await MongoDatasetCollection.deleteMany({ await MongoDatasetCollection.deleteMany({
teamId, teamId,
datasetId: { $in: datasetIds } datasetId: { $in: datasetIds }
}).session(session); }).session(session);
// Delete all dataset files
for (const datasetId of datasetIds) { for (const datasetId of datasetIds) {
await getS3DatasetSource().deleteDatasetFilesByPrefix({ datasetId }); await getS3DatasetSource().deleteDatasetFilesByPrefix({ datasetId });
} }
} }
export const deleteDatasets = async ({
teamId,
datasets
}: {
teamId: string;
datasets: DatasetSchemaType[];
}) => {
const datasetIds = datasets.map((d) => d._id);
// delete collection.tags
await MongoDatasetCollectionTags.deleteMany({
teamId,
datasetId: { $in: datasetIds }
});
// Remove cron job
await Promise.all(
datasets.map((dataset) => {
return removeDatasetSyncJobScheduler(dataset._id);
})
);
// delete all dataset.data and pg data
await mongoSessionRun(async (session) => {
// delete dataset data
await delDatasetRelevantData({ datasets, session });
// delete dataset
await MongoDataset.deleteMany(
{
_id: { $in: datasetIds }
},
{ session }
);
for await (const dataset of datasets) {
await removeImageByPath(dataset.avatar, session);
}
});
};
packages/service/core/dataset/data/controller.ts
import { replaceDatasetQuoteTextWithJWT } from '../../../core/dataset/utils'; import { replaceS3KeyToPreviewUrl } from '../../../core/dataset/utils';
import { addEndpointToImageUrl } from '../../../common/file/image/utils'; import { addEndpointToImageUrl } from '../../../common/file/image/utils';
import type { DatasetDataSchemaType } from '@fastgpt/global/core/dataset/type'; import type { DatasetDataSchemaType } from '@fastgpt/global/core/dataset/type';
import { addDays } from 'date-fns'; import { addDays } from 'date-fns';
...@@ -53,8 +53,8 @@ export const formatDatasetDataValue = ({ ...@@ -53,8 +53,8 @@ export const formatDatasetDataValue = ({
if (!imageId) { if (!imageId) {
return { return {
q: replaceDatasetQuoteTextWithJWT(q, addDays(new Date(), 90)), q: replaceS3KeyToPreviewUrl(q, addDays(new Date(), 90)),
a: a ? replaceDatasetQuoteTextWithJWT(a, addDays(new Date(), 90)) : undefined a: a ? replaceS3KeyToPreviewUrl(a, addDays(new Date(), 90)) : undefined
}; };
} }
......
packages/service/core/dataset/data/schema.ts
...@@ -105,9 +105,6 @@ try { ...@@ -105,9 +105,6 @@ try {
// rebuild data // rebuild data
DatasetDataSchema.index({ rebuilding: 1, teamId: 1, datasetId: 1 }); DatasetDataSchema.index({ rebuilding: 1, teamId: 1, datasetId: 1 });
// 为查询 initJieba 字段不存在的数据添加索引
DatasetDataSchema.index({ initJieba: 1, updateTime: 1 });
// Cron clear invalid data // Cron clear invalid data
DatasetDataSchema.index({ updateTime: 1 }); DatasetDataSchema.index({ updateTime: 1 });
} catch (error) { } catch (error) {
......
packages/service/core/dataset/delete/index.ts 0 → 100644
import { getQueue, getWorker, QueueNames } from '../../../common/bullmq';
import { datasetDeleteProcessor } from './processor';
export type DatasetDeleteJobData = {
teamId: string;
datasetId: string;
};
// 创建工作进程
export const initDatasetDeleteWorker = () => {
return getWorker<DatasetDeleteJobData>(QueueNames.datasetDelete, datasetDeleteProcessor, {
concurrency: 1, // 确保同时只有1个删除任务
removeOnFail: {
age: 30 * 24 * 60 * 60 // 保留30天失败记录
}
});
};
// 添加删除任务
export const addDatasetDeleteJob = (data: DatasetDeleteJobData) => {
// 创建删除队列
const datasetDeleteQueue = getQueue<DatasetDeleteJobData>(QueueNames.datasetDelete, {
defaultJobOptions: {
attempts: 10,
backoff: {
type: 'exponential',
delay: 5000
},
removeOnComplete: true,
removeOnFail: { age: 30 * 24 * 60 * 60 } // 保留30天失败记录
}
});
const jobId = `${data.teamId}:${data.datasetId}`;
// 使用去重机制，避免重复删除
return datasetDeleteQueue.add(jobId, data, {
deduplication: { id: jobId },
delay: 1000 // 延迟1秒执行，确保API响应完成
});
};
packages/service/core/dataset/delete/processor.ts 0 → 100644
import type { Processor } from 'bullmq';
import type { DatasetDeleteJobData } from './index';
import { delDatasetRelevantData, findDatasetAndAllChildren } from '../controller';
import { addLog } from '../../../common/system/log';
import type { DatasetSchemaType } from '@fastgpt/global/core/dataset/type';
import { MongoDatasetCollectionTags } from '../tag/schema';
import { removeDatasetSyncJobScheduler } from '../datasetSync';
import { mongoSessionRun } from '../../../common/mongo/sessionRun';
import { MongoDataset } from '../schema';
import { removeImageByPath } from '../../../common/file/image/controller';
import { MongoDatasetTraining } from '../training/schema';
export const deleteDatasetsImmediate = async ({
teamId,
datasets
}: {
teamId: string;
datasets: DatasetSchemaType[];
}) => {
const datasetIds = datasets.map((d) => d._id);
// delete training data
MongoDatasetTraining.deleteMany({
teamId,
datasetId: { $in: datasetIds }
});
// Remove cron job
await Promise.all(
datasets.map((dataset) => {
// 只处理已标记删除的数据集
if (datasetIds.includes(dataset._id)) {
return removeDatasetSyncJobScheduler(dataset._id);
}
})
);
};
export const deleteDatasets = async ({
teamId,
datasets
}: {
teamId: string;
datasets: DatasetSchemaType[];
}) => {
const datasetIds = datasets.map((d) => d._id);
// delete collection.tags
await MongoDatasetCollectionTags.deleteMany({
teamId,
datasetId: { $in: datasetIds }
});
// Delete dataset avatar
for await (const dataset of datasets) {
await removeImageByPath(dataset.avatar);
}
// delete all dataset.data and pg data
await mongoSessionRun(async (session) => {
// delete dataset data
await delDatasetRelevantData({
datasets,
session
});
});
// delete dataset
await MongoDataset.deleteMany({
_id: { $in: datasetIds }
});
};
export const datasetDeleteProcessor: Processor<DatasetDeleteJobData> = async (job) => {
const { teamId, datasetId } = job.data;
const startTime = Date.now();
addLog.info(`[Dataset Delete] Start deleting dataset: ${datasetId} for team: ${teamId}`);
try {
// 1. 查找知识库及其所有子知识库
const datasets = await findDatasetAndAllChildren({
teamId,
datasetId
});
if (!datasets || datasets.length === 0) {
addLog.warn(`[Dataset Delete] Dataset not found: ${datasetId}`);
return;
}
// 2. 安全检查：确保所有要删除的数据集都已标记为 deleteTime
const markedForDelete = await MongoDataset.find(
{
_id: { $in: datasets.map((d) => d._id) },
teamId,
deleteTime: { $ne: null }
},
{ _id: 1 }
).lean();
if (markedForDelete.length !== datasets.length) {
addLog.warn(
`[Dataset Delete] Safety check: ${markedForDelete.length}/${datasets.length} datasets marked for deletion`,
{
markedDatasetIds: markedForDelete.map((d) => d._id),
totalDatasetIds: datasets.map((d) => d._id)
}
);
}
// 3. 执行真正的删除操作（只删除已经标记为 deleteTime 的数据）
await deleteDatasets({
teamId,
datasets
});
addLog.info(
`[Dataset Delete] Successfully deleted dataset: ${datasetId} and ${datasets.length - 1} children`,
{
duration: Date.now() - startTime,
totalDatasets: datasets.length,
datasetIds: datasets.map((d) => d._id)
}
);
} catch (error: any) {
addLog.error(`[Dataset Delete] Failed to delete dataset: ${datasetId}`, error);
throw error;
}
};
packages/service/core/dataset/image/controller.ts
import { addMinutes } from 'date-fns';
import { bucketName, MongoDatasetImageSchema } from './schema'; import { bucketName, MongoDatasetImageSchema } from './schema';
import { connectionMongo, Types } from '../../../common/mongo'; import { connectionMongo, Types } from '../../../common/mongo';
import fs from 'fs';
import type { FileType } from '../../../common/file/multer';
import fsp from 'fs/promises';
import { computeGridFsChunSize } from '../../../common/file/gridfs/utils';
import { setCron } from '../../../common/system/cron';
import { checkTimerLock } from '../../../common/system/timerLock/utils';
import { TimerIdEnum } from '../../../common/system/timerLock/constants';
import { addLog } from '../../../common/system/log';
import { UserError } from '@fastgpt/global/common/error/utils'; import { UserError } from '@fastgpt/global/common/error/utils';
import { getS3DatasetSource, S3DatasetSource } from '../../../common/s3/sources/dataset';
import { isS3ObjectKey } from '../../../common/s3/utils';
const getGridBucket = () => { const getGridBucket = () => {
return new connectionMongo.mongo.GridFSBucket(connectionMongo.connection.db!, { return new connectionMongo.mongo.GridFSBucket(connectionMongo.connection.db!, {
...@@ -19,53 +8,6 @@ const getGridBucket = () => { ...@@ -19,53 +8,6 @@ const getGridBucket = () => {
}); });
}; };
export const createDatasetImage = async ({
teamId,
datasetId,
file,
expiredTime = addMinutes(new Date(), 30)
}: {
teamId: string;
datasetId: string;
file: FileType;
expiredTime?: Date;
}): Promise<{ imageId: string; previewUrl: string }> => {
const path = file.path;
const gridBucket = getGridBucket();
const metadata = {
teamId: String(teamId),
datasetId: String(datasetId),
expiredTime
};
const stats = await fsp.stat(path);
if (!stats.isFile()) return Promise.reject(`${path} is not a file`);
const readStream = fs.createReadStream(path, {
highWaterMark: 256 * 1024
});
const chunkSizeBytes = computeGridFsChunSize(stats.size);
const stream = gridBucket.openUploadStream(file.originalname, {
metadata,
contentType: file.mimetype,
chunkSizeBytes
});
// save to gridfs
await new Promise((resolve, reject) => {
readStream
.pipe(stream as any)
.on('finish', resolve)
.on('error', reject);
});
return {
imageId: String(stream.id),
previewUrl: ''
};
};
export const getDatasetImageReadData = async (imageId: string) => { export const getDatasetImageReadData = async (imageId: string) => {
// Get file metadata to get contentType // Get file metadata to get contentType
const fileInfo = await MongoDatasetImageSchema.findOne({ const fileInfo = await MongoDatasetImageSchema.findOne({
...@@ -81,93 +23,3 @@ export const getDatasetImageReadData = async (imageId: string) => { ...@@ -81,93 +23,3 @@ export const getDatasetImageReadData = async (imageId: string) => {
fileInfo fileInfo
}; };
}; };
export const getDatasetImageBase64 = async (imageId: string) => {
// Get file metadata to get contentType
const fileInfo = await MongoDatasetImageSchema.findOne({
_id: new Types.ObjectId(imageId)
}).lean();
if (!fileInfo) {
return Promise.reject(new UserError('Image not found'));
}
// Get image stream from GridFS
const { stream } = await getDatasetImageReadData(imageId);
// Convert stream to buffer
const chunks: Buffer[] = [];
return new Promise<string>((resolve, reject) => {
stream.on('data', (chunk: Buffer) => {
chunks.push(chunk);
});
stream.on('end', () => {
// Combine all chunks into a single buffer
const buffer = Buffer.concat(chunks);
// Convert buffer to base64 string
const base64 = buffer.toString('base64');
const dataUrl = `data:${fileInfo.contentType || 'image/jpeg'};base64,${base64}`;
resolve(dataUrl);
});
stream.on('error', reject);
});
};
export const deleteDatasetImage = async (imageId: string) => {
const gridBucket = getGridBucket();
try {
if (isS3ObjectKey(imageId, 'dataset')) {
await getS3DatasetSource().deleteDatasetFileByKey(imageId);
} else {
await gridBucket.delete(new Types.ObjectId(imageId));
}
} catch (error: any) {
const msg = error?.message;
if (msg.includes('File not found')) {
addLog.warn('Delete dataset image error', error);
return;
} else {
return Promise.reject(error);
}
}
};
export const clearExpiredDatasetImageCron = async () => {
const gridBucket = getGridBucket();
const clearExpiredDatasetImages = async () => {
addLog.debug('Clear expired dataset image start');
const data = await MongoDatasetImageSchema.find(
{
'metadata.expiredTime': { $lt: new Date() }
},
'_id'
).lean();
for (const item of data) {
try {
await gridBucket.delete(new Types.ObjectId(item._id));
} catch (error) {
addLog.error('Delete expired dataset image error', error);
}
}
addLog.debug('Clear expired dataset image end');
};
setCron('*/10 * * * *', async () => {
if (
await checkTimerLock({
timerId: TimerIdEnum.clearExpiredDatasetImage,
lockMinuted: 9
})
) {
try {
await clearExpiredDatasetImages();
} catch (error) {
addLog.error('clearExpiredDatasetImageCron error', error);
}
}
});
};
packages/service/core/dataset/image/utils.ts deleted 100644 → 0
import { ERROR_ENUM } from '@fastgpt/global/common/error/errorCode';
import { Types, type ClientSession } from '../../../common/mongo';
import { deleteDatasetImage } from './controller';
import { MongoDatasetImageSchema } from './schema';
import { addMinutes } from 'date-fns';
import jwt from 'jsonwebtoken';
import { EndpointUrl } from '@fastgpt/global/common/file/constants';
export const removeDatasetImageExpiredTime = async ({
ids = [],
collectionId,
session
}: {
ids?: string[];
collectionId: string;
session?: ClientSession;
}) => {
if (ids.length === 0) return;
return MongoDatasetImageSchema.updateMany(
{
_id: {
$in: ids
.filter((id) => Types.ObjectId.isValid(id))
.map((id) => (typeof id === 'string' ? new Types.ObjectId(id) : id))
}
},
{
$unset: { 'metadata.expiredTime': '' },
$set: {
'metadata.collectionId': String(collectionId)
}
},
{ session }
);
};
export const getDatasetImagePreviewUrl = ({
imageId,
teamId,
datasetId,
expiredMinutes
}: {
imageId: string;
teamId: string;
datasetId: string;
expiredMinutes: number;
}) => {
const expiredTime = Math.floor(addMinutes(new Date(), expiredMinutes).getTime() / 1000);
const key = (process.env.FILE_TOKEN_KEY as string) ?? 'filetoken';
const token = jwt.sign(
{
teamId: String(teamId),
datasetId: String(datasetId),
imageId: String(imageId),
exp: expiredTime
},
key
);
return `${EndpointUrl}/api/file/datasetImg/${token}.jpeg`;
};
export const authDatasetImagePreviewUrl = (token?: string) =>
new Promise<{
teamId: string;
datasetId: string;
imageId: string;
}>((resolve, reject) => {
if (!token) {
return reject(ERROR_ENUM.unAuthFile);
}
const key = (process.env.FILE_TOKEN_KEY as string) ?? 'filetoken';
jwt.verify(token, key, (err, decoded: any) => {
if (err || !decoded?.teamId || !decoded?.datasetId) {
reject(ERROR_ENUM.unAuthFile);
return;
}
resolve({
teamId: decoded.teamId,
datasetId: decoded.datasetId,
imageId: decoded.imageId
});
});
});
export const clearDatasetImages = async (datasetIds: string[]) => {
if (datasetIds.length === 0) return;
const images = await MongoDatasetImageSchema.find(
{
'metadata.datasetId': { $in: datasetIds.map((item) => String(item)) }
},
'_id'
).lean();
await Promise.all(images.map((image) => deleteDatasetImage(String(image._id))));
};
export const clearCollectionImages = async (collectionIds: string[]) => {
if (collectionIds.length === 0) return;
const images = await MongoDatasetImageSchema.find(
{
'metadata.collectionId': { $in: collectionIds.map((item) => String(item)) }
},
'_id'
).lean();
await Promise.all(images.map((image) => deleteDatasetImage(String(image._id))));
};
packages/service/core/dataset/migration/schema.ts
...@@ -11,7 +11,7 @@ export type DatasetMigrationLogSchemaType = { ...@@ -11,7 +11,7 @@ export type DatasetMigrationLogSchemaType = {
migrationVersion: string; // 如 'v4.14.3' migrationVersion: string; // 如 'v4.14.3'
// 资源类型和标识 // 资源类型和标识
resourceType: 'collection' | 'image' | 'chat_file'; // 支持不同类型的文件迁移 resourceType: 'collection' | 'dataset_image'; // 支持不同类型的文件迁移
resourceId: string; // collection._id 或 image._id resourceId: string; // collection._id 或 image._id
teamId: string; teamId: string;
datasetId?: string; // collection 有，image 可能没有 datasetId?: string; // collection 有，image 可能没有
...@@ -101,7 +101,7 @@ const DatasetMigrationLogSchema = new Schema({ ...@@ -101,7 +101,7 @@ const DatasetMigrationLogSchema = new Schema({
// 资源类型和标识 // 资源类型和标识
resourceType: { resourceType: {
type: String, type: String,
enum: ['collection', 'image', 'chat_file'], enum: ['collection', 'dataset_image'],
required: true required: true
}, },
resourceId: { resourceId: {
......
packages/service/core/dataset/schema.ts
...@@ -131,6 +131,12 @@ const DatasetSchema = new Schema({ ...@@ -131,6 +131,12 @@ const DatasetSchema = new Schema({
apiDatasetServer: Object, apiDatasetServer: Object,
// 软删除标记字段
deleteTime: {
type: Date,
default: null // null表示未删除，有值表示删除时间
},
// abandoned // abandoned
autoSync: Boolean, autoSync: Boolean,
externalReadUrl: String, externalReadUrl: String,
...@@ -143,6 +149,7 @@ const DatasetSchema = new Schema({ ...@@ -143,6 +149,7 @@ const DatasetSchema = new Schema({
try { try {
DatasetSchema.index({ teamId: 1 }); DatasetSchema.index({ teamId: 1 });
DatasetSchema.index({ type: 1 }); DatasetSchema.index({ type: 1 });
DatasetSchema.index({ deleteTime: 1 }); // 添加软删除字段索引
} catch (error) { } catch (error) {
console.log(error); console.log(error);
} }
......
packages/service/core/dataset/search/controller.ts
...@@ -33,7 +33,7 @@ import { datasetSearchQueryExtension } from './utils'; ...@@ -33,7 +33,7 @@ import { datasetSearchQueryExtension } from './utils';
import type { RerankModelItemType } from '@fastgpt/global/core/ai/model.d'; import type { RerankModelItemType } from '@fastgpt/global/core/ai/model.d';
import { formatDatasetDataValue } from '../data/controller'; import { formatDatasetDataValue } from '../data/controller';
import { pushTrack } from '../../../common/middle/tracks/utils'; import { pushTrack } from '../../../common/middle/tracks/utils';
import { replaceDatasetQuoteTextWithJWT } from '../../../core/dataset/utils'; import { replaceS3KeyToPreviewUrl } from '../../../core/dataset/utils';
import { addDays, addHours } from 'date-fns'; import { addDays, addHours } from 'date-fns';
export type SearchDatasetDataProps = { export type SearchDatasetDataProps = {
...@@ -81,9 +81,11 @@ export type SearchDatasetDataResponse = { ...@@ -81,9 +81,11 @@ export type SearchDatasetDataResponse = {
usingSimilarityFilter: boolean; usingSimilarityFilter: boolean;
queryExtensionResult?: { queryExtensionResult?: {
model: string; llmModel: string;
embeddingModel: string;
inputTokens: number; inputTokens: number;
outputTokens: number; outputTokens: number;
embeddingTokens: number;
query: string; query: string;
}; };
deepSearchResult?: { model: string; inputTokens: number; outputTokens: number }; deepSearchResult?: { model: string; inputTokens: number; outputTokens: number };
...@@ -906,7 +908,7 @@ export async function searchDatasetData( ...@@ -906,7 +908,7 @@ export async function searchDatasetData(
const filterMaxTokensResult = await filterDatasetDataByMaxTokens(scoreFilter, maxTokens); const filterMaxTokensResult = await filterDatasetDataByMaxTokens(scoreFilter, maxTokens);
const finalResult = filterMaxTokensResult.map((item) => { const finalResult = filterMaxTokensResult.map((item) => {
item.q = replaceDatasetQuoteTextWithJWT(item.q, addDays(new Date(), 90)); item.q = replaceS3KeyToPreviewUrl(item.q, addDays(new Date(), 90));
return item; return item;
}); });
...@@ -938,32 +940,32 @@ export const defaultSearchDatasetData = async ({ ...@@ -938,32 +940,32 @@ export const defaultSearchDatasetData = async ({
const query = props.queries[0]; const query = props.queries[0];
const histories = props.histories; const histories = props.histories;
const extensionModel = datasetSearchUsingExtensionQuery const { searchQueries, reRankQuery, aiExtensionResult } = await datasetSearchQueryExtension({
? getLLMModel(datasetSearchExtensionModel) query,
: undefined; llmModel: datasetSearchUsingExtensionQuery
? getLLMModel(datasetSearchExtensionModel).model
const { concatQueries, extensionQueries, rewriteQuery, aiExtensionResult } = : undefined,
await datasetSearchQueryExtension({ embeddingModel: props.model,
query, extensionBg: datasetSearchExtensionBg,
extensionModel, histories
extensionBg: datasetSearchExtensionBg, });
histories
});
const result = await searchDatasetData({ const result = await searchDatasetData({
...props, ...props,
reRankQuery: rewriteQuery, reRankQuery: reRankQuery,
queries: concatQueries queries: searchQueries
}); });
return { return {
...result, ...result,
queryExtensionResult: aiExtensionResult queryExtensionResult: aiExtensionResult
? { ? {
model: aiExtensionResult.model, llmModel: aiExtensionResult.llmModel,
inputTokens: aiExtensionResult.inputTokens, inputTokens: aiExtensionResult.inputTokens,
outputTokens: aiExtensionResult.outputTokens, outputTokens: aiExtensionResult.outputTokens,
query: extensionQueries.join('\n') embeddingModel: aiExtensionResult.embeddingModel,
embeddingTokens: aiExtensionResult.embeddingTokens,
query: searchQueries.join('\n')
} }
: undefined : undefined
}; };
......
packages/service/core/dataset/search/utils.ts
import { type LLMModelItemType } from '@fastgpt/global/core/ai/model.d';
import { queryExtension } from '../../ai/functions/queryExtension'; import { queryExtension } from '../../ai/functions/queryExtension';
import { type ChatItemType } from '@fastgpt/global/core/chat/type'; import { type ChatItemType } from '@fastgpt/global/core/chat/type';
import { hashStr } from '@fastgpt/global/common/string/tools'; import { hashStr } from '@fastgpt/global/common/string/tools';
import { chatValue2RuntimePrompt } from '@fastgpt/global/core/chat/adapt'; import { addLog } from '../../../common/system/log';
export const datasetSearchQueryExtension = async ({ export const datasetSearchQueryExtension = async ({
query, query,
extensionModel, llmModel,
embeddingModel,
extensionBg = '', extensionBg = '',
histories = [] histories = []
}: { }: {
query: string; query: string;
extensionModel?: LLMModelItemType; llmModel?: string;
embeddingModel?: string;
extensionBg?: string; extensionBg?: string;
histories?: ChatItemType[]; histories?: ChatItemType[];
}) => { }) => {
...@@ -28,19 +29,8 @@ export const datasetSearchQueryExtension = async ({ ...@@ -28,19 +29,8 @@ export const datasetSearchQueryExtension = async ({
return filterSameQueries; return filterSameQueries;
}; };
let { queries, rewriteQuery, alreadyExtension } = (() => { // 检查传入的 query 是否已经进行过扩展
// concat query let { queries, reRankQuery, alreadyExtension } = (() => {
let rewriteQuery =
histories.length > 0
? `${histories
.map((item) => {
return `${item.obj}: ${chatValue2RuntimePrompt(item.value).text}`;
})
.join('\n')}
Human: ${query}
`
: query;
/* if query already extension, direct parse */ /* if query already extension, direct parse */
try { try {
const jsonParse = JSON.parse(query); const jsonParse = JSON.parse(query);
...@@ -48,41 +38,45 @@ Human: ${query} ...@@ -48,41 +38,45 @@ Human: ${query}
const alreadyExtension = Array.isArray(jsonParse); const alreadyExtension = Array.isArray(jsonParse);
return { return {
queries, queries,
rewriteQuery: alreadyExtension ? queries.join('\n') : rewriteQuery, reRankQuery: alreadyExtension ? queries.join('\n') : query,
alreadyExtension: alreadyExtension alreadyExtension
}; };
} catch (error) { } catch (error) {
return { return {
queries: [query], queries: [query],
rewriteQuery, reRankQuery: query,
alreadyExtension: false alreadyExtension: false
}; };
} }
})(); })();
// ai extension // Use LLM to generate extension queries
const aiExtensionResult = await (async () => { const aiExtensionResult = await (async () => {
if (!extensionModel || alreadyExtension) return; if (!llmModel || !embeddingModel || alreadyExtension) return;
const result = await queryExtension({
chatBg: extensionBg, try {
query, const result = await queryExtension({
histories, chatBg: extensionBg,
model: extensionModel.model query,
}); histories,
if (result.extensionQueries?.length === 0) return; llmModel,
return result; embeddingModel
});
if (result.extensionQueries?.length === 0) return;
return result;
} catch (error) {
addLog.error('Failed to generate extension queries', error);
}
})(); })();
const extensionQueries = filterSamQuery(aiExtensionResult?.extensionQueries || []);
if (aiExtensionResult) { if (aiExtensionResult) {
queries = filterSamQuery(queries.concat(extensionQueries)); queries = queries.concat(aiExtensionResult.extensionQueries);
rewriteQuery = queries.join('\n'); reRankQuery = queries.join('\n');
} }
return { return {
extensionQueries, searchQueries: queries,
concatQueries: queries, reRankQuery,
rewriteQuery,
aiExtensionResult aiExtensionResult
}; };
}; };
packages/service/core/dataset/utils.ts
...@@ -44,12 +44,12 @@ export const filterDatasetsByTmbId = async ({ ...@@ -44,12 +44,12 @@ export const filterDatasetsByTmbId = async ({
* *
* ```typescript * ```typescript
* const datasetQuoteText = '![image.png](dataset/68fee42e1d416bb5ddc85b19/6901c3071ba2bea567e8d8db/aZos7D-214afce5-4d42-4356-9e05-8164d51c59ae.png)'; * const datasetQuoteText = '![image.png](dataset/68fee42e1d416bb5ddc85b19/6901c3071ba2bea567e8d8db/aZos7D-214afce5-4d42-4356-9e05-8164d51c59ae.png)';
* const replacedText = await replaceDatasetQuoteTextWithJWT(datasetQuoteText, addDays(new Date(), 90)) * const replacedText = await replaceS3KeyToPreviewUrl(datasetQuoteText, addDays(new Date(), 90))
* console.log(replacedText) * console.log(replacedText)
* // '![image.png](http://localhost:3000/api/system/file/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvYmplY3RLZXkiOiJjaGF0LzY5MWFlMjlkNDA0ZDA0Njg3MTdkZDc0Ny82OGFkODVhNzQ2MzAwNmM5NjM3OTlhMDcvalhmWHk4eWZHQUZzOVdKcGNXUmJBaFYyL3BhcnNlZC85YTBmNGZlZC00ZWRmLTQ2MTMtYThkNi01MzNhZjVhZTUxZGMucG5nIiwiaWF0IjoxNzYzMzcwOTYwLCJleHAiOjk1MzkzNzA5NjB9.tMDWg0-ZWRnWPNp9Hakd0w1hhaO8jj2oD98SU0wAQYQ)' * // '![image.png](http://localhost:3000/api/system/file/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvYmplY3RLZXkiOiJjaGF0LzY5MWFlMjlkNDA0ZDA0Njg3MTdkZDc0Ny82OGFkODVhNzQ2MzAwNmM5NjM3OTlhMDcvalhmWHk4eWZHQUZzOVdKcGNXUmJBaFYyL3BhcnNlZC85YTBmNGZlZC00ZWRmLTQ2MTMtYThkNi01MzNhZjVhZTUxZGMucG5nIiwiaWF0IjoxNzYzMzcwOTYwLCJleHAiOjk1MzkzNzA5NjB9.tMDWg0-ZWRnWPNp9Hakd0w1hhaO8jj2oD98SU0wAQYQ)'
* ``` * ```
*/ */
export function replaceDatasetQuoteTextWithJWT(documentQuoteText: string, expiredTime: Date) { export function replaceS3KeyToPreviewUrl(documentQuoteText: string, expiredTime: Date) {
if (!documentQuoteText || typeof documentQuoteText !== 'string') if (!documentQuoteText || typeof documentQuoteText !== 'string')
return documentQuoteText as string; return documentQuoteText as string;
......
packages/service/core/workflow/dispatch/ai/chat.ts
...@@ -41,9 +41,6 @@ import { i18nT } from '../../../../../web/i18n/utils'; ...@@ -41,9 +41,6 @@ import { i18nT } from '../../../../../web/i18n/utils';
import { postTextCensor } from '../../../chat/postTextCensor'; import { postTextCensor } from '../../../chat/postTextCensor';
import { createLLMResponse } from '../../../ai/llm/request'; import { createLLMResponse } from '../../../ai/llm/request';
import { formatModelChars2Points } from '../../../../support/wallet/usage/utils'; import { formatModelChars2Points } from '../../../../support/wallet/usage/utils';
import { replaceDatasetQuoteTextWithJWT } from '../../../dataset/utils';
import { getFileS3Key } from '../../../../common/s3/utils';
import { addDays } from 'date-fns';
export type ChatProps = ModuleDispatchProps< export type ChatProps = ModuleDispatchProps<
AIChatNodeProps & { AIChatNodeProps & {
...@@ -307,7 +304,6 @@ async function filterDatasetQuote({ ...@@ -307,7 +304,6 @@ async function filterDatasetQuote({
: ''; : '';
return { return {
// datasetQuoteText: replaceDatasetQuoteTextWithJWT(datasetQuoteText, addDays(new Date(), 90))
datasetQuoteText datasetQuoteText
}; };
} }
......
packages/service/core/workflow/dispatch/dataset/search.ts
import { import { type DispatchNodeResultType } from '@fastgpt/global/core/workflow/runtime/type.d';
type DispatchNodeResponseType,
type DispatchNodeResultType
} from '@fastgpt/global/core/workflow/runtime/type.d';
import { formatModelChars2Points } from '../../../../support/wallet/usage/utils'; import { formatModelChars2Points } from '../../../../support/wallet/usage/utils';
import type { SelectedDatasetType } from '@fastgpt/global/core/workflow/type/io'; import type { SelectedDatasetType } from '@fastgpt/global/core/workflow/type/io';
import type { SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type'; import type { SearchDataResponseItemType } from '@fastgpt/global/core/dataset/type';
...@@ -117,7 +114,7 @@ export async function dispatchDatasetSearch( ...@@ -117,7 +114,7 @@ export async function dispatchDatasetSearch(
return emptyResult; return emptyResult;
} }
// get vector // Get vector model
const vectorModel = getEmbeddingModel( const vectorModel = getEmbeddingModel(
(await MongoDataset.findById(datasets[0].datasetId, 'vectorModel').lean())?.vectorModel (await MongoDataset.findById(datasets[0].datasetId, 'vectorModel').lean())?.vectorModel
); );
...@@ -165,7 +162,7 @@ export async function dispatchDatasetSearch( ...@@ -165,7 +162,7 @@ export async function dispatchDatasetSearch(
// count bill results // count bill results
const nodeDispatchUsages: ChatNodeUsageType[] = []; const nodeDispatchUsages: ChatNodeUsageType[] = [];
// vector // 1. Search vector
const { totalPoints: embeddingTotalPoints, modelName: embeddingModelName } = const { totalPoints: embeddingTotalPoints, modelName: embeddingModelName } =
formatModelChars2Points({ formatModelChars2Points({
model: vectorModel.model, model: vectorModel.model,
...@@ -177,96 +174,98 @@ export async function dispatchDatasetSearch( ...@@ -177,96 +174,98 @@ export async function dispatchDatasetSearch(
model: embeddingModelName, model: embeddingModelName,
inputTokens: embeddingTokens inputTokens: embeddingTokens
}); });
// Rerank // 2. Rerank
const { totalPoints: reRankTotalPoints, modelName: reRankModelName } = formatModelChars2Points({
model: rerankModelData?.model,
inputTokens: reRankInputTokens
});
if (usingReRank) { if (usingReRank) {
const { totalPoints: reRankTotalPoints, modelName: reRankModelName } =
formatModelChars2Points({
model: rerankModelData?.model,
inputTokens: reRankInputTokens
});
nodeDispatchUsages.push({ nodeDispatchUsages.push({
totalPoints: reRankTotalPoints, totalPoints: reRankTotalPoints,
moduleName: node.name, moduleName: i18nT('account_usage:rerank'),
model: reRankModelName, model: reRankModelName,
inputTokens: reRankInputTokens inputTokens: reRankInputTokens
}); });
} }
// Query extension // 3. Query extension
(() => { if (queryExtensionResult) {
if (queryExtensionResult) { const { totalPoints: llmPoints, modelName: llmModelName } = formatModelChars2Points({
const { totalPoints, modelName } = formatModelChars2Points({ model: queryExtensionResult.llmModel,
model: queryExtensionResult.model, inputTokens: queryExtensionResult.inputTokens,
inputTokens: queryExtensionResult.inputTokens, outputTokens: queryExtensionResult.outputTokens
outputTokens: queryExtensionResult.outputTokens });
}); nodeDispatchUsages.push({
nodeDispatchUsages.push({ totalPoints: llmPoints,
totalPoints, moduleName: i18nT('common:core.module.template.Query extension'),
moduleName: i18nT('common:core.module.template.Query extension'), model: llmModelName,
model: modelName, inputTokens: queryExtensionResult.inputTokens,
inputTokens: queryExtensionResult.inputTokens, outputTokens: queryExtensionResult.outputTokens
outputTokens: queryExtensionResult.outputTokens });
});
return {
totalPoints
};
}
return {
totalPoints: 0
};
})();
// Deep search
(() => {
if (deepSearchResult) {
const { totalPoints, modelName } = formatModelChars2Points({
model: deepSearchResult.model,
inputTokens: deepSearchResult.inputTokens,
outputTokens: deepSearchResult.outputTokens
});
nodeDispatchUsages.push({
totalPoints,
moduleName: i18nT('common:deep_rag_search'),
model: modelName,
inputTokens: deepSearchResult.inputTokens,
outputTokens: deepSearchResult.outputTokens
});
return {
totalPoints
};
}
return {
totalPoints: 0
};
})();
const { totalPoints: embeddingPoints, modelName: embeddingModelName } =
formatModelChars2Points({
model: queryExtensionResult.embeddingModel,
inputTokens: queryExtensionResult.embeddingTokens
});
nodeDispatchUsages.push({
totalPoints: embeddingPoints,
moduleName: `${i18nT('account_usage:ai.query_extension_embedding')}`,
model: embeddingModelName,
inputTokens: queryExtensionResult.embeddingTokens,
outputTokens: 0
});
}
// 4. Deep search
if (deepSearchResult) {
const { totalPoints, modelName } = formatModelChars2Points({
model: deepSearchResult.model,
inputTokens: deepSearchResult.inputTokens,
outputTokens: deepSearchResult.outputTokens
});
nodeDispatchUsages.push({
totalPoints,
moduleName: i18nT('common:deep_rag_search'),
model: modelName,
inputTokens: deepSearchResult.inputTokens,
outputTokens: deepSearchResult.outputTokens
});
}
const totalPoints = nodeDispatchUsages.reduce((acc, item) => acc + item.totalPoints, 0); const totalPoints = nodeDispatchUsages.reduce((acc, item) => acc + item.totalPoints, 0);
const responseData: DispatchNodeResponseType & { totalPoints: number } = {
totalPoints,
query: userChatInput,
embeddingModel: vectorModel.name,
embeddingTokens,
similarity: usingSimilarityFilter ? similarity : undefined,
limit,
searchMode,
embeddingWeight:
searchMode === DatasetSearchModeEnum.mixedRecall ? embeddingWeight : undefined,
// Rerank
...(searchUsingReRank && {
rerankModel: rerankModelData?.name,
rerankWeight: rerankWeight,
reRankInputTokens
}),
searchUsingReRank,
// Results
quoteList: searchRes,
queryExtensionResult,
deepSearchResult
};
return { return {
data: { data: {
quoteQA: searchRes quoteQA: searchRes
}, },
[DispatchNodeResponseKeyEnum.nodeResponse]: responseData, [DispatchNodeResponseKeyEnum.nodeResponse]: {
totalPoints,
query: userChatInput,
embeddingModel: vectorModel.name,
embeddingTokens,
similarity: usingSimilarityFilter ? similarity : undefined,
limit,
searchMode,
embeddingWeight:
searchMode === DatasetSearchModeEnum.mixedRecall ? embeddingWeight : undefined,
// Rerank
...(searchUsingReRank && {
rerankModel: rerankModelData?.name,
rerankWeight: rerankWeight,
reRankInputTokens
}),
searchUsingReRank,
queryExtensionResult: queryExtensionResult
? {
model: queryExtensionResult.llmModel,
inputTokens: queryExtensionResult.inputTokens,
outputTokens: queryExtensionResult.outputTokens,
query: queryExtensionResult.query
}
: undefined,
deepSearchResult,
// Results
quoteList: searchRes
},
nodeDispatchUsages, nodeDispatchUsages,
[DispatchNodeResponseKeyEnum.toolResponses]: [DispatchNodeResponseKeyEnum.toolResponses]:
searchRes.length > 0 searchRes.length > 0
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment