Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 225da9bc by Xianquan Committed by GitHub
Options

fix: make mongo index sync non-destructive (#7313) 

* fix: make mongo index sync non-destructive

* feat: add safe mongo index maintenance

* docs: document mongo index sync modes

* refactor: simplify mongo index synchronization

* refactor: reduce mongo index sync logs

* refactor: unify mongo index declarations

* docs: consolidate mongo index design notes

* refactor: simplify mongo index declarations

* docs: clarify legacy index cleanup upgrade path

* docs: clarify MongoDB deprecated index cleanup

* fix: match deprecated text indexes by weights, drop options check

Allow name+key cleanup for MongoDB text indexes whose stored key is
rewritten to _fts/_ftsx, and stop requiring options equality so valid
deprecations are not blocked by declaration noise.

* fix: keep agent skill folder tree index active

Restore { teamId, parentId, deleteTime } as a current Schema index so
findSkillAndAllChildren can use it and startup cleanup will not drop it.

* fix: align pro submodule for admin build

---------

Co-authored-by: Archer <545436317@qq.com>
parent 9aa9709a
Showing with 2128 additions and 743 deletions
.agents/design/common/mongo-index-sync-strategy.md 0 → 100644
# MongoDB 索引安全同步设计
## 文档状态
- 状态：已落地
- 适用范围：主 Service、日志库、Marketplace 和 `pro/admin` 中由 FastGPT 管理的 MongoDB Schema
- 最终结论：启动时只补建当前 Schema 索引；仅当业务 Schema 明确登记废弃索引时才执行精确清理，任何未知索引一律保留。当前没有业务 Schema 登记废弃索引
## 问题与根因
私有化部署客户可能通过 `mongosh`、运维脚本或数据库管理平台添加自定义索引。原实现会在 Mongoose model 加载时调用 `model.syncIndexes()`，其语义是：
1. 比较 Schema 与数据库已有索引。
2. 删除 Schema 中不存在的索引。
3. 创建 Schema 中缺失的索引。
因此，客户自建索引在服务重启时被删除是 `syncIndexes()` 的确定性行为，并非异常分支。原有 `SYNC_INDEX=false` 虽然可以阻止删除，但也会阻止 FastGPT 新版本补建必需索引，无法作为默认解决方案。
该问题同时影响主业务库、日志库和 Marketplace；多实例启动还会放大重复执行和错误日志问题。
## 最终决策
1. 禁止在启动索引管理中调用 `syncIndexes()` 或 `cleanIndexes()`。
2. `SYNC_INDEX` 弃用；启动时固定检查差异并补建当前 Schema 缺失的索引。
3. `MONGO_DEPRECATE_INDEX` 仅控制是否清理显式声明的 FastGPT 系统内置废弃索引，默认值为 `true`；设置为 `false` 不影响缺失索引的创建。
4. 不提供可切换回全量删除的启动模式。
5. 当前索引和废弃索引统一通过 `defineIndex()` 声明；废弃索引必须声明在所属 Schema 文件中。
6. Schema 外且未被显式声明为废弃的索引只告警、不删除。
7. 删除废弃索引前必须先成功创建当前 Schema 索引；创建失败时不进入删除阶段。
8. 主 Service、日志库和 Marketplace 统一复用 `MongoIndexManager`，保持同步语义与错误处理一致。
## 索引声明
`packages/service/common/mongo/schemaIndexes.ts` 提供统一入口：
```ts
defineIndex(ChatSchema, {
key: { appId: 1, chatId: 1 },
options: { unique: true }
});
defineIndex(ChatSchema, {
key: { legacyField: 1 },
options: { name: 'legacyField_1' },
deprecated: true
});
```
声明规则：
- 未设置 `deprecated` 时，`defineIndex()` 代理 `Schema.index()`，该索引属于当前 Schema。
- 显式设置 `deprecated: true` 时，只在 Schema 实例上登记清理元数据，不再把索引加入 Mongoose Schema。
- 废弃索引元数据包含索引名、key，以及可选的 options 快照（仅供阅读/排查，不参与删除匹配）；不包含 collection name，collection 由 model 推导。
- 未显式提供索引名时，按 MongoDB 默认规则由 key 推导。
- 同一 Schema 内重复登记同名废弃索引属于配置错误，应立即抛错。
- FastGPT 管理的索引不再使用字段级 `index: true` 或 `unique: true` 隐式声明，避免绕开统一入口。
废弃索引元数据使用私有 symbol 挂在 Schema 实例上，使当前索引与历史清理声明在同一业务文件中完成 review，并避免中心清单与 Schema 演进脱节。
## 启动同步流程
每个 model 启动时执行以下流程：
1. `diffIndexes({ indexOptionsToCreate: true })` 生成 `toCreate` 和 `toDrop`。
2. `toDrop` 仅表示数据库中存在但当前 Schema 未声明的索引，记录 `warn` 后保留。
3. `createIndexes({ background: true })` 创建当前 Schema 索引。
4. 当 `MONGO_DEPRECATE_INDEX=true` 时，从 `model.schema` 读取废弃索引声明。
5. 在 model 对应 collection 中按 name 查找废弃索引。
6. 仅删除 name 与 key 匹配的索引；options 不参与匹配。
key 匹配规则：
- 普通索引：`listIndexes` 的 key 与声明 key 按字段顺序精确相等。
- text 索引：MongoDB 会把 key 改写为 `{ _fts: "text", _ftsx: 1 }`，因此改为比较声明中的 text 字段集合与 `weights` 字段集合。
- options（`unique` / `sparse` / TTL / partial / collation）故意不参与匹配，减少重复声明成本；声明方需自行确认同名同 key 的索引确实可删。
清理结果分为：
- `drop`：定义匹配，已删除或在 dry-run 中可删除。
- `skip_missing`：索引不存在或已被其他实例删除。
- `skip_mismatch`：同名索引的 key 不匹配，保留并告警。
- `error`：查询或删除失败，保留错误信息。
同一进程内同一 model 的并发调用复用正在执行的任务；任务完成后移除缓存，允许热加载或重连再次检查。多实例重复清理时，`IndexNotFound` 视为幂等跳过。
## 模块职责
- `packages/service/common/mongo/indexManager.ts`
- `inspectModelIndexes()`：只计算差异，不创建或删除索引。
- `syncModelIndexes()`：执行安全同步并复用同一 model 的进行中任务。
- `cleanupModelDeprecatedIndexes()`：按 Schema 本地声明检查或清理废弃索引。
- `summarizeCleanupReport()` / `formatCleanupReport()`：提供结构化结果与可读报告。
- `packages/service/common/mongo/schemaIndexes.ts`
- `defineIndex()`：声明当前索引或登记废弃索引。
- `getDeprecatedIndexes()`：读取当前 Schema 自身的废弃索引元数据。
- `packages/service/common/mongo/index.ts`
- 主 Service 和日志库的 model 注册入口，只负责按环境条件触发 manager。
- `projects/marketplace/src/service/mongo/index.ts`
- Marketplace 的 model 注册入口，复用同一 manager 并完整捕获异步错误。
原中心废弃索引清单已删除。当前 chat、sandbox instance 和 Agent Skill 均未登记废弃索引，因此启动同步不会自动删除任何历史索引；manager 仅保留显式清理能力供后续经过单独确认的迁移使用。
## 日志与失败处理
- 无差异且无清理动作时不输出同步摘要。
- `info`：实际创建或删除索引时输出 collection 级摘要。
- `warn`：发现 Schema 外索引，或废弃声明与数据库同名索引不匹配。
- `error`：当前索引创建失败，或废弃索引检查、删除失败。
索引任务失败不阻止 model 注册，但必须记录 model、collection 和错误信息。`createIndexes()` 的同名、同 key 或 options 冲突由 MongoDB/Mongoose 抛错并进入错误日志，不自动修正。
## 安全边界
1. 未登记的 Schema 外索引不会被删除，包括客户自建索引和无法确认所有权的历史索引。
2. `createIndexes()` 不会修改已存在索引的 options。TTL、唯一约束、partial filter 等变化必须通过明确迁移处理。
3. 错误的 Schema 本地废弃声明会在启动时触发清理，因此 name + key 匹配和代码 review 是必须保留的防线。
4. 启动流程不暴露 Mongoose 全量同步能力；需要诊断时复用 manager 的 inspect/dry-run 能力。
5. `MONGO_DEPRECATE_INDEX=false` 只关闭废弃索引清理，当前 Schema 缺失的索引仍会创建。
6. 客户自建索引应显式设置自定义名称，不使用 MongoDB 按 key 生成的默认名称，避免与 FastGPT 系统内置索引重名。
## 已知暂不处理的历史索引
### `llm_request_records.requestId_1`
- `76d6234de V4.14.7 features (#6406)` 首次在 `requestId` path 上声明 `unique: true`，MongoDB 创建 `requestId_1`。
- `f008ea971 feat: teamId in reacord llm` 将约束调整为 `{ teamId: 1, requestId: 1 }` 复合唯一索引。
- 旧索引对当前 Schema 已无必要，并可能继续施加跨团队全局唯一约束。
- 本次不新增其废弃声明。后续如需清理，必须在 `packages/service/core/ai/record/schema.ts` 单独声明并补充回归测试。
## 验收标准
1. 启动时创建当前 Schema 缺失索引，并保留所有未登记的 Schema 外索引。
2. Schema 未登记废弃索引时不执行删除。
3. 只有 name 与 key 匹配的废弃索引会被删除；text 索引通过 weights 字段集合匹配。
4. 当前索引创建失败时不删除废弃索引。
5. 同名但定义不同的索引保留并告警。
6. 已不存在的废弃索引和多实例并发重复清理保持幂等。
7. 当前所有业务 Schema 均未登记废弃索引，启动同步不会自动删除历史索引。
8. 主 Service、日志库和 Marketplace 始终调用同一 manager，并由 `MONGO_DEPRECATE_INDEX` 统一控制废弃索引清理。
## 后续事项
以下事项不影响当前方案交付：
1. 增加 Root 管理员 inspect/apply API 或等价脚本，提供启动日志之外的诊断入口；apply 仍只允许执行 manager 定义的安全动作。
2. 单独评估并迁移 `llm_request_records.requestId_1`。
3. 为新索引逐步采用 `fg_<collection>_<purpose>` 显式命名规范；旧索引不做批量改名。
AGENTS.md
...@@ -89,6 +89,13 @@ FastGPT 是一个 AI Agent 构建平台,通过 Flow 提供开箱即用的数据 ...@@ -89,6 +89,13 @@ FastGPT 是一个 AI Agent 构建平台,通过 Flow 提供开箱即用的数据
- 所有代码编写、修改、重构和测试调整都必须遵守 [FastGPT 代码规范](./.agents/code/syntax.md)。开始改动前先查看相关规范；如果规范与当前实现习惯冲突，优先按规范执行，并只在有明确业务或兼容性理由时说明例外。 - 所有代码编写、修改、重构和测试调整都必须遵守 [FastGPT 代码规范](./.agents/code/syntax.md)。开始改动前先查看相关规范；如果规范与当前实现习惯冲突，优先按规范执行，并只在有明确业务或兼容性理由时说明例外。
### MongoDB Schema 与索引维护
- 所有由 FastGPT 管理的当前索引和废弃索引都必须通过 `defineIndex(schema, { key, options, deprecated })` 声明：`deprecated` 默认是 `false`，当前索引省略该字段；只有废弃索引显式使用 `deprecated: true`。不要直接调用 `schema.index()`，也不要在字段定义中使用 `index: true` 或 `unique: true` 隐式创建索引。
- 每当新增、修改、删除或重命名 MongoDB/Mongoose Schema 字段、索引定义、唯一约束、TTL、partialFilterExpression、collation 等索引相关配置时，必须同步检查是否有 FastGPT 旧版本创建的索引不再被当前 Schema 使用。
- 如果历史索引可能继续影响写入约束、查询计划或存储成本，应在所属 Schema 文件中通过 `defineIndex(schema, { key, options, deprecated: true })` 紧邻当前索引声明登记删除定义，并补充/调整 `packages/service/test/common/mongo/indexManager.test.ts` 的清理行为覆盖。索引名默认由 key 推导，也可通过 `options.name` 显式指定；删除前按 name 定位，再精确匹配 key（text 索引兼容 `_fts/_ftsx` 与 weights）；options 不参与匹配。
- 不要登记客户自建索引、无法确认来源的索引，或仅凭当前 Schema 未声明就推断为废弃的索引。主动同步只允许删除 FastGPT 明确创建过、明确废弃且与 Schema 本地声明精确匹配的历史索引。
### API 入参校验 ### API 入参校验
- 编写或修改 NextJS API 路由时，如果需要校验接口入参（`req.body`、`req.query`、`req.params`），必须使用 `parseApiInput`，不要直接写 `SomeSchema.parse(req.body)`、`SomeSchema.parse(req.query)` 或 `SomeSchema.parse(req.params)`。 - 编写或修改 NextJS API 路由时，如果需要校验接口入参（`req.body`、`req.query`、`req.params`），必须使用 `parseApiInput`，不要直接写 `SomeSchema.parse(req.body)`、`SomeSchema.parse(req.query)` 或 `SomeSchema.parse(req.params)`。
...@@ -139,7 +146,7 @@ function agent_loop(用户需求){ ...@@ -139,7 +146,7 @@ function agent_loop(用户需求){
提出问题，让用户提供答案; 提出问题，让用户提供答案;
调整需求文档; 调整需求文档;
} }
// 2. 开发文档编写 // 2. 开发文档编写
while(开发文档编写未完成){ while(开发文档编写未完成){
编写开发文档; 编写开发文档;
......
document/content/self-host/config/env.en.mdx
...@@ -23,7 +23,7 @@ These variables are mainly validated by `packages/service/env.ts` and apply to ` ...@@ -23,7 +23,7 @@ These variables are mainly validated by `packages/service/env.ts` and apply to `
| Variable | Default | Description | | Variable | Default | Description |
| --------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | --------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DB_MAX_LINK` | `5` | Maximum connection pool size for MongoDB, PG, OceanBase, openGauss, and other databases. | | `DB_MAX_LINK` | `5` | Maximum connection pool size for MongoDB, PG, OceanBase, openGauss, and other databases. |
| `SYNC_INDEX` | `true` | Whether MongoDB indexes are synchronized at startup. | | `SYNC_INDEX` | `true` | Whether to create missing MongoDB indexes and remove explicitly declared deprecated indexes at startup. Maintain indexes manually when disabled. |
| `FILE_TOKEN_KEY` | None, **required** | Secret for file read and file authorization flows. Must be at least 6 characters. | | `FILE_TOKEN_KEY` | None, **required** | Secret for file read and file authorization flows. Must be at least 6 characters. |
| `AES256_SECRET_KEY` | None, **required** | Secret used by AES encryption and decryption. Must be at least 6 characters. | | `AES256_SECRET_KEY` | None, **required** | Secret used by AES encryption and decryption. Must be at least 6 characters. |
| `INVOKE_TOKEN_SECRET` | None, **required** | JWT secret for Invoke reverse calls. Must be at least 32 characters. | | `INVOKE_TOKEN_SECRET` | None, **required** | JWT secret for Invoke reverse calls. Must be at least 32 characters. |
......
document/content/self-host/config/env.mdx
...@@ -23,7 +23,7 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说 ...@@ -23,7 +23,7 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说
| 变量 | 默认值 | 说明 | | 变量 | 默认值 | 说明 |
| --------------------- | ------------------ | ----------------------------------------------------------------------------------------------------- | | --------------------- | ------------------ | ----------------------------------------------------------------------------------------------------- |
| `DB_MAX_LINK` | `5` | MongoDB、PG、OceanBase、openGauss 等数据库连接池最大连接数。 | | `DB_MAX_LINK` | `5` | MongoDB、PG、OceanBase、openGauss 等数据库连接池最大连接数。 |
| `SYNC_INDEX` | `true` | 启动时是否同步 MongoDB 索引。 | | `SYNC_INDEX` | `true` | 是否在启动时创建缺失的 MongoDB 索引并清理显式声明的废弃索引；关闭后需自行维护索引。 |
| `FILE_TOKEN_KEY` | 无，**必填** | 文件读取、文件鉴权相关密钥，长度至少 6 位。 | | `FILE_TOKEN_KEY` | 无，**必填** | 文件读取、文件鉴权相关密钥，长度至少 6 位。 |
| `AES256_SECRET_KEY` | 无，**必填** | AES 加解密密钥，长度至少 6 位。 | | `AES256_SECRET_KEY` | 无，**必填** | AES 加解密密钥，长度至少 6 位。 |
| `INVOKE_TOKEN_SECRET` | 无，**必填** | Invoke 反向调用 JWT 密钥，长度至少 32 位。 | | `INVOKE_TOKEN_SECRET` | 无，**必填** | Invoke 反向调用 JWT 密钥，长度至少 32 位。 |
......
document/content/self-host/upgrading/4-15/4154.en.mdx
...@@ -14,6 +14,31 @@ production; local development can use `http://localhost:3000`. ...@@ -14,6 +14,31 @@ production; local development can use `http://localhost:3000`.
```bash ```bash
FE_DOMAIN=https://fastgpt.example.com FE_DOMAIN=https://fastgpt.example.com
``` ```
### MongoDB Index Synchronization Changes
Starting with V4.15.4, `SYNC_INDEX` is deprecated and replaced by `MONGO_DEPRECATE_INDEX`. The new variable controls whether indexes explicitly marked as deprecated by a schema are removed and defaults to `true`. Setting it to `false` skips only deprecated-index cleanup; missing current schema indexes are still created.
FastGPT now performs safe index synchronization automatically at startup:
- Creates indexes that are missing from the current FastGPT schemas.
- Removes only built-in historical FastGPT indexes explicitly marked as deprecated by the corresponding schema and whose name, key, and relevant options match exactly.
- Preserves custom indexes and any other indexes that are not explicitly declared as deprecated.
This process does not call Mongoose's full `syncIndexes()` operation, so indexes are never removed simply because they are absent from a FastGPT schema.
> **Default and deletion boundary: `MONGO_DEPRECATE_INDEX` defaults to `true`. It removes only built-in indexes that a FastGPT schema explicitly marks as deprecated and whose definitions match exactly; customer-created indexes are not removed. Give every custom index an explicit name instead of relying on MongoDB's key-derived default name to prevent collisions with built-in FastGPT index names.**
> **Legacy index cleanup: V4.15.4 does not mark any existing historical indexes as deprecated, so upgrading to this version does not automatically remove old indexes. Future releases will explicitly mark verified obsolete indexes in their schemas and remove them incrementally.**
To fully remove obsolete indexes before upgrading to V4.15.4:
1. Upgrade to and start V4.15.3 once.
2. Set `SYNC_INDEX=true`, restart the services, and wait for index synchronization to finish.
3. After confirming that index synchronization succeeded, upgrade to V4.15.4.
V4.15.3 removes every index that is not declared in its schemas, which may include custom indexes. Back up your database and review the existing indexes before following this procedure. If custom indexes must be preserved, record their definitions and recreate them after synchronization, or do not use V4.15.3 for full cleanup.
Setting `MONGO_DEPRECATE_INDEX=false` skips deprecated-index cleanup that may be introduced in future releases, but does not skip creation of missing indexes.
## 🚀 New Features ## 🚀 New Features
......
document/content/self-host/upgrading/4-15/4154.mdx
...@@ -12,6 +12,31 @@ FastGPT 服务启动时会校验 `FE_DOMAIN`。请将它配置为客户端访问 ...@@ -12,6 +12,31 @@ FastGPT 服务启动时会校验 `FE_DOMAIN`。请将它配置为客户端访问
```bash ```bash
FE_DOMAIN=https://fastgpt.example.com FE_DOMAIN=https://fastgpt.example.com
``` ```
### MongoDB 索引同步调整
V4.15.4 起，`SYNC_INDEX` 弃用，新增 `MONGO_DEPRECATE_INDEX` 环境变量，用于控制是否清理 Schema 显式标记的废弃索引，默认值为 `true`。设置为 `false` 时只跳过废弃索引清理，不影响当前 Schema 缺失索引的创建。
FastGPT 启动时会自动执行安全的主动同步：
- 创建当前 FastGPT Schema 中缺失的索引。
- 仅删除对应 Schema 明确标记为废弃、且 name、key 和关键 options 完全匹配的 FastGPT 系统内置历史索引。
- 保留客户自建索引及其他未声明的索引。
该同步不会调用 Mongoose 的全量 `syncIndexes()`，因此不会按“未在 Schema 中声明”这一条件批量删除索引。
> **默认开启与删除边界：`MONGO_DEPRECATE_INDEX` 默认为 `true`，仅删除 FastGPT Schema 显式声明为废弃、且索引定义精确匹配的系统内置索引，不会删除客户自建索引。建议为自建索引显式设置自定义名称，不要使用 MongoDB 按 key 生成的默认名称，避免与 FastGPT 系统内置索引重名。**
> **旧索引清理说明：V4.15.4 不会把任何已有历史索引标记为废弃，因此升级到该版本时不会自动删除旧索引。后续版本会在确认安全后，通过 Schema 中的显式废弃标记逐步清理对应索引。**
如需在升级 V4.15.4 前完整删除历史过期索引，请按以下顺序操作：
1. 先升级并启动一次 V4.15.3。
2. 设置 `SYNC_INDEX=true`，重启服务并等待索引同步完成。
3. 确认索引同步成功后，再升级至 V4.15.4。
V4.15.3 的索引同步会删除所有未在当时 Schema 中声明的索引，其中可能包含客户自建索引。执行上述步骤前，请先备份数据库并检查现有索引；如需保留自建索引，请记录其定义并在同步后重新创建，或不要使用 V4.15.3 进行全量清理。
`MONGO_DEPRECATE_INDEX=false` 会跳过未来版本可能声明的废弃索引清理，但不会跳过缺失索引的创建。
## 🚀 新增内容 ## 🚀 新增内容
......
document/data/doc-last-modified.json
...@@ -320,8 +320,8 @@ ...@@ -320,8 +320,8 @@
"content/self-host/upgrading/4-15/41507.mdx": "2026-06-30T17:31:43+08:00", "content/self-host/upgrading/4-15/41507.mdx": "2026-06-30T17:31:43+08:00",
"content/self-host/upgrading/4-15/4151.en.mdx": "2026-07-21T12:01:28+08:00", "content/self-host/upgrading/4-15/4151.en.mdx": "2026-07-21T12:01:28+08:00",
"content/self-host/upgrading/4-15/4151.mdx": "2026-07-21T12:01:28+08:00", "content/self-host/upgrading/4-15/4151.mdx": "2026-07-21T12:01:28+08:00",
"content/self-host/upgrading/4-15/4152.en.mdx": "2026-07-17T19:17:14+08:00", "content/self-host/upgrading/4-15/4152.en.mdx": "2026-07-20T15:11:00+08:00",
"content/self-host/upgrading/4-15/4152.mdx": "2026-07-18T12:51:59+08:00", "content/self-host/upgrading/4-15/4152.mdx": "2026-07-20T15:11:00+08:00",
"content/self-host/upgrading/4-15/4153.en.mdx": "2026-07-18T12:51:59+08:00", "content/self-host/upgrading/4-15/4153.en.mdx": "2026-07-18T12:51:59+08:00",
"content/self-host/upgrading/4-15/4153.mdx": "2026-07-18T12:51:59+08:00", "content/self-host/upgrading/4-15/4153.mdx": "2026-07-18T12:51:59+08:00",
"content/self-host/upgrading/4-15/4154.en.mdx": "2026-07-22T11:49:20+08:00", "content/self-host/upgrading/4-15/4154.en.mdx": "2026-07-22T11:49:20+08:00",
...@@ -468,4 +468,4 @@ ...@@ -468,4 +468,4 @@
"content/self-host/upgrading/upgrade-intruction.mdx": "2026-04-26T21:08:47+08:00", "content/self-host/upgrading/upgrade-intruction.mdx": "2026-04-26T21:08:47+08:00",
"content/toc.en.mdx": "2026-07-21T13:14:50+08:00", "content/toc.en.mdx": "2026-07-21T13:14:50+08:00",
"content/toc.mdx": "2026-07-21T13:14:50+08:00" "content/toc.mdx": "2026-07-21T13:14:50+08:00"
} }
\ No newline at end of file
packages/service/common/buffer/tts/schema.ts
import { Schema, getMongoModel } from '../../../common/mongo'; import { defineIndex, Schema, getMongoModel } from '../../../common/mongo';
import { type TTSBufferSchemaType } from './type'; import { type TTSBufferSchemaType } from './type';
import { getLogger, LogCategories } from '../../logger'; import { getLogger, LogCategories } from '../../logger';
...@@ -25,12 +25,11 @@ const TTSBufferSchema = new Schema({ ...@@ -25,12 +25,11 @@ const TTSBufferSchema = new Schema({
} }
}); });
try { defineIndex(TTSBufferSchema, { key: { bufferId: 1 } });
TTSBufferSchema.index({ bufferId: 1 }); // 24 hour
// 24 hour defineIndex(TTSBufferSchema, {
TTSBufferSchema.index({ createTime: 1 }, { expireAfterSeconds: 24 * 60 * 60 }); key: { createTime: 1 },
} catch (error) { options: { expireAfterSeconds: 24 * 60 * 60 }
logger.error('Failed to build TTS buffer indexes', { error }); });
}
export const MongoTTSBuffer = getMongoModel<TTSBufferSchemaType>(collectionName, TTSBufferSchema); export const MongoTTSBuffer = getMongoModel<TTSBufferSchemaType>(collectionName, TTSBufferSchema);
packages/service/common/file/gridfs/schema.ts
import { Schema, getMongoModel } from '../../mongo'; import { defineIndex, Schema, getMongoModel } from '../../mongo';
const DatasetFileSchema = new Schema({ const DatasetFileSchema = new Schema({
metadata: Object metadata: Object
...@@ -7,10 +7,10 @@ const ChatFileSchema = new Schema({ ...@@ -7,10 +7,10 @@ const ChatFileSchema = new Schema({
metadata: Object metadata: Object
}); });
DatasetFileSchema.index({ uploadDate: -1 }); defineIndex(DatasetFileSchema, { key: { uploadDate: -1 } });
ChatFileSchema.index({ uploadDate: -1 }); defineIndex(ChatFileSchema, { key: { uploadDate: -1 } });
ChatFileSchema.index({ 'metadata.chatId': 1 }); defineIndex(ChatFileSchema, { key: { 'metadata.chatId': 1 } });
export const MongoDatasetFileSchema = getMongoModel('dataset.files', DatasetFileSchema); export const MongoDatasetFileSchema = getMongoModel('dataset.files', DatasetFileSchema);
export const MongoChatFileSchema = getMongoModel('chat.files', ChatFileSchema); export const MongoChatFileSchema = getMongoModel('chat.files', ChatFileSchema);
packages/service/common/file/image/schema.ts
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { Schema, getMongoModel } from '../../mongo'; import { defineIndex, Schema, getMongoModel } from '../../mongo';
import { type MongoImageSchemaType } from '@fastgpt/global/common/file/image/type'; import { type MongoImageSchemaType } from '@fastgpt/global/common/file/image/type';
import { getLogger, LogCategories } from '../../logger'; import { getLogger, LogCategories } from '../../logger';
...@@ -20,20 +20,19 @@ const ImageSchema = new Schema({ ...@@ -20,20 +20,19 @@ const ImageSchema = new Schema({
metadata: Object metadata: Object
}); });
try { // tts expired（60 Minutes）
// tts expired（60 Minutes） defineIndex(ImageSchema, {
ImageSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 60 * 60 }); key: { expiredTime: 1 },
ImageSchema.index({ type: 1 }); options: { expireAfterSeconds: 60 * 60 }
// delete related img });
ImageSchema.index({ teamId: 1, 'metadata.relatedId': 1 }); defineIndex(ImageSchema, { key: { type: 1 } });
// delete related img
defineIndex(ImageSchema, { key: { teamId: 1, 'metadata.relatedId': 1 } });
// Cron clear invalid img // Cron clear invalid img
ImageSchema.index( defineIndex(ImageSchema, {
{ createTime: 1 }, key: { createTime: 1 },
{ partialFilterExpression: { 'metadata.relatedId': { $exists: true } } } options: { partialFilterExpression: { 'metadata.relatedId': { $exists: true } } }
); });
} catch (error) {
logger.error('Failed to build image indexes', { error });
}
export const MongoImage = getMongoModel<MongoImageSchemaType>('image', ImageSchema); export const MongoImage = getMongoModel<MongoImageSchemaType>('image', ImageSchema);
packages/service/common/middle/tracks/schema.ts
import { type TrackSchemaType } from '@fastgpt/global/common/middle/tracks/type'; import { type TrackSchemaType } from '@fastgpt/global/common/middle/tracks/type';
import { getMongoModel, Schema } from '../../mongo'; import { defineIndex, getMongoModel, Schema } from '../../mongo';
import { TrackEnum } from '@fastgpt/global/common/middle/tracks/constants'; import { TrackEnum } from '@fastgpt/global/common/middle/tracks/constants';
const TrackSchema = new Schema({ const TrackSchema = new Schema({
...@@ -11,24 +11,24 @@ const TrackSchema = new Schema({ ...@@ -11,24 +11,24 @@ const TrackSchema = new Schema({
data: Object data: Object
}); });
TrackSchema.index({ event: 1 }); defineIndex(TrackSchema, { key: { event: 1 } });
// Dataset search index // Dataset search index
TrackSchema.index( defineIndex(TrackSchema, {
{ event: 1, teamId: 1, 'data.datasetId': 1, createTime: -1 }, key: { event: 1, teamId: 1, 'data.datasetId': 1, createTime: -1 },
{ options: {
partialFilterExpression: { partialFilterExpression: {
event: TrackEnum.datasetSearch event: TrackEnum.datasetSearch
} }
} }
); });
// QPM index // QPM index
TrackSchema.index( defineIndex(TrackSchema, {
{ event: 1, createTime: -1, 'data.requestCount': 1 }, key: { event: 1, createTime: -1, 'data.requestCount': 1 },
{ options: {
partialFilterExpression: { partialFilterExpression: {
event: TrackEnum.teamChatQPM event: TrackEnum.teamChatQPM
} }
} }
); });
export const TrackModel = getMongoModel<TrackSchemaType>('tracks', TrackSchema); export const TrackModel = getMongoModel<TrackSchemaType>('tracks', TrackSchema);
packages/service/common/mongo/index.ts
...@@ -9,6 +9,7 @@ import type { ...@@ -9,6 +9,7 @@ import type {
} from 'mongoose'; } from 'mongoose';
import mongoose, { Mongoose } from 'mongoose'; import mongoose, { Mongoose } from 'mongoose';
import { serviceEnv } from '../../env'; import { serviceEnv } from '../../env';
import { MongoIndexManager } from './indexManager';
const logger = getLogger(LogCategories.INFRA.MONGO); const logger = getLogger(LogCategories.INFRA.MONGO);
...@@ -136,7 +137,6 @@ export const getMongoModel = <T>(name: string, schema: mongoose.Schema): Model<T ...@@ -136,7 +137,6 @@ export const getMongoModel = <T>(name: string, schema: mongoose.Schema): Model<T
const model = connectionMongo.model(name, schema) as Model<T>; const model = connectionMongo.model(name, schema) as Model<T>;
// Sync index
syncMongoIndex(model); syncMongoIndex(model);
return model; return model;
...@@ -148,13 +148,12 @@ export const getMongoLogModel = <T>(name: string, schema: mongoose.Schema): Mode ...@@ -148,13 +148,12 @@ export const getMongoLogModel = <T>(name: string, schema: mongoose.Schema): Mode
const model = connectionLogMongo.model(name, schema) as Model<T>; const model = connectionLogMongo.model(name, schema) as Model<T>;
// Sync index
syncMongoIndex(model); syncMongoIndex(model);
return model; return model;
}; };
const syncMongoIndex = async (model: Model<any>) => { const syncMongoIndex = (model: Model<any>) => {
if ( if (
process.env.NODE_ENV === 'test' || process.env.NODE_ENV === 'test' ||
process.env.NEXT_PHASE === 'phase-production-build' || process.env.NEXT_PHASE === 'phase-production-build' ||
...@@ -164,11 +163,34 @@ const syncMongoIndex = async (model: Model<any>) => { ...@@ -164,11 +163,34 @@ const syncMongoIndex = async (model: Model<any>) => {
return; return;
} }
try { void MongoIndexManager.syncModelIndexes({
await model.syncIndexes({ background: true }); model,
} catch (error) { logger
logger.error('Failed to sync MongoDB indexes', { modelName: model.modelName, error }); }).catch((error) => {
} logger.error('Failed to ensure MongoDB indexes', {
modelName: model.modelName,
collectionName: model.collection.collectionName,
error
});
});
}; };
export const ReadPreference = connectionMongo.mongo.ReadPreference; export const ReadPreference = connectionMongo.mongo.ReadPreference;
export { MongoIndexManager } from './indexManager';
export {
getDeprecatedIndexes as getSchemaDeprecatedMongoIndexes,
defineIndex
} from './schemaIndexes';
export type {
MongoIndexCleanupAction,
MongoIndexCleanupReport,
MongoIndexCleanupReportItem,
MongoIndexCleanupSummary,
MongoIndexSyncResult
} from './indexManager';
export type {
DefineMongoIndexOptions,
DeprecatedMongoIndexDefinition,
DeprecatedMongoIndexOptions
} from './schemaIndexes';
packages/service/common/mongo/indexManager.ts 0 → 100644
import { getLogger, LogCategories } from '../logger';
import type { Model } from 'mongoose';
import { getDeprecatedIndexes, type DeprecatedMongoIndexDefinition } from './schemaIndexes';
const defaultLogger = getLogger(LogCategories.INFRA.MONGO);
type MongoIndexLogger = {
debug: (message: string, data?: Record<string, unknown>) => void;
info: (message: string, data?: Record<string, unknown>) => void;
warn: (message: string, data?: Record<string, unknown>) => void;
error: (message: string, data?: Record<string, unknown>) => void;
};
type MongooseDiffIndexesResult = {
toDrop: string[];
toCreate: unknown[];
};
export type MongoIndexSyncResult = {
modelName: string;
collectionName: string;
toDrop: string[];
toCreate: unknown[];
cleanupReport: MongoIndexCleanupReport;
};
export type MongoIndexDescription = {
name?: string;
key?: Record<string, unknown>;
unique?: boolean;
sparse?: boolean;
expireAfterSeconds?: number;
partialFilterExpression?: unknown;
collation?: unknown;
weights?: Record<string, unknown>;
textIndexVersion?: number;
};
export type MongoIndexCleanupAction = 'drop' | 'skip_missing' | 'skip_mismatch' | 'error';
export type MongoIndexCleanupReportItem = {
collectionName: string;
indexName: string;
action: MongoIndexCleanupAction;
applied: boolean;
reason: string;
error?: string;
};
export type MongoIndexCleanupReport = {
apply: boolean;
items: MongoIndexCleanupReportItem[];
};
export type MongoIndexCleanupSummary = {
total: number;
dropped: number;
droppable: number;
skippedMissing: number;
skippedMismatch: number;
errors: number;
};
type SyncModelIndexesParams = {
model: Model<any>;
logger?: MongoIndexLogger;
};
/**
* MongoDB 索引管理入口。
*
* 每个 model 固定执行安全同步：补建当前 Schema 索引，再删除该 Schema 明确登记且
* name + key 匹配的历史索引。Schema 外未知索引只记录不删除，以保护客户自建索引。
*/
export class MongoIndexManager {
private static modelIndexTasks = new Map<Model<any>, Promise<MongoIndexSyncResult>>();
private static getCollectionName(model: Model<any>) {
return model.collection.collectionName;
}
/**
* 只计算当前 Schema 和数据库索引的差异，不创建也不删除任何索引。
*
* `toDrop` 仅表示 Mongoose 认为 Schema 外存在的索引，不能直接作为删除清单。
*/
static async inspectModelIndexes(
model: Model<any>
): Promise<Pick<MongoIndexSyncResult, 'modelName' | 'collectionName' | 'toDrop' | 'toCreate'>> {
const diff = (await model.diffIndexes({
indexOptionsToCreate: true
})) as MongooseDiffIndexesResult;
return {
modelName: model.modelName,
collectionName: MongoIndexManager.getCollectionName(model),
toDrop: diff.toDrop,
toCreate: diff.toCreate
};
}
/**
* 主动同步单个 Model 的索引。
*
* 当前索引必须先创建成功，之后才会清理 Schema 本地登记的废弃索引。同一进程内
* 针对同一个 Model 的并发调用复用进行中的任务，完成后允许重连或热加载再次检查。
*/
static async syncModelIndexes(params: SyncModelIndexesParams): Promise<MongoIndexSyncResult> {
const existingTask = MongoIndexManager.modelIndexTasks.get(params.model);
if (existingTask) {
return existingTask;
}
const task = MongoIndexManager.syncModelIndexesInner(params);
MongoIndexManager.modelIndexTasks.set(params.model, task);
try {
return await task;
} finally {
if (MongoIndexManager.modelIndexTasks.get(params.model) === task) {
MongoIndexManager.modelIndexTasks.delete(params.model);
}
}
}
private static async syncModelIndexesInner({
model,
logger = defaultLogger
}: SyncModelIndexesParams): Promise<MongoIndexSyncResult> {
const inspection = await MongoIndexManager.inspectModelIndexes(model);
if (inspection.toDrop.length > 0) {
logger.warn('Detected MongoDB indexes not declared by FastGPT schema', {
collectionName: inspection.collectionName,
indexNames: inspection.toDrop
});
}
await model.createIndexes({ background: true });
const cleanupReport = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: true,
logger
});
const result: MongoIndexSyncResult = {
...inspection,
cleanupReport
};
const cleanupSummary = MongoIndexManager.summarizeCleanupReport(cleanupReport);
if (inspection.toCreate.length > 0 || cleanupSummary.dropped > 0) {
logger.info('MongoDB indexes synchronized', {
collectionName: inspection.collectionName,
created: inspection.toCreate.length,
dropped: cleanupSummary.dropped
});
}
return result;
}
/**
* 清理当前 Model 所属 Schema 明确登记的废弃索引。
*
* 只有 name 与 key 匹配时才允许删除；key 不匹配或未知索引均保留。text 索引会
* 兼容 MongoDB 返回的 `_fts/_ftsx` 形态。`apply=false` 仅供诊断入口复用，启动
* 同步固定传 true。
*/
static async cleanupModelDeprecatedIndexes({
model,
apply,
logger
}: {
model: Model<any>;
apply: boolean;
logger?: MongoIndexLogger;
}): Promise<MongoIndexCleanupReport> {
const collectionName = MongoIndexManager.getCollectionName(model);
const definitions = getDeprecatedIndexes(model.schema);
const items: MongoIndexCleanupReportItem[] = [];
if (definitions.length === 0) {
return { apply, items };
}
for (const definition of definitions) {
try {
const currentIndexes = (await model.collection.indexes().catch((error) => {
if (MongoIndexManager.isNamespaceNotFoundError(error)) {
return [];
}
throw error;
})) as MongoIndexDescription[];
const targetIndex = currentIndexes.find((index) => index.name === definition.indexName);
if (!targetIndex) {
const item = MongoIndexManager.buildCleanupItem({
collectionName,
definition,
action: 'skip_missing',
reason: 'Deprecated index does not exist'
});
items.push(item);
continue;
}
if (!MongoIndexManager.isDeprecatedIndexMatched({ definition, index: targetIndex })) {
const item = MongoIndexManager.buildCleanupItem({
collectionName,
definition,
action: 'skip_mismatch',
reason: 'Index definition does not match Schema declaration'
});
logger?.warn('Deprecated MongoDB index definition mismatched', {
collectionName,
indexName: definition.indexName
});
items.push(item);
continue;
}
if (apply) {
try {
await model.collection.dropIndex(definition.indexName);
} catch (error) {
if (MongoIndexManager.isIndexNotFoundError(error)) {
const item = MongoIndexManager.buildCleanupItem({
collectionName,
definition,
action: 'skip_missing',
reason: 'Deprecated index was already removed'
});
items.push(item);
continue;
}
throw error;
}
}
const item = MongoIndexManager.buildCleanupItem({
collectionName,
definition,
action: 'drop',
applied: apply,
reason: apply ? 'Deprecated index dropped' : 'Deprecated index can be dropped'
});
items.push(item);
} catch (error) {
const item = MongoIndexManager.buildCleanupItem({
collectionName,
definition,
action: 'error',
reason: 'Failed to inspect or cleanup deprecated index',
error: MongoIndexManager.getErrorMessage(error)
});
logger?.error('Failed to cleanup deprecated MongoDB index', {
collectionName,
indexName: definition.indexName,
error: item.error
});
items.push(item);
}
}
return { apply, items };
}
static summarizeCleanupReport(report: MongoIndexCleanupReport): MongoIndexCleanupSummary {
return report.items.reduce<MongoIndexCleanupSummary>(
(summary, item) => {
summary.total += 1;
if (item.action === 'drop' && item.applied) {
summary.dropped += 1;
} else if (item.action === 'drop') {
summary.droppable += 1;
} else if (item.action === 'skip_missing') {
summary.skippedMissing += 1;
} else if (item.action === 'skip_mismatch') {
summary.skippedMismatch += 1;
} else if (item.action === 'error') {
summary.errors += 1;
}
return summary;
},
{
total: 0,
dropped: 0,
droppable: 0,
skippedMissing: 0,
skippedMismatch: 0,
errors: 0
}
);
}
static formatCleanupReport(report: MongoIndexCleanupReport) {
const lines = [
`MongoDB deprecated index cleanup ${report.apply ? 'apply' : 'dry-run'} report`,
`Total: ${report.items.length}`
];
for (const item of report.items) {
lines.push(
[
`- [${item.action}]`,
item.applied ? 'applied' : 'not-applied',
`${item.collectionName}.${item.indexName}`,
`reason=${item.reason}`,
item.error ? `error=${item.error}` : undefined
]
.filter(Boolean)
.join(' ')
);
}
return lines.join('\n');
}
private static normalizeForCompare(
value: unknown,
{ sortObjectKeys }: { sortObjectKeys: boolean }
): unknown {
if (Array.isArray(value)) {
return value.map((item) => MongoIndexManager.normalizeForCompare(item, { sortObjectKeys }));
}
if (value && typeof value === 'object') {
const keys = Object.keys(value);
const orderedKeys = sortObjectKeys ? keys.sort() : keys;
return orderedKeys.reduce<Record<string, unknown>>((result, key) => {
result[key] = MongoIndexManager.normalizeForCompare(Reflect.get(value, key), {
sortObjectKeys
});
return result;
}, {});
}
return value;
}
private static isSameValue(
left: unknown,
right: unknown,
{ sortObjectKeys = true }: { sortObjectKeys?: boolean } = {}
) {
return (
JSON.stringify(MongoIndexManager.normalizeForCompare(left, { sortObjectKeys })) ===
JSON.stringify(MongoIndexManager.normalizeForCompare(right, { sortObjectKeys }))
);
}
/**
* 判断声明 key 是否为 text 索引（字段值包含 `"text"`）。
*
* MongoDB 创建后会把 text 索引 key 改写为 `{ _fts: "text", _ftsx: 1 }`，
* 因此清理匹配不能直接用声明 key 和 listIndexes 的 key 做对象相等比较。
*/
private static isTextIndexDefinition(key: DeprecatedMongoIndexDefinition['key']) {
return Object.values(key as Record<string, unknown>).some((value) => value === 'text');
}
/** 判断 listIndexes 返回的索引是否为 text 索引。 */
private static isStoredTextIndex(index: MongoIndexDescription) {
return (
index.key?._fts === 'text' ||
typeof index.textIndexVersion === 'number' ||
(index.weights != null && typeof index.weights === 'object')
);
}
/**
* 从废弃声明中提取 text 字段列表，保持声明顺序。
* 非 text 前缀/后缀字段暂不参与匹配，当前 FastGPT 未使用混合 text 复合索引。
*/
private static getDeclaredTextFields(key: DeprecatedMongoIndexDefinition['key']) {
return Object.entries(key as Record<string, unknown>)
.filter(([, value]) => value === 'text')
.map(([field]) => field);
}
/**
* 从数据库索引描述中提取 text 字段列表。
* 优先使用 `weights`（字段 -> 权重），这是 listIndexes 暴露业务字段的权威来源。
*/
private static getStoredTextFields(index: MongoIndexDescription) {
if (index.weights && typeof index.weights === 'object') {
return Object.keys(index.weights);
}
return [];
}
/**
* 废弃索引删除前的安全校验：name 已由调用方定位，这里只校验 key。
*
* - 普通索引：key 对象按声明顺序精确相等
* - text 索引：声明字段集合与 weights 字段集合相等（忽略 `_fts/_ftsx` 形态差异）
* - options（unique/sparse/TTL 等）不参与匹配，避免重复声明成本；同名同 key 下
* option 冲突极少，需由声明方自行确认
*/
private static isDeprecatedIndexMatched({
definition,
index
}: {
definition: DeprecatedMongoIndexDefinition;
index: MongoIndexDescription;
}) {
const definitionIsText = MongoIndexManager.isTextIndexDefinition(definition.key);
const storedIsText = MongoIndexManager.isStoredTextIndex(index);
if (definitionIsText || storedIsText) {
if (!definitionIsText || !storedIsText) {
return false;
}
// weights 字段顺序不一定等于声明顺序，按字段名集合比较即可
return MongoIndexManager.isSameValue(
[...MongoIndexManager.getDeclaredTextFields(definition.key)].sort(),
[...MongoIndexManager.getStoredTextFields(index)].sort(),
{ sortObjectKeys: false }
);
}
return MongoIndexManager.isSameValue(index.key, definition.key, { sortObjectKeys: false });
}
private static buildCleanupItem({
collectionName,
definition,
action,
applied = false,
reason,
error
}: {
collectionName: string;
definition: DeprecatedMongoIndexDefinition;
action: MongoIndexCleanupAction;
applied?: boolean;
reason: string;
error?: string;
}): MongoIndexCleanupReportItem {
return {
collectionName,
indexName: definition.indexName,
action,
applied,
reason,
error
};
}
private static getErrorMessage(error: unknown) {
if (error instanceof Error) {
return error.message;
}
return String(error);
}
private static isNamespaceNotFoundError(error: unknown) {
if (typeof error !== 'object' || error === null) {
return false;
}
const codeName = Reflect.get(error, 'codeName');
const message = Reflect.get(error, 'message');
return (
codeName === 'NamespaceNotFound' ||
(typeof message === 'string' && message.includes('ns does not exist'))
);
}
private static isIndexNotFoundError(error: unknown) {
if (typeof error !== 'object' || error === null) {
return false;
}
const code = Reflect.get(error, 'code');
const codeName = Reflect.get(error, 'codeName');
return code === 27 || codeName === 'IndexNotFound';
}
}
packages/service/common/mongo/schemaIndexes.ts 0 → 100644
import type { IndexDefinition, IndexOptions, Schema } from 'mongoose';
const deprecatedMongoIndexesKey = Symbol.for('fastgpt.mongo.deprecatedIndexes');
export type DeprecatedMongoIndexOptions = Pick<
IndexOptions,
'unique' | 'sparse' | 'expireAfterSeconds' | 'partialFilterExpression' | 'collation'
>;
export type DeprecatedMongoIndexDefinition = {
indexName: string;
key: IndexDefinition;
options?: DeprecatedMongoIndexOptions;
};
export type DefineMongoIndexOptions = {
key: IndexDefinition;
options?: IndexOptions;
deprecated?: true;
};
/**
* 统一声明当前索引和 FastGPT 明确废弃的历史索引。
*
* `deprecated` 默认是 `false`，当前索引直接代理 `Schema.index()`；显式设置为
* `true` 时只登记清理元数据，不能继续写入 Mongoose Schema，否则启动同步会先
* 重新创建该索引。废弃索引未显式命名时，按 MongoDB 的默认规则从 key 推导名称。
*/
export const defineIndex = (
schema: Schema,
{ key, options, deprecated }: DefineMongoIndexOptions
) => {
if (deprecated !== true) {
schema.index(key, options);
return;
}
const registeredIndexes = getDeprecatedIndexes(schema);
const indexName =
options?.name ??
Object.entries(key)
.map(([field, order]) => `${field}_${order}`)
.join('_');
const duplicateIndexName = registeredIndexes.some((index) => index.indexName === indexName);
if (duplicateIndexName) {
throw new Error(`Duplicate deprecated MongoDB index declaration: ${indexName}`);
}
const deprecatedOptions: DeprecatedMongoIndexOptions = {
unique: options?.unique,
sparse: options?.sparse,
expireAfterSeconds: options?.expireAfterSeconds,
partialFilterExpression: options?.partialFilterExpression,
collation: options?.collation
};
const hasDeprecatedOptions = Object.values(deprecatedOptions).some(
(value) => value !== undefined
);
Reflect.set(schema, deprecatedMongoIndexesKey, [
...registeredIndexes,
{
indexName,
key,
options: hasDeprecatedOptions ? deprecatedOptions : undefined
}
] satisfies DeprecatedMongoIndexDefinition[]);
};
/** 读取某个 Schema 自身登记的废弃索引，不聚合其他集合或全局清单。 */
export const getDeprecatedIndexes = (schema: Schema): readonly DeprecatedMongoIndexDefinition[] => {
const indexes: unknown = Reflect.get(schema, deprecatedMongoIndexesKey);
return Array.isArray(indexes) ? indexes : [];
};
packages/service/common/s3/accessLink/downloadAlias/schema.ts
import { getLogger, LogCategories } from '../../../logger'; import { getLogger, LogCategories } from '../../../logger';
import { getMongoModel, Schema } from '../../../mongo'; import { defineIndex, getMongoModel, Schema } from '../../../mongo';
import type { S3DownloadAliasType } from '../type'; import type { S3DownloadAliasType } from '../type';
export const S3DownloadAliasCollectionName = 's3_download_aliases'; export const S3DownloadAliasCollectionName = 's3_download_aliases';
...@@ -44,14 +44,21 @@ const S3DownloadAliasMongoSchema = new Schema({ ...@@ -44,14 +44,21 @@ const S3DownloadAliasMongoSchema = new Schema({
disabledAt: Date disabledAt: Date
}); });
try { defineIndex(S3DownloadAliasMongoSchema, {
S3DownloadAliasMongoSchema.index({ aliasId: 1 }, { unique: true }); key: { aliasId: 1 },
S3DownloadAliasMongoSchema.index({ aliasKey: 1 }, { unique: true }); options: { unique: true }
S3DownloadAliasMongoSchema.index({ purgeAt: 1 }, { expireAfterSeconds: 0 }); });
S3DownloadAliasMongoSchema.index({ bucketName: 1, objectKey: 1 }); defineIndex(S3DownloadAliasMongoSchema, {
} catch (error) { key: { aliasKey: 1 },
logger.error('Failed to build S3 download alias indexes', { error }); options: { unique: true }
} });
defineIndex(S3DownloadAliasMongoSchema, {
key: { purgeAt: 1 },
options: { expireAfterSeconds: 0 }
});
defineIndex(S3DownloadAliasMongoSchema, {
key: { bucketName: 1, objectKey: 1 }
});
export const MongoS3DownloadAlias = getMongoModel<S3DownloadAliasType>( export const MongoS3DownloadAlias = getMongoModel<S3DownloadAliasType>(
S3DownloadAliasCollectionName, S3DownloadAliasCollectionName,
......
packages/service/common/s3/accessLink/uploadSession/schema.ts
import { getLogger, LogCategories } from '../../../logger'; import { getLogger, LogCategories } from '../../../logger';
import { getMongoModel, Schema } from '../../../mongo'; import { defineIndex, getMongoModel, Schema } from '../../../mongo';
import type { S3UploadSessionType } from '../type'; import type { S3UploadSessionType } from '../type';
export const S3UploadSessionCollectionName = 's3_upload_sessions'; export const S3UploadSessionCollectionName = 's3_upload_sessions';
...@@ -42,13 +42,17 @@ const S3UploadSessionMongoSchema = new Schema({ ...@@ -42,13 +42,17 @@ const S3UploadSessionMongoSchema = new Schema({
revokedAt: Date revokedAt: Date
}); });
try { defineIndex(S3UploadSessionMongoSchema, {
S3UploadSessionMongoSchema.index({ tokenHash: 1 }, { unique: true }); key: { tokenHash: 1 },
S3UploadSessionMongoSchema.index({ expiresAt: 1 }, { expireAfterSeconds: 0 }); options: { unique: true }
S3UploadSessionMongoSchema.index({ bucketName: 1, objectKey: 1 }); });
} catch (error) { defineIndex(S3UploadSessionMongoSchema, {
logger.error('Failed to build S3 upload session indexes', { error }); key: { expiresAt: 1 },
} options: { expireAfterSeconds: 0 }
});
defineIndex(S3UploadSessionMongoSchema, {
key: { bucketName: 1, objectKey: 1 }
});
export const MongoS3UploadSession = getMongoModel<S3UploadSessionType>( export const MongoS3UploadSession = getMongoModel<S3UploadSessionType>(
S3UploadSessionCollectionName, S3UploadSessionCollectionName,
......
packages/service/common/s3/models/ttl.ts
import { Schema, getMongoModel } from '../../mongo'; import { defineIndex, Schema, getMongoModel } from '../../mongo';
import { type S3TtlSchemaType } from '@fastgpt/global/common/file/s3TTL/type'; import { type S3TtlSchemaType } from '@fastgpt/global/common/file/s3TTL/type';
const collectionName = 's3_ttls'; const collectionName = 's3_ttls';
...@@ -18,7 +18,7 @@ const S3TTLSchema = new Schema({ ...@@ -18,7 +18,7 @@ const S3TTLSchema = new Schema({
} }
}); });
S3TTLSchema.index({ expiredTime: 1 }); defineIndex(S3TTLSchema, { key: { expiredTime: 1 } });
S3TTLSchema.index({ bucketName: 1, minioKey: 1 }); defineIndex(S3TTLSchema, { key: { bucketName: 1, minioKey: 1 } });
export const MongoS3TTL = getMongoModel<S3TtlSchemaType>(collectionName, S3TTLSchema); export const MongoS3TTL = getMongoModel<S3TtlSchemaType>(collectionName, S3TTLSchema);
packages/service/common/system/config/schema.ts
import { type SystemConfigsType } from '@fastgpt/global/common/system/config/type'; import { type SystemConfigsType } from '@fastgpt/global/common/system/config/type';
import { connectionMongo, getMongoModel, type Model } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel, type Model } from '../../../common/mongo';
import { SystemConfigsTypeMap } from '@fastgpt/global/common/system/config/constants'; import { SystemConfigsTypeMap } from '@fastgpt/global/common/system/config/constants';
import { getLogger, LogCategories } from '../../logger'; import { getLogger, LogCategories } from '../../logger';
...@@ -23,11 +23,7 @@ const systemConfigSchema = new Schema({ ...@@ -23,11 +23,7 @@ const systemConfigSchema = new Schema({
} }
}); });
try { defineIndex(systemConfigSchema, { key: { type: 1 } });
systemConfigSchema.index({ type: 1 });
} catch (error) {
logger.error('Failed to build system config indexes', { error });
}
export const MongoSystemConfigs = getMongoModel<SystemConfigsType>( export const MongoSystemConfigs = getMongoModel<SystemConfigsType>(
collectionName, collectionName,
......
packages/service/common/system/countLimit/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import type { CountLimitType } from './type'; import type { CountLimitType } from './type';
import { getLogger, LogCategories } from '../../logger'; import { getLogger, LogCategories } from '../../logger';
...@@ -26,11 +26,13 @@ const CountLimitSchema = new Schema({ ...@@ -26,11 +26,13 @@ const CountLimitSchema = new Schema({
} }
}); });
try { defineIndex(CountLimitSchema, {
CountLimitSchema.index({ type: 1, key: 1 }, { unique: true }); key: { type: 1, key: 1 },
CountLimitSchema.index({ createTime: 1 }, { expireAfterSeconds: 60 * 60 * 24 * 30 }); // ttl 30天 options: { unique: true }
} catch (error) { });
logger.error('Failed to build count limit indexes', { error }); defineIndex(CountLimitSchema, {
} key: { createTime: 1 },
options: { expireAfterSeconds: 60 * 60 * 24 * 30 }
}); // ttl 30天
export const MongoCountLimit = getMongoModel<CountLimitType>(collectionName, CountLimitSchema); export const MongoCountLimit = getMongoModel<CountLimitType>(collectionName, CountLimitSchema);
packages/service/common/system/frequencyLimit/schema.ts
import { getMongoModel, Schema } from '../../mongo'; import { defineIndex, getMongoModel, Schema } from '../../mongo';
import type { FrequencyLimitSchemaType } from './type'; import type { FrequencyLimitSchemaType } from './type';
const FrequencyLimitSchema = new Schema({ const FrequencyLimitSchema = new Schema({
...@@ -16,10 +16,11 @@ const FrequencyLimitSchema = new Schema({ ...@@ -16,10 +16,11 @@ const FrequencyLimitSchema = new Schema({
} }
}); });
try { defineIndex(FrequencyLimitSchema, { key: { eventId: 1, expiredTime: 1 } });
FrequencyLimitSchema.index({ eventId: 1, expiredTime: 1 }); defineIndex(FrequencyLimitSchema, {
FrequencyLimitSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 0 }); key: { expiredTime: 1 },
} catch (error) {} options: { expireAfterSeconds: 0 }
});
export const MongoFrequencyLimit = getMongoModel<FrequencyLimitSchemaType>( export const MongoFrequencyLimit = getMongoModel<FrequencyLimitSchemaType>(
'frequency_limit', 'frequency_limit',
......
packages/service/common/system/log/schema.ts
import { getMongoLogModel as getMongoModel, Schema } from '../../../common/mongo'; import { defineIndex, getMongoLogModel as getMongoModel, Schema } from '../../../common/mongo';
import { type SystemLogType } from './type'; import { type SystemLogType } from './type';
import { LogLevelEnum } from './constant'; import { LogLevelEnum } from './constant';
...@@ -22,8 +22,11 @@ export const getMongoLog = () => { ...@@ -22,8 +22,11 @@ export const getMongoLog = () => {
metadata: Object metadata: Object
}); });
SystemLogSchema.index({ time: 1 }, { expires: '15d' }); defineIndex(SystemLogSchema, {
SystemLogSchema.index({ level: 1 }); key: { time: 1 },
options: { expires: '15d' }
});
defineIndex(SystemLogSchema, { key: { level: 1 } });
return getMongoModel<SystemLogType>(LogCollectionName, SystemLogSchema); return getMongoModel<SystemLogType>(LogCollectionName, SystemLogSchema);
}; };
packages/service/common/system/timerLock/schema.ts
import { connectionMongo, getMongoModel } from '../../mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type TimerLockSchemaType } from './type'; import { type TimerLockSchemaType } from './type';
import { getLogger, LogCategories } from '../../logger'; import { getLogger, LogCategories } from '../../logger';
...@@ -9,8 +9,7 @@ const logger = getLogger(LogCategories.INFRA.MONGO); ...@@ -9,8 +9,7 @@ const logger = getLogger(LogCategories.INFRA.MONGO);
const TimerLockSchema = new Schema({ const TimerLockSchema = new Schema({
timerId: { timerId: {
type: String, type: String,
required: true, required: true
unique: true
}, },
expiredTime: { expiredTime: {
type: Date, type: Date,
...@@ -18,10 +17,13 @@ const TimerLockSchema = new Schema({ ...@@ -18,10 +17,13 @@ const TimerLockSchema = new Schema({
} }
}); });
try { defineIndex(TimerLockSchema, {
TimerLockSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 5 }); key: { timerId: 1 },
} catch (error) { options: { unique: true }
logger.error('Failed to build timer lock indexes', { error }); });
} defineIndex(TimerLockSchema, {
key: { expiredTime: 1 },
options: { expireAfterSeconds: 5 }
});
export const MongoTimerLock = getMongoModel<TimerLockSchemaType>(collectionName, TimerLockSchema); export const MongoTimerLock = getMongoModel<TimerLockSchemaType>(collectionName, TimerLockSchema);
packages/service/core/ai/config/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { connectionMongo, defineIndex, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import type { SystemModelSchemaType } from '../type'; import type { SystemModelSchemaType } from '../type';
...@@ -6,8 +6,7 @@ const SystemModelSchema = new Schema( ...@@ -6,8 +6,7 @@ const SystemModelSchema = new Schema(
{ {
model: { model: {
type: String, type: String,
required: true, required: true
unique: true
}, },
metadata: { metadata: {
type: Object, type: Object,
...@@ -20,6 +19,11 @@ const SystemModelSchema = new Schema( ...@@ -20,6 +19,11 @@ const SystemModelSchema = new Schema(
} }
); );
defineIndex(SystemModelSchema, {
key: { model: 1 },
options: { unique: true }
});
export const MongoSystemModel = getMongoModel<SystemModelSchemaType>( export const MongoSystemModel = getMongoModel<SystemModelSchemaType>(
'system_models', 'system_models',
SystemModelSchema SystemModelSchema
......
packages/service/core/ai/record/schema.ts
import { getMongoLogModel, Schema } from '../../../common/mongo'; import { defineIndex, getMongoLogModel, Schema } from '../../../common/mongo';
import type { LLMRequestRecordSchemaType } from '@fastgpt/global/openapi/core/ai/api'; import type { LLMRequestRecordSchemaType } from '@fastgpt/global/openapi/core/ai/api';
import { serviceEnv } from '../../../env'; import { serviceEnv } from '../../../env';
...@@ -30,7 +30,10 @@ const LLMRequestRecordSchema = new Schema({ ...@@ -30,7 +30,10 @@ const LLMRequestRecordSchema = new Schema({
} }
}); });
LLMRequestRecordSchema.index({ teamId: 1, requestId: 1 }, { unique: true }); defineIndex(LLMRequestRecordSchema, {
key: { teamId: 1, requestId: 1 },
options: { unique: true }
});
export const MongoLLMRequestRecord = getMongoLogModel<LLMRequestRecordSchemaType>( export const MongoLLMRequestRecord = getMongoLogModel<LLMRequestRecordSchemaType>(
LLMRequestRecordCollectionName, LLMRequestRecordCollectionName,
......
packages/service/core/ai/sandbox/infrastructure/instance/schema.ts
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
* *
* 只描述本地实例记录结构，不编排 provider、归档或运行态流程。 * 只描述本地实例记录结构，不编排 provider、归档或运行态流程。
*/ */
import { connectionMongo, getMongoModel } from '../../../../../common/mongo'; import { connectionMongo, defineIndex, getMongoModel } from '../../../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import type { SandboxInstanceSchemaType } from '../../type'; import type { SandboxInstanceSchemaType } from '../../type';
import { SandboxStatusEnum, SandboxTypeEnum } from '@fastgpt/global/core/ai/sandbox/constants'; import { SandboxStatusEnum, SandboxTypeEnum } from '@fastgpt/global/core/ai/sandbox/constants';
...@@ -75,12 +75,25 @@ const SandboxInstanceSchema = new Schema({ ...@@ -75,12 +75,25 @@ const SandboxInstanceSchema = new Schema({
} }
}); });
SandboxInstanceSchema.index({ provider: 1, sandboxId: 1 }, { unique: true }); defineIndex(SandboxInstanceSchema, {
SandboxInstanceSchema.index({ sourceType: 1, sourceId: 1, chatId: 1 }); key: { provider: 1, sandboxId: 1 },
SandboxInstanceSchema.index({ sourceType: 1, status: 1, provider: 1, 'metadata.archive.state': 1 }); options: { unique: true }
SandboxInstanceSchema.index({ status: 1, lastActiveAt: 1, 'metadata.archive.state': 1 }); });
SandboxInstanceSchema.index({ 'metadata.archive.state': 1, 'metadata.archive.startedAt': 1 }); defineIndex(SandboxInstanceSchema, {
SandboxInstanceSchema.index({ 'metadata.archive.state': 1, 'metadata.archive.deleteStartedAt': 1 }); key: { sourceType: 1, sourceId: 1, chatId: 1 }
});
defineIndex(SandboxInstanceSchema, {
key: { sourceType: 1, status: 1, provider: 1, 'metadata.archive.state': 1 }
});
defineIndex(SandboxInstanceSchema, {
key: { status: 1, lastActiveAt: 1, 'metadata.archive.state': 1 }
});
defineIndex(SandboxInstanceSchema, {
key: { 'metadata.archive.state': 1, 'metadata.archive.startedAt': 1 }
});
defineIndex(SandboxInstanceSchema, {
key: { 'metadata.archive.state': 1, 'metadata.archive.deleteStartedAt': 1 }
});
/** /**
* sandbox 实例 Mongo model。 * sandbox 实例 Mongo model。
......
packages/service/core/ai/skill/model/schema.ts
import { connectionMongo, getMongoModel } from '../../../../common/mongo'; import { connectionMongo, defineIndex, getMongoModel } from '../../../../common/mongo';
import { import {
agentSkillsCollectionName, agentSkillsCollectionName,
agentSkillsVersionCollectionName, agentSkillsVersionCollectionName,
...@@ -121,13 +121,17 @@ const AgentSkillsSchema = new Schema({ ...@@ -121,13 +121,17 @@ const AgentSkillsSchema = new Schema({
}); });
// 名称和描述用于列表页搜索。 // 名称和描述用于列表页搜索。
AgentSkillsSchema.index({ teamId: 1, name: 'text', description: 'text' }); defineIndex(AgentSkillsSchema, {
key: { teamId: 1, name: 'text', description: 'text' }
});
// 列表页按来源、团队、删除状态和创建时间过滤排序。 // 列表页按来源、团队、删除状态和创建时间过滤排序。
AgentSkillsSchema.index({ source: 1, teamId: 1, deleteTime: 1, createTime: -1 }); defineIndex(AgentSkillsSchema, {
key: { source: 1, teamId: 1, deleteTime: 1, createTime: -1 }
});
// 分类筛选。 // 分类筛选。
AgentSkillsSchema.index({ category: 1 }); defineIndex(AgentSkillsSchema, { key: { category: 1 } });
// 文件夹树查询。 // 文件夹树查询：findSkillAndAllChildren 按 teamId + parentId + deleteTime 逐层查子节点。
AgentSkillsSchema.index({ teamId: 1, parentId: 1, deleteTime: 1 }); defineIndex(AgentSkillsSchema, { key: { teamId: 1, parentId: 1, deleteTime: 1 } });
export const MongoAgentSkills = getMongoModel<MongoAgentSkillSchemaType>( export const MongoAgentSkills = getMongoModel<MongoAgentSkillSchemaType>(
agentSkillsCollectionName, agentSkillsCollectionName,
......
packages/service/core/ai/skill/version/schema.ts
import { connectionMongo, getMongoModel } from '../../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../../common/mongo';
import { import {
agentSkillsCollectionName, agentSkillsCollectionName,
agentSkillsVersionCollectionName agentSkillsVersionCollectionName
...@@ -52,7 +52,9 @@ const AgentSkillsVersionSchema = new Schema({ ...@@ -52,7 +52,9 @@ const AgentSkillsVersionSchema = new Schema({
}); });
// 版本列表按 skillId 查询并按创建时间倒序展示。 // 版本列表按 skillId 查询并按创建时间倒序展示。
AgentSkillsVersionSchema.index({ skillId: 1, createdAt: -1, _id: -1 }); defineIndex(AgentSkillsVersionSchema, {
key: { skillId: 1, createdAt: -1, _id: -1 }
});
export const MongoAgentSkillsVersion = getMongoModel<AgentSkillsVersionSchemaType>( export const MongoAgentSkillsVersion = getMongoModel<AgentSkillsVersionSchemaType>(
agentSkillsVersionCollectionName, agentSkillsVersionCollectionName,
......
packages/service/core/app/evaluation/evalItemSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { EvaluationCollectionName } from './evalSchema'; import { EvaluationCollectionName } from './evalSchema';
import { import {
EvaluationStatusEnum, EvaluationStatusEnum,
...@@ -48,7 +48,7 @@ const EvalItemSchema = new Schema({ ...@@ -48,7 +48,7 @@ const EvalItemSchema = new Schema({
errorMessage: String errorMessage: String
}); });
EvalItemSchema.index({ evalId: 1, status: 1 }); defineIndex(EvalItemSchema, { key: { evalId: 1, status: 1 } });
export const MongoEvalItem = getMongoModel<EvalItemSchemaType>( export const MongoEvalItem = getMongoModel<EvalItemSchemaType>(
EvalItemCollectionName, EvalItemCollectionName,
......
packages/service/core/app/evaluation/evalSchema.ts
...@@ -2,7 +2,7 @@ import { ...@@ -2,7 +2,7 @@ import {
TeamCollectionName, TeamCollectionName,
TeamMemberCollectionName TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant'; } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { AppCollectionName } from '../schema'; import { AppCollectionName } from '../schema';
import type { EvaluationSchemaType } from '@fastgpt/global/core/app/evaluation/type'; import type { EvaluationSchemaType } from '@fastgpt/global/core/app/evaluation/type';
import { UsageCollectionName } from '../../../support/wallet/usage/constants'; import { UsageCollectionName } from '../../../support/wallet/usage/constants';
...@@ -49,7 +49,7 @@ const EvaluationSchema = new Schema({ ...@@ -49,7 +49,7 @@ const EvaluationSchema = new Schema({
errorMessage: String errorMessage: String
}); });
EvaluationSchema.index({ teamId: 1 }); defineIndex(EvaluationSchema, { key: { teamId: 1 } });
export const MongoEvaluation = getMongoModel<EvaluationSchemaType>( export const MongoEvaluation = getMongoModel<EvaluationSchemaType>(
EvaluationCollectionName, EvaluationCollectionName,
......
packages/service/core/app/logs/chatLogsSchema.ts
import type { AppChatLogSchema } from '@fastgpt/global/core/app/logs/type'; import type { AppChatLogSchema } from '@fastgpt/global/core/app/logs/type';
import { getMongoLogModel, Schema } from '../../../common/mongo'; import { defineIndex, getMongoLogModel, Schema } from '../../../common/mongo';
import { AppCollectionName } from '../schema'; import { AppCollectionName } from '../schema';
export const ChatLogCollectionName = 'app_chat_logs'; export const ChatLogCollectionName = 'app_chat_logs';
...@@ -69,17 +69,21 @@ const ChatLogSchema = new Schema({ ...@@ -69,17 +69,21 @@ const ChatLogSchema = new Schema({
}); });
// Get chart data // Get chart data
ChatLogSchema.index({ teamId: 1, appId: 1, source: 1, updateTime: -1 }); defineIndex(ChatLogSchema, {
key: { teamId: 1, appId: 1, source: 1, updateTime: -1 }
});
// Get chart data isFirstChat // Get chart data isFirstChat
ChatLogSchema.index({ isFirstChat: 1, teamId: 1, appId: 1, source: 1, createTime: -1 }); defineIndex(ChatLogSchema, {
key: { isFirstChat: 1, teamId: 1, appId: 1, source: 1, createTime: -1 }
});
// Get userStats // Get userStats
ChatLogSchema.index({ teamId: 1, appId: 1, userId: 1 }); defineIndex(ChatLogSchema, { key: { teamId: 1, appId: 1, userId: 1 } });
// Admin get chat form data - optimized for aggregation with appId/chatId grouping // Admin get chat form data - optimized for aggregation with appId/chatId grouping
ChatLogSchema.index({ createTime: -1, appId: 1, chatId: 1 }); defineIndex(ChatLogSchema, { key: { createTime: -1, appId: 1, chatId: 1 } });
// Init shell // Init shell
ChatLogSchema.index({ teamId: 1, appId: 1, chatId: 1 }); defineIndex(ChatLogSchema, { key: { teamId: 1, appId: 1, chatId: 1 } });
export const MongoAppChatLog = getMongoLogModel<AppChatLogSchema>( export const MongoAppChatLog = getMongoLogModel<AppChatLogSchema>(
ChatLogCollectionName, ChatLogCollectionName,
......
packages/service/core/app/logs/logkeysSchema.ts
import type { AppLogKeysSchemaType } from '@fastgpt/global/core/app/logs/type'; import type { AppLogKeysSchemaType } from '@fastgpt/global/core/app/logs/type';
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { AppCollectionName } from '../schema'; import { AppCollectionName } from '../schema';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
...@@ -24,7 +24,7 @@ const AppLogKeysSchema = new Schema({ ...@@ -24,7 +24,7 @@ const AppLogKeysSchema = new Schema({
} }
}); });
AppLogKeysSchema.index({ teamId: 1, appId: 1 }); defineIndex(AppLogKeysSchema, { key: { teamId: 1, appId: 1 } });
export const MongoAppLogKeys = getMongoModel<AppLogKeysSchemaType>( export const MongoAppLogKeys = getMongoModel<AppLogKeysSchemaType>(
AppLogKeysCollectionEnum, AppLogKeysCollectionEnum,
......
packages/service/core/app/record/schema.ts
...@@ -2,7 +2,7 @@ import { ...@@ -2,7 +2,7 @@ import {
TeamCollectionName, TeamCollectionName,
TeamMemberCollectionName TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant'; } from '@fastgpt/global/support/user/team/constant';
import { getMongoModel, Schema } from '../../../common/mongo'; import { defineIndex, getMongoModel, Schema } from '../../../common/mongo';
import { AppCollectionName } from '../schema'; import { AppCollectionName } from '../schema';
import type { AppRecordType } from './type'; import type { AppRecordType } from './type';
...@@ -35,9 +35,12 @@ const AppRecordSchema = new Schema( ...@@ -35,9 +35,12 @@ const AppRecordSchema = new Schema(
} }
); );
AppRecordSchema.index({ tmbId: 1, lastUsedTime: -1 }); // 查询用户最近使用的应用 defineIndex(AppRecordSchema, { key: { tmbId: 1, lastUsedTime: -1 } }); // 查询用户最近使用的应用
AppRecordSchema.index({ tmbId: 1, appId: 1 }, { unique: true }); // 防止重复记录 defineIndex(AppRecordSchema, {
AppRecordSchema.index({ teamId: 1, appId: 1 }); // 用于清理权限失效的记录 key: { tmbId: 1, appId: 1 },
options: { unique: true }
}); // 防止重复记录
defineIndex(AppRecordSchema, { key: { teamId: 1, appId: 1 } }); // 用于清理权限失效的记录
export const MongoAppRecord = getMongoModel<AppRecordType>( export const MongoAppRecord = getMongoModel<AppRecordType>(
AppRecordCollectionName, AppRecordCollectionName,
......
packages/service/core/app/schema.ts
import { AppTypeEnum } from '@fastgpt/global/core/app/constants'; import { AppTypeEnum } from '@fastgpt/global/core/app/constants';
import { Schema, getMongoModel } from '../../common/mongo'; import { defineIndex, Schema, getMongoModel } from '../../common/mongo';
import type { AppSchemaType as AppType } from '@fastgpt/global/core/app/type'; import type { AppSchemaType as AppType } from '@fastgpt/global/core/app/type';
import { import {
TeamCollectionName, TeamCollectionName,
...@@ -134,24 +134,26 @@ const AppSchema = new Schema( ...@@ -134,24 +134,26 @@ const AppSchema = new Schema(
} }
); );
AppSchema.index({ teamId: 1, updateTime: -1 }); defineIndex(AppSchema, { key: { teamId: 1, updateTime: -1 } });
AppSchema.index({ teamId: 1, type: 1 }); defineIndex(AppSchema, { key: { teamId: 1, type: 1 } });
AppSchema.index({ teamId: 1, deleteTime: 1, 'resourceRefs.skillIds': 1 }); defineIndex(AppSchema, {
key: { teamId: 1, deleteTime: 1, 'resourceRefs.skillIds': 1 }
});
// Schedule // Schedule
AppSchema.index( defineIndex(AppSchema, {
{ scheduledTriggerConfig: 1, scheduledTriggerNextTime: -1 }, key: { scheduledTriggerConfig: 1, scheduledTriggerNextTime: -1 },
{ options: {
partialFilterExpression: { partialFilterExpression: {
scheduledTriggerConfig: { $exists: true } scheduledTriggerConfig: { $exists: true }
} }
} }
); });
// Admin count // Admin count
AppSchema.index({ type: 1 }); defineIndex(AppSchema, { key: { type: 1 } });
AppSchema.index({ deleteTime: 1 }); defineIndex(AppSchema, { key: { deleteTime: 1 } });
// Admin search // Admin search
AppSchema.index({ name: 1 }); defineIndex(AppSchema, { key: { name: 1 } });
export const MongoApp = getMongoModel<AppType>(AppCollectionName, AppSchema); export const MongoApp = getMongoModel<AppType>(AppCollectionName, AppSchema);
packages/service/core/app/templates/templateSchema.ts
import { type AppTemplateSchemaType } from '@fastgpt/global/core/app/type'; import { type AppTemplateSchemaType } from '@fastgpt/global/core/app/type';
import { connectionMongo, getMongoModel } from '../../../common/mongo/index'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo/index';
import { UserTagsSchema } from '@fastgpt/global/support/user/type'; import { UserTagsSchema } from '@fastgpt/global/support/user/type';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
...@@ -39,7 +39,7 @@ const AppTemplateSchema = new Schema({ ...@@ -39,7 +39,7 @@ const AppTemplateSchema = new Schema({
workflow: Object workflow: Object
}); });
AppTemplateSchema.index({ templateId: 1 }); defineIndex(AppTemplateSchema, { key: { templateId: 1 } });
export const MongoAppTemplate = getMongoModel<AppTemplateSchemaType>( export const MongoAppTemplate = getMongoModel<AppTemplateSchemaType>(
collectionName, collectionName,
......
packages/service/core/app/version/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type AppVersionSchemaType } from '@fastgpt/global/core/app/version/type'; import { type AppVersionSchemaType } from '@fastgpt/global/core/app/version/type';
import { AppCollectionName, chatConfigType } from '../schema'; import { AppCollectionName, chatConfigType } from '../schema';
...@@ -48,7 +48,7 @@ const AppVersionSchema = new Schema( ...@@ -48,7 +48,7 @@ const AppVersionSchema = new Schema(
} }
); );
AppVersionSchema.index({ appId: 1, time: -1 }); defineIndex(AppVersionSchema, { key: { appId: 1, time: -1 } });
export const MongoAppVersion = getMongoModel<AppVersionSchemaType>( export const MongoAppVersion = getMongoModel<AppVersionSchemaType>(
AppVersionCollectionName, AppVersionCollectionName,
......
packages/service/core/chat/chatItemResponseSchema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import type { ChatItemResponseSchemaType } from '@fastgpt/global/core/chat/type'; import type { ChatItemResponseSchemaType } from '@fastgpt/global/core/chat/type';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
...@@ -42,16 +42,20 @@ const ChatItemResponseSchema = new Schema({ ...@@ -42,16 +42,20 @@ const ChatItemResponseSchema = new Schema({
/* TODO: 未全面检查操作，所以这里暂时不加 sourceType 的索引。 */ /* TODO: 未全面检查操作，所以这里暂时不加 sourceType 的索引。 */
// 按 chat item 拉取完整 nodeResponse rows；复合索引包含 _id，避免详情读取时额外排序。 // 按 chat item 拉取完整 nodeResponse rows；复合索引包含 _id，避免详情读取时额外排序。
ChatItemResponseSchema.index({ appId: 1, chatId: 1, chatItemDataId: 1, _id: 1 }); defineIndex(ChatItemResponseSchema, {
ChatItemResponseSchema.index({ key: { appId: 1, chatId: 1, chatItemDataId: 1, _id: 1 }
sourceType: 1, });
appId: 1, defineIndex(ChatItemResponseSchema, {
chatId: 1, key: {
chatItemDataId: 1, sourceType: 1,
_id: 1 appId: 1,
chatId: 1,
chatItemDataId: 1,
_id: 1
}
}); });
// Clear expired response // Clear expired response
ChatItemResponseSchema.index({ teamId: 1, time: -1 }); defineIndex(ChatItemResponseSchema, { key: { teamId: 1, time: -1 } });
export const MongoChatItemResponse = getMongoModel<ChatItemResponseSchemaType>( export const MongoChatItemResponse = getMongoModel<ChatItemResponseSchemaType>(
ChatItemResponseCollectionName, ChatItemResponseCollectionName,
......
packages/service/core/chat/chatItemSchema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type ChatItemDBSchemaType } from '@fastgpt/global/core/chat/type'; import { type ChatItemDBSchemaType } from '@fastgpt/global/core/chat/type';
import { ChatRoleMap, ChatSourceTypeEnum } from '@fastgpt/global/core/chat/constants'; import { ChatRoleMap, ChatSourceTypeEnum } from '@fastgpt/global/core/chat/constants';
...@@ -103,17 +103,25 @@ const ChatItemSchema = new Schema({ ...@@ -103,17 +103,25 @@ const ChatItemSchema = new Schema({
delete by chat id; delete by chat id;
close custom feedback; close custom feedback;
*/ */
ChatItemSchema.index({ appId: 1, chatId: 1, dataId: 1 }); defineIndex(ChatItemSchema, { key: { appId: 1, chatId: 1, dataId: 1 } });
ChatItemSchema.index({ sourceType: 1, appId: 1, chatId: 1, dataId: 1 }); defineIndex(ChatItemSchema, {
key: { sourceType: 1, appId: 1, chatId: 1, dataId: 1 }
});
// Get histories // Get histories
ChatItemSchema.index({ appId: 1, chatId: 1, deleteTime: 1 }); defineIndex(ChatItemSchema, { key: { appId: 1, chatId: 1, deleteTime: 1 } });
ChatItemSchema.index({ sourceType: 1, appId: 1, chatId: 1, deleteTime: 1 }); defineIndex(ChatItemSchema, {
key: { sourceType: 1, appId: 1, chatId: 1, deleteTime: 1 }
});
// get chatitem list,Anchor filter // get chatitem list,Anchor filter
ChatItemSchema.index({ appId: 1, chatId: 1, _id: -1 }); defineIndex(ChatItemSchema, { key: { appId: 1, chatId: 1, _id: -1 } });
ChatItemSchema.index({ sourceType: 1, appId: 1, chatId: 1, _id: -1 }); defineIndex(ChatItemSchema, {
key: { sourceType: 1, appId: 1, chatId: 1, _id: -1 }
});
// Query by role (AI/Human), get latest chat item, permission check // Query by role (AI/Human), get latest chat item, permission check
ChatItemSchema.index({ appId: 1, chatId: 1, obj: 1, _id: -1 }); defineIndex(ChatItemSchema, { key: { appId: 1, chatId: 1, obj: 1, _id: -1 } });
ChatItemSchema.index({ sourceType: 1, appId: 1, chatId: 1, obj: 1, _id: -1 }); defineIndex(ChatItemSchema, {
key: { sourceType: 1, appId: 1, chatId: 1, obj: 1, _id: -1 }
});
export const MongoChatItem = getMongoModel<ChatItemDBSchemaType>( export const MongoChatItem = getMongoModel<ChatItemDBSchemaType>(
ChatItemCollectionName, ChatItemCollectionName,
......
packages/service/core/chat/chatSchema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo'; import { connectionMongo, defineIndex, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type ChatSchemaType } from '@fastgpt/global/core/chat/type'; import { type ChatSchemaType } from '@fastgpt/global/core/chat/type';
import { import {
...@@ -133,120 +133,127 @@ const ChatSchema = new Schema({ ...@@ -133,120 +133,127 @@ const ChatSchema = new Schema({
userId: Schema.Types.ObjectId userId: Schema.Types.ObjectId
}); });
ChatSchema.index({ chatId: 1 }); defineIndex(ChatSchema, { key: { chatId: 1 } });
// Delete by appid; init chat; update chat; auth chat; // Delete by appid; init chat; update chat; auth chat;
ChatSchema.index({ sourceType: 1, appId: 1, chatId: 1 }, { unique: true }); defineIndex(ChatSchema, {
key: { sourceType: 1, appId: 1, chatId: 1 },
options: { unique: true }
});
// timer, clear history // timer, clear history
ChatSchema.index({ updateTime: -1, teamId: 1 }); defineIndex(ChatSchema, { key: { updateTime: -1, teamId: 1 } });
ChatSchema.index({ teamId: 1, updateTime: -1 }); defineIndex(ChatSchema, { key: { teamId: 1, updateTime: -1 } });
// get user history(Cookie) // get user history(Cookie)
ChatSchema.index({ tmbId: 1, appId: 1, deleteTime: 1, top: -1, updateTime: -1 }); defineIndex(ChatSchema, {
key: { tmbId: 1, appId: 1, deleteTime: 1, top: -1, updateTime: -1 }
});
/* ===== 条件索引 ===== */ /* ===== 条件索引 ===== */
// Clear history(share),Init 4121 // Clear history(share),Init 4121
ChatSchema.index( defineIndex(ChatSchema, {
{ appId: 1, outLinkUid: 1, tmbId: 1 }, key: { appId: 1, outLinkUid: 1, tmbId: 1 },
{ options: {
partialFilterExpression: { partialFilterExpression: {
outLinkUid: { $exists: true } outLinkUid: { $exists: true }
} }
} }
); });
// get share chat history // get share chat history
ChatSchema.index( defineIndex(ChatSchema, {
{ shareId: 1, outLinkUid: 1, updateTime: -1 }, key: { shareId: 1, outLinkUid: 1, updateTime: -1 },
{ options: {
partialFilterExpression: { partialFilterExpression: {
shareId: { $exists: true } shareId: { $exists: true }
} }
} }
); });
/* get chat logs */ /* get chat logs */
// 1. Common get // 1. Common get
ChatSchema.index({ appId: 1, updateTime: -1 }); defineIndex(ChatSchema, { key: { appId: 1, updateTime: -1 } });
// Get history(tmbId) // Get history(tmbId)
ChatSchema.index({ appId: 1, tmbId: 1, updateTime: -1 }); defineIndex(ChatSchema, { key: { appId: 1, tmbId: 1, updateTime: -1 } });
// clearHistory(API) // clearHistory(API)
ChatSchema.index({ appId: 1, source: 1, tmbId: 1, updateTime: -1 }); defineIndex(ChatSchema, {
key: { appId: 1, source: 1, tmbId: 1, updateTime: -1 }
});
// Periodic cleanup for chats stuck in generating state. // Periodic cleanup for chats stuck in generating state.
ChatSchema.index( defineIndex(ChatSchema, {
{ chatGenerateStatus: 1, updateTime: 1 }, key: { chatGenerateStatus: 1, updateTime: 1 },
{ options: {
partialFilterExpression: { partialFilterExpression: {
chatGenerateStatus: ChatGenerateStatusEnum.generating chatGenerateStatus: ChatGenerateStatusEnum.generating
} }
} }
); });
/* 反馈过滤的索引 */ /* 反馈过滤的索引 */
// 2. Has good feedback filter // 2. Has good feedback filter
ChatSchema.index( defineIndex(ChatSchema, {
{ key: {
appId: 1, appId: 1,
hasGoodFeedback: 1, hasGoodFeedback: 1,
updateTime: -1 updateTime: -1
}, },
{ options: {
partialFilterExpression: { partialFilterExpression: {
hasGoodFeedback: true hasGoodFeedback: true
} }
} }
); });
// Has bad feedback filter // Has bad feedback filter
ChatSchema.index( defineIndex(ChatSchema, {
{ key: {
appId: 1, appId: 1,
hasBadFeedback: 1, hasBadFeedback: 1,
updateTime: -1 updateTime: -1
}, },
{ options: {
partialFilterExpression: { partialFilterExpression: {
hasBadFeedback: true hasBadFeedback: true
} }
} }
); });
// 3. Has unread good feedback filter // 3. Has unread good feedback filter
ChatSchema.index( defineIndex(ChatSchema, {
{ key: {
appId: 1, appId: 1,
hasUnreadGoodFeedback: 1, hasUnreadGoodFeedback: 1,
updateTime: -1 updateTime: -1
}, },
{ options: {
partialFilterExpression: { partialFilterExpression: {
hasUnreadGoodFeedback: true hasUnreadGoodFeedback: true
} }
} }
); });
// Has unread bad feedback filter // Has unread bad feedback filter
ChatSchema.index( defineIndex(ChatSchema, {
{ key: {
appId: 1, appId: 1,
hasUnreadBadFeedback: 1, hasUnreadBadFeedback: 1,
updateTime: -1 updateTime: -1
}, },
{ options: {
partialFilterExpression: { partialFilterExpression: {
hasUnreadBadFeedback: true hasUnreadBadFeedback: true
} }
} }
); });
// Has error filter // Has error filter
ChatSchema.index( defineIndex(ChatSchema, {
{ key: {
appId: 1, appId: 1,
errorCount: 1, errorCount: 1,
updateTime: -1 updateTime: -1
}, },
{ options: {
partialFilterExpression: { partialFilterExpression: {
errorCount: { $gt: 0 } errorCount: { $gt: 0 }
} }
} }
); });
export const MongoChat = getMongoModel<ChatSchemaType>(chatCollectionName, ChatSchema); export const MongoChat = getMongoModel<ChatSchemaType>(chatCollectionName, ChatSchema);
packages/service/core/chat/favouriteApp/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { type ChatFavouriteAppType } from '@fastgpt/global/core/chat/favouriteApp/type'; import { type ChatFavouriteAppType } from '@fastgpt/global/core/chat/favouriteApp/type';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { AppCollectionName } from '../../app/schema'; import { AppCollectionName } from '../../app/schema';
...@@ -28,7 +28,7 @@ const ChatFavouriteAppSchema = new Schema({ ...@@ -28,7 +28,7 @@ const ChatFavouriteAppSchema = new Schema({
} }
}); });
ChatFavouriteAppSchema.index({ teamId: 1, appId: 1 }); defineIndex(ChatFavouriteAppSchema, { key: { teamId: 1, appId: 1 } });
export const MongoChatFavouriteApp = getMongoModel<ChatFavouriteAppType>( export const MongoChatFavouriteApp = getMongoModel<ChatFavouriteAppType>(
ChatFavouriteAppCollectionName, ChatFavouriteAppCollectionName,
......
packages/service/core/chat/inputGuide/schema.ts
import { AppCollectionName } from '../../app/schema'; import { AppCollectionName } from '../../app/schema';
import { connectionMongo, getMongoModel, type Model } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel, type Model } from '../../../common/mongo';
const { Schema, model, models } = connectionMongo; const { Schema, model, models } = connectionMongo;
import type { ChatInputGuideSchemaType } from '@fastgpt/global/core/chat/inputGuide/type'; import type { ChatInputGuideSchemaType } from '@fastgpt/global/core/chat/inputGuide/type';
import { getLogger, LogCategories } from '../../../common/logger';
export const ChatInputGuideCollectionName = 'chat_input_guides'; export const ChatInputGuideCollectionName = 'chat_input_guides';
...@@ -18,12 +17,10 @@ const ChatInputGuideSchema = new Schema({ ...@@ -18,12 +17,10 @@ const ChatInputGuideSchema = new Schema({
} }
}); });
try { defineIndex(ChatInputGuideSchema, {
ChatInputGuideSchema.index({ appId: 1, text: 1 }, { unique: true }); key: { appId: 1, text: 1 },
} catch (error) { options: { unique: true }
const logger = getLogger(LogCategories.INFRA.MONGO); });
logger.error('Failed to build chat input guide indexes', { error });
}
export const MongoChatInputGuide = getMongoModel<ChatInputGuideSchemaType>( export const MongoChatInputGuide = getMongoModel<ChatInputGuideSchemaType>(
ChatInputGuideCollectionName, ChatInputGuideCollectionName,
......
packages/service/core/chat/setting/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { type ChatSettingModelType } from '@fastgpt/global/core/chat/setting/type'; import { type ChatSettingModelType } from '@fastgpt/global/core/chat/setting/type';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { AppCollectionName } from '../../app/schema'; import { AppCollectionName } from '../../app/schema';
...@@ -53,7 +53,7 @@ ChatSettingSchema.virtual('quickAppList', { ...@@ -53,7 +53,7 @@ ChatSettingSchema.virtual('quickAppList', {
foreignField: '_id' foreignField: '_id'
}); });
ChatSettingSchema.index({ teamId: 1 }); defineIndex(ChatSettingSchema, { key: { teamId: 1 } });
export const MongoChatSetting = getMongoModel<ChatSettingModelType>( export const MongoChatSetting = getMongoModel<ChatSettingModelType>(
ChatSettingCollectionName, ChatSettingCollectionName,
......
packages/service/core/dataset/collection/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { getLogger, LogCategories } from '../../../common/logger';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type DatasetCollectionSchemaType } from '@fastgpt/global/core/dataset/type'; import { type DatasetCollectionSchemaType } from '@fastgpt/global/core/dataset/type';
import { DatasetCollectionTypeMap } from '@fastgpt/global/core/dataset/constants'; import { DatasetCollectionTypeMap } from '@fastgpt/global/core/dataset/constants';
...@@ -94,43 +93,46 @@ DatasetCollectionSchema.virtual('dataset', { ...@@ -94,43 +93,46 @@ DatasetCollectionSchema.virtual('dataset', {
justOne: true justOne: true
}); });
try { // auth file
// auth file defineIndex(DatasetCollectionSchema, { key: { teamId: 1, fileId: 1 } });
DatasetCollectionSchema.index({ teamId: 1, fileId: 1 });
// list collection; deep find collections // list collection; deep find collections
DatasetCollectionSchema.index({ defineIndex(DatasetCollectionSchema, {
key: {
teamId: 1, teamId: 1,
datasetId: 1, datasetId: 1,
parentId: 1, parentId: 1,
updateTime: -1 updateTime: -1
}); }
});
// Tag filter
DatasetCollectionSchema.index({ teamId: 1, datasetId: 1, tags: 1 }); // Tag filter
// create time filter defineIndex(DatasetCollectionSchema, {
DatasetCollectionSchema.index({ teamId: 1, datasetId: 1, createTime: 1 }); key: { teamId: 1, datasetId: 1, tags: 1 }
});
// Get collection by external file id // create time filter
DatasetCollectionSchema.index( defineIndex(DatasetCollectionSchema, {
{ datasetId: 1, externalFileId: 1 }, key: { teamId: 1, datasetId: 1, createTime: 1 }
{ });
unique: true,
partialFilterExpression: { // Get collection by external file id
externalFileId: { $exists: true } defineIndex(DatasetCollectionSchema, {
} key: { datasetId: 1, externalFileId: 1 },
options: {
unique: true,
partialFilterExpression: {
externalFileId: { $exists: true }
} }
); }
});
// Clear invalid image // Clear invalid image
DatasetCollectionSchema.index({ defineIndex(DatasetCollectionSchema, {
key: {
teamId: 1, teamId: 1,
'metadata.relatedImgId': 1 'metadata.relatedImgId': 1
}); }
} catch (error) { });
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset collection indexes', { error });
}
export const MongoDatasetCollection = getMongoModel<DatasetCollectionSchemaType>( export const MongoDatasetCollection = getMongoModel<DatasetCollectionSchemaType>(
DatasetColCollectionName, DatasetColCollectionName,
......
packages/service/core/dataset/data/dataTextSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type DatasetDataTextSchemaType } from '@fastgpt/global/core/dataset/type'; import { type DatasetDataTextSchemaType } from '@fastgpt/global/core/dataset/type';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { DatasetCollectionName } from '../schema'; import { DatasetCollectionName } from '../schema';
import { DatasetColCollectionName } from '../collection/schema'; import { DatasetColCollectionName } from '../collection/schema';
import { DatasetDataCollectionName } from './schema'; import { DatasetDataCollectionName } from './schema';
import { getLogger, LogCategories } from '../../../common/logger';
export const DatasetDataTextCollectionName = 'dataset_data_texts'; export const DatasetDataTextCollectionName = 'dataset_data_texts';
...@@ -33,20 +32,17 @@ const DatasetDataTextSchema = new Schema({ ...@@ -33,20 +32,17 @@ const DatasetDataTextSchema = new Schema({
fullTextToken: String fullTextToken: String
}); });
try { defineIndex(DatasetDataTextSchema, {
DatasetDataTextSchema.index( key: { teamId: 1, fullTextToken: 'text' },
{ teamId: 1, fullTextToken: 'text' }, options: {
{ name: 'teamId_1_fullTextToken_text',
name: 'teamId_1_fullTextToken_text', default_language: 'none'
default_language: 'none' }
} });
); defineIndex(DatasetDataTextSchema, {
DatasetDataTextSchema.index({ teamId: 1, datasetId: 1, collectionId: 1 }); key: { teamId: 1, datasetId: 1, collectionId: 1 }
DatasetDataTextSchema.index({ dataId: 'hashed' }); });
} catch (error) { defineIndex(DatasetDataTextSchema, { key: { dataId: 'hashed' } });
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset data text indexes', { error });
}
export const MongoDatasetDataText = getMongoModel<DatasetDataTextSchemaType>( export const MongoDatasetDataText = getMongoModel<DatasetDataTextSchemaType>(
DatasetDataTextCollectionName, DatasetDataTextCollectionName,
......
packages/service/core/dataset/data/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema, model, models } = connectionMongo; const { Schema, model, models } = connectionMongo;
import { type DatasetDataSchemaType } from '@fastgpt/global/core/dataset/type'; import { type DatasetDataSchemaType } from '@fastgpt/global/core/dataset/type';
import { import {
...@@ -8,7 +8,6 @@ import { ...@@ -8,7 +8,6 @@ import {
import { DatasetCollectionName } from '../schema'; import { DatasetCollectionName } from '../schema';
import { DatasetColCollectionName } from '../collection/schema'; import { DatasetColCollectionName } from '../collection/schema';
import { DatasetDataIndexTypeEnum } from '@fastgpt/global/core/dataset/data/constants'; import { DatasetDataIndexTypeEnum } from '@fastgpt/global/core/dataset/data/constants';
import { getLogger, LogCategories } from '../../../common/logger';
export const DatasetDataCollectionName = 'dataset_datas'; export const DatasetDataCollectionName = 'dataset_datas';
...@@ -89,26 +88,27 @@ const DatasetDataSchema = new Schema({ ...@@ -89,26 +88,27 @@ const DatasetDataSchema = new Schema({
initJieba: Boolean initJieba: Boolean
}); });
try { // list collection and count data; list data; delete collection(relate data)
// list collection and count data; list data; delete collection(relate data) defineIndex(DatasetDataSchema, {
DatasetDataSchema.index({ key: {
teamId: 1, teamId: 1,
datasetId: 1, datasetId: 1,
collectionId: 1, collectionId: 1,
chunkIndex: 1, chunkIndex: 1,
updateTime: -1 updateTime: -1
}); }
// Recall vectors after data matching });
DatasetDataSchema.index({ teamId: 1, datasetId: 1, collectionId: 1, 'indexes.dataId': 1 }); // Recall vectors after data matching
// rebuild data defineIndex(DatasetDataSchema, {
DatasetDataSchema.index({ rebuilding: 1, teamId: 1, datasetId: 1 }); key: { teamId: 1, datasetId: 1, collectionId: 1, 'indexes.dataId': 1 }
});
// rebuild data
defineIndex(DatasetDataSchema, {
key: { rebuilding: 1, teamId: 1, datasetId: 1 }
});
// Cron clear invalid data // Cron clear invalid data
DatasetDataSchema.index({ updateTime: 1 }); defineIndex(DatasetDataSchema, { key: { updateTime: 1 } });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset data indexes', { error });
}
export const MongoDatasetData = getMongoModel<DatasetDataSchemaType>( export const MongoDatasetData = getMongoModel<DatasetDataSchemaType>(
DatasetDataCollectionName, DatasetDataCollectionName,
......
packages/service/core/dataset/image/schema.ts
import type { Types } from '../../../common/mongo'; import type { Types } from '../../../common/mongo';
import { getMongoModel, Schema } from '../../../common/mongo'; import { defineIndex, getMongoModel, Schema } from '../../../common/mongo';
export const bucketName = 'dataset_image'; export const bucketName = 'dataset_image';
...@@ -16,9 +16,13 @@ const MongoDatasetImage = new Schema({ ...@@ -16,9 +16,13 @@ const MongoDatasetImage = new Schema({
expiredTime: { type: Date, required: true } expiredTime: { type: Date, required: true }
} }
}); });
MongoDatasetImage.index({ 'metadata.datasetId': 'hashed' }); defineIndex(MongoDatasetImage, {
MongoDatasetImage.index({ 'metadata.collectionId': 'hashed' }); key: { 'metadata.datasetId': 'hashed' }
MongoDatasetImage.index({ 'metadata.expiredTime': -1 }); });
defineIndex(MongoDatasetImage, {
key: { 'metadata.collectionId': 'hashed' }
});
defineIndex(MongoDatasetImage, { key: { 'metadata.expiredTime': -1 } });
export const MongoDatasetImageSchema = getMongoModel<{ export const MongoDatasetImageSchema = getMongoModel<{
_id: Types.ObjectId; _id: Types.ObjectId;
......
packages/service/core/dataset/migration/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { getLogger, LogCategories } from '../../../common/logger';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
export const DatasetMigrationLogCollectionName = 'dataset_migration_logs'; export const DatasetMigrationLogCollectionName = 'dataset_migration_logs';
...@@ -91,8 +90,7 @@ const DatasetMigrationLogSchema = new Schema({ ...@@ -91,8 +90,7 @@ const DatasetMigrationLogSchema = new Schema({
// 批次信息 // 批次信息
batchId: { batchId: {
type: String, type: String,
required: true, required: true
index: true
}, },
migrationVersion: { migrationVersion: {
type: String, type: String,
...@@ -107,17 +105,14 @@ const DatasetMigrationLogSchema = new Schema({ ...@@ -107,17 +105,14 @@ const DatasetMigrationLogSchema = new Schema({
}, },
resourceId: { resourceId: {
type: Schema.Types.ObjectId, type: Schema.Types.ObjectId,
required: true, required: true
index: true
}, },
teamId: { teamId: {
type: Schema.Types.ObjectId, type: Schema.Types.ObjectId,
required: true, required: true
index: true
}, },
datasetId: { datasetId: {
type: Schema.Types.ObjectId, type: Schema.Types.ObjectId
index: true
}, },
// 存储信息 // 存储信息
...@@ -151,15 +146,13 @@ const DatasetMigrationLogSchema = new Schema({ ...@@ -151,15 +146,13 @@ const DatasetMigrationLogSchema = new Schema({
type: String, type: String,
enum: ['pending', 'processing', 'completed', 'failed', 'rollback', 'verified'], enum: ['pending', 'processing', 'completed', 'failed', 'rollback', 'verified'],
default: 'pending', default: 'pending',
required: true, required: true
index: true
}, },
// 时间戳 // 时间戳
createdAt: { createdAt: {
type: Date, type: Date,
default: () => new Date(), default: () => new Date()
index: true
}, },
startedAt: Date, startedAt: Date,
completedAt: Date, completedAt: Date,
...@@ -227,29 +220,38 @@ const DatasetMigrationLogSchema = new Schema({ ...@@ -227,29 +220,38 @@ const DatasetMigrationLogSchema = new Schema({
}); });
// 索引优化 // 索引优化
try { defineIndex(DatasetMigrationLogSchema, { key: { batchId: 1 } });
// 查询某个批次的迁移状态 defineIndex(DatasetMigrationLogSchema, { key: { resourceId: 1 } });
DatasetMigrationLogSchema.index({ batchId: 1, status: 1 }); defineIndex(DatasetMigrationLogSchema, { key: { teamId: 1 } });
defineIndex(DatasetMigrationLogSchema, { key: { datasetId: 1 } });
// 查询某个资源的迁移历史 defineIndex(DatasetMigrationLogSchema, { key: { status: 1 } });
DatasetMigrationLogSchema.index({ resourceType: 1, resourceId: 1 }); defineIndex(DatasetMigrationLogSchema, { key: { createdAt: 1 } });
// 查询某个批次的迁移状态
defineIndex(DatasetMigrationLogSchema, { key: { batchId: 1, status: 1 } });
// 查询某个资源的迁移历史
defineIndex(DatasetMigrationLogSchema, {
key: { resourceType: 1, resourceId: 1 }
});
// 查询失败的迁移（需要重试） // 查询失败的迁移（需要重试）
DatasetMigrationLogSchema.index({ defineIndex(DatasetMigrationLogSchema, {
key: {
status: 1, status: 1,
attemptCount: 1, attemptCount: 1,
lastAttemptAt: 1 lastAttemptAt: 1
}); }
});
// 查询某个团队的迁移情况 // 查询某个团队的迁移情况
DatasetMigrationLogSchema.index({ teamId: 1, status: 1 }); defineIndex(DatasetMigrationLogSchema, { key: { teamId: 1, status: 1 } });
// 唯一索引：同一个资源在同一个批次只能有一条记录 // 唯一索引：同一个资源在同一个批次只能有一条记录
DatasetMigrationLogSchema.index({ batchId: 1, resourceType: 1, resourceId: 1 }, { unique: true }); defineIndex(DatasetMigrationLogSchema, {
} catch (error) { key: { batchId: 1, resourceType: 1, resourceId: 1 },
const logger = getLogger(LogCategories.INFRA.MONGO); options: { unique: true }
logger.error('Failed to build dataset migration indexes', { error }); });
}
export const MongoDatasetMigrationLog = getMongoModel<DatasetMigrationLogSchemaType>( export const MongoDatasetMigrationLog = getMongoModel<DatasetMigrationLogSchemaType>(
DatasetMigrationLogCollectionName, DatasetMigrationLogCollectionName,
......
packages/service/core/dataset/schema.ts
import { getMongoModel, Schema } from '../../common/mongo'; import { defineIndex, getMongoModel, Schema } from '../../common/mongo';
import { import {
ChunkSettingModeEnum, ChunkSettingModeEnum,
ChunkTriggerConfigTypeEnum, ChunkTriggerConfigTypeEnum,
...@@ -14,7 +14,6 @@ import { ...@@ -14,7 +14,6 @@ import {
} from '@fastgpt/global/support/user/team/constant'; } from '@fastgpt/global/support/user/team/constant';
import { userCollectionName } from '../../support/user/schema'; import { userCollectionName } from '../../support/user/schema';
import type { DatasetSchemaType } from '@fastgpt/global/core/dataset/type'; import type { DatasetSchemaType } from '@fastgpt/global/core/dataset/type';
import { getLogger, LogCategories } from '../../common/logger';
export const DatasetCollectionName = 'datasets'; export const DatasetCollectionName = 'datasets';
...@@ -152,13 +151,8 @@ const DatasetSchema = new Schema({ ...@@ -152,13 +151,8 @@ const DatasetSchema = new Schema({
yuqueServer: Object yuqueServer: Object
}); });
try { defineIndex(DatasetSchema, { key: { teamId: 1 } });
DatasetSchema.index({ teamId: 1 }); defineIndex(DatasetSchema, { key: { type: 1 } }); // Admin count
DatasetSchema.index({ type: 1 }); // Admin count defineIndex(DatasetSchema, { key: { deleteTime: 1 } }); // 添加软删除字段索引
DatasetSchema.index({ deleteTime: 1 }); // 添加软删除字段索引
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset indexes', { error });
}
export const MongoDataset = getMongoModel<DatasetSchemaType>(DatasetCollectionName, DatasetSchema); export const MongoDataset = getMongoModel<DatasetSchemaType>(DatasetCollectionName, DatasetSchema);
packages/service/core/dataset/tag/schema.ts
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel, type Model } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel, type Model } from '../../../common/mongo';
import { DatasetCollectionName } from '../schema'; import { DatasetCollectionName } from '../schema';
import { type DatasetCollectionTagsSchemaType } from '@fastgpt/global/core/dataset/type'; import { type DatasetCollectionTagsSchemaType } from '@fastgpt/global/core/dataset/type';
import { getLogger, LogCategories } from '../../../common/logger';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
export const DatasetCollectionTagsName = 'dataset_collection_tags'; export const DatasetCollectionTagsName = 'dataset_collection_tags';
...@@ -24,12 +23,9 @@ const DatasetCollectionTagsSchema = new Schema({ ...@@ -24,12 +23,9 @@ const DatasetCollectionTagsSchema = new Schema({
} }
}); });
try { defineIndex(DatasetCollectionTagsSchema, {
DatasetCollectionTagsSchema.index({ teamId: 1, datasetId: 1, tag: 1 }); key: { teamId: 1, datasetId: 1, tag: 1 }
} catch (error) { });
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset tag indexes', { error });
}
export const MongoDatasetCollectionTags = getMongoModel<DatasetCollectionTagsSchemaType>( export const MongoDatasetCollectionTags = getMongoModel<DatasetCollectionTagsSchemaType>(
DatasetCollectionTagsName, DatasetCollectionTagsName,
......
packages/service/core/dataset/training/schema.ts
/* 模型的知识库 */ /* 模型的知识库 */
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type DatasetTrainingSchemaType } from '@fastgpt/global/core/dataset/type'; import { type DatasetTrainingSchemaType } from '@fastgpt/global/core/dataset/type';
import { TrainingModeEnum } from '@fastgpt/global/core/dataset/constants'; import { TrainingModeEnum } from '@fastgpt/global/core/dataset/constants';
...@@ -119,16 +119,23 @@ TrainingDataSchema.virtual('data', { ...@@ -119,16 +119,23 @@ TrainingDataSchema.virtual('data', {
}); });
// lock training data(teamId); delete training data // lock training data(teamId); delete training data
TrainingDataSchema.index({ teamId: 1, datasetId: 1 }); defineIndex(TrainingDataSchema, { key: { teamId: 1, datasetId: 1 } });
// collection 级状态、错误列表、删除、详情 // collection 级状态、错误列表、删除、详情
TrainingDataSchema.index({ defineIndex(TrainingDataSchema, {
teamId: 1, key: {
datasetId: 1, teamId: 1,
collectionId: 1 datasetId: 1,
collectionId: 1
}
}); });
// get training data and sort // get training data and sort
TrainingDataSchema.index({ mode: 1, retryCount: 1, lockTime: 1, weight: -1 }); defineIndex(TrainingDataSchema, {
TrainingDataSchema.index({ expireAt: 1 }, { expireAfterSeconds: 7 * 24 * 60 * 60 }); // 7 days key: { mode: 1, retryCount: 1, lockTime: 1, weight: -1 }
});
defineIndex(TrainingDataSchema, {
key: { expireAt: 1 },
options: { expireAfterSeconds: 7 * 24 * 60 * 60 }
}); // 7 days
export const MongoDatasetTraining = getMongoModel<DatasetTrainingSchemaType>( export const MongoDatasetTraining = getMongoModel<DatasetTrainingSchemaType>(
DatasetTrainingCollectionName, DatasetTrainingCollectionName,
......
packages/service/core/plugin/schema/teamInstalledPluginSchema.ts
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo/index'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo/index';
import type { TeamInstalledPluginSchemaType } from '@fastgpt/global/core/plugin/schema/type'; import type { TeamInstalledPluginSchemaType } from '@fastgpt/global/core/plugin/schema/type';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
...@@ -29,7 +29,10 @@ const TeamInstalledPluginSchema = new Schema({ ...@@ -29,7 +29,10 @@ const TeamInstalledPluginSchema = new Schema({
} }
}); });
TeamInstalledPluginSchema.index({ teamId: 1, pluginId: 1 }, { unique: true }); defineIndex(TeamInstalledPluginSchema, {
key: { teamId: 1, pluginId: 1 },
options: { unique: true }
});
export const MongoTeamInstalledPlugin = getMongoModel<TeamInstalledPluginSchemaType>( export const MongoTeamInstalledPlugin = getMongoModel<TeamInstalledPluginSchemaType>(
collectionName, collectionName,
......
packages/service/core/plugin/tool/pluginGroupSchema.ts
// 已弃用 // 已弃用
import type { I18nStringStrictType } from '@fastgpt/global/sdk/fastgpt-plugin'; import type { I18nStringStrictType } from '@fastgpt/global/sdk/fastgpt-plugin';
import { connectionMongo, getMongoModel } from '../../../common/mongo/index'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo/index';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
export const collectionName = 'app_plugin_groups'; export const collectionName = 'app_plugin_groups';
...@@ -40,7 +40,10 @@ const PluginGroupSchema = new Schema({ ...@@ -40,7 +40,10 @@ const PluginGroupSchema = new Schema({
} }
}); });
PluginGroupSchema.index({ groupId: 1 }, { unique: true }); defineIndex(PluginGroupSchema, {
key: { groupId: 1 },
options: { unique: true }
});
export const MongoToolGroups = getMongoModel<SystemToolGroupSchemaType>( export const MongoToolGroups = getMongoModel<SystemToolGroupSchemaType>(
collectionName, collectionName,
......
packages/service/core/plugin/tool/systemToolSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo/index'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo/index';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import type { SystemPluginToolCollectionType } from '@fastgpt/global/core/plugin/tool/type'; import type { SystemPluginToolCollectionType } from '@fastgpt/global/core/plugin/tool/type';
import type { PluginStatusType } from '@fastgpt/global/core/plugin/type'; import type { PluginStatusType } from '@fastgpt/global/core/plugin/type';
...@@ -120,7 +120,7 @@ const SystemToolSchema = new Schema({ ...@@ -120,7 +120,7 @@ const SystemToolSchema = new Schema({
} }
}); });
SystemToolSchema.index({ pluginId: 1 }); defineIndex(SystemToolSchema, { key: { pluginId: 1 } });
export const MongoSystemTool = getMongoModel<SystemPluginToolCollectionType>( export const MongoSystemTool = getMongoModel<SystemPluginToolCollectionType>(
collectionName, collectionName,
......
packages/service/core/plugin/tool/tagSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo/index'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo/index';
import type { SystemPluginToolTagType } from '@fastgpt/global/core/plugin/type'; import type { SystemPluginToolTagType } from '@fastgpt/global/core/plugin/type';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
...@@ -7,8 +7,7 @@ export const collectionName = 'system_plugin_tool_tags'; ...@@ -7,8 +7,7 @@ export const collectionName = 'system_plugin_tool_tags';
const SystemPluginToolTagSchema = new Schema({ const SystemPluginToolTagSchema = new Schema({
tagId: { tagId: {
type: String, type: String,
required: true, required: true
unique: true
}, },
tagName: { tagName: {
type: Schema.Types.Mixed, type: Schema.Types.Mixed,
...@@ -24,7 +23,11 @@ const SystemPluginToolTagSchema = new Schema({ ...@@ -24,7 +23,11 @@ const SystemPluginToolTagSchema = new Schema({
} }
}); });
SystemPluginToolTagSchema.index({ tagOrder: 1 }); defineIndex(SystemPluginToolTagSchema, {
key: { tagId: 1 },
options: { unique: true }
});
defineIndex(SystemPluginToolTagSchema, { key: { tagOrder: 1 } });
export const MongoPluginToolTag = getMongoModel<SystemPluginToolTagType>( export const MongoPluginToolTag = getMongoModel<SystemPluginToolTagType>(
collectionName, collectionName,
......
packages/service/env.ts
...@@ -29,7 +29,6 @@ export const serviceEnv = createEnv({ ...@@ -29,7 +29,6 @@ export const serviceEnv = createEnv({
server: { server: {
// ==================== 基础配置 ==================== // ==================== 基础配置 ====================
DB_MAX_LINK: IntSchema.min(1).default(5), DB_MAX_LINK: IntSchema.min(1).default(5),
SYNC_INDEX: BoolSchema.default(true),
// ==================== 密钥 ==================== // ==================== 密钥 ====================
ROOT_KEY: z ROOT_KEY: z
...@@ -152,6 +151,9 @@ export const serviceEnv = createEnv({ ...@@ -152,6 +151,9 @@ export const serviceEnv = createEnv({
'mongodb://myusername:mypassword@localhost:27017/fastgpt?authSource=admin&directConnection=true' 'mongodb://myusername:mypassword@localhost:27017/fastgpt?authSource=admin&directConnection=true'
), ),
MONGODB_LOG_URI: z.string().optional(), MONGODB_LOG_URI: z.string().optional(),
SYNC_INDEX: BoolSchema.default(true).meta({
description: '是否在启动时创建当前 MongoDB 索引并清理显式声明的废弃索引'
}),
// VectorDB // VectorDB
VECTOR_VQ_LEVEL: IntSchema.default(32).meta({ VECTOR_VQ_LEVEL: IntSchema.default(32).meta({
......
packages/service/support/activity/promotion/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type PromotionRecordSchema as PromotionRecordType } from '@fastgpt/global/support/activity/type'; import { type PromotionRecordSchema as PromotionRecordType } from '@fastgpt/global/support/activity/type';
import { userCollectionName } from '../../user/schema'; import { userCollectionName } from '../../user/schema';
...@@ -30,7 +30,7 @@ const PromotionRecordSchema = new Schema({ ...@@ -30,7 +30,7 @@ const PromotionRecordSchema = new Schema({
} }
}); });
PromotionRecordSchema.index({ userId: 1 }); defineIndex(PromotionRecordSchema, { key: { userId: 1 } });
export const MongoPromotionRecord = getMongoModel<PromotionRecordType>( export const MongoPromotionRecord = getMongoModel<PromotionRecordType>(
'promotionRecord', 'promotionRecord',
......
packages/service/support/appRegistration/schema.ts
import { Schema, getMongoModel } from '../../common/mongo/index'; import { defineIndex, Schema, getMongoModel } from '../../common/mongo/index';
import { AppCollectionName } from '../../core/app/schema'; import { AppCollectionName } from '../../core/app/schema';
import { TeamMemberCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamMemberCollectionName } from '@fastgpt/global/support/user/team/constant';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
...@@ -25,7 +25,7 @@ const AppRegistrationSchema = new Schema({ ...@@ -25,7 +25,7 @@ const AppRegistrationSchema = new Schema({
} }
}); });
AppRegistrationSchema.index({ teamId: 1 }); defineIndex(AppRegistrationSchema, { key: { teamId: 1 } });
export const MongoAppRegistration = getMongoModel( export const MongoAppRegistration = getMongoModel(
AppRegistrationCollectionName, AppRegistrationCollectionName,
......
packages/service/support/mcp/schema.ts
...@@ -2,7 +2,7 @@ import { ...@@ -2,7 +2,7 @@ import {
TeamCollectionName, TeamCollectionName,
TeamMemberCollectionName TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant'; } from '@fastgpt/global/support/user/team/constant';
import { Schema, getMongoModel } from '../../common/mongo'; import { defineIndex, Schema, getMongoModel } from '../../common/mongo';
import { type McpKeyType } from '@fastgpt/global/support/mcp/type'; import { type McpKeyType } from '@fastgpt/global/support/mcp/type';
import { getNanoid } from '@fastgpt/global/common/string/tools'; import { getNanoid } from '@fastgpt/global/common/string/tools';
import { AppCollectionName } from '../../core/app/schema'; import { AppCollectionName } from '../../core/app/schema';
...@@ -17,7 +17,6 @@ const McpKeySchema = new Schema({ ...@@ -17,7 +17,6 @@ const McpKeySchema = new Schema({
key: { key: {
type: String, type: String,
required: true, required: true,
unique: true,
default: () => getNanoid(24) default: () => getNanoid(24)
}, },
teamId: { teamId: {
...@@ -53,4 +52,9 @@ const McpKeySchema = new Schema({ ...@@ -53,4 +52,9 @@ const McpKeySchema = new Schema({
} }
}); });
defineIndex(McpKeySchema, {
key: { key: 1 },
options: { unique: true }
});
export const MongoMcpKey = getMongoModel<McpKeyType>(mcpCollectionName, McpKeySchema); export const MongoMcpKey = getMongoModel<McpKeyType>(mcpCollectionName, McpKeySchema);
packages/service/support/openapi/schema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type OpenApiSchema } from '@fastgpt/global/support/openapi/type'; import { type OpenApiSchema } from '@fastgpt/global/support/openapi/type';
import { import {
TeamCollectionName, TeamCollectionName,
TeamMemberCollectionName TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant'; } from '@fastgpt/global/support/user/team/constant';
import { getLogger, LogCategories } from '../../common/logger';
const OpenApiSchema = new Schema( const OpenApiSchema = new Schema(
{ {
...@@ -70,15 +69,14 @@ const OpenApiSchema = new Schema( ...@@ -70,15 +69,14 @@ const OpenApiSchema = new Schema(
} }
); );
try { defineIndex(OpenApiSchema, { key: { teamId: 1 } });
OpenApiSchema.index({ teamId: 1 }); defineIndex(OpenApiSchema, { key: { apiKey: 1 } });
OpenApiSchema.index({ apiKey: 1 }); defineIndex(OpenApiSchema, {
OpenApiSchema.index({ teamId: 1, tmbId: 1, tagIds: 1, _id: -1 }); key: { teamId: 1, tmbId: 1, tagIds: 1, _id: -1 }
OpenApiSchema.index({ teamId: 1, tmbId: 1, appId: 1, _id: -1 }); });
OpenApiSchema.index({ teamId: 1, tmbId: 1, name: 1 }); defineIndex(OpenApiSchema, {
} catch (error) { key: { teamId: 1, tmbId: 1, appId: 1, _id: -1 }
const logger = getLogger(LogCategories.INFRA.MONGO); });
logger.error('Failed to build OpenAPI indexes', { error }); defineIndex(OpenApiSchema, { key: { teamId: 1, tmbId: 1, name: 1 } });
}
export const MongoOpenApi = getMongoModel<OpenApiSchema>('openapi', OpenApiSchema); export const MongoOpenApi = getMongoModel<OpenApiSchema>('openapi', OpenApiSchema);
packages/service/support/openapi/tag/schema.ts
...@@ -2,8 +2,7 @@ import { ...@@ -2,8 +2,7 @@ import {
TeamCollectionName, TeamCollectionName,
TeamMemberCollectionName TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant'; } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { getLogger, LogCategories } from '../../../common/logger';
import type { OpenApiTagType } from '@fastgpt/global/openapi/support/openapi/tag'; import type { OpenApiTagType } from '@fastgpt/global/openapi/support/openapi/tag';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
...@@ -55,13 +54,13 @@ const OpenApiTagSchema = new Schema({ ...@@ -55,13 +54,13 @@ const OpenApiTagSchema = new Schema({
} }
}); });
try { defineIndex(OpenApiTagSchema, {
OpenApiTagSchema.index({ teamId: 1, tmbId: 1, normalizedName: 1 }, { unique: true }); key: { teamId: 1, tmbId: 1, normalizedName: 1 },
OpenApiTagSchema.index({ teamId: 1, tmbId: 1, type: 1, order: 1 }); options: { unique: true }
} catch (error) { });
const logger = getLogger(LogCategories.INFRA.MONGO); defineIndex(OpenApiTagSchema, {
logger.error('Failed to build OpenAPI tag indexes', { error }); key: { teamId: 1, tmbId: 1, type: 1, order: 1 }
} });
export const MongoOpenApiTag = getMongoModel<OpenApiTagSchemaType>( export const MongoOpenApiTag = getMongoModel<OpenApiTagSchemaType>(
OpenApiTagCollectionName, OpenApiTagCollectionName,
......
packages/service/support/outLink/schema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type OutLinkSchemaType } from '@fastgpt/global/support/outLink/type'; import { type OutLinkSchemaType } from '@fastgpt/global/support/outLink/type';
import { import {
...@@ -111,12 +111,12 @@ OutLinkSchema.virtual('associatedApp', { ...@@ -111,12 +111,12 @@ OutLinkSchema.virtual('associatedApp', {
const logger = getLogger(LogCategories.INFRA.MONGO); const logger = getLogger(LogCategories.INFRA.MONGO);
OutLinkSchema.index({ shareId: -1 }); defineIndex(OutLinkSchema, { key: { shareId: -1 } });
OutLinkSchema.index({ teamId: 1, tmbId: 1, appId: 1 }); defineIndex(OutLinkSchema, { key: { teamId: 1, tmbId: 1, appId: 1 } });
// Wechat polling recovery: find online channels on startup // Wechat polling recovery: find online channels on startup
OutLinkSchema.index( defineIndex(OutLinkSchema, {
{ type: 1, 'app.status': 1 }, key: { type: 1, 'app.status': 1 },
{ partialFilterExpression: { type: 'wechat', 'app.status': 'online' } } options: { partialFilterExpression: { type: 'wechat', 'app.status': 'online' } }
); });
export const MongoOutLink = getMongoModel<OutLinkSchemaType>('outlinks', OutLinkSchema); export const MongoOutLink = getMongoModel<OutLinkSchemaType>('outlinks', OutLinkSchema);
packages/service/support/permission/memberGroup/groupMemberSchema.ts
import { TeamMemberCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamMemberCollectionName } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { MemberGroupCollectionName } from './memberGroupSchema'; import { MemberGroupCollectionName } from './memberGroupSchema';
import { type GroupMemberSchemaType } from '@fastgpt/global/support/permission/memberGroup/type'; import { type GroupMemberSchemaType } from '@fastgpt/global/support/permission/memberGroup/type';
import { GroupMemberRole } from '@fastgpt/global/support/permission/memberGroup/constant'; import { GroupMemberRole } from '@fastgpt/global/support/permission/memberGroup/constant';
...@@ -36,17 +36,17 @@ GroupMemberSchema.virtual('group', { ...@@ -36,17 +36,17 @@ GroupMemberSchema.virtual('group', {
const logger = getLogger(LogCategories.INFRA.MONGO); const logger = getLogger(LogCategories.INFRA.MONGO);
try { defineIndex(GroupMemberSchema, {
GroupMemberSchema.index({ key: {
groupId: 1 groupId: 1
}); }
});
GroupMemberSchema.index({ defineIndex(GroupMemberSchema, {
key: {
tmbId: 1 tmbId: 1
}); }
} catch (error) { });
logger.error('Failed to build group member indexes', { error });
}
export const MongoGroupMemberModel = getMongoModel<GroupMemberSchemaType>( export const MongoGroupMemberModel = getMongoModel<GroupMemberSchemaType>(
GroupMemberCollectionName, GroupMemberCollectionName,
......
packages/service/support/permission/memberGroup/memberGroupSchema.ts
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { type MemberGroupSchemaType } from '@fastgpt/global/support/permission/memberGroup/type'; import { type MemberGroupSchemaType } from '@fastgpt/global/support/permission/memberGroup/type';
import { getLogger, LogCategories } from '../../../common/logger'; import { getLogger, LogCategories } from '../../../common/logger';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
...@@ -35,19 +35,15 @@ export const MemberGroupSchema = new Schema( ...@@ -35,19 +35,15 @@ export const MemberGroupSchema = new Schema(
const logger = getLogger(LogCategories.INFRA.MONGO); const logger = getLogger(LogCategories.INFRA.MONGO);
try { defineIndex(MemberGroupSchema, {
MemberGroupSchema.index( key: {
{ teamId: 1,
teamId: 1, name: 1
name: 1 },
}, options: {
{ unique: true
unique: true }
} });
);
} catch (error) {
logger.error('Failed to build member group indexes', { error });
}
export const MongoMemberGroupModel = getMongoModel<MemberGroupSchemaType>( export const MongoMemberGroupModel = getMongoModel<MemberGroupSchemaType>(
MemberGroupCollectionName, MemberGroupCollectionName,
......
packages/service/support/permission/org/orgMemberSchema.ts
import { OrgCollectionName } from '@fastgpt/global/support/user/team/org/constant'; import { OrgCollectionName } from '@fastgpt/global/support/user/team/org/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { import {
TeamCollectionName, TeamCollectionName,
TeamMemberCollectionName TeamMemberCollectionName
...@@ -43,24 +43,22 @@ OrgMemberSchema.virtual('org', { ...@@ -43,24 +43,22 @@ OrgMemberSchema.virtual('org', {
const logger = getLogger(LogCategories.INFRA.MONGO); const logger = getLogger(LogCategories.INFRA.MONGO);
try { defineIndex(OrgMemberSchema, {
OrgMemberSchema.index( key: {
{
teamId: 1,
orgId: 1,
tmbId: 1
},
{
unique: true
}
);
OrgMemberSchema.index({
teamId: 1, teamId: 1,
orgId: 1,
tmbId: 1 tmbId: 1
}); },
} catch (error) { options: {
logger.error('Failed to build org member indexes', { error }); unique: true
} }
});
defineIndex(OrgMemberSchema, {
key: {
teamId: 1,
tmbId: 1
}
});
export const MongoOrgMemberModel = getMongoModel<OrgMemberSchemaType>( export const MongoOrgMemberModel = getMongoModel<OrgMemberSchemaType>(
OrgMemberCollectionName, OrgMemberCollectionName,
......
packages/service/support/permission/org/orgSchema.ts
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { OrgCollectionName } from '@fastgpt/global/support/user/team/org/constant'; import { OrgCollectionName } from '@fastgpt/global/support/user/team/org/constant';
import type { OrgSchemaType } from '@fastgpt/global/support/user/team/org/type'; import type { OrgSchemaType } from '@fastgpt/global/support/user/team/org/type';
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { OrgMemberCollectionName } from './orgMemberSchema'; import { OrgMemberCollectionName } from './orgMemberSchema';
import { getNanoid } from '@fastgpt/global/common/string/tools'; import { getNanoid } from '@fastgpt/global/common/string/tools';
import { DEFAULT_ORG_AVATAR } from '@fastgpt/global/common/system/constants'; import { DEFAULT_ORG_AVATAR } from '@fastgpt/global/common/system/constants';
...@@ -59,22 +59,20 @@ OrgSchema.virtual('members', { ...@@ -59,22 +59,20 @@ OrgSchema.virtual('members', {
const logger = getLogger(LogCategories.INFRA.MONGO); const logger = getLogger(LogCategories.INFRA.MONGO);
try { defineIndex(OrgSchema, {
OrgSchema.index({ key: {
teamId: 1, teamId: 1,
path: 1 path: 1
}); }
OrgSchema.index( });
{ defineIndex(OrgSchema, {
teamId: 1, key: {
pathId: 1 teamId: 1,
}, pathId: 1
{ },
unique: true options: {
} unique: true
); }
} catch (error) { });
logger.error('Failed to build org indexes', { error });
}
export const MongoOrgModel = getMongoModel<OrgSchemaType>(OrgCollectionName, OrgSchema); export const MongoOrgModel = getMongoModel<OrgSchemaType>(OrgCollectionName, OrgSchema);
packages/service/support/permission/schema.ts
...@@ -2,8 +2,7 @@ import { ...@@ -2,8 +2,7 @@ import {
TeamCollectionName, TeamCollectionName,
TeamMemberCollectionName TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant'; } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
import { getLogger, LogCategories } from '../../common/logger';
import type { ResourcePermissionType } from '@fastgpt/global/support/permission/type'; import type { ResourcePermissionType } from '@fastgpt/global/support/permission/type';
import { PerResourceTypeEnum } from '@fastgpt/global/support/permission/constant'; import { PerResourceTypeEnum } from '@fastgpt/global/support/permission/constant';
import { MemberGroupCollectionName } from './memberGroup/memberGroupSchema'; import { MemberGroupCollectionName } from './memberGroup/memberGroupSchema';
...@@ -78,169 +77,166 @@ ResourcePermissionSchema.virtual('org', { ...@@ -78,169 +77,166 @@ ResourcePermissionSchema.virtual('org', {
justOne: true justOne: true
}); });
try { defineIndex(ResourcePermissionSchema, {
ResourcePermissionSchema.index({ key: {
resourceType: 1, resourceType: 1,
teamId: 1 teamId: 1
}); }
});
// Indexes for resourceId-based resources
ResourcePermissionSchema.index( // Indexes for resourceId-based resources
{ defineIndex(ResourcePermissionSchema, {
resourceType: 1, key: {
teamId: 1, resourceType: 1,
resourceId: 1, teamId: 1,
groupId: 1 resourceId: 1,
}, groupId: 1
{ },
unique: true, options: {
partialFilterExpression: { unique: true,
groupId: { partialFilterExpression: {
$exists: true groupId: {
}, $exists: true
resourceId: { },
$exists: true resourceId: {
} $exists: true
} }
} }
); }
});
ResourcePermissionSchema.index(
{ defineIndex(ResourcePermissionSchema, {
resourceType: 1, key: {
teamId: 1, resourceType: 1,
resourceId: 1, teamId: 1,
orgId: 1 resourceId: 1,
}, orgId: 1
{ },
unique: true, options: {
partialFilterExpression: { unique: true,
orgId: { partialFilterExpression: {
$exists: true orgId: {
}, $exists: true
resourceId: { },
$exists: true resourceId: {
} $exists: true
} }
} }
); }
});
ResourcePermissionSchema.index(
{ defineIndex(ResourcePermissionSchema, {
resourceType: 1, key: {
teamId: 1, resourceType: 1,
resourceId: 1, teamId: 1,
tmbId: 1 resourceId: 1,
}, tmbId: 1
{ },
unique: true, options: {
partialFilterExpression: { unique: true,
tmbId: { partialFilterExpression: {
$exists: true tmbId: {
}, $exists: true
resourceId: { },
$exists: true resourceId: {
} $exists: true
} }
} }
); }
});
// General index for resourceId-based resources
ResourcePermissionSchema.index( // General index for resourceId-based resources
{ defineIndex(ResourcePermissionSchema, {
resourceType: 1, key: {
teamId: 1, resourceType: 1,
resourceId: 1 teamId: 1,
}, resourceId: 1
{ },
partialFilterExpression: { options: {
resourceId: { partialFilterExpression: {
$exists: true resourceId: {
} $exists: true
} }
} }
); }
});
// Indexes for resourceName-based resources
ResourcePermissionSchema.index( // Indexes for resourceName-based resources
{ defineIndex(ResourcePermissionSchema, {
resourceType: 1, key: {
teamId: 1, resourceType: 1,
resourceName: 1, teamId: 1,
groupId: 1 resourceName: 1,
}, groupId: 1
{ },
unique: true, options: {
partialFilterExpression: { unique: true,
groupId: { partialFilterExpression: {
$exists: true groupId: {
}, $exists: true
resourceName: { },
$exists: true resourceName: {
} $exists: true
} }
} }
); }
});
ResourcePermissionSchema.index(
{ defineIndex(ResourcePermissionSchema, {
resourceType: 1, key: {
teamId: 1, resourceType: 1,
resourceName: 1, teamId: 1,
orgId: 1 resourceName: 1,
}, orgId: 1
{ },
unique: true, options: {
partialFilterExpression: { unique: true,
orgId: { partialFilterExpression: {
$exists: true orgId: {
}, $exists: true
resourceName: { },
$exists: true resourceName: {
} $exists: true
} }
} }
); }
});
ResourcePermissionSchema.index(
{ defineIndex(ResourcePermissionSchema, {
resourceType: 1, key: {
teamId: 1, resourceType: 1,
resourceName: 1, teamId: 1,
tmbId: 1 resourceName: 1,
}, tmbId: 1
{ },
unique: true, options: {
partialFilterExpression: { unique: true,
tmbId: { partialFilterExpression: {
$exists: true tmbId: {
}, $exists: true
resourceName: { },
$exists: true resourceName: {
} $exists: true
} }
} }
); }
});
// General index for resourceName-based resources
ResourcePermissionSchema.index( // General index for resourceName-based resources
{ defineIndex(ResourcePermissionSchema, {
resourceType: 1, key: {
teamId: 1, resourceType: 1,
resourceName: 1 teamId: 1,
}, resourceName: 1
{ },
partialFilterExpression: { options: {
resourceName: { partialFilterExpression: {
$exists: true resourceName: {
} $exists: true
} }
} }
); }
} catch (error) { });
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build permission indexes', { error });
}
ResourcePermissionSchema.pre('save', function (next) { ResourcePermissionSchema.pre('save', function (next) {
if (!this.tmbId && !this.groupId && !this.orgId) { if (!this.tmbId && !this.groupId && !this.orgId) {
......
packages/service/support/tmpData/schema.ts
import { getMongoModel, Schema } from '../../common/mongo'; import { defineIndex, getMongoModel, Schema } from '../../common/mongo';
import type { TmpDataSchema as SchemaType } from '@fastgpt/global/support/tmpData/type'; import type { TmpDataSchema as SchemaType } from '@fastgpt/global/support/tmpData/type';
import { getLogger, LogCategories } from '../../common/logger';
const collectionName = 'tmp_datas'; const collectionName = 'tmp_datas';
const TmpDataSchema = new Schema({ const TmpDataSchema = new Schema({
dataId: { dataId: {
type: String, type: String,
required: true, required: true
unique: true
}, },
data: { data: {
type: Object type: Object
...@@ -19,12 +17,14 @@ const TmpDataSchema = new Schema({ ...@@ -19,12 +17,14 @@ const TmpDataSchema = new Schema({
} }
}); });
try { defineIndex(TmpDataSchema, {
TmpDataSchema.index({ dataId: -1 }); key: { dataId: 1 },
TmpDataSchema.index({ expireAt: -1 }, { expireAfterSeconds: 5 }); options: { unique: true }
} catch (error) { });
const logger = getLogger(LogCategories.INFRA.MONGO); defineIndex(TmpDataSchema, { key: { dataId: -1 } });
logger.error('Failed to build tmp data indexes', { error }); defineIndex(TmpDataSchema, {
} key: { expireAt: -1 },
options: { expireAfterSeconds: 5 }
});
export const MongoTmpData = getMongoModel<SchemaType<Object>>(collectionName, TmpDataSchema); export const MongoTmpData = getMongoModel<SchemaType<object>>(collectionName, TmpDataSchema);
packages/service/support/user/audit/schema.ts
import { Schema, getMongoLogModel } from '../../../common/mongo'; import { defineIndex, Schema, getMongoLogModel } from '../../../common/mongo';
import { type TeamAuditSchemaType } from '@fastgpt/global/support/user/audit/type'; import { type TeamAuditSchemaType } from '@fastgpt/global/support/user/audit/type';
import { AdminAuditEventEnum, AuditEventEnum } from '@fastgpt/global/support/user/audit/constants'; import { AdminAuditEventEnum, AuditEventEnum } from '@fastgpt/global/support/user/audit/constants';
import { import {
...@@ -34,8 +34,8 @@ const TeamAuditSchema = new Schema({ ...@@ -34,8 +34,8 @@ const TeamAuditSchema = new Schema({
} }
}); });
TeamAuditSchema.index({ teamId: 1, tmbId: 1, event: 1 }); defineIndex(TeamAuditSchema, { key: { teamId: 1, tmbId: 1, event: 1 } });
TeamAuditSchema.index({ timestamp: 1, teamId: 1 }); defineIndex(TeamAuditSchema, { key: { timestamp: 1, teamId: 1 } });
export const MongoTeamAudit = getMongoLogModel<TeamAuditSchemaType>( export const MongoTeamAudit = getMongoLogModel<TeamAuditSchemaType>(
TeamAuditCollectionName, TeamAuditCollectionName,
......
packages/service/support/user/auth/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import type { UserAuthSchemaType } from '@fastgpt/global/support/user/auth/type'; import type { UserAuthSchemaType } from '@fastgpt/global/support/user/auth/type';
import { userAuthTypeMap } from '@fastgpt/global/support/user/auth/constants'; import { userAuthTypeMap } from '@fastgpt/global/support/user/auth/constants';
import { addMinutes } from 'date-fns'; import { addMinutes } from 'date-fns';
import { getLogger, LogCategories } from '../../../common/logger';
const UserAuthSchema = new Schema({ const UserAuthSchema = new Schema({
key: { key: {
...@@ -32,12 +31,10 @@ const UserAuthSchema = new Schema({ ...@@ -32,12 +31,10 @@ const UserAuthSchema = new Schema({
} }
}); });
try { defineIndex(UserAuthSchema, { key: { key: 1, type: 1 } });
UserAuthSchema.index({ key: 1, type: 1 }); defineIndex(UserAuthSchema, {
UserAuthSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 0 }); key: { expiredTime: 1 },
} catch (error) { options: { expireAfterSeconds: 0 }
const logger = getLogger(LogCategories.INFRA.MONGO); });
logger.error('Failed to build user auth indexes', { error });
}
export const MongoUserAuth = getMongoModel<UserAuthSchemaType>('auth_codes', UserAuthSchema); export const MongoUserAuth = getMongoModel<UserAuthSchemaType>('auth_codes', UserAuthSchema);
packages/service/support/user/schema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { hashStr } from '@fastgpt/global/common/string/tools'; import { hashStr } from '@fastgpt/global/common/string/tools';
import { UserTagsSchema, type UserModelSchema } from '@fastgpt/global/support/user/type'; import { UserTagsSchema, type UserModelSchema } from '@fastgpt/global/support/user/type';
import { UserStatusEnum, userStatusMap } from '@fastgpt/global/support/user/constant'; import { UserStatusEnum, userStatusMap } from '@fastgpt/global/support/user/constant';
import { TeamMemberCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamMemberCollectionName } from '@fastgpt/global/support/user/team/constant';
import { LangEnum } from '@fastgpt/global/common/i18n/type'; import { LangEnum } from '@fastgpt/global/common/i18n/type';
import { getLogger, LogCategories } from '../../common/logger';
export const userCollectionName = 'users'; export const userCollectionName = 'users';
...@@ -18,8 +17,7 @@ const UserSchema = new Schema({ ...@@ -18,8 +17,7 @@ const UserSchema = new Schema({
username: { username: {
// 可以是手机/邮箱，新的验证都只用手机 // 可以是手机/邮箱，新的验证都只用手机
type: String, type: String,
required: true, required: true
unique: true // 唯一
}, },
password: { password: {
type: String, type: String,
...@@ -75,12 +73,12 @@ const UserSchema = new Schema({ ...@@ -75,12 +73,12 @@ const UserSchema = new Schema({
avatar: String avatar: String
}); });
try { // username 唯一。
// Admin charts defineIndex(UserSchema, {
UserSchema.index({ createTime: -1 }); key: { username: 1 },
} catch (error) { options: { unique: true }
const logger = getLogger(LogCategories.INFRA.MONGO); });
logger.error('Failed to build user indexes', { error }); // Admin charts
} defineIndex(UserSchema, { key: { createTime: -1 } });
export const MongoUser = getMongoModel<UserModelSchema>(userCollectionName, UserSchema); export const MongoUser = getMongoModel<UserModelSchema>(userCollectionName, UserSchema);
packages/service/support/user/team/teamMemberSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type TeamMemberSchema as TeamMemberType } from '@fastgpt/global/support/user/team/type'; import { type TeamMemberSchema as TeamMemberType } from '@fastgpt/global/support/user/team/type';
import { userCollectionName } from '../../user/schema'; import { userCollectionName } from '../../user/schema';
...@@ -8,7 +8,6 @@ import { ...@@ -8,7 +8,6 @@ import {
TeamCollectionName TeamCollectionName
} from '@fastgpt/global/support/user/team/constant'; } from '@fastgpt/global/support/user/team/constant';
import { getRandomUserAvatar } from '@fastgpt/global/support/user/utils'; import { getRandomUserAvatar } from '@fastgpt/global/support/user/utils';
import { getLogger, LogCategories } from '../../../common/logger';
const TeamMemberSchema = new Schema({ const TeamMemberSchema = new Schema({
teamId: { teamId: {
...@@ -67,13 +66,14 @@ TeamMemberSchema.virtual('user', { ...@@ -67,13 +66,14 @@ TeamMemberSchema.virtual('user', {
justOne: true justOne: true
}); });
try { defineIndex(TeamMemberSchema, {
TeamMemberSchema.index({ teamId: 1 }, { background: true }); key: { teamId: 1 },
TeamMemberSchema.index({ userId: 1 }, { background: true }); options: { background: true }
} catch (error) { });
const logger = getLogger(LogCategories.INFRA.MONGO); defineIndex(TeamMemberSchema, {
logger.error('Failed to build team member indexes', { error }); key: { userId: 1 },
} options: { background: true }
});
export const MongoTeamMember = getMongoModel<TeamMemberType>( export const MongoTeamMember = getMongoModel<TeamMemberType>(
TeamMemberCollectionName, TeamMemberCollectionName,
......
packages/service/support/user/team/teamSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type TeamSchema as TeamType } from '@fastgpt/global/support/user/team/type'; import { type TeamSchema as TeamType } from '@fastgpt/global/support/user/team/type';
import { userCollectionName } from '../../user/schema'; import { userCollectionName } from '../../user/schema';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { getLogger, LogCategories } from '../../../common/logger';
const TeamSchema = new Schema({ const TeamSchema = new Schema({
name: { name: {
...@@ -53,13 +52,11 @@ const TeamSchema = new Schema({ ...@@ -53,13 +52,11 @@ const TeamSchema = new Schema({
} }
}); });
try { defineIndex(TeamSchema, { key: { name: 1 } });
TeamSchema.index({ name: 1 }); defineIndex(TeamSchema, { key: { ownerId: 1 } });
TeamSchema.index({ ownerId: 1 }); defineIndex(TeamSchema, {
TeamSchema.index({ 'meta.wecom.corpId': 1 }, { sparse: true, unique: true }); key: { 'meta.wecom.corpId': 1 },
} catch (error) { options: { sparse: true, unique: true }
const logger = getLogger(LogCategories.INFRA.MONGO); });
logger.error('Failed to build team indexes', { error });
}
export const MongoTeam = getMongoModel<TeamType>(TeamCollectionName, TeamSchema); export const MongoTeam = getMongoModel<TeamType>(TeamCollectionName, TeamSchema);
packages/service/support/wallet/coupon/schema.ts
import { addDays } from 'date-fns'; import { addDays } from 'date-fns';
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import type { TeamCouponSchemaType } from '@fastgpt/global/support/wallet/sub/coupon/type'; import type { TeamCouponSchemaType } from '@fastgpt/global/support/wallet/sub/coupon/type';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { CouponTypeEnum } from '@fastgpt/global/support/wallet/sub/coupon/constants'; import { CouponTypeEnum } from '@fastgpt/global/support/wallet/sub/coupon/constants';
import { getLogger, LogCategories } from '../../../common/logger';
export const couponCollectionName = 'team_sub_coupons'; export const couponCollectionName = 'team_sub_coupons';
...@@ -43,12 +42,7 @@ const CouponSchema = new Schema({ ...@@ -43,12 +42,7 @@ const CouponSchema = new Schema({
} }
}); });
try { defineIndex(CouponSchema, { key: { key: 1 }, options: { unique: true } });
CouponSchema.index({ key: 1 }, { unique: true });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build coupon indexes', { error });
}
export const MongoTeamCoupon = getMongoModel<TeamCouponSchemaType>( export const MongoTeamCoupon = getMongoModel<TeamCouponSchemaType>(
couponCollectionName, couponCollectionName,
......
packages/service/support/wallet/discountCoupon/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { DiscountCouponTypeEnum } from '@fastgpt/global/support/wallet/sub/discountCoupon/constants'; import { DiscountCouponTypeEnum } from '@fastgpt/global/support/wallet/sub/discountCoupon/constants';
import type { DiscountCouponSchemaType } from '@fastgpt/global/openapi/support/wallet/discountCoupon/api'; import type { DiscountCouponSchemaType } from '@fastgpt/global/openapi/support/wallet/discountCoupon/api';
import { getLogger, LogCategories } from '../../../common/logger';
export const discountCouponCollectionName = 'team_discount_coupons'; export const discountCouponCollectionName = 'team_discount_coupons';
...@@ -30,13 +29,8 @@ const DiscountCouponSchema = new Schema({ ...@@ -30,13 +29,8 @@ const DiscountCouponSchema = new Schema({
} }
}); });
try { defineIndex(DiscountCouponSchema, { key: { status: 1, type: 1 } });
DiscountCouponSchema.index({ status: 1, type: 1 }); defineIndex(DiscountCouponSchema, { key: { teamId: 1, status: 1 } });
DiscountCouponSchema.index({ teamId: 1, status: 1 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build discount coupon indexes', { error });
}
export const MongoDiscountCoupon = getMongoModel<DiscountCouponSchemaType>( export const MongoDiscountCoupon = getMongoModel<DiscountCouponSchemaType>(
discountCouponCollectionName, discountCouponCollectionName,
......
packages/service/support/wallet/sub/schema.ts
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
1. type=standard: There will only be 1, and each team will have one 1. type=standard: There will only be 1, and each team will have one
2. type=extraDatasetSize/extraPoints: Can buy multiple 2. type=extraDatasetSize/extraPoints: Can buy multiple
*/ */
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { import {
...@@ -12,7 +12,6 @@ import { ...@@ -12,7 +12,6 @@ import {
SubTypeEnum SubTypeEnum
} from '@fastgpt/global/support/wallet/sub/constants'; } from '@fastgpt/global/support/wallet/sub/constants';
import type { TeamSubSchemaType } from '@fastgpt/global/support/wallet/sub/type'; import type { TeamSubSchemaType } from '@fastgpt/global/support/wallet/sub/type';
import { getLogger, LogCategories } from '../../../common/logger';
export const subCollectionName = 'team_subscriptions'; export const subCollectionName = 'team_subscriptions';
...@@ -81,30 +80,27 @@ const SubSchema = new Schema({ ...@@ -81,30 +80,27 @@ const SubSchema = new Schema({
currentExtraDatasetSize: Number currentExtraDatasetSize: Number
}); });
try { // Get plan by expiredTime
// Get plan by expiredTime defineIndex(SubSchema, { key: { expiredTime: -1, currentSubLevel: 1 } });
SubSchema.index({ expiredTime: -1, currentSubLevel: 1 });
// Get team plan // Get team plan
SubSchema.index({ teamId: 1, type: 1, expiredTime: -1 }); defineIndex(SubSchema, { key: { teamId: 1, type: 1, expiredTime: -1 } });
// timer task. Get standard plan;Get free plan;Clear expired extract plan // timer task. Get standard plan;Get free plan;Clear expired extract plan
SubSchema.index({ type: 1, expiredTime: -1, currentSubLevel: 1 }); defineIndex(SubSchema, {
key: { type: 1, expiredTime: -1, currentSubLevel: 1 }
});
// 修改后的唯一索引 // 修改后的唯一索引
SubSchema.index( defineIndex(SubSchema, {
{ key: {
teamId: 1, teamId: 1,
type: 1, type: 1,
currentSubLevel: 1 currentSubLevel: 1
}, },
{ options: {
unique: true, unique: true,
partialFilterExpression: { type: SubTypeEnum.standard } partialFilterExpression: { type: SubTypeEnum.standard }
} }
); });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build subscription indexes', { error });
}
export const MongoTeamSub = getMongoModel<TeamSubSchemaType>(subCollectionName, SubSchema); export const MongoTeamSub = getMongoModel<TeamSubSchemaType>(subCollectionName, SubSchema);
packages/service/support/wallet/usage/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import { type UsageSchemaType } from '@fastgpt/global/support/wallet/usage/type'; import { type UsageSchemaType } from '@fastgpt/global/support/wallet/usage/type';
import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants'; import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants';
...@@ -9,7 +9,6 @@ import { ...@@ -9,7 +9,6 @@ import {
import { UsageCollectionName, UsageItemCollectionName } from './constants'; import { UsageCollectionName, UsageItemCollectionName } from './constants';
import { AppCollectionName } from '../../../core/app/schema'; import { AppCollectionName } from '../../../core/app/schema';
import { DatasetCollectionName } from '../../../core/dataset/schema'; import { DatasetCollectionName } from '../../../core/dataset/schema';
import { getLogger, LogCategories } from '../../../common/logger';
import { agentSkillsCollectionName } from '@fastgpt/global/core/ai/skill/constants'; import { agentSkillsCollectionName } from '@fastgpt/global/core/ai/skill/constants';
const UsageSchema = new Schema( const UsageSchema = new Schema(
...@@ -76,13 +75,13 @@ UsageSchema.virtual('usageItems', { ...@@ -76,13 +75,13 @@ UsageSchema.virtual('usageItems', {
foreignField: 'usageId' foreignField: 'usageId'
}); });
try { defineIndex(UsageSchema, {
UsageSchema.index({ teamId: 1, tmbId: 1, source: 1, time: 1, appName: 1, _id: -1 }); key: { teamId: 1, tmbId: 1, source: 1, time: 1, appName: 1, _id: -1 }
});
UsageSchema.index({ time: 1 }, { expireAfterSeconds: 360 * 24 * 60 * 60 }); defineIndex(UsageSchema, {
} catch (error) { key: { time: 1 },
const logger = getLogger(LogCategories.INFRA.MONGO); options: { expireAfterSeconds: 360 * 24 * 60 * 60 }
logger.error('Failed to build usage indexes', { error }); });
}
export const MongoUsage = getMongoModel<UsageSchemaType>(UsageCollectionName, UsageSchema); export const MongoUsage = getMongoModel<UsageSchemaType>(UsageCollectionName, UsageSchema);
packages/service/support/wallet/usage/usageItemSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo'; import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo; const { Schema } = connectionMongo;
import type { UsageItemSchemaType } from '@fastgpt/global/support/wallet/usage/type'; import type { UsageItemSchemaType } from '@fastgpt/global/support/wallet/usage/type';
import { UsageCollectionName, UsageItemCollectionName } from './constants'; import { UsageCollectionName, UsageItemCollectionName } from './constants';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant'; import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { getLogger, LogCategories } from '../../../common/logger';
const UsageItemSchema = new Schema({ const UsageItemSchema = new Schema({
teamId: { teamId: {
...@@ -41,13 +40,11 @@ const UsageItemSchema = new Schema({ ...@@ -41,13 +40,11 @@ const UsageItemSchema = new Schema({
model: String model: String
}); });
try { defineIndex(UsageItemSchema, { key: { usageId: 'hashed' } });
UsageItemSchema.index({ usageId: 'hashed' }); defineIndex(UsageItemSchema, {
UsageItemSchema.index({ time: 1 }, { expireAfterSeconds: 360 * 24 * 60 * 60 }); key: { time: 1 },
} catch (error) { options: { expireAfterSeconds: 360 * 24 * 60 * 60 }
const logger = getLogger(LogCategories.INFRA.MONGO); });
logger.error('Failed to build usage item indexes', { error });
}
export const MongoUsageItem = getMongoModel<UsageItemSchemaType>( export const MongoUsageItem = getMongoModel<UsageItemSchemaType>(
UsageItemCollectionName, UsageItemCollectionName,
......
packages/service/test/common/mongo/indexManager.test.ts 0 → 100644
import { randomUUID } from 'node:crypto';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { connectionMongo, defineIndex, Schema } from '@fastgpt/service/common/mongo';
import type { DeprecatedMongoIndexDefinition } from '@fastgpt/service/common/mongo/schemaIndexes';
import { MongoIndexManager } from '@fastgpt/service/common/mongo/indexManager';
const logger = {
debug: vi.fn(),
info: vi.fn(),
warn: vi.fn(),
error: vi.fn()
};
const createModel = ({
schema,
prefix = 'MongoIndexManager'
}: {
schema: InstanceType<typeof Schema>;
prefix?: string;
}) => {
const suffix = randomUUID().replaceAll('-', '');
return connectionMongo.model(`${prefix}${suffix}`, schema, `${prefix.toLowerCase()}_${suffix}`);
};
const getIndexNames = async (model: ReturnType<typeof createModel>) =>
new Set((await model.collection.indexes()).map((index) => index.name));
const defineDeprecatedTestIndexes = (
schema: InstanceType<typeof Schema>,
indexes: DeprecatedMongoIndexDefinition[]
) => {
indexes.forEach(({ indexName, key, options }) => {
defineIndex(schema, {
key,
options: { ...options, name: indexName },
deprecated: true
});
});
};
const legacyDefinition = {
indexName: 'legacy_field_1',
key: { legacyField: 1 }
} as const;
describe('MongoIndexManager.syncModelIndexes', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('creates current indexes, removes declared legacy indexes, and preserves customer indexes', async () => {
const schema = new Schema(
{
currentField: String,
legacyField: String,
customerField: String
},
{ autoIndex: false }
);
defineIndex(schema, {
key: { currentField: 1 },
options: { name: 'current_field_1' }
});
defineDeprecatedTestIndexes(schema, [legacyDefinition]);
const model = createModel({ schema });
await model.collection.createIndex({ legacyField: 1 }, { name: 'legacy_field_1' });
await model.collection.createIndex({ customerField: 1 }, { name: 'customer_custom_1' });
const result = await MongoIndexManager.syncModelIndexes({ model, logger });
const indexNames = await getIndexNames(model);
expect(indexNames).toContain('current_field_1');
expect(indexNames).toContain('customer_custom_1');
expect(indexNames).not.toContain('legacy_field_1');
expect(result.cleanupReport.items).toEqual([
expect.objectContaining({
action: 'drop',
applied: true,
collectionName: model.collection.collectionName,
indexName: 'legacy_field_1'
})
]);
expect(logger.warn).toHaveBeenCalledWith(
'Detected MongoDB indexes not declared by FastGPT schema',
{
collectionName: model.collection.collectionName,
indexNames: expect.arrayContaining(['legacy_field_1', 'customer_custom_1'])
}
);
expect(logger.info).toHaveBeenCalledWith('MongoDB indexes synchronized', {
collectionName: model.collection.collectionName,
created: 1,
dropped: 1
});
});
it('does not delete schema-external indexes when the Schema has no deprecated declarations', async () => {
const schema = new Schema(
{ currentField: String, customerField: String },
{ autoIndex: false }
);
defineIndex(schema, {
key: { currentField: 1 },
options: { name: 'current_field_1' }
});
const model = createModel({ schema });
await model.collection.createIndex({ currentField: 1 }, { name: 'current_field_1' });
await model.collection.createIndex({ customerField: 1 }, { name: 'customer_custom_1' });
const result = await MongoIndexManager.syncModelIndexes({ model, logger });
expect(await getIndexNames(model)).toContain('customer_custom_1');
expect(result.cleanupReport.items).toEqual([]);
expect(logger.info).not.toHaveBeenCalled();
expect(logger.debug).not.toHaveBeenCalled();
});
it('reuses an in-flight task for concurrent calls on the same Model', async () => {
const schema = new Schema({ currentField: String }, { autoIndex: false });
defineIndex(schema, {
key: { currentField: 1 },
options: { name: 'current_field_1' }
});
const model = createModel({ schema });
const createIndexes = vi.spyOn(model, 'createIndexes');
const [firstResult, secondResult] = await Promise.all([
MongoIndexManager.syncModelIndexes({ model }),
MongoIndexManager.syncModelIndexes({ model })
]);
expect(firstResult).toBe(secondResult);
expect(createIndexes).toHaveBeenCalledTimes(1);
});
it('does not clean deprecated indexes when creating current indexes fails', async () => {
const schema = new Schema(
{ currentField: String, conflictingField: String, legacyField: String },
{ autoIndex: false }
);
defineIndex(schema, {
key: { currentField: 1 },
options: { name: 'current_field_1' }
});
defineDeprecatedTestIndexes(schema, [legacyDefinition]);
const model = createModel({ schema });
await model.collection.createIndex({ conflictingField: 1 }, { name: 'current_field_1' });
await model.collection.createIndex({ legacyField: 1 }, { name: 'legacy_field_1' });
await expect(MongoIndexManager.syncModelIndexes({ model })).rejects.toThrow();
expect(await getIndexNames(model)).toContain('legacy_field_1');
});
});
describe('MongoIndexManager.cleanupModelDeprecatedIndexes', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('supports dry-run without deleting a matched index', async () => {
const schema = new Schema({ legacyField: String }, { autoIndex: false });
defineDeprecatedTestIndexes(schema, [
{
...legacyDefinition,
options: { unique: true }
}
]);
const model = createModel({ schema });
await model.collection.createIndex(
{ legacyField: 1 },
{ name: 'legacy_field_1', unique: true }
);
const report = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: false,
logger
});
expect(report.items).toEqual([
expect.objectContaining({ action: 'drop', applied: false, indexName: 'legacy_field_1' })
]);
expect(await getIndexNames(model)).toContain('legacy_field_1');
expect(logger.info).not.toHaveBeenCalled();
expect(logger.debug).not.toHaveBeenCalled();
});
it('preserves same-name indexes when key or key order does not match', async () => {
const schema = new Schema(
{ customerField: String, legacyField: String, otherField: String },
{ autoIndex: false }
);
defineDeprecatedTestIndexes(schema, [
legacyDefinition,
{
indexName: 'legacy_compound_1',
key: { legacyField: 1, otherField: 1 }
}
]);
const model = createModel({ schema });
await model.collection.createIndex({ customerField: 1 }, { name: 'legacy_field_1' });
await model.collection.createIndex(
{ otherField: 1, legacyField: 1 },
{ name: 'legacy_compound_1' }
);
const report = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: true,
logger
});
expect(report.items).toEqual([
expect.objectContaining({ action: 'skip_mismatch', indexName: 'legacy_field_1' }),
expect.objectContaining({ action: 'skip_mismatch', indexName: 'legacy_compound_1' })
]);
expect(await getIndexNames(model)).toEqual(
expect.objectContaining(new Set(['_id_', 'legacy_field_1', 'legacy_compound_1']))
);
});
it('drops a deprecated index even when options differ, as long as key matches', async () => {
const schema = new Schema({ legacyField: String }, { autoIndex: false });
defineDeprecatedTestIndexes(schema, [
{
...legacyDefinition,
options: { unique: true }
}
]);
const model = createModel({ schema });
// 同名同 key 但 option 不同：只按 key 匹配，仍允许删除
await model.collection.createIndex({ legacyField: 1 }, { name: 'legacy_field_1' });
const report = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: true,
logger
});
expect(report.items).toEqual([
expect.objectContaining({ action: 'drop', applied: true, indexName: 'legacy_field_1' })
]);
expect(await getIndexNames(model)).not.toContain('legacy_field_1');
});
it('matches MongoDB text indexes via weights instead of the stored _fts key', async () => {
const schema = new Schema(
{ title: String, body: String, otherField: String },
{ autoIndex: false }
);
defineDeprecatedTestIndexes(schema, [
{
indexName: 'title_text_body_text',
key: { title: 'text', body: 'text' }
}
]);
const model = createModel({ schema });
await model.collection.createIndex(
{ title: 'text', body: 'text' },
{ name: 'title_text_body_text' }
);
const indexes = (await model.collection.indexes()) as Array<Record<string, unknown>>;
const textIndex = indexes.find((index) => index.name === 'title_text_body_text');
expect(textIndex?.key).toEqual({ _fts: 'text', _ftsx: 1 });
expect(textIndex?.weights).toEqual({ title: 1, body: 1 });
const report = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: true,
logger
});
expect(report.items).toEqual([
expect.objectContaining({
action: 'drop',
applied: true,
indexName: 'title_text_body_text'
})
]);
expect(await getIndexNames(model)).not.toContain('title_text_body_text');
});
it('skips text indexes when declared text fields do not match weights', async () => {
const schema = new Schema({ title: String, body: String }, { autoIndex: false });
defineDeprecatedTestIndexes(schema, [
{
indexName: 'title_text',
key: { title: 'text' }
}
]);
const model = createModel({ schema });
await model.collection.createIndex({ body: 'text' }, { name: 'title_text' });
const report = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: true,
logger
});
expect(report.items).toEqual([
expect.objectContaining({ action: 'skip_mismatch', indexName: 'title_text' })
]);
expect(await getIndexNames(model)).toContain('title_text');
});
it('reports a missing deprecated index and formats its report', async () => {
const schema = new Schema({ legacyField: String }, { autoIndex: false });
defineDeprecatedTestIndexes(schema, [legacyDefinition]);
const model = createModel({ schema });
const report = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: true
});
expect(report.items).toEqual([
expect.objectContaining({ action: 'skip_missing', indexName: 'legacy_field_1' })
]);
expect(MongoIndexManager.summarizeCleanupReport(report)).toEqual({
total: 1,
dropped: 0,
droppable: 0,
skippedMissing: 1,
skippedMismatch: 0,
errors: 0
});
expect(MongoIndexManager.formatCleanupReport(report)).toContain(
`${model.collection.collectionName}.legacy_field_1 reason=Deprecated index does not exist`
);
});
it('treats a concurrent IndexNotFound response as an idempotent skip', async () => {
const schema = new Schema({ legacyField: String }, { autoIndex: false });
defineDeprecatedTestIndexes(schema, [legacyDefinition]);
const model = createModel({ schema });
await model.collection.createIndex({ legacyField: 1 }, { name: 'legacy_field_1' });
vi.spyOn(model.collection, 'dropIndex').mockRejectedValueOnce({
codeName: 'IndexNotFound'
});
const report = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: true
});
expect(report.items).toEqual([
expect.objectContaining({
action: 'skip_missing',
reason: 'Deprecated index was already removed'
})
]);
});
it('recognizes the numeric MongoDB IndexNotFound code', async () => {
const schema = new Schema({ legacyField: String }, { autoIndex: false });
defineDeprecatedTestIndexes(schema, [legacyDefinition]);
const model = createModel({ schema });
await model.collection.createIndex({ legacyField: 1 }, { name: 'legacy_field_1' });
vi.spyOn(model.collection, 'dropIndex').mockRejectedValueOnce({ code: 27 });
const report = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: true
});
expect(report.items[0]).toMatchObject({
action: 'skip_missing',
reason: 'Deprecated index was already removed'
});
});
it('captures unexpected inspection errors in the cleanup report', async () => {
const schema = new Schema({ legacyField: String }, { autoIndex: false });
defineDeprecatedTestIndexes(schema, [legacyDefinition]);
const model = createModel({ schema });
vi.spyOn(model.collection, 'indexes').mockRejectedValueOnce(new Error('inspection failed'));
const report = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: true,
logger
});
expect(report.items).toEqual([
expect.objectContaining({
action: 'error',
error: 'inspection failed',
indexName: 'legacy_field_1'
})
]);
expect(logger.error).toHaveBeenCalledWith('Failed to cleanup deprecated MongoDB index', {
collectionName: model.collection.collectionName,
indexName: 'legacy_field_1',
error: 'inspection failed'
});
});
it('normalizes non-Error cleanup failures into report messages', async () => {
const schema = new Schema({ legacyField: String }, { autoIndex: false });
defineDeprecatedTestIndexes(schema, [legacyDefinition]);
const model = createModel({ schema });
await model.collection.createIndex({ legacyField: 1 }, { name: 'legacy_field_1' });
vi.spyOn(model.collection, 'dropIndex').mockRejectedValueOnce('drop failed');
const report = await MongoIndexManager.cleanupModelDeprecatedIndexes({
model,
apply: true
});
expect(report.items[0]).toMatchObject({ action: 'error', error: 'drop failed' });
});
it('summarizes every cleanup action and formats error details', () => {
const report = {
apply: true,
items: [
{
collectionName: 'test_collection',
indexName: 'legacy_drop_1',
action: 'drop' as const,
applied: false,
reason: 'Can drop',
error: 'test error'
},
{
collectionName: 'test_collection',
indexName: 'legacy_mismatch_1',
action: 'skip_mismatch' as const,
applied: false,
reason: 'Mismatch'
},
{
collectionName: 'test_collection',
indexName: 'legacy_error_1',
action: 'error' as const,
applied: false,
reason: 'Error'
}
]
};
expect(MongoIndexManager.summarizeCleanupReport(report)).toEqual({
total: 3,
dropped: 0,
droppable: 1,
skippedMissing: 0,
skippedMismatch: 1,
errors: 1
});
expect(MongoIndexManager.formatCleanupReport(report)).toContain('error=test error');
});
});
packages/service/test/common/mongo/schemaIndexes.test.ts 0 → 100644
import { describe, expect, it } from 'vitest';
import { Schema } from '@fastgpt/service/common/mongo';
import { defineIndex, getDeprecatedIndexes } from '@fastgpt/service/common/mongo/schemaIndexes';
describe('defineIndex', () => {
it('returns an empty list for a Schema without declarations', () => {
expect(getDeprecatedIndexes(new Schema())).toEqual([]);
});
it('defaults to an active index and delegates it to the Mongoose Schema', () => {
const schema = new Schema();
defineIndex(schema, {
key: { current: 1 },
options: { unique: true, name: 'current_unique' }
});
expect(schema.indexes()).toEqual([
[{ current: 1 }, { unique: true, name: 'current_unique', background: true }]
]);
expect(getDeprecatedIndexes(schema)).toEqual([]);
});
it('stores deprecated indexes without adding them to the Mongoose Schema', () => {
const schema = new Schema();
defineIndex(schema, {
key: { legacyA: 1 },
deprecated: true
});
defineIndex(schema, {
key: { teamId: 1, parentId: 1, deleteTime: 1 },
options: {},
deprecated: true
});
defineIndex(schema, {
key: { legacyB: -1 },
options: { name: 'custom_legacy_name', unique: true, background: true },
deprecated: true
});
expect(schema.indexes()).toEqual([]);
expect(getDeprecatedIndexes(schema)).toEqual([
{
indexName: 'legacyA_1',
key: { legacyA: 1 },
options: undefined
},
{
indexName: 'teamId_1_parentId_1_deleteTime_1',
key: { teamId: 1, parentId: 1, deleteTime: 1 },
options: undefined
},
{
indexName: 'custom_legacy_name',
key: { legacyB: -1 },
options: {
unique: true,
sparse: undefined,
expireAfterSeconds: undefined,
partialFilterExpression: undefined,
collation: undefined
}
}
]);
});
it('rejects duplicate index names within one Schema', () => {
const schema = new Schema();
const definition = { key: { legacy: 1 }, deprecated: true } as const;
defineIndex(schema, definition);
expect(() => defineIndex(schema, definition)).toThrow(
'Duplicate deprecated MongoDB index declaration: legacy_1'
);
});
});
packages/service/test/core/ai/sandbox/infrastructure/instance/schema.test.ts
import { describe, expect, it } from 'vitest'; import { describe, expect, it } from 'vitest';
import { MongoSandboxInstance } from '@fastgpt/service/core/ai/sandbox/infrastructure/instance/schema'; import { MongoSandboxInstance } from '@fastgpt/service/core/ai/sandbox/infrastructure/instance/schema';
import { getSchemaDeprecatedMongoIndexes } from '@fastgpt/service/common/mongo';
describe('MongoSandboxInstance schema indexes', () => { describe('MongoSandboxInstance schema indexes', () => {
it('declares provider sandbox uniqueness for remote resource records', () => { it('declares provider sandbox uniqueness for remote resource records', () => {
...@@ -21,6 +22,10 @@ describe('MongoSandboxInstance schema indexes', () => { ...@@ -21,6 +22,10 @@ describe('MongoSandboxInstance schema indexes', () => {
expect(legacyIndex).toBeUndefined(); expect(legacyIndex).toBeUndefined();
}); });
it('does not register historical sandbox indexes for automatic cleanup', () => {
expect(getSchemaDeprecatedMongoIndexes(MongoSandboxInstance.schema)).toEqual([]);
});
it('declares source lookup index for migrated sandbox instances', () => { it('declares source lookup index for migrated sandbox instances', () => {
const indexes = MongoSandboxInstance.schema.indexes(); const indexes = MongoSandboxInstance.schema.indexes();
const sourceChatIndex = indexes.find( const sourceChatIndex = indexes.find(
......
packages/service/test/core/chat/schema.test.ts
...@@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest'; ...@@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema'; import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema'; import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
import { MongoChatItemResponse } from '@fastgpt/service/core/chat/chatItemResponseSchema'; import { MongoChatItemResponse } from '@fastgpt/service/core/chat/chatItemResponseSchema';
import { getSchemaDeprecatedMongoIndexes } from '@fastgpt/service/common/mongo';
const hasIndex = ( const hasIndex = (
indexes: ReturnType<typeof MongoChat.schema.indexes>, indexes: ReturnType<typeof MongoChat.schema.indexes>,
...@@ -37,6 +38,10 @@ describe('chat schema indexes', () => { ...@@ -37,6 +38,10 @@ describe('chat schema indexes', () => {
expect(sourceAwareIndex?.[1]?.unique).toBe(true); expect(sourceAwareIndex?.[1]?.unique).toBe(true);
}); });
it('does not register historical chat indexes for automatic cleanup', () => {
expect(getSchemaDeprecatedMongoIndexes(MongoChat.schema)).toEqual([]);
});
it('keeps legacy chat item read and pagination indexes', () => { it('keeps legacy chat item read and pagination indexes', () => {
const indexes = MongoChatItem.schema.indexes(); const indexes = MongoChatItem.schema.indexes();
......
packages/service/test/env.test.ts
...@@ -9,6 +9,7 @@ const originalEnv = { ...@@ -9,6 +9,7 @@ const originalEnv = {
FILE_TOKEN_KEY: process.env.FILE_TOKEN_KEY, FILE_TOKEN_KEY: process.env.FILE_TOKEN_KEY,
FILE_DOWNLOAD_PUBLIC_URL_PREFIX: process.env.FILE_DOWNLOAD_PUBLIC_URL_PREFIX, FILE_DOWNLOAD_PUBLIC_URL_PREFIX: process.env.FILE_DOWNLOAD_PUBLIC_URL_PREFIX,
STORAGE_DOWNLOAD_URL_MODE: process.env.STORAGE_DOWNLOAD_URL_MODE, STORAGE_DOWNLOAD_URL_MODE: process.env.STORAGE_DOWNLOAD_URL_MODE,
SYNC_INDEX: process.env.SYNC_INDEX,
AES256_SECRET_KEY: process.env.AES256_SECRET_KEY, AES256_SECRET_KEY: process.env.AES256_SECRET_KEY,
INVOKE_TOKEN_SECRET: process.env.INVOKE_TOKEN_SECRET, INVOKE_TOKEN_SECRET: process.env.INVOKE_TOKEN_SECRET,
PRO_URL: process.env.PRO_URL, PRO_URL: process.env.PRO_URL,
...@@ -38,6 +39,7 @@ describe('serviceEnv', () => { ...@@ -38,6 +39,7 @@ describe('serviceEnv', () => {
vi.stubEnv('FILE_TOKEN_KEY', originalEnv.FILE_TOKEN_KEY); vi.stubEnv('FILE_TOKEN_KEY', originalEnv.FILE_TOKEN_KEY);
vi.stubEnv('FILE_DOWNLOAD_PUBLIC_URL_PREFIX', originalEnv.FILE_DOWNLOAD_PUBLIC_URL_PREFIX); vi.stubEnv('FILE_DOWNLOAD_PUBLIC_URL_PREFIX', originalEnv.FILE_DOWNLOAD_PUBLIC_URL_PREFIX);
vi.stubEnv('STORAGE_DOWNLOAD_URL_MODE', originalEnv.STORAGE_DOWNLOAD_URL_MODE); vi.stubEnv('STORAGE_DOWNLOAD_URL_MODE', originalEnv.STORAGE_DOWNLOAD_URL_MODE);
vi.stubEnv('SYNC_INDEX', originalEnv.SYNC_INDEX);
vi.stubEnv('AES256_SECRET_KEY', originalEnv.AES256_SECRET_KEY); vi.stubEnv('AES256_SECRET_KEY', originalEnv.AES256_SECRET_KEY);
vi.stubEnv('INVOKE_TOKEN_SECRET', originalEnv.INVOKE_TOKEN_SECRET); vi.stubEnv('INVOKE_TOKEN_SECRET', originalEnv.INVOKE_TOKEN_SECRET);
vi.stubEnv('PRO_URL', originalEnv.PRO_URL); vi.stubEnv('PRO_URL', originalEnv.PRO_URL);
...@@ -53,6 +55,22 @@ describe('serviceEnv', () => { ...@@ -53,6 +55,22 @@ describe('serviceEnv', () => {
vi.stubEnv('AGENT_SANDBOX_OPENSANDBOX_API_KEY', originalEnv.AGENT_SANDBOX_OPENSANDBOX_API_KEY); vi.stubEnv('AGENT_SANDBOX_OPENSANDBOX_API_KEY', originalEnv.AGENT_SANDBOX_OPENSANDBOX_API_KEY);
}); });
it('enables MongoDB index synchronization by default and supports disabling it', async () => {
vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey');
vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret');
vi.stubEnv('INVOKE_TOKEN_SECRET', validInvokeTokenSecret);
vi.stubEnv('SYNC_INDEX', undefined);
await expect(importServiceEnv()).resolves.toMatchObject({
serviceEnv: { SYNC_INDEX: true }
});
vi.stubEnv('SYNC_INDEX', 'false');
await expect(importServiceEnv()).resolves.toMatchObject({
serviceEnv: { SYNC_INDEX: false }
});
});
it('validates SYSTEM_MAX_STRING_LENGTH_M during service env init', async () => { it('validates SYSTEM_MAX_STRING_LENGTH_M during service env init', async () => {
vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey'); vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey');
vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret'); vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret');
......
projects/app/test/pages/api/admin/dataClean/cleanupDuplicateChats.test.ts
...@@ -37,7 +37,7 @@ const ensureLegacyDuplicateWritableCollection = async () => { ...@@ -37,7 +37,7 @@ const ensureLegacyDuplicateWritableCollection = async () => {
}; };
const restoreChatSchemaIndexes = async () => { const restoreChatSchemaIndexes = async () => {
await MongoChat.syncIndexes(); await MongoChat.createIndexes();
}; };
const createChatHeader = ({ const createChatHeader = ({
......
projects/marketplace/src/service/mongo/index.ts
...@@ -2,12 +2,12 @@ import type { Model, Schema } from 'mongoose'; ...@@ -2,12 +2,12 @@ import type { Model, Schema } from 'mongoose';
import { Mongoose } from 'mongoose'; import { Mongoose } from 'mongoose';
import { getLogger, LogCategories } from '../logger'; import { getLogger, LogCategories } from '../logger';
import { marketplaceEnv } from '../../env'; import { marketplaceEnv } from '../../env';
import { MongoIndexManager } from '@fastgpt/service/common/mongo/indexManager';
export const MONGO_URL = marketplaceEnv.MONGODB_URI; export const MONGO_URL = marketplaceEnv.MONGODB_URI;
const maxConnecting = Math.max(30, marketplaceEnv.DB_MAX_LINK); const maxConnecting = Math.max(30, marketplaceEnv.DB_MAX_LINK);
const logger = getLogger(LogCategories.INFRA.MONGO); const logger = getLogger(LogCategories.INFRA.MONGO);
const syncIndex = marketplaceEnv.SYNC_INDEX;
declare global { declare global {
var mongodb: Mongoose | undefined; var mongodb: Mongoose | undefined;
...@@ -32,16 +32,26 @@ export const getMongoModel = <T extends Schema>(name: string, schema: T) => { ...@@ -32,16 +32,26 @@ export const getMongoModel = <T extends Schema>(name: string, schema: T) => {
}; };
const syncMongoIndex = async (model: Model<any>) => { const syncMongoIndex = async (model: Model<any>) => {
if (syncIndex && process.env.NODE_ENV !== 'test') { if (process.env.NODE_ENV === 'test' || !marketplaceEnv.SYNC_INDEX || !MONGO_URL) {
try { return;
model.syncIndexes({ background: true }); }
} catch (error: any) {
logger.error('Create index error', { error }); try {
} await MongoIndexManager.syncModelIndexes({
model,
logger
});
} catch (error) {
logger.error('Failed to ensure MongoDB indexes', {
modelName: model.modelName,
collectionName: model.collection.collectionName,
error
});
} }
}; };
export const ReadPreference = connectionMongo.mongo.ReadPreference; export const ReadPreference = connectionMongo.mongo.ReadPreference;
export { defineIndex } from '@fastgpt/service/common/mongo/schemaIndexes';
export async function connectMongo(db: Mongoose, url: string): Promise<Mongoose> { export async function connectMongo(db: Mongoose, url: string): Promise<Mongoose> {
if (db.connection.readyState !== 0) { if (db.connection.readyState !== 0) {
...@@ -81,6 +91,7 @@ export async function connectMongo(db: Mongoose, url: string): Promise<Mongoose> ...@@ -81,6 +91,7 @@ export async function connectMongo(db: Mongoose, url: string): Promise<Mongoose>
serverSelectionTimeoutMS: 10000, // 服务器选择超时: 10秒,防止副本集故障时长时间阻塞 serverSelectionTimeoutMS: 10000, // 服务器选择超时: 10秒,防止副本集故障时长时间阻塞
heartbeatFrequencyMS: 5000 // 5s 进行一次健康检查 heartbeatFrequencyMS: 5000 // 5s 进行一次健康检查
}); });
return db; return db;
} catch (error) { } catch (error) {
logger.error('Mongo connect error', { error }); logger.error('Mongo connect error', { error });
......
projects/marketplace/src/service/mongo/models/download.ts
import { Schema } from 'mongoose'; import { Schema } from 'mongoose';
import z from 'zod'; import z from 'zod';
import { getMongoModel } from '..'; import { defineIndex, getMongoModel } from '..';
export const pluginTypeEnum = z.enum(['tool']); export const pluginTypeEnum = z.enum(['tool']);
...@@ -21,6 +21,9 @@ const downloadCountSchema = new Schema({ ...@@ -21,6 +21,9 @@ const downloadCountSchema = new Schema({
}); });
// 复合索引：type + toolId + time // 复合索引：type + toolId + time
downloadCountSchema.index({ type: 1, toolId: 1, time: 1 }, { unique: true }); defineIndex(downloadCountSchema, {
key: { type: 1, toolId: 1, time: 1 },
options: { unique: true }
});
export const MongoDownloadCount = getMongoModel('plugin_download_counts', downloadCountSchema); export const MongoDownloadCount = getMongoModel('plugin_download_counts', downloadCountSchema);
projects/marketplace/src/service/mongo/models/tool.ts
import { Schema } from 'mongoose'; import { Schema } from 'mongoose';
import z from 'zod'; import z from 'zod';
import { getMongoModel } from '..'; import { defineIndex, getMongoModel } from '..';
import { import {
MarketplaceOfficialSource, MarketplaceOfficialSource,
MarketplacePkgSourceSchema MarketplacePkgSourceSchema
...@@ -47,8 +47,13 @@ const marketplaceToolSchema = new Schema( ...@@ -47,8 +47,13 @@ const marketplaceToolSchema = new Schema(
} }
); );
marketplaceToolSchema.index({ pluginId: 1, version: 1 }, { unique: true }); defineIndex(marketplaceToolSchema, {
marketplaceToolSchema.index({ pluginId: 1, updateTime: -1 }); key: { pluginId: 1, version: 1 },
marketplaceToolSchema.index({ source: 1, pluginId: 1, updateTime: -1 }); options: { unique: true }
});
defineIndex(marketplaceToolSchema, { key: { pluginId: 1, updateTime: -1 } });
defineIndex(marketplaceToolSchema, {
key: { source: 1, pluginId: 1, updateTime: -1 }
});
export const MongoMarketplaceTool = getMongoModel('marketplace_tools', marketplaceToolSchema); export const MongoMarketplaceTool = getMongoModel('marketplace_tools', marketplaceToolSchema);
projects/marketplace/test/env.test.ts
import { afterEach, describe, expect, it, vi } from 'vitest'; import { afterEach, describe, expect, it, vi } from 'vitest';
const originalSyncIndex = process.env.SYNC_INDEX;
const originalCommunityAuthToken = process.env.COMMUNITY_AUTH_TOKEN; const originalCommunityAuthToken = process.env.COMMUNITY_AUTH_TOKEN;
const originalSyncIndex = process.env.SYNC_INDEX;
const importEnv = async () => { const importEnv = async () => {
vi.resetModules(); vi.resetModules();
...@@ -10,32 +10,20 @@ const importEnv = async () => { ...@@ -10,32 +10,20 @@ const importEnv = async () => {
describe('marketplace env', () => { describe('marketplace env', () => {
afterEach(() => { afterEach(() => {
vi.stubEnv('SYNC_INDEX', originalSyncIndex);
vi.stubEnv('COMMUNITY_AUTH_TOKEN', originalCommunityAuthToken); vi.stubEnv('COMMUNITY_AUTH_TOKEN', originalCommunityAuthToken);
vi.stubEnv('SYNC_INDEX', originalSyncIndex);
}); });
it('defaults SYNC_INDEX to true when it is not configured', async () => { it('enables MongoDB index synchronization by default and supports disabling it', async () => {
vi.stubEnv('SYNC_INDEX', undefined); vi.stubEnv('SYNC_INDEX', undefined);
await expect(importEnv()).resolves.toMatchObject({
marketplaceEnv: { SYNC_INDEX: true }
});
const { marketplaceEnv } = await importEnv();
expect(marketplaceEnv.SYNC_INDEX).toBe(true);
});
it('defaults SYNC_INDEX to true when it is empty', async () => {
vi.stubEnv('SYNC_INDEX', '');
const { marketplaceEnv } = await importEnv();
expect(marketplaceEnv.SYNC_INDEX).toBe(true);
});
it('parses explicit false-like SYNC_INDEX values', async () => {
vi.stubEnv('SYNC_INDEX', 'false'); vi.stubEnv('SYNC_INDEX', 'false');
await expect(importEnv()).resolves.toMatchObject({
const { marketplaceEnv } = await importEnv(); marketplaceEnv: { SYNC_INDEX: false }
});
expect(marketplaceEnv.SYNC_INDEX).toBe(false);
}); });
it('parses optional community auth token', async () => { it('parses optional community auth token', async () => {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment