Skip to content

赵月辉 / fastgpt-migrated

Go to a project
Commit 225da9bc by Xianquan Committed by GitHub
Options

fix: make mongo index sync non-destructive (#7313) 

* fix: make mongo index sync non-destructive

* feat: add safe mongo index maintenance

* docs: document mongo index sync modes

* refactor: simplify mongo index synchronization

* refactor: reduce mongo index sync logs

* refactor: unify mongo index declarations

* docs: consolidate mongo index design notes

* refactor: simplify mongo index declarations

* docs: clarify legacy index cleanup upgrade path

* docs: clarify MongoDB deprecated index cleanup

* fix: match deprecated text indexes by weights, drop options check

Allow name+key cleanup for MongoDB text indexes whose stored key is
rewritten to _fts/_ftsx, and stop requiring options equality so valid
deprecations are not blocked by declaration noise.

* fix: keep agent skill folder tree index active

Restore { teamId, parentId, deleteTime } as a current Schema index so
findSkillAndAllChildren can use it and startup cleanup will not drop it.

* fix: align pro submodule for admin build

---------

Co-authored-by: Archer <545436317@qq.com>
parent 9aa9709a
Showing with 1044 additions and 596 deletions
.agents/design/common/mongo-index-sync-strategy.md 0 → 100644
# MongoDB 索引安全同步设计
## 文档状态
- 状态：已落地
- 适用范围：主 Service、日志库、Marketplace 和 `pro/admin` 中由 FastGPT 管理的 MongoDB Schema
- 最终结论：启动时只补建当前 Schema 索引；仅当业务 Schema 明确登记废弃索引时才执行精确清理，任何未知索引一律保留。当前没有业务 Schema 登记废弃索引
## 问题与根因
私有化部署客户可能通过 `mongosh`、运维脚本或数据库管理平台添加自定义索引。原实现会在 Mongoose model 加载时调用 `model.syncIndexes()`，其语义是：
1. 比较 Schema 与数据库已有索引。
2. 删除 Schema 中不存在的索引。
3. 创建 Schema 中缺失的索引。
因此，客户自建索引在服务重启时被删除是 `syncIndexes()` 的确定性行为，并非异常分支。原有 `SYNC_INDEX=false` 虽然可以阻止删除，但也会阻止 FastGPT 新版本补建必需索引，无法作为默认解决方案。
该问题同时影响主业务库、日志库和 Marketplace；多实例启动还会放大重复执行和错误日志问题。
## 最终决策
1. 禁止在启动索引管理中调用 `syncIndexes()` 或 `cleanIndexes()`。
2. `SYNC_INDEX` 弃用；启动时固定检查差异并补建当前 Schema 缺失的索引。
3. `MONGO_DEPRECATE_INDEX` 仅控制是否清理显式声明的 FastGPT 系统内置废弃索引，默认值为 `true`；设置为 `false` 不影响缺失索引的创建。
4. 不提供可切换回全量删除的启动模式。
5. 当前索引和废弃索引统一通过 `defineIndex()` 声明；废弃索引必须声明在所属 Schema 文件中。
6. Schema 外且未被显式声明为废弃的索引只告警、不删除。
7. 删除废弃索引前必须先成功创建当前 Schema 索引；创建失败时不进入删除阶段。
8. 主 Service、日志库和 Marketplace 统一复用 `MongoIndexManager`，保持同步语义与错误处理一致。
## 索引声明
`packages/service/common/mongo/schemaIndexes.ts` 提供统一入口：
```ts
defineIndex(ChatSchema, {
key: { appId: 1, chatId: 1 },
options: { unique: true }
});
defineIndex(ChatSchema, {
key: { legacyField: 1 },
options: { name: 'legacyField_1' },
deprecated: true
});
```
声明规则：
- 未设置 `deprecated` 时，`defineIndex()` 代理 `Schema.index()`，该索引属于当前 Schema。
- 显式设置 `deprecated: true` 时，只在 Schema 实例上登记清理元数据，不再把索引加入 Mongoose Schema。
- 废弃索引元数据包含索引名、key，以及可选的 options 快照（仅供阅读/排查，不参与删除匹配）；不包含 collection name，collection 由 model 推导。
- 未显式提供索引名时，按 MongoDB 默认规则由 key 推导。
- 同一 Schema 内重复登记同名废弃索引属于配置错误，应立即抛错。
- FastGPT 管理的索引不再使用字段级 `index: true` 或 `unique: true` 隐式声明，避免绕开统一入口。
废弃索引元数据使用私有 symbol 挂在 Schema 实例上，使当前索引与历史清理声明在同一业务文件中完成 review，并避免中心清单与 Schema 演进脱节。
## 启动同步流程
每个 model 启动时执行以下流程：
1. `diffIndexes({ indexOptionsToCreate: true })` 生成 `toCreate` 和 `toDrop`。
2. `toDrop` 仅表示数据库中存在但当前 Schema 未声明的索引，记录 `warn` 后保留。
3. `createIndexes({ background: true })` 创建当前 Schema 索引。
4. 当 `MONGO_DEPRECATE_INDEX=true` 时，从 `model.schema` 读取废弃索引声明。
5. 在 model 对应 collection 中按 name 查找废弃索引。
6. 仅删除 name 与 key 匹配的索引；options 不参与匹配。
key 匹配规则：
- 普通索引：`listIndexes` 的 key 与声明 key 按字段顺序精确相等。
- text 索引：MongoDB 会把 key 改写为 `{ _fts: "text", _ftsx: 1 }`，因此改为比较声明中的 text 字段集合与 `weights` 字段集合。
- options（`unique` / `sparse` / TTL / partial / collation）故意不参与匹配，减少重复声明成本；声明方需自行确认同名同 key 的索引确实可删。
清理结果分为：
- `drop`：定义匹配，已删除或在 dry-run 中可删除。
- `skip_missing`：索引不存在或已被其他实例删除。
- `skip_mismatch`：同名索引的 key 不匹配，保留并告警。
- `error`：查询或删除失败，保留错误信息。
同一进程内同一 model 的并发调用复用正在执行的任务；任务完成后移除缓存，允许热加载或重连再次检查。多实例重复清理时，`IndexNotFound` 视为幂等跳过。
## 模块职责
- `packages/service/common/mongo/indexManager.ts`
- `inspectModelIndexes()`：只计算差异，不创建或删除索引。
- `syncModelIndexes()`：执行安全同步并复用同一 model 的进行中任务。
- `cleanupModelDeprecatedIndexes()`：按 Schema 本地声明检查或清理废弃索引。
- `summarizeCleanupReport()` / `formatCleanupReport()`：提供结构化结果与可读报告。
- `packages/service/common/mongo/schemaIndexes.ts`
- `defineIndex()`：声明当前索引或登记废弃索引。
- `getDeprecatedIndexes()`：读取当前 Schema 自身的废弃索引元数据。
- `packages/service/common/mongo/index.ts`
- 主 Service 和日志库的 model 注册入口，只负责按环境条件触发 manager。
- `projects/marketplace/src/service/mongo/index.ts`
- Marketplace 的 model 注册入口，复用同一 manager 并完整捕获异步错误。
原中心废弃索引清单已删除。当前 chat、sandbox instance 和 Agent Skill 均未登记废弃索引，因此启动同步不会自动删除任何历史索引；manager 仅保留显式清理能力供后续经过单独确认的迁移使用。
## 日志与失败处理
- 无差异且无清理动作时不输出同步摘要。
- `info`：实际创建或删除索引时输出 collection 级摘要。
- `warn`：发现 Schema 外索引，或废弃声明与数据库同名索引不匹配。
- `error`：当前索引创建失败，或废弃索引检查、删除失败。
索引任务失败不阻止 model 注册，但必须记录 model、collection 和错误信息。`createIndexes()` 的同名、同 key 或 options 冲突由 MongoDB/Mongoose 抛错并进入错误日志，不自动修正。
## 安全边界
1. 未登记的 Schema 外索引不会被删除，包括客户自建索引和无法确认所有权的历史索引。
2. `createIndexes()` 不会修改已存在索引的 options。TTL、唯一约束、partial filter 等变化必须通过明确迁移处理。
3. 错误的 Schema 本地废弃声明会在启动时触发清理，因此 name + key 匹配和代码 review 是必须保留的防线。
4. 启动流程不暴露 Mongoose 全量同步能力；需要诊断时复用 manager 的 inspect/dry-run 能力。
5. `MONGO_DEPRECATE_INDEX=false` 只关闭废弃索引清理，当前 Schema 缺失的索引仍会创建。
6. 客户自建索引应显式设置自定义名称，不使用 MongoDB 按 key 生成的默认名称，避免与 FastGPT 系统内置索引重名。
## 已知暂不处理的历史索引
### `llm_request_records.requestId_1`
- `76d6234de V4.14.7 features (#6406)` 首次在 `requestId` path 上声明 `unique: true`，MongoDB 创建 `requestId_1`。
- `f008ea971 feat: teamId in reacord llm` 将约束调整为 `{ teamId: 1, requestId: 1 }` 复合唯一索引。
- 旧索引对当前 Schema 已无必要，并可能继续施加跨团队全局唯一约束。
- 本次不新增其废弃声明。后续如需清理，必须在 `packages/service/core/ai/record/schema.ts` 单独声明并补充回归测试。
## 验收标准
1. 启动时创建当前 Schema 缺失索引，并保留所有未登记的 Schema 外索引。
2. Schema 未登记废弃索引时不执行删除。
3. 只有 name 与 key 匹配的废弃索引会被删除；text 索引通过 weights 字段集合匹配。
4. 当前索引创建失败时不删除废弃索引。
5. 同名但定义不同的索引保留并告警。
6. 已不存在的废弃索引和多实例并发重复清理保持幂等。
7. 当前所有业务 Schema 均未登记废弃索引，启动同步不会自动删除历史索引。
8. 主 Service、日志库和 Marketplace 始终调用同一 manager，并由 `MONGO_DEPRECATE_INDEX` 统一控制废弃索引清理。
## 后续事项
以下事项不影响当前方案交付：
1. 增加 Root 管理员 inspect/apply API 或等价脚本，提供启动日志之外的诊断入口；apply 仍只允许执行 manager 定义的安全动作。
2. 单独评估并迁移 `llm_request_records.requestId_1`。
3. 为新索引逐步采用 `fg_<collection>_<purpose>` 显式命名规范；旧索引不做批量改名。
AGENTS.md
......@@ -89,6 +89,13 @@ FastGPT 是一个 AI Agent 构建平台,通过 Flow 提供开箱即用的数据
- 所有代码编写、修改、重构和测试调整都必须遵守 [FastGPT 代码规范](./.agents/code/syntax.md)。开始改动前先查看相关规范；如果规范与当前实现习惯冲突，优先按规范执行，并只在有明确业务或兼容性理由时说明例外。
### MongoDB Schema 与索引维护
- 所有由 FastGPT 管理的当前索引和废弃索引都必须通过 `defineIndex(schema, { key, options, deprecated })` 声明：`deprecated` 默认是 `false`，当前索引省略该字段；只有废弃索引显式使用 `deprecated: true`。不要直接调用 `schema.index()`，也不要在字段定义中使用 `index: true` 或 `unique: true` 隐式创建索引。
- 每当新增、修改、删除或重命名 MongoDB/Mongoose Schema 字段、索引定义、唯一约束、TTL、partialFilterExpression、collation 等索引相关配置时，必须同步检查是否有 FastGPT 旧版本创建的索引不再被当前 Schema 使用。
- 如果历史索引可能继续影响写入约束、查询计划或存储成本，应在所属 Schema 文件中通过 `defineIndex(schema, { key, options, deprecated: true })` 紧邻当前索引声明登记删除定义，并补充/调整 `packages/service/test/common/mongo/indexManager.test.ts` 的清理行为覆盖。索引名默认由 key 推导，也可通过 `options.name` 显式指定；删除前按 name 定位，再精确匹配 key（text 索引兼容 `_fts/_ftsx` 与 weights）；options 不参与匹配。
- 不要登记客户自建索引、无法确认来源的索引，或仅凭当前 Schema 未声明就推断为废弃的索引。主动同步只允许删除 FastGPT 明确创建过、明确废弃且与 Schema 本地声明精确匹配的历史索引。
### API 入参校验
- 编写或修改 NextJS API 路由时，如果需要校验接口入参（`req.body`、`req.query`、`req.params`），必须使用 `parseApiInput`，不要直接写 `SomeSchema.parse(req.body)`、`SomeSchema.parse(req.query)` 或 `SomeSchema.parse(req.params)`。
......
document/content/self-host/config/env.en.mdx
......@@ -23,7 +23,7 @@ These variables are mainly validated by `packages/service/env.ts` and apply to `
| Variable | Default | Description |
| --------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DB_MAX_LINK` | `5` | Maximum connection pool size for MongoDB, PG, OceanBase, openGauss, and other databases. |
| `SYNC_INDEX` | `true` | Whether MongoDB indexes are synchronized at startup. |
| `SYNC_INDEX` | `true` | Whether to create missing MongoDB indexes and remove explicitly declared deprecated indexes at startup. Maintain indexes manually when disabled. |
| `FILE_TOKEN_KEY` | None, **required** | Secret for file read and file authorization flows. Must be at least 6 characters. |
| `AES256_SECRET_KEY` | None, **required** | Secret used by AES encryption and decryption. Must be at least 6 characters. |
| `INVOKE_TOKEN_SECRET` | None, **required** | JWT secret for Invoke reverse calls. Must be at least 32 characters. |
......
document/content/self-host/config/env.mdx
......@@ -23,7 +23,7 @@ description: projects/app、projects/code-sandbox 与 pro/admin 环境变量说
| 变量 | 默认值 | 说明 |
| --------------------- | ------------------ | ----------------------------------------------------------------------------------------------------- |
| `DB_MAX_LINK` | `5` | MongoDB、PG、OceanBase、openGauss 等数据库连接池最大连接数。 |
| `SYNC_INDEX` | `true` | 启动时是否同步 MongoDB 索引。 |
| `SYNC_INDEX` | `true` | 是否在启动时创建缺失的 MongoDB 索引并清理显式声明的废弃索引；关闭后需自行维护索引。 |
| `FILE_TOKEN_KEY` | 无，**必填** | 文件读取、文件鉴权相关密钥，长度至少 6 位。 |
| `AES256_SECRET_KEY` | 无，**必填** | AES 加解密密钥，长度至少 6 位。 |
| `INVOKE_TOKEN_SECRET` | 无，**必填** | Invoke 反向调用 JWT 密钥，长度至少 32 位。 |
......
document/content/self-host/upgrading/4-15/4154.en.mdx
......@@ -14,6 +14,31 @@ production; local development can use `http://localhost:3000`.
```bash
FE_DOMAIN=https://fastgpt.example.com
```
### MongoDB Index Synchronization Changes
Starting with V4.15.4, `SYNC_INDEX` is deprecated and replaced by `MONGO_DEPRECATE_INDEX`. The new variable controls whether indexes explicitly marked as deprecated by a schema are removed and defaults to `true`. Setting it to `false` skips only deprecated-index cleanup; missing current schema indexes are still created.
FastGPT now performs safe index synchronization automatically at startup:
- Creates indexes that are missing from the current FastGPT schemas.
- Removes only built-in historical FastGPT indexes explicitly marked as deprecated by the corresponding schema and whose name, key, and relevant options match exactly.
- Preserves custom indexes and any other indexes that are not explicitly declared as deprecated.
This process does not call Mongoose's full `syncIndexes()` operation, so indexes are never removed simply because they are absent from a FastGPT schema.
> **Default and deletion boundary: `MONGO_DEPRECATE_INDEX` defaults to `true`. It removes only built-in indexes that a FastGPT schema explicitly marks as deprecated and whose definitions match exactly; customer-created indexes are not removed. Give every custom index an explicit name instead of relying on MongoDB's key-derived default name to prevent collisions with built-in FastGPT index names.**
> **Legacy index cleanup: V4.15.4 does not mark any existing historical indexes as deprecated, so upgrading to this version does not automatically remove old indexes. Future releases will explicitly mark verified obsolete indexes in their schemas and remove them incrementally.**
To fully remove obsolete indexes before upgrading to V4.15.4:
1. Upgrade to and start V4.15.3 once.
2. Set `SYNC_INDEX=true`, restart the services, and wait for index synchronization to finish.
3. After confirming that index synchronization succeeded, upgrade to V4.15.4.
V4.15.3 removes every index that is not declared in its schemas, which may include custom indexes. Back up your database and review the existing indexes before following this procedure. If custom indexes must be preserved, record their definitions and recreate them after synchronization, or do not use V4.15.3 for full cleanup.
Setting `MONGO_DEPRECATE_INDEX=false` skips deprecated-index cleanup that may be introduced in future releases, but does not skip creation of missing indexes.
## 🚀 New Features
......
document/content/self-host/upgrading/4-15/4154.mdx
......@@ -12,6 +12,31 @@ FastGPT 服务启动时会校验 `FE_DOMAIN`。请将它配置为客户端访问
```bash
FE_DOMAIN=https://fastgpt.example.com
```
### MongoDB 索引同步调整
V4.15.4 起，`SYNC_INDEX` 弃用，新增 `MONGO_DEPRECATE_INDEX` 环境变量，用于控制是否清理 Schema 显式标记的废弃索引，默认值为 `true`。设置为 `false` 时只跳过废弃索引清理，不影响当前 Schema 缺失索引的创建。
FastGPT 启动时会自动执行安全的主动同步：
- 创建当前 FastGPT Schema 中缺失的索引。
- 仅删除对应 Schema 明确标记为废弃、且 name、key 和关键 options 完全匹配的 FastGPT 系统内置历史索引。
- 保留客户自建索引及其他未声明的索引。
该同步不会调用 Mongoose 的全量 `syncIndexes()`，因此不会按“未在 Schema 中声明”这一条件批量删除索引。
> **默认开启与删除边界：`MONGO_DEPRECATE_INDEX` 默认为 `true`，仅删除 FastGPT Schema 显式声明为废弃、且索引定义精确匹配的系统内置索引，不会删除客户自建索引。建议为自建索引显式设置自定义名称，不要使用 MongoDB 按 key 生成的默认名称，避免与 FastGPT 系统内置索引重名。**
> **旧索引清理说明：V4.15.4 不会把任何已有历史索引标记为废弃，因此升级到该版本时不会自动删除旧索引。后续版本会在确认安全后，通过 Schema 中的显式废弃标记逐步清理对应索引。**
如需在升级 V4.15.4 前完整删除历史过期索引，请按以下顺序操作：
1. 先升级并启动一次 V4.15.3。
2. 设置 `SYNC_INDEX=true`，重启服务并等待索引同步完成。
3. 确认索引同步成功后，再升级至 V4.15.4。
V4.15.3 的索引同步会删除所有未在当时 Schema 中声明的索引，其中可能包含客户自建索引。执行上述步骤前，请先备份数据库并检查现有索引；如需保留自建索引，请记录其定义并在同步后重新创建，或不要使用 V4.15.3 进行全量清理。
`MONGO_DEPRECATE_INDEX=false` 会跳过未来版本可能声明的废弃索引清理，但不会跳过缺失索引的创建。
## 🚀 新增内容
......
document/data/doc-last-modified.json
......@@ -320,8 +320,8 @@
"content/self-host/upgrading/4-15/41507.mdx": "2026-06-30T17:31:43+08:00",
"content/self-host/upgrading/4-15/4151.en.mdx": "2026-07-21T12:01:28+08:00",
"content/self-host/upgrading/4-15/4151.mdx": "2026-07-21T12:01:28+08:00",
"content/self-host/upgrading/4-15/4152.en.mdx": "2026-07-17T19:17:14+08:00",
"content/self-host/upgrading/4-15/4152.mdx": "2026-07-18T12:51:59+08:00",
"content/self-host/upgrading/4-15/4152.en.mdx": "2026-07-20T15:11:00+08:00",
"content/self-host/upgrading/4-15/4152.mdx": "2026-07-20T15:11:00+08:00",
"content/self-host/upgrading/4-15/4153.en.mdx": "2026-07-18T12:51:59+08:00",
"content/self-host/upgrading/4-15/4153.mdx": "2026-07-18T12:51:59+08:00",
"content/self-host/upgrading/4-15/4154.en.mdx": "2026-07-22T11:49:20+08:00",
......
packages/service/common/buffer/tts/schema.ts
import { Schema, getMongoModel } from '../../../common/mongo';
import { defineIndex, Schema, getMongoModel } from '../../../common/mongo';
import { type TTSBufferSchemaType } from './type';
import { getLogger, LogCategories } from '../../logger';
......@@ -25,12 +25,11 @@ const TTSBufferSchema = new Schema({
}
});
try {
TTSBufferSchema.index({ bufferId: 1 });
// 24 hour
TTSBufferSchema.index({ createTime: 1 }, { expireAfterSeconds: 24 * 60 * 60 });
} catch (error) {
logger.error('Failed to build TTS buffer indexes', { error });
}
defineIndex(TTSBufferSchema, { key: { bufferId: 1 } });
// 24 hour
defineIndex(TTSBufferSchema, {
key: { createTime: 1 },
options: { expireAfterSeconds: 24 * 60 * 60 }
});
export const MongoTTSBuffer = getMongoModel<TTSBufferSchemaType>(collectionName, TTSBufferSchema);
packages/service/common/file/gridfs/schema.ts
import { Schema, getMongoModel } from '../../mongo';
import { defineIndex, Schema, getMongoModel } from '../../mongo';
const DatasetFileSchema = new Schema({
metadata: Object
......@@ -7,10 +7,10 @@ const ChatFileSchema = new Schema({
metadata: Object
});
DatasetFileSchema.index({ uploadDate: -1 });
defineIndex(DatasetFileSchema, { key: { uploadDate: -1 } });
ChatFileSchema.index({ uploadDate: -1 });
ChatFileSchema.index({ 'metadata.chatId': 1 });
defineIndex(ChatFileSchema, { key: { uploadDate: -1 } });
defineIndex(ChatFileSchema, { key: { 'metadata.chatId': 1 } });
export const MongoDatasetFileSchema = getMongoModel('dataset.files', DatasetFileSchema);
export const MongoChatFileSchema = getMongoModel('chat.files', ChatFileSchema);
packages/service/common/file/image/schema.ts
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { Schema, getMongoModel } from '../../mongo';
import { defineIndex, Schema, getMongoModel } from '../../mongo';
import { type MongoImageSchemaType } from '@fastgpt/global/common/file/image/type';
import { getLogger, LogCategories } from '../../logger';
......@@ -20,20 +20,19 @@ const ImageSchema = new Schema({
metadata: Object
});
try {
// tts expired（60 Minutes）
ImageSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 60 * 60 });
ImageSchema.index({ type: 1 });
// delete related img
ImageSchema.index({ teamId: 1, 'metadata.relatedId': 1 });
// tts expired（60 Minutes）
defineIndex(ImageSchema, {
key: { expiredTime: 1 },
options: { expireAfterSeconds: 60 * 60 }
});
defineIndex(ImageSchema, { key: { type: 1 } });
// delete related img
defineIndex(ImageSchema, { key: { teamId: 1, 'metadata.relatedId': 1 } });
// Cron clear invalid img
ImageSchema.index(
{ createTime: 1 },
{ partialFilterExpression: { 'metadata.relatedId': { $exists: true } } }
);
} catch (error) {
logger.error('Failed to build image indexes', { error });
}
// Cron clear invalid img
defineIndex(ImageSchema, {
key: { createTime: 1 },
options: { partialFilterExpression: { 'metadata.relatedId': { $exists: true } } }
});
export const MongoImage = getMongoModel<MongoImageSchemaType>('image', ImageSchema);
packages/service/common/middle/tracks/schema.ts
import { type TrackSchemaType } from '@fastgpt/global/common/middle/tracks/type';
import { getMongoModel, Schema } from '../../mongo';
import { defineIndex, getMongoModel, Schema } from '../../mongo';
import { TrackEnum } from '@fastgpt/global/common/middle/tracks/constants';
const TrackSchema = new Schema({
......@@ -11,24 +11,24 @@ const TrackSchema = new Schema({
data: Object
});
TrackSchema.index({ event: 1 });
defineIndex(TrackSchema, { key: { event: 1 } });
// Dataset search index
TrackSchema.index(
{ event: 1, teamId: 1, 'data.datasetId': 1, createTime: -1 },
{
defineIndex(TrackSchema, {
key: { event: 1, teamId: 1, 'data.datasetId': 1, createTime: -1 },
options: {
partialFilterExpression: {
event: TrackEnum.datasetSearch
}
}
);
});
// QPM index
TrackSchema.index(
{ event: 1, createTime: -1, 'data.requestCount': 1 },
{
defineIndex(TrackSchema, {
key: { event: 1, createTime: -1, 'data.requestCount': 1 },
options: {
partialFilterExpression: {
event: TrackEnum.teamChatQPM
}
}
);
});
export const TrackModel = getMongoModel<TrackSchemaType>('tracks', TrackSchema);
packages/service/common/mongo/index.ts
......@@ -9,6 +9,7 @@ import type {
} from 'mongoose';
import mongoose, { Mongoose } from 'mongoose';
import { serviceEnv } from '../../env';
import { MongoIndexManager } from './indexManager';
const logger = getLogger(LogCategories.INFRA.MONGO);
......@@ -136,7 +137,6 @@ export const getMongoModel = <T>(name: string, schema: mongoose.Schema): Model<T
const model = connectionMongo.model(name, schema) as Model<T>;
// Sync index
syncMongoIndex(model);
return model;
......@@ -148,13 +148,12 @@ export const getMongoLogModel = <T>(name: string, schema: mongoose.Schema): Mode
const model = connectionLogMongo.model(name, schema) as Model<T>;
// Sync index
syncMongoIndex(model);
return model;
};
const syncMongoIndex = async (model: Model<any>) => {
const syncMongoIndex = (model: Model<any>) => {
if (
process.env.NODE_ENV === 'test' ||
process.env.NEXT_PHASE === 'phase-production-build' ||
......@@ -164,11 +163,34 @@ const syncMongoIndex = async (model: Model<any>) => {
return;
}
try {
await model.syncIndexes({ background: true });
} catch (error) {
logger.error('Failed to sync MongoDB indexes', { modelName: model.modelName, error });
}
void MongoIndexManager.syncModelIndexes({
model,
logger
}).catch((error) => {
logger.error('Failed to ensure MongoDB indexes', {
modelName: model.modelName,
collectionName: model.collection.collectionName,
error
});
});
};
export const ReadPreference = connectionMongo.mongo.ReadPreference;
export { MongoIndexManager } from './indexManager';
export {
getDeprecatedIndexes as getSchemaDeprecatedMongoIndexes,
defineIndex
} from './schemaIndexes';
export type {
MongoIndexCleanupAction,
MongoIndexCleanupReport,
MongoIndexCleanupReportItem,
MongoIndexCleanupSummary,
MongoIndexSyncResult
} from './indexManager';
export type {
DefineMongoIndexOptions,
DeprecatedMongoIndexDefinition,
DeprecatedMongoIndexOptions
} from './schemaIndexes';
packages/service/common/mongo/schemaIndexes.ts 0 → 100644
import type { IndexDefinition, IndexOptions, Schema } from 'mongoose';
const deprecatedMongoIndexesKey = Symbol.for('fastgpt.mongo.deprecatedIndexes');
export type DeprecatedMongoIndexOptions = Pick<
IndexOptions,
'unique' | 'sparse' | 'expireAfterSeconds' | 'partialFilterExpression' | 'collation'
>;
export type DeprecatedMongoIndexDefinition = {
indexName: string;
key: IndexDefinition;
options?: DeprecatedMongoIndexOptions;
};
export type DefineMongoIndexOptions = {
key: IndexDefinition;
options?: IndexOptions;
deprecated?: true;
};
/**
* 统一声明当前索引和 FastGPT 明确废弃的历史索引。
*
* `deprecated` 默认是 `false`，当前索引直接代理 `Schema.index()`；显式设置为
* `true` 时只登记清理元数据，不能继续写入 Mongoose Schema，否则启动同步会先
* 重新创建该索引。废弃索引未显式命名时，按 MongoDB 的默认规则从 key 推导名称。
*/
export const defineIndex = (
schema: Schema,
{ key, options, deprecated }: DefineMongoIndexOptions
) => {
if (deprecated !== true) {
schema.index(key, options);
return;
}
const registeredIndexes = getDeprecatedIndexes(schema);
const indexName =
options?.name ??
Object.entries(key)
.map(([field, order]) => `${field}_${order}`)
.join('_');
const duplicateIndexName = registeredIndexes.some((index) => index.indexName === indexName);
if (duplicateIndexName) {
throw new Error(`Duplicate deprecated MongoDB index declaration: ${indexName}`);
}
const deprecatedOptions: DeprecatedMongoIndexOptions = {
unique: options?.unique,
sparse: options?.sparse,
expireAfterSeconds: options?.expireAfterSeconds,
partialFilterExpression: options?.partialFilterExpression,
collation: options?.collation
};
const hasDeprecatedOptions = Object.values(deprecatedOptions).some(
(value) => value !== undefined
);
Reflect.set(schema, deprecatedMongoIndexesKey, [
...registeredIndexes,
{
indexName,
key,
options: hasDeprecatedOptions ? deprecatedOptions : undefined
}
] satisfies DeprecatedMongoIndexDefinition[]);
};
/** 读取某个 Schema 自身登记的废弃索引，不聚合其他集合或全局清单。 */
export const getDeprecatedIndexes = (schema: Schema): readonly DeprecatedMongoIndexDefinition[] => {
const indexes: unknown = Reflect.get(schema, deprecatedMongoIndexesKey);
return Array.isArray(indexes) ? indexes : [];
};
packages/service/common/s3/accessLink/downloadAlias/schema.ts
import { getLogger, LogCategories } from '../../../logger';
import { getMongoModel, Schema } from '../../../mongo';
import { defineIndex, getMongoModel, Schema } from '../../../mongo';
import type { S3DownloadAliasType } from '../type';
export const S3DownloadAliasCollectionName = 's3_download_aliases';
......@@ -44,14 +44,21 @@ const S3DownloadAliasMongoSchema = new Schema({
disabledAt: Date
});
try {
S3DownloadAliasMongoSchema.index({ aliasId: 1 }, { unique: true });
S3DownloadAliasMongoSchema.index({ aliasKey: 1 }, { unique: true });
S3DownloadAliasMongoSchema.index({ purgeAt: 1 }, { expireAfterSeconds: 0 });
S3DownloadAliasMongoSchema.index({ bucketName: 1, objectKey: 1 });
} catch (error) {
logger.error('Failed to build S3 download alias indexes', { error });
}
defineIndex(S3DownloadAliasMongoSchema, {
key: { aliasId: 1 },
options: { unique: true }
});
defineIndex(S3DownloadAliasMongoSchema, {
key: { aliasKey: 1 },
options: { unique: true }
});
defineIndex(S3DownloadAliasMongoSchema, {
key: { purgeAt: 1 },
options: { expireAfterSeconds: 0 }
});
defineIndex(S3DownloadAliasMongoSchema, {
key: { bucketName: 1, objectKey: 1 }
});
export const MongoS3DownloadAlias = getMongoModel<S3DownloadAliasType>(
S3DownloadAliasCollectionName,
......
packages/service/common/s3/accessLink/uploadSession/schema.ts
import { getLogger, LogCategories } from '../../../logger';
import { getMongoModel, Schema } from '../../../mongo';
import { defineIndex, getMongoModel, Schema } from '../../../mongo';
import type { S3UploadSessionType } from '../type';
export const S3UploadSessionCollectionName = 's3_upload_sessions';
......@@ -42,13 +42,17 @@ const S3UploadSessionMongoSchema = new Schema({
revokedAt: Date
});
try {
S3UploadSessionMongoSchema.index({ tokenHash: 1 }, { unique: true });
S3UploadSessionMongoSchema.index({ expiresAt: 1 }, { expireAfterSeconds: 0 });
S3UploadSessionMongoSchema.index({ bucketName: 1, objectKey: 1 });
} catch (error) {
logger.error('Failed to build S3 upload session indexes', { error });
}
defineIndex(S3UploadSessionMongoSchema, {
key: { tokenHash: 1 },
options: { unique: true }
});
defineIndex(S3UploadSessionMongoSchema, {
key: { expiresAt: 1 },
options: { expireAfterSeconds: 0 }
});
defineIndex(S3UploadSessionMongoSchema, {
key: { bucketName: 1, objectKey: 1 }
});
export const MongoS3UploadSession = getMongoModel<S3UploadSessionType>(
S3UploadSessionCollectionName,
......
packages/service/common/s3/models/ttl.ts
import { Schema, getMongoModel } from '../../mongo';
import { defineIndex, Schema, getMongoModel } from '../../mongo';
import { type S3TtlSchemaType } from '@fastgpt/global/common/file/s3TTL/type';
const collectionName = 's3_ttls';
......@@ -18,7 +18,7 @@ const S3TTLSchema = new Schema({
}
});
S3TTLSchema.index({ expiredTime: 1 });
S3TTLSchema.index({ bucketName: 1, minioKey: 1 });
defineIndex(S3TTLSchema, { key: { expiredTime: 1 } });
defineIndex(S3TTLSchema, { key: { bucketName: 1, minioKey: 1 } });
export const MongoS3TTL = getMongoModel<S3TtlSchemaType>(collectionName, S3TTLSchema);
packages/service/common/system/config/schema.ts
import { type SystemConfigsType } from '@fastgpt/global/common/system/config/type';
import { connectionMongo, getMongoModel, type Model } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel, type Model } from '../../../common/mongo';
import { SystemConfigsTypeMap } from '@fastgpt/global/common/system/config/constants';
import { getLogger, LogCategories } from '../../logger';
......@@ -23,11 +23,7 @@ const systemConfigSchema = new Schema({
}
});
try {
systemConfigSchema.index({ type: 1 });
} catch (error) {
logger.error('Failed to build system config indexes', { error });
}
defineIndex(systemConfigSchema, { key: { type: 1 } });
export const MongoSystemConfigs = getMongoModel<SystemConfigsType>(
collectionName,
......
packages/service/common/system/countLimit/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import type { CountLimitType } from './type';
import { getLogger, LogCategories } from '../../logger';
......@@ -26,11 +26,13 @@ const CountLimitSchema = new Schema({
}
});
try {
CountLimitSchema.index({ type: 1, key: 1 }, { unique: true });
CountLimitSchema.index({ createTime: 1 }, { expireAfterSeconds: 60 * 60 * 24 * 30 }); // ttl 30天
} catch (error) {
logger.error('Failed to build count limit indexes', { error });
}
defineIndex(CountLimitSchema, {
key: { type: 1, key: 1 },
options: { unique: true }
});
defineIndex(CountLimitSchema, {
key: { createTime: 1 },
options: { expireAfterSeconds: 60 * 60 * 24 * 30 }
}); // ttl 30天
export const MongoCountLimit = getMongoModel<CountLimitType>(collectionName, CountLimitSchema);
packages/service/common/system/frequencyLimit/schema.ts
import { getMongoModel, Schema } from '../../mongo';
import { defineIndex, getMongoModel, Schema } from '../../mongo';
import type { FrequencyLimitSchemaType } from './type';
const FrequencyLimitSchema = new Schema({
......@@ -16,10 +16,11 @@ const FrequencyLimitSchema = new Schema({
}
});
try {
FrequencyLimitSchema.index({ eventId: 1, expiredTime: 1 });
FrequencyLimitSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 0 });
} catch (error) {}
defineIndex(FrequencyLimitSchema, { key: { eventId: 1, expiredTime: 1 } });
defineIndex(FrequencyLimitSchema, {
key: { expiredTime: 1 },
options: { expireAfterSeconds: 0 }
});
export const MongoFrequencyLimit = getMongoModel<FrequencyLimitSchemaType>(
'frequency_limit',
......
packages/service/common/system/log/schema.ts
import { getMongoLogModel as getMongoModel, Schema } from '../../../common/mongo';
import { defineIndex, getMongoLogModel as getMongoModel, Schema } from '../../../common/mongo';
import { type SystemLogType } from './type';
import { LogLevelEnum } from './constant';
......@@ -22,8 +22,11 @@ export const getMongoLog = () => {
metadata: Object
});
SystemLogSchema.index({ time: 1 }, { expires: '15d' });
SystemLogSchema.index({ level: 1 });
defineIndex(SystemLogSchema, {
key: { time: 1 },
options: { expires: '15d' }
});
defineIndex(SystemLogSchema, { key: { level: 1 } });
return getMongoModel<SystemLogType>(LogCollectionName, SystemLogSchema);
};
packages/service/common/system/timerLock/schema.ts
import { connectionMongo, getMongoModel } from '../../mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../mongo';
const { Schema } = connectionMongo;
import { type TimerLockSchemaType } from './type';
import { getLogger, LogCategories } from '../../logger';
......@@ -9,8 +9,7 @@ const logger = getLogger(LogCategories.INFRA.MONGO);
const TimerLockSchema = new Schema({
timerId: {
type: String,
required: true,
unique: true
required: true
},
expiredTime: {
type: Date,
......@@ -18,10 +17,13 @@ const TimerLockSchema = new Schema({
}
});
try {
TimerLockSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 5 });
} catch (error) {
logger.error('Failed to build timer lock indexes', { error });
}
defineIndex(TimerLockSchema, {
key: { timerId: 1 },
options: { unique: true }
});
defineIndex(TimerLockSchema, {
key: { expiredTime: 1 },
options: { expireAfterSeconds: 5 }
});
export const MongoTimerLock = getMongoModel<TimerLockSchemaType>(collectionName, TimerLockSchema);
packages/service/core/ai/config/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { connectionMongo, defineIndex, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import type { SystemModelSchemaType } from '../type';
......@@ -6,8 +6,7 @@ const SystemModelSchema = new Schema(
{
model: {
type: String,
required: true,
unique: true
required: true
},
metadata: {
type: Object,
......@@ -20,6 +19,11 @@ const SystemModelSchema = new Schema(
}
);
defineIndex(SystemModelSchema, {
key: { model: 1 },
options: { unique: true }
});
export const MongoSystemModel = getMongoModel<SystemModelSchemaType>(
'system_models',
SystemModelSchema
......
packages/service/core/ai/record/schema.ts
import { getMongoLogModel, Schema } from '../../../common/mongo';
import { defineIndex, getMongoLogModel, Schema } from '../../../common/mongo';
import type { LLMRequestRecordSchemaType } from '@fastgpt/global/openapi/core/ai/api';
import { serviceEnv } from '../../../env';
......@@ -30,7 +30,10 @@ const LLMRequestRecordSchema = new Schema({
}
});
LLMRequestRecordSchema.index({ teamId: 1, requestId: 1 }, { unique: true });
defineIndex(LLMRequestRecordSchema, {
key: { teamId: 1, requestId: 1 },
options: { unique: true }
});
export const MongoLLMRequestRecord = getMongoLogModel<LLMRequestRecordSchemaType>(
LLMRequestRecordCollectionName,
......
packages/service/core/ai/sandbox/infrastructure/instance/schema.ts
......@@ -3,7 +3,7 @@
*
* 只描述本地实例记录结构，不编排 provider、归档或运行态流程。
*/
import { connectionMongo, getMongoModel } from '../../../../../common/mongo';
import { connectionMongo, defineIndex, getMongoModel } from '../../../../../common/mongo';
const { Schema } = connectionMongo;
import type { SandboxInstanceSchemaType } from '../../type';
import { SandboxStatusEnum, SandboxTypeEnum } from '@fastgpt/global/core/ai/sandbox/constants';
......@@ -75,12 +75,25 @@ const SandboxInstanceSchema = new Schema({
}
});
SandboxInstanceSchema.index({ provider: 1, sandboxId: 1 }, { unique: true });
SandboxInstanceSchema.index({ sourceType: 1, sourceId: 1, chatId: 1 });
SandboxInstanceSchema.index({ sourceType: 1, status: 1, provider: 1, 'metadata.archive.state': 1 });
SandboxInstanceSchema.index({ status: 1, lastActiveAt: 1, 'metadata.archive.state': 1 });
SandboxInstanceSchema.index({ 'metadata.archive.state': 1, 'metadata.archive.startedAt': 1 });
SandboxInstanceSchema.index({ 'metadata.archive.state': 1, 'metadata.archive.deleteStartedAt': 1 });
defineIndex(SandboxInstanceSchema, {
key: { provider: 1, sandboxId: 1 },
options: { unique: true }
});
defineIndex(SandboxInstanceSchema, {
key: { sourceType: 1, sourceId: 1, chatId: 1 }
});
defineIndex(SandboxInstanceSchema, {
key: { sourceType: 1, status: 1, provider: 1, 'metadata.archive.state': 1 }
});
defineIndex(SandboxInstanceSchema, {
key: { status: 1, lastActiveAt: 1, 'metadata.archive.state': 1 }
});
defineIndex(SandboxInstanceSchema, {
key: { 'metadata.archive.state': 1, 'metadata.archive.startedAt': 1 }
});
defineIndex(SandboxInstanceSchema, {
key: { 'metadata.archive.state': 1, 'metadata.archive.deleteStartedAt': 1 }
});
/**
* sandbox 实例 Mongo model。
......
packages/service/core/ai/skill/model/schema.ts
import { connectionMongo, getMongoModel } from '../../../../common/mongo';
import { connectionMongo, defineIndex, getMongoModel } from '../../../../common/mongo';
import {
agentSkillsCollectionName,
agentSkillsVersionCollectionName,
......@@ -121,13 +121,17 @@ const AgentSkillsSchema = new Schema({
});
// 名称和描述用于列表页搜索。
AgentSkillsSchema.index({ teamId: 1, name: 'text', description: 'text' });
defineIndex(AgentSkillsSchema, {
key: { teamId: 1, name: 'text', description: 'text' }
});
// 列表页按来源、团队、删除状态和创建时间过滤排序。
AgentSkillsSchema.index({ source: 1, teamId: 1, deleteTime: 1, createTime: -1 });
defineIndex(AgentSkillsSchema, {
key: { source: 1, teamId: 1, deleteTime: 1, createTime: -1 }
});
// 分类筛选。
AgentSkillsSchema.index({ category: 1 });
// 文件夹树查询。
AgentSkillsSchema.index({ teamId: 1, parentId: 1, deleteTime: 1 });
defineIndex(AgentSkillsSchema, { key: { category: 1 } });
// 文件夹树查询：findSkillAndAllChildren 按 teamId + parentId + deleteTime 逐层查子节点。
defineIndex(AgentSkillsSchema, { key: { teamId: 1, parentId: 1, deleteTime: 1 } });
export const MongoAgentSkills = getMongoModel<MongoAgentSkillSchemaType>(
agentSkillsCollectionName,
......
packages/service/core/ai/skill/version/schema.ts
import { connectionMongo, getMongoModel } from '../../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../../common/mongo';
import {
agentSkillsCollectionName,
agentSkillsVersionCollectionName
......@@ -52,7 +52,9 @@ const AgentSkillsVersionSchema = new Schema({
});
// 版本列表按 skillId 查询并按创建时间倒序展示。
AgentSkillsVersionSchema.index({ skillId: 1, createdAt: -1, _id: -1 });
defineIndex(AgentSkillsVersionSchema, {
key: { skillId: 1, createdAt: -1, _id: -1 }
});
export const MongoAgentSkillsVersion = getMongoModel<AgentSkillsVersionSchemaType>(
agentSkillsVersionCollectionName,
......
packages/service/core/app/evaluation/evalItemSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { EvaluationCollectionName } from './evalSchema';
import {
EvaluationStatusEnum,
......@@ -48,7 +48,7 @@ const EvalItemSchema = new Schema({
errorMessage: String
});
EvalItemSchema.index({ evalId: 1, status: 1 });
defineIndex(EvalItemSchema, { key: { evalId: 1, status: 1 } });
export const MongoEvalItem = getMongoModel<EvalItemSchemaType>(
EvalItemCollectionName,
......
packages/service/core/app/evaluation/evalSchema.ts
......@@ -2,7 +2,7 @@ import {
TeamCollectionName,
TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { AppCollectionName } from '../schema';
import type { EvaluationSchemaType } from '@fastgpt/global/core/app/evaluation/type';
import { UsageCollectionName } from '../../../support/wallet/usage/constants';
......@@ -49,7 +49,7 @@ const EvaluationSchema = new Schema({
errorMessage: String
});
EvaluationSchema.index({ teamId: 1 });
defineIndex(EvaluationSchema, { key: { teamId: 1 } });
export const MongoEvaluation = getMongoModel<EvaluationSchemaType>(
EvaluationCollectionName,
......
packages/service/core/app/logs/chatLogsSchema.ts
import type { AppChatLogSchema } from '@fastgpt/global/core/app/logs/type';
import { getMongoLogModel, Schema } from '../../../common/mongo';
import { defineIndex, getMongoLogModel, Schema } from '../../../common/mongo';
import { AppCollectionName } from '../schema';
export const ChatLogCollectionName = 'app_chat_logs';
......@@ -69,17 +69,21 @@ const ChatLogSchema = new Schema({
});
// Get chart data
ChatLogSchema.index({ teamId: 1, appId: 1, source: 1, updateTime: -1 });
defineIndex(ChatLogSchema, {
key: { teamId: 1, appId: 1, source: 1, updateTime: -1 }
});
// Get chart data isFirstChat
ChatLogSchema.index({ isFirstChat: 1, teamId: 1, appId: 1, source: 1, createTime: -1 });
defineIndex(ChatLogSchema, {
key: { isFirstChat: 1, teamId: 1, appId: 1, source: 1, createTime: -1 }
});
// Get userStats
ChatLogSchema.index({ teamId: 1, appId: 1, userId: 1 });
defineIndex(ChatLogSchema, { key: { teamId: 1, appId: 1, userId: 1 } });
// Admin get chat form data - optimized for aggregation with appId/chatId grouping
ChatLogSchema.index({ createTime: -1, appId: 1, chatId: 1 });
defineIndex(ChatLogSchema, { key: { createTime: -1, appId: 1, chatId: 1 } });
// Init shell
ChatLogSchema.index({ teamId: 1, appId: 1, chatId: 1 });
defineIndex(ChatLogSchema, { key: { teamId: 1, appId: 1, chatId: 1 } });
export const MongoAppChatLog = getMongoLogModel<AppChatLogSchema>(
ChatLogCollectionName,
......
packages/service/core/app/logs/logkeysSchema.ts
import type { AppLogKeysSchemaType } from '@fastgpt/global/core/app/logs/type';
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { AppCollectionName } from '../schema';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
......@@ -24,7 +24,7 @@ const AppLogKeysSchema = new Schema({
}
});
AppLogKeysSchema.index({ teamId: 1, appId: 1 });
defineIndex(AppLogKeysSchema, { key: { teamId: 1, appId: 1 } });
export const MongoAppLogKeys = getMongoModel<AppLogKeysSchemaType>(
AppLogKeysCollectionEnum,
......
packages/service/core/app/record/schema.ts
......@@ -2,7 +2,7 @@ import {
TeamCollectionName,
TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant';
import { getMongoModel, Schema } from '../../../common/mongo';
import { defineIndex, getMongoModel, Schema } from '../../../common/mongo';
import { AppCollectionName } from '../schema';
import type { AppRecordType } from './type';
......@@ -35,9 +35,12 @@ const AppRecordSchema = new Schema(
}
);
AppRecordSchema.index({ tmbId: 1, lastUsedTime: -1 }); // 查询用户最近使用的应用
AppRecordSchema.index({ tmbId: 1, appId: 1 }, { unique: true }); // 防止重复记录
AppRecordSchema.index({ teamId: 1, appId: 1 }); // 用于清理权限失效的记录
defineIndex(AppRecordSchema, { key: { tmbId: 1, lastUsedTime: -1 } }); // 查询用户最近使用的应用
defineIndex(AppRecordSchema, {
key: { tmbId: 1, appId: 1 },
options: { unique: true }
}); // 防止重复记录
defineIndex(AppRecordSchema, { key: { teamId: 1, appId: 1 } }); // 用于清理权限失效的记录
export const MongoAppRecord = getMongoModel<AppRecordType>(
AppRecordCollectionName,
......
packages/service/core/app/schema.ts
import { AppTypeEnum } from '@fastgpt/global/core/app/constants';
import { Schema, getMongoModel } from '../../common/mongo';
import { defineIndex, Schema, getMongoModel } from '../../common/mongo';
import type { AppSchemaType as AppType } from '@fastgpt/global/core/app/type';
import {
TeamCollectionName,
......@@ -134,24 +134,26 @@ const AppSchema = new Schema(
}
);
AppSchema.index({ teamId: 1, updateTime: -1 });
AppSchema.index({ teamId: 1, type: 1 });
AppSchema.index({ teamId: 1, deleteTime: 1, 'resourceRefs.skillIds': 1 });
defineIndex(AppSchema, { key: { teamId: 1, updateTime: -1 } });
defineIndex(AppSchema, { key: { teamId: 1, type: 1 } });
defineIndex(AppSchema, {
key: { teamId: 1, deleteTime: 1, 'resourceRefs.skillIds': 1 }
});
// Schedule
AppSchema.index(
{ scheduledTriggerConfig: 1, scheduledTriggerNextTime: -1 },
{
defineIndex(AppSchema, {
key: { scheduledTriggerConfig: 1, scheduledTriggerNextTime: -1 },
options: {
partialFilterExpression: {
scheduledTriggerConfig: { $exists: true }
}
}
);
});
// Admin count
AppSchema.index({ type: 1 });
AppSchema.index({ deleteTime: 1 });
defineIndex(AppSchema, { key: { type: 1 } });
defineIndex(AppSchema, { key: { deleteTime: 1 } });
// Admin search
AppSchema.index({ name: 1 });
defineIndex(AppSchema, { key: { name: 1 } });
export const MongoApp = getMongoModel<AppType>(AppCollectionName, AppSchema);
packages/service/core/app/templates/templateSchema.ts
import { type AppTemplateSchemaType } from '@fastgpt/global/core/app/type';
import { connectionMongo, getMongoModel } from '../../../common/mongo/index';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo/index';
import { UserTagsSchema } from '@fastgpt/global/support/user/type';
const { Schema } = connectionMongo;
......@@ -39,7 +39,7 @@ const AppTemplateSchema = new Schema({
workflow: Object
});
AppTemplateSchema.index({ templateId: 1 });
defineIndex(AppTemplateSchema, { key: { templateId: 1 } });
export const MongoAppTemplate = getMongoModel<AppTemplateSchemaType>(
collectionName,
......
packages/service/core/app/version/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import { type AppVersionSchemaType } from '@fastgpt/global/core/app/version/type';
import { AppCollectionName, chatConfigType } from '../schema';
......@@ -48,7 +48,7 @@ const AppVersionSchema = new Schema(
}
);
AppVersionSchema.index({ appId: 1, time: -1 });
defineIndex(AppVersionSchema, { key: { appId: 1, time: -1 } });
export const MongoAppVersion = getMongoModel<AppVersionSchemaType>(
AppVersionCollectionName,
......
packages/service/core/chat/chatItemResponseSchema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo;
import type { ChatItemResponseSchemaType } from '@fastgpt/global/core/chat/type';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
......@@ -42,16 +42,20 @@ const ChatItemResponseSchema = new Schema({
/* TODO: 未全面检查操作，所以这里暂时不加 sourceType 的索引。 */
// 按 chat item 拉取完整 nodeResponse rows；复合索引包含 _id，避免详情读取时额外排序。
ChatItemResponseSchema.index({ appId: 1, chatId: 1, chatItemDataId: 1, _id: 1 });
ChatItemResponseSchema.index({
defineIndex(ChatItemResponseSchema, {
key: { appId: 1, chatId: 1, chatItemDataId: 1, _id: 1 }
});
defineIndex(ChatItemResponseSchema, {
key: {
sourceType: 1,
appId: 1,
chatId: 1,
chatItemDataId: 1,
_id: 1
}
});
// Clear expired response
ChatItemResponseSchema.index({ teamId: 1, time: -1 });
defineIndex(ChatItemResponseSchema, { key: { teamId: 1, time: -1 } });
export const MongoChatItemResponse = getMongoModel<ChatItemResponseSchemaType>(
ChatItemResponseCollectionName,
......
packages/service/core/chat/chatItemSchema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo;
import { type ChatItemDBSchemaType } from '@fastgpt/global/core/chat/type';
import { ChatRoleMap, ChatSourceTypeEnum } from '@fastgpt/global/core/chat/constants';
......@@ -103,17 +103,25 @@ const ChatItemSchema = new Schema({
delete by chat id;
close custom feedback;
*/
ChatItemSchema.index({ appId: 1, chatId: 1, dataId: 1 });
ChatItemSchema.index({ sourceType: 1, appId: 1, chatId: 1, dataId: 1 });
defineIndex(ChatItemSchema, { key: { appId: 1, chatId: 1, dataId: 1 } });
defineIndex(ChatItemSchema, {
key: { sourceType: 1, appId: 1, chatId: 1, dataId: 1 }
});
// Get histories
ChatItemSchema.index({ appId: 1, chatId: 1, deleteTime: 1 });
ChatItemSchema.index({ sourceType: 1, appId: 1, chatId: 1, deleteTime: 1 });
defineIndex(ChatItemSchema, { key: { appId: 1, chatId: 1, deleteTime: 1 } });
defineIndex(ChatItemSchema, {
key: { sourceType: 1, appId: 1, chatId: 1, deleteTime: 1 }
});
// get chatitem list,Anchor filter
ChatItemSchema.index({ appId: 1, chatId: 1, _id: -1 });
ChatItemSchema.index({ sourceType: 1, appId: 1, chatId: 1, _id: -1 });
defineIndex(ChatItemSchema, { key: { appId: 1, chatId: 1, _id: -1 } });
defineIndex(ChatItemSchema, {
key: { sourceType: 1, appId: 1, chatId: 1, _id: -1 }
});
// Query by role (AI/Human), get latest chat item, permission check
ChatItemSchema.index({ appId: 1, chatId: 1, obj: 1, _id: -1 });
ChatItemSchema.index({ sourceType: 1, appId: 1, chatId: 1, obj: 1, _id: -1 });
defineIndex(ChatItemSchema, { key: { appId: 1, chatId: 1, obj: 1, _id: -1 } });
defineIndex(ChatItemSchema, {
key: { sourceType: 1, appId: 1, chatId: 1, obj: 1, _id: -1 }
});
export const MongoChatItem = getMongoModel<ChatItemDBSchemaType>(
ChatItemCollectionName,
......
packages/service/core/chat/chatSchema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo';
import { connectionMongo, defineIndex, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo;
import { type ChatSchemaType } from '@fastgpt/global/core/chat/type';
import {
......@@ -133,120 +133,127 @@ const ChatSchema = new Schema({
userId: Schema.Types.ObjectId
});
ChatSchema.index({ chatId: 1 });
defineIndex(ChatSchema, { key: { chatId: 1 } });
// Delete by appid; init chat; update chat; auth chat;
ChatSchema.index({ sourceType: 1, appId: 1, chatId: 1 }, { unique: true });
defineIndex(ChatSchema, {
key: { sourceType: 1, appId: 1, chatId: 1 },
options: { unique: true }
});
// timer, clear history
ChatSchema.index({ updateTime: -1, teamId: 1 });
ChatSchema.index({ teamId: 1, updateTime: -1 });
defineIndex(ChatSchema, { key: { updateTime: -1, teamId: 1 } });
defineIndex(ChatSchema, { key: { teamId: 1, updateTime: -1 } });
// get user history(Cookie)
ChatSchema.index({ tmbId: 1, appId: 1, deleteTime: 1, top: -1, updateTime: -1 });
defineIndex(ChatSchema, {
key: { tmbId: 1, appId: 1, deleteTime: 1, top: -1, updateTime: -1 }
});
/* ===== 条件索引 ===== */
// Clear history(share),Init 4121
ChatSchema.index(
{ appId: 1, outLinkUid: 1, tmbId: 1 },
{
defineIndex(ChatSchema, {
key: { appId: 1, outLinkUid: 1, tmbId: 1 },
options: {
partialFilterExpression: {
outLinkUid: { $exists: true }
}
}
);
});
// get share chat history
ChatSchema.index(
{ shareId: 1, outLinkUid: 1, updateTime: -1 },
{
defineIndex(ChatSchema, {
key: { shareId: 1, outLinkUid: 1, updateTime: -1 },
options: {
partialFilterExpression: {
shareId: { $exists: true }
}
}
);
});
/* get chat logs */
// 1. Common get
ChatSchema.index({ appId: 1, updateTime: -1 });
defineIndex(ChatSchema, { key: { appId: 1, updateTime: -1 } });
// Get history(tmbId)
ChatSchema.index({ appId: 1, tmbId: 1, updateTime: -1 });
defineIndex(ChatSchema, { key: { appId: 1, tmbId: 1, updateTime: -1 } });
// clearHistory(API)
ChatSchema.index({ appId: 1, source: 1, tmbId: 1, updateTime: -1 });
defineIndex(ChatSchema, {
key: { appId: 1, source: 1, tmbId: 1, updateTime: -1 }
});
// Periodic cleanup for chats stuck in generating state.
ChatSchema.index(
{ chatGenerateStatus: 1, updateTime: 1 },
{
defineIndex(ChatSchema, {
key: { chatGenerateStatus: 1, updateTime: 1 },
options: {
partialFilterExpression: {
chatGenerateStatus: ChatGenerateStatusEnum.generating
}
}
);
});
/* 反馈过滤的索引 */
// 2. Has good feedback filter
ChatSchema.index(
{
defineIndex(ChatSchema, {
key: {
appId: 1,
hasGoodFeedback: 1,
updateTime: -1
},
{
options: {
partialFilterExpression: {
hasGoodFeedback: true
}
}
);
});
// Has bad feedback filter
ChatSchema.index(
{
defineIndex(ChatSchema, {
key: {
appId: 1,
hasBadFeedback: 1,
updateTime: -1
},
{
options: {
partialFilterExpression: {
hasBadFeedback: true
}
}
);
});
// 3. Has unread good feedback filter
ChatSchema.index(
{
defineIndex(ChatSchema, {
key: {
appId: 1,
hasUnreadGoodFeedback: 1,
updateTime: -1
},
{
options: {
partialFilterExpression: {
hasUnreadGoodFeedback: true
}
}
);
});
// Has unread bad feedback filter
ChatSchema.index(
{
defineIndex(ChatSchema, {
key: {
appId: 1,
hasUnreadBadFeedback: 1,
updateTime: -1
},
{
options: {
partialFilterExpression: {
hasUnreadBadFeedback: true
}
}
);
});
// Has error filter
ChatSchema.index(
{
defineIndex(ChatSchema, {
key: {
appId: 1,
errorCount: 1,
updateTime: -1
},
{
options: {
partialFilterExpression: {
errorCount: { $gt: 0 }
}
}
);
});
export const MongoChat = getMongoModel<ChatSchemaType>(chatCollectionName, ChatSchema);
packages/service/core/chat/favouriteApp/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { type ChatFavouriteAppType } from '@fastgpt/global/core/chat/favouriteApp/type';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { AppCollectionName } from '../../app/schema';
......@@ -28,7 +28,7 @@ const ChatFavouriteAppSchema = new Schema({
}
});
ChatFavouriteAppSchema.index({ teamId: 1, appId: 1 });
defineIndex(ChatFavouriteAppSchema, { key: { teamId: 1, appId: 1 } });
export const MongoChatFavouriteApp = getMongoModel<ChatFavouriteAppType>(
ChatFavouriteAppCollectionName,
......
packages/service/core/chat/inputGuide/schema.ts
import { AppCollectionName } from '../../app/schema';
import { connectionMongo, getMongoModel, type Model } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel, type Model } from '../../../common/mongo';
const { Schema, model, models } = connectionMongo;
import type { ChatInputGuideSchemaType } from '@fastgpt/global/core/chat/inputGuide/type';
import { getLogger, LogCategories } from '../../../common/logger';
export const ChatInputGuideCollectionName = 'chat_input_guides';
......@@ -18,12 +17,10 @@ const ChatInputGuideSchema = new Schema({
}
});
try {
ChatInputGuideSchema.index({ appId: 1, text: 1 }, { unique: true });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build chat input guide indexes', { error });
}
defineIndex(ChatInputGuideSchema, {
key: { appId: 1, text: 1 },
options: { unique: true }
});
export const MongoChatInputGuide = getMongoModel<ChatInputGuideSchemaType>(
ChatInputGuideCollectionName,
......
packages/service/core/chat/setting/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { type ChatSettingModelType } from '@fastgpt/global/core/chat/setting/type';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { AppCollectionName } from '../../app/schema';
......@@ -53,7 +53,7 @@ ChatSettingSchema.virtual('quickAppList', {
foreignField: '_id'
});
ChatSettingSchema.index({ teamId: 1 });
defineIndex(ChatSettingSchema, { key: { teamId: 1 } });
export const MongoChatSetting = getMongoModel<ChatSettingModelType>(
ChatSettingCollectionName,
......
packages/service/core/dataset/collection/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { getLogger, LogCategories } from '../../../common/logger';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import { type DatasetCollectionSchemaType } from '@fastgpt/global/core/dataset/type';
import { DatasetCollectionTypeMap } from '@fastgpt/global/core/dataset/constants';
......@@ -94,43 +93,46 @@ DatasetCollectionSchema.virtual('dataset', {
justOne: true
});
try {
// auth file
DatasetCollectionSchema.index({ teamId: 1, fileId: 1 });
// auth file
defineIndex(DatasetCollectionSchema, { key: { teamId: 1, fileId: 1 } });
// list collection; deep find collections
DatasetCollectionSchema.index({
// list collection; deep find collections
defineIndex(DatasetCollectionSchema, {
key: {
teamId: 1,
datasetId: 1,
parentId: 1,
updateTime: -1
});
}
});
// Tag filter
DatasetCollectionSchema.index({ teamId: 1, datasetId: 1, tags: 1 });
// create time filter
DatasetCollectionSchema.index({ teamId: 1, datasetId: 1, createTime: 1 });
// Tag filter
defineIndex(DatasetCollectionSchema, {
key: { teamId: 1, datasetId: 1, tags: 1 }
});
// create time filter
defineIndex(DatasetCollectionSchema, {
key: { teamId: 1, datasetId: 1, createTime: 1 }
});
// Get collection by external file id
DatasetCollectionSchema.index(
{ datasetId: 1, externalFileId: 1 },
{
// Get collection by external file id
defineIndex(DatasetCollectionSchema, {
key: { datasetId: 1, externalFileId: 1 },
options: {
unique: true,
partialFilterExpression: {
externalFileId: { $exists: true }
}
}
);
});
// Clear invalid image
DatasetCollectionSchema.index({
// Clear invalid image
defineIndex(DatasetCollectionSchema, {
key: {
teamId: 1,
'metadata.relatedImgId': 1
});
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset collection indexes', { error });
}
}
});
export const MongoDatasetCollection = getMongoModel<DatasetCollectionSchemaType>(
DatasetColCollectionName,
......
packages/service/core/dataset/data/dataTextSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import { type DatasetDataTextSchemaType } from '@fastgpt/global/core/dataset/type';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { DatasetCollectionName } from '../schema';
import { DatasetColCollectionName } from '../collection/schema';
import { DatasetDataCollectionName } from './schema';
import { getLogger, LogCategories } from '../../../common/logger';
export const DatasetDataTextCollectionName = 'dataset_data_texts';
......@@ -33,20 +32,17 @@ const DatasetDataTextSchema = new Schema({
fullTextToken: String
});
try {
DatasetDataTextSchema.index(
{ teamId: 1, fullTextToken: 'text' },
{
defineIndex(DatasetDataTextSchema, {
key: { teamId: 1, fullTextToken: 'text' },
options: {
name: 'teamId_1_fullTextToken_text',
default_language: 'none'
}
);
DatasetDataTextSchema.index({ teamId: 1, datasetId: 1, collectionId: 1 });
DatasetDataTextSchema.index({ dataId: 'hashed' });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset data text indexes', { error });
}
});
defineIndex(DatasetDataTextSchema, {
key: { teamId: 1, datasetId: 1, collectionId: 1 }
});
defineIndex(DatasetDataTextSchema, { key: { dataId: 'hashed' } });
export const MongoDatasetDataText = getMongoModel<DatasetDataTextSchemaType>(
DatasetDataTextCollectionName,
......
packages/service/core/dataset/data/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema, model, models } = connectionMongo;
import { type DatasetDataSchemaType } from '@fastgpt/global/core/dataset/type';
import {
......@@ -8,7 +8,6 @@ import {
import { DatasetCollectionName } from '../schema';
import { DatasetColCollectionName } from '../collection/schema';
import { DatasetDataIndexTypeEnum } from '@fastgpt/global/core/dataset/data/constants';
import { getLogger, LogCategories } from '../../../common/logger';
export const DatasetDataCollectionName = 'dataset_datas';
......@@ -89,26 +88,27 @@ const DatasetDataSchema = new Schema({
initJieba: Boolean
});
try {
// list collection and count data; list data; delete collection(relate data)
DatasetDataSchema.index({
// list collection and count data; list data; delete collection(relate data)
defineIndex(DatasetDataSchema, {
key: {
teamId: 1,
datasetId: 1,
collectionId: 1,
chunkIndex: 1,
updateTime: -1
});
// Recall vectors after data matching
DatasetDataSchema.index({ teamId: 1, datasetId: 1, collectionId: 1, 'indexes.dataId': 1 });
// rebuild data
DatasetDataSchema.index({ rebuilding: 1, teamId: 1, datasetId: 1 });
}
});
// Recall vectors after data matching
defineIndex(DatasetDataSchema, {
key: { teamId: 1, datasetId: 1, collectionId: 1, 'indexes.dataId': 1 }
});
// rebuild data
defineIndex(DatasetDataSchema, {
key: { rebuilding: 1, teamId: 1, datasetId: 1 }
});
// Cron clear invalid data
DatasetDataSchema.index({ updateTime: 1 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset data indexes', { error });
}
// Cron clear invalid data
defineIndex(DatasetDataSchema, { key: { updateTime: 1 } });
export const MongoDatasetData = getMongoModel<DatasetDataSchemaType>(
DatasetDataCollectionName,
......
packages/service/core/dataset/image/schema.ts
import type { Types } from '../../../common/mongo';
import { getMongoModel, Schema } from '../../../common/mongo';
import { defineIndex, getMongoModel, Schema } from '../../../common/mongo';
export const bucketName = 'dataset_image';
......@@ -16,9 +16,13 @@ const MongoDatasetImage = new Schema({
expiredTime: { type: Date, required: true }
}
});
MongoDatasetImage.index({ 'metadata.datasetId': 'hashed' });
MongoDatasetImage.index({ 'metadata.collectionId': 'hashed' });
MongoDatasetImage.index({ 'metadata.expiredTime': -1 });
defineIndex(MongoDatasetImage, {
key: { 'metadata.datasetId': 'hashed' }
});
defineIndex(MongoDatasetImage, {
key: { 'metadata.collectionId': 'hashed' }
});
defineIndex(MongoDatasetImage, { key: { 'metadata.expiredTime': -1 } });
export const MongoDatasetImageSchema = getMongoModel<{
_id: Types.ObjectId;
......
packages/service/core/dataset/migration/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { getLogger, LogCategories } from '../../../common/logger';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
export const DatasetMigrationLogCollectionName = 'dataset_migration_logs';
......@@ -91,8 +90,7 @@ const DatasetMigrationLogSchema = new Schema({
// 批次信息
batchId: {
type: String,
required: true,
index: true
required: true
},
migrationVersion: {
type: String,
......@@ -107,17 +105,14 @@ const DatasetMigrationLogSchema = new Schema({
},
resourceId: {
type: Schema.Types.ObjectId,
required: true,
index: true
required: true
},
teamId: {
type: Schema.Types.ObjectId,
required: true,
index: true
required: true
},
datasetId: {
type: Schema.Types.ObjectId,
index: true
type: Schema.Types.ObjectId
},
// 存储信息
......@@ -151,15 +146,13 @@ const DatasetMigrationLogSchema = new Schema({
type: String,
enum: ['pending', 'processing', 'completed', 'failed', 'rollback', 'verified'],
default: 'pending',
required: true,
index: true
required: true
},
// 时间戳
createdAt: {
type: Date,
default: () => new Date(),
index: true
default: () => new Date()
},
startedAt: Date,
completedAt: Date,
......@@ -227,29 +220,38 @@ const DatasetMigrationLogSchema = new Schema({
});
// 索引优化
try {
// 查询某个批次的迁移状态
DatasetMigrationLogSchema.index({ batchId: 1, status: 1 });
// 查询某个资源的迁移历史
DatasetMigrationLogSchema.index({ resourceType: 1, resourceId: 1 });
defineIndex(DatasetMigrationLogSchema, { key: { batchId: 1 } });
defineIndex(DatasetMigrationLogSchema, { key: { resourceId: 1 } });
defineIndex(DatasetMigrationLogSchema, { key: { teamId: 1 } });
defineIndex(DatasetMigrationLogSchema, { key: { datasetId: 1 } });
defineIndex(DatasetMigrationLogSchema, { key: { status: 1 } });
defineIndex(DatasetMigrationLogSchema, { key: { createdAt: 1 } });
// 查询某个批次的迁移状态
defineIndex(DatasetMigrationLogSchema, { key: { batchId: 1, status: 1 } });
// 查询某个资源的迁移历史
defineIndex(DatasetMigrationLogSchema, {
key: { resourceType: 1, resourceId: 1 }
});
// 查询失败的迁移（需要重试）
DatasetMigrationLogSchema.index({
// 查询失败的迁移（需要重试）
defineIndex(DatasetMigrationLogSchema, {
key: {
status: 1,
attemptCount: 1,
lastAttemptAt: 1
});
}
});
// 查询某个团队的迁移情况
DatasetMigrationLogSchema.index({ teamId: 1, status: 1 });
// 查询某个团队的迁移情况
defineIndex(DatasetMigrationLogSchema, { key: { teamId: 1, status: 1 } });
// 唯一索引：同一个资源在同一个批次只能有一条记录
DatasetMigrationLogSchema.index({ batchId: 1, resourceType: 1, resourceId: 1 }, { unique: true });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset migration indexes', { error });
}
// 唯一索引：同一个资源在同一个批次只能有一条记录
defineIndex(DatasetMigrationLogSchema, {
key: { batchId: 1, resourceType: 1, resourceId: 1 },
options: { unique: true }
});
export const MongoDatasetMigrationLog = getMongoModel<DatasetMigrationLogSchemaType>(
DatasetMigrationLogCollectionName,
......
packages/service/core/dataset/schema.ts
import { getMongoModel, Schema } from '../../common/mongo';
import { defineIndex, getMongoModel, Schema } from '../../common/mongo';
import {
ChunkSettingModeEnum,
ChunkTriggerConfigTypeEnum,
......@@ -14,7 +14,6 @@ import {
} from '@fastgpt/global/support/user/team/constant';
import { userCollectionName } from '../../support/user/schema';
import type { DatasetSchemaType } from '@fastgpt/global/core/dataset/type';
import { getLogger, LogCategories } from '../../common/logger';
export const DatasetCollectionName = 'datasets';
......@@ -152,13 +151,8 @@ const DatasetSchema = new Schema({
yuqueServer: Object
});
try {
DatasetSchema.index({ teamId: 1 });
DatasetSchema.index({ type: 1 }); // Admin count
DatasetSchema.index({ deleteTime: 1 }); // 添加软删除字段索引
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset indexes', { error });
}
defineIndex(DatasetSchema, { key: { teamId: 1 } });
defineIndex(DatasetSchema, { key: { type: 1 } }); // Admin count
defineIndex(DatasetSchema, { key: { deleteTime: 1 } }); // 添加软删除字段索引
export const MongoDataset = getMongoModel<DatasetSchemaType>(DatasetCollectionName, DatasetSchema);
packages/service/core/dataset/tag/schema.ts
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel, type Model } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel, type Model } from '../../../common/mongo';
import { DatasetCollectionName } from '../schema';
import { type DatasetCollectionTagsSchemaType } from '@fastgpt/global/core/dataset/type';
import { getLogger, LogCategories } from '../../../common/logger';
const { Schema } = connectionMongo;
export const DatasetCollectionTagsName = 'dataset_collection_tags';
......@@ -24,12 +23,9 @@ const DatasetCollectionTagsSchema = new Schema({
}
});
try {
DatasetCollectionTagsSchema.index({ teamId: 1, datasetId: 1, tag: 1 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build dataset tag indexes', { error });
}
defineIndex(DatasetCollectionTagsSchema, {
key: { teamId: 1, datasetId: 1, tag: 1 }
});
export const MongoDatasetCollectionTags = getMongoModel<DatasetCollectionTagsSchemaType>(
DatasetCollectionTagsName,
......
packages/service/core/dataset/training/schema.ts
/* 模型的知识库 */
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import { type DatasetTrainingSchemaType } from '@fastgpt/global/core/dataset/type';
import { TrainingModeEnum } from '@fastgpt/global/core/dataset/constants';
......@@ -119,16 +119,23 @@ TrainingDataSchema.virtual('data', {
});
// lock training data(teamId); delete training data
TrainingDataSchema.index({ teamId: 1, datasetId: 1 });
defineIndex(TrainingDataSchema, { key: { teamId: 1, datasetId: 1 } });
// collection 级状态、错误列表、删除、详情
TrainingDataSchema.index({
defineIndex(TrainingDataSchema, {
key: {
teamId: 1,
datasetId: 1,
collectionId: 1
}
});
// get training data and sort
TrainingDataSchema.index({ mode: 1, retryCount: 1, lockTime: 1, weight: -1 });
TrainingDataSchema.index({ expireAt: 1 }, { expireAfterSeconds: 7 * 24 * 60 * 60 }); // 7 days
defineIndex(TrainingDataSchema, {
key: { mode: 1, retryCount: 1, lockTime: 1, weight: -1 }
});
defineIndex(TrainingDataSchema, {
key: { expireAt: 1 },
options: { expireAfterSeconds: 7 * 24 * 60 * 60 }
}); // 7 days
export const MongoDatasetTraining = getMongoModel<DatasetTrainingSchemaType>(
DatasetTrainingCollectionName,
......
packages/service/core/plugin/schema/teamInstalledPluginSchema.ts
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo/index';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo/index';
import type { TeamInstalledPluginSchemaType } from '@fastgpt/global/core/plugin/schema/type';
const { Schema } = connectionMongo;
......@@ -29,7 +29,10 @@ const TeamInstalledPluginSchema = new Schema({
}
});
TeamInstalledPluginSchema.index({ teamId: 1, pluginId: 1 }, { unique: true });
defineIndex(TeamInstalledPluginSchema, {
key: { teamId: 1, pluginId: 1 },
options: { unique: true }
});
export const MongoTeamInstalledPlugin = getMongoModel<TeamInstalledPluginSchemaType>(
collectionName,
......
packages/service/core/plugin/tool/pluginGroupSchema.ts
// 已弃用
import type { I18nStringStrictType } from '@fastgpt/global/sdk/fastgpt-plugin';
import { connectionMongo, getMongoModel } from '../../../common/mongo/index';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo/index';
const { Schema } = connectionMongo;
export const collectionName = 'app_plugin_groups';
......@@ -40,7 +40,10 @@ const PluginGroupSchema = new Schema({
}
});
PluginGroupSchema.index({ groupId: 1 }, { unique: true });
defineIndex(PluginGroupSchema, {
key: { groupId: 1 },
options: { unique: true }
});
export const MongoToolGroups = getMongoModel<SystemToolGroupSchemaType>(
collectionName,
......
packages/service/core/plugin/tool/systemToolSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo/index';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo/index';
const { Schema } = connectionMongo;
import type { SystemPluginToolCollectionType } from '@fastgpt/global/core/plugin/tool/type';
import type { PluginStatusType } from '@fastgpt/global/core/plugin/type';
......@@ -120,7 +120,7 @@ const SystemToolSchema = new Schema({
}
});
SystemToolSchema.index({ pluginId: 1 });
defineIndex(SystemToolSchema, { key: { pluginId: 1 } });
export const MongoSystemTool = getMongoModel<SystemPluginToolCollectionType>(
collectionName,
......
packages/service/core/plugin/tool/tagSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo/index';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo/index';
import type { SystemPluginToolTagType } from '@fastgpt/global/core/plugin/type';
const { Schema } = connectionMongo;
......@@ -7,8 +7,7 @@ export const collectionName = 'system_plugin_tool_tags';
const SystemPluginToolTagSchema = new Schema({
tagId: {
type: String,
required: true,
unique: true
required: true
},
tagName: {
type: Schema.Types.Mixed,
......@@ -24,7 +23,11 @@ const SystemPluginToolTagSchema = new Schema({
}
});
SystemPluginToolTagSchema.index({ tagOrder: 1 });
defineIndex(SystemPluginToolTagSchema, {
key: { tagId: 1 },
options: { unique: true }
});
defineIndex(SystemPluginToolTagSchema, { key: { tagOrder: 1 } });
export const MongoPluginToolTag = getMongoModel<SystemPluginToolTagType>(
collectionName,
......
packages/service/env.ts
......@@ -29,7 +29,6 @@ export const serviceEnv = createEnv({
server: {
// ==================== 基础配置 ====================
DB_MAX_LINK: IntSchema.min(1).default(5),
SYNC_INDEX: BoolSchema.default(true),
// ==================== 密钥 ====================
ROOT_KEY: z
......@@ -152,6 +151,9 @@ export const serviceEnv = createEnv({
'mongodb://myusername:mypassword@localhost:27017/fastgpt?authSource=admin&directConnection=true'
),
MONGODB_LOG_URI: z.string().optional(),
SYNC_INDEX: BoolSchema.default(true).meta({
description: '是否在启动时创建当前 MongoDB 索引并清理显式声明的废弃索引'
}),
// VectorDB
VECTOR_VQ_LEVEL: IntSchema.default(32).meta({
......
packages/service/support/activity/promotion/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import { type PromotionRecordSchema as PromotionRecordType } from '@fastgpt/global/support/activity/type';
import { userCollectionName } from '../../user/schema';
......@@ -30,7 +30,7 @@ const PromotionRecordSchema = new Schema({
}
});
PromotionRecordSchema.index({ userId: 1 });
defineIndex(PromotionRecordSchema, { key: { userId: 1 } });
export const MongoPromotionRecord = getMongoModel<PromotionRecordType>(
'promotionRecord',
......
packages/service/support/appRegistration/schema.ts
import { Schema, getMongoModel } from '../../common/mongo/index';
import { defineIndex, Schema, getMongoModel } from '../../common/mongo/index';
import { AppCollectionName } from '../../core/app/schema';
import { TeamMemberCollectionName } from '@fastgpt/global/support/user/team/constant';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
......@@ -25,7 +25,7 @@ const AppRegistrationSchema = new Schema({
}
});
AppRegistrationSchema.index({ teamId: 1 });
defineIndex(AppRegistrationSchema, { key: { teamId: 1 } });
export const MongoAppRegistration = getMongoModel(
AppRegistrationCollectionName,
......
packages/service/support/mcp/schema.ts
......@@ -2,7 +2,7 @@ import {
TeamCollectionName,
TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant';
import { Schema, getMongoModel } from '../../common/mongo';
import { defineIndex, Schema, getMongoModel } from '../../common/mongo';
import { type McpKeyType } from '@fastgpt/global/support/mcp/type';
import { getNanoid } from '@fastgpt/global/common/string/tools';
import { AppCollectionName } from '../../core/app/schema';
......@@ -17,7 +17,6 @@ const McpKeySchema = new Schema({
key: {
type: String,
required: true,
unique: true,
default: () => getNanoid(24)
},
teamId: {
......@@ -53,4 +52,9 @@ const McpKeySchema = new Schema({
}
});
defineIndex(McpKeySchema, {
key: { key: 1 },
options: { unique: true }
});
export const MongoMcpKey = getMongoModel<McpKeyType>(mcpCollectionName, McpKeySchema);
packages/service/support/openapi/schema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo;
import { type OpenApiSchema } from '@fastgpt/global/support/openapi/type';
import {
TeamCollectionName,
TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant';
import { getLogger, LogCategories } from '../../common/logger';
const OpenApiSchema = new Schema(
{
......@@ -70,15 +69,14 @@ const OpenApiSchema = new Schema(
}
);
try {
OpenApiSchema.index({ teamId: 1 });
OpenApiSchema.index({ apiKey: 1 });
OpenApiSchema.index({ teamId: 1, tmbId: 1, tagIds: 1, _id: -1 });
OpenApiSchema.index({ teamId: 1, tmbId: 1, appId: 1, _id: -1 });
OpenApiSchema.index({ teamId: 1, tmbId: 1, name: 1 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build OpenAPI indexes', { error });
}
defineIndex(OpenApiSchema, { key: { teamId: 1 } });
defineIndex(OpenApiSchema, { key: { apiKey: 1 } });
defineIndex(OpenApiSchema, {
key: { teamId: 1, tmbId: 1, tagIds: 1, _id: -1 }
});
defineIndex(OpenApiSchema, {
key: { teamId: 1, tmbId: 1, appId: 1, _id: -1 }
});
defineIndex(OpenApiSchema, { key: { teamId: 1, tmbId: 1, name: 1 } });
export const MongoOpenApi = getMongoModel<OpenApiSchema>('openapi', OpenApiSchema);
packages/service/support/openapi/tag/schema.ts
......@@ -2,8 +2,7 @@ import {
TeamCollectionName,
TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { getLogger, LogCategories } from '../../../common/logger';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import type { OpenApiTagType } from '@fastgpt/global/openapi/support/openapi/tag';
const { Schema } = connectionMongo;
......@@ -55,13 +54,13 @@ const OpenApiTagSchema = new Schema({
}
});
try {
OpenApiTagSchema.index({ teamId: 1, tmbId: 1, normalizedName: 1 }, { unique: true });
OpenApiTagSchema.index({ teamId: 1, tmbId: 1, type: 1, order: 1 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build OpenAPI tag indexes', { error });
}
defineIndex(OpenApiTagSchema, {
key: { teamId: 1, tmbId: 1, normalizedName: 1 },
options: { unique: true }
});
defineIndex(OpenApiTagSchema, {
key: { teamId: 1, tmbId: 1, type: 1, order: 1 }
});
export const MongoOpenApiTag = getMongoModel<OpenApiTagSchemaType>(
OpenApiTagCollectionName,
......
packages/service/support/outLink/schema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo;
import { type OutLinkSchemaType } from '@fastgpt/global/support/outLink/type';
import {
......@@ -111,12 +111,12 @@ OutLinkSchema.virtual('associatedApp', {
const logger = getLogger(LogCategories.INFRA.MONGO);
OutLinkSchema.index({ shareId: -1 });
OutLinkSchema.index({ teamId: 1, tmbId: 1, appId: 1 });
defineIndex(OutLinkSchema, { key: { shareId: -1 } });
defineIndex(OutLinkSchema, { key: { teamId: 1, tmbId: 1, appId: 1 } });
// Wechat polling recovery: find online channels on startup
OutLinkSchema.index(
{ type: 1, 'app.status': 1 },
{ partialFilterExpression: { type: 'wechat', 'app.status': 'online' } }
);
defineIndex(OutLinkSchema, {
key: { type: 1, 'app.status': 1 },
options: { partialFilterExpression: { type: 'wechat', 'app.status': 'online' } }
});
export const MongoOutLink = getMongoModel<OutLinkSchemaType>('outlinks', OutLinkSchema);
packages/service/support/permission/memberGroup/groupMemberSchema.ts
import { TeamMemberCollectionName } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { MemberGroupCollectionName } from './memberGroupSchema';
import { type GroupMemberSchemaType } from '@fastgpt/global/support/permission/memberGroup/type';
import { GroupMemberRole } from '@fastgpt/global/support/permission/memberGroup/constant';
......@@ -36,17 +36,17 @@ GroupMemberSchema.virtual('group', {
const logger = getLogger(LogCategories.INFRA.MONGO);
try {
GroupMemberSchema.index({
defineIndex(GroupMemberSchema, {
key: {
groupId: 1
});
}
});
GroupMemberSchema.index({
defineIndex(GroupMemberSchema, {
key: {
tmbId: 1
});
} catch (error) {
logger.error('Failed to build group member indexes', { error });
}
}
});
export const MongoGroupMemberModel = getMongoModel<GroupMemberSchemaType>(
GroupMemberCollectionName,
......
packages/service/support/permission/memberGroup/memberGroupSchema.ts
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { type MemberGroupSchemaType } from '@fastgpt/global/support/permission/memberGroup/type';
import { getLogger, LogCategories } from '../../../common/logger';
const { Schema } = connectionMongo;
......@@ -35,19 +35,15 @@ export const MemberGroupSchema = new Schema(
const logger = getLogger(LogCategories.INFRA.MONGO);
try {
MemberGroupSchema.index(
{
defineIndex(MemberGroupSchema, {
key: {
teamId: 1,
name: 1
},
{
options: {
unique: true
}
);
} catch (error) {
logger.error('Failed to build member group indexes', { error });
}
});
export const MongoMemberGroupModel = getMongoModel<MemberGroupSchemaType>(
MemberGroupCollectionName,
......
packages/service/support/permission/org/orgMemberSchema.ts
import { OrgCollectionName } from '@fastgpt/global/support/user/team/org/constant';
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import {
TeamCollectionName,
TeamMemberCollectionName
......@@ -43,24 +43,22 @@ OrgMemberSchema.virtual('org', {
const logger = getLogger(LogCategories.INFRA.MONGO);
try {
OrgMemberSchema.index(
{
defineIndex(OrgMemberSchema, {
key: {
teamId: 1,
orgId: 1,
tmbId: 1
},
{
options: {
unique: true
}
);
OrgMemberSchema.index({
});
defineIndex(OrgMemberSchema, {
key: {
teamId: 1,
tmbId: 1
});
} catch (error) {
logger.error('Failed to build org member indexes', { error });
}
}
});
export const MongoOrgMemberModel = getMongoModel<OrgMemberSchemaType>(
OrgMemberCollectionName,
......
packages/service/support/permission/org/orgSchema.ts
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { OrgCollectionName } from '@fastgpt/global/support/user/team/org/constant';
import type { OrgSchemaType } from '@fastgpt/global/support/user/team/org/type';
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
import { OrgMemberCollectionName } from './orgMemberSchema';
import { getNanoid } from '@fastgpt/global/common/string/tools';
import { DEFAULT_ORG_AVATAR } from '@fastgpt/global/common/system/constants';
......@@ -59,22 +59,20 @@ OrgSchema.virtual('members', {
const logger = getLogger(LogCategories.INFRA.MONGO);
try {
OrgSchema.index({
defineIndex(OrgSchema, {
key: {
teamId: 1,
path: 1
});
OrgSchema.index(
{
}
});
defineIndex(OrgSchema, {
key: {
teamId: 1,
pathId: 1
},
{
options: {
unique: true
}
);
} catch (error) {
logger.error('Failed to build org indexes', { error });
}
});
export const MongoOrgModel = getMongoModel<OrgSchemaType>(OrgCollectionName, OrgSchema);
packages/service/support/permission/schema.ts
......@@ -2,8 +2,7 @@ import {
TeamCollectionName,
TeamMemberCollectionName
} from '@fastgpt/global/support/user/team/constant';
import { connectionMongo, getMongoModel } from '../../common/mongo';
import { getLogger, LogCategories } from '../../common/logger';
import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
import type { ResourcePermissionType } from '@fastgpt/global/support/permission/type';
import { PerResourceTypeEnum } from '@fastgpt/global/support/permission/constant';
import { MemberGroupCollectionName } from './memberGroup/memberGroupSchema';
......@@ -78,21 +77,22 @@ ResourcePermissionSchema.virtual('org', {
justOne: true
});
try {
ResourcePermissionSchema.index({
defineIndex(ResourcePermissionSchema, {
key: {
resourceType: 1,
teamId: 1
});
}
});
// Indexes for resourceId-based resources
ResourcePermissionSchema.index(
{
// Indexes for resourceId-based resources
defineIndex(ResourcePermissionSchema, {
key: {
resourceType: 1,
teamId: 1,
resourceId: 1,
groupId: 1
},
{
options: {
unique: true,
partialFilterExpression: {
groupId: {
......@@ -103,16 +103,16 @@ try {
}
}
}
);
});
ResourcePermissionSchema.index(
{
defineIndex(ResourcePermissionSchema, {
key: {
resourceType: 1,
teamId: 1,
resourceId: 1,
orgId: 1
},
{
options: {
unique: true,
partialFilterExpression: {
orgId: {
......@@ -123,16 +123,16 @@ try {
}
}
}
);
});
ResourcePermissionSchema.index(
{
defineIndex(ResourcePermissionSchema, {
key: {
resourceType: 1,
teamId: 1,
resourceId: 1,
tmbId: 1
},
{
options: {
unique: true,
partialFilterExpression: {
tmbId: {
......@@ -143,33 +143,33 @@ try {
}
}
}
);
});
// General index for resourceId-based resources
ResourcePermissionSchema.index(
{
// General index for resourceId-based resources
defineIndex(ResourcePermissionSchema, {
key: {
resourceType: 1,
teamId: 1,
resourceId: 1
},
{
options: {
partialFilterExpression: {
resourceId: {
$exists: true
}
}
}
);
});
// Indexes for resourceName-based resources
ResourcePermissionSchema.index(
{
// Indexes for resourceName-based resources
defineIndex(ResourcePermissionSchema, {
key: {
resourceType: 1,
teamId: 1,
resourceName: 1,
groupId: 1
},
{
options: {
unique: true,
partialFilterExpression: {
groupId: {
......@@ -180,16 +180,16 @@ try {
}
}
}
);
});
ResourcePermissionSchema.index(
{
defineIndex(ResourcePermissionSchema, {
key: {
resourceType: 1,
teamId: 1,
resourceName: 1,
orgId: 1
},
{
options: {
unique: true,
partialFilterExpression: {
orgId: {
......@@ -200,16 +200,16 @@ try {
}
}
}
);
});
ResourcePermissionSchema.index(
{
defineIndex(ResourcePermissionSchema, {
key: {
resourceType: 1,
teamId: 1,
resourceName: 1,
tmbId: 1
},
{
options: {
unique: true,
partialFilterExpression: {
tmbId: {
......@@ -220,27 +220,23 @@ try {
}
}
}
);
});
// General index for resourceName-based resources
ResourcePermissionSchema.index(
{
// General index for resourceName-based resources
defineIndex(ResourcePermissionSchema, {
key: {
resourceType: 1,
teamId: 1,
resourceName: 1
},
{
options: {
partialFilterExpression: {
resourceName: {
$exists: true
}
}
}
);
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build permission indexes', { error });
}
});
ResourcePermissionSchema.pre('save', function (next) {
if (!this.tmbId && !this.groupId && !this.orgId) {
......
packages/service/support/tmpData/schema.ts
import { getMongoModel, Schema } from '../../common/mongo';
import { defineIndex, getMongoModel, Schema } from '../../common/mongo';
import type { TmpDataSchema as SchemaType } from '@fastgpt/global/support/tmpData/type';
import { getLogger, LogCategories } from '../../common/logger';
const collectionName = 'tmp_datas';
const TmpDataSchema = new Schema({
dataId: {
type: String,
required: true,
unique: true
required: true
},
data: {
type: Object
......@@ -19,12 +17,14 @@ const TmpDataSchema = new Schema({
}
});
try {
TmpDataSchema.index({ dataId: -1 });
TmpDataSchema.index({ expireAt: -1 }, { expireAfterSeconds: 5 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build tmp data indexes', { error });
}
defineIndex(TmpDataSchema, {
key: { dataId: 1 },
options: { unique: true }
});
defineIndex(TmpDataSchema, { key: { dataId: -1 } });
defineIndex(TmpDataSchema, {
key: { expireAt: -1 },
options: { expireAfterSeconds: 5 }
});
export const MongoTmpData = getMongoModel<SchemaType<Object>>(collectionName, TmpDataSchema);
export const MongoTmpData = getMongoModel<SchemaType<object>>(collectionName, TmpDataSchema);
packages/service/support/user/audit/schema.ts
import { Schema, getMongoLogModel } from '../../../common/mongo';
import { defineIndex, Schema, getMongoLogModel } from '../../../common/mongo';
import { type TeamAuditSchemaType } from '@fastgpt/global/support/user/audit/type';
import { AdminAuditEventEnum, AuditEventEnum } from '@fastgpt/global/support/user/audit/constants';
import {
......@@ -34,8 +34,8 @@ const TeamAuditSchema = new Schema({
}
});
TeamAuditSchema.index({ teamId: 1, tmbId: 1, event: 1 });
TeamAuditSchema.index({ timestamp: 1, teamId: 1 });
defineIndex(TeamAuditSchema, { key: { teamId: 1, tmbId: 1, event: 1 } });
defineIndex(TeamAuditSchema, { key: { timestamp: 1, teamId: 1 } });
export const MongoTeamAudit = getMongoLogModel<TeamAuditSchemaType>(
TeamAuditCollectionName,
......
packages/service/support/user/auth/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import type { UserAuthSchemaType } from '@fastgpt/global/support/user/auth/type';
import { userAuthTypeMap } from '@fastgpt/global/support/user/auth/constants';
import { addMinutes } from 'date-fns';
import { getLogger, LogCategories } from '../../../common/logger';
const UserAuthSchema = new Schema({
key: {
......@@ -32,12 +31,10 @@ const UserAuthSchema = new Schema({
}
});
try {
UserAuthSchema.index({ key: 1, type: 1 });
UserAuthSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 0 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build user auth indexes', { error });
}
defineIndex(UserAuthSchema, { key: { key: 1, type: 1 } });
defineIndex(UserAuthSchema, {
key: { expiredTime: 1 },
options: { expireAfterSeconds: 0 }
});
export const MongoUserAuth = getMongoModel<UserAuthSchemaType>('auth_codes', UserAuthSchema);
packages/service/support/user/schema.ts
import { connectionMongo, getMongoModel } from '../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../common/mongo';
const { Schema } = connectionMongo;
import { hashStr } from '@fastgpt/global/common/string/tools';
import { UserTagsSchema, type UserModelSchema } from '@fastgpt/global/support/user/type';
import { UserStatusEnum, userStatusMap } from '@fastgpt/global/support/user/constant';
import { TeamMemberCollectionName } from '@fastgpt/global/support/user/team/constant';
import { LangEnum } from '@fastgpt/global/common/i18n/type';
import { getLogger, LogCategories } from '../../common/logger';
export const userCollectionName = 'users';
......@@ -18,8 +17,7 @@ const UserSchema = new Schema({
username: {
// 可以是手机/邮箱，新的验证都只用手机
type: String,
required: true,
unique: true // 唯一
required: true
},
password: {
type: String,
......@@ -75,12 +73,12 @@ const UserSchema = new Schema({
avatar: String
});
try {
// Admin charts
UserSchema.index({ createTime: -1 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build user indexes', { error });
}
// username 唯一。
defineIndex(UserSchema, {
key: { username: 1 },
options: { unique: true }
});
// Admin charts
defineIndex(UserSchema, { key: { createTime: -1 } });
export const MongoUser = getMongoModel<UserModelSchema>(userCollectionName, UserSchema);
packages/service/support/user/team/teamMemberSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import { type TeamMemberSchema as TeamMemberType } from '@fastgpt/global/support/user/team/type';
import { userCollectionName } from '../../user/schema';
......@@ -8,7 +8,6 @@ import {
TeamCollectionName
} from '@fastgpt/global/support/user/team/constant';
import { getRandomUserAvatar } from '@fastgpt/global/support/user/utils';
import { getLogger, LogCategories } from '../../../common/logger';
const TeamMemberSchema = new Schema({
teamId: {
......@@ -67,13 +66,14 @@ TeamMemberSchema.virtual('user', {
justOne: true
});
try {
TeamMemberSchema.index({ teamId: 1 }, { background: true });
TeamMemberSchema.index({ userId: 1 }, { background: true });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build team member indexes', { error });
}
defineIndex(TeamMemberSchema, {
key: { teamId: 1 },
options: { background: true }
});
defineIndex(TeamMemberSchema, {
key: { userId: 1 },
options: { background: true }
});
export const MongoTeamMember = getMongoModel<TeamMemberType>(
TeamMemberCollectionName,
......
packages/service/support/user/team/teamSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import { type TeamSchema as TeamType } from '@fastgpt/global/support/user/team/type';
import { userCollectionName } from '../../user/schema';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { getLogger, LogCategories } from '../../../common/logger';
const TeamSchema = new Schema({
name: {
......@@ -53,13 +52,11 @@ const TeamSchema = new Schema({
}
});
try {
TeamSchema.index({ name: 1 });
TeamSchema.index({ ownerId: 1 });
TeamSchema.index({ 'meta.wecom.corpId': 1 }, { sparse: true, unique: true });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build team indexes', { error });
}
defineIndex(TeamSchema, { key: { name: 1 } });
defineIndex(TeamSchema, { key: { ownerId: 1 } });
defineIndex(TeamSchema, {
key: { 'meta.wecom.corpId': 1 },
options: { sparse: true, unique: true }
});
export const MongoTeam = getMongoModel<TeamType>(TeamCollectionName, TeamSchema);
packages/service/support/wallet/coupon/schema.ts
import { addDays } from 'date-fns';
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import type { TeamCouponSchemaType } from '@fastgpt/global/support/wallet/sub/coupon/type';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { CouponTypeEnum } from '@fastgpt/global/support/wallet/sub/coupon/constants';
import { getLogger, LogCategories } from '../../../common/logger';
export const couponCollectionName = 'team_sub_coupons';
......@@ -43,12 +42,7 @@ const CouponSchema = new Schema({
}
});
try {
CouponSchema.index({ key: 1 }, { unique: true });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build coupon indexes', { error });
}
defineIndex(CouponSchema, { key: { key: 1 }, options: { unique: true } });
export const MongoTeamCoupon = getMongoModel<TeamCouponSchemaType>(
couponCollectionName,
......
packages/service/support/wallet/discountCoupon/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { DiscountCouponTypeEnum } from '@fastgpt/global/support/wallet/sub/discountCoupon/constants';
import type { DiscountCouponSchemaType } from '@fastgpt/global/openapi/support/wallet/discountCoupon/api';
import { getLogger, LogCategories } from '../../../common/logger';
export const discountCouponCollectionName = 'team_discount_coupons';
......@@ -30,13 +29,8 @@ const DiscountCouponSchema = new Schema({
}
});
try {
DiscountCouponSchema.index({ status: 1, type: 1 });
DiscountCouponSchema.index({ teamId: 1, status: 1 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build discount coupon indexes', { error });
}
defineIndex(DiscountCouponSchema, { key: { status: 1, type: 1 } });
defineIndex(DiscountCouponSchema, { key: { teamId: 1, status: 1 } });
export const MongoDiscountCoupon = getMongoModel<DiscountCouponSchemaType>(
discountCouponCollectionName,
......
packages/service/support/wallet/sub/schema.ts
......@@ -3,7 +3,7 @@
1. type=standard: There will only be 1, and each team will have one
2. type=extraDatasetSize/extraPoints: Can buy multiple
*/
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import {
......@@ -12,7 +12,6 @@ import {
SubTypeEnum
} from '@fastgpt/global/support/wallet/sub/constants';
import type { TeamSubSchemaType } from '@fastgpt/global/support/wallet/sub/type';
import { getLogger, LogCategories } from '../../../common/logger';
export const subCollectionName = 'team_subscriptions';
......@@ -81,30 +80,27 @@ const SubSchema = new Schema({
currentExtraDatasetSize: Number
});
try {
// Get plan by expiredTime
SubSchema.index({ expiredTime: -1, currentSubLevel: 1 });
// Get plan by expiredTime
defineIndex(SubSchema, { key: { expiredTime: -1, currentSubLevel: 1 } });
// Get team plan
SubSchema.index({ teamId: 1, type: 1, expiredTime: -1 });
// timer task. Get standard plan;Get free plan;Clear expired extract plan
SubSchema.index({ type: 1, expiredTime: -1, currentSubLevel: 1 });
// Get team plan
defineIndex(SubSchema, { key: { teamId: 1, type: 1, expiredTime: -1 } });
// timer task. Get standard plan;Get free plan;Clear expired extract plan
defineIndex(SubSchema, {
key: { type: 1, expiredTime: -1, currentSubLevel: 1 }
});
// 修改后的唯一索引
SubSchema.index(
{
// 修改后的唯一索引
defineIndex(SubSchema, {
key: {
teamId: 1,
type: 1,
currentSubLevel: 1
},
{
options: {
unique: true,
partialFilterExpression: { type: SubTypeEnum.standard }
}
);
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build subscription indexes', { error });
}
});
export const MongoTeamSub = getMongoModel<TeamSubSchemaType>(subCollectionName, SubSchema);
packages/service/support/wallet/usage/schema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import { type UsageSchemaType } from '@fastgpt/global/support/wallet/usage/type';
import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants';
......@@ -9,7 +9,6 @@ import {
import { UsageCollectionName, UsageItemCollectionName } from './constants';
import { AppCollectionName } from '../../../core/app/schema';
import { DatasetCollectionName } from '../../../core/dataset/schema';
import { getLogger, LogCategories } from '../../../common/logger';
import { agentSkillsCollectionName } from '@fastgpt/global/core/ai/skill/constants';
const UsageSchema = new Schema(
......@@ -76,13 +75,13 @@ UsageSchema.virtual('usageItems', {
foreignField: 'usageId'
});
try {
UsageSchema.index({ teamId: 1, tmbId: 1, source: 1, time: 1, appName: 1, _id: -1 });
defineIndex(UsageSchema, {
key: { teamId: 1, tmbId: 1, source: 1, time: 1, appName: 1, _id: -1 }
});
UsageSchema.index({ time: 1 }, { expireAfterSeconds: 360 * 24 * 60 * 60 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build usage indexes', { error });
}
defineIndex(UsageSchema, {
key: { time: 1 },
options: { expireAfterSeconds: 360 * 24 * 60 * 60 }
});
export const MongoUsage = getMongoModel<UsageSchemaType>(UsageCollectionName, UsageSchema);
packages/service/support/wallet/usage/usageItemSchema.ts
import { connectionMongo, getMongoModel } from '../../../common/mongo';
import { defineIndex, connectionMongo, getMongoModel } from '../../../common/mongo';
const { Schema } = connectionMongo;
import type { UsageItemSchemaType } from '@fastgpt/global/support/wallet/usage/type';
import { UsageCollectionName, UsageItemCollectionName } from './constants';
import { TeamCollectionName } from '@fastgpt/global/support/user/team/constant';
import { getLogger, LogCategories } from '../../../common/logger';
const UsageItemSchema = new Schema({
teamId: {
......@@ -41,13 +40,11 @@ const UsageItemSchema = new Schema({
model: String
});
try {
UsageItemSchema.index({ usageId: 'hashed' });
UsageItemSchema.index({ time: 1 }, { expireAfterSeconds: 360 * 24 * 60 * 60 });
} catch (error) {
const logger = getLogger(LogCategories.INFRA.MONGO);
logger.error('Failed to build usage item indexes', { error });
}
defineIndex(UsageItemSchema, { key: { usageId: 'hashed' } });
defineIndex(UsageItemSchema, {
key: { time: 1 },
options: { expireAfterSeconds: 360 * 24 * 60 * 60 }
});
export const MongoUsageItem = getMongoModel<UsageItemSchemaType>(
UsageItemCollectionName,
......
packages/service/test/common/mongo/schemaIndexes.test.ts 0 → 100644
import { describe, expect, it } from 'vitest';
import { Schema } from '@fastgpt/service/common/mongo';
import { defineIndex, getDeprecatedIndexes } from '@fastgpt/service/common/mongo/schemaIndexes';
describe('defineIndex', () => {
it('returns an empty list for a Schema without declarations', () => {
expect(getDeprecatedIndexes(new Schema())).toEqual([]);
});
it('defaults to an active index and delegates it to the Mongoose Schema', () => {
const schema = new Schema();
defineIndex(schema, {
key: { current: 1 },
options: { unique: true, name: 'current_unique' }
});
expect(schema.indexes()).toEqual([
[{ current: 1 }, { unique: true, name: 'current_unique', background: true }]
]);
expect(getDeprecatedIndexes(schema)).toEqual([]);
});
it('stores deprecated indexes without adding them to the Mongoose Schema', () => {
const schema = new Schema();
defineIndex(schema, {
key: { legacyA: 1 },
deprecated: true
});
defineIndex(schema, {
key: { teamId: 1, parentId: 1, deleteTime: 1 },
options: {},
deprecated: true
});
defineIndex(schema, {
key: { legacyB: -1 },
options: { name: 'custom_legacy_name', unique: true, background: true },
deprecated: true
});
expect(schema.indexes()).toEqual([]);
expect(getDeprecatedIndexes(schema)).toEqual([
{
indexName: 'legacyA_1',
key: { legacyA: 1 },
options: undefined
},
{
indexName: 'teamId_1_parentId_1_deleteTime_1',
key: { teamId: 1, parentId: 1, deleteTime: 1 },
options: undefined
},
{
indexName: 'custom_legacy_name',
key: { legacyB: -1 },
options: {
unique: true,
sparse: undefined,
expireAfterSeconds: undefined,
partialFilterExpression: undefined,
collation: undefined
}
}
]);
});
it('rejects duplicate index names within one Schema', () => {
const schema = new Schema();
const definition = { key: { legacy: 1 }, deprecated: true } as const;
defineIndex(schema, definition);
expect(() => defineIndex(schema, definition)).toThrow(
'Duplicate deprecated MongoDB index declaration: legacy_1'
);
});
});
packages/service/test/core/ai/sandbox/infrastructure/instance/schema.test.ts
import { describe, expect, it } from 'vitest';
import { MongoSandboxInstance } from '@fastgpt/service/core/ai/sandbox/infrastructure/instance/schema';
import { getSchemaDeprecatedMongoIndexes } from '@fastgpt/service/common/mongo';
describe('MongoSandboxInstance schema indexes', () => {
it('declares provider sandbox uniqueness for remote resource records', () => {
......@@ -21,6 +22,10 @@ describe('MongoSandboxInstance schema indexes', () => {
expect(legacyIndex).toBeUndefined();
});
it('does not register historical sandbox indexes for automatic cleanup', () => {
expect(getSchemaDeprecatedMongoIndexes(MongoSandboxInstance.schema)).toEqual([]);
});
it('declares source lookup index for migrated sandbox instances', () => {
const indexes = MongoSandboxInstance.schema.indexes();
const sourceChatIndex = indexes.find(
......
packages/service/test/core/chat/schema.test.ts
......@@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest';
import { MongoChat } from '@fastgpt/service/core/chat/chatSchema';
import { MongoChatItem } from '@fastgpt/service/core/chat/chatItemSchema';
import { MongoChatItemResponse } from '@fastgpt/service/core/chat/chatItemResponseSchema';
import { getSchemaDeprecatedMongoIndexes } from '@fastgpt/service/common/mongo';
const hasIndex = (
indexes: ReturnType<typeof MongoChat.schema.indexes>,
......@@ -37,6 +38,10 @@ describe('chat schema indexes', () => {
expect(sourceAwareIndex?.[1]?.unique).toBe(true);
});
it('does not register historical chat indexes for automatic cleanup', () => {
expect(getSchemaDeprecatedMongoIndexes(MongoChat.schema)).toEqual([]);
});
it('keeps legacy chat item read and pagination indexes', () => {
const indexes = MongoChatItem.schema.indexes();
......
packages/service/test/env.test.ts
......@@ -9,6 +9,7 @@ const originalEnv = {
FILE_TOKEN_KEY: process.env.FILE_TOKEN_KEY,
FILE_DOWNLOAD_PUBLIC_URL_PREFIX: process.env.FILE_DOWNLOAD_PUBLIC_URL_PREFIX,
STORAGE_DOWNLOAD_URL_MODE: process.env.STORAGE_DOWNLOAD_URL_MODE,
SYNC_INDEX: process.env.SYNC_INDEX,
AES256_SECRET_KEY: process.env.AES256_SECRET_KEY,
INVOKE_TOKEN_SECRET: process.env.INVOKE_TOKEN_SECRET,
PRO_URL: process.env.PRO_URL,
......@@ -38,6 +39,7 @@ describe('serviceEnv', () => {
vi.stubEnv('FILE_TOKEN_KEY', originalEnv.FILE_TOKEN_KEY);
vi.stubEnv('FILE_DOWNLOAD_PUBLIC_URL_PREFIX', originalEnv.FILE_DOWNLOAD_PUBLIC_URL_PREFIX);
vi.stubEnv('STORAGE_DOWNLOAD_URL_MODE', originalEnv.STORAGE_DOWNLOAD_URL_MODE);
vi.stubEnv('SYNC_INDEX', originalEnv.SYNC_INDEX);
vi.stubEnv('AES256_SECRET_KEY', originalEnv.AES256_SECRET_KEY);
vi.stubEnv('INVOKE_TOKEN_SECRET', originalEnv.INVOKE_TOKEN_SECRET);
vi.stubEnv('PRO_URL', originalEnv.PRO_URL);
......@@ -53,6 +55,22 @@ describe('serviceEnv', () => {
vi.stubEnv('AGENT_SANDBOX_OPENSANDBOX_API_KEY', originalEnv.AGENT_SANDBOX_OPENSANDBOX_API_KEY);
});
it('enables MongoDB index synchronization by default and supports disabling it', async () => {
vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey');
vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret');
vi.stubEnv('INVOKE_TOKEN_SECRET', validInvokeTokenSecret);
vi.stubEnv('SYNC_INDEX', undefined);
await expect(importServiceEnv()).resolves.toMatchObject({
serviceEnv: { SYNC_INDEX: true }
});
vi.stubEnv('SYNC_INDEX', 'false');
await expect(importServiceEnv()).resolves.toMatchObject({
serviceEnv: { SYNC_INDEX: false }
});
});
it('validates SYSTEM_MAX_STRING_LENGTH_M during service env init', async () => {
vi.stubEnv('FILE_TOKEN_KEY', 'filetokenkey');
vi.stubEnv('AES256_SECRET_KEY', 'fastgptsecret');
......
projects/app/test/pages/api/admin/dataClean/cleanupDuplicateChats.test.ts
......@@ -37,7 +37,7 @@ const ensureLegacyDuplicateWritableCollection = async () => {
};
const restoreChatSchemaIndexes = async () => {
await MongoChat.syncIndexes();
await MongoChat.createIndexes();
};
const createChatHeader = ({
......
projects/marketplace/src/service/mongo/index.ts
......@@ -2,12 +2,12 @@ import type { Model, Schema } from 'mongoose';
import { Mongoose } from 'mongoose';
import { getLogger, LogCategories } from '../logger';
import { marketplaceEnv } from '../../env';
import { MongoIndexManager } from '@fastgpt/service/common/mongo/indexManager';
export const MONGO_URL = marketplaceEnv.MONGODB_URI;
const maxConnecting = Math.max(30, marketplaceEnv.DB_MAX_LINK);
const logger = getLogger(LogCategories.INFRA.MONGO);
const syncIndex = marketplaceEnv.SYNC_INDEX;
declare global {
var mongodb: Mongoose | undefined;
......@@ -32,16 +32,26 @@ export const getMongoModel = <T extends Schema>(name: string, schema: T) => {
};
const syncMongoIndex = async (model: Model<any>) => {
if (syncIndex && process.env.NODE_ENV !== 'test') {
try {
model.syncIndexes({ background: true });
} catch (error: any) {
logger.error('Create index error', { error });
if (process.env.NODE_ENV === 'test' || !marketplaceEnv.SYNC_INDEX || !MONGO_URL) {
return;
}
try {
await MongoIndexManager.syncModelIndexes({
model,
logger
});
} catch (error) {
logger.error('Failed to ensure MongoDB indexes', {
modelName: model.modelName,
collectionName: model.collection.collectionName,
error
});
}
};
export const ReadPreference = connectionMongo.mongo.ReadPreference;
export { defineIndex } from '@fastgpt/service/common/mongo/schemaIndexes';
export async function connectMongo(db: Mongoose, url: string): Promise<Mongoose> {
if (db.connection.readyState !== 0) {
......@@ -81,6 +91,7 @@ export async function connectMongo(db: Mongoose, url: string): Promise<Mongoose>
serverSelectionTimeoutMS: 10000, // 服务器选择超时: 10秒,防止副本集故障时长时间阻塞
heartbeatFrequencyMS: 5000 // 5s 进行一次健康检查
});
return db;
} catch (error) {
logger.error('Mongo connect error', { error });
......
projects/marketplace/src/service/mongo/models/download.ts
import { Schema } from 'mongoose';
import z from 'zod';
import { getMongoModel } from '..';
import { defineIndex, getMongoModel } from '..';
export const pluginTypeEnum = z.enum(['tool']);
......@@ -21,6 +21,9 @@ const downloadCountSchema = new Schema({
});
// 复合索引：type + toolId + time
downloadCountSchema.index({ type: 1, toolId: 1, time: 1 }, { unique: true });
defineIndex(downloadCountSchema, {
key: { type: 1, toolId: 1, time: 1 },
options: { unique: true }
});
export const MongoDownloadCount = getMongoModel('plugin_download_counts', downloadCountSchema);
projects/marketplace/src/service/mongo/models/tool.ts
import { Schema } from 'mongoose';
import z from 'zod';
import { getMongoModel } from '..';
import { defineIndex, getMongoModel } from '..';
import {
MarketplaceOfficialSource,
MarketplacePkgSourceSchema
......@@ -47,8 +47,13 @@ const marketplaceToolSchema = new Schema(
}
);
marketplaceToolSchema.index({ pluginId: 1, version: 1 }, { unique: true });
marketplaceToolSchema.index({ pluginId: 1, updateTime: -1 });
marketplaceToolSchema.index({ source: 1, pluginId: 1, updateTime: -1 });
defineIndex(marketplaceToolSchema, {
key: { pluginId: 1, version: 1 },
options: { unique: true }
});
defineIndex(marketplaceToolSchema, { key: { pluginId: 1, updateTime: -1 } });
defineIndex(marketplaceToolSchema, {
key: { source: 1, pluginId: 1, updateTime: -1 }
});
export const MongoMarketplaceTool = getMongoModel('marketplace_tools', marketplaceToolSchema);
projects/marketplace/test/env.test.ts
import { afterEach, describe, expect, it, vi } from 'vitest';
const originalSyncIndex = process.env.SYNC_INDEX;
const originalCommunityAuthToken = process.env.COMMUNITY_AUTH_TOKEN;
const originalSyncIndex = process.env.SYNC_INDEX;
const importEnv = async () => {
vi.resetModules();
......@@ -10,32 +10,20 @@ const importEnv = async () => {
describe('marketplace env', () => {
afterEach(() => {
vi.stubEnv('SYNC_INDEX', originalSyncIndex);
vi.stubEnv('COMMUNITY_AUTH_TOKEN', originalCommunityAuthToken);
vi.stubEnv('SYNC_INDEX', originalSyncIndex);
});
it('defaults SYNC_INDEX to true when it is not configured', async () => {
it('enables MongoDB index synchronization by default and supports disabling it', async () => {
vi.stubEnv('SYNC_INDEX', undefined);
const { marketplaceEnv } = await importEnv();
expect(marketplaceEnv.SYNC_INDEX).toBe(true);
});
it('defaults SYNC_INDEX to true when it is empty', async () => {
vi.stubEnv('SYNC_INDEX', '');
const { marketplaceEnv } = await importEnv();
expect(marketplaceEnv.SYNC_INDEX).toBe(true);
await expect(importEnv()).resolves.toMatchObject({
marketplaceEnv: { SYNC_INDEX: true }
});
it('parses explicit false-like SYNC_INDEX values', async () => {
vi.stubEnv('SYNC_INDEX', 'false');
const { marketplaceEnv } = await importEnv();
expect(marketplaceEnv.SYNC_INDEX).toBe(false);
await expect(importEnv()).resolves.toMatchObject({
marketplaceEnv: { SYNC_INDEX: false }
});
});
it('parses optional community auth token', async () => {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment