phsl / new-api

Record successful and failed API token operations with safe target metadata. Capture quota adjustments in a transaction, synchronize committed cache differences, and correlate audit and top-up records.

Show operation targets, changes, quota balances, and failure details consistently across audit and usage logs, with translations for all seven locales.

Validated controller, middleware, and model tests; 78 frontend tests; typecheck and lint; real SQLite 3.50.4, MySQL 8.4.11, and PostgreSQL 16.15 with shared and separate log databases.

Require scoped, single-use verification for account bindings and password
operations. Bind OAuth authorization and email confirmations to the initiating
session; preserve the last usable login method and audit operation outcomes.

Apply Unicode-aware password length limits, Argon2id writes with bcrypt
compatibility, and long-password encryption.
Return has_password with the existing profile SELECT without extra queries.
Reuse the existing security dialogs and add all seven locale translations.

Validation:
- Go: go test ./common ./model ./service ./middleware ./controller ./router -count=1
- DB: SQLite 3.50.4, MySQL 8.4.11, PostgreSQL 16.15; separate main/log databases
- MySQL/PostgreSQL: TEST_SECURITY_DIALECT=<dialect> with TEST_<DIALECT>_DSN,
  go test ./controller -run '^(TestSecurityAccount|TestSecurityEnrollment|TestGenerateOAuthCode|TestOAuthBind|TestTelegramOAuth)' -count=1 -v
- Web: relevant Vitest suites, bun run typecheck, targeted oxlint/format,
  bun run i18n:sync, and bun run build

Roll out dual-format readers to every instance with
ACCOUNT_PASSWORD_HASH_ALGORITHM=bcrypt before enabling Argon2id writes
and the new UI. Rollbacks must retain Argon2id and v2 envelope readers.

Relevant controls: ASVS 5.0.0 6.2.1-6.2.3, 6.2.5-6.2.9, 6.3.7, 7.4.3, 7.5.1;
this change does not assert application-wide ASVS certification.

Use authorization code flow with PKCE and verified ID tokens for Telegram login, binding, and security verification. Preserve existing bindings and require administrator OAuth configuration.

Keep the restricted WeChat first-enrollment session proof, fix missing-target authentication errors, and preserve callback requests after OAuth popups close.

Require single-use operation proofs for passkey enrollment, two-factor setup, and channel key access. Add password and OAuth verification flows, enforce session-bound enrollment, and redact OAuth callback secrets from logs.

Validation: affected Go packages pass; frontend typecheck, changed-file lint, and 111 tests pass. Security enrollment regressions pass on SQLite 3.50.4, MySQL 8.0.46, and PostgreSQL 16.15. Full frontend lint has pre-existing errors outside the changed files.

Upgrade the SQLite GORM driver to correctly distinguish unique indexes from constraints. Normalize equivalent MySQL decimal defaults and PostgreSQL CHAR metadata during migration comparison while preserving real schema changes.

Validation: 27 fresh-start and rc.26 upgrade scenarios using SQLite 3.50.4, MySQL 5.7.44, PostgreSQL 9.6.24, and ClickHouse 25.8.33.6; 15 upgraded databases passed uniqueness checks. Relational databases issue no DDL on unchanged restarts. Existing ClickHouse TTL synchronization remains unchanged.

Move account security settings into a dedicated page and add token status, rotation, revocation, and access history.

Store audit events with role snapshots and JSON metadata, add audit.read authorization and an independent audit page, and upgrade the ClickHouse driver to v2.46.0.

* fix(relay): treat gpt-5 and later generations alike for max_completion_tokens

IsOpenAIGPT5Model matched on the literal prefix "gpt-5", so gpt-6-astra
(and every generation after it) fell through the gpt-5 request rules:
max_tokens was forwarded as-is and the provider rejected it with
"Unsupported parameter: 'max_tokens' is not supported with this model.
Use 'max_completion_tokens' instead." The same gap left temperature,
top_p and logprobs untouched, each of which the provider also rejects,
and made the channel test button report a 400 for a healthy deployment.

Match on the major version instead (gpt-<n>... with n >= 5). Callers are
unchanged: ConvertOpenAIRequest, GetSystemRoleName, buildTestRequest and
the health check all go through this one helper. buildTestRequest now
sends max_completion_tokens for these models directly instead of relying
on the later conversion. gpt-4.1, gpt-4o, gpt-oss, gpt-image and
gpt-realtime names still do not match.

Verified against Azure OpenAI gpt-6-astra (2026-09-03): with the old
prefix max_tokens / temperature / top_p / logprobs each returned 400,
while gpt-5.6-luna with the same payload returned 200.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016bwz8o5UeoRtrtDusKaayp

* fix(relay): separate OpenAI chat model compatibility rules

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: CaIon <i@caion.me>

* perf(relay): bulk-copy Responses raw JSON fields

* perf(common): share RawMessage deep-copy optimization

---------

Co-authored-by: CaIon <i@caion.me>

Model-name post-processing is rebuilt around an explicit trailing
@key:value modifier syntax (thinking/effort/temperature/topp) that
overrides request fields, survives model mapping, and records
conversion diagnostics on the consume log.

- Legacy naked aliases (-thinking, -nothinking, -thinking-<budget>,
  effort tails) now parse only for positively matched families
  (gpt-*/o-series, claude-*, gemini-*, incl. vendor/ namespaces);
  names like qwen-max stay opaque. EffortTailModelIDs remains the
  escape hatch for real in-family IDs such as gpt-5.1-codex-max.
- Billing identity resolves once in ModelPriceHelper via a ladder:
  configured request name first (legacy wildcard entries intact), then
  canonical billing names rebuilt from parsed intent
  (base@effort:E@thinking:S, then base@thinking:S; order, duplicates,
  and budget values are irrelevant; temperature/topp never priced),
  then base. Routing and token limits fall back through
  RoutingMatchModelName; pricing lookups stay wildcard-only.
- Pass-through stays byte-identical: modifiers and aliases are neither
  parsed nor validated there and forward verbatim for the upstream
  (or a chained gateway) to interpret.
- Unknown modifier keys and invalid known-key values are rejected with
  400; models whose real names contain @tag:value are exempted via the
  thinking-suffix blacklist, which now supports re:-prefixed Go regex
  entries.
- Claude reasoning render coerces unsupported combinations (disable,
  adaptive, budgets) with warning diagnostics instead of erroring;
  native-protocol requests without host syntax pass through untouched.

BREAKING(openrouter): drop the host-invented "-thinking" model-name
alias (added in 4f6d16e3) that trimmed any *-thinking model on
OpenRouter channels and injected reasoning.enabled. It matched too
broadly and mangled real model IDs such as kimi-k2-thinking.
Migration: use some-model@thinking:on, or keep the old public name via
a channel model mapping {"some-model-thinking": "some-model@thinking:on"}.
Claude/Gemini family aliases (incl. anthropic/claude-*-thinking) keep
working via the family whitelist.

A model declared by a task plugin is served only by that plugin's channels.
When the claiming plugin has no enabled channel in the request group, the
distributor answered with the generic "no available channel" text, which hides
the actual cause and led operators to expect channel model_mapping on another
plugin's channel to take over (#7185). That expectation is not supported:
plugin declarations own model names statically, and channel availability must
not silently reassign ownership at request time. The supported fixes are the
existing operator tools, disabling the factory plugin per key or overriding it.

Both no-channel 503 sites in the distributor now route through
noAvailableChannelMessage. When the request is pinned to a task plugin, the
message names the claiming plugin and points to disabling or overriding it;
non-plugin requests keep the generic message. Added in en, zh-CN, zh-TW.

feat(task): give polling hooks a real query context, host HTTP classification, and bounded poll failures

Plugin polling hooks previously ran against a hollow context: parseTaskResult
and parseBatchResult received {} / nil, buildQueryRequest received a
{task_id, action} map under the misleading name requestBody, and batch hooks
saw only bare task ids. The per-task poller also never looked at the upstream
HTTP status, and every built-in plugin papered over unrecognized bodies with
`|| "IN_PROGRESS"`, so a 404, a revoked key, or a shape the plugin did not
know would sit in IN_PROGRESS for the full 24h TASK_TIMEOUT_MINUTES while
holding the user's pre-charged quota.

Contract (docs/plugin-api v1.d.ts, v1.md, v1.schema.json):
- TaskQueryContext is declared separately from DriverContext and rebuilt from
  the persisted Task row: taskId, publicTaskId, action, model, upstreamModel,
  baseUrl, apiKey, authHeader, auth, data, state. Query-side requestBody is
  removed; the original request is not persisted and hooks that need a
  request-derived value must save it into state at submit time.
- parseTaskResult / parseBatchResult receive a third {status, headers}
  argument. Batch hooks receive tasks[] with one TaskQueryContext per task.
- NormalizedTaskResult accepts status "UNKNOWN" meaning "I do not recognize
  this body". Falling back to IN_PROGRESS for unknown shapes is forbidden;
  `plugin lint` warns on the literal.
- parseSubmitResponse / parseTaskResult / parseBatchResult may return `state`.
  Task.Data remains a per-round snapshot overwritten on every valid parse;
  state is plugin-owned, persisted in TaskPrivateData.PluginState, preserved
  when a hook omits it, byte-capped like taskData, and never exposed through
  presenter views.

Host (service/task_polling.go, relay/channel/task/jsplugin/adaptor.go):
- TaskPollingAdaptor / BatchTaskPollingAdaptor take *model.Task and the
  *http.Response so the adaptor can build the full context; jsplugin is the
  only implementation.
- HTTP classification before the plugin sees the body: 2xx -> plugin;
  404/410 -> FAILURE and refund; 401/403 -> poll failure plus a channel-scoped
  warning, no auto-disable; 429/5xx/transport -> poll failure; other 4xx ->
  plugin with the status visible, counted as unrecognized if the plugin still
  reports a non-terminal state.
- TaskPrivateData.PollFailures counts consecutive poll failures (transient
  HTTP, auth, transport, hook error, UNKNOWN). It is persisted through the
  existing UpdateWithStatus CAS so a concurrent terminal transition on another
  instance is never clobbered, and reset on any valid 2xx non-terminal parse.
  Reaching TASK_POLL_MAX_FAILURES (default 20, <= 0 disables) fails the task
  with the last classification and HTTP code in fail_reason and runs the
  existing settle/refund chain exactly once. sweepTimedOutTasks and its
  1440-minute default are unchanged as the outer backstop.
- Unrecognized bodies are logged at WARN with a bounded redacted copy since
  Task.Data is intentionally not overwritten on that path.

Plugins (all ten bumped one patch version):
- jimeng persists the outbound req_key in state and reads it back in
  buildQueryRequest, replacing dead reads of ctx.data / ctx.requestBody that
  never resolved.
- sunoapi batch hooks read tasks[] instead of the removed requestBody.
- hailuo treats base_resp.status_code != 0 as FAILURE before the status table.
- kling, vidu, sora, alibaba, doubao, hailuo, jimeng return UNKNOWN with the
  raw upstream status in reason on table miss.
- google and vertex-ai treat a missing `done` as in-progress: Google
  long-running operations omit proto3 default fields, so a running Veo
  operation has no `done` key at all. Only a body without an operation name is
  UNKNOWN. plugins/veo_poll_test.go locks this so the poll-failure cutoff can
  never fail a rendering Veo task.

Tests cover the classification table end to end against a real DB (404
immediate refund, 429xN refund, 401 increments without status change, 2xx
reset, UNKNOWN increments, state preserved vs replaced, PollFailures survives
the CAS write), the query-context shape, UNKNOWN on unrecognized bodies, and
the absence of PluginState/PollFailures from TaskView. Controller tests derive
the kling factory version from the embedded manifest instead of hardcoding it.

- Drop the unreachable user-visibility branch in LogOther.toMap and the
  receiver-mutating normalizeLegacyAdminFields; JSONString/Snapshot/
  MarshalJSON now share one full serialization
- Define legacySensitiveLogOtherKeys once and reference it from both
  SetPublic rejection and the user read-side projection
- Return the original JSON for every role when formatLogOtherJSON
  removed nothing, avoiding a re-marshal on the user log list path
- Log rejected OtherRatios keys in taskBillingOther instead of dropping
  them silently
- Use Snapshot() with typed assertions in service tests

* fix: reduce public bootstrap requests and revalidate content

* fix(controller): use a weak ETag for revalidated public JSON

/api is gzip-compressed by middleware that runs after the handler returns,
and the validator is computed over the uncompressed body. The compressed and
identity forms of one payload therefore share a validator, which a strong ETag
must not do -- it asserts byte-for-byte equality across representations
(RFC 9110 8.8.1). Serve W/ instead.

Weak comparison ignores W/ on both operands, so etagMatches now strips it from
the served validator as well as from each candidate. Stripping only the
candidate would make a weak served validator match nothing and silently
disable every 304.

Vary: Accept-Encoding stays. Weakening the validator makes revalidation
correct, but it does not separate the two encodings in a shared cache.

* fix(test): align response cookie helper name

* fix(auth): revalidate stale route sessions

* Update web/src/features/about/api.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* test: remove newly added PR tests

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* feat(plugin): add MiniMax-H3 /v2 video generation to the hailuo task plugin

MiniMax-H3 speaks a different contract from the other Hailuo models, so the
hailuo task plugin now branches on the upstream model instead of adding a Go
adaptor:

- submit builds /v2/video_generation with a multimodal `content` array
  (text, first/last frame images, reference video/audio, or a full
  `metadata.content` passthrough), an explicit `ratio`, and 768P/2K
  resolutions; `metadata.callback_url` and `metadata.aigc_watermark` pass
  through
- query uses /v2/query/video_generation/{task_id} and parses the
  `{"task": {...}}` envelope, falling back to the /v1 shapes for every other
  model
- the /v2 result is a public CDN URL, so its artifact is proxied
  credentialless instead of through /v1/files/download
- request bounds (duration 4-15, resolution 768P/2K, ratio whitelist, at most
  2 frame images and 9/3/3 reference images/videos/audios) are enforced while
  the request body is built, which the host runs during validation, so an
  out-of-range duration is rejected with a 400 before it can become a billing
  multiplier
- duration and resolution are reported as usage facts only. Like the rest of
  this plugin, extractUsage returns no billing ratios, so per-call pricing is
  flat and 2K/duration pricing is expressed through the model's tiered billing
  expression over those facts.

Query hooks are driver hooks and are documented to receive `ctx.model` and
`ctx.upstreamModel`, but polling has no relay info and never populated them.
The polling and realtime-fetch call sites now carry the persisted task model
properties and the plugin adaptor maps them onto the query context, with
`upstreamModel` falling back to the origin name for tasks submitted without a
channel mapping.

* fix(plugin): validate Hailuo H3 requests and errors

Deferred follow-ups from the relaykit-tools review cycle, verified by
live end-to-end billing tests:

- billing: normalize Gemini modality keys consistently between stream
  merge and settlement (case/whitespace variants no longer drop
  independent audio/image pricing) and sum duplicate modality entries
  on both paths
- billing: sync legacy flat Claude cache-creation fields from the
  CacheCreation sub-object (including zeroing) and fall back to flat
  fields only when the snapshot never carried a sub-object, closing a
  stale 1h-cache overcharge path in cascaded deployments
- relay: move Chat-to-Claude and Chat-to-Gemini stream conversion state
  from gin.Context onto RelayInfo and reset it with SendResponseCount in
  InitChannelMeta, so channel retries start clean while per-request
  state (stream error collection, conversion diagnostics, channel
  chain, billing accumulators) survives
- relay: Claude channel now serves Gemini-format clients (request via
  registry conversion, response and stream composed through the Chat
  pivot), removing the last unimplemented conversion direction
- relaykit: recognize legacy pseudo tool names (googleSearch,
  codeExecution, urlContext) in the toolconv decode stage and drop the
  string-matching bypass in the Chat-to-Gemini converter; native Gemini
  tool output is restored and non-Gemini targets follow standard loss
  diagnostics
- relaykit: attach upstream Gemini usage (with billing_usage sidecar)
  to intermediate stream chunks so converted Claude streams report
  upstream truth from message_start, and preserve the sidecar through
  Claude stream usage merges; billing settlement unchanged
- billing: clamp negative Total-Prompt completion derivation, OR the
  Estimated flag across cross-dialect snapshot replacement, and fill
  canonical OpenAI prompt details via field-wise merge

feat(relay): hosted-tool conversion fidelity, reasoning normalization, and billing usage integrity (#7137)

* feat(relaykit): preserve hosted tools across conversions

- add protocol-neutral hosted-tool DTOs, conversion metadata, and loss policies
- bridge citations, grounding metadata, and hosted-tool stream lifecycles
- document the public conversion behavior and channel policy controls

* refactor(relaykit): normalize reasoning and thinking intent

- centralize provider-neutral reasoning intent, effort, and budget mappings
- parse model suffixes at the host entry boundary while preserving provider-owned tails
- keep adaptive Claude thinking and explicit zero-token compatibility consistent

* fix(billing): preserve authoritative usage across relay hops

- carry native BillingUsage sidecars through direct and streamed protocol bridges
- merge partial and terminal usage monotonically with safe fallback settlement
- retain cache metadata, penultimate usage, and per-call Gemini tool surcharges

* feat(relay): bridge Responses with Claude and Gemini protocols

- add direct request, response, and stream converters across supported relay formats
- expose Claude count_tokens and Chat-to-Responses compatibility endpoints
- carry conversion diagnostics through the host while retaining the curated public goldens

* fix(relay): wire relaykit conversions into host channels

- connect handlers, adaptors, and channel settings to the standalone conversion layer
- keep model mapping, pricing identity, retries, and provider-specific suffix behavior aligned
- ignore local audit artifacts and retain focused public regression coverage

* fix(log): preserve quota in usage statistics

This reverts commit 69a41eea.

fix(relay): bound the wait for upstream response headers (fixes unbounded heap growth → OOM) (#6949)

* fix(relay): bound the wait for upstream response headers (fixes unbounded heap growth)

The relay transport sets a dial timeout, a TLS handshake timeout and an expect-continue
timeout, but nothing bounds how long it waits for the upstream *response headers* after
the request has been written. An upstream that accepts the connection and then never
answers -- without sending FIN/RST, which is what happens when a NAT/firewall silently
drops the flow or the provider hangs -- parks the goroutine in
net/http.(*persistConn).roundTrip forever.

That goroutine keeps the whole request alive, which in practice means three copies of the
request body stay reachable for the lifetime of the process: the raw bytes from
io.ReadAll in CreateBodyStorageFromReader, the decoded messages held as json.RawMessage,
and the re-marshalled upstream body from common.Marshal. BodyStorageCleanup cannot help
here: it runs after c.Next() returns, and for these requests c.Next() never returns.

Measured on v1.0.0-rc.23 in production (see #6947 for the full evidence):

  - 23 goroutines stuck in persistConn.roundTrip on a single 40h-old instance,
    blocked between 353 and 1894 minutes (5.9h to 31.5h)
  - 96.9% of the live heap, sampled after a forced GC, attributable to those three
    body copies (HeapAlloc 892 MiB surviving three GC cycles; HeapObjects dropping
    30x while bytes dropped only 25%)
  - the live floor grows with uptime: 33.7 MiB at 0.1h, 89.2 at 13.8h, 510.0 at 40.1h,
    955.2 at 146.8h, OOMKilled at 172.9h -- same image, same config, same load

Doubling the memory limit and adding GOMEMLIMIT only moved the OOM from 132h to 172.9h.

RELAY_TIMEOUT (http.Client.Timeout) cannot be used for this: it covers the whole response
read and would cut legitimate long streaming calls, which is why it defaults to 0.
ResponseHeaderTimeout only bounds the wait for the headers; streaming after they arrive is
unaffected.

The default is deliberately generous. Non-streaming upstreams usually send the response
headers only once generation has finished, so the value has to leave room for a long
completion. 1800s is 12x shorter than the shortest hang observed here while leaving
several times the headroom a normal non-streaming request needs; 0 restores the previous
unbounded behaviour.

The assignment goes next to the other transport.* lines rather than inside the else
branch: newRelayHTTPTransport() normally takes the http.DefaultTransport.Clone() path,
and DefaultTransport does not set ResponseHeaderTimeout either.

This repo already sets ResponseHeaderTimeout on its other outbound transports
(controller/model_sync.go, controller/ratio_sync.go); the relay path appears to have
been missed.

Refs #6947. Likely also the root cause of #6731, which reported the same symptom
(production OOM on /v1/responses after ~64h) but was closed for template reasons.

* review: clamp overflowing timeout values and switch the test to testify

Addresses the two CodeRabbit findings on this PR.

Overflow (common/init.go:113): a RELAY_RESPONSE_HEADER_TIMEOUT beyond ~9.2e9 seconds
overflows time.Duration and can wrap into a *tiny positive* timeout, which would cut
every relay request instead of only the stuck ones. The value is now clamped before the
conversion, with regression tests for both the negative and the overflowing input.

I did not add fail-on-startup validation for negative values, for two reasons: the
existing `if seconds > 0` guard already treats them as "disabled", and the neighbouring
env-driven timeouts in this file are less strict still -- RelayIdleConnTimeout is
converted with no guard at all. Failing startup on a bad value would be a behaviour
change out of step with the rest of the file; happy to add it if you'd prefer that
direction repo-wide.

Test style: switched to testify (require.Equal / require.Zero / require.Positive), which
is what every other test under service/ uses.

go build, go vet and go test ./common/... ./service/... pass.
(`go build ./...` fails on the `web/dist` embed both with and without this change -- the
frontend bundle is not checked in.)

With PrepareStmt disabled, PostgreSQL queries run over pgx's simple
protocol, which encodes every []byte parameter as a bytea hex literal
('\x...'). driver.Valuer implementations returning []byte from
json.Marshal therefore fail json-column writes with SQLSTATE 22P02
(reported on the channels UPDATE path via ChannelInfo).

Reproduced against a live PostgreSQL 16: []byte Valuer into a json
column fails under simple protocol, string succeeds; []byte into a
text column silently stores the hex literal (no such path exists in
the repo today — audited all Valuers, json.RawMessage fields, and raw
SQL call sites).

- ChannelInfo, Properties, TaskPrivateData, JSONValue Value() now
  return string; zero-value nil semantics unchanged. Task.Data
  (bare json.RawMessage) is unaffected — database/sql's default
  converter already passes it as expected.
- Their Scan() counterparts now accept both []byte and string via a
  shared jsonScanBytes helper: SQLite returns string for these columns
  once Value() emits string, and the old []byte-only assertions
  silently zeroed the field (caught by the model test suite).
- Add regression tests locking both contracts: json-column Valuers
  must return string (or nil for zero values), Scanners must accept
  []byte and string.

Verified end-to-end against PostgreSQL 16 with the real model types:
Channel create/update/read-back, Task json fields, PrefillGroup items.

fix(sqlite): enable WAL + working busy timeout + _txlock=immediate to stop concurrent write lockouts (#7030)

* fix(sqlite): enable WAL + working busy timeout + _txlock=immediate to stop concurrent write lockouts

Co-authored-by: Claude <noreply@anthropic.com>

Marketplace install/upgrade on a factory-served plugin actually created
a permanent override shadowing every future built-in release. The card
now shows an informational "Updates with the system" badge instead of
the action, while keeping the built-in vs marketplace version line and
the upgradable state badge visible. Deliberate overrides are untouched:
upload and marketplace actions on overridden or third-party plugins
behave as before, and the plugins table now hints when an override
lags behind the shipped built-in version so operators know deleting it
restores the newer factory plugin.