Instance admin

An instance-wide view and control panel for the people who run a TotisWorkers installation, alongside the org-scoped console every member has. It covers host health, ingress, the system app and its database, and management of every organisation, user and resource.

Who can use it

Instance owners only: the seed user created at first-run setup (users.instance_owner = 1) and anyone an owner later promotes. It is reached from the account menu as Instance (#/instance). Access is signed-in browser sessions only — API tokens are refused on every /v1/instance/* route, as a server-side invariant, not just a UI gate.

What it is — and is not

Instance owners see metadata across organisations (names, counts, sizes, request rates, status codes, error counts), never tenant content: no database rows, secret values, message bodies, site files or another org's app logs. Management acts on resources as objects — suspend, unpublish, delete, transfer — never by reading or editing their content.

Every management action requires a reason (5–500 chars) and is written to the instance audit log and to the affected organisation's audit log (actor shown as "Instance owner <email>"), so tenants can see what the operator did.

Observe

Read-only, per-user rate-limited, never cached:

Route	Shows
GET /v1/instance/overview	version/build, uptime, runtime kind, KEK source (warns when generated), instance email configured, counts, last-24 h requests and error rate, top apps by traffic and 5xx
GET /v1/instance/ingress?window=&surface=&target=	request series by status class, p50/p95/p99, bytes; breakdowns by surface, kind, target, status code; system API routes by template
GET /v1/instance/host?window=1h|6h	host samples: CPU, heap, GC, threads, disk, JS runtime pool stats
GET /v1/instance/system	the system app and db_system: tables with row counts, size, applied/pending migrations
GET /v1/instance/inventory?kind=apps|databases|queues|storage|email|orgs	every resource across orgs with owning org/project/env, dates, size/depth, 24 h traffic — metadata only
GET /v1/instance/logs?source=host|system-app&level=&q=&before=	host warnings/errors (5,000-line ring) and the system app's logs
GET /v1/instance/security?window=24h	failed logins, two-factor events (mfa.failed, mfa.lockout, mfa.enable, mfa.disable, mfa.recovery.used), rate-limit hits, device approvals, resets, token creations — no secrets

Ingress aggregates keep tenant traffic at (surface, app, kind, status, latency, bytes) — never raw paths, query strings, headers, bodies or client IPs. Only the system app's routes are kept by template.

Manage

Every management route is POST/PATCH/DELETE, browser session + instance owner, same-origin, and takes a mandatory reason; destructive ones also take confirm = the target's slug or email. The most sensitive ones — deleting, disabling or transferring, granting instance-owner, resetting a user's 2FA, and changing the instance 2FA requirement — also need a recent re-authentication (password, or a code for owners with 2FA): without one they answer 403 {reauth: true, method} and the console re-confirms before retrying.

Organisations (/v1/instance/orgs…): create, rename/re-slug, suspend/resume, add or transfer owners, set per-org limits, and delete (two-step with a 7-day grace, cancellable via restore; the system org can never be deleted). Suspension makes the org's apps answer 503, pauses its queue consumers, suspends email, and makes every tenant write return 423 while reads keep working.

Platform AI (/v1/instance/ai…). The console page is Limits & quotas → AI.

• GET returns the routing, the key status (a fingerprint, never the value), the grants and this node's usage.
• PUT {instance?, keys?, reason} sets the routing (aliases, fallbacks, model catalogue, provider base URLs, concurrency) and the OpenAI/Anthropic keys. Changing a key needs a recent re-authentication, and the audit log records only which keys changed.
• PUT /v1/instance/ai/{appId} {grant | null, reason} grants an app AI or removes the grant. PUT /v1/instance/ai/projects {projectId, envId, grant | null, reason} grants a whole project environment, with one shared quota. Every app is at zero until granted. Daily tokens and the monthly cost are enforced across the cluster from each node's usage file, which is flushed every 5 s.

Users (/v1/instance/users…): invite, disable/enable (kills sessions and tokens at once), revoke sessions or tokens, send a password-reset or one-time recovery link, reset a user's two-factor authentication (POST /v1/instance/users/{id}/mfa-reset {reason} — clears the factor and recovery codes, signs the user out everywhere including uncollected device approvals, leaves API tokens; refused on your own account), grant/remove instance-owner, edit memberships, delete. The user detail shows whether a user has 2FA on. The last owner of an org and the last instance owner are protected, and an owner cannot change their own instance-owner flag.

Instance 2FA requirement (/v1/instance/mfa): GET returns requireMfaForInstanceOwners and whether the caller is enrolled; PUT {requireMfaForInstanceOwners, reason} turns it on or off. When on, every instance owner needs a second factor at the next sign-in with no grace period; it can only be turned on by an owner who already has 2FA, and it is not settable through PUT /v1/instance/limits (which rejects the key). The console shows it on Instance → Security.

Resources (by id): suspend/resume or unpublish an app (hold cleared only by an instance owner), set per-app limits, delete an app, delete a database (refused while bound unless {force:true}), pause/resume/purge a queue, delete a storage or email connection, and release a disputed custom domain.

Outbound fetch (/v1/instance/fetch): GET lists the instance default and the organisations with their own setting; PUT {enabled, reason} sets the instance default and PUT /v1/instance/orgs/{id}/fetch {value: true|false|null, reason} sets or clears one organisation's. Off on a fresh install. The console shows it on the System page and on an organisation's Limits tab.

Limits (/v1/instance/limits):

• GET returns the per-app defaults and ceilings, the node settings and the private address allow-list.
• PUT {defaults?, ceilings?, settings?, privateAllow?, reason} changes them. A field set to null goes back to its built-in value.
• PUT /v1/instance/orgs/{id}/ceilings {ceilings | null, reason} raises or lowers one organisation's per-app ceilings.

The console shows these on the Instance → Limits page and on an organisation's Limits tab. Pool size and the cache budgets apply on the next start; everything else applies at once.

The Limits page's Features card carries the instance's boolean node settings (HTMLRewriter, and Database writes from the console). Database writes from the console is the consoleDataWrites setting (default on): when off it refuses row edits and inserts in the console table browser, write statements from the query console, and data API insert-rows — all with 403 forbidden ("data writes are disabled on this instance"). Apps writing through env.DB are never affected, and neither are write statements run through the data API query endpoint.

Suspension and holds are registry flags (OrgRecord.suspended, AppRecord.suspended, AppRecord.publishHold) enforced where requests are served, deploys run, and queue consumers are dispatched — so they hold even if the management app had a bug.

System database browsing

The system app's db_system can be browsed read-only from the instance view, but credential-bearing columns (password_hash, token_hash, secret_hash, token_value, device_hash, code_hash) are masked server-side — the mask list lives in the host op, not the UI.

Status

The instance-admin backend (ingress/host metrics, host ops, and the /v1/instance/* management and observe routes) is implemented. The console pages are under #/instance.