Superlog's docs are compact and current-looking. They ship llms.txt, llms-full.txt, and an OpenAPI 3.1 spec. But the guides, the OpenAPI spec, and the pricing page disagree on several things that matter in production: billing behavior, run limits, the security boundary of tag mode, and how to remove credentials and secrets.
1. Billing docs contradict the pricing page on plan names, overage, fallback, and seats (critical)
Problem: The two pages describe different billing models:
Plan name. The billing page lists "Max | $200 / month". The pricing page calls the $200 plan Team ("| Plan | Free | Pro | Team | Enterprise |").
Hitting the quota. The billing page says: "New runs and tag mode questions do not start until the quota resets at the start of the next billing period or you upgrade." The pricing page says the hard stop applies only to Free: "On Pro and Team, extra Superlog AI usage is billed at the model provider's API list price" and "Extra machine time | Hard stop at the cap | Per-hour rate | Lower per-hour rate".
Fallback to included usage. The pricing page advertises an opt-in "Fallback to usage credit — When your key or subscription runs out" setting ("Optional, off by default" on Pro, "Optional" on Team). No docs page mentions this setting or where to turn it on. The only statement on the subject is the Models page's flat "If the provider rejects your key… the run fails… It does not fall back to included usage." That reads as absolute even though, per the pricing page, an exhausted key can fall back when the setting is on.
Free quota. The billing page says the quota "resets at the start of the next billing period". The pricing page describes Free AI credit as "$5 (one-time grant)".
Seats. The pricing page shows "Seats | 1 | 1 | Unlimited" and "Shared workspaces and admin dashboard — Not included" for Free and Pro. Meanwhile, the Workspace page describes inviting teammates ("Under Invite a teammate, enter their email… and click Invite") with no plan caveat.
Consequence: A team can't tell from the docs whether going over quota stops its automations or adds charges, whether a failing BYO key silently moves spend onto Superlog credit, or whether its plan allows a second member. These are the questions that decide whether a production automation pipeline fails or costs money unexpectedly.
The fix: Make the billing page match the live pricing model. Rename Max to Team (or the reverse). Document overage billing per plan. Document the optional fallback-to-credit setting (which plans have it, where to turn it on, and which failures trigger it), and qualify the Models page's "does not fall back" sentence to say it applies only when the setting is off. State that the Free credit is a one-time grant. Note seat limits next to "Invite a teammate".
2. Tag mode is called "read-only" but can be given write access (critical)
Problem: The Tag mode page says: "Tag mode is read-only. It does not open pull requests, create tickets, or change anything in your tools." Other pages contradict this:
Supabase. The integrations table lists Supabase's tag mode access as "Configurable", and the Supabase section offers "Full database SQL access | SQL that can change data or schema."
Custom MCP servers. The integrations table says "Read" for tag mode. But the Custom MCP page is titled "Connect any remote MCP server so automations and tag mode can use its tools" and says "It does not filter them to read-only, so the agent can call write tools if the server has them." The Tag mode page lists "custom MCP servers" among the integrations tag mode can read.
Datadog. The same matrix labels the Datadog automation connector "Read". The Connectors page says its write capability is "Any tool the Datadog MCP server exposes", and the overview itself admits "Datadog and custom MCP servers pass their tools through without filtering."
Consequence: Tag mode answers anyone who can mention the bot in Slack. An admin who trusts the "read-only" promise and turns on a full-access Supabase connection or a custom MCP server with write tools has given every Slack user an agent that can change production data. This is a security-boundary error, not a wording issue.
The fix: Remove the absolute "read-only" claim or qualify it. For example: "Built-in integrations are read-only in tag mode, except Supabase with Full database SQL access and custom MCP servers, which pass through every tool." Change the matrix cells for Custom MCP (tag mode) and Datadog (connector) from "Read" to "All server tools", and add a warning box on the Tag mode page.
3. Removing a model credential: the API reference and the Models guide describe opposite behavior (significant)
Problem: The API reference says: "Removes a model credential. Automations that used it fall back to included usage." The Models guide says: "A key that an automation is using cannot be removed." The OpenAPI spec doesn't settle it. The 409 description for DELETE /model-credentials/{credentialId} is the generic "The request conflicts with the current state, such as a name already in use."
Consequence: A script that rotates or deletes keys can't know whether to expect 200, with automations silently moving to paid included usage, or a 409 it must handle. If the API page is right, deleting a key quietly moves inference spend onto the plan quota. If the guide is right, cleanup scripts fail with an unexplained conflict.
The fix: Pick the real behavior and document it in both places. If in-use keys are blocked, give the 409 a specific code (for example credential_in_use) and describe it on the endpoint. If removal falls back, remove the "cannot be removed" sentence and warn about the billing effect.
4. The documented 30-minute run limit is configurable up to 60 minutes in the API (significant)
Problem: The guides state a fixed limit: "A run can last up to 30 minutes", and the Limits table lists "Run time | 30 minutes". The OpenAPI configuration schema has "maxRuntimeSeconds": {"description": "Defaults to 1800.", "minimum": 60, "maximum": 3600}. It also has maxModelRequests (default 500, max 1000) and maxOutputTokensPerRequest (default 16000, max 100000), and none of these three settings appears in any guide.
Consequence: Teams whose runs time out at 30 minutes believe that's a hard platform limit and redesign their automations. The knob that would fix it exists only in a JSON schema. Runs that fail on the model-request cap get no explanation, because the cap isn't documented.
The fix: Describe the 30 minutes as a default, document maxRuntimeSeconds, maxModelRequests, and maxOutputTokensPerRequest in the Limits table, and say whether the UI exposes them or only the API does.
5. Run notifications: "schedule only" in the guides, "scheduled or Sentry" in the API (significant)
Problem: The Runs page says: "Notifications are available only for automations with a schedule trigger. Event-triggered automations report through their connectors". How Superlog works agrees: "For scheduled automations, the Slack channels that receive each finished run." The OpenAPI schema describes notifications as "Slack channels that receive the result of each scheduled or Sentry run."
Consequence: Someone building a Sentry-triage automation over the API can't tell whether notifications will post results or be ignored. An agent following the guide won't set it. An agent following the spec will, and gets behavior the guide says can't happen.
The fix: Fix whichever source is wrong. If Sentry runs support notifications, update the Runs and How-it-works pages and the UI description.
6. Workspace secrets: a stale app route, contradictory scoping, and no way to delete (significant)
Stale route. The "Create a secret" steps start with "Open /agents/new in Superlog." No other page uses an "agents" route. Every other page sends you to the automation editor's connector picker or Tag mode.
Scoping. Step 5 says that after you click Store and add, "The secret is now available to every automation in the workspace and to tag mode". The next section, and the Connectors page, say you must pick each secret per automation (up to 20) and choose it separately for tag mode.
No delete. Rotation is: "Secrets cannot be edited. To rotate a credential, create a new secret with a new name… and remove the old one from them." No page explains how to delete a secret from the workspace. The OpenAPI spec has only GET and POST /secrets, with no /secrets/{id} path.
Consequence: Developers follow a URL that may not match the current app. They may assume a secret is already in scope and skip selecting it, so runs see no variable. Each rotation leaves an old, unremovable credential (still allowed to reach its hosts) in the workspace.
The fix: Replace /agents/new with the real navigation path. Clarify that storing a secret makes it selectable, not active. Document secret deletion in the UI and add DELETE /secrets/{secretId}, or say plainly that deletion isn't supported.
7. The error reference lists codes the spec never declares, and no rate limits are documented (significant)
Problem: The Errors table on the API introduction includes "502, 503 | A provider or the run queue is unavailable. Try again." The OpenAPI spec declares only 200, 201, 202, 400, 401, 404, 409. No page mentions 429, rate limits, or retry/backoff guidance. And every endpoint reuses one 409 description, "such as a name already in use", even where a name can't conflict: POST /automations/{id}/runs, POST /runs/{runId}/messages, DELETE /model-credentials/{id}. The intro mentions "a stable code", but the only codes documented anywhere are invalid_request (shown in the docs example) and unauthorized (seen in a live probe). No list of codes is published.
Consequence: Generated clients and agents built from the spec won't handle 502/503/429. Callers who get a 409 when starting a run can't tell "run still active" from anything else without guessing. The intro does say "a run that is still active", but the endpoint doesn't.
The fix: Add 429/502/503 to the spec with the shared Error schema. Write endpoint-specific 409 descriptions. Publish the list of code values and the rate limits per key or workspace.
8. The self-hosting guide says "no specific cloud provider", but Google Cloud requires running on AWS (significant)
Problem: The Deployment page says: "The application does not depend on a specific cloud provider." The Environment page says: "The Google Cloud integration also requires AWS_INTEGRATION_PRINCIPAL_ARN… Self-hosted deployments must run on AWS with permission to assume that role."
Consequence: A team self-hosting on GCP, Azure, or bare metal finds out only after deploying that the Google Cloud integration (and AWS context) can't work for them.
The fix: Put the AWS dependency for the AWS and GCP integrations in a requirements section on the Self-hosting overview and Deployment pages, and qualify the "no specific cloud provider" claim.
9. The deployment steps don't run anything and omit a route the repository README requires (significant)
Problem: "Build the control plane" and "Run the worker" both show only docker build .... No step runs either container, and no step gives ports, health checks, or the environment needed at runtime. The routing step says to "route /api/* on the same origin to the control-plane service." The repository README, which the docs link as the source, says: "Route /api/* and /.well-known/* on that origin to the control-plane service." The README also says MCP clients "connect with OAuth: they register themselves".
Consequence: Operators who follow the docs end up with built images and no running service. A proxy that forwards only /api/* leaves out a route the maintainers' own deployment steps include. That is likely what MCP OAuth client registration and discovery need, but the docs never say what /.well-known/* serves. Whatever depends on it fails, and operators can't trace the failure back to the docs.
The fix: Add docker run (or compose) examples with ports and env files, and add /.well-known/* to the routing step, explaining what it serves.
10. Two public "superlog" repositories describe two different products (significant)
Problem: The docs say the code is at superloglabs/responder-oss, "where the project is named Responder". The same organization also publishes superloglabs/superlog, whose README says "Superlog is an open-source agentic telemetry system. It ingests traces, logs, and metrics…" That README describes an OTLP ingest proxy, ClickHouse, and different prerequisites ("Node.js 20+", "pnpm 9+" versus the docs' "Node.js 24 or newer", "pnpm 11"). Self-hosted operators also run into Responder naming everywhere: RESPONDER_* variables, ResponderInvestigationRole, the responder-investigation service account, and @responder/core.
Consequence: Developers and coding agents that search for "Superlog GitHub" land on the repo literally named superlog and follow setup steps for a different product. The docs never mention the other repository, so nothing tells them which one is current.
The fix: Add a note on the docs home and Self-hosting overview saying superloglabs/superlog is the earlier telemetry product (or its current status), and update that README to link to the docs and responder-oss. Consider a short Responder → Superlog naming glossary for operators.
11. The llms-full.txt export drops every environment-variable name on the Environment page (significant)
Problem: In llms-full.txt, each variable on the Environment page appears as <ParamField type="string"> followed by a description, with no name. For example: "<ParamField type="string"> Overrides the tag mode model. Defaults to gpt-5.6-sol. </ParamField>". The .md export keeps the name: <ParamField path="OPENAI_AGENT_MODEL" type="string" default="gpt-5.6-sol">. The same happens for DAYTONA_API_KEY, AI_GATEWAY_API_KEY, OPENAI_API_KEY, DAYTONA_API_URL, DAYTONA_TARGET, BILLING_ENABLED, AUTUMN_SECRET_KEY, and every other variable documented with <ParamField>.
Consequence: Coding agents that ingest llms-full.txt (the file the docs advertise for agents) see a list of descriptions with no variable names. They can't configure a self-hosted deployment from it and will guess names.
The fix: Render the path attribute into the llms-full.txt output, or rewrite these entries as a table with explicit names.
12. The Workspace page's roles and Settings tabs don't match the rest of the docs (minor)
Problem: The Workspace page lists only two roles, Admin and Member, and says "The tabs are Workspace, Models, and Billing." The API keys page says "Admins and owners can revoke any key", and is reached through "Settings → API keys", a tab that list leaves out. The OpenAPI role field is "owner, admin, or member."
Consequence: Admins can't tell what an owner can do or how ownership is assigned. Readers following the Workspace page won't find where API keys live.
The fix: Add Owner to the roles table with its permissions, and add API keys to the list of Settings tabs.
13. Run status names differ between the UI guide and the API (minor)
Problem: The Runs page status table uses Queued, Running, Completed, Failed, and Cancelled. The API enum is pending, running, succeeded, failed, cancelled. No page maps one set to the other.
Consequence: A developer or agent that reads the Runs page and polls until status == "completed" never sees that value and loops until it times out.
The fix: Add the API value next to each UI label in the Runs table (for example, Completed = succeeded, Queued = pending).
14. "The same operations are available as tools in the MCP server" isn't true (minor)
Problem: The API introduction says "The same operations are available as tools in the Superlog MCP server." The MCP page leaves several out ("Adding a workspace secret or a model API key is not available as a tool"), and its tools table also has no rotate-key tool. In the other direction, MCP has list_available_integrations and start_integration_connection, which have no REST equivalent.
Consequence: Agents and users who plan workflows from the introduction expect parity and hit missing tools partway through.
The fix: Replace the parity claim with a short mapping table, or "most operations, except…".
15. Schedule trigger schema requires fields the guide says hourly and daily schedules don't have (minor)
Problem: The guide says Every hour "Runs at the top of each hour" and Every day is "Pick the time, on the hour", so hourly takes no options and daily takes no day. The OpenAPI schedule trigger lists "required": ["frequency", "hour", "kind", "timezone", "weekday"], which requires hour and weekday for every frequency. The field descriptions say hour is "for daily and weekly runs" and weekday is "for weekly runs".
Consequence: API callers creating an hourly or daily schedule get a 400 unless they send placeholder values. Nothing says which values are safe or whether they're ignored.
The fix: Make hour and weekday conditionally required in the schema, or document that they're required and ignored for hourly and daily schedules.
16. No changelog or status page for a product with moving defaults (minor)
Problem: The 59-URL sitemap has no changelog, release notes, or status page. /changelog returns 404, and status.superlog.sh doesn't resolve. Meanwhile the docs pin version-sensitive defaults, such as the automation model "gpt-5.4" and the tag mode model "gpt-5.6-sol".
Consequence: When the default model or a limit changes, users and agents have no dated record to compare behavior against, and no place to check outages before debugging failed runs.
The fix: Publish a changelog page, include it in llms.txt, and link a status page from the docs.
17. Tag mode ignores workspace model keys, and its model isn't documented for hosted users (minor)
Problem: The docs home page promises "Choose your model and harness, and bring your own tokens and subscriptions." The Environment page says tag mode runs on a deployment-level key: "OpenAI API key the worker uses for tag mode. Workspace model keys in Settings → Models apply to automations only", with the model defaulting to gpt-5.6-sol. The Tag mode page, the one hosted users read, never mentions which model answers, never says BYO keys don't apply, and says only: "Tag mode questions use your plan's usage allowance." It also says "Slack automations with an App mentioned trigger run alongside tag mode" without saying whether one mention then produces two replies and two quota charges.
Consequence: Teams that add their own provider keys expecting all inference to move off the Superlog quota still find tag-mode questions draining it. They can't choose or even identify the model behind tag-mode answers. Channels that combine tag mode with an App mentioned automation may get duplicate responses and duplicate charges.
The fix: On the Tag mode page, state which model tag mode uses, that workspace model keys and ChatGPT subscriptions don't apply to it, and how its usage is metered. Explain exactly what happens when a mention matches both tag mode and an App mentioned automation.
What they do well
Agent-ready entry points exist: llms.txt, llms-full.txt, Markdown exports of every page, and an OpenAPI 3.1 spec whose operations map to MCP tool names.
The security page is concrete about credential handling: sandbox placeholders, a broker outside the sandbox, and AES-256-GCM encryption. The live API's {code, error} envelope matches the documented error shape.
Top 3 recommendations
Make the billing page match the pricing page (plan names, overage, fallback-to-credit, seats), and fix the contradiction over model-credential removal.
Remove or qualify "tag mode is read-only", and correct the integrations access matrix for Supabase, custom MCP, and Datadog.
Make the OpenAPI spec and the guides agree: run limits, notifications, schedule fields, error codes including 429/5xx, and secret deletion. Fix llms-full.txt so environment-variable names come through.
PASS: the JSON/TOML config and error examples have valid syntax, and the openssl secret-generation commands produce the documented formats (32-byte base64, 64-char hex). The documented error envelope (code, error) matches what the live API returned.
SKIP: the API calls couldn't be tested because the supplied workspace API key (used as SUPERLOG_API_KEY) was rejected by https://superlog.sh/api/v1 with HTTP 401 unauthorized. The claude CLI isn't available in the sandbox. Self-hosting commands need a checkout of the source repository plus Docker and Postgres, which are out of scope for this verifier.
No snippet failed in a way attributable to the documentation.
Required credentials
SUPERLOG_API_KEY: a Superlog workspace API key (slk_…) created under Settings → API keys. One was supplied, but the API rejected it with 401 for every authenticated call, so all authenticated snippets are SKIP.
AUTOMATION_ID: the ID of an existing automation (needed by the PATCH example). Not supplied.
A workspace-specific GitHub repository ID (repositoryIds) from GET /integrations (needed by the create-automation example). Not available.
Ran as written: HTTP 401 (supplied credential rejected). Even with a valid key, it needs a real repositoryIds value from GET /integrations. The UUID in the example is illustrative.
Needs a repository checkout and a Postgres DATABASE_URL.
14
openssl rand -base64 32 / openssl rand -hex 32
bash
PASS
Produced a value that decodes to 32 bytes and a 64-character hex string, matching the documented CREDENTIAL_ENCRYPTION_KEY and INTERNAL_INGEST_TOKEN formats.