Agnost AI Documentation Audit

The docs cover a respectable surface area — MCP SDKs in three languages, a conversation SDK in two, half a dozen OTel framework guides, and an OpenAPI spec — but three "Next steps" links from the Quickstart 404, the REST API reference pages are effectively empty, the package-name conventions invert between Python and TypeScript, the Apify guide ends by sending Python users to a Configuration page about a different SDK, and llms.txt (the only thing the docs explicitly advertise to AI agents) omits the Quickstart, Configuration, and Architecture pages entirely.

1. Three Quickstart "Next steps" links resolve to 404 (critical)

Location: /quickstart "Next steps" and "Troubleshooting" sections; verified against /authentication, /errors, /sdks, /architecture

Problem: The Quickstart's "Next steps" list points developers to four pages — "SDK index", "Authentication: API keys and JWT for the REST API", "Errors: error response schema and status codes", and "Architecture". Direct fetches show /authentication, /errors, and /sdks return HTTP 404; only /architecture resolves. The Quickstart's Troubleshooting block also tells users to "see Authentication" for "programmatic dashboard access" — same dead route. (Separately, /introduction also 404s, but the canonical landing page lives at /index and is correctly indexed in llms.txt, so that URL is a stale alias rather than a broken Next-steps link.)

Consequence: Every developer who finishes the Quickstart and tries to learn auth, error handling, or "what other SDKs exist" hits a wall. Agents following the links fail silently and can't reconstruct the intended sitemap.

The fix: Either ship the three missing pages (Authentication, Errors, SDK index) or remove their references from the Quickstart "Next steps" and Troubleshooting sections. Add a CI link-checker against rendered pages so future broken internal references fail the build. Optionally, redirect /introduction → /index so stray inbound links don't 404.

2. Package naming inverts between Python and TypeScript (critical)

Location: /python-conversation, /typescript-conversation, /typescript-sdk, /fastmcp, /anthropic-low-level, /apify

Problem: Naming is inconsistent across languages:

So in Python, agnost is the Conversation SDK and agnost-mcp is the MCP SDK. In TypeScript, agnost is the MCP SDK and agnostai is the Conversation SDK. The Apify page hands developers both at once — pip install agnost-mcp for Python and npm install agnost for Node.js — without flagging that the bare name agnost means two different things on the two runtimes.

Consequence: A developer or coding agent porting a Python integration to Node.js (or vice versa) will install the wrong package, get import errors at runtime, and have no doc page that contrasts the two. Search for "install agnost" returns five pages, all of which look correct in isolation and contradict each other in aggregate.

The fix: Either rename packages so the same name means the same thing across runtimes (e.g. @agnost/mcp + @agnost/conversation and agnost-mcp + agnost-conversation), or add a single "Which package do I install?" table at the top of the SDK section and link to it from every install snippet.

3. The REST API reference pages are stubs (critical)

Location: /capture-event, /capture-session

Problem: Both pages, as rendered, contain only a title, the endpoint (POST /api/v1/capture-event, POST /api/v1/capture-session), and a one-line description. No request body schema, no example payload, no response, no headers, no error codes — despite llms.txt listing them as Capture Event and Capture Session and the OpenAPI spec at /openapi.yaml clearly defining ingestion endpoints under the SDK tag.

Consequence: A developer (or agent) trying to send events directly without an SDK — exactly the use case for documenting raw endpoints — has nothing to copy. They have to download openapi.yaml and read YAML by hand, which defeats the purpose of having human-readable reference pages.

The fix: Render request/response schemas, an example curl, and the auth header (x-org-id) on both pages — Mintlify's <RequestExample> / <ResponseExample> components or the OpenAPI playground component handle this from the existing spec. At minimum, document the JSON body shape inline.

4. llms.txt omits Quickstart, Configuration, and Architecture — including the only page that documents key lifecycle methods (critical)

Location: /llms.txt

Problem: The published llms.txt lists 17 SDK / OTel / API pages but does not include /quickstart, /configuration, or /architecture, all of which exist (verified by direct fetch). The Quickstart is the canonical "five minute hello world", Configuration is the only place that documents lifecycle methods agnost.flush(), agnost.shutdown(), and agnost.set_debug_logs() (the /python-conversation page only mentions shutdown() in passing and never names flush() or set_debug_logs() at all), and Architecture is the only page that explains the ingestion pipeline.

Consequence: This is exactly the file AI coding agents consume to plan retrieval — and it omits the three pages an agent most needs to onboard a user. An agent told "set up Agnost in my Python app" will be routed to one of 17 framework-specific pages and miss the canonical Quickstart; even if it lands on /python-conversation, it will have no signal that flush() or set_debug_logs() exist.

The fix: Add the three missing entries to llms.txt, and consider auto-generating llms.txt from the Mintlify navigation so future pages don't drift out of the index. Either inline the lifecycle methods (flush, shutdown, set_debug_logs) into /python-conversation or cross-link to /configuration from it. Optionally publish llms-full.txt for agents that want the bundled content.

5. Apify page sends Python users to a Configuration guide that documents a different SDK (significant)

Location: /apify (footer link), /configuration

Problem: The Apify page just walked a Python developer through pip install agnost-mcp and track(server, "your-org-id") (the MCP analytics SDK). It then ends with: "See the full Configuration guide for more advanced identification patterns." But /configuration is explicitly titled "Configure the Agnost AI Python SDK for conversation tracking" and documents agnost.init(), agnost.begin(), agnost.identify() — none of which exist in agnost_mcp. The agnost_mcp.track(...) API takes an identify callable via config(...) (per /anthropic-low-level and /fastmcp), which /configuration never mentions.

Consequence: A Python Apify developer clicking through for "advanced identification patterns" lands on an unrelated API surface, will likely copy agnost.init(...) / agnost.identify(...) calls that don't exist in the package they installed, and get AttributeError at runtime. Agents will compound the confusion by stitching the two APIs together.

The fix: Either link to the MCP-side identify docs (/fastmcp or /anthropic-low-level, which show config(identify=...)), or write a dedicated MCP configuration page. At minimum, rename /configuration to make its scope explicit (e.g. "Python Conversation SDK configuration") and stop linking it from MCP-flavored pages.

6. OTel endpoint contradicts itself between Quickstart and OTel page (significant)

Location: /quickstart "Path C: OpenTelemetry" vs /otel "Quick Setup"

Problem: Quickstart shows:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.agnost.ai/v1/traces

The OTel page shows:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.agnost.ai

and then notes "For SDKs that take an explicit URL, use https://otel.agnost.ai/v1/traces." The OTLP spec is unforgiving about this: when OTEL_EXPORTER_OTLP_ENDPOINT is set (vs the signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT), exporters typically append /v1/traces. Putting /v1/traces in the base env var produces /v1/traces/v1/traces on many SDKs.

Consequence: Developers who copy the Quickstart snippet into a stock OTel-SDK app will get traces silently dropped (404 from the collector) and no helpful error. Agents will copy whichever page they hit first and the success rate of the integration becomes a coin flip.

The fix: Pick one convention (almost certainly https://otel.agnost.ai for the env var) and use it on both pages. Add a short "When to use the /v1/traces suffix" note for SDKs that take an explicit URL argument.

7. Authentication is referenced everywhere and explained nowhere — and the header name differs across surfaces (significant)

Location: OpenAPI /openapi.yaml, /quickstart Troubleshooting, all OTel pages, MCP server SDK pages

Problem: The OpenAPI spec describes three distinct auth mechanisms — x-org-id for SDK ingestion, Authorization: Bearer <jwt> + x-org-id for the dashboard, and x-api-key (agnost_<64-hex>) for programmatic dashboard access. The Quickstart promises an Authentication page covering "API keys and JWT for the REST API" — which 404s. No rendered page explains how to mint an API key, how long JWTs live, what x-api-key permits vs Authorization: Bearer, or what "Settings → API Keys" actually exposes.

Compounding this: the same organization identifier is sent under different header names depending on the surface. OpenAPI ingestion uses x-org-id; every OTel page and the Quickstart's OTLP example use X-Agnost-Org-ID. The Quickstart Troubleshooting even warns "confirm the header name is X-Agnost-Org-ID (case-insensitive but the dash placement matters)" — a tell that users are getting this wrong.

Consequence: A developer who wants to script against the dashboard (export sentiment data, list conversations) has the endpoints in openapi.yaml but no documented path from "I have a login" to "I have credentials a script can use". A developer mixing surfaces (OTel + direct ingestion) sees two header names for the same identifier and has to guess which goes where. Agents will guess at headers or invent flows.

The fix: Publish /authentication covering: (1) how SDK ingestion uses x-org-id, (2) how OTLP ingestion uses X-Agnost-Org-ID and why the name differs (or unify them), (3) how to create an API key in Settings → API Keys, (4) the difference between JWT and API key auth, (5) key format (agnost_<64-hex>), (6) expiry/rotation. Link it from every page that says "see Authentication".

8. "write_key" is mentioned once and used nowhere (significant)

Location: /quickstart step 1, /configuration "Basic Configuration"

Problem: Quickstart: "This ID is what we'll refer to as your org_id (also called write_key in some SDKs)." Configuration page header: "Initialize the SDK with your organization ID (passed positionally as write_key)". But none of the actual SDK code samples — Python conversation (agnost.init("your-org-id")), TypeScript conversation (agnost.init('your-org-id')), track(server, "your-org-id", ...) etc. — name the parameter write_key. The Python Conversation SDK options table calls it org_id.

Consequence: A developer reading "passed positionally as write_key" in the Configuration page will grep their codebase for write_key, find nothing, and lose confidence in the docs. An agent generating wrapper code may invent a write_key kwarg and crash.

The fix: Either drop the write_key references entirely (cleanest), or pick one term and use it everywhere — including in the SDK signatures themselves.

9. MCP Toolbox telemetry endpoint omits the URL scheme (minor)

Location: /mcp-toolbox "Enable Telemetry"

Problem: The page tells users to run:

./toolbox --tools-file "tools.yaml" --telemetry-otlp="otel.agnost.ai"

Every other page on the site spells the endpoint https://otel.agnost.ai (or https://otel.agnost.ai/v1/traces). Whether otel.agnost.ai works depends on whether genai-toolbox prepends a scheme — and on whether it defaults to HTTP or HTTPS.

Consequence: Best case it works; worst case the toolbox tries http://otel.agnost.ai, gets redirected/refused, and the user has nothing to grep for. The flag also reads as deliberately inconsistent with the rest of the docs.

The fix: Use --telemetry-otlp="https://otel.agnost.ai" and confirm the toolbox accepts a full URL there. If the flag strictly wants a host, say so explicitly.

10. MCP Toolbox install pins a specific version in copy-paste examples (minor)

Location: /mcp-toolbox "Installation"

Problem: Both Binary and Docker install snippets hardcode export VERSION=0.28.0. A developer who copies the snippet six months from now installs an old release; an agent that scrapes the snippet has no signal that the version should be parameterised.

Consequence: Users silently fall behind on toolbox security/bug fixes, or hit incompatibilities with newer tools.yaml features without a hint that the version is stale.

The fix: Either link to a "latest version" badge or show the install with a VERSION=$(curl -s …/latest) lookup. At minimum, add a "check the latest release here" link.

What they do well

• OpenAPI spec is published and discoverable via llms.txt, with tagged groups (SDK, Dashboard, Alerts, SOPs, etc.) — agents can at least enumerate endpoints even when the rendered pages are thin.
• OTel framework coverage is unusually broad — LangChain, Mastra, Vercel AI SDK, VoltAgent, OpenAI SDK each get a dedicated page with per-call userId / sessionId snippets, not just a generic "set these env vars" page.
• The OpenAI SDK page explicitly documents version-compatibility pinning between openai and @arizeai/openinference-instrumentation-openai, with the exact error message developers will hit otherwise — a quality of attention missing from many of the other pages.

Top 3 recommendations

1. Fix the dead Quickstart links and publish /authentication and /errors — every "Next steps" entry should resolve, and the three documented auth mechanisms (plus the x-org-id vs X-Agnost-Org-ID header split) need one canonical explanation.
2. Resolve the package-naming inversion between Python (agnost = conversation) and TypeScript (agnost = MCP, agnostai = conversation), and stop linking MCP-flavored pages (Apify, FastMCP, Anthropic low-level) to a /configuration page that only describes the Conversation SDK.
3. Make the agent-facing surface complete: add /quickstart, /configuration, and /architecture to llms.txt; cross-link the lifecycle methods (flush, shutdown, set_debug_logs) into /python-conversation; and fill out /capture-event and /capture-session with schemas and examples from the existing OpenAPI spec.