Agnost

Agnost AI Documentation Audit

The docs cover a respectable surface area — MCP SDKs in three languages, a conversation SDK in two, half a dozen OTel framework guides, and an OpenAPI spec — but three "Next steps" links from the Quickstart 404, the REST API reference pages are effectively empty, the package-name conventions invert between Python and TypeScript, the Apify guide ends by sending Python users to a Configuration page about a different SDK, and llms.txt (the only thing the docs explicitly advertise to AI agents) omits the Quickstart, Configuration, and Architecture pages entirely.

1. Three Quickstart "Next steps" links resolve to 404 (critical)

Location: /quickstart "Next steps" and "Troubleshooting" sections; verified against /authentication, /errors, /sdks, /architecture

Problem: The Quickstart's "Next steps" list points developers to four pages — "SDK index", "Authentication: API keys and JWT for the REST API", "Errors: error response schema and status codes", and "Architecture". Direct fetches show /authentication, /errors, and /sdks return HTTP 404; only /architecture resolves. The Quickstart's Troubleshooting block also tells users to "see Authentication" for "programmatic dashboard access" — same dead route. (Separately, /introduction also 404s, but the canonical landing page lives at /index and is correctly indexed in llms.txt, so that URL is a stale alias rather than a broken Next-steps link.)

Consequence: Every developer who finishes the Quickstart and tries to learn auth, error handling, or "what other SDKs exist" hits a wall. Agents following the links fail silently and can't reconstruct the intended sitemap.

The fix: Either ship the three missing pages (Authentication, Errors, SDK index) or remove their references from the Quickstart "Next steps" and Troubleshooting sections. Add a CI link-checker against rendered pages so future broken internal references fail the build. Optionally, redirect /introduction → /index so stray inbound links don't 404.

2. Package naming inverts between Python and TypeScript (critical)

Location: /python-conversation, /typescript-conversation, /typescript-sdk, /fastmcp, /anthropic-low-level, /apify

Problem: Naming is inconsistent across languages:

Agnost AI Documentation Audit

1. Three Quickstart "Next steps" links resolve to 404 (critical)

Location: /quickstart "Next steps" and "Troubleshooting" sections; verified against /authentication, /errors, /sdks, /architecture

2. Package naming inverts between Python and TypeScript (critical)

Location: /python-conversation, /typescript-conversation, /typescript-sdk, /fastmcp, /anthropic-low-level, /apify

Problem: Naming is inconsistent across languages:

Purpose	Python install	TypeScript install
MCP analytics	`pip install agnost-mcp`	`npm install agnost`
Conversation SDK	`pip install agnost`	`npm install agnostai`

#	Snippet	Status	Notes
1	`pip install agnost-mcp` (Path A FastMCP)	PASS	agnost-mcp 0.1.12 installed
2	Path A Python FastMCP code (`FastMCP("my-server")` + `@server.tool` + `track(...)`)	PASS	Imports/setup ran; SDK logged "MCP server tracking enabled successfully"
3	`pip install agnost-mcp` (Path A Python low-level)	PASS	Same package
4	Path A Python low-level code (`Server("my-server")` + `track(...)`)	PASS	Runs to completion; SDK logs an error "No CallToolRequest handler found" because the snippet uses `# ... your list_tools and call_tool handlers ...` as a placeholder, but the Python script itself does not raise
5	`npm install agnost` (Path A TS)	PASS	agnost 0.1.11 installed
6	Path A TS code (`trackMCP(server, 'your-org-id')`)	SKIP	Partial fragment — `// your existing server setup...` placeholder leaves `server` undefined, snippet throws `ReferenceError` when run verbatim
7	`go get github.com/agnostai/agnost-go/agnost` (Path A Go)	SKIP	Runtime unavailable (no Go) — repo URL resolves on GitHub
8	Path A Go integrate code	SKIP	Runtime unavailable
9	`pip install agnost` (Path B Python)	PASS	agnost 0.1.13 installed
10	Path B Python begin/end code	PASS	Runs; backend rejects placeholder org id with HTTP 400 as expected
11	`npm install agnostai` (Path B TS)	PASS	agnostai 0.1.2 installed
12	Path B TS begin/end code	PASS	Runs; backend rejects placeholder org id with HTTP 400 as expected
13	Path C OTel env vars (`OTEL_EXPORTER_OTLP_ENDPOINT` + `OTEL_EXPORTER_OTLP_HEADERS`)

#	Snippet	Status	Notes
1	`pip install agnost-mcp`	PASS
2	`Server("your-server-name")` + `track(..., config(endpoint=..., disable_input=False, disable_output=False))`	PASS	Runs; same "No CallToolRequest handler found" warning as the quickstart low-level snippet because the snippet shows handlers as a comment placeholder
3	Same with `identify=lambda req, env: {...}` config	PASS	Runs; lambda accepted by `config()`

#	Snippet	Status	Notes
1	`pip install agnost-mcp`	PASS
2	`FastMCP("Your Server Name")` + `track(..., config(...))`	PASS	Tracking enabled successfully (FastMCP without registered tools still patches cleanly)

#	Snippet	Status	Notes
1	`npm install agnost`	PASS
2	`pnpm add agnost`	SKIP	Runtime unavailable (pnpm not installed)
3	`yarn add agnost`	SKIP	Runtime unavailable (yarn not installed)
4	Integrate: `trackMCP(server, 'your-org-id')`	SKIP	Partial fragment — verbatim run throws `ReferenceError: server is not defined`; doc shows `// your existing server setup...`
5	Options: `trackMCP(server, 'your-org-id', { disableInput, disableOutput, identify })`	SKIP	Partial fragment — same undefined `server`
6	Checkpoints: `server.setRequestHandler(CallToolRequestSchema, async (request) => { checkpoint(...); ... })`	SKIP	Partial fragment — uses `server`, `db`, `format` not in scope

#	Snippet	Status	Notes
1	`npm install agnostai`	PASS
2	`pnpm add agnostai`	SKIP	Runtime unavailable
3	`yarn add agnostai`	SKIP	Runtime unavailable
4	`agnost.init('your-org-id')`	PASS
5	`begin({ userId, agentName, input })` + `interaction.end('...')`	PASS	API call shape works; backend returns 400 only because placeholder org id is not a UUID, as the docs warn
6	Error end: `interaction.end('Error: timeout', false)`	PASS
7	Group into conversations via `crypto.randomUUID()` + `conversationId`	PASS
8	`agnost.identify('u-123', { name, email, plan })`	PASS
9	`interaction.setProperties({ model, tokens })`	PASS
10	`agnost.getInteraction('req-1')`	PASS	Returns the begun interaction; `.end('...')` accepted
11	`agnost.init('your-org-id', { endpoint, debug: true })`	PASS
12	`await agnost.shutdown()`	PASS

#	Snippet	Status	Notes
1	`pip install agnost`	PASS
2	`uv add agnost`	SKIP	Setup unavailable (no `pyproject.toml`/uv project)
3	`agnost.init("your-org-id")`	PASS
4	`begin(user_id=, agent_name=, input=)` + `interaction.end(output=)`	PASS	Backend 400 on placeholder org id, as documented
5	Error: `interaction.end(output=..., success=False)`	PASS
6	`agnost.track(user_id=..., input=..., output=..., agent_name=...)`	PASS
7	Conversation grouping with `uuid.uuid4()` + `conversation_id`	PASS
8	`agnost.identify("u-123", {...})`	PASS
9	`interaction.set_property` / `set_properties`	PASS
10	`agnost.init("your-org-id", endpoint=..., debug=True)`	PASS
11	`agnost.shutdown()`	PASS

#	Snippet	Status	Notes
1	Basic `agnost.init("your-org-id")`	PASS
2	`agnost.init("your-org-id", debug=True)`	PASS
3	Lifecycle: `agnost.flush()`, `agnost.shutdown()`, `agnost.set_debug_logs(enabled)`	PASS	All three callable and accept documented signatures

#	Snippet	Status	Notes
1	`go get github.com/agnostai/agnost-go/agnost`	SKIP	Runtime unavailable (no `go` in sandbox); upstream repo at github.com/agnostai/agnost-go returns 200
2	`agnost.Track(s, "your-org-id", &agnost.Config{...})` integrate code	SKIP	Runtime unavailable

#	Snippet	Status	Notes
1	Quick Setup env vars	PASS
2	Step 1 Python install (`pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http`)	PASS
3	Step 1 TS install (`npm install @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto @opentelemetry/api`)	PASS
4	Step 2 Python "append to existing provider"	SKIP	Partial fragment — verbatim run raises `AttributeError: 'ProxyTracerProvider' object has no attribute 'add_span_processor'` because there is no pre-existing TracerProvider; the doc explicitly conditions this on "Already have an OTel TracerProvider?"
5	Step 2 TS "append to existing provider"	SKIP	Partial fragment — uses bare `provider` variable not defined in snippet
6	Step 2 Python full setup (`TracerProvider(resource=Resource.create(...))` + `BatchSpanProcessor(OTLPSpanExporter(...))`)	PASS
7	Step 2 TS full setup (`new NodeSDK({ traceExporter: new OTLPTraceExporter(...) }).start()`)	PASS
8	Step 3 Python `tracer.start_as_current_span("tool.search_web")` with `agnost.user_id` / `agnost.session_id` attrs	PASS
9	Step 3 TS span snippet	SKIP	Partial fragment — uses `query`, `searchWeb` not defined

#	Snippet	Status	Notes
1	`pip install "langsmith[otel]" langchain opentelemetry-sdk opentelemetry-exporter-otlp-proto-http`	PASS
2	Step 2 TracerProvider setup before importing LangChain + `LANGSMITH_OTEL_ENABLED/TRACING/OTEL_ONLY=true` env, then `from langchain_openai import ChatOpenAI`	PASS	All imports and env wiring succeed; `langchain_openai` imports cleanly after the provider is registered
3	Step 3 `agent.invoke({"messages": [...]}, config={"metadata": {...}})`	SKIP	Partial fragment — `agent` not defined in snippet

#	Snippet	Status	Notes
1	`npm install @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto`	PASS
2	Step 2 "append to existing provider" `provider.addSpanProcessor(...)`	SKIP	Partial fragment — `provider` not defined in snippet
3	Step 2 full `new NodeSDK({ traceExporter: new OTLPTraceExporter(...) }).start()`	PASS
4	Step 3 `generateText({ model: openai('gpt-4o'), prompt, experimental_telemetry: { isEnabled, metadata: { userId, sessionId } } })`	SKIP	Partial fragment — snippet imports `generateText` from `'ai'` but uses `openai('gpt-4o')` without importing an `openai` provider; also requires `OPENAI_API_KEY` which is not in scope

#	Snippet	Status	Notes
1	Step 1 Python install	PASS
2	Step 1 TS install	PASS
3	Step 2 Python "append to existing provider"	SKIP	Partial fragment — `os` not imported in this snippet block and no pre-existing provider
4	Step 2 TS "append to existing provider"	SKIP	Partial fragment — bare `provider`
5	Step 2 Python full setup with `OpenAIInstrumentor().instrument(...)`	PASS
6	Step 2 TS full setup with `NodeSDK({ instrumentations: [new OpenAIInstrumentation()] })` + `openAIInstrumentation.manuallyInstrument(OpenAI)`	PASS
7	Step 3 Python `using_attributes(user_id=, session_id=)` block	SKIP	Partial fragment — `client` not defined in snippet (snippet shows the wrapper, not the SDK client construction)
8	Step 3 TS `setUser`/`setSession` + `context.with(ctx, () => client...)`	SKIP	Partial fragment — `client` not defined
9	TS version compatibility table (`openai ^6.7.0` ↔ `@arizeai/openinference-instrumentation-openai ^4.0.0`)	PASS	Installed versions verified to match (`openai@6.37.0` + `@arizeai/openinference-instrumentation-openai@4.1.1`) and import without the documented `APIPromise` ESM error

#	Snippet	Status	Notes
1	`npm install @mastra/observability @mastra/otel-exporter`	PASS
2	Step 2 "append exporter" snippet (`exporters: [..., agnostExporter]` fragment)	SKIP	Partial fragment — shown explicitly as something to splice into an existing config
3	Step 2 full setup `new Mastra({ agents: { agent }, observability: new Observability({ ... }) })`	SKIP	Partial fragment — references `agent` not defined; verbatim run fails inside `Mastra.addAgent`
4	Step 3 per-call `agent.generate('Hello', { tracingOptions: { metadata: { userId, conversationId } } })`	SKIP	Partial fragment — `agent` not defined

#	Snippet	Status	Notes
1	Python install (`pip install agnost-mcp`)	PASS
2	Node install (`npm install agnost`)	PASS
3	Python `async with Actor:` body with `Server("your-mcp-server")` + `track(server, "your-org-id")`	PASS	Module parses; defines `main()` (which the snippet itself does not call)
4	Node `Actor.main(async () => { new Server({...}); trackMCP(server, "your-org-id"); await Actor.getInput(); })`	PASS	Actor boots, agnost SDK initializes, `Actor.main()` exits cleanly

#	Snippet	Status	Notes
1	`./toolbox version`	FAIL	Toolbox 0.28.0 has no `version` subcommand. Running `./toolbox version` exits 1 with no output. Diagnosis: `toolbox --help` lists only `completion`, `help`, `invoke`, `skills-generate`; version is exposed as the flag `--version` (`toolbox version 0.28.0+binary.linux.amd64.81253a0`). Suggested fix in the docs: replace `./toolbox version` with `./toolbox --version`. Not applied — public verifier is report-only.
2	Binary install (`curl -O https://storage.googleapis.com/.../toolbox` + `chmod +x toolbox`)	PASS	URL returns 200, binary downloads and is executable
3	Docker pull (`docker pull us-central1-docker.pkg.dev/database-toolbox/toolbox/toolbox:$VERSION`)	SKIP	Runtime unavailable (no docker)
4	Homebrew (`brew install mcp-toolbox`)	SKIP	Runtime unavailable (no brew)
5	`export OTEL_EXPORTER_OTLP_HEADERS="X-Agnost-Org-ID=<your-agnost-org-id>"`	PASS
6	`./toolbox --tools-file "tools.yaml" --telemetry-otlp="otel.agnost.ai"`	PASS	Flags accepted by 0.28.0; binary exits 0 after logging `ERROR "unable to read tool file at \"tools.yaml\""`, which is the documented user-supplied prerequisite

Check out Manicule.

Agnost AI Documentation Audit

1. Three Quickstart "Next steps" links resolve to 404 (critical)

2. Package naming inverts between Python and TypeScript (critical)

Runtime snippet checks

Prior reports

Sources

Check out Manicule.

Agnost

Agnost AI Documentation Audit

1. Three Quickstart "Next steps" links resolve to 404 (critical)

2. Package naming inverts between Python and TypeScript (critical)

Runtime snippet checks

Prior reports

Prior reports

Sources

3. The REST API reference pages are stubs (critical)

4. llms.txt omits Quickstart, Configuration, and Architecture — including the only page that documents key lifecycle methods (critical)

5. Apify page sends Python users to a Configuration guide that documents a different SDK (significant)

6. OTel endpoint contradicts itself between Quickstart and OTel page (significant)

7. Authentication is referenced everywhere and explained nowhere — and the header name differs across surfaces (significant)

8. "write_key" is mentioned once and used nowhere (significant)

9. MCP Toolbox telemetry endpoint omits the URL scheme (minor)

10. MCP Toolbox install pins a specific version in copy-paste examples (minor)

What they do well

Top 3 recommendations

Summary

Required credentials

Pages

Prior reports

4. `llms.txt` omits Quickstart, Configuration, and Architecture — including the only page that documents key lifecycle methods (critical)