The docs are well-structured for agents (llms.txt, page-level .md mirrors, MCP server, OpenAPI spec) but suffer from contradictions between the API reference and the feature guides, a Python SDK that's named two different things on two different pages, and a self-serve flow that quietly requires emailing support.
1. rerank default contradicts itself across pages (critical)
Location:/guides/get-started/multimodal-search and /api-reference/api-reference/query/collection-v-2
Problem: The multimodal search guide's request-fields table sets rerank default to true: "rerank | boolean | true | Enable reranking. Required when multimodal content is present". The Query API reference table sets the same parameter's default to false: "rerank | boolean | false | Improve relevance ordering". These are the two canonical reference tables for the same parameter on the same endpoint.
Consequence: A developer (or coding agent) reading the API reference assumes rerank is off and omits it from their request body, then gets surprise reranker billing or unexplained latency. A developer reading the multimodal guide assumes rerank is on by default and is shocked when their first text-only call returns un-reranked results. Agents can't infer which doc is canonical and will silently pick the wrong one.
The fix: Pick one default in code and update both tables to match. If multimodal requests truly require rerank=true, document that as a precondition rather than a default-flip per content type.
2. Python SDK has two different names across docs (critical)
Location:/guides/get-started/introduction and /api-reference/api-reference/indexing/parsing-scripts/validate-parsing-script-v-2
Problem: The introduction's Quick Reference says: "SDKs: Python (captain-sdk on PyPI), TypeScript (captain-sdk on npm)". The parsing-script validate page's Python example imports a different package and class: from runcaptain import Captain / client = Captain(key="YOUR_TOKEN_HERE"). Meanwhile the S3 TypeScript example uses import { CaptainClient } from "captain-sdk" — so the TS side at least matches the intro.
Consequence: A developer copy-pasting the validate-script example runs pip install captain-sdk (per the intro), then gets . Or they trust the example, run , and now have an SDK whose name doesn't match anything else in the docs. Agents using the docs as ground truth will emit broken installs either way.
The fix: Settle on one package name, audit every code sample for the right import, and add a one-line install command at the top of every code block (e.g. # pip install captain-sdk).
3. Self-serve flow requires emailing support for an External ID (critical)
Problem: Step 1 of cross-account IAM setup is: "Email support@runcaptain.com with your Captain organization ID, or ask your account manager directly … We'll reply with a value of the form cap_org_<24-hex>, usually same-day. Save it; you'll paste it into your IAM trust policy in step 2." The External ID is "NOT customer-generated. Captain mints it server-side per organization."
Consequence: Cross-account IAM is the AWS-recommended way to grant access to a third party — and the docs gate it behind a same-day email round-trip. A developer evaluating Runcaptain on a Friday afternoon can't finish setup until Monday. Agents driving onboarding can't proceed without human intervention, which breaks every "from spec to running pipeline" use case Runcaptain pitches.
The fix: Expose External ID via the dashboard (Settings → Organization → IAM External ID) or via an API endpoint (GET /v2/organizations/me). Email remains a fallback, not the primary path.
Problem: Fetching /welcome returns "Page Not Found — This page does not exist." The llms-full.txt manifest and the navigation refer to "Welcome Aboard" as the docs landing page, but the canonical URL is /guides/get-started/introduction. There's no redirect from /welcome.
Consequence: The friendliest, most-guessable entry URL for a docs site dead-ends. Any external link, tweet, or onboarding email pointing at docs.runcaptain.com/welcome (a natural guess) breaks. Agents crawling guessable paths waste a request and may treat the 404 as "no docs at this site."
The fix: Redirect /welcome to /guides/get-started/introduction (301), or render the introduction at both paths.
Location:/api-reference/api-reference/indexing/parsing-scripts/validate-parsing-script-v-2 (and its OpenAPI definition)
Problem: The parameters block for POST /v2/parsing-scripts/validate defines Authorization as an in: header parameter twice, with two different descriptions:
- name: Authorization
in: header
description: Bearer token authentication using API key
required: true
...
- name: Authorization
in: header
description: Bearer token - your Captain API key.
required: true
Consequence: OpenAPI 3.1 requires parameter name + in to be unique. Spec validators (Spectral, Redocly lint, openapi-generator) error on this, and code generators may produce broken clients that send the Authorization header twice or skip the operation. Anyone running the published spec through CI gets a failure.
The fix: Remove the duplicate. Define Authorization once in a shared securitySchemes block and reference it via security: so it can't drift again.
6. Response field for AI answers has two names (significant)
Location:/api-reference/api-reference/query/collection-v-2 vs /guides/get-started/multimodal-search
Problem: The Query API reference lists the AI answer under "summary / response: AI response (when inference=true)" — two field names for the same payload, with no explanation of which one clients should read. The multimodal search agent quick-reference lists per-result fields (score, content, document_id, filename, uri, modality, media_segment_start_sec, media_segment_end_sec, rerank_score, metadata) but never mentions either summary or response, leaving the answer-extraction path undefined for multimodal+inference queries.
Consequence: A developer writes result.summary and an agent's parser breaks the day the server returns result.response (or vice versa). For multimodal+inference calls — Runcaptain's headline feature — there's no documented field to read the LLM answer from.
The fix: Commit to one canonical field name in the response schema. If both names are returned for backwards compatibility, say so explicitly and mark one deprecated with a removal date.
7. Changelog advertises endpoints and protocols that aren't documented anywhere else (significant)
Location:/changelog/changelog (October and November 2025 entries)
Problem: The October 2025 changelog entry advertises "OpenAI SDK-compatible chat completions endpoint (/v1/chat/completions)" and "Vercel AI SDK support added." The November 2025 entry advertises "Tool calling support — OpenAI-compatible function calling with client-side execution." The introduction's endpoint inventory lists only /v2/... endpoints. There is no API reference page, no auth guide, no rate-limit doc, no model list, and no tool-schema reference for any of these features. They're mentioned in the changelog and nowhere else in the scraped corpus.
Consequence: Teams choosing Runcaptain for OpenAI-SDK drop-in compatibility or function calling — both explicitly advertised — have no documented schema, no auth example, no deviation list from the OpenAI spec, no error reference. They have to reverse-engineer behavior from OpenAI's docs and hope the server matches. For an agent-targeted product, undocumented agent surface area is the worst kind of gap.
The fix: Publish reference pages for /v1/chat/completions (request schema, supported models, deviations from OpenAI's spec, streaming behavior) and the tool-calling protocol (tool-schema shape, execution model, client-side vs server-side semantics). If either was rolled back, remove the claim from the changelog.
8. Scientific-medical page contradicts itself on client timeout (significant)
Problem: The Agent Quick Reference says "p50 ~8s, p95 ~12s. Set client timeouts ≥ 90s." The Limits section repeats "Set client timeouts ≥ 90s." But the body of the "Asking a Question" section says "First invocation of the day can hit a cold Lambda (~60s) + agent loop (~30-60s). set client timeouts to 180s and guard against non-JSON responses (504/502) so errors surface cleanly." 90s vs 180s on the same page, for the same endpoint, with the 180s figure explicitly justified by the documented worst-case cold-start.
Consequence: An agent that reads only the Quick Reference (which is what the docs train agents to do) sets a 90s timeout and gets a hard failure on the very cold-start path the docs warn about elsewhere. A human reader scrolls to the next section and gets a different answer with no indication which is canonical.
The fix: Pick one. If 180s is the safe upper bound, put 180s in the Quick Reference and Limits and explain that 90s works once Lambda is warm. If 90s is the right floor for the steady state, move the 180s cold-start guidance into a separate "First request" callout.
9. search_results is both a request parameter and a response field with different types (significant)
Problem: On the same Query API reference page, search_results appears in the request parameters table as search_results | boolean | false | Include context chunks with inference response, and then again in the Response Structure block as search_results: Array of SearchResult objects. Same name, two different types (boolean in, array out), no note connecting the two.
Consequence: A developer or agent typing response.search_results against a TypeScript type generated from the request schema gets a boolean and a type error. A developer setting search_results: true to opt-in to chunks has no documented guarantee about what comes back when the value is omitted vs. false. Code generators may collapse the two into a single conflicting field.
The fix: Rename one of them — e.g. include_search_results: boolean for the request flag — or explicitly document that the request boolean toggles the response array, with a worked example of the response shape under both values.
Problem: The "Filter Operators" table lists $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin — but not $and, $or, or $not. Later sections demonstrate $or ("Use $or to match chunks that satisfy any of the conditions") and "Implicit AND," so the operators clearly exist, but they're not in the reference table.
Consequence: An agent generating filter expressions from the reference table won't emit $or because it doesn't appear in the canonical operator list. A developer scanning the table for "how do I OR" assumes it isn't supported and either drops the query or runs two queries client-side. Anyone Ctrl-F'ing the page for $or finds it only in a usage example, not in a contract.
The fix: Add $and, $or, $not rows to the operator table with example syntax, nesting rules, and depth limits.
11. TypeScript S3 IAM-role example is missing the auth block (significant)
There is no field identifying which role ARN to assume, no auth / role_arn / external_id block, no reference to the cross-account setup from the companion guide. The label promises "IAM role" but the body looks identical to a public-bucket call.
Consequence: A developer copies this snippet expecting IAM-role auth, runs it against a private bucket, and gets a generic 403. They can't tell whether the SDK silently picked up ambient AWS credentials, whether a field is missing from the request, or whether IAM-role mode isn't actually wired up. Agents can't synthesize a working call from this example.
The fix: Show the full request body for IAM-role mode, including roleArn and externalId, and cross-link to /guides/get-started/s-3-cross-account-iam from the example.
12. Live Search advertises 10 sources but documents only one (significant)
Location:/guides/integrations/overview vs /guides/integrations/live-search-guides/*
Problem: The integrations overview lists Live Search sources as "Snowflake (SQL queries), Slack, Linear, Jira, Confluence, Asana, Gmail, Google Calendar, HubSpot, and Oracle NetSuite." Only one of those — Oracle NetSuite — has a dedicated guide in the scraped corpus. The other nine have no setup, auth, scope, or rate-limit documentation.
Consequence: A buyer comparing connectors thinks Runcaptain has nine more integrations than it has documentation for. Developers wiring up Slack or Jira get the connector name from marketing and then have nowhere to read about OAuth scopes, refresh behavior, or query semantics.
The fix: Ship a per-connector page for each Live Search source with the same structure as the NetSuite guide (authentication method, setup steps, supported query types, known limits). Until then, mark undocumented connectors as "Studio-only" or "private beta."
13. Cloud-storage setup has screenshots only for AWS (minor)
Problem: The cloud-storage page hard-codes image URLs to Vercel Blob (https://y2i6auvwuaxlu05f.public.blob.vercel-storage.com/aws_image.png, aws_image2.png, google_image.png). AWS gets two screenshots; GCS gets one labeled "Service Accounts" with no follow-up; Azure and R2 — both listed as first-class integrations — get no visuals at all.
Consequence: AWS users get a guided console walkthrough; everyone else gets prose-only setup against unfamiliar console UIs. Azure's "generate a SAS token from the Azure Portal" instruction is hand-waved past the click-path that actually matters.
The fix: Add parity screenshots for GCS service-account key creation, Azure SAS generation, and R2 token creation. Move images to a stable CDN under the runcaptain.com domain so a Vercel Blob URL rotation doesn't silently 404 every screenshot.
14. Changelog claims monthly cadence and is on track to miss May 2026 (minor)
Location:/changelog/changelog
Problem: The changelog header states "Monthly updates to the Captain API — new features, improvements, and fixes." As of 2026-05-23, the most recent entry is April 2026. The month is not over yet — but with one week left and no entry, May is on track to be the first missing month under the stated cadence, and there's no signal to the reader about whether something is coming or whether the cadence promise has lapsed.
Consequence: Developers can't tell whether (a) nothing has shipped in May, (b) something shipped and the changelog is behind, or (c) the changelog is no longer maintained. For a product that bills its docs as agent-friendly, the absence of a "nothing this month" note is a silent staleness signal.
The fix: Either post a May entry before month-end (even a short "no user-facing changes" note) or rephrase the header to drop the monthly promise and describe the cadence honestly.
What they do well
Strong agent-parsing surface:llms.txt, llms-full.txt, per-page .md mirrors, an MCP server at /_mcp/server, and "append .md to any page" — this is one of the more agent-aware docs sites in the category.
Per-page Agent Quick Reference blocks front-load the key constraints (endpoint, required params, hard limits) so an agent doesn't have to read 800 words of prose to call the API.
Honest limits sections: The parsers page lists sandbox restrictions (no require, no fetch, no setTimeout, 64 MB heap, 10s timeout) and the scientific dataset page documents the 8-call tool-call cap and p95 latency — both rare and useful.
Top 3 recommendations
Fix the contradictions first.rerank default (true vs false), Python SDK name (captain-sdk vs runcaptain), AI-response field name (summary vs response), and the 90s-vs-180s client timeout are all single-line changes that affect every agent-generated call.
Document /v1/chat/completions, the Vercel AI SDK surface, and tool calling — or retract the claims. OpenAI-SDK compatibility and function calling are serious selling points and currently exist only as changelog bullets.
Eliminate the email-for-External-ID handoff. Surface it in the dashboard or an API endpoint so AWS cross-account IAM setup is fully self-serve.
Executed the public Captain API documentation snippets for three in-scope pages (create-collection, plain-text indexing with metadata, and query). Of 13 documented snippets I attempted, 8 pass against the live https://api.runcaptain.com API using the supplied development credentials, 4 fail, and 1 is skipped because no Go runtime is available in this sandbox.
Both FAIL categories reproduce across two pages and share clear root causes:
Python SDK examples (from runcaptain import Captain … Captain(key="…")) fail with TypeError: __init__() missing 1 required keyword-only argument: 'organization_id'. The installed PyPI package captain-sdk (version 0.1.1, which exposes the runcaptain module) requires organization_id as a mandatory keyword-only argument, but every documented Python SDK snippet omits it. Suggested fix the docs could adopt: add organization_id="<org-id>" to each Captain(...) constructor example, since the API key alone is no longer sufficient to instantiate the SDK.
TypeScript SDK examples (import { CaptainClient } from "captain-sdk") fail with ERR_MODULE_NOT_FOUND because the npm package captain-sdk does not exist on the npm registry (404). The documentation's stated SDK distribution channel (captain-sdk on npm) is wrong; only @captain-sdk/captain-mcp is published under that npm scope. Suggested fix: update the snippets and the introduction page to reflect whatever the actual published npm package name is, or publish the documented captain-sdk package.
The "with reranking" query path works in every documented form (Python REST non-streaming, Python REST streaming SSE, TypeScript REST non-streaming, TypeScript REST streaming SSE). The "without reranking" case requested in the credentials note has no verbatim documented snippet on the in-scope pages — every documented Captain query example sets rerank: true — so it could not be verified from documented code without modifying a snippet.
The index/text snippets all return {job_id, status:"pending"} as documented, but they require a pre-existing collection: running the index snippet against my_collection before that collection was created returns HTTP 404 with "Collection 'my_collection' not found … Create it first using PUT /v2/collections/my_collection". Whether this counts as a docs gap depends on the page's surrounding instructions; the in-scope metadata-filtering.md page does not show the prerequisite PUT /v2/collections/my_collection call before its index/text example.
Required credentials
CAPTAIN_API_KEY — Captain API key, supplied by the user as a cap_dev_* development key, passed in Authorization: Bearer …. All endpoints exercised resolved the calling organization from the API key alone, so the X-[REDACTED]-ID header was not required for any documented snippet.
CAPTAIN_ORG_ID — supplied by the user but unused by any of the documented snippets that were executed (the Python SDK snippets that should have used it are exactly the ones that fail because the SDK constructor demands it).
PUT /v2/collections/my_documents returned HTTP 201 with the expected collection_name, collection_id, organization_id, created_at shape. Re-run returned HTTP 200 ("already exists"), matching the page's documented idempotency contract.
A2
python (SDK runcaptain)
FAIL
Captain(key=…) raises TypeError: __init__() missing 1 required keyword-only argument: 'organization_id'. Diagnosis: the published captain-sdk 0.1.1 SDK requires organization_id as a mandatory keyword-only arg; the documented snippet omits it. Suggested fix: add organization_id="<your-org-id>" to the Captain(...) example.
A3
typescript (REST)
PASS
PUT /v2/collections/my_documents via fetch returned HTTP 200 with the documented JSON body. Snippet runs as-is under Node 24 (no TS-specific syntax).
A4
typescript (SDK captain-sdk)
FAIL
import { CaptainClient } from "captain-sdk" raises ERR_MODULE_NOT_FOUND because no package named captain-sdk exists on the npm registry (404 from npm view captain-sdk). Diagnosis: the documented npm SDK distribution channel is incorrect. Suggested fix: update the snippet (and the "SDKs" line in the Quick Reference) to the actual published package name, or publish a captain-sdk package matching the docs.
python (REST, "Plain text" index/text with custom_metadata)
PASS
After pre-creating my_collection (the page does not show this prerequisite), the snippet returned HTTP 200 with {"job_id": "…", "status": "pending"} as documented. Note: against a non-existent collection the snippet returns HTTP 404 with "Collection 'my_collection' not found … Create it first using PUT /v2/collections/my_collection" — a setup gap on the page, not a snippet bug.
B2
typescript (REST, same "Plain text" example)
PASS
Same behavior as B1 via fetch under Node 24. HTTP 200 with {job_id, status:"pending"}.
Returned HTTP 200 with the documented QueryResponseV2 shape: success, inference, search_results, total_results, top_k, query, tokens_used, execution_time_ms, request_id. search_results was empty because my_documents had no indexed content yet; the response envelope matched the docs exactly. This is the documented "with reranking" path.
C2
python (SDK runcaptain, streaming inference)
FAIL
Same root cause as A2: Captain(key=…) raises TypeError: missing organization_id. Suggested fix identical to A2 — include organization_id= in the SDK example.
C3
python (REST, SSE streaming, rerank: true)
PASS
SSE stream opened, emitted the documented stream_complete event, and exited cleanly. No text.delta events because the collection had no relevant content; the event-type contract matched the docs.
C4
typescript (REST, non-streaming, rerank: true)
PASS
fetch POST returned HTTP 200 and the loop over result.search_results ran without error (zero results, as expected for an empty collection).
C5
typescript (REST, SSE streaming, rerank: true)
PASS
Streamed via the documented response.body!.getReader() + TextDecoder loop. Ran verbatim under node --experimental-strip-types (the ! non-null assertion is preserved). Emitted stream_complete as documented.
C6
typescript (SDK captain-sdk, streaming)
FAIL
Same root cause as A4: import { CaptainClient } from "captain-sdk" resolves to no npm package (ERR_MODULE_NOT_FOUND). Suggested fix identical to A4.
C7
go (non-streaming, net/http)
SKIP
Runtime unavailable: is not installed in this sandbox. Snippet not executed.
Note on the credentials-note request to also verify "querying without reranking": none of the in-scope query snippets set rerank: false or omit rerank. Verifying that path would require modifying a documented snippet, which this verifier does not do.