PromptLayer Documentation Audit

The public docs are broad and feature-rich, but the surface has serious hygiene problems: internal-only files are indexed in the public sitemap, the llms-full.txt bundle contains almost nothing usable, multiple pages contradict each other on the SDK API, reference pages disagree on basic conventions (base path, header case), and the navigation links to 404s.

1. Internal-only operations files are published as public docs pages (critical)

Location: https://docs.promptlayer.com/CLAUDE, https://docs.promptlayer.com/DOCS_UPDATER_SETUP_TODO, https://docs.promptlayer.com/TESTING_INSTRUCTIONS — all three are the first three entries in sitemap.xml.

Problem: /CLAUDE is an internal "Documentation Update Policy" describing what changes the docs team should/shouldn't write up. /DOCS_UPDATER_SETUP_TODO is a TODO for installing a GitHub app, generating a PAT, and storing GH_READ_TOKEN / ANTHROPIC_API_KEY secrets. /TESTING_INSTRUCTIONS is a test workspace setup guide referencing private prompt templates (multi-turn-assistant, multi-turn-assistant-with-tools) and the internal testing/ folder. These are the top three URLs in the production sitemap.

Consequence: Every search engine and every AI agent that crawls the sitemap indexes operational/internal docs as first-class PromptLayer content. Agents like Claude/Cursor that pull from llms-full.txt will literally surface "here's how to set up our docs-updater GitHub workflow" as if it were product documentation. It's also a minor disclosure issue (internal repo names, secret variable names, internal API routes like /api/dashboard/v2/*).

The fix: Move these three files out of docs/ (they are repo READMEs, not docs pages), remove them from sitemap.xml, and add a Mintlify ignore / noindex rule so any future *.md at the repo root doesn't get auto-published.

2. llms-full.txt contains only those internal files — not the actual docs (critical)

Location: https://docs.promptlayer.com/llms-full.txt (the "full" agent-facing bundle exposed at the docs root)

Problem: The advertised "full" bundle includes only five sections: CLAUDE.md, DOCS_UPDATER_SETUP_TODO, TESTING_INSTRUCTIONS, MCP, and Custom Providers. There is no Quickstart, no SDK reference, no API reference, no Prompt Registry, no Evaluations, no Webhooks, no OpenTelemetry — nothing an agent would actually need to integrate PromptLayer. (The docs root links to llms.txt as the "Complete documentation index"; llms-full.txt is the sibling "full" bundle and is the one that is broken.)

Consequence: PromptLayer is itself a platform for AI agents and LLM workflows — llms-full.txt is one of the first things a coding agent will fetch when a developer says "integrate PromptLayer." What the agent gets back is the internal docs-update policy and a testing TODO. The agent will then either hallucinate the SDK or give up. This single file undermines every other piece of documentation on the site.

The fix: Regenerate llms-full.txt from the full sitemap (excluding the three internal files above). At minimum it should contain Quickstart, both SDK pages, the REST reference, Prompt Registry, Webhooks, OpenTelemetry, Workflows, Evaluations overview, and the FAQ.

3. Two different SDK APIs documented for fetching a labeled prompt (significant)

Location: /features/prompt-registry/release-labels vs /sdks/python, /sdks/javascript, and /features/faq

Problem: The release-labels page tells developers to call promptlayer_client.templates.get("my_template", { "label": "prod" }) (and the equivalent JS). The Python and JavaScript SDK pages document a different shape — pl.run(..., prompt_release_label="prod") in Python and promptReleaseLabel in JS — and the FAQ also directs users to run(). Both forms are shown as canonical, on different pages, without any note that one supersedes the other.

Consequence: Developers (and agents) copying from the release-labels page get a different shape of API than the one the rest of the docs teach. There's no migration guidance, no deprecation banner, and no cross-link — so it looks like two valid APIs that take different parameter names (label vs prompt_release_label) for the same concept. Agents pattern-matching on the docs will pick whichever page they saw last.

The fix: Pick one (presumably pl.run(..., prompt_release_label=...)) and either rewrite the release-labels page to match, or add a clearly marked "legacy API" banner at the top with a link to the current SDK call site.

4. Inconsistent REST base path between docs pages (significant)

Location: /features/prompt-history/metadata vs /reference/introduction / /features/opentelemetry

Problem: The metadata page documents the REST endpoint as https://api.promptlayer.com/rest/track-metadata. Other parts of the docs (OpenTelemetry at https://api.promptlayer.com/v1/traces, the reference introduction, and the internal-policy file calling out "Public APIs (/api/v1/*)") use /v1/ or /rest/ interchangeably with no explanation of which base path applies to which family of endpoints.

Consequence: Developers writing direct HTTP integrations (especially anyone bypassing the SDK) cannot tell from the docs whether the base path is /rest/..., /v1/..., or /api/v1/.... There is no single "API base URL" statement in /reference/introduction. Agents trying to assemble request URLs will silently 404.

The fix: Add a "Base URL" section to /reference/introduction enumerating each path prefix (/rest/*, /v1/*) and which endpoints live under each, then update every reference page to use the same prefix it documents.

5. Auth header is spelled two different ways across the REST reference (significant)

Location: /reference/introduction (uses X-API-Key) vs /reference/log-request, /reference/templates-get, and /features/opentelemetry (all use X-API-KEY)

Problem: The reference introduction states authentication is via the X-API-Key header (mixed case). Every other reference page and the OTLP integration page write the header as X-API-KEY (all caps). HTTP headers are case-insensitive on the wire, but the docs disagree with themselves on the canonical spelling.

Consequence: Developers using strict client-code generators (OpenAPI, gRPC-gateway, code-gen tools that key off exact header names) get two different "official" spellings. Anyone grepping examples or building a typed wrapper around the headers will end up with two constants for one header. It also signals that no one is reviewing reference pages against each other.

The fix: Pick a single canonical spelling (most pages use X-API-KEY) and replace all occurrences. Update /reference/introduction to match.

6. Navigation and tutorial-videos page link to URLs that 404 or are missing from sitemap (significant)

Location: /tutorial-videos, /features/custom-providers, and /why-promptlayer/playground

Problem: /tutorial-videos lists four "Additional Resources" links — /onboarding-guides/prompt-management, /onboarding-guides/evaluation, /onboarding-guides/agentic-workflows, /onboarding-guides/observability. None of those four URLs appear in the published sitemap.xml. Some (e.g. /onboarding-guides/observability, /onboarding-guides/prompt-management) do resolve, but /onboarding-guides/evaluation and /onboarding-guides/agentic-workflows are not discoverable from the sitemap and may have been renamed. Separately, the sitemap contains /features/prompt-registry/custom-providers and /features/prompt-registry/playground even though the canonical pages live at /features/custom-providers and /why-promptlayer/playground — so the sitemap is shipping at least two stale paths.

Consequence: New users following the tutorial-videos "next steps" land on broken or stale pages. Search engines and agents indexing from the sitemap try to fetch the stale prompt-registry paths and get 404s, lowering trust in the full index.

The fix: Audit the sitemap against actually published pages, remove the stale /features/prompt-registry/custom-providers and /features/prompt-registry/playground entries, and either move the onboarding-guides pages under the sitemap or update the /tutorial-videos links to the current /features/... locations.

7. Mintlify component markup leaks into agent-facing content (significant)

Location: /features/evaluations/datasets-overview

Problem: The datasets overview page emits raw <CardGroup cols={2}> and <Card title="..." icon="..." href="..."> JSX into what is otherwise prose. This renders fine in the browser but leaks unprocessed markup into the text that agents and llms-full.txt-style aggregators see.

Consequence: Agents pulling the page get JSX literals instead of clean links/titles, which degrades summarization and link extraction.

The fix: Render the CardGroup section out to plain markdown links before serving the page to llms.txt consumers, or include a plain markdown list of the same "Next steps" links immediately below the CardGroup.

8. Parallel datasets pages create an ambiguous canonical URL (significant)

Location: /features/evaluations/datasets and /features/evaluations/datasets-overview — both present in the sitemap.

Problem: Two separate pages exist for the same concept ("Datasets" the feature). Both are indexed and reachable; neither declares itself canonical, and neither links to or supersedes the other. The -overview variant is the one with the JSX CardGroup (Finding 7), suggesting it is the newer rewrite — but the older /datasets URL is still live.

Consequence: Search results and agent indexes have two competing pages for the same topic. Internal links from other pages point at one or the other inconsistently. Developers don't know which is authoritative.

The fix: Pick one canonical page, 301-redirect the other, and update all in-docs links and the sitemap to point only at the survivor.

9. /overview is a near-duplicate of the docs root (significant)

Location: /overview vs the docs root /

Problem: The sidebar-listed /overview page restates the same five feature pillars (Registry, Evaluations, Dataset Management, Observability, Workflows) that the docs landing page already presents. There is no additional content on /overview that the root doesn't already carry.

Consequence: Same-content-at-two-URLs SEO and canonicalization problem. Agents indexing the site will see the same five-pillar paragraph twice and may pick whichever they land on first as the "overview." Sidebar links from within the docs that go to /overview instead of the root produce a confusing back-and-forth.

The fix: Either delete /overview and redirect it to the root, or differentiate it (turn it into a true product-tour page with depth the landing page doesn't have).

10. RBAC page claims 18 permissions but enumerates none (significant)

Location: /why-promptlayer/rbac

Problem: The page states the system includes "18 granular permissions spanning" six categories (prompt operations, workflow management, dataset handling, report functionality, metadata editing, API key management, administrative access) but never lists the 18 by name. There is no permissions matrix mapping each default role (Contributor, Publisher, Developer, Admin) to which permissions it actually grants.

Consequence: Security/compliance reviewers cannot answer "what exactly can a Contributor do vs a Publisher?" without contacting sales. For an enterprise feature gated on SOC 2 / SSO conversations, this is the page that gets read by procurement, and it doesn't contain the data needed.

The fix: Publish the full 18-permission list and a role-permission matrix (table of role × permission with ✓/✗). This is data the product already exposes in-app.

11. No documented error/status taxonomy in the REST reference outside one endpoint (significant)

Location: /reference/introduction and the rest of /reference/*

Problem: The only place HTTP status codes and error semantics are enumerated is /reference/log-request, which lists PROVIDER_TIMEOUT, PROVIDER_QUOTA_LIMIT, PROVIDER_RATE_LIMIT, TEMPLATE_RENDER_ERROR, VARIABLE_MISSING_OR_EMPTY, etc. There is no global error-code reference, no rate-limit documentation, and /reference/templates-get documents only 200 and 422. The reference introduction does not describe what 401/403/429 look like, what Retry-After is sent, or how quota errors are signaled outside the log-request payload.

Consequence: Developers writing retry logic or building dashboards on top of PromptLayer cannot tell from the docs which errors are retryable, which are fatal, or what rate limits to expect. The error vocabulary that does exist is buried inside one logging endpoint, which makes it look like an enum for that endpoint rather than the platform's error taxonomy.

The fix: Add a /reference/errors page enumerating HTTP status codes, the canonical error payload shape, the error_type enum, and rate-limit headers. Cross-link from /reference/introduction.

12. "Get prompt template" uses POST without explanation (minor)

Location: /reference/templates-get

Problem: The endpoint named "Get a Prompt Template" is documented as POST /prompt-templates/{identifier}. The page does not explain why a read operation uses POST (presumably because the call accepts a GetPromptTemplate request body with version, label, input_variables, metadata_filters, etc.). There is no note in /reference/introduction describing this convention either.

Consequence: Developers generating OpenAPI clients will classify this as a mutation. Caching layers and observability tools that key off HTTP verb will treat it as non-idempotent. Anyone scanning the API for "what's read-only?" gets a misleading answer.

The fix: Add a one-line note on the page (and in the reference introduction) explaining that some "read" endpoints use POST because they accept structured filter bodies, and confirm whether they are idempotent and cacheable.

13. "Custom Logging documentation" referenced without a link (minor)

Location: /migration

Problem: The migration page tells developers to see the "Custom Logging documentation" for "more details including format converters for OpenAI and Anthropic" but doesn't link to it. There is no page at /custom-logging in the sitemap; the closest matches are /reference/log-request and the SDK pages, but the docs don't say so.

Consequence: Users following the log_request path (specifically those migrating from an existing LLM client — the exact audience this page targets) hit a dead-end reference. Agents will not be able to resolve the link target.

The fix: Either create a /features/custom-logging page consolidating the log_request walkthrough and format converters, or replace the prose reference with a direct link to /reference/log-request and the SDK sections.

14. /running-requests/promptlayer-run is an empty stub (minor)

Location: /running-requests/promptlayer-run

Problem: The sitemap-listed page exists only to say "the run() method details have been relocated to SDK-specific guides" and link to /sdks/python and /sdks/javascript. It contains no content of its own.

Consequence: Search ranking goes to the empty redirect page, not the real SDK page. Developers (and the OG-search of any agent) land on a content-free shell.

The fix: Either redirect /running-requests/promptlayer-run server-side to /sdks/python#using-the-run-method-recommended, or fold the SDK examples back into the page so it can stand alone.

What they do well

• Strong machine-readable surface for AI agents: hosted MCP server at mcp.promptlayer.com/mcp with 60 tools, OTLP endpoint with documented GenAI semantic conventions, OpenAI-compatible custom-provider story.
• /reference/log-request is the strongest reference page on the site — it documents content-array shape, structured outputs, tool-choice variants, reasoning config per provider, and a clear error-type enum.
• The Prompt Registry / release-labels / dynamic-release-labels chain explains a non-trivial deployment pattern with concrete percentages and a useful caveat about logging the resolved version rather than the label.

Top 3 recommendations

1. Fix the agent-facing index. Remove /CLAUDE, /DOCS_UPDATER_SETUP_TODO, /TESTING_INSTRUCTIONS from the public sitemap and regenerate llms-full.txt so it actually contains the SDK, REST, Registry, and OTel docs.
2. Pick one SDK API for labeled retrieval and reconcile release-labels with the SDK pages and FAQ. Add a deprecation banner on whichever form is legacy.
3. Publish a real errors + rate-limit reference, resolve the /rest/ vs /v1/ base-path inconsistency in a single Base URL section on /reference/introduction, and normalize the X-API-Key / X-API-KEY header spelling across all reference pages.