PromptLayer

PromptLayer Documentation Audit

The public docs are broad and feature-rich, but the surface has serious hygiene problems: internal-only files are indexed in the public sitemap, the llms-full.txt bundle contains almost nothing usable, multiple pages contradict each other on the SDK API, reference pages disagree on basic conventions (base path, header case), and the navigation links to 404s.

1. Internal-only operations files are published as public docs pages (critical)

Location: https://docs.promptlayer.com/CLAUDE, https://docs.promptlayer.com/DOCS_UPDATER_SETUP_TODO, https://docs.promptlayer.com/TESTING_INSTRUCTIONS — all three are the first three entries in sitemap.xml.

Problem: /CLAUDE is an internal "Documentation Update Policy" describing what changes the docs team should/shouldn't write up. /DOCS_UPDATER_SETUP_TODO is a TODO for installing a GitHub app, generating a PAT, and storing GH_READ_TOKEN / ANTHROPIC_API_KEY secrets. /TESTING_INSTRUCTIONS is a test workspace setup guide referencing private prompt templates (multi-turn-assistant, multi-turn-assistant-with-tools) and the internal testing/ folder. These are the top three URLs in the production sitemap.

Consequence: Every search engine and every AI agent that crawls the sitemap indexes operational/internal docs as first-class PromptLayer content. Agents like Claude/Cursor that pull from llms-full.txt will literally surface "here's how to set up our docs-updater GitHub workflow" as if it were product documentation. It's also a minor disclosure issue (internal repo names, secret variable names, internal API routes like /api/dashboard/v2/*).

The fix: Move these three files out of docs/ (they are repo READMEs, not docs pages), remove them from sitemap.xml, and add a Mintlify ignore / noindex rule so any future *.md at the repo root doesn't get auto-published.

2. `llms-full.txt` contains only those internal files — not the actual docs (critical)

Location: https://docs.promptlayer.com/llms-full.txt (the "full" agent-facing bundle exposed at the docs root)

Problem: The advertised "full" bundle includes only five sections: , , , , and . There is no Quickstart, no SDK reference, no API reference, no Prompt Registry, no Evaluations, no Webhooks, no OpenTelemetry — nothing an agent would actually need to integrate PromptLayer. (The docs root links to as the "Complete documentation index"; is the sibling "full" bundle and is the one that is broken.)

PromptLayer Documentation Audit

1. Internal-only operations files are published as public docs pages (critical)

2. `llms-full.txt` contains only those internal files — not the actual docs (critical)

Location: https://docs.promptlayer.com/llms-full.txt (the "full" agent-facing bundle exposed at the docs root)

#	Lang	Status	Notes
1	bash	PASS	`pip install promptlayer` already satisfied.
2	bash	PASS	`npm install promptlayer` installs cleanly.
3	bash	PASS	`export PROMPTLAYER_API_KEY=... && export OPENAI_API_KEY=...` runs.
4	python	SKIP	`pl.run(prompt_name="cake-recipe", ...)` — prompt template `cake-recipe` not in workspace; also requires `OPENAI_API_KEY`.
5	js	SKIP	Same — `cake-recipe` not in workspace; requires `OPENAI_API_KEY`.

#	Lang	Status	Notes
1	bash	PASS	`pip install promptlayer`.
2	python	SKIP	`pl.run("my-prompt", ...)` — prompt missing + no `OPENAI_API_KEY`.
3	python	SKIP	`pl.run("your-prompt-name", ...)` — same.
4	python	SKIP	Streaming `pl.run("your-prompt", stream=True)` — same.
5	python	SKIP	partial fragment — kwargs-only `version`/`label` override on undefined `pl`.
6	python	SKIP	partial fragment — tags/metadata kwargs only.
7	python	SKIP	partial fragment — provider/model kwargs only.
8	python	SKIP	`run_workflow("Data Analysis Workflow", ...)` — workflow missing.
9	python	SKIP	Advanced `run_workflow(...)` — workflow missing.
10	python	PASS	`PromptLayer(api_key=..., cache_ttl_seconds=300)` initializes.
11	python	PASS	`AsyncPromptLayer(api_key=..., cache_ttl_seconds=300)` initializes.
12	python	SKIP	Custom logging via OpenAI — needs `openai` SDK + `OPENAI_API_KEY`.
13	python	SKIP	Custom logging via Anthropic — needs `anthropic` SDK + `ANTHROPIC_API_KEY`.
14	python	PASS	All 8 listed exception classes import cleanly.
15	python	PASS	`throw_on_error=True/False` constructor switch.
16	python	PASS	`templates.get("NonExistentTemplate")` raises `PromptLayerNotFoundError` exactly as documented.

#	Lang	Status	Notes
1	bash	PASS	`npm install promptlayer`.
2	js	SKIP	`run({promptName:"my-prompt",...})` — prompt missing + no `OPENAI_API_KEY`.
3	js	SKIP	`run({promptName:"your-prompt-name",...})` — same.
4	js	SKIP	Streaming `run({promptName:"your-prompt", stream:true})` — same.
5	js	SKIP	partial fragment — only override kwargs.
6	js	SKIP	partial fragment — tags/metadata kwargs only.
7	js	SKIP	partial fragment — provider/model kwargs only.
8	js	SKIP	`runWorkflow({workflowName:"Data Analysis Workflow",...})` — workflow missing.
9	js	SKIP	Advanced `runWorkflow(...)` — workflow missing.
10	js	PASS	`new PromptLayer({apiKey, cacheTtlSeconds:300})` initializes.
11	js	SKIP	OpenAI `logRequest` example — needs `openai` SDK + `OPENAI_API_KEY`.
12	js	SKIP	Anthropic `logRequest` example — needs `@anthropic-ai/sdk` + `ANTHROPIC_API_KEY`.
13	js	PASS	`new PromptLayer({apiKey, throwOnError:true/false})`.
14	js	PASS	`try/catch` around `templates.get("NonExistentTemplate")` — logs `Failed to get template: Prompt template not found`.
15	js	PASS	`throwOnError:false` returns `null` + logs `WARNING: ... Prompt template not found`.
16	js	SKIP

#	Lang	Status	Notes
1	python	SKIP	`pl.run("cake-recipe",...) + client.track.score(...)` — prompt missing + no `OPENAI_API_KEY`.
2	python	SKIP	partial fragment — `pl_request_id` undefined.
3	js	SKIP	partial fragment — `pl_request_id` undefined.
4	bash	PASS	`curl POST https://api.promptlayer.com/rest/track-metadata` with `X-API-KEY` returns `200 {"success":true}` for a non-existent `request_id` payload.

#	Lang	Status	Notes
1	python	SKIP	`pl_client.run("support-assistant", prompt_release_label="prod", ...)` — prompt missing + no `OPENAI_API_KEY`.
2	js	SKIP	Same in JS.

#	Lang	Status	Notes
1	python	SKIP	`templates.get("my_template", {"label":"prod"})` — `my_template` not in workspace.
2	js	SKIP	Same in JS.

#	Lang	Status	Notes
1	python	SKIP	`templates.get("educational_prompt", {...})` — template missing.
2	js	SKIP	Same JS variant.
3	python	SKIP	partial fragment — dict definition only.
4	js	SKIP	partial fragment — object definition only.
5	python	SKIP	partial fragment — jinja2 nested dict only.
6	js	SKIP	partial fragment — jinja2 object only.

#	Lang	Status	Notes
1	python	PASS	HMAC-SHA256 verification snippet executes; with placeholder signature prints `Signature is invalid` as expected.
2	js	PASS	Same JS snippet executes after `npm install json-stable-stringify`; prints `Signature is invalid` with placeholder secret.

#	Lang	Status	Notes
1	json	PASS	Claude Desktop / Cursor / Windsurf `mcpServers` config — valid JSON.
2	json	PASS	OpenAI Responses API `tools` MCP entry — valid JSON.
3	json	PASS	Local stdio `npx -y @promptlayer/mcp-server` config — valid JSON.

#	Lang	Status	Notes
1	python	PASS	`OTLPSpanExporter`, `TracerProvider`, `BatchSpanProcessor` setup executes.
2	js	PASS	`NodeSDK` + `OTLPTraceExporter` setup starts cleanly under Node 24.
3	python	PASS	`promptlayer.prompt.name` / `.label` / `gen_ai.*` span attributes set.
4	js	PASS	Same span-attribute pattern under `@opentelemetry/api`.
5	yaml	PASS	Collector `otlphttp/promptlayer` exporter config parses as valid YAML.

#	Lang	Status	Notes
1	bash	PASS	`pip install promptlayer` / `pip install openai` execute.
2	python	PASS	`os.environ["OPENAI_API_KEY"] = ...; PromptLayer(api_key=...)` initializes (no LLM call here).
3	python	SKIP	`promptlayer_client.run(prompt_name="ai-poet", ...)` — `ai-poet` missing + no `OPENAI_API_KEY`.
4	bash	PASS	`npm install promptlayer` / `npm install openai` execute.
5	js	PASS	`new PromptLayer({apiKey:"pl_****"})` initializes.
6	js	SKIP	`await promptLayerClient.run({promptName:"ai-poet", inputVariables:input_variables})` — prompt missing.

#	Lang	Status	Notes
1	python	PASS	`PromptLayer(api_key="YOUR_API_KEY")` initializes.
2	python	SKIP	`pl_client.run(prompt_name="ai-poet")` — prompt missing + no `OPENAI_API_KEY`.
3	js	PASS	`new PromptLayer({apiKey:"YOUR_API_KEY"})` initializes.
4	js	SKIP	`await promptLayerClient.run({promptName:"ai-poet", ...})` — prompt missing; snippet also references undefined `input_variables`.

#	Lang	Status	Notes
1	python	SKIP	`pl_client.run("order-summary", prompt_release_label="prod", ...)` — prompt missing.
2	js	SKIP	Same in JS.
3	python	SKIP	Labelled `# pseudocode` — undefined `db`.
4	python	SKIP	Pseudocode — undefined `openai`, `queue`, `cart_items`.
5	python	SKIP	`pl_client.templates.publish(prompt_name="order-summary", prompt_template="Your updated prompt template here")` — string `prompt_template` is not the documented blueprint shape.
6	bash	FAIL	`curl -X POST https://api.promptlayer.com/prompt-templates -H "X-API-KEY: $PROMPTLAYER_API_KEY" ...` returns `HTTP 401 {"msg":"Missing Authorization Header"}` with a valid key; the public publish endpoint is actually `POST /rest/publish-prompt-template` (verified — returns 400 schema-validation on a valid key), and `/prompt-templates` (POST) refuses `X-API-KEY` entirely. Diagnosis: docs snippet uses the wrong path / auth scheme. Suggested: change URL to `https://api.promptlayer.com/rest/publish-prompt-template` (keeping `X-API-KEY`) and ensure the body matches the `publish-prompt-template` schema. Not applied or re-verified.
7	python	SKIP	`run_workflow(workflow_name="customer_support_workflow", workflow_label_name="prod", ...)` — workflow missing.
8	ts	SKIP	Same workflow in TS.

#	Lang	Status	Notes
1	python	SKIP	Option 1 `pl.run("my-prompt", ...)` — prompt missing + no `OPENAI_API_KEY`.
2	js	SKIP	Option 1 JS — same.
3	python	SKIP	Option 2 `log_request` via OpenAI — needs `openai` + `OPENAI_API_KEY`.
4	js	SKIP	Option 2 `logRequest` via OpenAI — same.

Check out Manicule.

PromptLayer Documentation Audit

1. Internal-only operations files are published as public docs pages (critical)

2. llms-full.txt contains only those internal files — not the actual docs (critical)

Runtime snippet checks

Prior reports

Sources

Check out Manicule.

PromptLayer

PromptLayer Documentation Audit

1. Internal-only operations files are published as public docs pages (critical)

2. llms-full.txt contains only those internal files — not the actual docs (critical)

Runtime snippet checks

Prior reports

Prior reports

Sources

3. Two different SDK APIs documented for fetching a labeled prompt (significant)

4. Inconsistent REST base path between docs pages (significant)

5. Auth header is spelled two different ways across the REST reference (significant)

6. Navigation and tutorial-videos page link to URLs that 404 or are missing from sitemap (significant)

7. Mintlify component markup leaks into agent-facing content (significant)

8. Parallel datasets pages create an ambiguous canonical URL (significant)

9. /overview is a near-duplicate of the docs root (significant)

10. RBAC page claims 18 permissions but enumerates none (significant)

11. No documented error/status taxonomy in the REST reference outside one endpoint (significant)

12. "Get prompt template" uses POST without explanation (minor)

13. "Custom Logging documentation" referenced without a link (minor)

14. /running-requests/promptlayer-run is an empty stub (minor)

What they do well

Top 3 recommendations

Summary

Required credentials

Pages

Prior reports

2. `llms-full.txt` contains only those internal files — not the actual docs (critical)

2. `llms-full.txt` contains only those internal files — not the actual docs (critical)

9. `/overview` is a near-duplicate of the docs root (significant)

14. `/running-requests/promptlayer-run` is an empty stub (minor)