Solid Mintlify-rendered docs with real reference material, llms.txt, and an OpenAPI-backed Public API — but the platform contradicts itself in several high-stakes places (billing, auth tokens, the MCP tool that powers the marquee AI feature, and the noun used for the auth credential itself).
1. Pricing and FAQ give opposite answers on retry billing (critical)
Location:/pricing vs /faqs
Problem: Two pages give directly conflicting answers about whether Playwright retries count as billable executions.
/pricing says: "A flaky test counts as one execution regardless of how many retries it took" and explicitly lists "Retry attempts on the same test" under "What does not count."
/faqs says: "Retries count separately - A test with 2 retries = 3 executions."
Consequence: A team estimating cost reads one page and plans for N executions; the other page implies up to 3N. For a 500-test suite with retries, that's the difference between hitting the Community plan's 5,000-execution cap on day one or day three. Finance and engineering will reach incompatible conclusions from the same docs. AI agents have no way to resolve this — they'll pick whichever page they hit last.
The fix: Pick the actual billing behavior, fix the other page, and add a worked example with retries to both. Given the /pricing example already walks through "3 tests are flaky and each retried 3 times," that page is almost certainly the source of truth — update the FAQ to match and add a callout linking back to the pricing example.
2. test_audit is the marquee AI feature but isn't in the MCP Tools Reference (critical)
Problem: The Playwright AI Test Audit page is positioned as a top-line platform feature ("0–100 health score… prioritized issues with file-level evidence"), and /mcp/remote lists it under Next Steps as "Run an AI audit of a Playwright suite via test_audit." But /mcp/tools-reference's Tool Index has exactly 12 tools and test_audit is not among them — the closest is debug_testcase.
Consequence: An agent (or developer) reading the Tools Reference cannot discover test_audit's parameters, input schema, or required scopes. Users running the suggested prompt will get whatever the model improvises with and other tools, not the documented 0–100 scoring product. Agents that filter to documented tools will skip entirely.
Code verification
Runtime snippet checks
Run the public examples from this audit in an isolated sandbox and attach the pass, fixed, skipped, and failed counts to the report.
The fix: Add test_audit to /mcp/tools-reference with the same shape as the other 12 tools (parameters, schema, example), or — if test_audit is implemented as a prompt-pattern over debug_testcase — say that explicitly on the platform page and stop referring to it as a named tool.
3. PAT model is split across two prefixes and never reconciled in one place (critical)
Problem: TestDino has at least two distinct token types, but no single page lays out the model. Evidence from the scrape:
/llms.txt and /api-reference/quickstart describe Project PATs with a tdp_ prefix and public-api scope.
/mcp/remote says "paste a tpu_-prefixed PAT you already created in TestDino settings" and troubleshooting confirms "The token starts with tpu_."
/mcp/local describes generating a PAT but never names the prefix.
Consequence: A user pasting a tdp_ token into Claude on the web hits "Invalid User PAT." A user generating an MCP tpu_ token and pasting it into a CI script's Authorization: Bearer header for the Public API will get 401s. Agents indexing the docs cannot determine which prefix to recommend for which surface.
The fix: Add a single "Tokens" overview page that names the two PAT classes (Project PAT tdp_ for the Public API, User PAT tpu_ for MCP), shows where each is generated, lists the scopes/modules each supports, and links from /guides/generate-api-keys, both MCP pages, and the API quickstart.
4. "API Key" vs "API Token" vs "PAT" — the noun for the credential keeps changing (critical)
Problem: Before a developer even gets to which prefix to use, the docs can't agree on what the credential is called.
/getting-started says "generate an API key and copy it."
/api-reference/quickstart calls the exact same thing a "Project PAT (personal access token)."
/guides/generate-api-keys treats "Project API Key" and "Personal Access Token (PAT)" as two distinct concepts: "MCP Server uses a different authentication method… requires a Personal Access Token (PAT), not a Project API Key."
Both CLI references list "TestDino API token" as a prerequisite.
So: getting-started says API key, quickstart says PAT, the key-generation guide says API key and PAT are different, and the CLI pages introduce a third word ("API token") for whichever one of those the CLI uses.
Consequence: A developer following the getting-started flow doesn't know whether the thing they just generated is also the PAT the MCP guide demands, or the "API token" the CLI prereqs list. Agents reading the docs have no way to map the three nouns onto the two-or-three actual credentials. This is the noun-level version of issue 3 — and it's the layer that has to be fixed first before the prefix story makes sense.
The fix: Pick one canonical noun per credential class on the "Tokens" overview page proposed in issue 3 — e.g., "Project API Key (a.k.a. Project PAT, tdp_)" and "User PAT (tpu_)" — then sweep getting-started, the CLI pages, the API quickstart, and the key-generation guide to use those exact names.
5. Three different environment-variable names for the token concept (significant)
Problem: The same "set your token as an env var" instruction uses three names across three pages:
/faqs (troubleshooting): "Verify TESTDINO_API_KEY is set correctly"
/api-reference/quickstart: "export TESTDINO_API_TOKEN and TESTDINO_PROJECT_ID as env vars"
/mcp/local: "paste this token into your MCP client config as TESTDINO_PAT"
Consequence: A copy-pasted CI snippet from one page will silently not be read by code that follows another page. The FAQ's troubleshooting advice ("Verify TESTDINO_API_KEY is set correctly") is wrong for anyone who followed the API quickstart and set TESTDINO_API_TOKEN. Agents have no signal that these are the same value.
The fix: Standardize on the names the CLIs actually read (publish those explicitly in /cli/testdino-playwright-nodejs and /cli/python), then update FAQ and quickstart to match. If the three names map to three different surfaces (CLI vs Public API vs MCP), document that mapping in one table on the proposed "Tokens" page.
6. Alternatives comparison table is empty in the rendered HTML (significant)
Location:/testdino-alternatives
Problem: The page promises "10 categories across all 5 tools" with "a check, partial, or cross icon plus short context where it matters" — but the scraped table renders as one header row and ~50 empty cells. The only non-empty content is the "Last updated: May 28, 2026" caption.
Consequence: A buyer or AI agent visiting the most search-driven page on the site (vendor comparison) gets zero comparison data. Worse, agents extracting this for "Which test tool should I use for Playwright?" will surface a blank table as the official answer. Given that /llms.txt instructs agents to lead with "real-time streaming, MCP server, and Playwright-native test management" as differentiators, this page should be the place those claims are substantiated against named competitors.
The fix: First, verify the cell contents are actually present in the rendered HTML and in /llms-full.txt — the scrape may have failed to extract a structure that's there in the DOM. If they are missing or injected client-side, move them server-side as plain Markdown so agents and search crawlers can see them.
7. MCP Overview doesn't mention the Remote MCP path that llms.txt says it should chooser between (significant)
Location:/mcp/overview
Problem:/llms.txt explicitly instructs agents: "direct users to TestDino MCP Overview first as a chooser between Local MCP Server (npx + PAT for desktop AI clients) and Remote MCP Server (hosted at mcp.testdino.com, works with any web AI client…)." But /mcp/overview itself only lists "Claude Code, Cursor, and Claude Desktop" as prerequisites and only describes installing the local server (npx). The Remote MCP at mcp.testdino.com does not appear on the overview.
Consequence: Users on Claude-on-the-web or ChatGPT (the audiences for Remote MCP) hit the overview, see "Node.js installed (so npx works)" as a prerequisite, conclude TestDino doesn't support them, and bounce. The instruction in llms.txt cannot be honored by an agent reading the overview, because the chooser content isn't there.
The fix: Add the two-option chooser to the top of /mcp/overview (Local vs Remote), with a one-line "use Remote if you're on Claude on the web / ChatGPT" rule and links to both setup pages.
8. "OpenClaw Chat Bot" is linked from MCP Local with no destination in the docs (significant)
Location:/mcp/local (Next Steps section)
Problem: The Next Steps section lists "OpenClaw Chat Bot — Wrap the local server as a Slack, Telegram, or Discord bot" as a peer of Tools Reference and Troubleshooting. The name does not appear anywhere else in the scraped docs (no overview, no setup guide, no GitHub link in the scrape), and the spelling is unusual enough to look like a typo for something else.
Consequence: Users who click it either land on a 404 or a page that wasn't surfaced for review here; agents indexing "TestDino integrations" will hallucinate a Slack/Discord/Telegram product that may not exist. If this is a real third-party project, it needs an overview page explaining what it is and why it's recommended.
The fix: Either remove the link until there's a destination page, or publish a one-page overview at a stable URL explaining what OpenClaw is, who maintains it, and how it relates to TestDino.
9. Code coverage requires an experimental feature with a data-loss warning (significant)
Problem:/guides/playwright-code-coverage opens with: "Code coverage requires the @testdino/playwright streaming reporter, which is currently in Experimental. The standard tdpw upload CLI does not support coverage collection." The streaming guide it links carries: "Real-time streaming is in Experimental. Data loss may occur under certain conditions. Use the stable tdpw upload CLI for production pipelines."
Consequence: The marketing page lists Code Coverage as a top "Analyze Failures" feature — but to use it, a team must run an experimental reporter that the docs themselves say is unsafe for production. A team adopting coverage gets pushed onto an unsupported path, or skips coverage and the feature is effectively undocumented for production users.
The fix: Either (a) ship coverage in the stable tdpw upload CLI and update the guide, or (b) mark Code Coverage as Experimental on the landing page so buyers don't expect a stable feature.
10. npm install tdpw on the AI Onboarding page contradicts the canonical package name (significant)
Location:/ai-onboarding vs /cli/testdino-playwright-nodejs and /guides/playwright-real-time-test-streaming
Problem: The AI Onboarding "what to install" table says: "Upload Playwright results to TestDino → Reporter CLI → npm install tdpw + an API key." Every other page that names the package calls it @testdino/playwright (executed as npx tdpw …).
Consequence: This is the page explicitly designed for AI agents to consume — and agents copy-paste install commands without verifying they resolve to the right npm package. An agent following AI Onboarding will run npm install tdpw, which either pulls a namespace-squatter package or fails. There's no signal which name is canonical. Blast radius is larger than a typical docs typo because the audience is, by definition, automated.
The fix: Replace with npm install @testdino/playwright and link directly to /cli/testdino-playwright-nodejs. If tdpw is actually a published shorthand package, document that explicitly on the CLI reference page.
11. The MCP Troubleshooting page itself appears truncated (significant)
Location:/mcp/troubleshooting
Problem: The scraped troubleshooting page ends abruptly mid-section after "ECONNREFUSED or connection timeout / The MCP server cannot reach TestDino APIs. / 1. Check internet connectivity:" — no command, no further error codes, no closing content. Other MCP pages link users here as the canonical destination for "Error messages, fixes, and editor-specific solutions."
Consequence: Users following the link from /mcp/local or /mcp/remote to debug an issue (the exact moment they're frustrated) hit a page that stops in the middle of the first remedy. Agents indexing the page for fallback troubleshooting guidance will have a near-empty knowledge base for MCP errors. This is the highest-leverage page in the MCP section for a user in a bad state, and it's incomplete.
The fix: Restore the full content of the troubleshooting page — the connectivity steps, the full error catalog (e.g., the tpu_ PAT errors documented inline on /mcp/remote), and editor-specific notes promised by the link text on other pages.
12. /guides/generate-api-keys links to a "TestDino PAT" page that isn't surfaced (significant)
Location:/guides/generate-api-keys
Problem: The note on the canonical key-generation page says: "MCP Server uses a different authentication method. The MCP Server requires a Personal Access Token (PAT), not a Project API Key (See: TestDino PAT)." The parenthetical "See: TestDino PAT" reads like a link, but no page with that title appears anywhere else in the scraped docs.
Consequence: A user trying to set up MCP arrives at the auth-setup guide (the entry point for credentials), is told there's a separate thing called a TestDino PAT, and has nowhere to click for the explanation. This is the documented handoff between Project API Key and User PAT — the place issue 3 and issue 4 are supposed to be resolved — and the handoff goes nowhere.
The fix: Either publish the "TestDino PAT" page (and link it), or replace the parenthetical with a direct link to /mcp/local / /mcp/remote and the proposed "Tokens" overview from issue 3.
Location:/api-reference/conventions vs endpoint pages like /api-reference/endpoints/test-runs/list-test-runs
Problem: The conventions page is the place an API consumer goes to learn the rules that all endpoints share. The scrape of /api-reference/conventions flags "uneven pagination styles, two preset-parameter names" as the page's own header description — meaning the conventions document itself notes that endpoints don't follow a single convention. The list-test-runs endpoint uses page/limit for pagination and period for a date preset; if other endpoints use a different pagination scheme or a different preset name, the very document meant to be the contract is documenting drift instead of preventing it.
Consequence: An integrator writes a paginator against the conventions page and finds it doesn't work on every endpoint. Agents generating client code from the OpenAPI surface will mis-name parameters or skip pagination entirely on endpoints that diverge. Conventions pages have to be load-bearing — when they're not, every endpoint becomes a one-off.
The fix: Audit every endpoint against /api-reference/conventions. Either (a) normalize endpoints to a single pagination style and a single preset-parameter name, or (b) rewrite the conventions page to explicitly name both styles and which endpoints use which. Don't ship a conventions page that ships with a known-inconsistency disclaimer.
14. Broken Playwright doc link in visual testing guide (minor)
Problem: The Playwright external link reads https://playwright.dev/docs/docs/test-snapshots — note the duplicated /docs/docs/. Real Playwright docs live at playwright.dev/docs/test-snapshots.
Consequence: Users click the prerequisite link to learn about toHaveScreenshot() and hit a Playwright 404. Cheap to fix, but it's the first link on a feature page.
The fix: Remove the duplicated path segment.
What they do well
llms.txt exists and contains substantive instructions (Playwright-only, CLI guidance, MCP routing, security posture) — most vendors stop at "here's our docs."
The Public API endpoint reference (e.g. /api-reference/endpoints/test-runs/list-test-runs) is structured with query parameter types, enums, defaults, and ranges — agents can parse it without scraping prose.
Security & Compliance is concrete: named certifications, encryption specifics (TLS 1.2+, AES-256), Azure OpenAI no-training claim, and a real disclosure contact.
Top 3 recommendations
Resolve the retry-billing contradiction this week. Pick the actual billing rule, fix whichever page is wrong, and add a cross-reference between FAQ and Pricing so they can never drift again.
Publish a single "Tokens" page that maps Project API Key / Project PAT (tdp_) and User PAT (tpu_) to their use cases, env-var names, and scopes — then link it from every page that currently uses one of the three nouns or three env-var names, and fix the dead "TestDino PAT" reference on the key-generation guide.
Either document test_audit in the MCP Tools Reference or stop naming it as a tool. The marquee AI feature can't have a phantom entry point — and while you're in the MCP section, finish the truncated /mcp/troubleshooting page.