Ito

Ito Documentation Audit

Ito's docs read well page-by-page, but the moment you cross-reference them they fall apart: the API reference can't agree on its own hostname, two of the home page's primary navigation cards 404, and the REST API the home page sells is missing from the very index the docs tell agents to read first.

1. API reference uses two different base hostnames (critical)

Location: /docs/api/overview, /docs/api/test-runs, /docs/api/repositories

Problem: The overview declares one base URL — "Base URL https://api.ito.ai — All endpoints are relative to this base URL" — and the pull-requests page honors it. But the test-runs and repositories pages run every example against a different host: curl https://api.ito-qa.com/api/test-runs?... and curl "https://api.ito-qa.com/api/repositories?...".

Consequence: A developer (or an AI agent) copying the test-runs or repositories examples points requests at api.ito-qa.com while the documented base is api.ito.ai. One of the two domains is wrong, and nothing tells the reader which. The failure mode is a DNS/connection error or a silent wrong-host response — not an obvious 404 — so it's hard to diagnose from the docs alone.

The fix: Pick one canonical API host, replace every api.ito-qa.com occurrence with it, and add a lint check so example curls cannot diverge from the documented base URL again.

2. The entire REST API is absent from llms.txt and sitemap.xml (critical)

Location: /docs/llms.txt, /docs/sitemap.xml, /docs/api/*

Problem: The home page promotes the API ("Integrate Ito data into your own tools using the REST API") and the 404 pages tell readers to "Fetch the complete documentation index at: /docs/llms.txt — Use this file to discover all available pages before exploring further." Yet llms.txt lists only the 15 Concepts/Guides/Settings pages and contains "No /docs/api/* entries." sitemap.xml likewise has only those 15 URLs and none under /docs/api/.

Consequence: An AI coding agent — explicitly told to use llms.txt for discovery — will conclude Ito has no REST API and never read the seven /docs/api/* pages. The advertised programmatic surface is undiscoverable through the discovery mechanism the docs themselves point at.

The fix: Add all /docs/api/* pages to both llms.txt and sitemap.xml. Generate the index from the route table so new pages are never silently omitted.

3. Two of the docs home navigation cards point at 404s (critical)

Location: /docs (home), /docs/settings/testing-level, /docs/integrations/github

Problem: Of the primary "Explore the docs" cards, the Settings card links to /docs/settings/testing-level and the Integrations card links to /docs/integrations/github — both return HTTP 404 (Mintlify "Page Not Found"). The Settings card promises "testing depth, automation rules, and video hosting"; none of those three has a docs page. There is no /docs/integrations/* section at all.

Consequence: A new user clicking the most prominent entry points lands on dead ends on their first visit. The quickstart's "Testing level" card points at the same dead /docs/settings/testing-level, compounding the impression that core configuration is undocumented.

The fix: Either build the Settings and Integrations landing pages (and the testing-level page) before linking them, or repoint the cards to pages that exist. Add a CI link-checker against the rendered nav so a card can't ship pointing at a 404.

4. "Settings → Automation" is referenced repeatedly but does not exist (significant)

Location: /docs/concepts/test-runs, /docs/concepts/pr-comments, /docs/settings/automation (404)

Problem: test-runs says "See Settings → Automation to enable automatic testing" and lists "Settings → Automation — configure which PRs trigger automatic runs." pr-comments says "You can control what appears in PR comments from Settings → Automation. See Automation settings for the full reference." The target page /docs/settings/automation returns 404. All automation controls (Automatic Review, Automatic Incremental Review, Smart Defaults, Automated Reviews master switch) are described only inside the pr-comments concept page, never on a settings page.

Consequence: A reader following the most-cited cross-reference in the Concepts section hits a 404 every time. The changelog confirms "Repository automation controls" shipped in v1.0.12 (May 27, 2026), so a real feature exists with no settings documentation.

The fix: Create /docs/settings/automation as the canonical reference for these controls and link the concept pages to it, or rewrite the cross-references to point at wherever the automation options are actually documented.

5. Auth page and API examples disagree on the cookie name and header (significant)

Location: /docs/api/authentication, /docs/api/test-runs, /docs/api/repositories

Problem: The authentication page says the session cookie "begins with ito_session" and shows --cookie "<your-session-cookie>". But the test-runs and repositories examples use a different header and a different cookie name: -H "Cookie: session=<your-session-cookie>". So one set of docs calls the cookie ito_session and another calls it session.

Consequence: A developer can't tell whether the cookie they need to send is named ito_session or session. Sending the wrong name yields {"error": "Not authenticated"} (401) with no hint that the cookie name — not the value — is the problem.

The fix: Standardize on one cookie name and one header style across every API page, and state the exact cookie name once, authoritatively, on the authentication page.

6. Pagination limit contradicts itself across endpoints (significant)

Location: /docs/api/overview, /docs/api/pull-requests, /docs/api/repositories

Problem: The overview declares pagination global: "limit integer default 100 maximum 100 ... Paginated responses always include total, offset, and limit." pull-requests agrees ("default:100 ... Capped at 100"). But repositories says "limit ... Defaults to 20" with no maximum documented.

Consequence: An agent or script that relies on the overview's "default 100" will silently receive only 20 repositories per page and may assume it has the full list — a correctness bug that produces no error, just missing data.

The fix: Make pagination defaults/maxima consistent, or if /api/repositories genuinely defaults to 20, state that exception explicitly on the overview page rather than declaring the convention global.

7. Session-cookie-only auth undercuts the "integrate into your own tools" pitch (significant)

Location: /docs/api/authentication, /docs (API Reference card)

Problem: The docs pitch the API for integration ("Integrate Ito data into your own tools using the REST API"), but the only documented auth is a manually copied browser cookie: "Open your browser's developer tools, go to Application → Cookies, and copy the value of the Ito session cookie." There are no API keys or tokens, and the docs note the cookie expires "when you sign out or when your GitHub OAuth token is revoked."

Consequence: Cookie-from-devtools auth is unworkable for the automation the API is sold for — any unattended script breaks the next time the user signs out, and there's no programmatic way to obtain or refresh credentials. This is exactly the integration use case the home page advertises.

The fix: Document a token/API-key flow for programmatic access, or clearly scope the API as "dashboard-session-only, not for unattended automation" so readers don't build integrations on a credential that expires on sign-out.

8. The same four severity levels appear under four different names and wordings (significant)

Location: /docs/concepts/severity-levels, /docs/how-it-works, /docs/guides/reading-results, /docs/api/test-runs

Problem: The four-level scale is defined four times with diverging language. "Low" is "Minor issues or edge cases with limited impact on users. Often cosmetic" (severity-levels), "Edge cases or cosmetic issues with limited impact" (reading-results), and "minor issues or edge cases with limited user impact" (how-it-works). Worse, the API renames the field entirely: test-runs uses "risk": "high" with its own definitions ("medium" = "Moderate bug with a workaround"), while pull-requests uses maxUnrelatedSeverity (severity terminology).

Consequence: An agent mapping dashboard "severity" to the API can't find a severity field on test cases — it's risk — and the definitions don't line up, so any automated triage built on these values is unreliable. Humans get four subtly different rubrics for one concept.

The fix: Define the four levels once canonically, reuse that exact wording everywhere, and either rename the API field to severity or document risk/severity/maxUnrelatedSeverity as the same scale with a single shared definition.

9. Test-case Category taxonomy doesn't match the API, which also returns an undocumented field (significant)

Location: /docs/concepts/test-cases, /docs/api/test-runs

Problem: The test-cases page documents Category as one of "Happy-path, Edge, Adversarial, Logic, Accessibility, or Mobile." The API example instead returns "category": "checkout" — a value outside that taxonomy — plus a "tags": ["checkout", "payment"] array that no docs page mentions.

Consequence: A developer parsing category and expecting one of six enumerated values will hit unexpected strings, and an undocumented tags field carries data no consumer is told exists. Code that switches on category will fall through.

The fix: Reconcile the Category enum with what the API actually returns (document whether category is free-form or constrained), and document the tags field on both the concept and API pages.

10. Slack integration is advertised on the home page with zero documentation (significant)

Location: /docs (Integrations card), /docs/integrations/slack (404)

Problem: The Integrations card says "Set up GitHub and Slack integrations to fit your workflow," but /docs/integrations/slack returns 404 and Slack "is never mentioned again anywhere in the docs body."

Consequence: A team evaluating Ito for Slack notifications has no way to learn whether the feature exists, how to connect it, or what it does. The docs assert a capability they never describe.

The fix: Add a Slack integration page covering setup and behavior, or remove Slack from the Integrations card until it's documented.

11. The "Additional findings" indicator is ℹ️ on some pages and ⚠️ on others (minor)

Location: /docs/guides/reading-results, /docs/concepts/test-runs, /docs/concepts/pr-comments

Problem: The same "Additional findings" concept (pre-existing failures unrelated to the current PR) is rendered with an info icon ℹ️ in reading-results and test-runs, but with a warning emoji ⚠️ in pr-comments ("additional findings ⚠️", "⚠️ Additional Findings").

Consequence: Readers correlating a dashboard ℹ️ row with a ⚠️ line in a PR comment may think they're two different things, when they're the same category. Small, but it muddies an already-confusing pre-existing-vs-introduced distinction.

The fix: Standardize one glyph for "Additional findings" across dashboard and PR-comment docs.

12. "Testing level" vs "testing depth," and the FAQ points at a nonexistent "Settings → General" (minor)

Location: /docs (Settings card), /docs/how-it-works

Problem: The docs home Settings card calls the feature "testing depth," while how-it-works calls it "testing level" and its FAQ instructs "Go to Settings → General and choose a testing level ... See the testing level settings for details." Neither "Settings → General" nor a testing-level settings page exists (the link target /docs/settings/testing-level 404s).

Consequence: A user searching the UI for "testing depth" or "Settings → General" finds neither term, because the actual Settings sub-areas documented are Team, Plan & Billing, Context & Secrets, and Test Instructions.

The fix: Use one term consistently, and update the FAQ to name the Settings location that actually exists once the testing-level page is built.

13. skipped test-case status is documented but has no place to view it (minor)

Location: /docs/concepts/test-cases, /docs/guides/reading-results, /docs/concepts/test-runs

Problem: test-cases documents a Skipped status ("Skipped test cases were not executed in this run") and lists passed, failed, or skipped as the status values. But the dashboard test-case list documents only the tabs "All Test Cases / Passed / Failed / Additional findings" — no Skipped tab or filter appears anywhere.

Consequence: A user with skipped test cases has no documented way to find them in the UI; they're invisible against the documented All/Passed/Failed/Additional-findings tabs.

The fix: Document where skipped cases surface in the dashboard (a tab, a filter, or inline), or clarify that skipped cases appear only via the API.

14. Variable naming rule conflicts between guide and API (minor)

Location: /docs/guides/secrets-and-variables, /docs/api/secrets-variables

Problem: The guide says "Variable names are automatically uppercased" (so any case is accepted and normalized). The API says secret names "Must be uppercase with underscores (e.g. API_KEY)" — a validation requirement, not auto-normalization. The two pages describe different contracts for similar inputs.

Consequence: A developer creating a variable via the API in lowercase, expecting auto-uppercasing per the guide, may instead get a 400 if the API enforces the must-be-uppercase rule. The distinction between variables (auto-uppercased) and secrets (must be uppercase) is never stated explicitly.

The fix: State the casing rule per resource type explicitly — whether names are normalized or validated — on both the guide and the API page.

What they do well

• The /docs/api/pull-requests page is a clean reference model: correct base URL, consistent --cookie usage, and a status enum that matches the test-runs page — the other endpoint pages should be normalized to it.
• Strong, specific safety warnings where they matter most: "Never use production credentials as seed data," and the explicit caveat that Test Instructions "apply to everyone at your org by default" and are global unless scoped.
• The changelog is dated and maintained (v1.0.1 → v1.0.17), which makes it possible to date features and spot doc gaps like the missing automation-settings page.

Top 3 recommendations

1. Make the API reference internally consistent. One base hostname, one cookie name and header style, one pagination convention — enforced by a lint/CI check on example curls so api.ito.ai vs api.ito-qa.com and ito_session vs session can never diverge again.
2. Fix discovery and dead ends. Add all /docs/api/* pages to llms.txt and sitemap.xml, and either build the Settings/Integrations/Automation/testing-level pages or remove the cards and cross-references that 404.
3. Unify the severity model. Define the four levels (and their iconography) once, reuse that exact wording everywhere, and reconcile the API's risk field with the dashboard's severity so automated triage maps cleanly.