The docs are well-structured at the Mintlify level (llms.txt and llms-full.txt both exist, sitemap is clean, 14 locales are wired up), but core surface-area facts contradict each other across nearly every page that mentions them — policy counts, CLI support, scope semantics, exit codes, and file-extension support all disagree. For a "reliability tool for autonomous agents," the docs themselves are not reliably consistent.
2. "Local" scope refers to two different files in two different places (critical)
Location:/configuration vs /cli/install-policies
Problem: The --scope local flag and the "Local" configuration scope are documented as different things:
/configuration defines Local as .failproofai/policies-config.local.json — "Individual overrides (gitignored)"
/cli/install-policies defines --scope local as "Claude only, uses .claude/settings.local.json"
These are two different files, in two different directories, with two different purposes (failproofai's own config vs. Claude's settings file).
Consequence: A developer running failproofai policies --install --scope local reasonably expects to write to their personal .failproofai/policies-config.local.json override. Instead the command writes to Claude Code's .claude/settings.local.json, which has very different semantics (it controls Claude itself, not failproofai's policy params). The override they thought they made silently goes to a different file and may have no effect.
The fix: Rename one of these. Either call the CLI flag --scope claude-local (since it's Claude-specific), or align the file path so --scope local writes to .failproofai/policies-config.local.json as the configuration page describes. Document the exact file path each --scope value writes to in a small table.
3. Exit code 2 means two different things in the hook protocol (critical)
Location:/cli/hook vs /testing
Problem:/cli/hook defines the exit-code contract as:
Exit 0 (allow): Action proceeds
Exit 1 (deny): Action blocked
Exit 2 (instruct): Guidance injected into Claude's context
But /testing describes the same exit codes differently: "Stop directives exit with code 2, and allows produce empty stdout."
So exit code 2 is simultaneously "instruct" (per the hook spec) and "Stop" (per testing). Both are documented as authoritative.
Consequence: Anyone writing a custom hook integration, a test harness, or a wrapper script has to guess which interpretation Claude Code actually receives. If exit 2 is overloaded internally, an instruct decision could be misread as a Stop directive, silently terminating a session that should have continued with added context.
The fix: Clarify on /cli/hook whether exit code 2 carries different meanings based on event type (PreToolUse vs Stop), or pick one meaning. Add an explicit table mapping (event type, decision) → (exit code, stdout shape) so an integrator never has to infer.
4. The list of supported agent CLIs varies by page — five different claims (critical)
/cli/install-policies: --cli flag accepts only claude, codex, or copilot (3 names)
Consequence: A Cursor, Windsurf, OpenCode, Pi, or Gemini CLI user reads the dashboard or for-agents page, installs the tool, then can't actually wire it up — the install command rejects their CLI value. Windsurf in particular is named on /for-agents but is not an --cli option anywhere. A landing-page visitor sees only 3 CLIs and never learns Cursor or Gemini exist as targets. An agent fetching docs to answer "does this work with OpenCode?" will give a confidently wrong answer in either direction depending on which page it hit.
The fix: Maintain the canonical list in one place (e.g., the install command's --cli enum). Auto-generate the marketing, examples, for-agents, and getting-started lists from that source. If only 3 CLIs are truly installable today, fix every other claim; if 7 are supported, fix --cli to accept all 7 and add Windsurf (or remove it from /for-agents).
5. Custom-policy file-extension support contradicts itself (significant)
Location:/custom-policies and /llms-full.txt vs /getting-started
Problem:/custom-policies and /llms-full.txt say custom policies are auto-loaded from files matching *policies.{js,mjs,ts}. /getting-started says: "add policy files (.mjs format). These load automatically across the team without additional configuration steps once committed to version control."
Consequence: A team following the "Getting Started" path writes .ts policy files, commits them, and they fail to load — or vice versa. The mismatch quietly breaks team-shared policy enforcement, which is exactly the use case /getting-started is selling.
The fix: Update /getting-started to state all three supported extensions (.js, .mjs, .ts) and link to /custom-policies for loading-order rules. If .ts requires a build step or specific runtime, document it.
6. Custom policies advertise four event types; the hook handler ships 22 (significant)
Consequence: A developer who wants to write a custom policy reacting to SessionStart or SubagentStop reads /custom-policies, doesn't see the event, and assumes it isn't possible — when in fact the underlying handler accepts it. Conversely, if those events are not deliverable to custom policies, the hook page advertises an API surface that custom-policy authors can't reach.
The fix: State explicitly on /custom-policies which subset of /cli/hook's events are deliverable to user code, and either widen the type union in the API reference or add a note explaining why custom policies see a narrower set than built-ins.
7. PORT env var is unprefixed; collides with every other Node service on a developer's machine (significant)
Location:/cli/environment-variables
Problem: Every other env var on the page is namespaced — FAILPROOFAI_DISABLE_PAGES, FAILPROOFAI_ALLOWED_DEV_ORIGINS, FAILPROOFAI_LOG_LEVEL, FAILPROOFAI_HOOK_LOG_FILE, FAILPROOFAI_TELEMETRY_DISABLED, FAILPROOFAI_LLM_*. Then: PORT: Server listening port (defaults to 8020).
Consequence:PORT is the most-collided variable name in the Node ecosystem (Next.js, Express, Nest, Heroku, Vercel local dev all use it). A developer running failproofai in a shell that already exports PORT=3000 for their Next app will silently bind failproofai to 3000, possibly conflicting with their dev server. The --port flag exists but isn't called out as the safer override.
The fix: Rename to FAILPROOFAI_PORT (keep PORT as a deprecated fallback for one release), or at minimum add a callout warning that PORT is read globally and recommend --port for project shells.
8. The llm config field exists but no page documents which policies need it (significant)
Location:/configuration and /cli/environment-variables
Problem:/configuration says: "llm (optional object) — Configures LLM client for policies requiring AI calls, including model selection and API credentials." /cli/environment-variables documents FAILPROOFAI_LLM_BASE_URL, FAILPROOFAI_LLM_API_KEY, FAILPROOFAI_LLM_MODEL (default gpt-4o-mini). No page anywhere — including /built-in-policies — identifies which policies trigger LLM calls, when they fire, what they cost, or what happens if the key is missing.
Consequence: A user turning on "all" policies (failproofai policies --install all) may unknowingly start making outbound OpenAI calls from a tool that elsewhere markets itself as "operates entirely locally" and "no external data transmission" (/dashboard, /introduction). That's a privacy and cost surprise tied directly to the product's headline claim of locality.
The fix: On /built-in-policies, tag the policies that require LLM calls with a clear icon/label. Add a section to /configuration listing those policy names and the call shape (provider, model, payload). Update the "stays on your machine" claims to qualify them: "except policies that explicitly opt into LLM evaluation."
9. "Operates entirely locally" trust claim is materially incorrect for LLM-backed policies (critical)
Problem: Multiple pages emphasize local-only operation as a headline trust claim:
/introduction: "all processing stays on your machine with no external data transmission"
/dashboard: "operates entirely on your filesystem with no remote data transmission"
/llms-full.txt: "nothing uploads to remote servers"
But /cli/environment-variables documents FAILPROOFAI_LLM_BASE_URL: API endpoint (defaults to https://api.openai.com/v1). The default is a remote OpenAI endpoint, and /configuration confirms an llm config field exists for "policies requiring AI calls."
Consequence: This is not a documentation nit — it's a defect in the product's primary trust claim. A privacy-conscious developer or a regulated team picks failproofai specifically for the "no external transmission" guarantee, then enables an LLM-backed policy and unknowingly streams agent tool inputs (which can include source code, paths, and sanitization-target output) to OpenAI. The default endpoint is OpenAI's hosted API, not a local model. For HIPAA, SOC2, or any data-residency context, this is the difference between a tool that's safe to deploy and one that triggers a security incident.
The fix: Soften and qualify every "stays local" claim. Add: "LLM-backed policies are opt-in and will send context to the configured FAILPROOFAI_LLM_BASE_URL (default: OpenAI). To stay fully local, leave the llm config unset or point it at a self-hosted endpoint." Also list which built-in policies are LLM-backed.
10. Activity-log scope disagrees: only non-allow vs all events (significant)
Location:/architecture vs /dashboard
Problem:/architecture says: "Each hook event generates one JSONL record to ~/.failproofai/hook-activity.jsonl, capturing timestamp, decision, tool name, and policy name. Only non-allow decisions are logged to minimize file size." /dashboard describes the Activity tab as: "Paginated history of all hook events with filtering by decision, event type, CLI, policy name, or session ID."
Consequence: A user investigating "why did my agent do X?" expects to find the full timeline on the Activity tab. If the underlying JSONL only captures non-allow decisions, the tab can never show the allow events the page promises to filter on. Conversely, if all events are logged, the architecture page's "minimize file size" claim is misleading and the log will grow much faster than expected.
The fix: Decide which is true. If only non-allow events are persisted, update /dashboard to remove the implication that allow events appear in Activity. If everything is persisted, drop the "non-allow only" line from /architecture and document the actual retention/rotation policy.
11. --scope options are asymmetric between install and uninstall (significant)
Location:/cli/install-policies vs /cli/remove-policies
Problem: The install command documents three scope values — user, project, local. The uninstall command documents four — user, project, local, and all ("removes from every configuration level simultaneously"). There is no corresponding --scope all on install.
Consequence: A user who runs failproofai policies --uninstall --scope all to clean up cannot symmetrically reinstall in one command — they have to invoke install three times, once per scope. Worse, if a user assumes the commands are symmetric and tries --install --scope all, the behavior is undocumented: either the flag is silently accepted (and does something), silently ignored, or errors. None of those is the expected "round-trip" behavior. The asymmetry also obscures whether install/uninstall manipulate the same set of files.
The fix: Either add --scope all to install (with the same semantics — write entries to every scope) or remove it from uninstall and require explicit scope iteration. Document the exact files each scope value reads from and writes to, in a single table shared by both pages.
12. Branding split between befailproof.ai and failproofai is never reconciled (minor)
Location: Sitewide
Problem: The docs domain is docs.befailproof.ai. The product name varies: "Failproof AI" (/introduction, /examples), "failproofai" (CLI, npm package). The npm package is failproofai — no be- prefix. The /package-aliases page lists typo-squat protections for failproof, failprof, failprof-ai, faliproof, faliproof-ai, but befailproof (the actual domain) is not among them. The npm aliases for failproof-ai and fail-proof-ai are listed as "pending npm approval."
Consequence: Developers Googling the product reach befailproof.ai but npm install befailproof fails (no such package). Agents asked "what's the package name?" can easily produce befailproof, failproof-ai, or failproofai depending on which page they parsed first. The typosquat list itself omits the most likely typo (the brand's own domain prefix).
The fix: Either rename the package to befailproofai to match the domain, or surface a clear "Package: failproofai, domain: befailproof.ai" callout on /introduction and /getting-started. Add befailproof, befailproofai, and be-failproof-ai to the typosquat registration list.
13. No machine-readable policy schema or registry (minor)
Location:/built-in-policies, /llms-full.txt
Problem: The 38–39 built-in policies are documented in prose with ad-hoc parameter mentions (allowPatterns, allowPaths, protectedBranches, thresholdKb, etc.). There is no JSON Schema, no OpenAPI-style spec, and no programmatically-fetchable policy registry. The customPolicies.add() signature is shown inline as TypeScript-ish prose.
Consequence: An agent building a config for a new user has to scrape prose to learn which policies exist and what parameters they accept. The /cli/list-policies page notes that the CLI flags unrecognized keys, meaning a typoed param fails late — there's no way to validate a policies-config.json against a schema before shipping it. Teams can't generate type-safe wrappers without reading every page.
The fix: Publish a JSON Schema for policies-config.json and a machine-readable index of built-in policies (name, description, parameter schema). Link them from /llms.txt so coding agents discover them automatically. Ship .d.ts for the customPolicies API.
14. FAILPROOFAI_HOOK_LOG_FILE "default location" is never defined (minor)
Location:/cli/environment-variables and /custom-policies
Problem:/cli/environment-variables documents FAILPROOFAI_HOOK_LOG_FILE: Custom logging destination or true for default location — but no page states what that default path actually is. /custom-policies mentions in passing that custom-policy errors can be viewed with tail -f ~/.failproofai/hook.log, but never connects that path to the env var or confirms it's the default.
Consequence: A developer debugging a hook integration sets FAILPROOFAI_HOOK_LOG_FILE=true, expecting logs to land somewhere predictable, then has to guess the path. They may grep ~/.failproofai/, search their cwd, or check /tmp. For an opt-in logging feature, the missing default destination defeats the point.
The fix: On /cli/environment-variables, replace "default location" with the actual path (presumably ~/.failproofai/hook.log based on /custom-policies). Cross-link the two pages so anyone hunting for hook errors lands in the same place.
What they do well
Both llms.txt and llms-full.txt exist and are advertised in the navigation — uncommon and welcome.
Eighteen pages localized into fourteen languages with a consistent sitemap.
Strong proactive typosquatting story on /package-aliases (even if the brand's own prefix is missing from the list).
Top 3 recommendations
Reconcile the contradictions table. Pick the canonical value for: policy count, supported CLI list (across landing, examples, for-agents, getting-started, dashboard, install), custom-policy file extensions, activity-log scope, --scope options on install vs uninstall, and --scope local target file. Add CI that fails when the documented count or list drifts from the source of truth.
Fix the locality claim. This is a trust-claim defect, not a doc nit. Either qualify "stays on your machine" everywhere it appears, or make LLM-backed policies obvious and opt-in with a clear list of which built-ins call out and a default that isn't a remote OpenAI endpoint.
Publish a machine-readable policy registry. JSON Schema for policies-config.json + a structured index of built-in policies and their parameters, linked from llms.txt. This is the single highest-leverage change for agent-driven adoption of a product whose audience is, definitionally, AI coding agents.
Fixed
0
Skip
52
Fail
2
Failing pages
https://docs.befailproof.ai/getting-started
https://docs.befailproof.ai/custom-policies
Summary
Verified 119 fenced code blocks across 19 public documentation pages on docs.befailproof.ai. The failproofai CLI (v0.0.10) was installed globally via npm to enable execution of CLI snippets, JSON config snippets were syntax-validated, and JavaScript custom-policy snippets were validated through failproofai policies --install --custom <file> (which loads and registers each hook in the validator).
65 PASS — All JSON config examples on /built-in-policies, /architecture, /configuration, /examples parse correctly. The CLI commands --version, -v, policies (list), policies --install <names>, policies --install all, policies --install --cli <claude|codex|copilot|cursor|opencode|pi|gemini>, policies --uninstall [...], policies --uninstall --custom, --hook PreToolUse/PostToolUse (with stdin), npm install [-g] failproofai, npm info failproof, and npx skills add https://docs.befailproof.ai[/...] all work as written. The 9 distinct JavaScript custom-policy files all load and register their hooks.
52 SKIP — Long-running dashboard server commands (failproofai, --port, env-var-prefixed launchers), bun-prefixed commands (bun runtime not installed), npm run dev commands that require the failproofai source repo, text/diagram blocks treated as expected output, TypeScript type signatures and test fixtures that are partial fragments (importing ../../src/... from the upstream repo), syntax-form lines containing literal placeholders (<EventType>, [policy-names...], ./my-policies.js, sk-...), the interactive policies --install prompt, and the bare claude REPL launch.
2 FAIL — Both failures are the same underlying docs bug: cp node_modules/failproofai/examples/convention-policies/*.mjs .failproofai/policies/ (in /getting-started) and the examples/convention-policies/*.mjs equivalent (in /custom-policies). The examples/ directory referenced by these snippets is not present in the published npm package — package.json lists files: ["bin/", "src/", "scripts/", "lib/", "pi-extension/", ".next/standalone/", "dist/", "README.md"] and the examples/ directory was confirmed absent after both global and local installs. Suggested resolution: either add "examples/" to the files array in the package manifest so the convention-policy starter files ship with the package, or update both docs pages to point users at the GitHub repo (e.g. curl -L https://raw.githubusercontent.com/exospherehost/failproofai/main/examples/convention-policies/...mjs) instead of node_modules/failproofai/examples/.... The public verifier does not apply or test fixes.
A minor docs-drift item observed but not classified as FAIL: /package-aliases instructs users to look for "ExosphereHost Inc." in the npm info failproof maintainers field; the live output instead lists niveditjain <nivedit@exosphere.host>. The command itself runs successfully (PASS); only the human-readable guidance is stale.
Required credentials
None. All verifiable snippets ran with placeholder values where env vars were referenced. Specifically:
SLACK_WEBHOOK_URL — supplied as the harmless placeholder https://hooks.slack.com/test-fake only to exercise the install-time validation of the Slack policy file; the webhook is not invoked at install time.
FAILPROOFAI_LLM_BASE_URL, FAILPROOFAI_LLM_API_KEY, FAILPROOFAI_LLM_MODEL — assigned with the documentation's own placeholder values (sk-..., etc.) to verify shell-syntax validity; no LLM call was made.
DEPENDENCY_FREEZE — referenced only inside a custom-policy body that never executed.
cp: cannot stat ...: No such file or directory. The examples/ directory is not declared in the package manifest's files array, so it is not shipped to node_modules/failproofai/. Suggested fix: add "examples/" to files in the package's package.json, or rewrite the doc snippet to fetch the starter files from the GitHub repo.
Validated 1 custom hook(s): confirm-branch-status (with allPushed defined locally so the file evaluates).
approval-gate transitive imports example
js
SKIP
Partial fragment — imports ./utils.js and ./approval-client.js, neither of which is provided in the docs. failproofai validator confirms the load error: Cannot find module '.../utils.js'.
Each variant tested individually: claude/codex/copilot/cursor/opencode/pi/gemini all install hooks into the corresponding .codex/hooks.json, .cursor/hooks.json, .gemini/settings.json, etc. The final aggregate --cli claude codex copilot cursor opencode pi gemini form also succeeds.
Partial fragment — <EventType> is a literal placeholder. (Verified separately that echo '{...}' | failproofai --hook PreToolUse and ... PostToolUse exit 0.)
Syntax form with literal brackets — partial fragment.
Examples block (8 invocations)
bash
SKIP
Block leads with failproofai policies --install which opens an interactive prompt without [names...]; cannot be executed as-is. Individual variants verified in the /configuration page row above and elsewhere.
npm install -g failproofai / # or / bun add -g failproofai (alternative install block)
bash
PASS
npm path verified. The # or separator marks the bun line as a documented alternative (not a sequential script); bun runtime is unavailable in this environment but the docs intent is clearly an either/or choice.
npm info failproof + # Look for: "ExosphereHost Inc." in the maintainers field
bash
PASS
Command runs successfully and prints the alias package metadata. Note (docs drift, not failure): the maintainers field currently shows niveditjain <nivedit@exosphere.host>, not the literal string ExosphereHost Inc. mentioned in the comment.
mkdir -p .failproofai/policies + cp examples/convention-policies/*.mjs .failproofai/policies/ (project and user variants)
bash
FAIL
cp: cannot stat 'examples/convention-policies/*.mjs': No such file or directory. The relative examples/ path resolves only inside the failproofai source repo and is not shipped to the npm package. Suggested fix: ship the directory (add "examples/" to files in package.json) and reference it as node_modules/failproofai/examples/..., or rewrite the snippet to download the starter files from the GitHub repo.