The product is real and shipping (an MCP server, a model router, product pages, a security/compliance posture), but the documentation surface is fractured across three domains and two GitHub repos, the one file built for AI agents (llms.txt) points every product link tested at a 404, and the platform's headline compliance and data-access claims contradict each other depending on which page you land on. There is no public API reference despite the product exposing concrete, documented APIs in code, and the self-hosted router ships insecure-by-default secrets.
1. Every product link tested in llms.txt 404s — the agent-facing index is broken (critical)
Problem: The llms.txt## Docs block links to five product pages under /product/...: /product/ai-agent, /product/ai-insights, /product/code-output, /product/code-quality, and /product/code-review. The three tested (/product/ai-agent, /product/ai-insights, /product/code-output) all return HTTP 404 Not Found. The live pages exist under a different path prefix — /product-pages/ai-usage resolves, /product/... does not — so the remaining two almost certainly share the broken pattern.
Consequence:llms.txt exists specifically so AI coding agents can index docs efficiently. Here it does the opposite: an agent that follows the index hits 404s for the product capabilities it tries and silently concludes those features are undocumented or gone. Humans clicking from the index get the same dead ends. The single artifact meant to make the docs machine-readable is among the most broken things on the site.
The fix: Regenerate llms.txt from live routes (/product-pages/*), add a CI check that fails the build if any URL in llms.txt returns non-200, and add a 301 redirect from /product/* to /product-pages/* to catch existing inbound links.
2. "We do NOT access your source code" contradicts the product pages that measure your source code (critical)
Problem: The FAQ states flatly: "We do NOT access your actual source code or proprietary information." The AI Usage product page describes measuring "the exact portion of your codebase generated or modified by AI," attribution via "Git co-authorship" from "commit metadata," and correlating usage against "bug ratios and cycle times." The cursor-plugin README reinforces this with metrics like code_loc ("Lines of code changed"), ai_code_percentage ("% of code written with AI"), and ai_efficiency_index (a "Composite score (volume, usage, cost, churn)"); the pricing page lists "Code turnover." You cannot compute "% of code modified by AI" or lines-of-code churn without reading code or diffs.
Consequence: A security reviewer or procurement team reading the FAQ approves a tool they believe never touches source; a reviewer reading the product page assumes it ingests diffs. Both can't be right, and the gap is exactly the kind of question that kills an enterprise security review. Developers and buyers cannot determine what Weave actually reads.
The fix: Reconcile the claim. State precisely what is accessed (e.g., "commit metadata, diff statistics, and co-authorship trailers — not full file contents, retained as X") and remove the absolute "we do NOT access your source code" line if any diff/LOC analysis occurs. Put one authoritative data-access spec in the docs and link the FAQ and product pages to it.
3. Homepage and Enterprise page assert certifications the Security page never mentions (critical)
Problem: The homepage and Enterprise page assert as fact: "SOC 2 Type II Certified," "AICPA Type II Certified," "GDPR Compliant," "HIPAA Compliant." The dedicated Security page — the page a buyer goes to verify exactly this — lists policy categories ("Information Security and Governance," "Vulnerability Management," etc.) but never mentions SOC 2, AICPA, GDPR, or HIPAA at all. It is stamped "Last Updated: December 9th, 2024" (~18 months stale as of this audit). Meanwhile the FAQ hedges the same claim into a maybe: "SOC 2 Type II compliance (contact us for detailed compliance documentation)."
Consequence: Three different confidence levels for the same compliance posture — asserted fact (homepage), total silence (Security page), and hedged "contact us" (FAQ). A security team trying to verify SOC 2 finds the authoritative page silent and the marketing pages loud, which reads as unsubstantiated marketing and stalls the deal. HIPAA "Compliant" with no detail is an especially load-bearing claim to leave unsupported.
The fix: Make the Security page the single source of truth: list each certification with audit period, auditor, and how to request the report under NDA. Update the "Last Updated" date. Align the homepage, Enterprise page, and FAQ to whatever the Security page can substantiate — don't assert on marketing pages what the compliance page won't.
4. Router ships insecure-by-default secrets that the docs make easy to miss (critical)
Problem: Two security-critical defaults are documented as table rows that are trivial to skim past. ROUTER_ADMIN_PASSWORD "Defaults to admin" for the dashboard (username not documented). EXTERNAL_KEY_ENCRYPTION_KEY is unset by default, and "If unset, BYOK secrets are stored unencrypted" — meaning customer-supplied upstream provider keys sit in plaintext. Both emit only a startup WARN. The README's quickstart path ("No clone, no Docker, no Postgres") gets a user running before they'd ever read the configuration reference.
Consequence: A team that follows the fast path and deploys the router internet-facing exposes an admin dashboard protected only by the default password admin and stores other people's LLM provider keys unencrypted — a credential-theft jackpot. A startup WARN in logs nobody reads is not a control. This is the highest real-world risk in these docs, which is why it sits among the criticals.
The fix: Refuse to start (or auto-generate a random password) when ROUTER_DEPLOYMENT_MODE is internet-facing and ROUTER_ADMIN_PASSWORD/EXTERNAL_KEY_ENCRYPTION_KEY are unset, rather than warning. In the docs, add a prominent "Before you expose this" security checklist at the top of both the README and CONFIGURATION.md, not buried mid-table.
5. The product has at least three names and lives on three domains (significant)
Location: Across https://workweave.dev/, llms.txt, the FAQ, the Security/Subprocessors page titles, and the GitHub org/READMEs
Problem: The marketing site is workweave.dev; llms.txt self-describes the product as "Workweave.ai"; the FAQ says "Visit workweave.ai"; the app and all GitHub README links point to app.workweave.ai / www.workweave.ai; the GitHub org is workweave; the npm scope is @workweave; and the product itself is branded "Weave." The tagline is also unstable: the homepage leads with "Engineering intelligence for the AI era," while the Security and Subprocessors page titles use the variant "Weave - AI to measure AI." So the brand a developer must track is simultaneously WorkWeave, Workweave.ai, workweave.dev, and Weave, split across .dev (marketing) and .ai (app) domains with no page stating which is canonical.
Consequence: An agent or developer that grabs "workweave.dev" as the base domain and tries to reach the dashboard or API hits the wrong host (everything operational is on .ai). Link-building, API base-URL inference, and even trust signals ("is workweave.ai the same company as workweave.dev?") all break. This is the kind of ambiguity that makes automated tooling silently target the wrong origin.
The fix: State the canonical domain split explicitly in the docs and llms.txt ("marketing: workweave.dev, app/API: app.workweave.ai"), fix the llms.txt self-description to match the actual domain, and standardize one product name and one tagline across pages.
6. No public API reference, yet concrete APIs and 40+ metrics are documented only in a repo named "cursor-plugin" (significant)
Problem: The FAQ answers "Does Weave offer an API? Yes - contact our technical team for API documentation." There is no API reference anywhere on the docs site, and llms.txt's ## Docs section is ~75 blog/marketing links with zero reference material. Yet the cursor-plugin README already documents four concrete MCP tools (get_metric_overview, get_metric_drill_down, get_accounts, get_teams) and 40+ named metrics (code_output, pr_cycle_time, ai_code_percentage, ai_efficiency_index, output_per_ai_dollar, …). The Weave Router exposes /admin/v1/* endpoints in its config reference. The API exists and is documented — just not where anyone or any agent would look, and behind a repo whose name ("cursor-plugin") doesn't match its GitHub description ("engineering analytics MCP server").
Consequence: Developers are told to email for docs that are already public on GitHub; agents indexing workweave.dev find no machine-readable endpoint or metric catalog and can't programmatically discover anything. The richest reference Weave ships is effectively hidden by naming and by absence from the docs index.
The fix: Publish the MCP tool list, the 40+ metric definitions, and the router /admin/v1/* endpoints as a first-class reference on the docs site; link it from llms.txt; and change the FAQ answer to point at that reference instead of "contact our technical team."
7. Multiple unrelated API-key schemes, no unified auth doc (significant)
Problem: The cursor-plugin uses keys formatted wkey_abc123... (placeholder YOUR_WEAVE_API_KEY), obtained from app.workweave.ai/organization/settings. The Router uses a different model entirely: its own router keys (documented as rk_...) plus customer-supplied upstream provider keys (sk-/sk-ant-...) that the EXTERNAL_KEY_ENCRYPTION_KEY is meant to encrypt at rest. There is no single page that explains which key is which, where each comes from, what scope each has, or whether a wkey_ and a router key are related. Each repo documents its own keys in isolation.
Consequence: A developer using both Weave products has to reverse-engineer two unrelated auth models from two READMEs. An agent generating a request can't tell which token format a given endpoint wants and will paste a wkey_ where a router/provider key is required (or vice versa) and get an opaque auth failure.
The fix: Publish one authentication page covering every key type (wkey_ analytics keys, the router's own keys, and sk-/sk-ant- BYOK provider keys): what each authorizes, where to generate it, format, and rotation/expiry. Cross-link both READMEs to it.
8. On-prem and data residency are "contact us" in the FAQ but "available" facts on the Enterprise page — and the Subprocessors page already answers them (significant)
Problem: The FAQ treats deployment and residency as open questions: on-prem is "Contact our team to discuss," and "Where is data stored? Contact our team for specific information about data residency." The Enterprise page asserts the answers as shipping features: "Air-gapped deployment available," "Full data sovereignty," "Your data never leaves your network." Meanwhile the Subprocessors page already discloses residency concretely — every subprocessor (Google Cloud, GitHub, Stripe, PostHog, etc.) lists "Country of location: USA."
Consequence: A prospect asking the two highest-stakes enterprise questions — "can we self-host air-gapped?" and "where does our data live?" — gets "contact us" from the FAQ, "yes, available" from Enterprise marketing, and a concrete USA-only answer from the Subprocessors page, with no way to know which is authoritative. For non-US customers with data-residency requirements, the unacknowledged US-only subprocessor footprint vs. the "full data sovereignty" claim is a direct conflict.
The fix: State data residency plainly in the FAQ and Security page ("primary processing in the USA; see Subprocessors"), and qualify the Enterprise "air-gapped / data sovereignty" claims with availability conditions (which plan, GA vs. roadmap) so they match the FAQ's caution.
9. FAQ and the cursor-plugin README list different integrations (significant)
Problem: The FAQ says Weave connects "your development tools (GitHub, Cursor, Claude, and other AI tools)." The cursor-plugin README's Requirements section says you need "A Weave account with connected data sources (GitHub, GitLab, Linear, Jira, etc.)." The two lists barely overlap — the FAQ names Cursor and Claude (AI tools) and the README names GitLab, Linear, and Jira (project/SCM tools), with only GitHub in common. Neither page presents a single authoritative integrations list.
Consequence: A buyer evaluating fit can't tell which integrations actually exist. A team on GitLab + Jira reads the FAQ (which never mentions either) and assumes Weave won't work for them; a team expecting Claude support reads the README (which never mentions it) and concludes the opposite. There is no canonical "supported integrations" page to settle it.
The fix: Publish one integrations reference listing every supported data source with its capability level, and have both the FAQ and the README link to it instead of each maintaining its own partial list.
10. The only install step for the public MCP server points to a generic marketplace root (significant)
Problem: Setup step 1 reads: "Install the plugin from the Cursor Marketplace." The link is a generic marketplace root with no plugin slug, no search term, and no package identifier — nothing that points to the actual "weave" plugin. This is the first onboarding step for the only public developer artifact Weave ships.
Consequence: A developer following the documented path lands on a generic page (if it resolves at all) with no way to locate the specific plugin, and an agent following the link can't extract an installable target. Onboarding stalls at step 1 of the one tool that's actually documented for developers.
The fix: Link directly to the plugin's marketplace listing (deep link or exact slug), or document the precise install command / search term. Add a link-check so the install step can't silently rot to a generic URL.
11. The Pro-plan agent appears under conflicting names (minor)
Problem: The pricing page lists the Pro-tier feature as "Wooly AI Agent." The llms.txt index and its linked blog entry refer to the same agent capability as "Mycroft" ("Mycroft integration and use cases," slug /blog/mycroft). So the agent is "Wooly" where you choose a plan and "Mycroft" where it's explained — two unrelated codenames for what appears to be one feature, with no page introducing one as an alias of the other.
Consequence: A developer comparing plans reads "Wooly AI Agent" under Pro, then finds only "Mycroft" in the docs index, and can't tell whether these are the same feature, two features, or whether the agent they're paying for is documented at all. Agents indexing the docs will treat "Wooly" and "Mycroft" as distinct products.
The fix: Pick one canonical product name and use it everywhere, introducing any codename ("Wooly," "Mycroft") once as an explicit alias on first mention.
12. Pricing and product pages render their content two-to-three times (minor)
Problem: The full three-tier pricing table renders three times back-to-back on /pricing. On the AI Usage product page, the H1 ("See exactly how your team uses AI coding tools.") and the content blocks are duplicated verbatim. These are rendering defects, not intentional repetition.
Consequence: Humans see a page that looks broken and lose trust at the exact moment they're evaluating cost; an agent extracting the pricing tiers or feature list ingests every value two-to-three times and may double-count or report conflicting duplicates.
The fix: Fix the component/loop emitting duplicate nodes so each section renders once.
Problem: The configuration reference shows placeholder paths where a version segment has collapsed away: the Docker build downloads assets into /opt/router/assets// (note the empty //), and the committed cluster artifacts "live under internal/router/cluster/artifacts/v/" (empty segment after v). These read like angle-bracket <version> placeholders that were stripped during Markdown rendering, leaving nothing to signal that a value is missing.
Consequence: A developer or agent copying /opt/router/assets// or artifacts/v/ constructs an invalid path with a missing version segment, and the command fails with no indication of what was supposed to go there. Empty placeholders are worse than labeled ones because nothing flags the gap.
The fix: Replace stripped placeholders with explicitly marked tokens that survive Markdown rendering (e.g., /opt/router/assets/{version}/, artifacts/{version}/), escaping angle brackets so they render.
14. README and its own config reference disagree on the quickstart's name (minor)
Problem: The README titles its setup section "30-second quickstart." CONFIGURATION.md, referring to that same section, calls it "the 60-second quickstart."
Consequence: Trivial on its own, but it's a direct cross-document contradiction about the same section — a small tell that the two docs aren't reviewed together, which undermines confidence in the rest of the cross-references.
The fix: Standardize the name in both files (and ideally drop the time-claim).
What they do well
Honest about attribution fidelity. The AI Usage page lays out a clear hierarchy — high-fidelity API data → Git co-authorship → time-series fallback "clearly labeled as estimated" — which is a more candid description of measurement confidence than most analytics vendors offer.
Subprocessors page is concrete and complete. Every processor is named with country and purpose, which (ironically) answers the residency question the FAQ ducks.
The Router config reference is genuinely thorough — it documents env vars, defaults, and security implications (encryption, admin password) in one place, even if those defaults need to fail closed.
Top 3 recommendations
Fix the agent-facing surface first: regenerate llms.txt against live routes (kill the /product/* 404s), add a link-checker to CI, and publish the already-existing MCP tools, 40+ metrics, and router endpoints as a real API reference linked from llms.txt.
Reconcile the contradictions that block enterprise sales: what data Weave accesses (source code yes/no), which compliance certifications are real (make the Security page authoritative), whether on-prem/air-gapped/residency is shipping or "contact us," and which integrations actually exist.
Unify brand and auth, and harden the router: declare the canonical domain split (workweave.dev marketing vs. app.workweave.ai app/API), pick one product and one agent name, publish a single authentication page covering wkey_, router, and sk- keys, and make the router's admin-password/unencrypted-BYOK defaults fail-closed.
Fixed
0
Skip
6
Fail
0
Summary
Verified every public Workweave source in scope. Runnable code lives only in the two GitHub repos (workweave/router README and its docs/CONFIGURATION.md). The marketing/site pages (homepage, /product-pages/ai-usage, /security, /enterprise, /pricing, /subprocessors, the FAQ blog post, and llms.txt) contain no executable code, and the cursor-plugin README documents only GUI setup steps plus natural-language prompts — nothing to run.
PASS (1): the CONFIGURATION.md Jina-embedding asset fetch downloaded both public HuggingFace files as documented (162 MB ONNX with a valid onnx.quantize header; 2.5 MB valid-JSON tokenizer).
SKIP (6): blocked by environment/policy, not by docs defects — localhost:8080 curl examples (private-network policy), make-based self-host/install steps (require cloning the repo; make absent), tinkey keyset generation (tool absent), and the npx @workweave/router installer + flag variants (need an interactive TTY and jq, both documented as required and both unavailable here).
The npx @workweave/router quickstart did resolve and run: it installed @workweave/router@0.2.2 from npm and launched its interactive installer exactly as written, stopping only on the documented jq dependency and the lack of a TTY. The package and the pinned @0.1.0 version both exist on npm, so the command itself is valid; I just could not complete the interactive, account-gated flow.
Note (not code-execution findings, surfaced for context): llms.txt links every product page under /product/*, which 404 — the live pages are under /product-pages/*. None of those are runnable snippets, so they are not scored above.
Required credentials
None were supplied, and none were needed for the one snippet that passed (public HuggingFace assets). The skipped snippets reference these credential names/placeholders only (no values used or printed):
rk_... — Weave Router key (Bearer token for the localhost:8080 curl examples).
OPENROUTER_API_KEY (sk-or-v1-...), ANTHROPIC_API_KEY (sk-ant-...), OPENAI_API_KEY (sk-...), GOOGLE_API_KEY — upstream provider keys for the self-host path.
ROUTER_ADMIN_PASSWORD, EXTERNAL_KEY_ENCRYPTION_KEY — router admin/BYOK config referenced in CONFIGURATION.md.
wkey_... — Weave API key for the cursor-plugin MCP setup (GUI-only; no runnable snippet).
Resolved + installed @workweave/router@0.2.2 and launched the installer as written, then exited on jq is required but not installed and missing /dev/tty. Both are documented prerequisites ("Claude Code and opencode paths also need jq") and are unavailable in this non-interactive environment. Package valid; flow is interactive/account-gated, so cannot be completed here.
Same interactive installer entrypoint as above; needs a TTY, jq, and a Weave account/router key. Pinned @0.1.0 confirmed to exist on npm (latest is 0.2.2).
echo "OPENROUTER_API_KEY=sk-or-v1-..." >> .env.local + make full-setup (self-host)
bash
SKIP
make is not installed and the target requires the cloned repo (Postgres + build); cloning is out of scope for this verifier.
tinkey runtime not available in this environment; cannot execute as written.
Jina embedder asset fetch: mkdir -p + two curl -L (model + tokenizer) + echo ... >> .env.local
bash
PASS
Executed as written. model.onnx = 161,895,621 bytes with valid ONNX protobuf header (onnx.quantize...); tokenizer.json = 2,561,316 bytes, valid JSON; .env.local written with ROUTER_ONNX_ASSETS_DIR. Both pinned-revision HuggingFace URLs returned HTTP 200.
Setup ("Install from Cursor Marketplace", paste wkey_... in Cursor Settings) + example chat prompts
n/a
—
No runnable code: GUI-only setup steps and natural-language prompts. No code block to execute; not counted.
https://workweave.dev/ and other site pages (/product-pages/ai-usage, /security, /enterprise, /pricing, /subprocessors, FAQ blog, /llms.txt); dead /product/ai-agent, /product/ai-insights, /product/code-output
Snippet
Lang
Status
Notes
(none)
n/a
—
No executable code blocks on any marketing/reference page or in llms.txt. The /product/* paths return HTTP 404 (live equivalents are /product-pages/*), but these are links, not runnable snippets; not counted.