Repository navigation
AR-20261009-1: emit truthful docs sitemap lastmod - #882
warp-agent-staging[bot] wants to merge 4 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
This PR was generated with Warp. Comment |
|
Autoresearch: AR-20261009-1 (docs half) is ready for review.
|
|
@warp-agent-staging[bot] I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
There was a problem hiding this comment.
Overview
This PR replaces the default Astro sitemap integration with a wrapper that emits lastmod from rendered Starlight article:modified_time metadata, restores full git history for build environments when possible, omits shallow-boundary dates when history cannot be restored, and adds CI coverage for the new behavior.
Concerns
- No blocking correctness, security, comment-quality, test-quality, or spec-alignment concerns found in the changed lines.
spec_context.mdcontains no approved or repository spec context to validate against.
Verdict
Found: 0 critical, 0 important, 0 suggestions
Approve
Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
Summary
Experiment: AR-20261009-1 (docs half)
Emit per-URL sitemap lastmod from the exact git-derived date Starlight renders. The serializer reads article:modified_time from generated HTML; it does not guess source paths or use build time. Pages without dates omit lastmod. CI now checks out full history and runs regression tests; Vercel reuses the existing Starlight git-date source.
Hypothesis: truthful content-change timestamps help Bing prioritize fresh clean documentation URLs, improving their availability to ChatGPT retrieval alongside the separate www IndexNow experiment.
Metric: share of ChatGPT retrievals of docs Factories URLs using the clean (non-?trk) URL, Peec 14-day windows; baseline 2 of 167 (1.2%), Sep 25–Oct 8, across 7 Factories URLs. Fast metric: ora sitemap-lastmod fail (0/1) → pass, at least 95% URL coverage with varied dates.
Prediction: ora passes immediately; clean-URL share rises toward >=10% within 4 weeks of merge combined with www IndexNow (directional; verify Bing indexing separately).
Related issues
2026-10-09 AEO autoresearch brief.
Validation
How verified:
npm ci;npm run typecheck && npm run test:heading-vars && npm run test:sitemap-lastmod && npm run build && npm run test:heading-tocs && npm run test:homepage-json-ld: passed, 0 type errors, 5 heading tests, 3 lastmod/history tests, 3,353 TOC entries across 396 HTML files, JSON-LD valid..vercel/output/static/sitemap-0.xml: 394/395 (99.75%) lastmod coverage, 79 distinct timestamps; /api/ omitted. Factories quickstart matches Starlight metadata (2026-10-06T22:10:23Z), not build time.python3 .agents/skills/style_lint/style_lint.py --changedandpython3 .agents/skills/validate_ui_refs/validate_ui_refs.py --changed --require-provenance: no findings (no changed documentation pages).npm run lint/npm run fmt: unavailable because Trunk is not installed;git diff --checkpassed.curl -fLsS https://docs-git-factory-ar-20261009-1-docs-lastmod-warpdotdev.vercel.app/sitemap-0.xml -o preview-sitemap.xml: final public output has 394/395 URLs dated (99.75%), 79 distinct timestamps; /api/ correctly omits lastmod.git log --format=%ct -- src/content/docs/<source>.mdx(maximum timestamp, exactly Starlight's build-date selection). Five samples match:npx -y @ora-ai/ax@0.5 audit https://docs.warp.dev --json > prod.json: canonical 62/C. Raw preview 55/C reflects preview-host identity/search/SDK discovery differences; the orchestrator approved unchanged Automated SEO fixes: lengthen short API errors title and trim long description #872 as control.npx -y @ora-ai/ax@0.5 audit https://docs-r7655tvgt-warpdotdev.vercel.app --json > control.jsonandnpx -y @ora-ai/ax@0.5 audit https://docs-git-factory-ar-20261009-1-docs-lastmod-warpdotdev.vercel.app --json > preview.json; timestamps 2026-10-09T18:15:42.351Z / 2026-10-09T18:10:47.97+00:00. All 125 score/status pairs match; no regressions. No forced scans.ora sitemap-lastmod verifiable only post-merge: unchanged robots.txt and sitemap-index.xml pin sitemap URLs to production, so ora reads production's undated XML on preview hosts. G2 target evidence is the fetched final preview XML, not a claimed ora pass.
Shallow-history and build-cost verification
Vercel's shallow clone initially yielded only 7 distinct dates, falsely treating old unchanged pages as modified at the clone horizon; it also makes production's visible “Last updated” dates inaccurate. Prebuild now tries to restore history (origin, or public GitHub URL from Vercel owner/slug metadata if origin is absent), nonfatally. If unavailable, clone-horizon dates are omitted from metadata, sitemap and visible last-updated line; explicit frontmatter dates are preserved.
git clone --depth 10 --branch factory/ar-20261009-1-docs-lastmod https://github.com/warpdotdev/docs.git /tmp/docs-shallow-repro && npm --prefix /tmp/docs-shallow-repro ci && npm --prefix /tmp/docs-shallow-repro run build: passed; unshallow 1.1s, 394/395 coverage, 79 distinct dates. Earlier network measurement: 22.28 MiB, 8,597 objects, 1.2s. Full-history checkout has no network fetch.git clone --depth 10 --branch factory/ar-20261009-1-docs-lastmod file:///workspace/docs /tmp/docs-no-origin-repro && git -C /tmp/docs-no-origin-repro remote remove origin && npm --prefix /tmp/docs-no-origin-repro ci && npm --prefix /tmp/docs-no-origin-repro run build: passed without remote/metadata; remains shallow, omits horizon dates (9/395 URLs, 4 distinct genuine dates).VERCEL_GIT_REPO_OWNER=warpdotdev VERCEL_GIT_REPO_SLUG=docs npm --prefix /tmp/docs-no-origin-repro run build: public fallback passed, unshallow 1.1s, restored full-history output.Rollback: revert this PR; sitemap returns to undated entries without affecting page content or URLs.
Follow-ups
Slow metric and Bing indexing require post-merge monitoring.