_Published alongside the study "can you vibecode ahrefs?". Timestamps as written; unedited apart from removing server paths, hostnames and port numbers for publication. Where this file compares the clone against "193" referring domains, that was Ahrefs' count for canivibecodeit.com on 7 August 2026; the study itself uses the count on Rob's screen on 9 September 2026, 346._

# Ahrefs clone attempt — LOG

- 2026-09-07 17:56 UTC — START. Time box ends 2026-09-07 23:56 UTC. Brief read in full. Scratch folder: the project folder/
- 17:59 UTC — Env: no pip on system python; bootstrapped pip into a venv via get-pip.py (no system packages installed). Installed flask, requests, bs4, lxml, pytrends.
- 17:59 UTC — Source probes so far: HN Algolia OK (1 hit: HN submission of canivibecodeit.com, 2026-08-01). Wikipedia exturlusage: 0 results for canivibecodeit.com. Wayback CDX OK (first capture of canivibecodeit.com 2026-07-30). Common Crawl CDX index OK (site present in CC-MAIN-2026-34, crawled 2026-08-10). DuckDuckGo html/lite via curl: bot-blocked ("anomaly"/"challenge"). Reddit search JSON via curl: "Blocked". Google suggest endpoint (suggestqueries.google.com) OK. DDG autocomplete OK. Bing HTML via curl returns a page with b_algo results (parsing pending).
- 17:59 UTC — Found Common Crawl web graph cc-main-2026-jun-jul-aug (domain level: 119.7M nodes, 2.45B arcs). Files: domain-vertices 0.89GB, domain-edges 9.43GB, domain-ranks 2.36GB. Measured download ~17MB/s -> edges ~10min. Decision: download all three into data/ and scan locally for in-links to target domains. This is the primary backlink source.
- 17:59 UTC — Note: previous three entries were stamped 18:07 by mistake; corrected to actual clock time 17:59 (all times UTC, from `date -u`).
- 18:00 UTC — DuckDuckGo blocks this server IP even in headless Chromium (duck-picking CAPTCHA, screenshot probes/ddg-browser.png). Not solving CAPTCHAs; DDG dropped as a SERP source. Bing via curl returns a degraded bot SERP (junk/irrelevant results); testing Bing in a real browser next.
- 18:02 UTC — Search-engine probes from this server IP: Brave = CAPTCHA, Mojeek = CAPTCHA, Qwant = DataDome JS challenge, Yandex = empty response, DDG = CAPTCHA (browser too). Bing responds via curl, headless browser, and legacy RSS (format=rss) but the results look degraded: "notion alternatives" returns only notion.com pages; "can i vibecode it" returns dictionary entries for the word "can". Testing a human-like Bing flow (homepage -> type -> Enter) to compare.
- 18:02 UTC — Google Trends (pytrends, after dropping the urllib3 Retry kwarg): interest_over_time works for the 3 test keywords US/12-month (relative 0-100 index, NOT a volume). related_queries hit HTTP 429 immediately. Will use interest_over_time only, cached in SQLite.
- 18:02 UTC — Decision: Startpage / SearXNG proxies of Google results are NOT used; they are Google SERPs by another door and the brief forbids Google scraping.
- 18:04 UTC — All 3 CC graph files downloaded (12.7GB). Target vertex ids: canivibecodeit.com=24457970, ahrefs.com=15154848, granola.ai=219818, notion.so=110257719, notion.com=53473819. Starting full edges scan (grep on 2.45B arcs) for in-links to these ids.
- 18:10 UTC — Edge scan finished in 4.5 min (18:04:57→18:09:29). Referring-domain counts in cc-main-2026-jun-jul-aug: ahrefs.com (15154848): 116776 referring domains;notion.so (110257719): 51150 referring domains;notion.com (53473819): 16867 referring domains;granola.ai (219818): 1675 referring domains;canivibecodeit.com (24457970): 5 referring domains;
- 18:11 UTC — Triggered real backlink scans through the dashboard for canivibecodeit.com and ahrefs.com (the well-known site I chose; Ahrefs publishes its own numbers so the gap is checkable).
- 18:12 UTC — Dashboard up on the local dashboard (pidfile data/app.pid). Mishap: my first restart used pkill -f on a pattern that also matched my own shell and killed the turn; switched to a pidfile + port check. Both dashboard-triggered scans (canivibecodeit.com, ahrefs.com) are running (vertex lookup took 4s with grep -m1 + page cache).
- 18:13 UTC — Second rank check run manually (track.py) so the history strip has two data points; in real use it runs daily from cron. Result unchanged: canivibecodeit.com not in Bing top 10 for any of the 10 keywords.
- 18:15 UTC — canivibecodeit.com referring domains in the CC graph: zoer.ai, alobbs.com, microsaasexamples.com, mattiarighetti.net, superx.so (5). Added a Common Crawl CDX+WARC fallback (clone/cc_index.py) to find an actual linking page when the homepage does not carry the link: found https://www.microsaasexamples.com/p/can-i-vibecode-it (crawled 2026-08-19) in 3.4s; zoer.ai: no linking page in the 25-page sample.
- 18:19 UTC — Scans done: canivibecodeit.com 5 referring domains (18:12→18:17), ahrefs.com 116,776 (18:12→18:17). Two fixes: (1) stored sample for big domains was the first 5000 ids, not the most authoritative — now resolve all ids, rank by CC harmonic centrality, store top 5000; (2) CC index fallback found 0 inside the app because index.commoncrawl.org rate-limits (429/503) — added backoff. Re-running the ahrefs.com scan through the dashboard.
- 18:20 UTC — index.commoncrawl.org started refusing connections from this IP mid-run (this is why the in-app CC-index fallback found 0). Added connection-error backoff. App restarted; ahrefs.com scan re-triggered with the authority-ranked sampling.
- 18:26 UTC — index.commoncrawl.org reachable again at 18:23:37 (refused 18:18–18:23). Re-ran the CC-index linking-page fallback for canivibecodeit.com.
- 18:32 UTC — ahrefs.com re-scan done (18:20→18:27): 116,776 referring domains, 8,282 of them inside the CC top-1M ranking; stored top 5,000 by authority (google.com, linkedin.com, cloudflare.com, wikipedia.org, wordpress.org, github.com ...). Live homepage probe of the top 40 found the link on none of the homepages (expected: links live on inner pages). index.commoncrawl.org still refusing connections (000) at 18:29, so no CC-index linking pages for either domain in the app; the one standalone hit at 18:15 (microsaasexamples.com/p/can-i-vibecode-it, crawled 2026-08-19) stands as the only page-level evidence. App restarted with in-process daily scheduler (06:00 UTC).
- 18:34 UTC — REPORT.md written.
- 18:37 UTC — index.commoncrawl.org still refusing connections (down since 18:18). Finalising. END OF ATTEMPT at 18:37 UTC; ~45 min wall-clock used of 6h.
- 18:39 UTC — Addenda received (read queue + brief bottom): app renamed vibehrefs, home the project folder/, a process-manager entry named vibehrefs on the local dashboardxx, local git only, Ahrefs product screens as design reference (PM already saved them to the study's research folder/), screenshot every wall with timestamps, report to parent AND PM. Box unchanged (ends 23:56). Applying now; REPORT.md will be rewritten at the end.
- 18:40 UTC — Design references copied to the project folder/reference/ (9 files, source URLs in reference/SOURCES.md; all from ahrefs.com product/help/blog pages, saved by the PM on 2026-09-07).
- 18:42 UTC — Wall screenshots taken (shots/wall-YYYYMMDD-HHMM-*.png): Common Crawl index refused, Wikipedia exturlusage empty, Reddit blocked, Brave CAPTCHA, DuckDuckGo CAPTCHA, Bing degraded RSS for 'notion alternatives', plus a rendered copy of the terminal-only walls (Trends 429, Bing pagination, index refusal).
- 18:44 UTC — Addenda applied: vibehrefs at the project folder/ (git: 2 commits, local only), the process manager on the local dashboard, Ahrefs-style UI, re-screenshotted (old UI kept as *-v1-generic-ui.png), wall screenshots taken, REPORT.md §8 added. Final report sent to parent + PM.
- 18:46 UTC — index.commoncrawl.org back at 18:44 (refused 18:18–18:44). Fallback re-run for the 3 unchecked domains: 2 linking pages found (microsaasexamples.com/p/can-i-vibecode-it crawled 2026-08-19, Wayback first capture 2026-08-22; mattiarighetti.net/just-because-you-can-it-doesnt-mean-you-should/ crawled 2026-08-15, not in Wayback); superx.so none. Code now takes the earlier of CC capture and Wayback first capture (commit 3). Backlinks screenshot retaken. REPORT.md updated.
- 17:11 UTC — PM re-run request: superx.so backlink scan, 'calendly alternatives' keyword, re-shoot ahrefs.com/superx.so/calendly/granola into the study's assets folder/, append 'Re-run for the final figure set' to REPORT.md. Same app, same method, no new features.
- 17:27 UTC — Re-run done: superx.so 299 referring domains (0 linking pages: CC index rate-limited all 11 attempts), calendly alternatives 91 ideas / Trends 429 then OK on retry / KD proxy 23; screenshots into the study's assets folder/; REPORT.md §9 appended and copied to assets/clone-REPORT.md.