Roughly **99.9% of AI-crawler traffic is the gitweb interface**, up ~15× in twelve days, while actual fetching of posts *declined* over the same window. Meta/Facebook AI logged 69,074 requests and **zero** content pages in one day; GPTBot 37,122 requests for one page.
-This is permitted behavior, not abuse: `provisioning/robots.txt` is `Allow: /` with **no `Disallow` lines at all**, and git history is a near-infinite URL space (commits × files × view modes × blame/diff/raw). The cheap experiment is `Disallow: /blog/source` — the digests confirm ClaudeBot, Applebot, PerplexityBot, and OAI-SearchBot all reliably fetch robots.txt, so compliant bots would stop; whatever keeps hammering afterward is then a much clearer signal about who's ignoring it. Not done as of 2026-08-07. Weigh it against actually wanting the git history publicly crawlable, which is a real thing to want and the reason it's exposed.
+This is permitted behavior, not abuse: `provisioning/robots.txt` is `Allow: /` with **no `Disallow` lines at all**, and git history is a near-infinite URL space (commits × files × view modes × blame/diff/raw). The cheap experiment is `Disallow: /blog/source` — the digests confirm ClaudeBot, Applebot, PerplexityBot, and OAI-SearchBot all reliably fetch robots.txt, so compliant bots would stop; whatever keeps hammering afterward is then a much clearer signal about who's ignoring it. Not done as of 2026-08-07. Weigh it against actually wanting the git history publicly crawlable, which is a real thing to want and the reason it's exposed — see the next section, which is about not having to choose.
+
+### Wanted: crawlers see drafts and notes, but not the SHA cross-product (undecided, 2026-08-11)
+
+The blanket `Disallow: /blog/source` above is coarser than what we actually want. The *want* is asymmetric: unpublished writing in the working tree (`content/drafts/`, `notes/`) being legible to AI crawlers is a feature — a model knowing things about ZMD that aren't on the public site is the interesting part — while the every-file-at-every-SHA explosion is pure waste. Those are separable; the blanket rule is only necessary if gitweb is the delivery mechanism for both, and it isn't.
+
+**Drafts don't need gitweb at all — they're already served.** `Status: draft` posts (all of `content/drafts/`) build to `/blog/drafts/<slug>.html` via Pelican's default `DRAFT_SAVE_AS`, and `robots.txt` is `Allow: /`, so they are *already* crawlable today. What's missing is discovery: they're excluded from the index, feeds, and tag pages, and nothing else links to them, so no crawler has ever found one — **zero** `/blog/drafts/` fetches across every digest in `notes/ai_bot_digests/`. So the cheapest version of the whole want is "advertise `/blog/drafts/`, keep `Disallow: /blog/source` blanket," and no fine-grained gitweb rule is needed.
+
+They're also invisible to the LLM-legibility path specifically: `_prepare_markdown_mirrors` iterates `generator.articles`, which excludes drafts, so drafts get no `.md` mirror and no `llms.txt` entry. Extending it to `generator.drafts` is the direct fix — but note `article.markdown_url` is computed as `dirname(save_as) + '.md'`, and a draft's `save_as` is `drafts/<slug>.html` (one path segment, not the `YYYY/Mon/slug/index.html` shape), so that expression collapses every draft to a single `drafts.md`. Special-case it or the mirrors overwrite each other silently.
+
+**`notes/` is the part that genuinely needs gitweb**, since it isn't Pelican content and has no URL otherwise. Two ways in:
+
+* **Fine-grained robots.txt rules.** The obstacle is that gitweb here uses query-string URLs — *every* page is path `/blog/source` with `?p=…;a=…;f=…` — so path-prefix rules are all-or-nothing by construction. Distinguishing views requires matching against the query string, which Google and Bing document (along with `*`/`$` wildcards, codified in RFC 9309) but which is less uniformly implemented than plain prefixes. The precise, wildcard-free version: keep `Disallow: /blog/source`, then add one **exact** `Allow:` line per file, e.g. `Allow: /blog/source/?p=an_algorithmic_lucidity.git;a=blob_plain;f=notes/foo.md;hb=HEAD`. Longest-match `Allow` beats `Disallow` under the Google spec, `hb=HEAD` keeps the URL stable while the content underneath changes, and `blob_plain` skips the HTML chrome. Generating those lines means `robots.txt` stops being a hand-deployed static file (see the `STATIC_PATHS` comment in `pelicanconf.py`).
+* **Make the URL space path-shaped instead.** gitweb's `pathinfo` feature (`$feature{'pathinfo'}{'default'} = [1]`) yields `/blog/source/repo.git/blob_plain/HEAD:/notes/foo.md`, after which ordinary path prefixes do the whole job with no query-matching question. Cost: it changes every link gitweb generates, and the long `@stylesheets` comment in `provisioning/gitweb.conf` is premised on the query-string style (its whole argument is that gitweb never emits a `<base href>` — under pathinfo it does), so re-verify the CSS/JS still load.
+
+**Measurement gotcha:** `ai_bot_digest.py:313` drops the query string before counting, so the digests cannot tell which gitweb *views* are being hammered, and can't confirm afterward that an `Allow:` exception is being honored — they'd just show `/blog/source` going down. The raw access log keeps the full request line; checking the effect means grepping that, or teaching the digest to bucket by `a=` parameter.
+
+**Don't expect politeness headers to help with load.** gitweb already emits `<meta name="robots" content="index, nofollow"/>` on every HTML page (`/usr/share/gitweb/gitweb.cgi:4214`) and the trap still drew 312,894 requests in a day. That isn't proof of defiance — `nofollow` is about endorsement/ranking, not crawl budget — but it's the shape of the lesson: `blob_plain` responses aren't HTML and carry no meta tag at all, and an nginx `X-Robots-Tag` (which *could* key off `$args` via a `map`) governs indexing rather than fetching. **robots.txt is the only lever here that reduces requests.**
+
+### Sitemap as the draft-discovery mechanism (undecided, same thread)
+
+A sitemap is exactly the "advertise URLs nothing links to" mechanism, which is precisely the drafts' problem — so the missing `sitemap.xml` noted below is the natural home for this, listing `/blog/drafts/<slug>.html` alongside (or instead of) the published posts. Constraints:
+
+* **A sitemap can't override robots.txt.** A `Disallow`ed URL doesn't get fetched just because it's listed. Via `/blog/drafts/` that's a non-issue (already allowed); via gitweb blob URLs the exact `Allow:` lines above become mandatory, not optional.
+* **Placement is the same problem `llms.txt` has** — it must be at the true domain root, so it needs copying out of `output/` in `provisioning/pelican_scheduler.py` (or generating straight to the webroot), plus a `Sitemap:` line in `robots.txt`, which is reliably fetched.
+* **Decide whether drafts go in the same sitemap as published posts.** Listing them invites ordinary search indexing too, not just AI ingestion — a half-finished post ranking for its own title is a different outcome than a model having read it. If the want is read-but-don't-index, that's `X-Robots-Tag: noindex` on a `location /blog/drafts` block, with the caveat that training crawlers have no particular obligation to honor it (that's the point, here). A separate `drafts-sitemap.xml` keeps the two audiences distinguishable.
+* **Scope note:** any gitweb-side version of this exposes whatever else is in the working tree, not just the files aimed at — `notes/` currently holds crawler-digest session logs and a tag proposal alongside anything else that lands there. Enumerating files in `Allow:` lines is the version that stays deliberate; a `Disallow`-with-`f=notes/*`-hole is the version that quietly publishes the next thing added.
### `llms.txt` is generated where nothing will find it (unfixed, low priority)
projects / An_Algorithmic_Lucidity.git / commitdiff