From fc683b02b919d67202189e15e1ea63bd9264c593 Mon Sep 17 00:00:00 2001 From: "Zack M. Davis" Date: Wed, 5 Aug 2026 16:55:24 -0700 Subject: [PATCH] provisioning updates: server-wide charset, documenting root pages --- CLAUDE.md | 12 ++++++++++ content/extra/robots.txt | 3 --- provisioning/nginx_siteconf | 47 +++++++++++++++++++++++-------------- 3 files changed, 41 insertions(+), 21 deletions(-) delete mode 100644 content/extra/robots.txt diff --git a/CLAUDE.md b/CLAUDE.md index d7cae69..15e1fec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,6 +14,10 @@ The blog is a DigitalOcean VPS. `provisioning/` holds the server's config, but n | `ai_bot_digest.py` | `/usr/local/bin/ai_bot_digest` | | `systemd/*` | `/etc/systemd/system/` | | `pelican_scheduler.py` | symlinked as the bare repo's `hooks/post-receive` | +| `root_index.html` | `/home/blogmistress/zackmdavis.net/index.html` — **renamed on the way over**; scp'ing it under its repo name lands a dead file beside the real one and changes nothing visible | +| `robots.txt` | `/home/blogmistress/zackmdavis.net/robots.txt` (the true domain root, not `/blog` — see the `STATIC_PATHS` comment in `pelicanconf.py` for why Pelican deliberately doesn't generate it) | + +The last two live under the webroot — `root /home/blogmistress/zackmdavis.net` in `nginx_siteconf`, which is also what serves `/docs` and `/media`. Being inside `blogmistress`'s own home, they're the only two deployable without root; `install` as root would leave them root-owned. After nginx edits, `nginx -t && systemctl reload nginx` — `nginx -t` catches a `conf.d` file that didn't land, since the site config references things defined there. After unit edits, `systemctl daemon-reload` **and** `systemctl restart ai-bot-digest.timer`; a daemon-reload alone leaves an already-scheduled timer on its old schedule. @@ -54,3 +58,11 @@ The fix, when it's worth doing: build into a sibling directory and flip a symlin ### AI-crawler observability `provisioning/ai_bot_digest.py` runs daily via systemd and files `-.txt` into `/var/log/ai-bot-digest/`, summarizing which crawlers fetched which posts. Its own docstring covers usage and the second-site story. Two structural limits worth knowing before trusting it: User-Agents are forgeable (it quarantines UAs whose traffic is ≥60% 404s as likely impostors), and Google/Apple AI-training use is invisible in principle, since `Google-Extended`/`Applebot-Extended` are robots.txt tokens that no request carries. + +### `llms.txt` is generated where nothing will find it (unfixed, low priority) + +`_write_llms_txt` in `pelicanconf.py` emits it into the Pelican output tree, so it lands at `/blog/llms.txt` — but `llms.txt` is a domain-root convention like `robots.txt`, so a crawler that goes looking checks `zackmdavis.net/llms.txt` and gets a 404. Nothing links to the real one either, from `robots.txt` or from any page. It has **zero fetches** across every digest in `notes/ai_bot_digests/` (2026-07-25 through 2026-08-03) — the file is currently dead weight. + +Note this means the heavy `.md` fetching visible in those digests is *not* evidence llms.txt works: the mirrors are reachable from the HTML anyway, via the `` in `theme/templates/article.html` and the visible "Markdown source" link in `theme/templates/includes/post_card.html`. + +The fix is placement, not content — `_canonical_url` already builds absolute `https://zackmdavis.net/blog/...` links, so a copy served from the webroot works unmodified. It's the same problem `STATIC_PATHS` in `pelicanconf.py` already solves for `robots.txt` by deploying that standalone, but `llms.txt` is build-generated, so it needs copying out of `output/` after each build (`provisioning/pelican_scheduler.py`) rather than a one-time `scp`. While in there: there's no `sitemap.xml` at all and no `Sitemap:` line in `robots.txt`, which is the same missing-machine-discovery-surface problem — robots.txt *is* reliably fetched (ClaudeBot, Applebot, PerplexityBot, OAI-SearchBot all hit it), so it's the better hook of the two. diff --git a/content/extra/robots.txt b/content/extra/robots.txt deleted file mode 100644 index cb10b4b..0000000 --- a/content/extra/robots.txt +++ /dev/null @@ -1,3 +0,0 @@ -User-Agent: * -Content-Signal: search=yes, ai-input=yes, ai-train=yes -Allow: / diff --git a/provisioning/nginx_siteconf b/provisioning/nginx_siteconf index a3dfbcb..6d0a42b 100644 --- a/provisioning/nginx_siteconf +++ b/provisioning/nginx_siteconf @@ -56,6 +56,32 @@ server { root /home/blogmistress/zackmdavis.net; index index.html; + # .md is served as text/markdown via a `text/markdown md;` line added + # to the OS-managed mime.types on the server (not part of this repo). + # mime.types can't carry a charset parameter, though, and without one + # a client handed "Content-Type: text/markdown" falls back to its own + # default (typically windows-1252) and renders every em dash and curly + # quote as mojibake -- there's no for it to recover + # from, the way there is in HTML. So declare it here. (Distinct from + # the gitweb case in provisioning/gitweb.conf: there CGI.pm attached a + # *wrong* charset; here nginx attaches none at all.) + # + # Deliberately at server scope rather than inside location /blog/, + # where it originally lived: .md files are served from /docs too (the + # diary entries), and those are NOT under the Pelican tree, so a + # /blog/-scoped charset never reached them. Locations inherit this + # unless one overrides it, so /blog/ is unaffected by the move. + # + # charset_types REPLACES the default list rather than extending it, so + # the stock entries are restated alongside text/markdown; only + # text/html is implicit and can be omitted. Since these are static + # files that carry no charset of their own, nginx just appends -- no + # recoding is attempted. + charset utf-8; + charset_types text/markdown text/xml text/plain + text/vnd.wap.wml application/javascript + application/rss+xml; + location / { try_files $uri $uri/ =404; } @@ -122,24 +148,9 @@ server { # served from /blog/ (e.g. llms.txt, from _write_llms_txt in # pelicanconf.py) as text/markdown too, not just .md files. # - # mime.types can't carry a charset parameter, though, and without - # one a browser handed "Content-Type: text/markdown" falls back to - # its own default (typically windows-1252) and renders every em - # dash and curly quote in a .md mirror as mojibake -- there's no - # for it to recover from, the way there is in HTML. - # So declare it here. (Distinct from the gitweb case in - # provisioning/gitweb.conf: there CGI.pm attached a *wrong* - # charset; here nginx attaches none at all.) - # - # charset_types REPLACES the default list rather than extending it, - # so the stock entries are restated alongside text/markdown; only - # text/html is implicit and can be omitted. Since these are static - # files that carry no charset of their own, nginx just appends -- - # no recoding is attempted. - charset utf-8; - charset_types text/markdown text/xml text/plain - text/vnd.wap.wml application/javascript - application/rss+xml; + # The utf-8 charset those .md mirrors need is declared once at + # server scope above and inherited here -- see the comment there + # for why it isn't in this block. # See conf.d/markdown_negotiation_map.conf for # $aal_uri_sans_trailing_slash and $aal_wants_markdown_suffix. -- 2.53.0