| `ai_bot_digest.py` | `/usr/local/bin/ai_bot_digest` |
| `systemd/*` | `/etc/systemd/system/` |
| `pelican_scheduler.py` | symlinked as the bare repo's `hooks/post-receive` |
+| `root_index.html` | `/home/blogmistress/zackmdavis.net/index.html` — **renamed on the way over**; scp'ing it under its repo name lands a dead file beside the real one and changes nothing visible |
+| `robots.txt` | `/home/blogmistress/zackmdavis.net/robots.txt` (the true domain root, not `/blog` — see the `STATIC_PATHS` comment in `pelicanconf.py` for why Pelican deliberately doesn't generate it) |
+
+The last two live under the webroot — `root /home/blogmistress/zackmdavis.net` in `nginx_siteconf`, which is also what serves `/docs` and `/media`. Being inside `blogmistress`'s own home, they're the only two deployable without root; `install` as root would leave them root-owned.
After nginx edits, `nginx -t && systemctl reload nginx` — `nginx -t` catches a `conf.d` file that didn't land, since the site config references things defined there. After unit edits, `systemctl daemon-reload` **and** `systemctl restart ai-bot-digest.timer`; a daemon-reload alone leaves an already-scheduled timer on its old schedule.
### AI-crawler observability
`provisioning/ai_bot_digest.py` runs daily via systemd and files `<site>-<date>.txt` into `/var/log/ai-bot-digest/`, summarizing which crawlers fetched which posts. Its own docstring covers usage and the second-site story. Two structural limits worth knowing before trusting it: User-Agents are forgeable (it quarantines UAs whose traffic is ≥60% 404s as likely impostors), and Google/Apple AI-training use is invisible in principle, since `Google-Extended`/`Applebot-Extended` are robots.txt tokens that no request carries.
+
+### `llms.txt` is generated where nothing will find it (unfixed, low priority)
+
+`_write_llms_txt` in `pelicanconf.py` emits it into the Pelican output tree, so it lands at `/blog/llms.txt` — but `llms.txt` is a domain-root convention like `robots.txt`, so a crawler that goes looking checks `zackmdavis.net/llms.txt` and gets a 404. Nothing links to the real one either, from `robots.txt` or from any page. It has **zero fetches** across every digest in `notes/ai_bot_digests/` (2026-07-25 through 2026-08-03) — the file is currently dead weight.
+
+Note this means the heavy `.md` fetching visible in those digests is *not* evidence llms.txt works: the mirrors are reachable from the HTML anyway, via the `<link rel="alternate">` in `theme/templates/article.html` and the visible "Markdown source" link in `theme/templates/includes/post_card.html`.
+
+The fix is placement, not content — `_canonical_url` already builds absolute `https://zackmdavis.net/blog/...` links, so a copy served from the webroot works unmodified. It's the same problem `STATIC_PATHS` in `pelicanconf.py` already solves for `robots.txt` by deploying that standalone, but `llms.txt` is build-generated, so it needs copying out of `output/` after each build (`provisioning/pelican_scheduler.py`) rather than a one-time `scp`. While in there: there's no `sitemap.xml` at all and no `Sitemap:` line in `robots.txt`, which is the same missing-machine-discovery-surface problem — robots.txt *is* reliably fetched (ClaudeBot, Applebot, PerplexityBot, OAI-SearchBot all hit it), so it's the better hook of the two.
root /home/blogmistress/zackmdavis.net;
index index.html;
+ # .md is served as text/markdown via a `text/markdown md;` line added
+ # to the OS-managed mime.types on the server (not part of this repo).
+ # mime.types can't carry a charset parameter, though, and without one
+ # a client handed "Content-Type: text/markdown" falls back to its own
+ # default (typically windows-1252) and renders every em dash and curly
+ # quote as mojibake -- there's no <meta charset> for it to recover
+ # from, the way there is in HTML. So declare it here. (Distinct from
+ # the gitweb case in provisioning/gitweb.conf: there CGI.pm attached a
+ # *wrong* charset; here nginx attaches none at all.)
+ #
+ # Deliberately at server scope rather than inside location /blog/,
+ # where it originally lived: .md files are served from /docs too (the
+ # diary entries), and those are NOT under the Pelican tree, so a
+ # /blog/-scoped charset never reached them. Locations inherit this
+ # unless one overrides it, so /blog/ is unaffected by the move.
+ #
+ # charset_types REPLACES the default list rather than extending it, so
+ # the stock entries are restated alongside text/markdown; only
+ # text/html is implicit and can be omitted. Since these are static
+ # files that carry no charset of their own, nginx just appends -- no
+ # recoding is attempted.
+ charset utf-8;
+ charset_types text/markdown text/xml text/plain
+ text/vnd.wap.wml application/javascript
+ application/rss+xml;
+
location / {
try_files $uri $uri/ =404;
}
# served from /blog/ (e.g. llms.txt, from _write_llms_txt in
# pelicanconf.py) as text/markdown too, not just .md files.
#
- # mime.types can't carry a charset parameter, though, and without
- # one a browser handed "Content-Type: text/markdown" falls back to
- # its own default (typically windows-1252) and renders every em
- # dash and curly quote in a .md mirror as mojibake -- there's no
- # <meta charset> for it to recover from, the way there is in HTML.
- # So declare it here. (Distinct from the gitweb case in
- # provisioning/gitweb.conf: there CGI.pm attached a *wrong*
- # charset; here nginx attaches none at all.)
- #
- # charset_types REPLACES the default list rather than extending it,
- # so the stock entries are restated alongside text/markdown; only
- # text/html is implicit and can be omitted. Since these are static
- # files that carry no charset of their own, nginx just appends --
- # no recoding is attempted.
- charset utf-8;
- charset_types text/markdown text/xml text/plain
- text/vnd.wap.wml application/javascript
- application/rss+xml;
+ # The utf-8 charset those .md mirrors need is declared once at
+ # server scope above and inherited here -- see the comment there
+ # for why it isn't in this block.
# See conf.d/markdown_negotiation_map.conf for
# $aal_uri_sans_trailing_slash and $aal_wants_markdown_suffix.
projects / An_Algorithmic_Lucidity.git / commitdiff