[Home](https://unprompted.dirkprimbs.de/index.md) · [← Previous: COSMIC on Linux Mint](https://unprompted.dirkprimbs.de/posts/cosmic-on-linux-mint.md)

# Markdown Web - Yes, you read that right.

> Dave Winer asked for a web of Markdown, Andrew Shell built a browser for it, and my engine now publishes an llms.txt and a Markdown twin of every page.

- Published: 2026-10-16
- Tags: [Blogging](https://unprompted.dirkprimbs.de/tags/blogging.md), [Engine Updates](https://unprompted.dirkprimbs.de/tags/engine-updates.md), [Markdown](https://unprompted.dirkprimbs.de/tags/markdown.md), [AI](https://unprompted.dirkprimbs.de/tags/ai.md), [WebDev](https://unprompted.dirkprimbs.de/tags/webdev.md), [OSS](https://unprompted.dirkprimbs.de/tags/oss.md)
- Web page: https://unprompted.dirkprimbs.de/posts/engine-update-markdown.html

---

On October 6th Dave Winer posted [The web of Markdown](http://scripting.com/2026/10/06/005559.html). It's short enough to quote. "How about a Markdown web browser. No CSS, no frameworks, just Markdown files. That would be how you would restart the web in 2026." A day later Andrew Shell [replied](https://andrewshell.org/2026/10/the-web-of-markdown) that a lot of this already exists, because plenty of sites now serve Markdown for LLMs - and then spent a short session building [Markdown Browser](https://md.geekity.com/), a proxy that lets you browse exactly that web.

I came at it from the other end. I'd found the [llms.txt proposal](https://llmstxt.org) opened in Andrew's browser, read it, and thought that my engine already has all the ingredients. Every post I write starts as Markdown anyway and the HTML is generated from it. So I added it.

### First I had to unblock the robots

When I looked into this I found that my own `robots.txt` blocked every AI crawler I could name, on all six of my sites. I had put that in at the end of August after GPTBot and ClaudeBot together were hitting my hosting account with something like 5,000 requests a day, and then promptly forgot about it.

That's gone now. I'd actually like LLMs to read what I write. If someone asks an assistant about something I've blogged about, I want it to have read the post, and publishing a map for AI agents while telling them to stay out would have been a bit silly. The SEO crawlers stay blocked, since not a single reader has ever come here through Ahrefs or Semrush.

### What the engine does now

There's a file at [/llms.txt](https://unprompted.dirkprimbs.de/llms.txt) that follows the proposal: the site's name, a one-line description, a short paragraph about who writes here, and then lists of every post and page with a one-line summary each, which is where an agent starts.

Every page on this site now also exists as Markdown. Take any address that ends in `.html`, replace that with `.md`, and you get the same page as plain text. That works for posts, for the about page, for tag pages, for the paginated front page (so there's an [index.md](https://unprompted.dirkprimbs.de/index.md) now) and even for the little player pages the podcast sites hand out for embedding. The build checks this every time it runs and complains if any page it wrote is missing its twin, because a promise like "every page" quietly stops being true the first time I add a new kind of page and forget.

Each Markdown page starts with one line of navigation: a link home to `index.md`, and links to the previous and next page in whatever set it belongs to. For posts that's older and newer, like the links at the bottom of the HTML version, while the front page and the tag pages step through their own page 2, page 3 and so on. Links from one post to another point at the other post's Markdown, so once you're in there you can walk the whole site without ever touching HTML. Images and audio stay links to the actual files.

The HTML of my [podcast post](https://unprompted.dirkprimbs.de/posts/engine-update-podcasts.md) is about 27 KB and its Markdown twin about 7 KB, almost all of it the words themselves - no header, no menu, no comment widget, no fonts, no scripts. That's a lot less for a model to read and get wrong, and in a terminal it's simply pleasant to read.

### How to try it on this site

The quickest test is the Markdown twin of this very post, [engine-update-markdown.md](https://unprompted.dirkprimbs.de/posts/engine-update-markdown.md). Depending on your browser it shows up as text or as a download.

In a terminal:

```
curl https://unprompted.dirkprimbs.de/llms.txt
curl https://unprompted.dirkprimbs.de/index.md
```

Or use Andrew's browser. Open [Markdown Browser](https://md.geekity.com/) and type `unprompted.dirkprimbs.de` into the address bar. It doesn't find Markdown at the bare address, so it falls back to `llms.txt` and shows you that, rendered, with every link clickable. You can also [start straight at index.md](https://md.geekity.com/?url=https%3A%2F%2Funprompted.dirkprimbs.de%2Findex.md) and click your way through with the Home / Previous / Next line at the top of every page. Pasting a regular `.html` address won't work, because the browser doesn't swap the extension for you, so start from the site's address or from `index.md`.

### Doing this on your own site

All of this lives in my blog engine, which is open source: [unprompted-blog-engine on GitHub](https://github.com/dirkprimbs/unprompted-blog-engine). If you run it, it's one line in `site.yaml`:

```
site:
  llms_txt: true
```

It's off by default, because whether AI agents should read a site is up to whoever runs it. If you switch it on, check that your `robots.txt` doesn't block the agents you're inviting in. Mine did, for seven weeks.