Making a personal site discoverable by LLM-powered search
Classical search engine optimization is a ranking problem: get a page to show up high in a list of blue links, and a human chooses whether to click. The new generation of search powered by language models is a different problem: the answer is generated, the sources are sometimes cited, and the source pages may never be visited. A site that wants to be in those answers has to be quotable and machine-readable, not just indexed.
I recently went through this layer on pcbrom.com. This post is the catalogue of moves that made the site more useful to LLM-powered search engines, written in the order I applied them, with the parts that do not work and the parts that do. The whole sweep took one afternoon and added no JavaScript, no tracking, and no third-party requests. The site stays as private as it was before.
What changed in search
Three behaviors define the LLM-powered layer. First, the answer is composed: a model reads a small set of sources, writes a synthesis, and may cite the sources at the end. The relevant unit is the sentence the model lifts, not the page it ranked. Second, the model selects sources by a mix of classic ranking signals and an evaluation of how citable the page is. Third, the source pages are crawled by a new family of user agents whose names (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others) the standard robots.txt of most sites do not address.
A page optimized only for classical ranking can land in the candidate set and still go unused: if the prose is wandering, if the page is mostly JavaScript, if the relevant fact is buried, the model lifts somebody else’s sentence. Three moves cover most of the gap.
Move 1: /llms.txt and /llms-full.txt
The llms.txt convention is a single markdown file at the root of the site that describes what the site contains, in a structured way a model can read in one shot. It is a hand-curated table of contents: who I am, the projects I have, where the papers live, the recent blog posts. The shape is the same as a top-of-README.
The companion file /llms-full.txt is the same information expanded to the actual prose of the site: a longer plaintext dump that includes the body of recent blog posts. It is the file a model can read when it wants the substance, not the index.
Both files take care of an asymmetry in how language models discover a site. Search engines crawl pages, follow links, and reconstruct site structure from the link graph. A model that lands on a page in the middle has no way to know that the rest of the site exists, unless the page tells it. llms.txt tells it.
The convention is still emerging; not every model crawler honors it yet. The cost of adding the file is two small text artifacts at the root and a five-line entry in the build script that generates them from the same curation layer the landing page uses. Worth doing now: by the time the convention is widely honored, the file is in place.
Move 2: granular Schema.org structured data
A site that publishes papers, software, and a blog has three different kinds of artifact, and Schema.org has a vocabulary for each. The classical advice is to add a Person block for the site author and call it a day. That is a floor, not a ceiling.
For each paper, add a ScholarlyArticle block with headline, url (the DOI), description, and an author reference back to the Person block via @id. For each software repository in the featured grid, add a SoftwareSourceCode block with name, codeRepository, description, programmingLanguage, and the same author reference. The result is a graph: the model that lifts a sentence about a paper or a repo also sees, in the same page, the structured assertion that the same author wrote it.
The third Schema.org type that helps is FAQPage. I curated seven questions a reader (human or model) might ask about the site: who is the author, what is each project, where to find the papers. Each question has a one-paragraph answer, declarative and quotable. The block is rendered as a collapsible disclosure for human readers and as a FAQPage JSON-LD entity that the model can read directly. The text stays in the HTML regardless of whether the disclosure is open, so the crawler does not depend on JavaScript to see the answers.
The combined JSON-LD on the landing page now lists one Person, four ScholarlyArticle, twelve SoftwareSourceCode, one FAQPage with seven question-answer pairs. Each block is one short <script type="application/ld+json"> injection. None of it changes the visual layout.
Move 3: an explicit robots.txt for AI crawlers
The standard robots.txt of most personal sites is a single User-agent: * block that allows everything. That works for classical search but leaves the new AI crawlers in a grey zone: a few of them honor the wildcard, a few read site-specific rules and ignore the wildcard if none exist, a few do neither. The friction-free fix is to add an explicit Allow: / block per user agent.
I added blocks for seventeen: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, cohere-ai, DuckAssistBot, and MistralAI-User. The list will grow; the maintenance cost is one line per new crawler.
Two things to keep in mind. The first is policy: if you do not want your content used to train models, the right move is to block these crawlers, not allow them; this post takes the opposite direction because the goal is discoverability. The second is enforcement: robots.txt is a polite request, not a security boundary. A crawler that does not honor it crawls anyway. The block-list is for the polite crawlers, and the polite ones are the ones whose answer engines are most worth being inside.
Move 4: quotable opening sentences
The earlier three moves are mechanical. This one is editorial and harder.
A model that picks which sentence to lift looks for a sentence that stands on its own: a declarative statement, with the relevant concrete fact in it, that a reader who has not read the rest of the page can still understand. The first sentence of a blog post or of a project description should be that sentence. The conventional advice (do not bury the lede) carries over with one twist: the lede is not what the page is about, it is what the page claims, in a form that survives extraction.
This is harder than the other moves because it requires rereading the prose with the LLM’s behavior in mind, and rewriting the parts that do not survive a one-sentence lift. The payoff is the page being represented by its own first sentence in any AI-generated answer, rather than by a paraphrase the model invents.
What does not work
Three classical SEO tactics do not carry over and can hurt.
The first is keyword density. Models rank pages by semantic relevance to the query, not by lexical match counts. A page that stuffs the keyword in three paragraphs reads as authored to game the ranker, and the model penalizes the page accordingly. The right move is to write the page once, well, with the natural vocabulary of the topic.
The second is duplicate content for different keyword targets. Where classical SEO sometimes accepted near-duplicate landing pages tuned for different long-tail keywords, model-powered search treats them as one source and chooses the canonical version, which may not be the one you want.
The third is JavaScript-rendered prose. AI crawlers vary in how aggressively they execute JavaScript, and several do not at all. Anything that lives only in a client-side render is invisible to that subset of crawlers. The mitigation is server-side rendering or static HTML for the substance of every page; this site renders everything at build time, so this constraint is satisfied by accident.
Cost
The whole sweep took one afternoon: the llms.txt and llms-full.txt generator, the additional JSON-LD blocks, the FAQ section, the expanded robots.txt, and a pass over the opening sentences of the existing pages. The site added zero JavaScript, zero third-party fetches, and zero cookies. The CSP did not need to be loosened. Nothing about the visitor’s experience changed.
What may or may not change is whether the site shows up in AI-generated answers. The honest answer is that none of the AI search providers publish their ranking models, so the experiment is open-ended. The reason to do the sweep anyway is that machine-readability and quotability are leverage in every reading the model performs, and the cost is small enough that “let me wait and see if it matters” is the wrong default.
Reference list
- llms.txt convention. https://llmstxt.org/
- Schema.org structured data vocabulary. https://schema.org/
- Google Search Central: AI Overviews. https://developers.google.com/search/docs/appearance/ai-features
- Anthropic. Robots policy and
ClaudeBot. https://support.anthropic.com/en/articles/8896518