llms.txt is a proposal, with no search visibility guarantee
In a technical SEO audit, do not prioritize llms.txt ahead of crawlability, indexing and accurate content. Jeremy Howard published the proposal on September 3, 2024; its v2 update appeared on August 10, 2026. It guides agents toward useful documents when seeking information. It is not a compulsory protocol automatically implemented by every model.
Google’s official guidance says Search does not use these files and they neither help nor harm Search visibility or rankings. Google’s generative AI search features do not require special files either. Another system’s documentation workflow may differ; that is a separate question from Google ranking.
A provider publishing a file for its own developer documentation does not establish that it automatically consumes every customer website’s file when choosing answers. “Publication completed” and “the target tool used this content” are separate outcomes. An agency proposal should specify the deliverable and the evidence used to assess it.
It serves a different purpose from robots.txt and sitemaps
The crawling and indexing distinction in the robots.txt guide still applies. llms.txt does not authorize access, implement noindex or protect private files. Listing a document does not replace access controls. Similar filenames do not make these tools interchangeable.
Model training, automatic search crawling and user-initiated document retrieval are different flows. OpenAI’s official crawler documentation describes GPTBot and OAI-SearchBot as separate preferences. Publishing llms.txt does not change those robots preferences, grant training permission or establish a data-use policy enforced across every provider.
| Tool | Purpose | What it does not establish |
|---|---|---|
| Robots.txt | Regulate URL-path crawling by cooperating bots. | Access security or certain removal from an index. |
| XML sitemap | Tell search engines about appropriate URLs. | Guaranteed indexing of every listed URL. |
| llms.txt | Provide a curated entry point to information and documents. | Guaranteed use, citations or rankings. |
| Authentication | Restrict unauthorized access. | Protection cannot be provided by an explanatory text file. |
FROM READING TO A NEXT STEP
Define a useful scope and maintenance plan
Share your document structure, intended agent task and content owner so we can assess generation, validation and measurement against a concrete need.
Choose a use case worth maintaining
A content marketing plan should identify who the file will help. Versioned API documentation for a software library has different needs from a small business site with a few service pages. Replace a hypothetical visibility gain with a testable task, such as finding the installation instructions for the correct product version.
Company information, product guides, return policies, support documents and educational material can be candidates. Manually summarizing stock, prices, promotions or changing requirements can create another source of inconsistency. Start with limited scope and keep the original page as the information source. The file should support a usable website, not replace it.
Document access
Describe a real support or development task. Identify the sources and versions needed for the correct answer and which documents are secondary.
Content ownership
When the product changes, who updates the page, Markdown version and guide links? A summary without an owner can quickly become stale.
Work priorities
Fix broken pages, missing service explanations and incorrect information first. Do not make file creation a success metric that hides unfinished foundational work.
Write a short introduction and clear document links
Within your content update process, verify information on the original page before adding it. Exclude unpublished customer data, private contracts, personal information and internal paths. Do not select documents solely by traffic: a rarely visited integration limitation may be essential to completing a task correctly.
The proposal uses an H1 project name, a blockquote summary, H2 document groups and annotated Markdown links. For example: # Example Project The address is illustrative; do not list a .md URL that has not actually been published.
> Installation and usage documentation.
## Documents
- [Start](https://example.com/docs/start.md): Initial setup.
Describe each document’s scope and, where relevant, language and version. Preserve tables, code samples, exceptions and links when extracting page content. Removing an important limitation in pursuit of cleaner text improves appearance while damaging accuracy. Use the actual URL for each language instead of deriving an English address from another language’s slug.
Choose publication methods that fit the platform
Your CMS management approach determines whether the file is static, plugin-generated or built during deployment. FTP and hosting panels work for some sites; public_html and identical filesystem permissions are not universal. Check the actual server ownership and access arrangement instead of applying a blanket “set permissions to 644” instruction.
The v2 proposal allows path-specific files such as /docs/llms.txt as well as a root file. It recommends alternate and describedby link relations for discovering Markdown documents and the relevant guide. Verify that the intended client supports the method; root publication alone does not establish automatic use.
Assign one generator
Prevent conflict between a static file and dynamic plugin response. Record editing, publication and rollback locations for the team.
Check WordPress behavior
Yoast SEO documents generation and weekly updates, with manual page selection available. Existing files and permissions affect generation. Do not assume every SEO plugin offers the same capability.
Read the live response
Fetch the file on the correct hostname and path. Check for login pages, HTML errors, old caches and document links pointing to the wrong version.
Separate file validation from provider adoption
A technical audit should check accessibility, format and content accuracy separately. Inspect UTF-8 characters, headings, Markdown links, HTTP responses and broken URLs. A passing validator does not prove that every agent uses the file. Record which proposal version the tool actually validates.
The Lighthouse agentic browsing documentation describes the file as optional and treats a missing file returning 404 as N/A. This audit is not a Google Search ranking assessment. Do not report a passing technical check as evidence of visibility or sales.
Giving a chatbot the file URL and asking it to find a particular document is a controlled task test. Record the expected answer, source, date, tool and prompt; review missing exceptions or incorrect versions. It does not show that the tool will discover the file independently during ordinary questions or produce the same answer every time.
Define maintenance and measurement before launch
Your content measurement plan should separate file requests, task accuracy, external citations and website conversions. A /llms.txt request in server logs demonstrates access; it does not prove use in an answer. A user-agent string alone also does not conclusively verify the requester’s identity.
A bot request is different from a visitor session. If main pages were updated during the same period, do not attribute subsequent traffic changes to this file. Keep baseline observations, publication dates and concurrent changes. Tool and model changes can also affect comparisons between repeated task tests.
Review the file after version releases, URL moves, document removal and policy changes. Inspect automatically selected pages rather than assuming generation guarantees relevance. Reducing scope that has no verified use and consumes maintenance time is reasonable. The goal is sustainable access to accurate information, rather than increasing the number of published files.
BEFORE YOU DECIDE
Frequently asked questions
Does llms.txt improve Google rankings?
Google says Search does not use these files. Publishing one neither helps nor harms Google Search visibility and rankings. Usage by other systems must be assessed separately.
Must the file be at the website root?
The current v2 proposal allows root files and scoped paths such as /docs/. Check the intended client and platform’s discovery method instead of following older claims that every subdirectory is invalid.
Can the file be written in Turkish?
Yes. Use clear descriptions and real document links. Specify language and version scope for multilingual content, and do not invent English Markdown URLs.
Does WordPress require a file manager plugin?
No. Static publication, an appropriate CMS generation process or a documented plugin feature can work. Choose based on permissions, existing files and maintenance ownership; an additional file manager is not compulsory.
Does a correct chatbot summary prove successful deployment?
It is evidence from a controlled content-reading test for that prompt and time. It does not prove automatic discovery, support by all providers or guaranteed citations in ordinary answers.
Can llms.txt replace robots.txt?
No. Content guidance and crawler rules have different purposes. Neither replaces authentication or access permissions for private information.
LET’S DEFINE THE SCOPE
Define a useful scope and maintenance plan
Share your document structure, intended agent task and content owner so we can assess generation, validation and measurement against a concrete need.
Discuss content and technical scope