Making a WordPress Site Agent-Ready: A Practical Tutorial

Last Updated on July 10, 2026

What inspired me to get my websites better ready for Agentic Browsing? I had read one of his articles, liked the design of his website, and kept reading.

Three sources I found the most useful come from the same person – Joost de Valk, of Yoast fame.

https://joost.blog/agent-native an excellent article
https://github.com/progressplanner/markdown-alternate a WordPress plugin
https://isitagentready.com ( pagespeed.web.dev checks your site now too) check your website for agentic access

I used claude.ai to assist with the updates. AI is very useful when we want it to be.

Tools like isitagentready.com scan sites against an emerging checklist of standards for AI agent discovery – Link headers, robots.txt content signals, API catalogs, OAuth metadata, payment protocols, and more. Most of these checklists are written with a generic site in mind, so not every item applies to every site.

This tutorial walks through how to figure out what’s actually relevant, and how to implement the parts that are – using a real WordPress content/brochure site as the example.

New to this? A quick primer

If terms like “header,” “.htaccess,” or “curl” are new to you, start here before Step 1.

What’s a “header”? Every time your browser loads a webpage, your server sends back the page content plus a small bundle of extra information that never actually appears on the page itself — things like “this is HTML,” “here’s when this page was last changed,” or, in our case, “here’s a link to more info for AI agents.” That extra bundle is called an HTTP header. It’s invisible unless you go looking for it with a specific tool.

What’s .htaccess? It’s a small configuration file that lives in your website’s root folder and gives instructions directly to your web server (rather than to WordPress). It’s powerful, which means mistakes in it can break your site — always keep a backup copy before editing it.

What’s curl? A command-line tool for making web requests and seeing exactly what a server sends back — including headers, which a normal browser tab hides from you. You don’t need to install anything special; it’s built into Mac and Linux terminals, and available in Windows PowerShell too.

What you’ll need before starting:

Access to your site’s files. This usually means an FTP program (like FileZilla) or your hosting provider’s built-in “File Manager,” found in your hosting control panel (cPanel, Plesk, etc.). If you only ever log into yoursite.com/wp-admin, you have dashboard access but not file access — ask your host how to get FTP or File Manager credentials.
A way to check raw headers. Either a terminal (Mac/Linux have one built in; Windows has PowerShell) or a plain-text web tool like reqbin.com/curl.
A backup of any file before you edit it. Especially .htaccess and robots.txt — download a copy first, every time.

How to know if a step worked: Throughout this tutorial, “verify with curl” means: open a terminal, type curl -sI https://yoursite.com/, hit enter, and read the block of text it prints out. You’re looking for the line you just added to appear in that list, exactly as you wrote it.

With that groundwork in place, here’s how to work through the actual checklist.

Step 1: Know what kind of site you’re running

Before touching any code, answer two questions:

Do you have file/server access (FTP, SFTP, SSH, hosting file manager), or only the WordPress admin dashboard?
Does your site have APIs, paid content, or e-commerce checkout that an AI agent would need to authenticate with or pay for?

These two answers eliminate most of the checklist immediately:

If your site is…	Skip these items
Dashboard-only (no file access)	Anything requiring `.htaccess` or `/.well-known/` files — use plugins instead
A content/brochure site with no APIs	OAuth/OIDC discovery, OAuth Protected Resource Metadata, `auth.md`, API catalog, MCP Server Card, WebMCP
Not selling anything programmatically	x402, MPP, UCP, ACP (agent payment protocols)

For a typical small-business or personal WordPress site, that usually leaves only two checklist items worth doing:

Content Signals in robots.txt
Link response headers (RFC 8288)

Everything else (DNS-AID, payment protocols, MCP server cards) is either too early-stage, requires infrastructure you don’t have, or simply doesn’t apply to a site that isn’t running its own AI agent or API.

Step 2: Add Content Signals to robots.txt

WordPress generates a virtual robots.txt by default. To add custom directives, upload a physical file to your site root (same folder as wp-config.php):

User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=no

Sitemap: https://yoursite.com/sitemap.xml

Adjust ai-train / search / ai-input to your actual preference. Also double-check your Sitemap: line matches whatever URL your SEO plugin actually generates — it’s not always /sitemap.xml.

Watch out for conflicts: if your SEO plugin (Yoast, RankMath, etc.) has its own robots.txt editor active, disable it – a physical file and a plugin-managed one can fight each other.

Step 3: Add a Link response header

This is where most of the real troubleshooting happens. A few important lessons learned along the way:

Use a real, registered relation type

Don’t invent your own rel value (e.g. rel="ai-usage"). Agent-discovery checkers look for registered IANA link relations – things like service-doc, help, privacy-policy, terms-of-service. If your site has no API, point to genuinely useful pages using real relation types:

Header set Link '</privacy-policy/>; rel="privacy-policy", </faq/>; rel="help"'

Prefer `.htaccess` over an mu-plugin, if you’re on a caching setup

Adding the header via WordPress (add_action('send_headers', ...) in an mu-plugin) is theme-independent and normally works fine, but if your host or a caching plugin serves full-page HTML snapshots, WordPress’s PHP may never run on cache hits, so the header silently disappears. Setting the header at the web-server level avoids this entirely:

# Place near the TOP of .htaccess, before "# BEGIN WordPress"
<IfModule mod_headers.c>
    Header set Link '</privacy-policy/>; rel="privacy-policy", </faq/>; rel="help"'
</IfModule>

Always back up .htaccess before editing it.

Verify with raw headers, not a browser-based checker

Some online “check my headers” tools render results through HTML, which can silently eat parts of your header value – specifically, anything that looks like an HTML tag, such as </path>. If a checker shows a truncated or missing header even after you’ve confirmed it’s set, verify with a raw tool instead:

curl -sI https://yoursite.com/

or a plain-text tool like reqbin.com/curl. This was the exact issue encountered here – the header was correct all along, but the checking tool’s rendering was stripping it visually.

A generic scanner error isn’t always about your site

If an automated checker reports something like “Could not check Link headers: the operation was aborted,” that’s often the scanner’s own request failing (firewall block, rate limit, timeout) – not proof that your header is missing. Confirm independently with curl before chasing a phantom problem.

Step 4: Consider `llms.txt` (optional, but a good fit for content sites)

Separate from the formal checklist above, llms.txt is a community-driven (non-IETF) convention: a plain Markdown file at your site root (/llms.txt) that gives AI systems a concise summary of your site and links to key pages — similar in spirit to robots.txt or sitemap.xml, but written for language models instead of crawlers or search engines. Unlike most of the checklists above, this one is:

Genuinely designed for content/marketing sites, not APIs
A single static file – no server config or plugins strictly required
Increasingly recognized by tools like Chrome Lighthouse’s agentic browsing audits

If your site already generates Markdown versions of pages (look for <link rel="alternate" type="text/markdown"> in your page source – some SEO/AI plugins add this automatically), you likely already have the groundwork in place to add an llms.txt index quickly.

Summary checklist for a typical WordPress content site

[x] Content Signals in robots.txt (physical file at site root)
[x] Link response header with registered rel types, set via .htaccess
[ ] llms.txt (optional, high value for content-focused sites)
Everything else on typical “agent readiness” checklists: skip, unless you later add APIs, an MCP server, or agent-based payments

Key takeaways

Read the checklist critically – it’s written for a generic site, and most items target API- or commerce-driven use cases.
Match implementation method to your access level – .htaccess/file uploads if you have FTP/SSH, plugins/mu-plugins if you’re dashboard-only.
Caching is the most common silent failure for server-set headers – test with caching both on and off.
Verify with raw tools (curl -I), not browser-rendered checkers, since HTML rendering can hide parts of header values.
A scanner error doesn’t always mean your site is broken – cross-check before spending time “fixing” something that already works.