How to Make Your Website Visible to AI Agents: A Practical Guide to llms.txt

AI agents are becoming decision-makers, not just assistants. They're researching products, comparing vendors, and summarizing information for busy executives. But here's the problem: if your website is only designed for humans, it's invisible to these AI agents.

Think about how you discover software today. You might ask ChatGPT "What are the best competitive intelligence tools?" or use Claude to research vendor options. These AI agents are crawling websites, but they're struggling to understand which pages matter and what your product actually does.

By the end of this guide, you'll have implemented llms.txt—a simple file that makes your website crystal clear to AI agents.

What is llms.txt?

llms.txt is a markdown file placed at your website's root (yoursite.com/llms.txt) that helps large language models understand your site's structure and content. Think of it as a "cheat sheet" for AI agents.

Here's an analogy: robots.txt tells crawlers what NOT to index; llms.txt tells AI what TO focus on.

The specification was created by Jeremy Howard, founder of fast.ai and one of the most influential figures in practical AI. The idea is beautifully simple: instead of making AI agents parse your entire website (with all its navigation, footers, popups, and marketing fluff), you give them a clean, structured summary of what matters.

Who's already using it? Anthropic, Cloudflare, Docker, HubSpot, Mintlify, and hundreds of other companies have adopted llms.txt. It's becoming the de facto standard for AI-friendly websites.

The Format

Here's the basic structure:

# Company Name

> One-line description

Brief overview paragraph explaining what you do.

## Documentation

- [Page Title](url): Description of what this page contains

## Key Pages

- [Pricing](url): Pricing tiers and plans

- [Features](url): Core product capabilities

## Optional

- [Blog](url): Secondary resources that can be skipped

Key rules to follow:

• Must use Markdown format
• H1 (# Company Name) is required—this is your site name
• Blockquote (> text) provides a one-line summary (optional but recommended)
• H2 sections (## Category) group your links logically
• Each link includes a description explaining the VALUE, not just the content
• "Optional" section contains content that AI can skip if context is limited

Step-by-Step Implementation

Step 1: Identify your 5-10 most important pages

Start by listing the pages that matter most for someone evaluating your product:

• Homepage or main landing page
• Pricing or contact page
• Core product/feature pages
• Documentation (if applicable)
• Key comparison pages (You vs Competitor)

Don't include every page. Be ruthless. AI agents work better with focused, high-quality information than with comprehensive dumps.

Step 2: Create markdown versions (optional but recommended)

For maximum effectiveness, create .md versions of your key pages that strip out navigation, scripts, ads, and other noise. Pure content only. Some companies add a /page.md endpoint alongside their regular /page URL.

Step 3: Write your llms.txt file

Open a text editor and start writing:

• Start with your company name as H1
• Add a one-line description in blockquote format
• Write a brief overview paragraph (2-3 sentences)
• Group your pages into logical categories (Core Product, Pricing, Comparisons, etc.)
• For each link, write a description that explains VALUE—what will someone learn or find here?

Step 4: Deploy

Place your llms.txt file at your website root. It should be accessible at yoursite.com/llms.txt with no authentication required. The MIME type should be text/plain or text/markdown.

For most web frameworks:

• Static sites: Just add llms.txt to your public folder
• Rails: Add to /public/llms.txt
• Next.js: Add to /public/llms.txt
• WordPress: Upload to your root directory via FTP or file manager

Step 5: Test

Visit yoursite.com/llms.txt in your browser to confirm it's accessible. Then test it with AI: ask ChatGPT or Claude to "fetch and summarize the llms.txt file from yoursite.com" and see what they understand.

Real Example: DigBI's llms.txt

Here's how we structured our own llms.txt file:

# DigBI - AI-Powered Competitive Intelligence

> Real-time competitive intelligence platform with AI agents that monitor competitors 24/7.

DigBI helps product managers, marketing teams, and executives track competitors automatically...

## Core Product

- [Platform Overview](/platform_overview): How DigBI works

- [AI Agents](/ai_agents): Our autonomous AI agents

## Comparisons

- [DigBI vs Klue](/digbi_vs_klue): Feature comparison

- [DigBI vs Crayon](/digbi_vs_crayon): Feature comparison

Notice what we prioritized: our core differentiator (AI agents), our comparison pages (critical for evaluation), and our use cases (who we serve). We put integrations and API docs in "Optional" because they're secondary for most visitors.

See it live at digbi.co/llms.txt

Best Practices

• Keep it concise: Aim for under 500 lines. AI agents have context limits.
• Update when key pages change: If you launch a new product or redesign your pricing, update llms.txt.
• Prioritize commercial pages: Pricing, features, and comparison pages should be front and center.
• Include differentiators explicitly: Don't make AI guess what makes you special. State it clearly.
• Avoid marketing fluff: AI values facts over superlatives. "50% faster deployment" beats "blazingly fast."
• Test with multiple AI agents: Try ChatGPT, Claude, and Perplexity to see how each interprets your file.

What's Next: GEO (Generative Engine Optimization)

llms.txt is just the beginning. The broader field of Generative Engine Optimization (GEO) is emerging as the AI equivalent of SEO. It encompasses everything from structured data to content formatting that AI agents prefer.

Key areas to watch:

• Schema markup optimized for AI comprehension
• FAQ pages that AI can easily parse
• Comparison content that helps AI make recommendations
• Citations and sources that establish credibility

Conclusion

llms.txt is a low-effort, high-optionality investment. It takes 30 minutes to implement and costs nothing, but it positions your website for an AI-first future.

If AI agents are becoming the new gatekeepers—researching on behalf of humans, summarizing options, and making recommendations—make sure they can read your pitch.

The companies that make themselves visible to AI today will have a significant advantage as AI-driven discovery becomes the norm.