Introduction

Robots Meta Tag: Controlling How Search Engines Crawl Your Site

6 min read
View contents

The robots meta tag is an HTML instruction that controls how search engines index and follow links on specific pages. According to Web.dev’s HTML metadata documentation² and Ahrefs’ practical examples¹, it’s fundamental for managing what content appears in search results and how crawl budget is distributed.

What is the robots meta tag?

The robots meta tag is a <meta> element in your HTML <head> that gives direct instructions to search engine crawlers (like Googlebot). According to Web.dev’s HTML metadata best practices², this element should be properly placed within the <head> section of the document.

Basic example¹:

<head>
  <meta name="robots" content="index, follow" />
</head>

How it works:

  1. Googlebot visits your page
  2. Reads the <meta name="robots"> tag
  3. Obeys the instructions (index/don’t index, follow links/don’t follow)
  4. Processes the page according to directives

Difference between robots.txt and robots meta tag

Aspect robots.txt robots meta tag
Location Domain root (/robots.txt) <head> of each HTML page
Scope Entire site or sections Individual page
Function Blocks crawling Controls indexing and link following
Priority First barrier Second instruction (if page was crawled)

Important: robots.txt blocks crawling, robots meta tag controls indexing after crawling.

Main robots meta tag directives

1. index vs noindex

index (default): Allows the page to appear in search results.

<meta name="robots" content="index" />

noindex: Prevents the page from appearing in search results.

<meta name="robots" content="noindex" />

When to use noindex:

  • Login/registration pages
  • Order confirmation pages
  • Internal admin pages
  • Temporary duplicate content
  • Thank you pages after form submission
  • Internal search results pages

Real example - E-commerce:

<!-- Product page: index -->
<head>
  <title>iPhone 15 Pro Max - Buy Online</title>
  <meta name="robots" content="index, follow" />
</head>

<!-- Shopping cart: DON'T index -->
<head>
  <title>Shopping Cart</title>
  <meta name="robots" content="noindex, follow" />
</head>

2. follow vs nofollow

follow (default): Googlebot follows links on the page to discover more content.

<meta name="robots" content="follow" />

nofollow: Googlebot does NOT follow links on this page. As explained in Web.dev’s HTML links documentation³, the nofollow directive tells crawlers not to follow the links present on the page.

<meta name="robots" content="nofollow" />

When to use nofollow:

  • Pages with many unverified external links
  • User comment pages (if not moderated)
  • Thank you pages with external links
  • Temporary campaign pages

Note: nofollow in robots meta tag is different from rel="nofollow" on individual links³.

3. Common combinations

As demonstrated by Ahrefs’ SEO meta tags examples¹, the most commonly used combinations are:

<!-- Index page and follow links (DEFAULT) -->
<meta name="robots" content="index, follow" />

<!-- DON'T index but DO follow links (internal navigation pages) -->
<meta name="robots" content="noindex, follow" />

<!-- Index but DON'T follow links (pages with external links) -->
<meta name="robots" content="index, nofollow" />

<!-- DON'T index NOR follow links (complete block) -->
<meta name="robots" content="noindex, nofollow" />

Shortcut: You can use all or none:

<!-- Equivalent to: index, follow -->
<meta name="robots" content="all" />

<!-- Equivalent to: noindex, nofollow -->
<meta name="robots" content="none" />

4. Advanced Google directives

noarchive

Prevents Google from showing the “Cached” link in search results.

<meta name="robots" content="noarchive" />

When to use:

  • Frequently changing content (prices, availability)
  • Time-sensitive information pages
  • Subscription/paywall content

nosnippet

Prevents Google from showing description/snippet in results (only shows title and URL).

<meta name="robots" content="nosnippet" />

Side effect: Also automatically applies noarchive.

max-snippet

Limits snippet length to N characters.

<!-- Maximum 160 characters of snippet -->
<meta name="robots" content="max-snippet:160" />

<!-- No snippet -->
<meta name="robots" content="max-snippet:0" />

<!-- No limit (default) -->
<meta name="robots" content="max-snippet:-1" />

max-image-preview

Controls maximum size of image preview.

<!-- No image preview -->
<meta name="robots" content="max-image-preview:none" />

<!-- Standard preview -->
<meta name="robots" content="max-image-preview:standard" />

<!-- Large preview (recommended for articles) -->
<meta name="robots" content="max-image-preview:large" />

max-video-preview

Limits video snippet duration.

<!-- Maximum 30 seconds -->
<meta name="robots" content="max-video-preview:30" />

<!-- No limit -->
<meta name="robots" content="max-video-preview:-1" />

notranslate

Prevents Google from offering to translate the page.

<meta name="robots" content="notranslate" />

noimageindex

Prevents images on the page from being indexed.

<meta name="robots" content="noimageindex" />

When to use:

  • Images with protected copyright
  • Product images before official launch

Bot-specific directives

You can give different instructions to specific bots:

<!-- Instruction for all bots -->
<meta name="robots" content="noindex, follow" />

<!-- Specific instruction for Googlebot (takes priority) -->
<meta name="googlebot" content="index, follow" />

<!-- Instruction for Bingbot -->
<meta name="bingbot" content="index, follow" />

Common bots:

  • googlebot - Google
  • bingbot - Bing
  • slurp - Yahoo (now uses Bingbot)
  • duckduckbot - DuckDuckGo
  • baiduspider - Baidu

Practical example:

<!-- Allow Google but block Bing -->
<meta name="robots" content="noindex, nofollow" />
<meta name="googlebot" content="index, follow" />

Common use cases

Case 1: Login page

<head>
  <title>Login - My Site</title>
  <meta name="robots" content="noindex, nofollow" />
</head>

Why: You don’t want login pages appearing in searches or Googlebot wasting time crawling internal links behind login.

Case 2: Category pages with pagination

<!-- Page 1 (main) -->
<head>
  <title>Products - Page 1</title>
  <meta name="robots" content="index, follow" />
</head>

<!-- Page 2, 3, 4... (pagination) -->
<head>
  <title>Products - Page 2</title>
  <meta name="robots" content="noindex, follow" />
</head>

Why: Avoid duplicate content by indexing only the first page, but still allow Googlebot to discover products on subsequent pages.

Case 3: Temporary/seasonal content

<!-- Black Friday campaign (valid only in November) -->
<head>
  <title>Black Friday 2025 - Special Offers</title>
  <meta name="robots" content="noindex, follow" />
</head>

Why: Avoid Google indexing content that will be irrelevant in 1 month.

Case 4: Thank you pages

<head>
  <title>Thank You for Your Purchase</title>
  <meta name="robots" content="noindex, nofollow" />
</head>

Why: No SEO value, users shouldn’t arrive directly from Google.

Case 5: Low quality or thin content

<!-- Archive page with only 2 posts -->
<head>
  <title>Archive: January 2020</title>
  <meta name="robots" content="noindex, follow" />
</head>

Why: Protect your site from possible penalties for thin content.

Common mistakes and how to avoid them

Mistake 1: Block in robots.txt + noindex

<!-- ❌ BAD: robots.txt blocks /admin/ -->

# robots.txt
User-agent: *
Disallow: /admin/

<!-- This page will NEVER be seen by Googlebot -->
<head>
  <meta name="robots" content="noindex, follow" />
</head>

Problem: If robots.txt blocks the page, Googlebot never sees the noindex meta tag. The page could still appear in results (without description) if it has external links.

Solution:

# robots.txt - DON'T block
User-agent: *
# Disallow: /admin/  ← REMOVE THIS

<!-- Use only meta tag -->
<meta name="robots" content="noindex, nofollow" />

Mistake 2: noindex on entire page by error

<!-- ❌ BAD: noindex on homepage -->
<head>
  <title>My Site - Homepage</title>
  <meta name="robots" content="noindex, follow" />
</head>

Consequence: Your site disappears from Google.

Solution: Regularly audit which pages have noindex.

Mistake 3: Incorrect syntax

<!-- ❌ BAD: Incorrect spacing -->
<meta name="robots" content="noindex,follow" />

<!-- ❌ BAD: Incorrect order (doesn't matter but confusing) -->
<meta name="robots" content="follow, noindex" />

<!-- ✅ GOOD: Space after comma -->
<meta name="robots" content="noindex, follow" />

Mistake 4: Using nofollow when you meant noindex

<!-- ❌ BAD: Want it NOT indexed but wrote only nofollow -->
<meta name="robots" content="nofollow" />
<!-- Result: Page IS indexed -->

<!-- ✅ GOOD -->
<meta name="robots" content="noindex, follow" />

Mistake 5: Forgetting that noindex takes time

Problem: You add noindex and expect the page to disappear from Google immediately.

Reality: Google needs to re-crawl the page to see the change. Can take days or weeks.

Quick solution: Use Google Search Console → “Request Indexing” to speed up the process.

How to verify robots meta tags with UXR SEO Analyzer

Our Chrome extension automatically checks:

  1. Robots meta tag presence: Detects if it exists
  2. Active directives: Shows which instructions are configured
  3. Conflicts with robots.txt: Alerts if robots.txt blocks but has noindex
  4. Pages with accidental noindex: Identifies important pages that shouldn’t have noindex
  5. Suspicious combinations: Detects uncommon configurations

How to use the extension:

  1. Install UXR SEO Analyzer from Chrome Web Store
  2. Visit any page on your site
  3. Open extension → “Basic SEO” tab
  4. Review “Robots Meta Tag” section
  5. Follow specific recommendations

Next steps

Deepen your knowledge about crawling and indexing control:

<<<<<<< HEAD

RAG References

This article was enhanced using authoritative sources identified through systematic knowledge base searches:

References

This article cites the following authoritative sources:

24a07d01237e8d2ce45f0032ef83094634b50223

[1] Ahrefs Blog: SEO Meta Tags - Robots Examples (aa1a856f-2c67-423e-aa05-6ac80b44970e) https://ahrefs.com/blog/seo-meta-tags Practical robots meta tag examples with common directives including index/follow and noindex/follow combinations. Appeared in 2 searches (scores 0.780 and 0.572), providing consistent code examples for real-world robots meta tag implementation. Ahrefs is a leading SEO industry authority.

[2] web.dev: Metadata - General Meta Tags (18a2f541-ce14-4323-ab7c-44ae51e25afd) https://web.dev/learn/html/metadata Comprehensive HTML metadata implementation guidance covering proper meta tag structure and W3C-aligned best practices. Appeared in 2 searches (scores 0.720 and 0.436), ensuring correct placement of robots directives within the document head section.

[3] web.dev: HTML Links - nofollow Attribute (97648c75-021d-4e31-8acb-20d977014199) https://web.dev/learn/html/links W3C-aligned best practices for HTML link attributes including nofollow directive. Appeared in 2 searches with identical scores (0.724), explaining how nofollow functions both as a link attribute and meta robots directive.

<<<<<<< HEAD RAG Coverage: GOOD - Combines industry SEO authority (Ahrefs practical examples) with W3C-aligned best practices (web.dev). 3 sources across 2 knowledge bases providing comprehensive coverage of robots meta tag fundamentals and implementation.

24a07d01237e8d2ce45f0032ef83094634b50223

External resources

Need to verify your robots meta tag? Use our UXR SEO Analyzer extension for instant directive analysis, conflict detection, and specific recommendations.

Related articles

Related version

Detailed guide

Robots Meta Tag: Complete Technical Guide to Crawl and Indexing Control

The robots meta tag is a powerful tool for surgically precise control over what content gets indexed and how crawl budget is distributed¹

Category hub

Hub

Basic SEO Fundamentals: The Essential Elements Every Website Needs

Every successful website is built on a foundation of solid SEO fundamentals

In the same category

Introduction

Canonical URL: Avoiding Duplicate Content in Google

The canonical URL (canonical tag) is an HTML element that tells Google which is the preferred version of a page when multiple URLs exist with similar or...

Introduction

XML Sitemaps: Helping Search Engines Discover Your Content

XML Sitemaps are a fundamental tool for communicating directly with search engines