min read

No Index Tag

Written by

Search Historian

Edited by

Emanuel Skrobonja

TL;DR:

The No Index Tag is a code snippet in the head section of your webpage that tells search engines not to index the page. As a result, pages with a noindex tag won't appear in search results.

What is Noindex Tag?

Noindex is an HTML meta tag that is used to stop search engines from indexing your web page.

With Noindex, you have the ability to selectively block certain webpages from being indexed by search engines.

This helps ensure that search engines prioritize indexing the most relevant and valuable pages on your site, improving overall visibility and traffic.

It looks like this:

<meta name="robots" content="noindex">

Adding this code to the head tag of your site will stop all search engines from indexing your page.

To stop Google specifically without blocking other engines, you can use:

<meta name="googlebot" content="noindex">

What is NoFollow Tag?

Nofollow tag is another HTML meta robots tag that tells crawlers to ignore links and doesn’t attribute PageRank to any specific link.

When used not within a specific link, but within metadata, the Nofollow tag tells search engines to ignore all links on that page.

That’s why it is commonly used together with Noindex tag:

<meta name="robots" content="noindex, nofollow">

Note that you can also use the none tag to achieve the same thing as Noindex, nofollow like so:

<meta name="googlebot" content="none">

How to Prevent Pages from Indexing in Webflow?

Static Pages

To prevent pages from indexing, you can toggle Sitemap Indexing in your Page Settings

The Sitemap indexing toggle excludes your page from the auto-generated sitemap, and adds <meta name="robots" content="noindex"> to your site page.

This prevents the page from being crawled and indexed by search engines.

Dynamic Pages

For dynamic pages, you don't have this toggle, so you can't remove the page from the autogenerated sitemap.

However, you can still add a robots meta tag to your <head> code, and simply bind it with an CMS Option field.

This way, you can choose which items in your collection you want indexed.

Here's how to do it:

Collection settings — 1. Add an Indexing field to your CMS collection

Setting indexing for individual CMS items — 2. Set the indexing option for all of your items

Binding your CMS field to your robots tag — 3. Bind your CMS field with the robots meta tag

Create an option field in your CMS collection with the following options:
- none - for pages that shouldn't be indexed
- all - for pages that need to be indexed
Select 'all' or 'none' for each item in your collection
Add the a robots meta tag to your <head> code, and set the content dynamically
Publish your site!

When to Use Noindex

The Noindex tag doesn’t stop users from using or finding that page within your Webflow website.

It just stops the possibility of users landing on your page from search engines directly.

In other words, pages that are not indexed will not be shown in search results.

This means that you should set Noindex tags for pages that are:

Duplicate pages
Thin content pages
Landing pages (not search intent optimized)
Marketing or ad traffic pages
Certain types of user-generated content pages
Optionally, you can also add a Noindex tag to protected pages (behind a login).

As you can see, you should add Noindex tag to all pages on your Webflow website that:

Have a better or alternative page version
Don’t have value for search engine results

Why Noindex Matters for SEO?

Search engine rankings are based on trust in your website.

Based on the trust search engines have for your website, they will allocate a certain amount of “crawl budget”.

Think of the crawl budget as a rule for crawlers for how much energy it’s worth investing in crawling your website, before pausing for a while.

To keep their index up to date, crawlers recrawl each page they have on their index, from time to time.

Now, let's say that 50% of your indexed pages are duplicates, marketing landing pages, and other pages that have not been optimized for any type of search intent.

Well, this means crawlers will be checking those pages anyway to ensure you haven’t improved or optimized them.

And if you never plan to optimize them? Then you are wasting your crawl budget on pages that will never rank anyway.

That’s when you want to use a Noindex tag.

Disclaimer! Adding a Noindex tag should be done ONLY for pages that you never will want to show up in search results.

Not even in the future.

For cases where you are not sure yet, adding a canonical tag can be enough.

Can Search Engines Index Pages with Noindex Tag?

No, search engines should not index your page if it has a Noindex tag in it.

Difference Between Robots Disallow and NoIndex Tag

Disallow within your robots.txt tells crawlers not to index or access specific parts of a website (Pages, CMS Collections or folders).

On the other hand, the Noindex meta tag tells not to index specific pages to crawlers after they have already started crawling that page.

So to sum up, Noindex instructs crawlers that they shouldn’t index a particular page, whereas Disallow in your robots.txt file tells crawlers that they shouldn’t even access that page.

Now you might be wondering…

What Happens if You Add No Index Tag to a Page That is Already Indexed?

If a search engine has indexed your page before, it must crawl the page again to detect a new Noindex tag.

When Googlebot crawls your site again, it will stumble upon the Noindex tag, and your page will be removed from search engine results.

But, there's one thing you need to keep in mind.

Be careful with adding disallow to robots.txt

Adding a disallow rule for the page in robots.txt can prevent the search engine from crawling it again, keeping your page indexed unintentionally.

This is because the crawler won't visit the page to learn it should be de-indexed.

To ensure a page is removed from the search engine index, let it be crawled once more before blocking further crawls with robots.txt.

Combining NoIndex Tag Together with Robots.txt Disallow

In most cases, you will want to add all slugs you set the Noindex tag for into your robots.txt file in your Webflow project.

As mentioned before, if the page is already indexed don’t forget that search engines should crawl your page at least once before you disallow crawling it in your robots.txt file!

To sum up

Using the Noindex tag together with robots.txt disallow in your Webflow site's SEO strategy can greatly benefit your search engine indexing efforts.

With Noindex, you have the ability to selectively block certain webpages from being indexed by search engines.

This can be particularly useful for pages that contain duplicate content, are still under development, or are intended for internal use only.

The Noindex tag helps ensure that search engines prioritize indexing the most relevant and valuable pages on your site, improving overall visibility and traffic.