Rating 0 stars - 0 votes

Set Magento 2 Noindex and Nofollow Tags

When you're starting a new project on dev server or make some transactional pages, you may need to hide some of your pages or whole website from search engines not to be indexed beforehand. The best way to tell robots visiting your site not to index or follow your content is to use robots.txt file and nofollow and noindex meta tags in html head of the page(s). Magento website allows you to configure such settings from the admin panel with no code used.


What Is robots.txt File And Why Do You Need It?

Let's say, some web robots want to visit your site https://store.emagicone.com and index or track its content. Before starting the indexation the robots will first enter your https://store.emagicone.com/robots.txt file and check it. This file is located in the root of your Magento site and contains the instructions for robots on how to process or not process your site.


Setting Noindex and Nofollow in Magento 2

There are several methods to instruct the robots not to index certain store content. Below we will describe how to close from indexing the entire store, pages, products and categories.


Method 1. Setting Noindex/Nofollow For Magento 2 Store

If you run Magento 2.1.x go to Stores > Configuration > General (On the left) > Design section

Magento 2 robots.txt file settings
Access Magento 2 robots.txt file settings via store admin

If you run Magento 2.2.0, 2.2.1 or 2.2.2 store in the admin panel choose Content > Design section — Configuration:


Magento 2 Admin Panel Content Configuration
To close store from indexing choose Content section in the admin panel

You will find there a table containing all your storeviews. Click “Edit” next to a store configuration:

Magento 2 Edit Design Configuration
Edit Design Configuration to close the store from indexing

The window with “Global” settings will open. Scroll down to “Other Settings” and click on “Search Engine Robots” to expand this section. Here you will see the “Default Robots” field. Click to see the drop-down list:

Magento 2 Content Global Settings
To close the store from indexing select required action from the drop-down list

There are four “Default Robots” options available:

  • INDEX, FOLLOW - allows web robots to index your site and track it for changes;
  • NOINDEX, FOLLOW - instructs robots not to index your site, but allows to track it for changes;
  • INDEX, NOFOLLOW - instructs robots to index your site, but not to check it for changes furtherly.
  • NOINDEX, NOFOLLOW - forbids the robots neither to index nor to track you site.

You can select the option based on your purposes or use “Edit custom instruction of robots.txt File” field, in case you need to block access to some specific folders or URLs.

Input the following directives for robots into this field depending on your needs:


To block all robots from certain folders:

User-agent: *

Disallow: /folder name


To block specific robots from certain folders:

User-agent: robot name

Disallow: /folder name


To block some bots to access one folder, but allow to access another one:

User-agent: robot name

Disallow: /folder name Allow: /folder name


Method 2. Setting Noindex/Nofollow For Specific Magento 2 Pages

There is another way to block robots from processing your web pages.


From the admin panel on the left, select “Content” section. Under the “Elements” select “Pages”. You will find a table containing all pages of your store. Click “Select” on the right and choose the “Edit' option from the drop-down list:

Magento 2 Edit Page to Close from Indexing
Edit Page to close it from indexing

Expand the “Design” section and find the “Layout Update XML” field. In order to prevent this page from indexing and following, insert this code into the field:

Magento 2 Close Page From Indexing
Insert the code and save the page to close it from indexing

Method 3: Setting Noindex/Nofollow For Specific Magento 2 Categories and Products

The above indicated code can also be used to configure permissions to some of your category or product pages whether in 2.1.x or in 2.2.


To block the product page select “Products” from the “Catalog” section on the admin panel. Press “Edit” on the right side of the product field:

Magento 2 Edit Product to Close From Indexing
Edit the product to close it from Indexing

Scroll down and find the “Design” section. Expand it and enter the piece of code into the 'Layout Update XML' field. Press “Save”:

Magento 2 Close Product From Indexing
Insert the code to close the product from indexing

To block category page select “Categories” from the “Catalog” section and find the necessary category in the list on the right. Find 'Design' section and enter the above indicated code into 'Layout Update XML' field:

Magento 2 Close Category from Indexing
Insert the code and save the category to close it from indexing

Conclusion: Do I Need to Block Access to Some of My Links?

Definitely, yes. It is essential for an online store owner to be able to manage access to some of his/her site’s folders or URLs by forbidding or allowing web robots to index or follow them. Beside limiting access for unknown crawlers, you can considerably improve your SEO by applying these restrictions.


The thing is that, when filling your site with content, you probably strive to optimize it for Google as much as possible in order to show up in the first positions of search results. Although, some of your site links might have some negative influence on your Google rate, so it would be appropriate to disallow search engine robots to index or track them.


Magento provides its users with a majority of ways to manage site access in few clicks directly from admin panel. Take advantage of any of them and keep your web store protected from any undesirable indexing or tracking.




Related Articles:


Card image

How to Optimize your Magento 2 SEO?

Detect SEO issues of the whole Magento store at once and automatically solve them clicking on a button. Read More...

Card image

Magento 2 Meta Tags Optimization

Learn how to optimize your Magento 2 meta tags and move your pages higher in search results. Read More...

Card image

Magento 2 XML Sitemap

How to make that web crawlers correctly understood your website organization and indexed its content faster. Read More...

Card image

Magento 2 SEO Features

Find out how track, analyze and optimize your SEO Magento 2 on regular basis and omprove your SEO ranking. Read More...



COMMENTS
Be the first to comment.