Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Our Robots.txt Generator tool is designed to help webmasters, SEOs, and marketers generate their robots.txt files without a lot of technical knowledge. Please be careful though, as creating your robots.txt file can have a significant impact on Google being able to access your website.

If you’re interested in setting up your very own robots.txt file on your website, there are a few things you should know first! Your robots.txt file tells search engine crawlers which files to index and which ones to ignore, and can also be used to stop bots from accessing certain pages altogether if necessary. However, there are some common pitfalls that website owners make when setting up their robots.txt files, so this Robots.txt Generator tool was designed to help webmasters, SEOs, and marketers generate their robots.txt files without a lot of technical knowledge required!

Although our tool is straightforward to use, we would suggest you familiarize yourself with Google’s instructions before using it. This is because incorrect implementation can lead to search engines like Google being unable to crawl critical pages on your site or even your entire domain, which can very negatively impact your SEO.

What Is a Robots.txt File?

A robots.txt file is a really simple, plain text format file. Its core function is to prevent certain search engine crawlers like Google from crawling and indexing content on a website for SEO.

If you’re not certain whether your website or your client’s website has a robots.txt file, it’s easy to check:

Simply type yourdomain.com/robots.txt. You’ll either find an error page or a simple format page.  If you are using WordPress and you have Yoast installed, then Yoast can also build the text file for you as well.

User-agent:

Since each search engine has its own crawler (the most common being Googlebot), the ‘user-agent’ allows you to notify certain search engines that the following set of instructions is for them.

You’ll commonly find ‘user-agent’ followed by a *, otherwise known as a wildcard. This indicates that all search engines should take note of the next set of instructions.There is also typically a default phrase following the wildcard that tells all search engines not to index any webpage on your site.

The default phrase is to disallow the symbol ‘/’ from being indexed, which essentially prohibits every internal page except your main URL from the bots. It’s important that you check for this phrase and immediately remove it from your robots.txt page.

It will look something like this:

User-agent: *
Disallow: 

Disallow:

The term ‘Disallow’ followed by a URL slug of any kind gives strict instructions to the aforementioned user-agent, which should appear on the line above.

For instance, you’re able to block certain pages from search engines that you feel are of no use to users. These commonly include WordPress login pages or cart pages, which is generally why you see the following lines of text within the robots.txt files of WordPress sites:

User-agent: *
Disallow: /wp-admin/

Test Your Robots.txt File

If you are not a webmaster or developer, you may be asking yourself: What is robots.txt? Robots.txt is a text file that can be placed in any directory on your site to tell search engines and bots what they can and cannot do in relation to that specific folder or page. It's essentially instructions for bots that crawl your site looking for pages to rank (Google), index (Bing) and serve up in search results (Baidu). Think of it as a set of rules all bots are required to follow so they don't get confused on how they should crawl your site, index it, serve it up etc., while also respecting your wishes and being respectful of your content at all times.