Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Robots.txt generator is a tool which will help you to create the robots.txt file for your website in just few steps.

What is robots.txt?

It is a file which holds information for search engine bots to tell them which section to be crawled and which not to be crawled.

If you do not have configured robots.txt file in your website, then bots will crawl the pages that you don't want to be crawled such as admin pages.

What are benefits of robot.txt generator?

This is a best tool to protect your blog or website form getting its unwanted portions to be visible on search engines.

If your admin page is directly visible on search engine, it will be more vulnerable for your website protection.

Sometimes there are some portions you don't want to be ranked on search engines, this file will protect those portions from getting crawled by search engine bots.

How it works?

The robot.txt file disallows search engines to crawl the protected portions of your website.

It have the codes which decides which portion of the website search engine bots can access and which portion is disallowed for search engine to access.

Because of this, search engine bots can not be able to crawl the disallowed sections of your website and only needed sections gets crawled.

How to use it?

This tool is completely easy to use as only three step action is required to get robots.txt file for your website.

Step1: Enter your website address you want to generate robots.txt file for (Make sure robots are allowed).

Step2: Now enter the sections path without your website address you want to disallow in 6 boxes given in 'Redirected Directories' section.

Step3: Click on 'create' or 'create and save' button depending on your need. And Done! You get the robots.txt code or file depending on your button click.

You can check more of our useful SEO tools like Keyword Suggestion Tool, Meta Tag Generator and Htaccess Redirect Generator.