Don't miss

Wednesday, 17 December 2014

Tips And Tricks For Robots txt file


By on Wednesday, December 17, 2014

What Is Robots.txt?

When a search engine crawler comes to index your website, it will look for a special file on your website. That file is called robots.txt and it tells the search engine spiders, which Web pages of your website should be indexed and which Web pages should be ignored in your websites.
robot.txt file, robot.txt tricks, robot.txt file tips, what is robot.txt, pawan seo world, pawan sharma bhardwaj
Robot.txt

The robots.txt file is a simple text file (without use of HTML language), that must be placed in your root directory, for example:
    http://www.yourwebsite.com/robots.txt
unprotected website, protected website, site without robot.txt, site with robot.txt, robots.txt file, tips and tricks, robots.txt tips, robot.txt file tricks
Difference's between protected website & unprotected website 
1. How to allow all search engine spiders to index all files
    Use the following content for your robots.txt file if you want to allow all search engine spiders to index all files of your Web site:
    User-agent: *
    Disallow:
2. How to disallow all spiders to index any file
    If you don't want search engines to index any file of your Web site, use the following:
    User-agent: *
    Disallow: /
3. Where to find more complex examples.
    If you want to see more complex examples, of robots.txt files, view the robots.txt files of big Web sites:
Your Web site should have a proper robots.txt file if you want to have good rankings on search engines. Only if search engines know what to do with your pages, they can give you a good ranking.
Post by- Pawan Seo World

Share/Bookmark
My Photo,google seo expert, google expert, pawan seo world, pawan sharma bhardwaj, digital marketing expert, google blog in india, google seo

About Pawan Sharma Bhardwaj

I am (Google Seo Experts), I am a Freelancer Google SEO. I work from Home, Hence, i can spend more time on your site, I have Experence, Hence, I can get your Site on Top on Search, I never Ask you Money to pay in advance until i show you what is my SEO service is. I dont force you to take my service. But You Can Mail ME or Call Me, i will explain you more things on how to proceed For better seo.

9 comments:

Post a Comment