What is Robot.txt ?

Discussion in 'SEO & Traffic Discussion' started by fish2012, Sep 27, 2012.

  1. fish2012

    fish2012 Member

    Joined:
    Aug 14, 2012
    Messages:
    91
    Likes Received:
    56
    Trophy Points:
    18
    Location:
    U.S
    Hi friends,

    I would like to know that what is robot.txt and why it use in web pages.... thanks
     
    #1
    Jerardx likes this.
  2. wannabeseowizzard

    wannabeseowizzard Super Moderator Super Moderator

    Joined:
    Mar 30, 2011
    Messages:
    2,895
    Likes Received:
    2,096
    Trophy Points:
    113
    Gender:
    Male
    Occupation:
    Web developer and SEO specialist
    Location:
    Scottsdale, Arizona
    Quit spamming the forum so people see your "live links".... Read the forum rules and follow the like the rest of us need to.. They should ban you but probable wont..
     
    #2
    Jerardx likes this.
  3. mukundguruji

    mukundguruji Active Member

    Joined:
    Apr 27, 2010
    Messages:
    121
    Likes Received:
    121
    Trophy Points:
    43
    Spammer, don't use live links. Moderators please step in. I have seen this fellow posting live spamming links in other threads also. Take action!.
     
    Last edited by a moderator: Sep 27, 2012
    #3
    Jerardx likes this.
  4. surajsingh

    surajsingh New Member

    Joined:
    Oct 1, 2012
    Messages:
    7
    Likes Received:
    7
    Trophy Points:
    3
    Robots.txt is a file through which also you can guide search engines crawl or not to crawl certain sections of your website.
    Google specifically follows instructions given in this robots.txt file..
     
    #4
    Jerardx likes this.
  5. Neocon

    Neocon Banned

    Joined:
    Oct 4, 2012
    Messages:
    253
    Likes Received:
    64
    Trophy Points:
    28
    Robot.txt is a map through which you show your pages to google and increase index rate
     
    #5
    Jerardx likes this.
  6. Walker

    Walker Well-Known Member

    Joined:
    Aug 4, 2012
    Messages:
    186
    Likes Received:
    479
    Trophy Points:
    63
    Why ask here, just Google it F.C.O.L.

    Walker
     
    #6
    Jerardx likes this.
  7. jennifersmith2012

    jennifersmith2012 New Member

    Joined:
    Oct 4, 2012
    Messages:
    8
    Likes Received:
    2
    Trophy Points:
    3
    Robot.txt is a file which gives instructions to the Google about the website that what section or part of the site must crawl and other not. There are some disallow options available which shows to robots of search engines that this part is official and not for their use so they don’t have to crawl that part.
     
    #7
    Jerardx likes this.
  8. anishkumar854

    anishkumar854 New Member

    Joined:
    Jun 11, 2012
    Messages:
    2
    Likes Received:
    2
    Trophy Points:
    3
    Home Page:
    Robots.txt is a text file you put on your site to tell search robots that which website page you dont want to crawl.
     
    #8
    Jerardx likes this.
  9. dean1122

    dean1122 New Member

    Joined:
    Sep 21, 2012
    Messages:
    10
    Likes Received:
    7
    Trophy Points:
    3
    Home Page:
    hi,
    instead of doing discussion about spamming the forum or not . i will share the ans.

    Robots.txt is a text file that tells search engine spiders , also known as search engine robots, which parts of your website they can enter and which parts they can’t .

    dean
     
    #9
    Jerardx likes this.
  10. Orange Creator

    Orange Creator New Member

    Joined:
    Sep 21, 2012
    Messages:
    23
    Likes Received:
    7
    Trophy Points:
    3
    Occupation:
    Software Engineer
    Location:
    United Kingdom
    Robots.txt are files you place in the root directories of the pages you don't want Google to crawl which means they won't index these pages and none of them will be found in the search results. Best example would be a Members page or download page. If a member has to make a purchase to go to a downloads page, make sure the page has a robots.txt file in the root folder so the big 'G' won't allow people to find it in search engines, stopping users from going direct to your download pages and taking all your products without a purchase.

    Hope this info helps.
     
    #10
    Jerardx likes this.
  11. guptaabhijit318

    guptaabhijit318 New Member

    Joined:
    Dec 18, 2012
    Messages:
    10
    Likes Received:
    7
    Trophy Points:
    3
    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. It is important to explain that robots.txt is not a way from preventing search engines from crawling your site.
     
    #11
    Jerardx likes this.
  12. sutjipto

    sutjipto New Member

    Joined:
    Dec 22, 2012
    Messages:
    11
    Likes Received:
    4
    Trophy Points:
    3
    Robots.txt are files contained in the root directory of a site that allows you to control certain pages to be indexed by the search engine spiders.
     
    #12
    Jerardx likes this.
  13. DavidLuis

    DavidLuis New Member

    Joined:
    Dec 22, 2012
    Messages:
    25
    Likes Received:
    9
    Trophy Points:
    3
    Gender:
    Male
    Location:
    Bandung, Indonesia
    Home Page:
    What I do know, robots.txt is a list of commands used to instruct robots search engines (like Google, Bing, Yahoo, etc.) index file or files on the server. I think this is important, because Search Engines in mengcrawling a website it blindly. and sometimes there is a folder in your website or confidential files that do not want us to tell the search engines. (because if the index by search engines, it can be seen also by others).
     
    #13
    Jerardx likes this.
  14. CheshireCatSX

    CheshireCatSX New Member

    Joined:
    Nov 11, 2012
    Messages:
    7
    Likes Received:
    2
    Trophy Points:
    3
    Location:
    Moscow
    Home Page:
    Robots.txt - a text file located on a site that is intended for search engine robots. In this file, the webmaster can specify the index of the site for all the robots at once, and for each search engine individually.

    Use any text editor (such as Notepad or WordPad'om), create a file called "robots.txt" and fill it out according to the rules. After this, you must download the file to the root directory of your website.
     
    #14
    Jerardx likes this.
  15. topdog5

    topdog5 Well-Known Member

    Joined:
    Dec 4, 2010
    Messages:
    424
    Likes Received:
    644
    Trophy Points:
    93
    Occupation:
    marketing, making money
    Location:
    USA
    Admin, please close this thread. It's just being used for spam. Look at the repeat answers from new members with just 2 or 3 posts.
     
    #15
    Jerardx likes this.
  16. DavidLuis

    DavidLuis New Member

    Joined:
    Dec 22, 2012
    Messages:
    25
    Likes Received:
    9
    Trophy Points:
    3
    Gender:
    Male
    Location:
    Bandung, Indonesia
    Home Page:
    Robots.txt can be called a file that allows you to adjust the flow of the search engine robots (search engines) on our site. For web robots.txt which we can arrange to index and which ones should not. and we also can avoid duplicate content.
     
    #16
    Jerardx likes this.
  17. ruworth

    ruworth Banned

    Joined:
    Jun 9, 2010
    Messages:
    973
    Likes Received:
    615
    Trophy Points:
    93
    Location:
    UK
    Home Page:
    A Robot.txt file turns all your .txt files into robots :) Happy New Year.
     
    #17
    Jerardx likes this.
  18. pdforum

    pdforum New Member

    Joined:
    Jan 21, 2013
    Messages:
    17
    Likes Received:
    5
    Trophy Points:
    3
    Robots.txt is a text file which describe spiders of search engine and crawls of your website.
     
    #18
    Jerardx likes this.
  19. turbo354

    turbo354 Active Member

    Joined:
    Jun 10, 2010
    Messages:
    161
    Likes Received:
    32
    Trophy Points:
    28
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enterâ€￾ on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
     
    #19
    Jerardx likes this.
  20. King John

    King John Well-Known Member

    Joined:
    Aug 22, 2011
    Messages:
    1,120
    Likes Received:
    2,442
    Trophy Points:
    113
    Whilst it doesn't happen very often it is useful to check the robots.txt of sites for download directories. Sometimes unsuspecting site owner will put dl directories in the robots.txt file to prevent them being indexed. There are of course other ways to protect downloads (not against seasoned BHT'ers of course).
     
    #20
    Jerardx likes this.

Share This Page