What is Robot.txt ?

Discussion in 'SEO & Traffic Discussion' started by fish2012, Sep 27, 2012.

  1. liamgood

    liamgood New Member

    Joined:
    Mar 23, 2018
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    Gender:
    Male
    Robots.txt file is a text file that can be saved to a website’s server. It determines if and when the search engine crawlers can visit a website’s subpages and include them in their index. In doing this, certain subpages can be excluded from the search results. Talking about file recognizing tools and converters, I know nothing better than Filetypeadvisor .
     
    #21
  2. islf

    islf Member

    Joined:
    Aug 31, 2017
    Messages:
    33
    Likes Received:
    2
    Trophy Points:
    8
    The robots.txt file, also known as the robots exclusion protocol or standard, is atext file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. ... The slash after “Disallow” tells the robot to not visit any pages on the site.
     
    #22
  3. deepakweb

    deepakweb New Member

    Joined:
    Nov 5, 2018
    Messages:
    15
    Likes Received:
    4
    Trophy Points:
    3
    Location:
    Malaysia
    Home Page:
    Robots.txt is a text file that is used to tell the Google to crawl the page or not.
     
    #23
  4. islf

    islf Member

    Joined:
    Aug 31, 2017
    Messages:
    33
    Likes Received:
    2
    Trophy Points:
    8
    The robots.txt file, also known as the robots exclusion protocol or standard, is atext file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. ... The slash after “Disallow” tells the robot to not visit any pages on the site.
     
    #24
  5. tsavo

    tsavo New Member

    Joined:
    Jan 9, 2019
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    Gender:
    Male
    Occupation:
    TEXTILE
    Location:
    TEXTILE COLONY LUDHIANA 141003 (INDIA)
    Home Page:
    Robots.txt file is used to communicate to the search engines-Google, Yahoo, Bing, etc.
     
    #25
  6. Katson

    Katson New Member

    Joined:
    Oct 29, 2018
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    Gender:
    Male
    Location:
    Ludhiana
    Robots.txt is used to give instruction to the search engine for telling them not to visit to pages reciting in the file.

    user agent: *
    Disallow: /
    ----------------
    user agent: *
    allow: /
     
    #26
  7. tsavo

    tsavo New Member

    Joined:
    Jan 9, 2019
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    Gender:
    Male
    Occupation:
    TEXTILE
    Location:
    TEXTILE COLONY LUDHIANA 141003 (INDIA)
    Home Page:
    Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
    User-agent:*
    Disallow:/
     
    #27
  8. MarvinFew

    MarvinFew Member

    Joined:
    Feb 12, 2019
    Messages:
    271
    Likes Received:
    0
    Trophy Points:
    16
  9. deepakweb

    deepakweb New Member

    Joined:
    Nov 5, 2018
    Messages:
    15
    Likes Received:
    4
    Trophy Points:
    3
    Location:
    Malaysia
    Home Page:
    Robots.txt is used to keep the confidential page from the search engine index. It will discourage the search engine to crawl or index.
     
    #29

Share This Page