What is Robot.txt ?

Discussion in 'SEO & Traffic Discussion' started by fish2012, Sep 27, 2012.

  1. liamgood

    liamgood New Member

    Joined:
    Mar 23, 2018
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    Gender:
    Male
    Robots.txt file is a text file that can be saved to a website’s server. It determines if and when the search engine crawlers can visit a website’s subpages and include them in their index. In doing this, certain subpages can be excluded from the search results. Talking about file recognizing tools and converters, I know nothing better than Filetypeadvisor .
     
    #21
  2. islf

    islf New Member

    Joined:
    Aug 31, 2017
    Messages:
    28
    Likes Received:
    2
    Trophy Points:
    3
    The robots.txt file, also known as the robots exclusion protocol or standard, is atext file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. ... The slash after “Disallow” tells the robot to not visit any pages on the site.
     
    #22
  3. deepakweb

    deepakweb New Member

    Joined:
    Nov 5, 2018
    Messages:
    10
    Likes Received:
    3
    Trophy Points:
    3
    Location:
    Malaysia
    Home Page:
    Robots.txt is a text file that is used to tell the Google to crawl the page or not.
     
    #23

Share This Page