robots google

  FOEYDO 18:12 07 Apr 06

Hi, I have uploaded an xml sitemap to Google and they say my robots file is 404 error, I do not have a robots file as I want no exclusions, how do I write a file that syas this? Also how long do you think it takes to get indexed in Google?Thanks

  AndySD 21:06 07 Apr 06

Use Notepad

type in

User-agent: *


save the file as


then upload this file to the website.

  beynac 22:45 07 Apr 06

Isn't this the wrong way round? Your text file stops robots, I thought that FOEYDO was asking for a way to encourage them.

  AndySD 23:35 07 Apr 06

User-agent: *

The asterisk (*) in the User-agent field is shorthand for "all robots".


Because nothing is disallowed, everything is allowed.

It allows all robots to crawl all the files and directories.

  beynac 07:17 08 Apr 06

I think that we will have to agree to differ. As far as I'm concerned "User Agent: * Disallow" is saying "all robots: disallow". I don't understand where you get "nothing is disallowed" from.

  beynac 08:38 08 Apr 06

I suggest that you contact Google and ask them what they need. I would expect it to be a text file on your website called "robots.txt" which contains:

User-agent: *

I don't want Google, or any other search engine to list my website (it is only there for me to experiment with html, css, php and other coding). Googlebots visit my site regularly but only look at my "robots.txt" file, which says "Disallow". This stops them looking any further.

  FOEYDO 08:42 08 Apr 06

I want to allow them all.

  beynac 09:08 08 Apr 06

I have been thinking about your post and had another look at my "robots.txt" file. I have a forward slash after the "Disallow:". I now understand where you got the "nothing is disallowed" from.

Ignore my previous posts: AndySD is right. I would still suggest that you contact Google.

This thread is now locked and can not be replied to.

Surface Pro (2017) vs Surface Pro 4

Where HTML5 is headed next

MacBook Pro v Surface Pro 5