Problem with wordpress robots.txt?

Question

I submitted my wordpress site to google, and on google webmaster tools, it came up with a crawl error that says "restricted by robots.txt".

What is this? Will it stop my site being indexed by google? And if it does, how do I fix it?

Answer 1

There's nothing wrong with restricting access with robots.txt - there are many legitimate reasons to do that (for example, on directories that have nothing but executable code or program data). It should not affect your indexing or ranking of the other pages that are not protected by it.



Now, if there's a robots.txt on a path that you DO want to be indexed (worst case would be on the directory where your home page is), then you'll need to remove or modify that file. There is more than enough online documentation on using robots.txt, so you shouldn't have any trouble modifying it, if that's what you choose to do.

Answer 2

Hi Reece,



Your robots.txt file is separate from your WordPress dashboard, unless you have a plugin to manage your robots.txt file. You can use either an FTP program or your web host's File Manager to view and edit your robots.txt file. Look for something similar to this:



User-agent: *

Disallow: /



That tells all search engines to not index an entire site. For more information on robots.txt:



http://www.webhostinghub.com/support/website/how-to/using-robotstxt



Hope this helps!

Answer 3

Login to your wordpress admin.

Go to settings/privacy

Make sure the radio button for Allow search engines to index this site. is checked. If it's not, then select it and save changes.

Give it a couple of minutes and go back to webmaster tools and try again.

Answer 4

Hello Reece,



Googlebot download robots.txt file time to time. Now see the time when googlebot downloaded your site robots.txt previously and I think that time crawler was restricted. Let googlebot download it again and restriction will automatically disapear.

Answer 5

To download WordPress for free you can click here http://j.mp/1yyndd5



If you want to get started in blogging, you really can't make a much better choice than WordPress.