+- Twitist Forums (http://twitist.com)
+-- Forum: General Social Media & Marketing Forums (/forum-8.html)
+--- Forum: SEO Search Engine Optimization (/forum-30.html)
+--- Thread: Webmasters and seo people, why can't Google crawl my new site? (/thread-149248.html)
Webmasters and seo people, why can't Google crawl my new site? - Dustin - 04-28-2014 05:22 PM
I have a new site I purchased on domain.com and I am using the Word Press platform and theme. I am trying to use the Google webmasters tool.
1) I received an error message that reads "Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. "
2) when I go to robots.txt page on my site the only thing I see on the page is "User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/"
3) I don't know how to allow Google to crawl my site, I thought It's done automatically as with my previous sites..
Please if anyone knows exactly the procedures of how I can enable Google bot to crawl my site I will really appreciate it. Thank you! Mike
- Borislav Arapchev - 04-28-2014 05:31 PM
Hi!
Last recommendations from Google are to NOT use at all robots.txt.
I usually don't use it, years ago and I'm OK.