+- Twitist Forums (http://twitist.com)
+-- Forum: General Social Media & Marketing Forums (/forum-8.html)
+--- Forum: SEO Search Engine Optimization (/forum-30.html)
+--- Thread: robots.txt file and SEO? (/thread-164340.html)
robots.txt file and SEO? - thegrons - 06-15-2014 09:07 PM
I currently have the standard robots.txt file that Wordpress includes in their themes. As a result many of my pictures on the website have been indexed by google. If I add a robots.txt file to my site now, will the 404 errors hurt my rankings? Will adding a robots.txt file like the one below help my site in terms of page rank and SERP's?
User-agent: *
Disallow: /wp-content/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /trackback/
Disallow: /cgi-bin/
Allow: /wp-content/uploads/
User-agent: Googlebot
Disallow: /*.pdf$
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
Disallow: /*?*
Disallow: /wp-*
Disallow: */trackback/
Disallow: /cgi-bin/
Disallow: /go/
Allow: /wp-content/uploads/
User-agent: Googlebot-Image
Allow: /*
- Maxinder - 06-15-2014 09:17 PM
That robot txt file will more than likely hurt your rankings, you're blocking off a lot of important things which when indexed can help you get traffic, like for example - your images. I don't recommend it. You're also telling it not to index ".php" files --- if i recall, wordpress is a php based engine, you might end up preventing your website from being indexed.
I would stick with the default robots txt.