+- Twitist Forums (http://twitist.com)
+-- Forum: Other forums (/forum-31.html)
+--- Forum: General Internet related Qustions (/forum-32.html)
+--- Thread: How can I block semalt.com crawler? (/thread-155344.html)
How can I block semalt.com crawler? - Ted Pack - 05-11-2014 04:45 PM
I use sitemeter.com's free version, which shows me the last 100 visitors. Lately, on some of my infrequently read sites, 80 - 90 of the last 100 have been http://semalt.com/crawler.
They come from different URLs each time, as far as I can tell. It is distorting my visit count and hiding the real visitors. Is there some magic word I can put in my .htaccess file to block it?
- Ngoc - 05-11-2014 05:01 PM
# block visits from semalt.com
RewriteEngine on
RewriteCond %{HTTP_REFERER} semalt\.com [NC]
RewriteRule .* - [F]
or you can contact with alex@semalt.com
- hendro - 05-11-2014 05:10 PM
RewriteCond %{HTTP_USER_AGENT} ^.*(semalt|otherbadbot|otherbadcrawler).*$ [NC]
RewriteRule . - [F,L]
separate some bot that you want to block using pipeline |
you don't need to write full name of the crawler only specific name like semalt, google, yahoo, baidu or other bot that you want to block. because it will be match with wildcard name wildcard