This Forum has been archived there is no more new posts or threads ... use this link to report any abusive content
==> Report abusive content in this page <==
Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How can I block semalt.com crawler?
05-11-2014, 04:45 PM
Post: #1
How can I block semalt.com crawler?
I use sitemeter.com's free version, which shows me the last 100 visitors. Lately, on some of my infrequently read sites, 80 - 90 of the last 100 have been http://semalt.com/crawler.

They come from different URLs each time, as far as I can tell. It is distorting my visit count and hiding the real visitors. Is there some magic word I can put in my .htaccess file to block it?

Ads

Find all posts by this user
Quote this message in a reply
05-11-2014, 05:01 PM
Post: #2
 
# block visits from semalt.com
RewriteEngine on
RewriteCond %{HTTP_REFERER} semalt\.com [NC]
RewriteRule .* - [F]

or you can contact with alex@semalt.com

Ads

Find all posts by this user
Quote this message in a reply
05-11-2014, 05:10 PM
Post: #3
 
RewriteCond %{HTTP_USER_AGENT} ^.*(semalt|otherbadbot|otherbadcrawler).*$ [NC]
RewriteRule . - [F,L]

separate some bot that you want to block using pipeline |

you don't need to write full name of the crawler only specific name like semalt, google, yahoo, baidu or other bot that you want to block. because it will be match with wildcard name wildcard
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)