Results 1 to 4 of 4

Thread: Bot/Spider protection?

  1. #1

    Default Bot/Spider protection?

    Any take on how to keep unwanted guest away?

    I tried a few automatic ways, but now I relly more and more on just configuring the server with data from logwatch and the firewall.
    (i.e. any kind of stupid behavior that I notice gets you blacklisted).

    My nightly logwatch can look like this:
    Code:
    ################### Logwatch 7.4.3 (12/07/16) #################### 
            Processing Initiated: Sat Nov 24 06:25:16 2018
            Date Range Processed: yesterday
                                  ( 2018-Nov-23 )
                                  Period is day.
            Detail Level of Output: 0
            Type of Output/Format: mail / text
            Logfiles for Host: server.fantasy-freak.com
     ################################################################## 
    
     --------------------- Amavisd-new Begin ------------------------ 
    
            2   Total messages scanned ------------------  100.00%
       33.508K  Total bytes scanned                         34,312
     ========   ==================================================
    
            2   Passed ----------------------------------  100.00%
            2     Clean passed                             100.00%
     ========   ==================================================
    
            2   Ham -------------------------------------  100.00%
            2     Clean passed                             100.00%
     ========   ==================================================
    
    
     ---------------------- Amavisd-new End ------------------------- 
    
    
     --------------------- httpd Begin ------------------------ 
    
    
     A total of 2 sites probed the server 
        213.186.170.226
        5.188.210.12
    
     Requests with error response codes
        400 Bad Request
           null: 3 Time(s)
           /: 2 Time(s)
    http://5.188.210.12/echo.php: 1 Time(s)
           mstshash=Administr: 1 Time(s)
        403 Forbidden
           /: 244 Time(s)
        :
        499 (undefined)
           /index.php?action=dlattach;attach=81;type=avatar: 1 Time(s)
    
     ---------------------- httpd End ------------------------- 
    
    
     --------------------- Postfix Begin ------------------------ 
    
            2   Miscellaneous warnings                           2
    
       33.509K  Bytes accepted                              34,313
       34.831K  Bytes delivered                             35,667
       34.540K  Bytes forwarded                             35,369
     ========   ==================================================
    
            2   Accepted                                   100.00%
     --------   --------------------------------------------------
            2   Total                                      100.00%
     ========   ==================================================
    
            6   Removed from queue                               6
            2   Delivered                                        2
            2   Forwarded                                        2
          126   Postscreen                                     126
    
    
     ---------------------- Postfix End ------------------------- 
    
    
     --------------------- SSHD Begin ------------------------ 
    
    
     Users logging in through sshd:
        rille:
           192.168.1.99 (LAPTOP): 3 times
    
     **Unmatched Entries**
     syslogin_perform_logout: logout() returned an error : 3 time(s)
    
     ---------------------- SSHD End ------------------------- 
    
    
     --------------------- Disk Space Begin ------------------------ 
    
     Filesystem      Size  Used Avail Use% Mounted on
     /dev/root        30G  4.7G   24G  17% /
     /dev/mmcblk0p1   43M   22M   21M  52% /boot
    
    
     ---------------------- Disk Space End ------------------------- 
    
    
     ###################### Logwatch End ######################### 

  2. #2
    English language questions as captchas have always gotten me the most success. I've never cared about preventing access to bots though, just stopping spam.

    edit: Every time bots have gotten through the system, it's been because the bots reposted the question on Yahoo Answers or equivalent and copy + pasted the answer given there. You prevent that by asking questions specifically about your own site/forum, like what's the name of the second forum category on your forum. They still repost the question on Yahoo Answers, but humans can't answer it without knowing where they copy + pasted the question from, so it's safe.
    Last edited by Wraith; 11-24-2018 at 07:32 PM.

  3. #3
    Usually does the trick.

    I find that the most annoying scanners etc. are from Ukraine/Russia, China, Korea, Germany and U.S.

    The question is to whatever allow indexers or not. Can be hard to prevent without require login.

  4. #4
    Most indexers obey the robots.txt file, but they're generally harmless even if they don't.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •