Bot/Spider protection?

[rille]

Any take on how to keep unwanted guest away?

I tried a few automatic ways, but now I relly more and more on just configuring the server with data from logwatch and the firewall.
(i.e. any kind of stupid behavior that I notice gets you blacklisted).

My nightly logwatch can look like this:

################### Logwatch 7.4.3 (12/07/16) #################### 
        Processing Initiated: Sat Nov 24 06:25:16 2018
        Date Range Processed: yesterday
                              ( 2018-Nov-23 )
                              Period is day.
        Detail Level of Output: 0
        Type of Output/Format: mail / text
        Logfiles for Host: server.fantasy-freak.com
 ################################################################## 

 --------------------- Amavisd-new Begin ------------------------ 

        2   Total messages scanned ------------------  100.00%
   33.508K  Total bytes scanned                         34,312
 ========   ==================================================

        2   Passed ----------------------------------  100.00%
        2     Clean passed                             100.00%
 ========   ==================================================

        2   Ham -------------------------------------  100.00%
        2     Clean passed                             100.00%
 ========   ==================================================


 ---------------------- Amavisd-new End ------------------------- 


 --------------------- httpd Begin ------------------------ 


 A total of 2 sites probed the server 
    213.186.170.226
    5.188.210.12

 Requests with error response codes
    400 Bad Request
       null: 3 Time(s)
       /: 2 Time(s)
http://5.188.210.12/echo.php: 1 Time(s)
       mstshash=Administr: 1 Time(s)
    403 Forbidden
       /: 244 Time(s)
    :
    499 (undefined)
       /index.php?action=dlattach;attach=81;type=avatar: 1 Time(s)

 ---------------------- httpd End ------------------------- 


 --------------------- Postfix Begin ------------------------ 

        2   Miscellaneous warnings                           2

   33.509K  Bytes accepted                              34,313
   34.831K  Bytes delivered                             35,667
   34.540K  Bytes forwarded                             35,369
 ========   ==================================================

        2   Accepted                                   100.00%
 --------   --------------------------------------------------
        2   Total                                      100.00%
 ========   ==================================================

        6   Removed from queue                               6
        2   Delivered                                        2
        2   Forwarded                                        2
      126   Postscreen                                     126


 ---------------------- Postfix End ------------------------- 


 --------------------- SSHD Begin ------------------------ 


 Users logging in through sshd:
    rille:
       192.168.1.99 (LAPTOP): 3 times

 **Unmatched Entries**
 syslogin_perform_logout: logout() returned an error : 3 time(s)

 ---------------------- SSHD End ------------------------- 


 --------------------- Disk Space Begin ------------------------ 

 Filesystem      Size  Used Avail Use% Mounted on
 /dev/root        30G  4.7G   24G  17% /
 /dev/mmcblk0p1   43M   22M   21M  52% /boot


 ---------------------- Disk Space End ------------------------- 


 ###################### Logwatch End ######################### 

[Wraith]

English language questions as captchas have always gotten me the most success. I've never cared about preventing access to bots though, just stopping spam.

edit: Every time bots have gotten through the system, it's been because the bots reposted the question on Yahoo Answers or equivalent and copy + pasted the answer given there. You prevent that by asking questions specifically about your own site/forum, like what's the name of the second forum category on your forum. They still repost the question on Yahoo Answers, but humans can't answer it without knowing where they copy + pasted the question from, so it's safe.

[rille]

Usually does the trick.

I find that the most annoying scanners etc. are from Ukraine/Russia, China, Korea, Germany and U.S.

The question is to whatever allow indexers or not. Can be hard to prevent without require login.

[Wraith]

Most indexers obey the robots.txt file, but they're generally harmless even if they don't.