Fail2Ban v0.11.1 Release Notes

Release Date: 2020-01-11 // 10 days ago
  • Compatibility:

    • to v.0.10:
      • 0.11 is totally compatible to 0.10 (configuration- and API-related stuff), but the database got some new tables and fields (auto-converted during the first start), so once updated to 0.11, you have to remove the database /var/lib/fail2ban/fail2ban.sqlite3 (or its different to 0.10 schema) if you would need to downgrade to 0.10 for some reason.
    • to v.0.9:

      • Filter (or failregex) internal capture-groups:
      • If you've your own failregex or custom filters using conditional match (?P=host), you should rewrite the regex like in example below resp. using (?:(?P=ip4)|(?P=ip6) instead of (?P=host) (or (?:(?P=ip4)|(?P=ip6)|(?P=dns)) corresponding your usedns and raw settings).

      Of course you can always define your own capture-group (like below _cond_ip_) to do this.

        testln="1500000000 failure from 192.0.2.1: bad host 192.0.2.1"
        fail2ban-regex "$testln" "^\s*failure from (?P<_cond_ip_><HOST>): bad host (?P=_cond_ip_)$"
      
      • New internal groups (currently reserved for internal usage): ip4, ip6, dns, fid, fport, additionally user and another captures in lower case if mapping from tag <F-*> used in failregex (e. g. user by <F-USER>).
      • v.0.10 and 0.11 use more precise date template handling, that can be theoretically incompatible to some user configurations resp. datepattern.
      • Since v0.10 fail2ban supports the matching of IPv6 addresses, but not all ban actions are IPv6-capable now.

    🛠 Fixes

    • purge database will be executed now (within observer).
    • 🛠 restoring currently banned ip after service restart fixed (now < timeofban + bantime), ignore old log failures (already banned)
    • ⚡️ upgrade database: update new created table bips with entries from table bans (allows restore current bans after upgrade from version <= 0.10)

    🆕 New Features

    • Increment ban time (+ observer) functionality introduced.
    • Database functionality extended with bad ips.
    • 🆕 New tags (usable in actions):
      • <bancount> - ban count of this offender if known as bad (started by 1 for unknown)
      • <bantime> - current ban-time of the ticket (prolongation can be retarded up to 10 sec.)
    • ⏱ Introduced new action command actionprolong to prolong ban-time (e. g. set new timeout if expected); Several actions (like ipset, etc.) rewritten using net logic with actionprolong. Note: because ban-time is dynamic, it was removed from jail.conf as timeout argument (check jail.local).

    ✨ Enhancements

    • ⚡️ algorithm of restore current bans after restart changed: update the restored ban-time (and therefore end of ban) of the ticket with ban-time of jail (as maximum), for all tickets with ban-time greater (or persistent); not affected if ban-time of the jail is unchanged between stop/start.
    • ➕ added new setup-option --without-tests to skip building and installing of tests files (gh-2287).
    • ➕ added new command fail2ban-client get <JAIL> banip ?sep-char|--with-time? to get the banned ip addresses (gh-1916).

Previous changes from v0.10.5

  • Yes, Hrrrm...

    🛠 Fixes

    • 0️⃣ [compatibility] systemd backend: default flags changed to SYSTEM_ONLY(4), fixed in gh-2444 in order to ignore user session files per default, so could prevent "Too many open files" errors on a lot of user sessions (see gh-2392)
    • 📜 [grave] fixed parsing of multi-line filters (maxlines > 1) together with systemd backend, now systemd-filter replaces newlines in message from systemd journal with \n (otherwise multi-line parsing may be broken, because removal of matched string from multi-line buffer window is confused by such extra new-lines, so they are retained and got matched on every followed message, see gh-2431)
    • [stability] prevent race condition - no unban if the bans occur continuously (gh-2410); now an unban-check will happen not later than 10 tickets get banned regardless there are still active bans available (precedence of ban over unban-check is 10 now)
    • 🛠 fixed read of included config-files (.local overwrites options of .conf for config-files included with before/after)
    • action.d/abuseipdb.conf: switched to use AbuseIPDB API v2 (gh-2302)
    • 🛠 action.d/badips.py: fixed start of banaction on demand (which may be IP-family related), gh-2390
    • action.d/helpers-common.conf: rewritten grep arguments, now options -wF used to match only whole words and fixed string (not as pattern), gh-2298
    • filter.d/apache-auth.conf:
      • ignore errors from mod_evasive in normal mode (mode-controlled now) (gh-2548);
      • extended with option mode - normal (default) and aggressive
    • filter.d/sshd.conf:
      • matches Bad protocol version identification in ddos and aggressive modes (gh-2404).
      • captures Disconnecting ...: Change of username or service not allowed (gh-2239, gh-2279)
      • captures Disconnected from ... [preauth], preauth phase only, different handling by extra (with supplied user only) and ddos/aggressive mode (gh-2115, gh-2239, gh-2279)
    • filter.d/mysqld-auth.conf:
      • MYSQL 8.0.13 compatibility (log-error-verbosity = 3), log-format contains few additional words enclosed in brackets after "[Note]" (gh-2314)
    • filter.d/sendmail-reject.conf:
      • mode=extra now captures port IDs of TLSMTA and MSA (defaults for ports 465 and 587 on some distros)
    • 🛠 files/fail2ban.service.in: fixed systemd-unit template - missing nftables dependency (gh-2313)
    • 🛠 several action.d/mail*: fixed usage with multiple log files (ultimate fix for gh-976, gh-2341)
    • 🛠 filter.d/sendmail-reject.conf: fixed journal usage for some systems (e. g. CentOS): if only identifier set to sm-mta (no unit sendmail) for some messages (gh-2385)
    • 🔊 filter.d/asterisk.conf: asterisk can log additional timestamp if logs into systemd-journal (regex extended with optional part matching this, gh-2383)
    • filter.d/postfix.conf:
      • regexp's accept variable suffix code in status of postfix for precise messages (gh-2442)
      • extended with new postfix filter mode errors to match "too many errors" (gh-2439), also included within modes normal, more (extra and aggressive), since postfix parameter smtpd_hard_error_limit is default 20 (additionally consider maxretry)
    • filter.d/named-refused.conf:
      • support BIND 9.11.0 log format (includes an additional field @0xXXX..., gh-2406);
      • prefregex extended, more selective now (denied/NOTAUTH suffix moved from failregex, so no catch-all there anymore)
    • filter.d/sendmail-auth.conf, filter.d/sendmail-reject.conf :
      • ID in prefix can be longer as 14 characters (gh-2563);
    • all filters would accept square brackets around IPv4 addresses also (e. g. monit-filter, gh-2494)
    • 👻 avoids unhandled exception during flush (gh-2588)
    • 🛠 fixes pass2allow-ftp jail - due to inverted handling, action should prohibit access per default for any IP, therefore reset start on demand parameter for this action (it will be started immediately by repair);
    • auto-detection of IPv6 subsystem availability (important for not on-demand actions or jails, like pass2allow);

    🆕 New Features

    • 🆕 new replacement tags for failregex to match subnets in form of IP-addresses with CIDR mask (gh-2559):
      • <CIDR> - helper regex to match CIDR (simple integer form of net-mask);
      • <SUBNET> - regex to match sub-net adresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional);
    • grouped tags (<ADDR>, <HOST>, <SUBNET>) recognize IP addresses enclosed in square brackets
    • 🆕 new failregex-flag tag <F-MLFGAINED> for failregex, signaled that the access to service was gained (ATM used similar to tag <F-NOFAIL>, but it does not add the log-line to matches, gh-2279)
    • 🔧 filters: introduced new configuration parameter logtype (default file for file-backends, and journal for journal-backends, gh-2387); can be also set to rfc5424 to force filters (which include common.conf) to use RFC 5424 conform prefix-line per default (gh-2467);
    • 🐎 for better performance and safety the option logtype can be also used to select short prefix-line for file-backends too for all filters using __prefix_line (common.conf), if message logged only with hostname svc[nnnn] prefix (often the case on several systems): ini [jail] backend = auto filter = flt[logtype=short]
    • filter.d/common.conf: differentiate __prefix_line for file/journal logtype's (speedup and fix parsing of systemd-journal);
    • filter.d/traefik-auth.conf: used to ban hosts, that were failed through traefik
    • filter.d/znc-adminlog.conf: new filter for ZNC (IRC bouncer); requires the adminlog module to be loaded

    ✨ Enhancements

    • introduced new options: dbmaxmatches (fail2ban.conf) and maxmatches (jail.conf) to contol how many matches per ticket fail2ban can hold in memory and store in database (gh-2402, gh-2118);
    • 🔧 fail2ban.conf: introduced new section [Thread] and option stacksize to configure default size of the stack for threads running in fail2ban (gh-2356), it could be set in fail2ban.local to avoid runtime error "can't start new thread" (see gh-969);
    • 👍 jail-reader extended (amend to gh-1622): actions support multi-line options now (interpolations containing new-line);
    • 👀 fail2ban-client: extended to ban/unban multiple tickets (see gh-2351, gh-2349); Syntax:
      • fail2ban-client set <jain> banip <ip1> ... <ipN>
      • fail2ban-client set <jain> unbanip [--report-absent] <ip1> ... <ipN>
    • fail2ban-client: extended with new feature which allows to inform fail2ban about single or multiple attempts (failure) for IP (resp. failure-ID), see gh-2351; Syntax:
      • fail2ban-client set <jail> attempt <ip> [<failure-message1> ... <failure-messageN>]
    • action.d/nftables.conf:
      • isolate fail2ban rules into a dedicated table and chain (gh-2254)
      • nftables-allports supports multiple protocols in single rule now
      • combined nftables actions to single action nftables:
      • nftables-common is removed (replaced with single action nftables now)
      • nftables-allports is obsolete, superseded by nftables[type=allports]
      • nftables-multiport is obsolete, superseded by nftables[type=multiport]
      • allowed multiple protocols in nftables[type=multiport] action (single set with multiple rules in chain), following configuration in jail would replace 3 separate actions, see https://github.com/fail2ban/fail2ban/pull/2254#issuecomment-534684675
    • action.d/badips.py: option loglevel extended with level of summary message, following example configuration logging summary with NOTICE and rest with DEBUG log-levels: action = badips.py[loglevel="debug, notice"]
    • ✅ samplestestcase.py (testSampleRegexsFactory) extended:
      • allow coverage of journal logtype;
      • new option fileOptions to set common filter/test options for whole test-file;
    • large enhancement: auto-reban, improved invariant check and conditional operations (gh-2588):
      • improves invariant check and repair (avoid unhandled exception, consider family on conditional operations, etc), prepared for bulk re-ban in repair case (if bulk-ban becomes implemented);
      • automatic reban (repeat banning action) after repair/restore sane environment, if already logged ticket causes new failures (via new action operation actionreban or actionban if still not defined in action);
      • introduces banning epoch for actions and tickets (to distinguish or recognize removed set of the tickets);
      • invariant check avoids repair by unban/stop (unless parameter actionrepair_on_unban set to true);
      • better handling for all conditional operations (distinguish families for certain operations like repair/flush/stop, prepared for other families, e. g. if different handling for subnets expected, etc);
      • partially implements gh-980 (more breakdown safe handling);
      • closes gh-1680 (better as large-scale banning implementation with on-demand reban by failure, at least unless a bulk-ban gets implemented);
    • 🛠 fail2ban-regex - several enhancements and fixes:
      • improved usage output (don't put a long help if an error occurs);
      • new option --no-check-all to avoid check of all regex's (first matched only);
      • new option -o, --out to set token only provided in output (disables check-all and outputs only expected data).