Differences

This shows you the differences between two versions of the page.

--- system:annoyances [2021-04-25T18:49:34+0000]
michael_paoli
+++ system:annoyances [2021-05-06T06:05:20+0000] (current)
michael_paoli reverted the temporarily increase of max queue time from 4 days to 7 days
@@ Line 385: / Line 385: @@
 Analyzed logs further, notably for web and email traffic/attempts.  Looks like most all that problematic email was from bad web bots repeatedly and voluminously subscribing (well, attempting to subscribe) that, and one other email address, to BALUG's various lists - causing confirmation emails to be queued.  Looks like two such emails got delivered, but all (or almost all?) of the others got deferred by the receiving MTAS (there were only 2 email addresses).  So, perhaps bad bot trying to do DoS/DDoS against those two target emails?  Could potentially block the IP address but ... whack-a-mole - would likely just pop up on another IP.\\
 Checked the mail queue again - after subtracting out target addresses that have already been successfully delivered to, there remain at the moment only 6 unique email addresses presently showing any delivery issues.
+More anti-spam to do ... SPF ... looks like config files can have that enabled ...\\
+<file>
+conf.d/acl/30_exim4-config_check_rcpt
+  # This is quite costly in terms of DNS lookups (~6 lookups per mail).  Do not
+  # enable if that's an issue.  Also note that if you enable this, you must
+  # install "spf-tools-perl" which provides the spfquery command.
+  # Missing spf-tools-perl will trigger the "Unexpected error in
+  # SPF check" warning.
+  .ifdef CHECK_RCPT_SPF
+  deny
+    message = [SPF] $sender_host_address is not allowed to send mail from \
+              ${if def:sender_address_domain {$sender_address_domain}{$sender_helo_name}}.  \
+              Please see \
+              http://www.openspf.org/Why?scope=${if def:sender_address_domain \
+$ dpkg -l spf-tools-perl | grep '^ii '
+ii  spf-tools-perl 2.9.0-4      all          SPF tools (spfquery, spfd) based on the Mail::SPF Perl module
+$ nc -z www.openspf.org. 80
+nc: unable to connect to address www.openspf.org., service 80
+$ nc -z www.openspf.org. 443
+nc: unable to connect to address www.openspf.org., service 443
+$
+So, is spf-tools-perl still applicable, or is it just the diagnostic that's out-of-date referring to a service that's no longer (at least pesently)
+reachable?
+$ dpkg -L spf-tools-perl | sort | grep -e bin/ -e '/man/.*spf'
+/usr/bin/spfquery.mail-spf-perl
+/usr/sbin/spfd.mail-spf-perl
+/usr/share/man/man1/spfquery.mail-spf-perl.1p.gz
+/usr/share/man/man8/spfd.mail-spf-perl.8p.gz
+$ man spfquery
+...
+$ spfquery --scope mfrom --identity balug.org --ip-address $(dig +short balug.org. A)
+pass
+balug.org: 96.86.170.229 is authorized to use 'balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)
+balug.org: 96.86.170.229 is authorized to use 'balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)
+Received-SPF: pass (balug.org: 96.86.170.229 is authorized to use 'balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)) receiver=balug-sf-lug-v2.balug.org; identity=mailfrom; envelope-from=balug.org; client-ip=96.86.170.229
+$ echo $?
+$ spfquery --scope mfrom --identity balug.org --ip-address 8.8.8.8; echo $?
+neutral
+balug.org: Default neutral result due to no mechanism matches
+balug.org: Default neutral result due to no mechanism matches
+Received-SPF: neutral (balug.org: Default neutral result due to no mechanism matches) receiver=balug-sf-lug-v2.balug.org; identity=mailfrom; envelope-from=balug.org; client-ip=8.8.8.8
+$
+neutral ? - are we missing something that ought say that should fail???
+Anyway, looks like spfquery probably works fine, but the web site may be no longer available (DDoS from spammers, or ???).
+$ spfquery --scope mfrom --identity lists.balug.org --ip-address $(dig +short balug.org. A)
+pass
+lists.balug.org: 96.86.170.229 is authorized to use 'lists.balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)
+lists.balug.org: 96.86.170.229 is authorized to use 'lists.balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)
+Received-SPF: pass (lists.balug.org: 96.86.170.229 is authorized to use 'lists.balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)) receiver=balug-sf-lug-v2.balug.org; identity=mailfrom; envelope-from=lists.balug.org; client-ip=96.86.170.229
+$ spfquery --scope mfrom --identity lists.balug.org --ip-address 8.8.8.8
+neutral
+lists.balug.org: Default neutral result due to no mechanism matches
+lists.balug.org: Default neutral result due to no mechanism matches
+Received-SPF: neutral (lists.balug.org: Default neutral result due to no mechanism matches) receiver=balug-sf-lug-v2.balug.org; identity=mailfrom; envelope-from=lists.balug.org; client-ip=8.8.8.8
+$
+Again with the neutral.  Those ought be hard fail.
+... Ah ...:
+balug.org. IN TXT "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2"
+We're missing the -all at the end.
+Should check all our SPF records, and fix as appropriate.
+Should probably also add spf version 2, but first things first ...
+</file>
+<file>
+So ... we have ...:
+balug.org.              600     IN      SPF     "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2"
+balug.org.              600     IN      TXT     "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2"
+tmp.balug.org.          300     IN      TXT     "v=spf1 ip4:96.86.170.228 ip6:2001:470:1f05:19e::f"
+lists.balug.org.        600     IN      SPF     "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2"
+lists.balug.org.        600     IN      TXT     "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2"
+berkeleylug.com.        172800  IN      SPF     "v=spf1 -all"
+berkeleylug.com.        172800  IN      TXT     "v=spf1 -all"
+sf-lug.com.             172800  IN      SPF     "v=spf1 -all"
+sf-lug.com.             172800  IN      TXT     "v=spf1 -all"
+sf-lug.net.             172800  IN      SPF     "v=spf1 -all"
+sf-lug.net.             172800  IN      TXT     "v=spf1 -all"
+sflug.com.              172800  IN      SPF     "v=spf1 -all"
+sflug.com.              172800  IN      TXT     "v=spf1 -all"
+sflug.net.              172800  IN      SPF     "v=spf1 -all"
+sflug.net.              172800  IN      TXT     "v=spf1 -all"
+sflug.org.              86400   IN      SPF     "v=spf1 -all"
+sflug.org.              86400   IN      TXT     "v=spf1 -all"
+We should:
+remove the RRs of type SPF (superseded/obsoleted, per RFC(s))
+add trailing " -all" for those that don't have it
+Our active sending TTLs look rather short, should probably nudge 'em up to ... 3600 or so? ... at least after they're tested out okay.
+</file>
+<file>
+And after updating, we have:
+balug.org.              3600    IN      TXT     "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2 -all"
+lists.balug.org.        3600    IN      TXT     "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2 -all"
+tmp.balug.org.          3600    IN      TXT     "v=spf1 ip4:96.86.170.228 ip6:2001:470:1f05:19e::f -all"
+berkeleylug.com.        172800  IN      TXT     "v=spf1 -all"
+sf-lug.com.             172800  IN      TXT     "v=spf1 -all"
+sf-lug.net.             172800  IN      TXT     "v=spf1 -all"
+sflug.com.              172800  IN      TXT     "v=spf1 -all"
+sflug.net.              172800  IN      TXT     "v=spf1 -all"
+sflug.org.              86400   IN      TXT     "v=spf1 -all"
+So ... that now looks better.
+And let's do a little retest on our earlier:
+$ spfquery --scope mfrom --identity balug.org --ip-address $(dig +short balug.org. A); echo "$?"
+pass
+balug.org: 96.86.170.229 is authorized to use 'balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)
+balug.org: 96.86.170.229 is authorized to use 'balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)
+Received-SPF: pass (balug.org: 96.86.170.229 is authorized to use 'balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)) receiver=balug-sf-lug-v2.balug.org; identity=mailfrom; envelope-from=balug.org; client-ip=96.86.170.229
+$ spfquery --scope mfrom --identity lists.balug.org --ip-address $(dig +short balug.org. A); echo "$?"
+pass
+lists.balug.org: 96.86.170.229 is authorized to use 'lists.balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)
+lists.balug.org: 96.86.170.229 is authorized to use 'lists.balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)
+Received-SPF: pass (lists.balug.org: 96.86.170.229 is authorized to use 'lists.balug.org' in 'mfrom' identity (mechanism 'ip4:96.86.170.229' matched)) receiver=balug-sf-lug-v2.balug.org; identity=mailfrom; envelope-from=lists.balug.org; client-ip=96.86.170.229
+$ spfquery --scope mfrom --identity balug.org --ip-address 8.8.8.8; echo "$?"
+fail
+Please see http://www.openspf.org/Why?s=mfrom;id=balug.org;ip=8.8.8.8;r=balug-sf-lug-v2.balug.org
+balug.org: Sender is not authorized by default to use 'balug.org' in 'mfrom' identity (mechanism '-all' matched)
+Received-SPF: fail (balug.org: Sender is not authorized by default to use 'balug.org' in 'mfrom' identity (mechanism '-all' matched)) receiver=balug-sf-lug-v2.balug.org; identity=mailfrom; envelope-from=balug.org; client-ip=8.8.8.8
+$ spfquery --scope mfrom --identity lists.balug.org --ip-address 8.8.8.8; echo "$?"
+fail
+Please see http://www.openspf.org/Why?s=mfrom;id=lists.balug.org;ip=8.8.8.8;r=balug-sf-lug-v2.balug.org
+lists.balug.org: Sender is not authorized by default to use 'lists.balug.org' in 'mfrom' identity (mechanism '-all' matched)
+Received-SPF: fail (lists.balug.org: Sender is not authorized by default to use 'lists.balug.org' in 'mfrom' identity (mechanism '-all' matched)) receiver=balug-sf-lug-v2.balug.org; identity=mailfrom; envelope-from=lists.balug.org; client-ip=8.8.8.8
+$
+So, that looks much better now.
+</file>
+<file>
+wordpress also sends mail:
+From www-data@balug.org Tue Apr 27 02:12:48 2021
+From: WordPress <wordpress@berkeleylug.com>
+So, @berkeleylug.com needs to be set up to send - and at least minimally receive, email (e.g. postmaster ...)
+So, ... SPF first, as that has the longer TTL presently ...
+from:
+berkeleylug.com.        172800  IN      TXT     "v=spf1 -all"
+to:
+berkeleylug.com.        3600    IN      TXT     "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2 -all"
+And, added bit more for digitalwitness.org. and sf-lug.org. (latter of which thus far still uses @linuxmafia.com for mail), now have:
+balug.org.              3600    IN      TXT     "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2 -all"
+lists.balug.org.        3600    IN      TXT     "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2 -all"
+tmp.balug.org.          3600    IN      TXT     "v=spf1 ip4:96.86.170.228 ip6:2001:470:1f05:19e::f -all"
+berkeleylug.com.        3600    IN      TXT     "v=spf1 ip4:96.86.170.229 ip6:2001:470:1f05:19e::2 -all"
+digitalwitness.org.     86400   IN      TXT     "v=spf1 -all"
+sf-lug.com.             172800  IN      TXT     "v=spf1 -all"
+sf-lug.net.             172800  IN      TXT     "v=spf1 -all"
+sf-lug.org.             86400   IN      TXT     "v=spf1 -all"
+sflug.com.              172800  IN      TXT     "v=spf1 -all"
+sflug.net.              172800  IN      TXT     "v=spf1 -all"
+sflug.org.              86400   IN      TXT     "v=spf1 -all"
+SPF version 2 could be good/better ... but later, not a top priority.
+</file>
+<file>
+So, let's look into enabling SPF checking upon receipt of incoming ...
+I also noticed what looks like something about a daemon - which may be preferable for large volumes/streams of incoming ...
+let's look at documentation bit more ...
+$ man spfd.mail-spf-perl
+$ systemctl list-unit-files | fgrep spf
+$
+So, nothin' in systemd unit files nor exim4 config that supports the spf daemon, so doing that would mean fair bit more manual configuring.
+For now let's presume spfquery (non-daemonized) is quite "good enough" for now - we can change later if we need to.
+So ... let's configure that ...
+added ...:
+# tail -n 1 conf.d/main/000_localmacros
+CHECK_RCPT_SPF = true
+# systemctl restart exim4.service
+# That should be enough for that to now be operational - that should stop >> 50% of the incoming spam (attempts).  Should see results in logs
+quite soon (if not already).
+</file>
+<file>
+Not seeing an SPF failure in the logs ... quite yet.
+Let's test something that should fail ...
+Drats - test made it through, even though the config should'a rejected it.
+Oh, let's also add berkeleylug.com to the email domains, so that should work.
+# DEBIAN_PRIORITY=medium dpkg-reconfigure exim4-config
+# systemctl start exim4.service
+Let's try sending to postmaster@berkeleylug.com
+and yes, that got delivered fine.
+So ... why is SPF check not working?
+</file>
+<file>
+# systemctl stop exim4.service
+# ls -d /usr/*bin/*exim*conf*
+/usr/sbin/update-exim4.conf  /usr/sbin/update-exim4.conf.template
+# update-exim4.conf
+# systemctl start exim4.service
+SPF check still not working.
+</file>
+<file>
+Wordpress email ... something to circle back on later.
+For now, for header it uses:
+From: WordPress <wordpress@berkeleylug.com>
+Looks like the only bit of that that's easy to change is the domain.  Looks like it uses php mail.  There are plugins to change that, but
+that's then more complications.  As for envelope, since it's using Apache, between that and exim, that ends up as:
+MAIL FROM:<www-data@balug.org>
+Again, not simple to change that.  More to circle back on for later.
+For now, dropped in aliases for www-data and wordpress, so at least attempts to those - and for now at least, won't bounce at those domains if
+attempted.  So, that should help deliverability (and, on the receiving side, probably some more spam for postmaster as I presently aliased those to
+postmaster ... "good enough" for now).
+</file>
+<file>
+Looks like the SPF checks are now working.
+I also found an older spdf process running and killed that off - maybe that made the difference?
+So, yes, and seeing SPF fail/rejects in the log e.g.:
+# fgrep -ai spf rejectlog
+-04-28 02:29:33 H=(sweja-se.mail.protection.outlook.com) [183.199.220.44] F=<oefydgodea@ottawa.ca> rejected RCPT <rsvp@balug.org>: SPF check failed.
+-04-28 03:50:56 H=(smail1.vub.sk) [222.77.253.120] F=<jhylunrrhc@swebolt.se> rejected RCPT <rsvp@balug.org>: SPF check failed.
+# dig +noall +answer +nottl ottawa.ca. TXT ottawa.ca. SPF swebolt.se. TXT swebolt.se. SPF | fgrep \"v=spf
+ottawa.ca.              IN      TXT     "v=spf1 include:spf.protection.outlook.com include:_spf.esolutionsgroup.ca include:emsd1.com -all"
+swebolt.se.             IN      TXT     "v=spf1 mx ip4:167.99.44.246 include:spf.protection.outlook.com a:smtp05.dgcsystems.net -all"
+# spfquery --scope mfrom --id oefydgodea@ottawa.ca --ip 183.199.220.44; echo "$?"
+fail
+Please see http://www.openspf.org/Why?s=mfrom;id=oefydgodea%40ottawa.ca;ip=183.199.220.44;r=balug-sf-lug-v2.balug.org
+ottawa.ca: Sender is not authorized by default to use 'oefydgodea@ottawa.ca' in 'mfrom' identity (mechanism '-all' matched)
+Received-SPF: fail (ottawa.ca: Sender is not authorized by default to use 'oefydgodea@ottawa.ca' in 'mfrom' identity (mechanism '-all' matched)) receiver=balug-sf-lug-v2.balug.org; identity=mailfrom; envelope-from="oefydgodea@ottawa.ca"; client-ip=183.199.220.44
+# spfquery --scope mfrom --id jhylunrrhc@swebolt.se --ip 222.77.253.120; echo "$?"
+fail
+Please see http://www.openspf.org/Why?s=mfrom;id=jhylunrrhc%40swebolt.se;ip=222.77.253.120;r=balug-sf-lug-v2.balug.org
+swebolt.se: Sender is not authorized by default to use 'jhylunrrhc@swebolt.se' in 'mfrom' identity (mechanism '-all' matched)
+Received-SPF: fail (swebolt.se: Sender is not authorized by default to use 'jhylunrrhc@swebolt.se' in 'mfrom' identity (mechanism '-all' matched)) receiver=balug-sf-lug-v2.balug.org; identity=mailfrom; envelope-from="jhylunrrhc@swebolt.se"; client-ip=222.77.253.120
+#
+</file>
+<file>
+Wrote a handy little program to summarize the exim rejectlog failure from the most recent few such log files:
+# Rejectlog_report
+Unrouteable address
+relay not permitted
+SPF check failed
+SMTP protocol synchronization error (input sent without waiting for greeting)
+maximum allowed line length
+unqualified address not permitted
+SMTP protocol synchronization error (next input sent too soon: pipelining was not advertised)
+missing or malformed local part
+syntactically invalid
+#
+Look at least the top couple items would be good candidates for adding configurations for fail2ban.
+Some others beyond that may also be worth doing - but not as high a priority.
+</file>
+<file>
+// reverted the temporarily increase of max queue time from 4 days to 7 days:
+# awk '{if($1~/^[^#]/||$1~/^#\*/||$0~/^# temp/)print;}' conf.d/retry/30_exim4-config
+*                      *           F,2h,15m; G,16h,1h,1.5; F,4d,6h
+# systemctl reload exim4.service
+#
+</file>

BALUG Wiki

User Tools

Site Tools

Differences

Page Tools