This is an old revision of the document!
Wiki page to cover BALUG's email and lists
Not an entire history, but most releant as of the time of this writing, BALUG's email and lists are still hosted on DreamHost.Com. For numerous reasons we wish to
Where those two objectives may conflict, in general the former will take precedence over the latter.
Lists - lists are presently on lists.balug.org. There is also some mail on balug.org. High-level of migration strategy off of DreamHost.Com (to be implemented as feasible, but we do need to get off of DreamHost.Com):
Note also that presently much of BALUG's content is already self-hosted - e.g. [www.]balug.org is mostly just rsynced from "master" that's self-hosted; primary traffic can be put to that master with some DNS changes - when that and anything else that needs/ought be migrated/pulled from DreamHost.Com is done, then it's safe to pull/cut the cord / flip the switch, and move off of DreamHost.Com.
(Work-in-progress), and may be mostly documented as we go along the way)
checked lists, BALUG-Talk and BALUG-Admin lists:
obscure_addresses (privacy): Show member addresses so they're not directly recognizable as email addresses? Setting this option causes member email addresses to be transformed when they are presented on list web pages (both in text and as links), so they're not trivially recognizable as email addresses. The intention is to prevent the addresses from being snarfed up by automated web scanners for use by spammers.
changed from Yes to No (BALUG-Announce was already set to No).
From walking the admin menus, the above seems the only setting of relevance to unobfuscate email addresses, but since they're obfuscated on the BALUG-Announce list, where that settings isn't and wasn't set, and where the email addresses in the archive are obfuscated anyway, seems we don't have a setting we can access to turn off that behavior.
# (umask 022 && DEBIAN_PRIORITY=critical DEBIAN_FRONTEND=noninteractive apt-get -y --purge install exim4-daemon-heavy exim4-doc-\* sendmail\*-)
$ ss -nlt '( sport = :25 or sport = :587 )' State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 10 127.0.0.1:25 *:* LISTEN 0 10 127.0.0.1:587 *:* $
$ ss -nlt '( sport = :25 or sport = :587 )' State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 20 127.0.0.1:25 *:* LISTEN 0 20 ::1:25 :::* $
238.194.144.198.in-addr.arpa. 10800 IN CNAME 238.net232.194.144.198.in-addr.arpa. 238.net232.194.144.198.in-addr.arpa. 14400 IN PTR temp.balug.org.
mx.temp.balug.org. 14400 IN AAAA 2001:470:1f04:19e::2
email/List stuff to (re)test - results (Y - good, N - failed, ? - to be tested)
o Y? (partially fixed?) greylisting: shouldn't cause excessive delays o Y? (partially fixed?) greylisting: shouldn't cause false permanent blocks o Y flood protection: enable and reasonably reject spam floods o Y? (partially fixed?) flood protection: when enabled, shouldn't cause excessive delays o Y? (partially fixed?) flood protection: when enabled, shouldn't cause false permanent blocks o N don't reject empty body (e.g. legitimate mailman command in subject with empty body) o Y If rejecting empty body, implement work-around (suitable updates to texts, etc.) o Y SMTP TLS - should use STARTTLS on sending when available o Y (fixed) SMTP TLS - should offer working STARTTLS on receiving with CA signed cert for applicable domain(s) o Y SMTP TLS - set up separate cert for MTA to have read access to private key with just {temp.,}balug.org (and later to add lists.balug.org, and eventually drop temp.balug.org) o N outbound IPv6 SMTP to TCP port 25 should be open and operational (blocked by the (IPv6) tunnel provider by default, (probably) won't be able to get this opened until after balug.org is fully migrated off of DreamHost.com hosting) o Y (implemented, need to verify) if outbound IPv6 SMTP to TCP port 25 is not open, apply workaround: changed config line in /etc/exim4/eximconfig/config/ignore_target_hosts to: <; 127.0.0.1/8 ; 0.0.0.0/32 ; 192.168.0.0/16 ; 172.16.0.0/12 ; 10.0.0.0/8 ; 2000::/3 o Y relevant list user URLs should generally work: info/subscribe/unsubscribe/archive (need more complete list) o Y relevant list admin URLs should generally work: per-list and overall admin, roster, etc. (need more complete list) o Y http[s]://temp.balug.org/[whatever] should not redirect to other domains (except in future to lists.balug.org) o Y http[s]://temp.balug.org/[mailman-prefix]/[whatever] should only give out https o Y http[s]://temp.balug.org/[mailman-prefix]/[whatever] should only use https: http should redirect to https o Y http[s]://temp.balug.org/{,cgi-bin{,/{,mailman{,/}}}} redirect to https://temp.balug.org/cgi-bin/mailman/listinfo o Y (fixed) http[s]://temp.balug.org/robots.txt Disallow: / # will later update more appropriate under lists.balug.org domain o Y full mbox archive publicly available; procedure: add: PUBLIC_MBOX = Yes to file: /etc/mailman/mm_cfg.py restart mailman for existing lists, toggling archive from public to private and back again seems sufficient to then create the needed link o N full mbox archive should be publicly available via public rsync o ? should be able to completely reload archive, add/drop messages from archive, etc. & document procedure thereof o Y (working or mostly working?) mailman commands should work via email: subscribe/unsubscribe/help (need more complete list) o ? (untested) mailman admin commands should work via email (need more complete list) o (partially staged) should accept legitimate email for legitimate @balug.org addresses o N (future) default sending domain of host: @balug.org o N (future) add appropriate SPF records for @lists.balug.org, @balug.org o (http://lists.balug.org/ staged, remainder pending DNS & SSL cert, etc.) lists should use URLs starting with: https://lists.balug.org/ o (MTA partially staged, remainder pending DNS, SSL certs, reconfigurations) lists should use email addresses ending with: @lists.balug.org o (requested 2017-08-19, Ticket #7876434) get raw mbox of archives from DreamHost.Com if feasible o (emailed requeste of primary account holder 2017-08-20, reminder sent 2017-08-24 & 2017-08-30; I called and left voicemail message 2017-09-06; 2017-09-13: called and left voicemail again, sent email again, also sent cellular text message; 2017-09-14 called and left voicemail again and sent email again also gave additional option to have primary user transfer DreamHost primary user and billing to Michael Paoli) above requires DreamHost support ticket opened requesting such from primary account holder o (pending) DreamHost primary account holder to open support ticket with DreamHost.com to get raw archives in mbox format. o N (future) http[s]://temp.balug.org/robots.txt - set up appropriately o (staged) http[s]://lists.balug.org/robots.txt - set up appropriately o (staged) http[s]://lists.balug.org/[mailman-prefix]/ should only use https o (staged) http[s]://lists.balug.org/{,cgi-bin{,/{,mailman{,/}}}} should redirect to https://lists.balug.org/cgi-bin/mailman/listinfo o N (future) all of http[s]://temp.balug.org/ should 301 redirect to corresponding https://lists.balug.org/ URLs o (staged) legacy http://lists.balug.org URLs should 301 redirect to new locations (where different) o Y http[s]://{temp,www}.balug.org/lists/balug-announce-do-not-auto-add.html update to use temp.balug.org o Y http[s]://{temp,www}.balug.org/lists/balug-announce-do-not-auto-add.html update to exclude BALUG-Test list o N (future) http[s]://{lists,temp,www}.balug.org/lists/balug-announce-do-not-auto-add.html update to use lists.balug.org o N (future) http[s]://{lists,temp,www}.balug.org/lists/balug-announce-do-not-auto-add.html canonicalize to https://lists.balug.org/lists/balug-announce-do-not-auto-add.html o ? (future) decommission temp.balug.org domain?
per member configuration (options) and transfer thereof (as feasible)
Mailman stores the data in a Python pickle (.pck) format per-list file.
Have written program that scrapes the web pages, and gathers almost all that data of relevance.
That essentially gives us everything except: passwords
So … that's probably darn good enough overall.
Format - as various /usr/lib/mailman/bin/dumpdb shows us for output (dump) we have essentially this, and likely need to have massage the data to quite similar form to load(/edit) it into Python pickle data to adjust user configurations. For most of the options, they're in a binary format (yes/no or on/off, etc.), and Python use a power of 2 weighting of each and combines them into a single decimal number. Here's the basic high-level of that data and the binary weightings and also the format of how strings are stored/represented (likely Phython convention):
'members': { 'canonical_lowercase_email': 'case_preserved_email', 'canonical_lowercase_email': 0 (0 when user entered without uppercase), ... } 'usernames': { 'canonical_lowercase_email': u'Your name (optional)' (empty string if not set), ... } disablemail: delivery_status (2, epoch_timestamp_to_6_points_after_decimal) digest: digest_members vs. members mime 8 +8 Plain dontreceive 2 +2 receive own: No ackposts 4 +4 receive ack: Yes remind 32 +32 reminder: No conceal 16 +16 Conceal: Yes rcvtopic 64 +64 receive messages that do not match: Yes nodupes 256 +256 avoid duplicates: Yes strings: ' quoted \' for ' \\ for \
Fix for: Show case preserved emails in the roster (to be considered and possibly tested/applied).
Added rewrite rules to remap old URLs to new - this will be useful most notably once we're hosting lists.balug.org away from DreamHost.com:
RewriteRule "^/*listinfo\.cgi/*$" https://%0/cgi-bin/mailman/listinfo [L,R=permanent] RewriteRule "^/*listinfo\.cgi/+(balug-(?:announce|talk|admin))-balug\.org$" "https://%0/cgi-bin/mailman/listinfo/$1" [L,R=permanent] RewriteRule "^/*pipermail/(balug-(?:announce|talk|admin))-balug\.org/(.*)$" "https://%0/pipermail/$1/$2" [L,R=permanent] RewriteRule "^/*admin\.cgi/(balug-(?:announce|talk|admin))-balug\.org$" "https://%0/cgi-bin/mailman/admin/$1" [L,R=permanent] RewriteRule "^/*roster\.cgi/(balug-(?:announce|talk|admin))-balug\.org$" "https://%0/cgi-bin/mailman/roster/$1" [L,R=permanent]
Added constraints to deny mailman cgi except for the correct vhost(s) using HTTPS:
<Directory "/usr/lib/cgi-bin/mailman/"> Require expr %{SERVER_NAME} == "temp.balug.org" && %{HTTPS} == "on" #Require expr %{SERVER_NAME} in { "lists.balug.org", "temp.balug.org" } && %{HTTPS} == "on" #Require expr %{SERVER_NAME} == "lists.balug.org" && %{HTTPS} == "on" </Directory>
added mailman-loop alias - this may not be optimal handling, but the alias needs to exist (needs to always be deliverable), and is probably at least "good enough" for now:
mailman-loop: postmaster