User Tools

Site Tools


balug:mail_and_lists

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
balug:mail_and_lists [2017-09-21T03:45:50+0000]
michael_paoli various status updates, and other minor tweaks
balug:mail_and_lists [2018-05-22T22:10:41+0000] (current)
michael_paoli
Line 13: Line 13:
 [Migration was fully completed off of DreamHost.com on 2017-09-18] [Migration was fully completed off of DreamHost.com on 2017-09-18]
 High-level of migration strategy off of DreamHost.Com (to be implemented as feasible, but we do need to get off of DreamHost.Com):​ High-level of migration strategy off of DreamHost.Com (to be implemented as feasible, but we do need to get off of DreamHost.Com):​
-  * create temp.balug.org (done - implemented ​for now as fully delegated sub-domain) +  * (done) (temporarily) ​create temp.balug.org (implemented ​initially ​as fully delegated sub-domain) 
-  * create test list on temp.balug.org +  * (done) ​create test list (initally ​on temp.balug.org) 
-  * migrate lists - from lowest traffic/​subscribers first, to highest, from lists.balug.org to temp.balug.org +  * (done) (initially) ​migrate lists - from lowest traffic/​subscribers first, to highest, from lists.balug.org to temp.balug.org 
-  * fully prepare email infrastructure on @balug.org (e.g. for various aliases there) for migration (and test as feasible, etc.) +  * (done) ​fully prepare email infrastructure on @balug.org (e.g. for various aliases there) for migration (and test as feasible, etc.) 
-  * copy/​migrate any remnant bits from DreamHost.Com if/as feasible +  * (done - all relevant data copied) ​copy/​migrate any remnant bits from DreamHost.Com if/as feasible 
-  * pull the plug on DreamHost.Com (remove DNS delegation from it), move forward to cancel any relevant DreamHost.Com account(s) or services thereof +  * (done) ​pull the plug on DreamHost.Com (remove DNS delegation from it), move forward to cancel any relevant DreamHost.Com account(s) or services thereof ​(relevant notification(s) sent) 
-  * temp.balug.org is intended to be //​temporary//​ - transitional - it will be migrated back to lists.balug.org (once we're done with DreamHost.Com and we can make that self-hosted),​ then we'll slowly phase out temp.balug.org (for some fair while we'll allow both to work, but lists.balug.org will be the canonical).+  * (done) ​temp.balug.org is intended to be //​temporary//​ - transitional - it will be migrated back to lists.balug.org (once we're done with DreamHost.Com and we can make that self-hosted),​ then we'll slowly phase out temp.balug.org (for some fair while we'll allow both to work, but lists.balug.org will be the canonical).
  
-Note also that presently ​much of BALUG'​s content is already self-hosted - e.g. [www.]balug.org is mostly just rsynced from "​master"​ that's self-hosted;​ primary traffic can be put to that master with some DNS changes - when that and anything else that needs/ought be migrated/​pulled from DreamHost.Com is done, then it's safe to pull/cut the cord / flip the switch, and move off of DreamHost.Com.+Note also that much of BALUG'​s content ​(was) is already self-hosted ​(now is) - e.g. [www.]balug.org ​(was) is mostly just rsynced from "​master"​ that's self-hosted;​ primary traffic can be put to that master with some DNS changes ​(done) ​- when that and anything else that needs/ought be migrated/​pulled from DreamHost.Com is done, then it's safe to pull/cut the cord / flip the switch, and move off of DreamHost.Com ​(done).
 ===== List migration step-by-step ===== ===== List migration step-by-step =====
 (Work-in-progress,​ and may be mostly documented as we go along the way) (Work-in-progress,​ and may be mostly documented as we go along the way)
-  * temp.balug.org [created] +  * temp.balug.org [created, moved to, moved from and deprecated, phased out, done
-  * temp.balug.org - any DNS records we need to create/add there? - notably not only what we need but implement as soon as feasible to avoid any DNS negative caching TTL issues [checked, done, more than sufficient] +  * temp.balug.org - any DNS records we need to create/add there? - notably not only what we need but implement as soon as feasible to avoid any DNS negative caching TTL issues [checked, done, more than sufficient ​... phased out ... done
-  * archives - will want to get raw mbox of archives from DreamHost.Com if feasible, next best get rawest forms we can manage to get of archives +  * archives - will want to get raw mbox of archives from DreamHost.Com if feasible, next best get rawest forms we can manage to get of archives ​[done] 
-  * disable any email obfuscation of archives [checked and adjusted as feasible] - that may/will slightly aid quality of archive bits we can get - especially if we're not able to get raw mbox format.\\+  * disable any email obfuscation of archives [checked and adjusted as feasible] - that may/will slightly aid quality of archive bits we can get - especially if/​where ​we're not able to get raw mbox format ​and/or complete history thereof.\\
 checked lists, BALUG-Talk and BALUG-Admin lists:\\ checked lists, BALUG-Talk and BALUG-Admin lists:\\
 <​file>​ <​file>​
-obscure_addresses (privacy): Show member addresses so they'​re not directly recognizable as email addresses?+obscure_addresses (privacy): Show member addresses so they'​re not directly recognizable as email addresses? ​[done - at least as feasible]
  
 Setting this option causes member email addresses to be transformed when they are presented on list web pages (both in text and as links), so they'​re not trivially recognizable as email addresses. The intention is to prevent the addresses from being snarfed up by automated web scanners for use by spammers. Setting this option causes member email addresses to be transformed when they are presented on list web pages (both in text and as links), so they'​re not trivially recognizable as email addresses. The intention is to prevent the addresses from being snarfed up by automated web scanners for use by spammers.
 </​file>​\\ </​file>​\\
 changed from Yes to No (BALUG-Announce was already set to No).\\ changed from Yes to No (BALUG-Announce was already set to No).\\
-From walking the admin menus, the above seems the only setting of relevance to unobfuscate email addresses, but since they'​re obfuscated on the BALUG-Announce list, where that settings ​isn't and wasn't set, and where the email addresses in the archive are obfuscated anyway, seems we didn't (on DreamHost) have a setting we could access to turn off that behavior. +From walking the admin menus, the above seems the only setting of relevance to unobfuscate email addresses, but since they'​re obfuscated on the BALUG-Announce list, where that setting ​isn't and wasn't set, and where the email addresses in the archive are obfuscated anyway, seems we didn't (on DreamHost) have a setting we could access to turn off that behavior. 
-  * for our at migration time target operating system (Debian GNU/Linux 8.8 (jessie) amd64) to work with our desired anti-spam software/​configuration,​ we'll want package exim4-daemon-heavy - have that installed, but need to more suitably configure that (hasn't really been used, but with very low setting on priority of questions asked when installed, probably not so close to default - we want (much) closer to default for baseline/​reference before we start integrating the other pieces).+  * for our at migration time target operating system (Debian GNU/Linux 8.8 (jessie) amd64) to work with our desired anti-spam software/​configuration,​ we'll want package exim4-daemon-heavy - have that installed, but need to more suitably configure that ([done] hadn't really been used, was installed initially ​with very low setting on priority of questions asked when installed, probably not so close to default - we want (much) closer to default for baseline/​reference before we start integrating the other pieces ​[done - reinstalled very near to defaults, then adjusted suitably from there]).
   * So, for clean reinstall of exim4-daemon-heavy,​ temporarily installed another (conflicting) MTA (sendmail) and purged the exim4 packages, then purged the sendmail packages and reinstalled the exim4 packages thusly for a nice clean base install:\\   * So, for clean reinstall of exim4-daemon-heavy,​ temporarily installed another (conflicting) MTA (sendmail) and purged the exim4 packages, then purged the sendmail packages and reinstalled the exim4 packages thusly for a nice clean base install:\\
 <​file>​ <​file>​
Line 50: Line 50:
 </​file>​ </​file>​
   * We also notice it's //not// listening at all on IPv6 (not even \[::1\]:25, \[::​1\]:​587). ​ That's perfectly fine ... at least for now, as ...   * We also notice it's //not// listening at all on IPv6 (not even \[::1\]:25, \[::​1\]:​587). ​ That's perfectly fine ... at least for now, as ...
-  * One or our existing key present design/​migration criteria is: "not worse than" - notably not worse than existing configuration/​functionality - and the existing DreamHost.Com email stuff (certainly at //least// all Internet facing) is purely IPv4 - with no IPv6 at all ... so we're perfectly fine with enabling IPv6 later at our leisure (certainly ought be done ... but no extreme rush for that part).+  * One or our existing key present design/​migration criteria is: "not worse than" - notably not worse than existing configuration/​functionality - and the existing(/​former) ​DreamHost.Com email stuff (certainly at //least// all Internet facing) is(was) purely IPv4 - with no IPv6 at all ... so we're perfectly fine with enabling IPv6 later at our leisure (certainly ought be done ... but no extreme rush for that part).
   * Turns out our former MTA wasn't fully cleaned out - was unlinked but still running, SIGTERMed it, started exim, and rechecked our listening IPs for our ports:\\   * Turns out our former MTA wasn't fully cleaned out - was unlinked but still running, SIGTERMed it, started exim, and rechecked our listening IPs for our ports:\\
 <​file>​ <​file>​
Line 62: Line 62:
   * also installed: sa-exim (probably needed, and dependencies thereof)   * also installed: sa-exim (probably needed, and dependencies thereof)
   * also installed: clamav and libclamunrar7 (and dependencies thereof) - probably not required, but if the resource consumption isn't too great, very possibly a "good to have" - notably help us from being a (mostly immune) carrier.   * also installed: clamav and libclamunrar7 (and dependencies thereof) - probably not required, but if the resource consumption isn't too great, very possibly a "good to have" - notably help us from being a (mostly immune) carrier.
-  * also added additional "​reverse"​ DNS for temp.balug.org.:​+  * also added additional "​reverse"​ DNS for temp.balug.org. ​(later also added lists.balug.org.,​ temp.balug.org. will yet later be phased out):
 <​file>​ <​file>​
 238.194.144.198.in-addr.arpa. 10800 IN  CNAME   ​238.net232.194.144.198.in-addr.arpa. 238.194.144.198.in-addr.arpa. 10800 IN  CNAME   ​238.net232.194.144.198.in-addr.arpa.
Line 68: Line 68:
 </​file>​ </​file>​
   * after much more configuration of eximconfig, exim4, and some adding of packages and further configuration also including clamav and spamassassin and spfd and related, got to semi-working configuration ...   * after much more configuration of eximconfig, exim4, and some adding of packages and further configuration also including clamav and spamassassin and spfd and related, got to semi-working configuration ...
-  * also, clamav quite the (virtual) memory resource hog ... increased the host (virtual machine) RAM up from 512 MiB to 1 GiB - that seems sufficient for at least present - but clamav still consumes over 50% of RAM much of the time.  At 512 MIB of system RAM, the OOM killer was kicking in. +  * also, clamav quite the (virtual) memory resource hog ... increased the host (virtual machine) RAM up from 512 MiB to 1 GiB - that seems sufficient for at least present - but clamav still consumes over 50% of RAM much of the time.  At 512 MiB of system RAM, the OOM killer was kicking in (later made some additional adjustments to prevent Apache RAM consumption from ballooning too big and triggering OOM killer)
-  * send some initial test email messages to the domain - seemed to go relatively okay - at least after the RAM increase noted above - still much to (better) configure/​optimize,​ but may be "good enough"​ for reasonable start, and first (test) list, etc ... still need to install and configure list software.+  * sent some initial test email messages to the domain - seemed to go relatively okay - at least after the RAM increase noted above - still (was) much to (better) configure/​optimize,​ but may be "good enough"​ for reasonable start, and first (test) list, etc ... still need to install and configure list software.
   * added AAAA record for our MX - not really any great reason not to at this point:   * added AAAA record for our MX - not really any great reason not to at this point:
 <​file>​ <​file>​
 mx.temp.balug.org. ​     14400   ​IN ​     AAAA    2001:​470:​1f04:​19e::​2 mx.temp.balug.org. ​     14400   ​IN ​     AAAA    2001:​470:​1f04:​19e::​2
 </​file>​ </​file>​
-  * did set up BALUG-Test list, fixed some various issues, it seems at least (partially) working, but there are still various issues to correct and address, more to configure, etc. - but was at least able to successfully subscribe a non-local email address to it ... but still much more to do (and test). +  * did set up BALUG-Test list, fixed some various issues, it seems it was at least (partially) working, but there are(were) ​still various issues to correct and address, more to configure, etc. - but was at least able to successfully subscribe a non-local email address to it ... but (was) still much more to do (and test). 
-  * should probably create a bullet list of stuff to test on (test) list and confirm it's all working (sort'​a like a set of regression tests) - a small bit of which has already been addressed/​corrected.+  * should probably create a bullet list of stuff to test on (test) list and confirm it's all working (sort'​a like a set of regression tests) - a small bit of which has already been addressed/​corrected ​(did create list, tested, etc.).
  
 email/List stuff to (re)test - results (Y - good, N - failed, ? - to be tested) email/List stuff to (re)test - results (Y - good, N - failed, ? - to be tested)
Line 112: Line 112:
     for existing lists, toggling archive from public to private and back again seems sufficient to then create the needed link     for existing lists, toggling archive from public to private and back again seems sufficient to then create the needed link
 o N full mbox archive should be publicly available via public rsync o N full mbox archive should be publicly available via public rsync
-should be able to completely reload archive, add/drop messages from archive, etc. & document procedure thereof+should be able to completely reload archive, add/drop messages from archive, etc. & document procedure thereof ​(basically uses mailman command arch, with --wipe option, and run it as id list)
 o Y (working or mostly working?) mailman commands should work via email: subscribe/​unsubscribe/​help (need more complete list) o Y (working or mostly working?) mailman commands should work via email: subscribe/​unsubscribe/​help (need more complete list)
-(untested) mailman admin commands should work via email (need more complete list) +(untested) mailman admin commands should work via email (need more complete list) 
-(partially staged) ​should accept legitimate email for legitimate @balug.org addresses +should accept legitimate email for legitimate @balug.org addresses 
-N (future) ​default sending domain of host: @balug.org +default sending domain of host: @balug.org ​(for non-list email, list email updated to use @lists.balug.org) 
-[Y/partial - need to phase out ~all soft-fail] ​add appropriate SPF records for @lists.balug.org,​ @balug.org +o Y add/​update ​appropriate SPF records for @lists.balug.org,​ @balug.org, @temp.balug.org 
-[Apache mostly set, need to complete Mailman configuration changes] ​lists should use URLs starting with: https://​lists.balug.org/​ +lists should use URLs starting with: https://​lists.balug.org/​ 
-[MTA set, need to complete Mailman configuration changes] ​lists should use email addresses ending with: @lists.balug.org+lists should use email addresses ending with: @lists.balug.org
 o Y get raw mbox of archives from DreamHost.Com (completed 2017-09-16) o Y get raw mbox of archives from DreamHost.Com (completed 2017-09-16)
 o Y (emailed request of primary account holder 2017-08-20, reminder sent 2017-08-24 & 2017-08-30; I called and left voicemail message 2017-09-06; 2017-09-13: called and left voicemail again, sent email again, also sent cellular text message; 2017-09-14 called and left voicemail again and sent email again also gave additional option to have primary user transfer DreamHost primary user and billing to Michael Paoli; 2017-09-16: Dreamhost primary user opened ticket with DreamHost, DreamHost made the files available to us, I transferred files from DreamHost and ran sanity checks on the files (appears to be good set of the expected data)) above requires DreamHost support ticket opened requesting such from primary account holder o Y (emailed request of primary account holder 2017-08-20, reminder sent 2017-08-24 & 2017-08-30; I called and left voicemail message 2017-09-06; 2017-09-13: called and left voicemail again, sent email again, also sent cellular text message; 2017-09-14 called and left voicemail again and sent email again also gave additional option to have primary user transfer DreamHost primary user and billing to Michael Paoli; 2017-09-16: Dreamhost primary user opened ticket with DreamHost, DreamHost made the files available to us, I transferred files from DreamHost and ran sanity checks on the files (appears to be good set of the expected data)) above requires DreamHost support ticket opened requesting such from primary account holder
Line 127: Line 127:
 o Y http[s]://​lists.balug.org/​[mailman-prefix]/​ should only use https (redirect http to https) o Y http[s]://​lists.balug.org/​[mailman-prefix]/​ should only use https (redirect http to https)
 o Y http[s]://​lists.balug.org/​{,​cgi-bin{,/​{,​mailman{,/​}}}} should redirect to https://​lists.balug.org/​cgi-bin/​mailman/​listinfo o Y http[s]://​lists.balug.org/​{,​cgi-bin{,/​{,​mailman{,/​}}}} should redirect to https://​lists.balug.org/​cgi-bin/​mailman/​listinfo
-N (future) ​all of http[s]://​temp.balug.org/​ should 301 redirect to corresponding https://​lists.balug.org/​ URLs+all of http[s]://​temp.balug.org/​ should ​permanent (301redirect to corresponding https://​lists.balug.org/​ URLs
 o Y legacy http://​lists.balug.org URLs should 301 redirect to new locations (where different) o Y legacy http://​lists.balug.org URLs should 301 redirect to new locations (where different)
-http[s]://​{temp,​www}.balug.org/​lists/​balug-announce-do-not-auto-add.html update to use temp.balug.org+[superceded] ​http[s]://​{temp,​www}.balug.org/​lists/​balug-announce-do-not-auto-add.html update to use temp.balug.org
 o Y http[s]://​{temp,​www}.balug.org/​lists/​balug-announce-do-not-auto-add.html update to exclude BALUG-Test list o Y http[s]://​{temp,​www}.balug.org/​lists/​balug-announce-do-not-auto-add.html update to exclude BALUG-Test list
-N (future) ​http[s]://​{lists,​temp,​www}.balug.org/​lists/​balug-announce-do-not-auto-add.html update to use lists.balug.org +http[s]://​{lists,​temp,​www}.balug.org/​lists/​balug-announce-do-not-auto-add.html update to use lists.balug.org 
-N (future) ​http[s]://​{lists,​temp,​www}.balug.org/​lists/​balug-announce-do-not-auto-add.html canonicalize to https://​lists.balug.org/​lists/​balug-announce-do-not-auto-add.html +http[s]://​{lists,​temp,​www}.balug.org/​lists/​balug-announce-do-not-auto-add.html canonicalize to http[s]://​lists.balug.org/​lists/​balug-announce-do-not-auto-add.html 
-o ? (future) decommission temp.balug.org domain?+? http://​www.balug.org/​lists/​balug-announce-do-not-auto-add.html - redirect to https? (all the others above go to https) 
 +o Y decommission temp.balug.org domain 
 +o Y add IPv6 to {www.,​lists.,​}balug.org 
 +o [partially done] review/​fix/​verify Mailman mailman-loop and other Mailman lists bounce processing 
 +o [future] Mailman - review/​use/​configure VERP - for better bounce/​backscatter processing/​identification 
 +o Y add DNSSEC for balug.org.
 </​file>​ </​file>​
  
Line 162: Line 167:
 Fix for: [[http://​bazaar.launchpad.net/​~mailman-coders/​mailman/​2.1/​revision/​1721|Show case preserved emails in the roster]] (to be considered and possibly tested/​applied). Fix for: [[http://​bazaar.launchpad.net/​~mailman-coders/​mailman/​2.1/​revision/​1721|Show case preserved emails in the roster]] (to be considered and possibly tested/​applied).
  
-Added rewrite rules to remap old URLs to new - this will be useful most notably once we're hosting lists.balug.org away from DreamHost.com:​+Added rewrite rules to remap old URLs to new - this will be useful most notably once we're hosting lists.balug.org away from DreamHost.com ​(done):
 <​file>​ <​file>​
 RewriteRule "​^/​*listinfo\.cgi/​*$"​ https://​%0/​cgi-bin/​mailman/​listinfo [L,​R=permanent] RewriteRule "​^/​*listinfo\.cgi/​*$"​ https://​%0/​cgi-bin/​mailman/​listinfo [L,​R=permanent]
Line 179: Line 184:
 </​Directory>​ </​Directory>​
 </​file>​ </​file>​
 +(and that's had lists.balug.org enabled, and temp.balug.org has been phased out)
 added mailman-loop alias - this may not be optimal handling, but the alias needs to exist (needs to always be deliverable),​ and is probably at least "good enough"​ for now: added mailman-loop alias - this may not be optimal handling, but the alias needs to exist (needs to always be deliverable),​ and is probably at least "good enough"​ for now:
 <​file>​ <​file>​
 mailman-loop:​ postmaster mailman-loop:​ postmaster
 </​file>​ </​file>​
balug/mail_and_lists.1505965550.txt.bz2 · Last modified: 2017-09-21T03:45:50+0000 by michael_paoli