Page MenuHomePhabricator
Create Task
Maniphest T111653

Encrypt all the things
Closed, ResolvedPublic
Actions

Description

All of our flows carrying private or sensitive information between datacenters (but in many cases between servers in the same datacenter as well) should be encrypted.

We currently have IPsec deployed for some limited use cases. We should expand this and/or deploy TLS where possible. This task will track remaining or upcoming work for this.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedLSobanskiT111653 Encrypt all the things
 ResolvedBBlackT92602 Secure inter-datacenter web request log (Kafka) traffic
 ResolvedOttomataT106581 Build 0.8.2.1 Kafka package and upgrade Kafka brokers
 DeclinedOttomataT98161 Build Kafka 0.8.1.1 package for Jessie and upgrade Brokers to Jessie.
 ResolvedOttomataT103106 Create jmxtrans Jessie package
 ResolvedOttomataT90640 Audit hyperthreading on analytics nodes.
 ResolvedEevansT108953 Cassandra inter-node encryption (TLS)
 ResolvedRobHT111382 codfw 3x spares for cassandra encryption testing
 ResolvedOttomataT97294 Turn off webrequest udp2log instances.
 ResolvedOttomataT97771 Backport? and install kafkacat (on stat1002?)
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
 ResolvedJgreenT110592 package udp-filter for Trusty, for use on fundraising banner_logger
 ResolvedJgreenT110740 build libcidr package for Trusty
 ResolvedJgreenT110739 build libanon package for trusty
 ResolvedJgreenT112139 build librdkafka for Trusty
 ResolvedJgreenT110591 reformulate kafkatee package to work with Trusty
Restricted Task
 ResolvedawightT97676 Verify kafkatee use for fundraising logs on erbium
 ResolvedcorenT97860 Add andyrussg to udp2log-users group to allow him to verify kafkatee generated fundraising log files on erbium
 ResolvedawightT116800 Impression log parsers should get sample rate from filenames
 ResolvedOttomataT117727 Turn off sqstat udp2log instance
 ResolvedOttomataT83580 Overhaul reqstats
 ResolvedJoeT118979 Migrate reqstats icinga alerts to new graphite metrics and deprecate or adapt reqstats gdash
 ResolvedjcrespoT111654 Set up TLS for MariaDB replication
 ResolvedjcrespoT120122 Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic)
 Resolved• CmjohnsonT120689 es1019 and its management interface are unresponsive
 ResolvedjcrespoT151995 Rolling restart of external storage servers for TLS certificate update
 ResolvedjcrespoT152029 Rolling restart of parsercache servers for TLS certificate update
 ResolvedjcrespoT152188 Restart pending mysql hosts with old TLS cert
 ResolvedMarosteguiT152364 db1047 out of disk space, eventlogging_sync spam
 DuplicateNoneT152595 Implement TLS expiration/validation checking for MariaDB certificates
 ResolvedjcrespoT156005 Reimage db1065 and db1066
 ResolvedfaidonT82576 Enable STARTTLS (both inbound and outbound) on lists
 ResolvedDzahnT105756 Mailman Upgrade (Jessie & Mailman 2.x) and migration to a VM
 ResolvedDzahnT108057 Phabricator NDA for John Lewis
Restricted Task
 ResolvedDzahnT108073 test importing of mailing list configs and archives on staging VM
 ResolvedDzahnT108071 export config and archive data from sodium
 ResolvedDzahnT108080 service IP can't be switched over
 ResolvedDzahnT108082 give John Lewis shell access on the mailman staging VM
 ResolvedDzahnT108070 install jessie on new VM for mailman
 ResolvedRobHT108065 eqiad: 1 VM request for mailman
 ResolvedDzahnT108383 create basic mailman setup on fermium (jessie) for testing import
 ResolvedRobHT109393 rename wikitech-announce.disabled.T100503
 ResolvedDzahnT109399 go through all directories in /var/lib/mailman and decide if migration is needed
ResolvedDzahnT109467 write migration plan for mailman
 ResolvedDzahnT109539 rename lists mwapi-team.disabled.T97148 and flowfunding.disabled.T97328 ?
 Resolved• JohnLewisT109624 move mailman server and service IPs to hiera / make it possible to run multiple instances at once
 ResolvedDzahnT109838 clean up mailman data directory (moderated messages > 0.5 million)
Restricted Task
 ResolvedDzahnT109890 reinstall fermium with public IP
ResolvedDzahnT109923 add public IP for fermium - DNS and DHCP change for reinstall
 DuplicateDzahnT109891 announce mailman downtime
 ResolvedDzahnT109921 setup rsyncd on fermium to copy files from sodium
 ResolvedDzahnT109922 write script to import mailing lists from other server
 DuplicateDzahnT109924 reinstall fermium with jessie and public IP
 ResolvedDzahnT109925 apply regular lists role on fermium and confirm no issues
 ResolvedDzahnT110695 mailman: listinfo template encoding
 ResolvedDzahnT110129 rsync all configs and archives one more time
 InvalidDzahnT110131 import all lists with the script we wrote for that
 ResolvedDzahnT110132 lower lists.wikimedia.org TTL to 5 min
 ResolvedDzahnT110133 announce scheduled downtime
 DeclinedDzahnT110135 right before the switch: lower TTL to 10 seconds
 ResolvedDzahnT110136 hold lists.wikimedia.org with exim
 ResolvedDzahnT110137 shut down mailman on sodium
 ResolvedDzahnT110138 rsync the diff since mail was held on sodium
 ResolvedDzahnT110139 switch over mailman service IP
 ResolvedDzahnT110140 send follow-up email, announce changes with new mailman version if any that have user impact
 Resolved• JohnLewisT110382 mailman cronjobs not running?
 ResolvedDzahnT110440 rsync exim spool directory
ResolvedDzahnT110441 test sending individual mails from fermium during migration
 ResolvedDzahnT112229 fermium needs to have exim4-daemon-heavy installed, not -light
 ResolvedDzahnT113020 run /var/lib/mailman/bin/update and ./check_perms
 ResolvedDzahnT113045 start exim on fermium / revert migration hack
 ResolvedfaidonT109239 Ensure mailman VM setup has adequate entropy for STARTTLS
 ResolvedfaidonT101452 Protect incoming emails with SMTP STARTLS
 ResolvedfaidonT113211 Replace primary mail relays (polonium/lead)
 ResolvedGehelT124444 Look into encrypting Elasticsearch traffic
 ResolvedGehelT128077 Create a PKI that can be used by Puppet and for general purpose certificates
 ResolvedEBernhardsonT130219 Implement connection pooling for elasticsearch connections
 ResolvedGehelT130365 Enable metric collection on nginx for elasticsearch
 ResolvedGehelT130366 Should we have a specific check for SSL certificate expiration on elasticsearch
 ResolvedGehelT131839 Activate SSL + connection pooling for CirrusSearch on PROD
 OpenNoneT126989 MediaWiki logging & encryption
 ResolvedfgiunchediT127455 Enable HTTPS for Swift traffic
 OpenNoneT127498 git/http operations in scap should be secure
 ResolvedfgiunchediT136312 Encrypt syslog traffic
 DeclinedArielGlennT123560 investigate rsync between dcs with encryption
 ResolvedJgreenT142993 encrypt fundraising kafka collector traffic
 ResolvedOttomataT166167 Write generic certificate management software for use with Puppet and Self Signing CAs.
 ResolvedJgreenT142994 configure TLS for fundraising syslog collection
 ResolvedJgreenT145116 replace indium (eqiad fundraising logger) with new hardware running jessie
Unknown Object (Task)
Unknown Object (Task)
Unknown Object (Task)
 ResolvedfaidonT159336 deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001)
 ResolvedJgreenT163127 rack and cable frlog1001
 Resolved• CmjohnsonT164748 configure RAID on frlog1001
 ResolvedaaronT160616 Enable HTTPS for swift clients
 ResolvedfgiunchediT161717 Point swiftrepl to swift HTTPS

Event Timeline

faidon created this task.Sep 6 2015, 8:58 PM
faidon raised the priority of this task from to Medium.
faidon updated the task description. (Show Details)
faidon added a project: acl*sre-team.
faidon subscribed.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptSep 6 2015, 8:58 PM
faidon added subtasks: T108580: HTTPS for internal service traffic, T92602: Secure inter-datacenter web request log (Kafka) traffic, T108953: Cassandra inter-node encryption (TLS), T97294: Turn off webrequest udp2log instances..Sep 6 2015, 8:59 PM
Krenair subscribed.Sep 6 2015, 9:05 PM
Platonides added subtasks: T82576: Enable STARTTLS (both inbound and outbound) on lists, T101452: Protect incoming emails with SMTP STARTLS.Sep 6 2015, 9:23 PM
Platonides subscribed.
MoritzMuehlenhoff set Security to None.Sep 7 2015, 6:20 AM
MoritzMuehlenhoff subscribed.
MarcoAurelio awarded a token.Sep 7 2015, 10:45 AM
MarcoAurelio subscribed.
faidon closed subtask T82576: Enable STARTTLS (both inbound and outbound) on lists as Resolved.Sep 21 2015, 11:43 AM
faidon closed subtask T101452: Protect incoming emails with SMTP STARTLS as Resolved.Sep 23 2015, 1:43 PM
fgiunchedi closed subtask T108953: Cassandra inter-node encryption (TLS) as Resolved.Dec 7 2015, 3:56 PM
JanZerebecki subscribed.Dec 7 2015, 4:08 PM
BBlack closed subtask T92602: Secure inter-datacenter web request log (Kafka) traffic as Resolved.Dec 15 2015, 7:01 PM
BBlack removed a subtask: T108580: HTTPS for internal service traffic.Dec 15 2015, 7:18 PM
Ottomata closed subtask T97294: Turn off webrequest udp2log instances. as Resolved.Dec 15 2015, 7:20 PM
Legoktm subscribed.Jan 22 2016, 6:12 PM
Volker_E subscribed.Jan 22 2016, 6:14 PM
faidon created subtask T127455: Enable HTTPS for Swift traffic.Feb 19 2016, 3:25 PM
• demon created subtask T127498: git/http operations in scap should be secure.Feb 19 2016, 6:41 PM
greg subscribed.Mar 15 2016, 4:10 PM
• csteipp subscribed.Mar 15 2016, 4:11 PM
• Deskana closed subtask T124444: Look into encrypting Elasticsearch traffic as Resolved.Apr 8 2016, 9:04 PM
fgiunchedi created subtask T136312: Encrypt syslog traffic.May 26 2016, 2:49 PM
ArielGlenn added a subtask: T123560: investigate rsync between dcs with encryption.May 30 2016, 4:49 PM
Paladox subscribed.May 30 2016, 4:50 PM
abian subscribed.Jun 4 2016, 10:26 AM
Jgreen added a subtask: T142993: encrypt fundraising kafka collector traffic.Aug 15 2016, 2:01 PM
Jgreen added a subtask: T142994: configure TLS for fundraising syslog collection.Aug 15 2016, 2:13 PM
Peachey88 subscribed.Aug 15 2016, 2:20 PM
fgiunchedi mentioned this in T150822: Internal PKI for secure communication - Barcelona Ops offsite 2016.Nov 16 2016, 1:23 AM
jcrespo closed subtask T111654: Set up TLS for MariaDB replication as Resolved.Feb 9 2017, 5:08 PM
Jgreen closed subtask T142994: configure TLS for fundraising syslog collection as Resolved.Feb 14 2017, 4:55 PM
fgiunchedi closed subtask T127455: Enable HTTPS for Swift traffic as Resolved.Mar 13 2017, 1:33 PM
fgiunchedi created subtask T160616: Enable HTTPS for swift clients.Mar 16 2017, 9:15 AM
fgiunchedi created subtask T161717: Point swiftrepl to swift HTTPS.Mar 29 2017, 4:07 PM
aaron closed subtask T160616: Enable HTTPS for swift clients as Resolved.Jul 5 2017, 11:56 PM
fgiunchedi closed subtask T161717: Point swiftrepl to swift HTTPS as Resolved.Jul 6 2017, 11:47 AM
Aklapper added a project: Epic.Dec 26 2017, 8:33 AM
Jgreen closed subtask T142993: encrypt fundraising kafka collector traffic as Resolved.Jun 18 2018, 10:05 PM
fgiunchedi closed subtask T136312: Encrypt syslog traffic as Resolved.Oct 30 2018, 2:41 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:18 PM
ArielGlenn closed subtask T123560: investigate rsync between dcs with encryption as Declined.Mar 20 2020, 9:14 AM
LSobanski closed this task as Resolved.Nov 28 2022, 3:05 PM
LSobanski claimed this task.
LSobanski subscribed.

The remaining two open action items are assigned to specific teams and the value of this task is limited so I'm resolving it.

Volker_E awarded a token.Nov 28 2022, 3:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits