Page MenuHomePhabricator
Create Task
Maniphest T286911

Upgrade MXes to Bullseye
Closed, ResolvedPublic
Actions

Description

mx1001 and mx2001 are currently running Stretch and we should upgrade them to Bullseye. In principal we have two ways to do that:

  1. Install new mx1002 and mx2002 systems with Bullseye and switch over the MX records. This involves an IP change which may or may not impact mail sending (further research needed)
  2. Test the role's compatibility with Bullseye beforehand. Then temporarily filter access to each MX and reimage in place to the new OS (starting with mx2001 and then keeping that around for 1-2 weeks, then move on with mx1001)

Details

SubjectRepoBranchLines +/-
mx: enable tainted data checkingoperations/puppetproduction+0 -7
acmechief: Remove mx2002operations/puppetproduction+1 -2
Configure remaining domains with equal weights for mx1001/mx2001operations/dnsmaster+14 -14
Revert "Prefer codfw wiki smarthost over eqiad one for mx1001 reimage"operations/puppetproduction+1 -1
Configure a few domains with equal weights for mx1001/mx2001operations/dnsmaster+5 -5
Revert "Prefer mx2001 over mx1001 for internal smarthosts"operations/puppetproduction+1 -1
profile::mail::mx: Remove OS checksoperations/puppetproduction+3 -7
Remove mx1002/mx2002operations/puppetproduction+0 -20
Temporarily filter port 25 on mx1001 for reimageoperations/homer/publicmaster+13 -0
Prefer codfw wiki smarthost over eqiad one for mx1001 reimageoperations/puppetproduction+1 -1
Temporarily filter port 25 on mx1001 for reimageoperations/homer/publicmaster+13 -0
dhcp: Switch mx1001 to bullseyeoperations/puppetproduction+1 -1
profile::mail::mx: add +dkim_verbose to log_selector on bullseyeoperations/puppetproduction+6 -1
Switch remaining MX records to mx2001operations/dnsmaster+28 -28
Switch a few domains over to mx2001operations/dnsmaster+10 -10
Prefer mx2001 over mx1001 for internal smarthostsoperations/puppetproduction+1 -1
Revert "Temporarily filter port 25 on mx2001 for reimage"operations/homer/publicmaster+0 -13
Temporarily filter port 25 on mx2001 for reimageoperations/homer/publicmaster+13 -0
Disable new config validation on Bullseyeoperations/puppetproduction+9 -0
acmechief: acmechief: allow mx2002operations/puppetproduction+2 -1
Apply MX role to mx2002operations/puppetproduction+1 -1
discard traffic to mx2002 tcp/25operations/homer/publicmaster+26 -0
Show related patches Customize query in gerrit

Related Objects

Event Timeline

• MoritzMuehlenhoff created this task.Jul 19 2021, 1:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 19 2021, 1:03 PM
herron added a comment.Jul 19 2021, 2:26 PM

+1 for option 2, I think that will be a more straightforward approach overall.

In either case let's include a step to route and flush queued mail to the MX in the other DC before erasing/retiring each stretch host.

• taavi subscribed.Jul 29 2021, 9:37 AM

The exim version in Bullseye (4.94) had some breaking changes - see https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html#idm1404. So I agree that option 2 is a better approach.

• MoritzMuehlenhoff added subscribers: cmooney, ayounsi.Aug 4 2021, 3:02 PM

Ok, so let's proceed with option two. There's a test instance mx2002.wikimedia.org which I'll setup with the mx role and bullseye next week.

To prevent this test server from accidentally messing with our existing production mail infrastructure, I'd like to also filter port 25 for mx2002.wikimedia.org on the router level. @cmooney or @ayounsi, is that something you could set up?

Legoktm subscribed.Aug 4 2021, 8:07 PM
gerritbot added a comment.Aug 9 2021, 10:39 AM

Change 710943 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] discard traffic to mx2002 tcp/25

https://gerrit.wikimedia.org/r/710943

gerritbot added a project: Patch-For-Review.Aug 9 2021, 10:39 AM
gerritbot added a comment.Aug 10 2021, 8:19 AM

Change 710943 merged by jenkins-bot:

[operations/homer/public@master] discard traffic to mx2002 tcp/25

https://gerrit.wikimedia.org/r/710943

jenkins-bot mentioned this in rOHPUe3144ba475c9: discard traffic to mx2002 tcp/25.Aug 10 2021, 8:23 AM
ayounsi added a comment.Aug 10 2021, 8:49 AM

To prevent this test server from accidentally messing with our existing production mail infrastructure, I'd like to also filter port 25 for mx2002.wikimedia.org on the router level. @cmooney or @ayounsi, is that something you could set up?

Done, let us know when to rollback.

Maintenance_bot removed a project: Patch-For-Review.Aug 10 2021, 9:10 AM
• MoritzMuehlenhoff added a comment.Aug 10 2021, 11:02 AM
In T286911#7271558, @ayounsi wrote:

To prevent this test server from accidentally messing with our existing production mail infrastructure, I'd like to also filter port 25 for mx2002.wikimedia.org on the router level. @cmooney or @ayounsi, is that something you could set up?

Done, let us know when to rollback.

Thanks, will do.

gerritbot added a comment.Aug 10 2021, 11:10 AM

Change 711123 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Apply MX role to mx2002

https://gerrit.wikimedia.org/r/711123

gerritbot added a project: Patch-For-Review.Aug 10 2021, 11:10 AM
Stashbot added a comment.Aug 11 2021, 2:23 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-11T14:23:18Z] <moritzm> installing mx2002 T286911

ayounsi moved this task from Backlog to In Progress on the Infrastructure-Foundations board.Aug 12 2021, 9:07 AM
gerritbot added a comment.Aug 12 2021, 12:15 PM

Change 711123 merged by Muehlenhoff:

[operations/puppet@production] Apply MX role to mx2002

https://gerrit.wikimedia.org/r/711123

gerritbot added a comment.Aug 12 2021, 12:22 PM

Change 712277 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] acmechief: acmechief: allow mx2002

https://gerrit.wikimedia.org/r/712277

gerritbot added a comment.Aug 16 2021, 2:32 PM

Change 712277 merged by Herron:

[operations/puppet@production] acmechief: acmechief: allow mx2002

https://gerrit.wikimedia.org/r/712277

herron added a comment.Sep 8 2021, 3:29 PM

Seeing errors like this in the paniclog unfortunately

mx2002:~# zcat /var/log/exim4/paniclog*.gz
2021-08-29 00:00:01 1mK8Ez-0095xJ-IE Tainted filename for search '/etc/exim4/aliases/wikimedia.org'
2021-08-22 00:00:01 1mHau9-005TiL-Hz Tainted filename for search '/etc/exim4/aliases/wikimedia.org'
2021-08-19 00:40:18 1mF3ZJ-001qIs-21 Tainted filename for search '/etc/exim4/aliases/wikimedia.org'

Which AIUI is due to our current use of $domain in the alias lookup, and will cause aliases to be rejected in the upgraded version.

aliases:
        driver = redirect
        domains = +local_domains
        require_files = CONFDIR/aliases/$domain
        data = ${lookup{$local_part}lsearch*{CONFDIR/aliases/$domain}}

From the docs at http://exim.org/exim-html-current/doc/html/spec_html/ch-string_expansions.html

$domain 
<snip>
If the origin of the data is an incoming message, the result of expanding this variable is tainted. When un untainted version is needed, one should be obtained from looking up the value in a local (therefore trusted) database. Often $domain_data is usable in this role.

And found some additional related information in https://github.com/Exim/exim/blob/master/src/README.UPDATING

Some Transports now refuse to use tainted data in constructing their delivery
location; this WILL BREAK configurations which are not updated accordingly.
In particular: any Transport use of $local_part which has been relying upon
check_local_user far away in the Router to make it safe, should be updated to
replace $local_part with $local_part_data.
gerritbot added a comment.Sep 9 2021, 10:43 AM

Change 719975 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Disable new config validation on Bullseye

https://gerrit.wikimedia.org/r/719975

gerritbot added a comment.Sep 9 2021, 2:10 PM

Change 719975 merged by Muehlenhoff:

[operations/puppet@production] Disable new config validation on Bullseye

https://gerrit.wikimedia.org/r/719975

gerritbot added a comment.Sep 10 2021, 11:09 AM

Change 720277 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Temporarily filter port 25 on mx2001 for reimage

https://gerrit.wikimedia.org/r/720277

Stashbot added a comment.Sep 13 2021, 2:44 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-13T14:44:28Z] <herron> drained mx2001 mail queue to mx1001 T286911

gerritbot added a comment.Sep 13 2021, 3:21 PM

Change 720277 merged by Muehlenhoff:

[operations/homer/public@master] Temporarily filter port 25 on mx2001 for reimage

https://gerrit.wikimedia.org/r/720277

Muehlenhoff mentioned this in rOHPUe65ac7083b4a: Temporarily filter port 25 on mx2001 for reimage.Sep 13 2021, 3:36 PM
Stashbot added a comment.Sep 13 2021, 3:54 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-13T15:54:45Z] <moritzm> filtered mx2001 on the routers for reimage T286911

gerritbot added a comment.Sep 13 2021, 4:04 PM

Change 720783 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Revert \"Temporarily filter port 25 on mx2001 for reimage\"

https://gerrit.wikimedia.org/r/720783

• MoritzMuehlenhoff added a comment.Sep 13 2021, 4:09 PM

mx2001 is now filtered on the routers, in case there are any issues, this can be reverted by merging https://gerrit.wikimedia.org/r/720783 and running 'homer "cr*" merge' on cumin2002.

Next, mx2001 will be reimaged to bullseye tomorrow after the mw switchover.

Stashbot added a comment.Sep 14 2021, 5:45 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-14T17:45:41Z] <moritzm> reimaging mx2001 to bullseye T286911

gerritbot added a comment.Sep 14 2021, 6:33 PM

Change 720783 merged by Muehlenhoff:

[operations/homer/public@master] Revert \"Temporarily filter port 25 on mx2001 for reimage\"

https://gerrit.wikimedia.org/r/720783

Muehlenhoff mentioned this in rOHPU7ee6bc158a6f: Revert "Temporarily filter port 25 on mx2001 for reimage".Sep 14 2021, 6:34 PM
Stashbot added a comment.Sep 14 2021, 6:48 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-14T18:48:57Z] <moritzm> removed filter for tcp/25 on mx2001, reimage is complete T286911

gerritbot added a comment.Sep 15 2021, 12:25 PM

Change 721289 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Prefer mx2001 over mx1001 for internal smarthosts

https://gerrit.wikimedia.org/r/721289

gerritbot added a comment.Sep 16 2021, 1:17 PM

Change 721289 merged by Muehlenhoff:

[operations/puppet@production] Prefer mx2001 over mx1001 for internal smarthosts

https://gerrit.wikimedia.org/r/721289

gerritbot added a comment.Sep 16 2021, 2:26 PM

Change 721554 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Switch a few domains over to mx2001

https://gerrit.wikimedia.org/r/721554

gerritbot added a comment.Sep 16 2021, 2:26 PM

Change 721555 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Switch remaining MX records to mx2001

https://gerrit.wikimedia.org/r/721555

gerritbot added a comment.Sep 16 2021, 3:00 PM

Change 721554 merged by Muehlenhoff:

[operations/dns@master] Switch a few domains over to mx2001

https://gerrit.wikimedia.org/r/721554

• MoritzMuehlenhoff added a comment.Sep 16 2021, 3:16 PM

Status update: mx2001 is reimaged to Bullseye and working fine so far. The smart hosts config on our servers has been switched to prefer mx2001 over mx1001 and the MX records of a handful of lesser used domains now point to mx2001.
If there's no further issues, the remaining DNS records will be updated on Monday and following that mx1001 will be reimaged some time mid next week.

• MoritzMuehlenhoff added a project: SRE.Sep 16 2021, 3:17 PM
• Marostegui triaged this task as Medium priority.Sep 20 2021, 5:08 AM
gerritbot added a comment.Sep 20 2021, 3:01 PM

Change 721555 merged by Muehlenhoff:

[operations/dns@master] Switch remaining MX records to mx2001

https://gerrit.wikimedia.org/r/721555

gerritbot added a comment.Sep 21 2021, 7:50 AM

Change 722551 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Temporarily filter port 25 on mx1001 for reimage

https://gerrit.wikimedia.org/r/722551

joanna_borun changed the task status from Open to In Progress.Sep 21 2021, 1:33 PM
gerritbot added a comment.Sep 21 2021, 2:59 PM

Change 722630 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] profile::mail::mx: add +dkim_verbose to log_selector on bullseye

https://gerrit.wikimedia.org/r/722630

gerritbot added a comment.Sep 21 2021, 7:39 PM

Change 722630 merged by Herron:

[operations/puppet@production] profile::mail::mx: add +dkim_verbose to log_selector on bullseye

https://gerrit.wikimedia.org/r/722630

herron added a comment.Sep 21 2021, 8:06 PM

Re: the above patch -- DKIM metrics dropped off on mx2001 beginning yesterday. Our Exim metrics are generated via mtail parsing of the Exim log, and Exim 4.90 introduced a new log_selector option for DKIM logging which is switched off by default. Since the DKIM log lines were disabled by default, the DKIM metrics dropped off as well.

After merging https://gerrit.wikimedia.org/r/722630 DKIM metrics are flowing again from the bullseye hosts, looking better!

Screen Shot 2021-09-21 at 3.48.09 PM.png (1×4 px, 447 KB)

gerritbot added a comment.Sep 22 2021, 6:57 AM

Change 722816 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] dhcp: Switch mx1001 to bullseye

https://gerrit.wikimedia.org/r/722816

gerritbot added a comment.Sep 22 2021, 7:00 AM

Change 722816 merged by Muehlenhoff:

[operations/puppet@production] dhcp: Switch mx1001 to bullseye

https://gerrit.wikimedia.org/r/722816

gerritbot added a comment.Sep 22 2021, 12:51 PM

Change 722551 merged by Muehlenhoff:

[operations/homer/public@master] Temporarily filter port 25 on mx1001 for reimage

https://gerrit.wikimedia.org/r/722551

Muehlenhoff mentioned this in rOHPUcdf6d02821da: Temporarily filter port 25 on mx1001 for reimage.Sep 22 2021, 12:52 PM
Stashbot added a comment.Sep 22 2021, 1:26 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-22T13:26:48Z] <moritzm> mx1001 filterered on the routers for forthcoming reimage to bullseye T286911

Stashbot added a comment.Sep 22 2021, 1:39 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-22T13:39:22Z] <herron> flushed mx1001 mail queue to mx2001 T286911

gerritbot added a comment.Sep 22 2021, 2:13 PM

Change 722870 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Prefer codfw wiki smarthost over eqiad one for mx1001 reimage

https://gerrit.wikimedia.org/r/722870

gerritbot added a comment.Sep 22 2021, 2:20 PM

Change 722870 merged by Muehlenhoff:

[operations/puppet@production] Prefer codfw wiki smarthost over eqiad one for mx1001 reimage

https://gerrit.wikimedia.org/r/722870

Stashbot added a comment.Sep 22 2021, 3:02 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-22T15:02:44Z] <moritzm> re-installing mx1001 with bullseye T286911

Stashbot added a comment.Sep 22 2021, 3:52 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-22T15:52:09Z] <moritzm> removed filters on mx1001 filterered on the routers due to an issue with the mx1001 reinstall T286911

gerritbot added a comment.Sep 23 2021, 10:14 AM

Change 723072 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Temporarily filter port 25 on mx1001 for reimage

https://gerrit.wikimedia.org/r/723072

gerritbot added a comment.Sep 23 2021, 10:35 AM

Change 723072 merged by Muehlenhoff:

[operations/homer/public@master] Temporarily filter port 25 on mx1001 for reimage

https://gerrit.wikimedia.org/r/723072

Muehlenhoff mentioned this in rOHPUe6317ed55819: Temporarily filter port 25 on mx1001 for reimage.Sep 23 2021, 10:35 AM
Stashbot added a comment.Sep 23 2021, 10:53 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-23T10:53:03Z] <moritzm> mx1001 filterered on the routers for forthcoming reimage to bullseye T286911

Stashbot added a comment.Sep 23 2021, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-23T13:27:40Z] <moritzm> reimaging mx1001 to bullseye T286911

Stashbot added a comment.Sep 23 2021, 2:19 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-23T14:19:45Z] <moritzm> removed routers filter for mx1001, reimage to bullseye complete T286911

• MoritzMuehlenhoff added a comment.Sep 23 2021, 2:37 PM

Both mx1001 and mx2001 are now running Bullseye. There's a little cleanup/followup work, but the core of the work is completed.

ops-monitoring-bot added a comment.Sep 24 2021, 7:42 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: mx1002.wikimedia.org

  • mx1002.wikimedia.org (WARN)
    • Host not found on Icinga, unable to downtme it
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
ops-monitoring-bot added a comment.Sep 24 2021, 8:08 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: mx2002.wikimedia.org

  • mx2002.wikimedia.org (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
gerritbot added a comment.Sep 24 2021, 8:13 AM

Change 723421 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove mx1002/mx2002

https://gerrit.wikimedia.org/r/723421

gerritbot added a comment.Sep 24 2021, 8:15 AM

Change 723422 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] acmechief: Remove mx2002

https://gerrit.wikimedia.org/r/723422

gerritbot added a comment.Sep 24 2021, 8:49 AM

Change 723421 merged by Muehlenhoff:

[operations/puppet@production] Remove mx1002/mx2002

https://gerrit.wikimedia.org/r/723421

gerritbot added a comment.Sep 24 2021, 9:04 AM

Change 723433 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Revert \"Prefer codfw wiki smarthost over eqiad one for mx1001 reimage\"

https://gerrit.wikimedia.org/r/723433

gerritbot added a comment.Sep 24 2021, 9:13 AM

Change 723434 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Revert \"Prefer mx2001 over mx1001 for internal smarthosts\"

https://gerrit.wikimedia.org/r/723434

gerritbot added a comment.Sep 24 2021, 10:15 AM

Change 723473 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Configure a few domains with equal weights for mx1001/mx2001

https://gerrit.wikimedia.org/r/723473

gerritbot added a comment.Sep 24 2021, 10:25 AM

Change 723482 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Configure remaining domains with equal weights for mx1001/mx2001

https://gerrit.wikimedia.org/r/723482

gerritbot added a comment.Sep 24 2021, 10:28 AM

Change 723487 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] profile::mail::mx: Remove OS checks

https://gerrit.wikimedia.org/r/723487

• MoritzMuehlenhoff added a comment.Sep 24 2021, 10:50 AM

The two VMs (mx1002/mx2002) which were used to test the Bullseye setup have been taken down.

gerritbot added a comment.Sep 27 2021, 7:22 AM

Change 723487 merged by Muehlenhoff:

[operations/puppet@production] profile::mail::mx: Remove OS checks

https://gerrit.wikimedia.org/r/723487

gerritbot added a comment.Sep 27 2021, 7:35 AM

Change 723434 merged by Muehlenhoff:

[operations/puppet@production] Revert \"Prefer mx2001 over mx1001 for internal smarthosts\"

https://gerrit.wikimedia.org/r/723434

gerritbot added a comment.Sep 27 2021, 12:29 PM

Change 723473 merged by Muehlenhoff:

[operations/dns@master] Configure a few domains with equal weights for mx1001/mx2001

https://gerrit.wikimedia.org/r/723473

gerritbot added a comment.Sep 27 2021, 12:50 PM

Change 723433 merged by Muehlenhoff:

[operations/puppet@production] Revert \"Prefer codfw wiki smarthost over eqiad one for mx1001 reimage\"

https://gerrit.wikimedia.org/r/723433

gerritbot added a comment.Sep 30 2021, 12:04 PM

Change 723482 merged by Muehlenhoff:

[operations/dns@master] Configure remaining domains with equal weights for mx1001/mx2001

https://gerrit.wikimedia.org/r/723482

Stashbot added a comment.Sep 30 2021, 12:17 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-30T12:17:20Z] <moritzm> adapted MX records to point to both mx1001.wikimedia.org and mx2001.wikimedia.org with equal weights T286911

• MoritzMuehlenhoff closed this task as Resolved.Sep 30 2021, 12:21 PM
• MoritzMuehlenhoff claimed this task.

mx1001/mx2001 have been reimaged to Bullseye (reusing the VM/IP for potential IP reputation issues).

Along with the update the setup has also been adapted to better spread the load across both servers:

  • For outbound mail (wiki-mail records and smarthosts of our servers) servers in eqiad/esams use mx1001 and codfw/eqsin/ulsfo use mx2001
  • For inbound mail the weight of the MX records are now equal, which should make mail servers do round-robin lookups for both.
herron awarded a token.Oct 1 2021, 7:13 PM
gerritbot added a comment.Oct 6 2021, 12:24 PM

Change 723422 merged by Muehlenhoff:

[operations/puppet@production] acmechief: Remove mx2002

https://gerrit.wikimedia.org/r/723422

gerritbot added a comment.May 31 2022, 6:49 PM

Change 801799 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] mx: enable tainted data checking

https://gerrit.wikimedia.org/r/801799

gerritbot added a comment.Jun 2 2022, 8:58 PM

Change 801799 merged by JHathaway:

[operations/puppet@production] mx: enable tainted data checking

https://gerrit.wikimedia.org/r/801799

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits