Thank you very much :)

I've rebased and implemented one of @Volans recommandation on the CR that had already been created by @RLazarus
Would love to have your eyeballs on it Data-Persistence

Thanks @jbond ! It'll update the file in max 30 minutes, that sounds good enough for me since I doubt a lot of pcc runs would be done right after the switchover. If need be we can still force a puppet-run on the compiler hosts.

I don't think this should be considered a blocker for T327920: March 2023 Datacenter Switchover.

I don't think this should be considered a blocker for T327920: March 2023 Datacenter Switchover
However, we should address it for mw-on-k8s and releases.

Directly related work by @Joe that could use a couple eyeballs https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/886038

Is this work still ongoing? If not, would it be possible to update the doc before T327920: March 2023 Datacenter Switchover ?

We are now correctly sending, ingesting and storing slowlogs in ECS format. Next step, dashboards.

@RLazarus You were working on rewriting the app server warmup script in Python, I suppose this is directly related.

Debugging steps:

• Launch helmfile -e staging apply
• From another deploy1002 shell

kube-env zotero staging
kubectl get pods
# Get the most recent pod
kubectl logs <pod_id> zotero-staging

In T328287#8579336, @Trizek-WMF wrote:

@Clement_Goubert Has anything major changed in your process since the last time (noticeable things that are worth being announced)?

Is this still relevant, does it need to be finished for T327920: March 2023 Datacenter Switchover, or can it be closed?

In T328287#8579474, @Trizek-WMF wrote:

As you gave 3 dates in the task description, can you confirm precisely when the wikis will be in a read-only mode noticeable by anyone? Thank you!

2 more mw hosts affected:

cgoubert@cumin1001:~$ sudo cumin 'mw23[26,29,32].codfw.wmnet' 'ipmi-sel | tail -n2'
3 hosts will be targeted:
mw[2326,2329,2332].codfw.wmnet
OK to proceed on 3 hosts? Enter the number of affected hosts to confirm or "q" to quit: 3
===== NODE GROUP =====                                                                                                                                  
(1) mw2329.codfw.wmnet                                                                                                                                  
----- OUTPUT of 'ipmi-sel | tail -n2' -----                                                                                                             
25  | Jan-30-2023 | 21:44:31 | Status           | Power Supply                | Power Supply input lost (AC/DC)                                         
26  | Jan-30-2023 | 21:45:55 | PS Redundancy    | Power Supply                | Fully Redundant                                                         
===== NODE GROUP =====                                                                                                                                  
(1) mw2332.codfw.wmnet                                                                                                                                  
----- OUTPUT of 'ipmi-sel | tail -n2' -----                                                                                                             
18  | Jan-30-2023 | 21:44:25 | Status           | Power Supply                | Power Supply input lost (AC/DC)                                         
19  | Jan-30-2023 | 21:44:30 | PS Redundancy    | Power Supply                | Redundancy Lost                                                         
===== NODE GROUP =====                                                                                                                                  
(1) mw2326.codfw.wmnet                                                                                                                                  
----- OUTPUT of 'ipmi-sel | tail -n2' -----                                                                                                             
13  | Sep-28-2022 | 15:05:49 | Status           | Power Supply                | Power Supply input lost (AC/DC)                                         
14  | Jan-30-2023 | 17:15:41 | Status           | Power Supply                | Power Supply input lost (AC/DC)                                         
================

Tagging @akosiaris and @Eevans for awareness as they are on-call this week too.

Handing off to this week's Clinic Duty SRE.
@herron you should just have to merge the CR and create the kerberos principal.

• Approval from @Ottomata or @odimitrijevic as group approvers
• Approval from @JanWMF as manager
• Out of band key verification

The new key has been pushed, please allow for 30 minutes from this post for it to be deployed. Feel free to reopen the task if you experience any issue.

Got out of band confirmation through slack, proceeding.

In T305979#8557029, @MoritzMuehlenhoff wrote:

In T305979#8557028, @Clement_Goubert wrote:

FYI this created warnings in cross-validate-accounts, CR incoming.

I made https://gerrit.wikimedia.org/r/c/operations/puppet/+/883500 earlier, review welcome :-)

FYI this created warnings in cross-validate-accounts, CR incoming.

Hi @RGaines_WMF
Sorry this did not get picked up earlier.
According to https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#Google_&_Bing_search_console_access SRE does not own access requests for GSC, Core Experiences does.
@SCherukuwada may be able to help if this is still relevant?

In T239862#8504801, @LSobanski wrote:

@Joe's patch mentioned above has been merged in Feb 2021 and the hardcoded IP config has since been moved to monitoring.pp [1]. @CDanis can the entry be removed at this point?

[1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/725298

@Samwalton9 your access to the relevant groups has been granted. Please wait 30m (as of this comment) before trying it out as the access propagates across the fleet. You should also have received an email regarding Kerberos, you can follow the instructions on there to set your credentials. If you didn't, please check your spam folder just in case.

Retention updated for mediawiki.httpd.accesslog in codfw

Ack. CR updated for kerberos access.

Hi @Samwalton9,
I need :

• Approval from @Ottomata or @odimitrijevic for the privilege extension to shell access, as group approvers
• Approval from @DannyH for the same, as manager

According to #acl*procurement-review access needs to be granted by @mark or @RobH

Being bold and closing this task since it's had no meaningful update in 7 years. Feel free to reopen if needed.

@taavi Access request merged, you should have your access around 30 minutes from now when puppet has run. Resolving, don't hesitate to reopen if needed.

@taavi Patch ready, assuming you don't need kerberos access. Here are the Data Access User Responsibilities

@Muhammad_Yasser_Jazirahly_WMDE your access to the relevant groups has been granted. Please wait 30m (as of this comment) before trying it out as the access propagates across the fleet. You should also have received an email regarding Kerberos, you can follow the instructions on there to set your credentials. If you didn't, please check your spam folder just in case.

You should have received an email regarding Kerberos, you can follow the instructions on there to set your credentials. If you didn't, please check your spam folder just in case.

@MShilova_WMF your access to the wmf group has been granted. Please wait 30m (as of this comment) before trying it out as the access propagates across the fleet. I'll resolve this task, feel free to reopen if you meet issues.

@Ollie.Shotton_WMDE your access to the relevant groups has been granted. Please wait 30m (as of this comment) before trying it out as the access propagates across the fleet. I'll resolve this task, feel free to reopen if you meet issues.

The link in the task description 404s. Being bold and closing as Invalid, feel free to reopen with up to date information if needed.

• Merge CR
• Grant LDAP group access