User Details
User Details
- User Since
- Sep 5 2023, 11:23 AM (161 w, 1 d)
- Availability
- Available
- IRC Nick
- brouberol
- LDAP User
- Brouberol
- MediaWiki User
- BRouberol-WMF [ Global Accounts ]
Mon, Oct 5
Mon, Oct 5
brouberol added a comment to T435595: [Spike] Evaluate OpenTofu against the RADOS Gateway for IAM and bucket provisioning.
The "fix" was adding a write bucket caps
$ sudo radosgw-admin caps add \ --uid=opentofu-root \ --caps="buckets=read,write"
brouberol added a comment to T435595: [Spike] Evaluate OpenTofu against the RADOS Gateway for IAM and bucket provisioning.
I tried to create a bucket with quotas, but it failed:
OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create
brouberol added a comment to T435595: [Spike] Evaluate OpenTofu against the RADOS Gateway for IAM and bucket provisioning.
I then deleted that resource from main.tf and was able to delete the bucket via tofu apply:
root@ceph-admin2001:/srv/tofu-infra# tofu plan radosgw_s3_bucket.example: Refreshing state... [id=my-example-bucket]
brouberol added a comment to T435595: [Spike] Evaluate OpenTofu against the RADOS Gateway for IAM and bucket provisioning.
I then added the following resource to main.tf and was able to create it via tofu apply:
resource "radosgw_s3_bucket" "example" { bucket = "my-example-bucket" }
brouberol added a comment to T435595: [Spike] Evaluate OpenTofu against the RADOS Gateway for IAM and bucket provisioning.
I manually defined the following main.tf file:
brouberol changed the status of T435595: [Spike] Evaluate OpenTofu against the RADOS Gateway for IAM and bucket provisioning, a subtask of T435472: Manage the Ceph RGW IAM and S3 user configuration as code, from Open to In Progress.
brouberol changed the status of T435595: [Spike] Evaluate OpenTofu against the RADOS Gateway for IAM and bucket provisioning from Open to In Progress.
brouberol added a comment to T435595: [Spike] Evaluate OpenTofu against the RADOS Gateway for IAM and bucket provisioning.
Creating an IAM account for opentofu:
root@cephosd2001:~# radosgw-admin account create --account-name=opentofu
{
"id": "RGW22654237449906372",
"tenant": "",
"name": "opentofu",
"email": "",
"quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"max_users": 1000,
"max_roles": 1000,
"max_groups": 1000,
"max_buckets": 1000,
"max_access_keys": 4
}brouberol added a comment to T435472: Manage the Ceph RGW IAM and S3 user configuration as code.
Indeed, something I forgot to mention is that I saw that WMCS was already using opentofu, and it was already added to our apt repo.
brouberol added a comment to T435472: Manage the Ceph RGW IAM and S3 user configuration as code.
Here are some monday morning thoughts.
Mon, Sep 28
Mon, Sep 28
brouberol added a comment to T438350: Set up a postgres database supporting the fundraising data for growthbook experiments.
I just talked to @phuedx who mentioned that we should put this on hold for a while. There might be a world in which the fundraising events live in the data lake and all of this is moot.
brouberol moved T438062: Dumps NFS mount down from Blocked/Waiting to Done on the Data-Platform-SRE (2026-09-18 - 2026-10-09) board.
brouberol added a comment to T438350: Set up a postgres database supporting the fundraising data for growthbook experiments.
brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --region=dpe --host-bucket=no mb s3://postgresql-growthbook-fundraising Bucket 's3://postgresql-growthbook-fundraising/' created
brouberol added a comment to T438350: Set up a postgres database supporting the fundraising data for growthbook experiments.
brouberol@cephosd1001:~$ sudo radosgw-admin user create --uid=postgresql-growthbook-fundraising --display-name="postgresql-growthbook-fundraising"
{
"user_id": "postgresql-growthbook-fundraising",
"display_name": "postgresql-growthbook-fundraising",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "postgresql-growthbook-fundraising",
"access_key": "[REDACTED]",
"secret_key": "[REDACTED]"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}brouberol changed the status of T438350: Set up a postgres database supporting the fundraising data for growthbook experiments, a subtask of T436940: [EPIC] Prepare GrowthBook for receiving experiment-related analytics events from donate.wikimedia.org, from Open to In Progress.
brouberol changed the status of T438350: Set up a postgres database supporting the fundraising data for growthbook experiments from Open to In Progress.
brouberol moved T439220: Repurpose ganeti-jumbo hosts as dse-k8s workers from In Progress to Done on the Data-Platform-SRE (2026-09-18 - 2026-10-09) board.
brouberol added a comment to T439220: Repurpose ganeti-jumbo hosts as dse-k8s workers.
The 3 ganeti-jumbo codfw nodes have been repurposed as dse-k8s-codfw workers
brouberol updated the task description for T429017: Delete orphan EventLogging topic `eventlogging_HomepageModulet`.
brouberol closed T429017: Delete orphan EventLogging topic `eventlogging_HomepageModulet` as Resolved.
brouberol added a comment to T429017: Delete orphan EventLogging topic `eventlogging_HomepageModulet`.
brouberol@kafka-jumbo1010:~$ kafka topics --describe --topic eventlogging_HomepageModulet Error while executing topic command : Topic 'eventlogging_HomepageModulet' does not exist as expected
brouberol updated subscribers of T436631: Airflow testing instances made with airflow-devenv aren't able to find DAG that's required within sensor.
@atsuko Could this have to do with changes made to the RESTExternalSensor ?
brouberol added a comment to T438790: airflow-devenv creates wrong envoy when using --name.
Ah, perfect!
brouberol added a comment to T439220: Repurpose ganeti-jumbo hosts as dse-k8s workers.
All 3 eqiad ganeti-jumbo hosts were repurposed as dse-k8s-eqiad workers
Fri, Sep 25
Fri, Sep 25
brouberol updated subscribers of T439241: dse-k8s-eqiad pods lost access to the urldownloader proxies.
[2026-09-25, 12:30:25 UTC] {opensearch_ipoid_download_and_index.py:91} INFO - Fetching metadata from endpoint: /v2/anonymous-residential/latest
[2026-09-25, 12:30:25 UTC] {base.py:84} INFO - Retrieving connection 'spurus'
[2026-09-25, 12:30:25 UTC] {opensearch_ipoid_download_and_index.py:108} INFO - Downloading file from: /v2/anonymous-residential/20260925/feed.json.gz to /tmp/20260925-feed.json.gz
[2026-09-25, 12:30:25 UTC] {base.py:84} INFO - Retrieving connection 'spurus'
[2026-09-25, 12:30:31 UTC] {opensearch_ipoid_download_and_index.py:121} INFO - Download completed successfully!
[2026-09-25, 12:30:31 UTC] {opensearch_ipoid_download_and_index.py:122} INFO - Downloaded size: 828,985,783 bytes
[2026-09-25, 12:30:31 UTC] {opensearch_ipoid_download_and_index.py:123} INFO - Expected size: 828,985,783 bytes
...brouberol raised the priority of T439241: dse-k8s-eqiad pods lost access to the urldownloader proxies from High to Unbreak Now!.
brouberol changed the status of T439241: dse-k8s-eqiad pods lost access to the urldownloader proxies from Open to In Progress.
brouberol changed the status of T439220: Repurpose ganeti-jumbo hosts as dse-k8s workers from Open to In Progress.
brouberol added a comment to T438351: Define the fundraising project in Growthbook.
Thanks!
Wed, Sep 23
Wed, Sep 23
brouberol added a comment to T438823: Medium Errors detected on cephosd1002:/dev/sdo.
Agreed. I think we need to commit the absent_osds hiera change, run puppet on the host, and then run the cookbook. With the caveat that an absent OSD should not prompt puppet to trigger its deletion. Simply stop managing it.
brouberol added a comment to T438823: Medium Errors detected on cephosd1002:/dev/sdo.
...
Error EBUSY: OSD(s) 24 have 1 pgs currently mapped to them.
OSD(s) 24 are safe to destroy without reducing data durability.
Tue, Sep 22
Tue, Sep 22
brouberol added a comment to T438823: Medium Errors detected on cephosd1002:/dev/sdo.
I'm running the commands that would have been run by puppet manually:
ceph osd ok-to-stop osd.24
{"ok_to_stop":true,"osds":[ ...root@cephosd1002:~# while ! ceph osd safe-to-destroy osd.24 ; do sleep 10 ; done Error EBUSY: OSD(s) 24 have 158 pgs currently mapped to them. ... Error EBUSY: OSD(s) 24 have 158 pgs currently mapped to them. Error EBUSY: OSD(s) 24 have 157 pgs currently mapped to them. Error EBUSY: OSD(s) 24 have 157 pgs currently mapped to them. Error EBUSY: OSD(s) 24 have 156 pgs currently mapped to them. Error EBUSY: OSD(s) 24 have 155 pgs currently mapped to them. Error EBUSY: OSD(s) 24 have 154 pgs currently mapped to them. Error EBUSY: OSD(s) 24 have 153 pgs currently mapped to them. Error EBUSY: OSD(s) 24 have 152 pgs currently mapped to them. ...
brouberol updated subscribers of T438823: Medium Errors detected on cephosd1002:/dev/sdo.
brouberol added a comment to T438823: Medium Errors detected on cephosd1002:/dev/sdo.
Thanks to that new ceph-osd-locations, we can now map the osd.24 OSD to the c0/e23/s4 disk location
brouberol@cephosd1002:~$ sudo ceph-osd-locations | grep osd.24 | grep sdo OSD DEVICE ID_WWN PERC_WWN INTF MEDIUM SERIAL LOCATION ------ -------- ---------------- ---------------- ---- ------ ------------ ---------- osd.24 /dev/sdo 5000C500D9BB5F4B 5000C500D9BB5F48 SAS HDD ZR5BP856 c0/e23/s4
brouberol changed the status of T438823: Medium Errors detected on cephosd1002:/dev/sdo from Open to In Progress.
brouberol added a comment to T436270: Airflow testing instances made with airflow-devenv aren't able to send emails.
I think you can make this work by passing --set config.airflow.config.email.email_backend=airflow.utils.email.send_email_smtp. Can you try that?
brouberol added a comment to T438790: airflow-devenv creates wrong envoy when using --name.
So far, everything I've seen seems to be working and I can't reproduce, so I'll need a bit more details and help from you to find the issue.
brouberol added a comment to T438790: airflow-devenv creates wrong envoy when using --name.
The DAG processor running on the scheduler seems to be constructing the URLs correctly:
brouberol added a comment to T438790: airflow-devenv creates wrong envoy when using --name.
Could I have the full traceback? I'd need to know how the URL was constructed in the first place.
brouberol added a comment to T438790: airflow-devenv creates wrong envoy when using --name.
brouberol@deploy1003:~$ k get cm airflow-dev-kcvelaga-api-daily-kubernetes-pod-templates -oyaml > cm.yaml Type "help", "copyright", "credits" or "license" for more information. >>> import yaml >>> cm = yaml.safe_load(open('cm.yaml')) >>> env_list = yaml.safe_load(cm['data']['kubernetes_executor_default_pod_template.yaml'])['spec']['containers'][0]['env'] >>> for item in env_list: ... if item['name'] == 'ENVOY_SERVICE_NAME': ... print(item) ... {'name': 'ENVOY_SERVICE_NAME', 'value': 'airflow-dev-kcvelaga-api-daily-envoy'}
brouberol added a comment to T438790: airflow-devenv creates wrong envoy when using --name.
Ok, I'm trying to reproduce this in a devenv that was created via the --name parameter: dev-kcvelaga-api-daily.
brouberol changed the status of T438790: airflow-devenv creates wrong envoy when using --name from Open to In Progress.
brouberol added a comment to T426610: Follow up on multiple RAID / drive issues.
Seems like https://phabricator.wikimedia.org/T438744 is another occurence.
Thu, Sep 17
Thu, Sep 17
brouberol added a comment to T438354: Define the LDAP groups determining access management in the Fundraising growthbook project.
@phuedx: who should receive notifications from IDM when users attempt to self-enroll?
brouberol updated the task description for T438354: Define the LDAP groups determining access management in the Fundraising growthbook project.
brouberol edited parent tasks for T438351: Define the fundraising project in Growthbook, added: T438353: Define a permission attribution pipeline specific to the fundraisng Growthbook project; removed: T436940: [EPIC] Prepare GrowthBook for receiving experiment-related analytics events from donate.wikimedia.org.
brouberol updated subscribers of T438351: Define the fundraising project in Growthbook.
@phuedx do you have any idea/preference in terms of project name? The one project we have atm is called "Wikimedia". We could call this one "Fundraising" I guess?
brouberol added a comment to T436940: [EPIC] Prepare GrowthBook for receiving experiment-related analytics events from donate.wikimedia.org.
@phuedx sure, no problem. AFAICT, it would make sense for DPE SRE to:
- setup the PG database
- creating the Growthbook project
- creating the LDAP groups
- hooking the LDAP groups into IDM with the proper manager approval structure
- setting up the permission sync pipeline between LDAP group members and the growthbook project
brouberol added a comment to T438249: Matomo: Enable “for kept data above, also keep segmented reports” option.
Sure thing! Let's mark this as solved and reopen (or create a new task) if you need something else!
brouberol added a comment to T438249: Matomo: Enable “for kept data above, also keep segmented reports” option.
All done:
brouberol changed the status of T438249: Matomo: Enable “for kept data above, also keep segmented reports” option from Open to In Progress.
Wed, Sep 16
Wed, Sep 16
brouberol updated Other Assignee for T431697: Metrics Superset dashboard access for all WMF, removed: brouberol.
brouberol added a comment to T435775: Heterogeneous kafka-jumbo-eqiad rack placement.
Let's wait a few weeks, no problem!
Tue, Sep 15
Tue, Sep 15
brouberol added a comment to T437988: Dumps pods sometimes take more than 5 minutes to start.
www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-9rrdb:/$ ls -alh /mnt/dumpsdata/ total 4.0K drwxrwsr-x 4 root www-data 2 May 15 2025 . drwxr-xr-x 1 root root 4.0K Sep 10 18:14 .. drwxrwsr-x 14 www-data www-data 12 Jul 28 2025 otherdumps drwxrwsr-x 5 www-data www-data 3 Mar 19 2025 xmldatadumps
Ok, we're back to gid 33 owning the volume. Closing.
brouberol reopened T437988: Dumps pods sometimes take more than 5 minutes to start as "In Progress".
brouberol added a comment to T437988: Dumps pods sometimes take more than 5 minutes to start.
I'm re-opening this as I've observed the root volume dir owned by gid 900 again:
www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-9rrdb:/$ ls -alh /mnt/dumpsdata/ total 4.0K drwxrwsr-x 4 root 900 2 May 15 2025 . drwxr-xr-x 1 root root 4.0K Sep 10 18:14 .. drwxrwsr-x 14 www-data 900 12 Jul 28 2025 otherdumps drwxrwsr-x 5 www-data 900 3 Mar 19 2025 xmldatadumps
brouberol added a comment to T437988: Dumps pods sometimes take more than 5 minutes to start.
www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-9rrdb:/$ ls -alh /mnt/dumpsdata/ total 4.0K drwxrwsr-x 4 root www-data 2 May 15 2025 . drwxr-xr-x 1 root root 4.0K Sep 10 18:14 .. drwxrwsr-x 14 www-data www-data 12 Jul 28 2025 otherdumps drwxrwsr-x 5 www-data www-data 3 Mar 19 2025 xmldatadumps www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-9rrdb:/$ getent group 33 www-data:x:33:
We're back to normal, where the root volume is owned by gid=33
brouberol added a comment to T437988: Dumps pods sometimes take more than 5 minutes to start.
I'm calling this done!
brouberol added a comment to T437988: Dumps pods sometimes take more than 5 minutes to start.
Same for the monitor pods
brouberol@deploy1003:~$ k get pod -o json mediawiki-sql-xml-dumps-monitor | jq .spec.securityContext
{
"fsGroup": 33,
"fsGroupChangePolicy": "OnRootMismatch"
}mediawiki-sql-xml-dumps-monitor 0/4 Pending 0 0s mediawiki-sql-xml-dumps-monitor 0/4 Pending 0 0s mediawiki-sql-xml-dumps-monitor 0/4 ContainerCreating 0 0s mediawiki-sql-xml-dumps-monitor 0/4 ContainerCreating 0 4s mediawiki-sql-xml-dumps-monitor 2/4 Running 0 11s mediawiki-sql-xml-dumps-monitor 3/4 Running 0 11s mediawiki-sql-xml-dumps-monitor 4/4 Running 0 12s mediawiki-sql-xml-dumps-monitor 3/4 NotReady 0 44s mediawiki-sql-xml-dumps-monitor 0/4 Completed 0 45s mediawiki-sql-xml-dumps-monitor 0/4 Completed 0 46s mediawiki-sql-xml-dumps-monitor 0/4 Completed 0 46s mediawiki-sql-xml-dumps-monitor 0/4 Completed 0 47s mediawiki-sql-xml-dumps-monitor 0/4 Completed 0 47s mediawiki-sql-xml-dumps-monitor-sync 0/1 Pending 0 0s mediawiki-sql-xml-dumps-monitor-sync 0/1 Pending 0 0s mediawiki-sql-xml-dumps-monitor-sync 0/1 ContainerCreating 0 0s mediawiki-sql-xml-dumps-monitor-sync 0/1 ContainerCreating 0 1s mediawiki-sql-xml-dumps-monitor-sync 1/1 Running 0 2s mediawiki-sql-xml-dumps-monitor-sync 0/1 Completed 0 4s mediawiki-sql-xml-dumps-monitor-sync 0/1 Completed 0 5s mediawiki-sql-xml-dumps-monitor-sync 0/1 Completed 0 6s mediawiki-sql-xml-dumps-monitor-sync 0/1 Completed 0 7s mediawiki-sql-xml-dumps-monitor-sync 0/1 Completed 0 7s