Page MenuHomePhabricator

jbond (John Bond)
Staff Site Reliability Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Jan 7 2019, 1:06 PM (212 w, 1 d)
Availability
Available
IRC Nick
jbond
LDAP User
Jbond
MediaWiki User
JBond (WMF) [ Global Accounts ]

Recent Activity

Today

jbond committed rLPRIe025465e045d: add profile::idm::server::oidc_secret (authored by jbond).
add profile::idm::server::oidc_secret
Tue, Jan 31, 11:46 AM

Fri, Jan 27

jbond added a comment to T323717: Feature request: sre.hardware.upgrade-firmware should allow option to defer NIC firmware installation to next reboot.
iDrac shouldn't upgrade to 6.00.00.00 (breaks https mgmt access), cap at 5.10.30.00.

FYI its safe to update to the most recent idrac version now, can you update where ever this information is?

This is from https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation#Urgent_Firmware_Revision_Notices: ... is it safe to do 6.x now?

Fri, Jan 27, 2:36 PM · SRE, Infrastructure-Foundations, Traffic
jbond added a comment to T323717: Feature request: sre.hardware.upgrade-firmware should allow option to defer NIC firmware installation to next reboot.

Hi ssingh ,

Fri, Jan 27, 2:35 PM · SRE, Infrastructure-Foundations, Traffic
jbond added a comment to T323717: Feature request: sre.hardware.upgrade-firmware should allow option to defer NIC firmware installation to next reboot.
Fri, Jan 27, 2:32 PM · SRE, Infrastructure-Foundations, Traffic
jbond closed T313825: Add requestctl support to ferm as Resolved.
Fri, Jan 27, 12:23 PM · SRE, conftool
jbond closed T313825: Add requestctl support to ferm, a subtask of T305580: requestctl v1 improvements, as Resolved.
Fri, Jan 27, 12:23 PM · SRE, conftool

Thu, Jan 26

jbond created P43407 (An Untitled Masterwork).
Thu, Jan 26, 1:49 PM
jbond added a comment to T323717: Feature request: sre.hardware.upgrade-firmware should allow option to defer NIC firmware installation to next reboot.

@ssingh i have created a patch to defer reboots until all drivers have been uploaded. Are you able to let me know a host i can test on?

Thu, Jan 26, 11:13 AM · SRE, Infrastructure-Foundations, Traffic

Wed, Jan 25

jbond added a comment to T247517: Request creation of 'sre-sandbox' VPS project.
  • did the emails informing @herron that the machine was due to be deleted go out correctly
  • where they received by @herron (spam filter etc)
  • why where they not acted upon (possibly not enough notice)

I don't recall seeing any email warnings re: instance deletion, and don't see any archived with a cursory search of my mailboxes. But I can double check. What should the from/subject headers have been?

It should come from do_not_reply@wmcloud.org

Wed, Jan 25, 4:49 PM · cloud-services-team (Kanban), SRE, Cloud-VPS (Project-requests)

Tue, Jan 24

jbond triaged T327768: cfssl: export metricts from sql database as Medium priority.
Tue, Jan 24, 11:21 AM · Observability-Metrics, Infrastructure-Foundations, CFSSL-PKI
jbond created T327768: cfssl: export metricts from sql database.
Tue, Jan 24, 11:21 AM · Observability-Metrics, Infrastructure-Foundations, CFSSL-PKI

Mon, Jan 23

jbond closed T325004: Requesting access to analytics-privatedata-users & analytics-product-users for Hxi-ctr as Resolved.

I have gone ahead and merged the changes to rename this account, please reopen if you have have any issues

Mon, Jan 23, 4:53 PM · SRE, SRE-Access-Requests
jbond closed T325004: Requesting access to analytics-privatedata-users & analytics-product-users for Hxi-ctr, a subtask of T324349: Onboarding for Hua, as Resolved.
Mon, Jan 23, 4:53 PM · Product-Analytics (Kanban)
jbond closed T325004: Requesting access to analytics-privatedata-users & analytics-product-users for Hxi-ctr, a subtask of T325857: Requesting Kerberos identity for Hxi-ctr, as Resolved.
Mon, Jan 23, 4:53 PM · Data-Engineering-Planning
jbond added a comment to T274230: Create a generic network performance profile.

@BCornwall thanks for reviving this. i think that this ultimately stalled as there was a questions of wether it would be usefull. from memory the main questions where

  • is there a set of universal sysctl settings that are useful across the board
  • does it make senses to try have theses settings configured in a base profile or is it better to have the settings in the application specific profile
Mon, Jan 23, 1:19 PM · Traffic-Icebox, Patch-For-Review, User-MoritzMuehlenhoff, SRE
jbond added a comment to T323484: Fine tune the SSHd config of the restricted bastion for better performances.

150:100:150 seems a bit more logical to me although i think they have the same end result so not a big deal

Mon, Jan 23, 11:34 AM · cloud-services-team (FY2022/2023-Q3), Infrastructure-Foundations
Jelto awarded T320390: migrate gitlab away from the CAS protocol a Like token.
Mon, Jan 23, 11:09 AM · GitLab (Auth & Access), serviceops-collab, CAS-SSO, Infrastructure-Foundations, SRE
jbond updated subscribers of T320390: migrate gitlab away from the CAS protocol.

fyi we now have OIDC support in production, currently been tested by @SLyngshede-WMF

Mon, Jan 23, 11:01 AM · GitLab (Auth & Access), serviceops-collab, CAS-SSO, Infrastructure-Foundations, SRE

Fri, Jan 13

jbond committed rLPRI19a8efc99c54: add oidc test data for idp-test (authored by jbond).
add oidc test data for idp-test
Fri, Jan 13, 1:56 PM

Wed, Jan 11

jbond added a comment to T277011: Automated removal of obsolete kernels.

With bullseye apt even does this automatically

wonder if we could backport this to buster, ignore stretch and call it done?

With Buster being around for nine months (and many migrations happening much earlier) we could even focus on building this based on apt/bullseye entirely. After all, the problem has been around for quite a while and still having it for some more months on a small subset of the fleet seems fine. Backporting would be a little tricky since it's a feature in apt itself and there are no backports for dpkg.

Wed, Jan 11, 3:14 PM · Infrastructure-Foundations, User-MoritzMuehlenhoff, SRE
jbond added a comment to T277011: Automated removal of obsolete kernels.

With bullseye apt even does this automatically

wonder if we could backport this to buster, ignore stretch and call it done?

Wed, Jan 11, 2:58 PM · Infrastructure-Foundations, User-MoritzMuehlenhoff, SRE
jbond added a comment to T326316: Misconfigured proxies on I/F hosts.

Both of theses look like manual user actions possibly would be fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/771568/ been rolled out globally

Only seeing that set of patches now, I left a comment on:
https://gerrit.wikimedia.org/r/c/operations/puppet/+/771411/8#message-232fee2f2106f145e77d72a41219ced4f1b8d120

I don't see any red-flag on rolling it out globally, if there are no blockers, let's draft an email for sre-at-large@ to plan a global roll out?

Wed, Jan 11, 12:46 PM · Infrastructure-Foundations

Tue, Jan 10

jbond added a comment to T326316: Misconfigured proxies on I/F hosts.

After a chat with @MoritzMuehlenhoff they are from the pbuilder environments

I took a look at this and the pbuilder environments already go direct however there is a systemd timer debian-weekly-rebuild.service that seems to be responsible for this traffic, CR sent.

Tue, Jan 10, 3:22 PM · Infrastructure-Foundations
jbond added a comment to T324670: Create partman recipe for cephosd servers.

@BTullis Seems you have allready gone through most of the issues i went through. Some addtional things to mention

Tue, Jan 10, 1:32 PM · Patch-For-Review, Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)), Data-Engineering
jbond committed rLPRIb8b0943c6a1c: add wikikube_staging_front_proxy-key.pem (authored by jbond).
add wikikube_staging_front_proxy-key.pem
Tue, Jan 10, 10:57 AM
jbond committed rLPRIf725e646d462: add wikikube_staging-key.pem (authored by jbond).
add wikikube_staging-key.pem
Tue, Jan 10, 10:54 AM
jbond committed rLPRI5120e741589d: add wikikube_front_proxy-key.pem (authored by jbond).
add wikikube_front_proxy-key.pem
Tue, Jan 10, 10:54 AM
jbond committed rLPRI5a363aaf3de6: add wikikube-key.pem (authored by jbond).
add wikikube-key.pem
Tue, Jan 10, 10:51 AM

Thu, Jan 5

jbond added a comment to T239464: Create a generic get_address puppet function.

For now i have added the dnsquery dns_lookup function

Thu, Jan 5, 2:49 PM · User-jbond
jbond added a comment to T233940: CLI tools for CAS administration.

also possibly useful: https://apereo.github.io/cas/6.5.x/installation/Configuring-Commandline-Shell.html

Thu, Jan 5, 2:47 PM · Infrastructure-Foundations, CAS-SSO, User-jbond, SRE
jbond created P42863 (An Untitled Masterwork).
Thu, Jan 5, 2:35 PM

Tue, Jan 3

jbond added a comment to P42722 puppet yaml fail.

this seems to be fixed in puppet 6.0.0 so wont get fixed in the 5 branch

Tue, Jan 3, 4:34 PM

Dec 21 2022

jbond closed T324606: cookbook sre.hardware.upgrade-firmware nic firmware comparison mismatch as Resolved.

Full command for what exactly? The downgrade issue is fixed with the commented info:

i meant the full cookbook command that you ran however...

Dec 21 2022, 12:17 PM · Infrastructure-Foundations, DC-Ops

Dec 19 2022

jbond added a comment to T324606: cookbook sre.hardware.upgrade-firmware nic firmware comparison mismatch.

what's the full command you are using robh?

Dec 19 2022, 12:00 PM · Infrastructure-Foundations, DC-Ops

Dec 16 2022

jbond created P42723 puppet yaml fail.
Dec 16 2022, 7:06 PM
jbond added a comment to P42722 puppet yaml fail.

Notice: Compiled catalog for jbond-laptop.home.arpa in environment production in 0.02 seconds
Notice: Applied catalog in 0.01 seconds

Dec 16 2022, 6:33 PM
jbond created P42722 puppet yaml fail.
Dec 16 2022, 6:33 PM
jbond created P42721 (An Untitled Masterwork).
Dec 16 2022, 2:42 PM
jbond added a comment to T300819: Gerritlab: Stacked pull requests.

leaving this here as it may also be useful https://olivernguyen.io/w/sapling/. disclaimer i have not tested myself

Dec 16 2022, 10:28 AM · GitLab (Misc), Release-Engineering-Team (Seen)
jbond renamed T300819: Gerritlab: Stacked pull requests from Gerritlab to Gerritlab: Stacked pull requests.
Dec 16 2022, 10:27 AM · GitLab (Misc), Release-Engineering-Team (Seen)

Dec 15 2022

jbond committed rLPRI547ef7e6ba32: add back old pass for diffs (authored by jbond).
add back old pass for diffs
Dec 15 2022, 6:05 PM
jbond committed rLPRI11db9abd8ab2: move peeringdb pass to role (authored by jbond).
move peeringdb pass to role
Dec 15 2022, 6:03 PM
jbond committed rLPRIa53134173e44: add new structure (authored by jbond).
add new structure
Dec 15 2022, 4:38 PM
jbond added a comment to T321783: Setup an initial bookworm host pair with Puppetdb 7.

upstream bug re java 11

Dec 15 2022, 3:18 PM · Patch-For-Review, Infrastructure-Foundations, SRE

Dec 14 2022

jbond committed rCCKBf9fe4544ad14: sre.hardware.upgrade-firmware: ensure folderes are group writable (authored by jbond).
sre.hardware.upgrade-firmware: ensure folderes are group writable
Dec 14 2022, 3:31 PM
jbond committed rCCKB3a6e6a3289a9: sre.dns.netbox: add call to sre.puppet.sync-netbox-hiera (authored by jbond).
sre.dns.netbox: add call to sre.puppet.sync-netbox-hiera
Dec 14 2022, 3:31 PM
jbond committed rCCKBb2c5af3c9717: upgrade-firmware: small fix to ensure files get saved in the correct path (authored by jbond).
upgrade-firmware: small fix to ensure files get saved in the correct path
Dec 14 2022, 3:31 PM
jbond committed rCCKB6d7898b5dfad: sre.hardware.upgrade-firmware: ensure we disable hostcheck (authored by jbond).
sre.hardware.upgrade-firmware: ensure we disable hostcheck
Dec 14 2022, 3:31 PM
jbond committed rCCKBa4a33e79d96d: sre.hardware.upgrade-firmware: add support for storage devices (authored by jbond).
sre.hardware.upgrade-firmware: add support for storage devices
Dec 14 2022, 3:31 PM
jbond committed rCCKB7c72a2036b2d: sre.hardware.upgrade-firmware: Fix usage (authored by Muehlenhoff).
sre.hardware.upgrade-firmware: Fix usage
Dec 14 2022, 3:31 PM
jbond committed rCCKB926a1aa7d394: sre.hardware.upgrade-firmware: Add support for driver updates (authored by jbond).
sre.hardware.upgrade-firmware: Add support for driver updates
Dec 14 2022, 3:31 PM
jbond committed rCCKB4b2069ffec4f: sre.hardware.upgrade-firmware: add a cache for firmware answers (authored by jbond).
sre.hardware.upgrade-firmware: add a cache for firmware answers
Dec 14 2022, 3:31 PM
jbond committed rCCKBd99ed1c65df8: sre.hardware.upgrade-firmware: use packaging.version.Version (authored by jbond).
sre.hardware.upgrade-firmware: use packaging.version.Version
Dec 14 2022, 3:31 PM
jbond committed rCCKB372c990764ec: sre.hardware.upgrade-firmware: ensure we create all directories (authored by jbond).
sre.hardware.upgrade-firmware: ensure we create all directories
Dec 14 2022, 3:31 PM
jbond committed rCCKBe8568a1439ab: sre.hosts.reboot-cluster: fix argument parsing (authored by jbond).
sre.hosts.reboot-cluster: fix argument parsing
Dec 14 2022, 3:31 PM
jbond committed rCCKBb255a28fc7c0: sre.puppet.sync-netbox-hiera: update prefix for common data (authored by jbond).
sre.puppet.sync-netbox-hiera: update prefix for common data
Dec 14 2022, 3:31 PM
jbond committed rCCKBa902a7eef76c: sre.hardware.upgrade-firmware: correct passed parameter (authored by jbond).
sre.hardware.upgrade-firmware: correct passed parameter
Dec 14 2022, 3:31 PM
jbond committed rCCKB70ce47d828aa: sre.hardware.upgrade-firmware: Present user with a list of current files (authored by jbond).
sre.hardware.upgrade-firmware: Present user with a list of current files
Dec 14 2022, 3:31 PM
jbond committed rCCKB0d13a9b6010d: sre.hardware.firmware-upgrade: fix typo (authored by jbond).
sre.hardware.firmware-upgrade: fix typo
Dec 14 2022, 3:31 PM
jbond committed rCCKB52d73249ba42: sre.hardware.upgrade-firmware: drop firmware-file flag (authored by jbond).
sre.hardware.upgrade-firmware: drop firmware-file flag
Dec 14 2022, 3:31 PM
jbond committed rCCKBfccbc5a1cff3: sre.hardware.upgrade-firmware: cast firmware_store_dir to Path (authored by jbond).
sre.hardware.upgrade-firmware: cast firmware_store_dir to Path
Dec 14 2022, 3:31 PM
jbond committed rCCKB65829451c3c6: sre.hardware.upgrade-firmware: create subfolderes for firmware type (authored by jbond).
sre.hardware.upgrade-firmware: create subfolderes for firmware type
Dec 14 2022, 3:31 PM
jbond committed rCCKB80f66ecdc5c3: sre.hardware.upgrade-firmware: read firmware_store from config (authored by jbond).
sre.hardware.upgrade-firmware: read firmware_store from config
Dec 14 2022, 3:31 PM
jbond committed rCCKBb81831acd523: sre.puppet.sync-netbox-hiera: check flag should be bool (authored by jbond).
sre.puppet.sync-netbox-hiera: check flag should be bool
Dec 14 2022, 3:31 PM
jbond committed rCCKB5df8946da56f: sre.puppet.sync-netbox-hiera: Add return check mode (authored by jbond).
sre.puppet.sync-netbox-hiera: Add return check mode
Dec 14 2022, 3:31 PM
jbond committed rCCKB535b1f3782e1: sre.hardware.firmware-upgrade: power on server for firmware updates (authored by jbond).
sre.hardware.firmware-upgrade: power on server for firmware updates
Dec 14 2022, 3:31 PM
jbond committed rCCKBbe19a00561c8: sre.hardware.upgrade-firmware: Add new flag (authored by jbond).
sre.hardware.upgrade-firmware: Add new flag
Dec 14 2022, 3:31 PM
jbond committed rCCKB18bf85574a93: sre.hardware.upgrade-firmware: If the system is new reboot with redfish (authored by jbond).
sre.hardware.upgrade-firmware: If the system is new reboot with redfish
Dec 14 2022, 3:30 PM
jbond committed rCCKB1e02d9458844: cookbook sre.puppet.sync-netbox-hiera: Fix exception handling (authored by jbond).
cookbook sre.puppet.sync-netbox-hiera: Fix exception handling
Dec 14 2022, 3:30 PM
jbond committed rCCKB742cf34bd8f4: sre.hardware.firmware-upgrade: create new cookbook for updating idrac and bios (authored by jbond).
sre.hardware.firmware-upgrade: create new cookbook for updating idrac and bios
Dec 14 2022, 3:30 PM
jbond committed rCCKB7f14c62924b9: SREBaseClass: Allow overriding actions (authored by jbond).
SREBaseClass: Allow overriding actions
Dec 14 2022, 3:30 PM
jbond committed rCCKB9287e3f5bff9: sre.ganeti.makevm: Clear the DNS cache before adding the ganeti instance (authored by jbond).
sre.ganeti.makevm: Clear the DNS cache before adding the ganeti instance
Dec 14 2022, 3:30 PM
jbond committed rCCKB0d682762e52b: sre.puppet.sync-netbox-hiera: Cookbook for syncing netbox puppet data (authored by jbond).
sre.puppet.sync-netbox-hiera: Cookbook for syncing netbox puppet data
Dec 14 2022, 3:30 PM
jbond committed rCCKB9141b53ff5a7: cookbook sre.hosts.decommision: dont sleep in dry-run mode (authored by jbond).
cookbook sre.hosts.decommision: dont sleep in dry-run mode
Dec 14 2022, 3:29 PM
jbond committed rCCKB4226e65c2892: cookbook sre.idm.u2f: add cookbook to enable/disable u2f (authored by jbond).
cookbook sre.idm.u2f: add cookbook to enable/disable u2f
Dec 14 2022, 3:29 PM
jbond committed rCCKBf1976646e935: cookbook sre.dns.wipe-cache: cookbook to clear stale DNS entries (authored by jbond).
cookbook sre.dns.wipe-cache: cookbook to clear stale DNS entries
Dec 14 2022, 3:29 PM
jbond committed rCCKB0ec27595055d: sre: add conftool aware SREBatchRunnerBase (authored by jbond).
sre: add conftool aware SREBatchRunnerBase
Dec 14 2022, 3:29 PM
jbond committed rCCKB26f5d423323c: cookbooks.sre: update to use correct icinga_hosts instance (authored by jbond).
cookbooks.sre: update to use correct icinga_hosts instance
Dec 14 2022, 3:29 PM
jbond committed rCCKB401a8b45598b: cookbooks sre: update run_scripts to accept a list of scripts not functions (authored by jbond).
cookbooks sre: update run_scripts to accept a list of scripts not functions
Dec 14 2022, 3:29 PM
jbond committed rCCKB99d8894b7614: cookbook sre: update SREBatchBase/SREBatchRunnerBase with minor fixes (authored by jbond).
cookbook sre: update SREBatchBase/SREBatchRunnerBase with minor fixes
Dec 14 2022, 3:29 PM
jbond committed rCCKB309963f371e3: sre.misc-clusters.thumbor: create batch action cook book for thumbor (authored by jbond).
sre.misc-clusters.thumbor: create batch action cook book for thumbor
Dec 14 2022, 3:29 PM
jbond committed rCCKB1864fc70c770: sre: convert the generic reboot functions to the cookbook class API (authored by jbond).
sre: convert the generic reboot functions to the cookbook class API
Dec 14 2022, 3:29 PM
jbond committed rCCKB9e5e7f005d5f: logout cookbook: Quote CN and UID (authored by Muehlenhoff).
logout cookbook: Quote CN and UID
Dec 14 2022, 3:28 PM
jbond committed rCCKBddf36a5728c4: sre.idm.cookbooks: fix typo uid -> cn (authored by jbond).
sre.idm.cookbooks: fix typo uid -> cn
Dec 14 2022, 3:28 PM
jbond committed rCCKB985eb51a6ce4: sre.idm.logout: create cookbook to logout users (authored by jbond).
sre.idm.logout: create cookbook to logout users
Dec 14 2022, 3:28 PM
jbond committed rCCKB0906563d2425: sre.puppet.renew-cert: correct typo allow_dns_alt_names (authored by jbond).
sre.puppet.renew-cert: correct typo allow_dns_alt_names
Dec 14 2022, 3:28 PM
jbond committed rCCKB3e9c66bf67d0: gitignore: ignore vi swap/tmp files (authored by jbond).
gitignore: ignore vi swap/tmp files
Dec 14 2022, 3:27 PM
jbond committed rCCKB78630334de1b: sre.hosts.decommission: fix pylint error (authored by jbond).
sre.hosts.decommission: fix pylint error
Dec 14 2022, 3:27 PM
jbond committed rCCKBf98020464223: sre.puppet.renew-cert: add support for allow_alt_names (authored by jbond).
sre.puppet.renew-cert: add support for allow_alt_names
Dec 14 2022, 3:27 PM
jbond committed rCCKBf4480cbe712b: sre.puppet.renew-cert: convert to class API (authored by jbond).
sre.puppet.renew-cert: convert to class API
Dec 14 2022, 3:27 PM
jbond committed rCCKBf59d0b7b0c82: sre.puppet.renew-cert: fix arguments for puppet_master commands (authored by jbond).
sre.puppet.renew-cert: fix arguments for puppet_master commands
Dec 14 2022, 3:27 PM
jbond committed rCCKBcd940ea8b090: sre.pdus.rotate-password: fix TypeError: 'tuple' object does not support item… (authored by jbond).
sre.pdus.rotate-password: fix TypeError: 'tuple' object does not support item…
Dec 14 2022, 3:26 PM
jbond committed rCCKB5e286a8cfa14: cookbook sre.pdu: Fix reboot logic and other minor fixes (authored by jbond).
cookbook sre.pdu: Fix reboot logic and other minor fixes
Dec 14 2022, 3:26 PM
jbond committed rCCKBa3c00f5b3fad: cookbook sre.puppet.renew-cert: add cookbook to renew a puppet cert (authored by jbond).
cookbook sre.puppet.renew-cert: add cookbook to renew a puppet cert
Dec 14 2022, 3:26 PM
jbond committed rCCKB5524ee82466d: cookbook sre.pdus: add reboot script (authored by jbond).
cookbook sre.pdus: add reboot script
Dec 14 2022, 3:25 PM
jbond committed rCCKB9bedf2ec6124: cookbooks sre.pdus: add uptime cookbook (authored by jbond).
cookbooks sre.pdus: add uptime cookbook
Dec 14 2022, 3:25 PM
jbond committed rCCKBda38ddea9815: cookbook sre.hosts.rotate-pdu-password: rename (authored by jbond).
cookbook sre.hosts.rotate-pdu-password: rename
Dec 14 2022, 3:25 PM
jbond committed rCCKB5a7f308d7d1b: cookbooks sre.hosts.rotate-pdu-password: reset SNMP (authored by jbond).
cookbooks sre.hosts.rotate-pdu-password: reset SNMP
Dec 14 2022, 3:25 PM
jbond committed rCCKB3a6dcc29c349: sre.pdus.rotate-password: split generic functions out to __init__.py (authored by jbond).
sre.pdus.rotate-password: split generic functions out to __init__.py
Dec 14 2022, 3:25 PM
jbond committed rCCKB7fa8ea13b216: cookbooks sre.hosts.rotate-pdu-password: refactor (authored by jbond).
cookbooks sre.hosts.rotate-pdu-password: refactor
Dec 14 2022, 3:25 PM