Page MenuHomePhabricator

jbond (John Bond)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Jan 7 2019, 1:06 PM (73 w, 4 d)
Availability
Available
IRC Nick
jbond42
LDAP User
Jbond
MediaWiki User
JBond (WMF) [ Global Accounts ]

Recent Activity

Today

jbond added a comment to T254480: Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file.

I agree. Example: the first attempt at ./modules/dumps/templates/web/fetches/analytics/job/rsync_script.sh.erb was here https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594773/2/modules/dumps/manifests/web/fetches/analytics/job.pp so you can see why erb was the natural approach in the end. Another file in there (modules/snapshot/templates/set_dump_dirs.sh.erb) literally just sets some vars to be picked up by the 'real' shell scripts that use it; I favour allowing scripts like these and strongly encouraging folks with long scripts as erb files to refactor them accordingly.

Fri, Jun 5, 9:05 PM · Release-Engineering-Team (CI & Testing services), Continuous-Integration-Config, Patch-For-Review, User-jbond, Puppet, Operations
jbond added a comment to T254480: Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file.

@CDanis yeah, sorry I noticed that after replying.
Totally agree with this approach, would be nice to have the CI check but we have a bunch of few liners that would become just more complex and probably not gaining a lot, see for example the first part of this list:

for scripts that are just one line i think it reasonable to just build the content in puppet, this is a bit of a cheat but for simple one liners i think its a fine compromise. I took a quick look through the files other then the oneliners, some of them don't actually need to be templates as there is no erb and a few others look like they are already config files so we could possibly just remove the shebang . however there are of course some that may be a bit awkward to fit into this model. Perhaps we should just whitelist them. It would at least mean going forward we don't have more violations added.

Fri, Jun 5, 8:38 PM · Release-Engineering-Team (CI & Testing services), Continuous-Integration-Config, Patch-For-Review, User-jbond, Puppet, Operations
jbond added a comment to T254480: Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file.

Another option is we could just ban templated scripts and add CI to reject any erb file with a shebang in it. This would mean updating any current templated scripts to read a new templated config. i.e. move all the dynamic bits to a template

Fri, Jun 5, 3:37 PM · Release-Engineering-Team (CI & Testing services), Continuous-Integration-Config, Patch-For-Review, User-jbond, Puppet, Operations
jbond added a comment to T254480: Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file.

Originally i wanted to do this checking in the Rake checks as that feels like the right place for them. however i cant think of a reasonable way to compile the erb files as we dont know the values of variables that would ordinarily be bound by the parent Puppet manifest. And different variables/hosts/roles etc could produce different outputs. As such im now leaning more to putting this in PCC, however i wonder if this should be in standard and in the main PCC code or if we should just have some helper scripts in utils. Regardless ill probably work on the latter to see how it looks but ideas welcome

Fri, Jun 5, 3:23 PM · Release-Engineering-Team (CI & Testing services), Continuous-Integration-Config, Patch-For-Review, User-jbond, Puppet, Operations
jbond added a comment to T254491: Puppet labs/private.git data loss incident affecting some projects.

i have made a first pass at the incident report. The main sections which could use some more input are

  • impact
  • detection
  • documentation
  • actionable
Fri, Jun 5, 1:03 PM · Puppet, cloud-services-team (Kanban), Cloud-VPS

Yesterday

jbond added a comment to T254480: Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file.

this is a bit better

Thu, Jun 4, 6:47 PM · Release-Engineering-Team (CI & Testing services), Continuous-Integration-Config, Patch-For-Review, User-jbond, Puppet, Operations
jbond added a comment to T254480: Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file.

quick glance suggest most are SC2086 & SC2006

Thu, Jun 4, 5:58 PM · Release-Engineering-Team (CI & Testing services), Continuous-Integration-Config, Patch-For-Review, User-jbond, Puppet, Operations
jbond added a comment to T254480: Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file.

This is a great idea, i think we may be able to do it in a rake task by adding something to task gen, the biggest issues with adding stuff like this is fixing all of the current issues. the wmf_style checks dose avoid this by only alerting on new errors but i have not looked at it. We should also add shellcheck for sh files in general

Thu, Jun 4, 5:56 PM · Release-Engineering-Team (CI & Testing services), Continuous-Integration-Config, Patch-For-Review, User-jbond, Puppet, Operations
jbond triaged T254480: Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file as Medium priority.
Thu, Jun 4, 4:44 PM · Release-Engineering-Team (CI & Testing services), Continuous-Integration-Config, Patch-For-Review, User-jbond, Puppet, Operations
jbond committed rLPRI4f2041f4f874: whitespace change to test puppet-merge (authored by jbond).
whitespace change to test puppet-merge
Thu, Jun 4, 11:08 AM

Wed, Jun 3

jbond added a comment to T233933: Replicated ticket registry.

We could resolve this by having mcrouter talk directly to memcache in the other DC however this requires 1.5.13 which is not currently in buster.

1.6.6 is available in testing and seems to build correctly on buster.

Wed, Jun 3, 3:48 PM · CAS-SSO, Patch-For-Review, User-jbond, Operations
jbond updated subscribers of T233933: Replicated ticket registry.

I have configured memcache and mcrouter for CAS however there is currently an error. If CAS talks directly to memcache then all works fine. however when CAS talks to mcrouter we get and amplification storm before a timeout is sent to CAS. After speaking with @elukey we think the following may be happening

Wed, Jun 3, 3:20 PM · CAS-SSO, Patch-For-Review, User-jbond, Operations
jbond added a comment to T244278: Document 2020-02-04 kartotherian incident.

Is this a duplicate of https://phabricator.wikimedia.org/T251340 ?

Wed, Jun 3, 12:04 PM · SRE-OnFire-Incident-Docs, Patch-For-Review, Operations
jbond moved T251340: 20200204-maps from In-review & unassigned to In-review & reviewer assigned on the SRE-OnFire-Incident-Docs board.
Wed, Jun 3, 11:49 AM · SRE-OnFire-Incident-Docs
jbond claimed T251340: 20200204-maps.
Wed, Jun 3, 11:48 AM · SRE-OnFire-Incident-Docs
jbond added a comment to T251340: 20200204-maps.

@CDanis you still have the example text for the "Where did we get lucky?" and "Links to relevant documentation" sections. Also still need tasks for

Wed, Jun 3, 11:48 AM · SRE-OnFire-Incident-Docs
jbond moved T252402: 20200501-vc-link-failure from In-review & unassigned to Finished on the SRE-OnFire-Incident-Docs board.
Wed, Jun 3, 11:40 AM · SRE-OnFire-Incident-Docs
jbond added a comment to T252402: 20200501-vc-link-failure.

Reviewed this looks good to publish, what is the procedure for that @CDanis ?

Wed, Jun 3, 11:39 AM · SRE-OnFire-Incident-Docs
jbond claimed T252402: 20200501-vc-link-failure.
Wed, Jun 3, 11:37 AM · SRE-OnFire-Incident-Docs
jbond claimed T251333: 20200207-mediawiki API down.
Wed, Jun 3, 11:36 AM · SRE-OnFire-Incident-Docs

Tue, Jun 2

jbond moved T247956: Ensure hiera only has profile:: qualified or global hiera keys from Unsorted 💣 to Friday tasks on the User-jbond board.
Tue, Jun 2, 2:40 PM · User-jbond, Patch-For-Review, User-Joe, Puppet, Operations
jbond moved T254249: Refactor puppet-merge from Unsorted 💣 to Patch for Review on the User-jbond board.
Tue, Jun 2, 2:40 PM · Patch-For-Review, User-jbond, Puppet
jbond triaged T254249: Refactor puppet-merge as Medium priority.
Tue, Jun 2, 2:37 PM · Patch-For-Review, User-jbond, Puppet
jbond moved T229397: Puppet: get row/rack info from Netbox from Watching 👀 to Patch for Review on the User-jbond board.
Tue, Jun 2, 2:30 PM · User-crusnov, User-jbond, Patch-For-Review, Puppet, Operations
jbond added a project to T247956: Ensure hiera only has profile:: qualified or global hiera keys: User-jbond.
Tue, Jun 2, 2:29 PM · User-jbond, Patch-For-Review, User-Joe, Puppet, Operations
jbond moved T251493: Updated java security policy in OpenJDK 8 u252 from Unsorted 💣 to Patch for Review on the User-jbond board.
Tue, Jun 2, 2:27 PM · User-jbond, User-MoritzMuehlenhoff, Patch-For-Review, Operations
jbond added a project to T251493: Updated java security policy in OpenJDK 8 u252: User-jbond.
Tue, Jun 2, 2:27 PM · User-jbond, User-MoritzMuehlenhoff, Patch-For-Review, Operations
jbond moved T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK) from Unsorted 💣 to Friday tasks on the User-jbond board.
Tue, Jun 2, 2:24 PM · User-jbond, netbox
jbond moved T253632: update profile::waf::apache2::administrative to use the new abuse_networks hiera key from Unsorted 💣 to Blocked 🚧 on the User-jbond board.
Tue, Jun 2, 2:24 PM · User-jbond, Operations
jbond moved T254248: Upgrade puppet to use hiera version 5 from Unsorted 💣 to Patch for Review on the User-jbond board.
Tue, Jun 2, 2:23 PM · Patch-For-Review, User-jbond, Puppet
jbond triaged T254248: Upgrade puppet to use hiera version 5 as Medium priority.
Tue, Jun 2, 2:23 PM · Patch-For-Review, User-jbond, Puppet
jbond moved T239334: Python3 style guide from Friday tasks to Patch for Review on the User-jbond board.
Tue, Jun 2, 2:20 PM · Patch-For-Review, User-ArielGlenn, User-jbond, Operations, Puppet
jbond moved T233933: Replicated ticket registry from Back Burner 🏛️ to Active 🚁 on the User-jbond board.
Tue, Jun 2, 2:18 PM · CAS-SSO, Patch-For-Review, User-jbond, Operations
jbond moved T233931: Cross data center setup for CAS from Back Burner 🏛️ to Active 🚁 on the User-jbond board.
Tue, Jun 2, 2:17 PM · CAS-SSO, User-jbond, Operations
jbond moved T239323: Investigate how automated tasks can authenticate against CAS from Active 🚁 to Patch for Review on the User-jbond board.
Tue, Jun 2, 2:17 PM · Patch-For-Review, CAS-SSO, User-jbond, Operations
jbond moved T236277: Extend Puppet CA Expiry date from Active 🚁 to Watching 👀 on the User-jbond board.
Tue, Jun 2, 2:17 PM · Patch-For-Review, User-jbond, Puppet, Operations
jbond moved T251104: puppet-merge: answering no to merging labs-private prevents puppet-merge from pushing to all puppet masters from Friday tasks to Patch for Review on the User-jbond board.
Tue, Jun 2, 2:14 PM · Puppet, User-jbond

Mon, Jun 1

jbond updated the task description for T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK).
Mon, Jun 1, 11:46 AM · User-jbond, netbox
jbond closed T245771: Adjust onboarding/offboarding logic to accommodate changes to #security (now acl*security) as Resolved.

Resolve as per Moritz comment

Mon, Jun 1, 10:57 AM · User-jbond, Security, Phabricator, Security-Team, Operations
jbond updated the task description for T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK).
Mon, Jun 1, 9:49 AM · User-jbond, netbox
jbond moved T233933: Replicated ticket registry from blocked to in progress on the CAS-SSO board.
Mon, Jun 1, 8:50 AM · CAS-SSO, Patch-For-Review, User-jbond, Operations

Fri, May 29

jbond updated the task description for T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK).
Fri, May 29, 12:16 PM · User-jbond, netbox
jbond added a comment to T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK).

when i checked stat1008 already had a AAAA record, not sure if someone fixed it or some issue in the script?

Fri, May 29, 12:07 PM · User-jbond, netbox
jbond updated the task description for T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK).
Fri, May 29, 12:06 PM · User-jbond, netbox
jbond added a comment to T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK).

Currently mongodb is not listening on IPv6 however[[ https://phabricator.wikimedia.org/T180761 | mongodb is going away ]] so we should wait until that work has been completed

Fri, May 29, 11:49 AM · User-jbond, netbox
jbond updated the task description for T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK).
Fri, May 29, 11:44 AM · User-jbond, netbox
jbond created T253986: update bacula-sd config so that it listens on IPv6.
Fri, May 29, 11:23 AM · netbox
jbond updated the task description for T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK).
Fri, May 29, 10:19 AM · User-jbond, netbox
jbond closed T233947: CAS build as a deb, a subtask of T233921: Further steps for CAS/web SSO, as Resolved.
Fri, May 29, 8:39 AM · CAS-SSO, Security-Team, User-jbond, Operations
jbond closed T233947: CAS build as a deb as Resolved.

Puppet is now deployed as a deb.

Fri, May 29, 8:39 AM · CAS-SSO, User-jbond, Operations

Thu, May 28

jbond added a comment to T251574: prepare mcrouter package for Debian Buster.

Awesome thanks

Thu, May 28, 8:48 AM · Patch-For-Review, serviceops, Packaging

Wed, May 27

jbond added a project to T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK): User-jbond.
Wed, May 27, 5:00 PM · User-jbond, netbox
jbond added a comment to T252890: scrape ripe atlas data for a few anchors at other large networks.

I think I'm leaning towards a few stable anchors in similar geographic locations to our PoPs. Maybe also a few root servers as well even though they're less apples-to-apples.

Wed, May 27, 12:55 PM · netops, Operations
jbond closed T251574: prepare mcrouter package for Debian Buster, a subtask of T213089: Upgrade memcached for Debian Stretch/Buster, as Resolved.
Wed, May 27, 12:31 PM · Patch-For-Review, User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
jbond closed T251574: prepare mcrouter package for Debian Buster, a subtask of T251294: Upgrade cloud-vps control plane to Debian Buster, as Resolved.
Wed, May 27, 12:31 PM · Patch-For-Review, cloud-services-team (Kanban)
jbond closed T251574: prepare mcrouter package for Debian Buster as Resolved.
Wed, May 27, 12:31 PM · Patch-For-Review, serviceops, Packaging
jbond added a comment to T251574: prepare mcrouter package for Debian Buster.

mcrouter 0.4.1 is now available in buster-wikimedia

Wed, May 27, 12:30 PM · Patch-For-Review, serviceops, Packaging

Tue, May 26

jbond edited P11304 sidecar.pp.
Tue, May 26, 3:45 PM
jbond edited P11304 sidecar.pp.
Tue, May 26, 3:36 PM
jbond created P11304 sidecar.pp.
Tue, May 26, 3:34 PM
chasemp awarded T253632: update profile::waf::apache2::administrative to use the new abuse_networks hiera key a Yellow Medal token.
Tue, May 26, 2:28 PM · User-jbond, Operations
jbond claimed T233935: Icinga Monitoring for CAS.
Tue, May 26, 1:56 PM · observability, CAS-SSO, User-jbond, Operations
jbond moved T233935: Icinga Monitoring for CAS from Backlog to in progress on the CAS-SSO board.
Tue, May 26, 1:55 PM · observability, CAS-SSO, User-jbond, Operations
jbond closed Restricted Task, a subtask of T233921: Further steps for CAS/web SSO, as Resolved.
Tue, May 26, 1:54 PM · CAS-SSO, Security-Team, User-jbond, Operations
jbond triaged T253632: update profile::waf::apache2::administrative to use the new abuse_networks hiera key as Medium priority.
Tue, May 26, 1:53 PM · User-jbond, Operations
jbond created T253632: update profile::waf::apache2::administrative to use the new abuse_networks hiera key.
Tue, May 26, 1:53 PM · User-jbond, Operations
jbond moved T239323: Investigate how automated tasks can authenticate against CAS from in progress to change pending on the CAS-SSO board.
Tue, May 26, 1:50 PM · Patch-For-Review, CAS-SSO, User-jbond, Operations
jbond added a comment to T239323: Investigate how automated tasks can authenticate against CAS.

I had a look at this on the CAS side and i think it would be doable to add some level of 2FA with an account in ldap. however i think most scripted users would only need to login to one service and not all services as such im not sure it makes sense to give scripted accounts a full SSO account as we would end up complicating things by applying additional ACL's for. Again we can restrict which services a specific account has access to on the CAS side however it seems to me that we are just adding complexity , both on CAS but also on the monitoring scripts which would need some type of session management.

Tue, May 26, 1:50 PM · Patch-For-Review, CAS-SSO, User-jbond, Operations
jbond edited P11302 test.pp.
Tue, May 26, 10:10 AM
jbond edited P11302 test.pp.
Tue, May 26, 10:10 AM
jbond created P11302 test.pp.
Tue, May 26, 10:09 AM
jbond moved T239323: Investigate how automated tasks can authenticate against CAS from Backlog to in progress on the CAS-SSO board.
Tue, May 26, 9:37 AM · Patch-For-Review, CAS-SSO, User-jbond, Operations
jbond moved T233933: Replicated ticket registry from Backlog to blocked on the CAS-SSO board.
Tue, May 26, 9:37 AM · CAS-SSO, Patch-For-Review, User-jbond, Operations
jbond added a comment to T233933: Replicated ticket registry.

Currently blocked waiting on mcrouter for buster

Tue, May 26, 9:36 AM · CAS-SSO, Patch-For-Review, User-jbond, Operations
jbond closed T233939: Wikimedia theme for SSO login page, a subtask of T233921: Further steps for CAS/web SSO, as Resolved.
Tue, May 26, 9:34 AM · CAS-SSO, Security-Team, User-jbond, Operations
jbond closed T233939: Wikimedia theme for SSO login page as Resolved.

A new skin is now in place

Tue, May 26, 9:34 AM · CAS-SSO, User-jbond, Operations
jbond closed T233948: Review ticket policies, a subtask of T233921: Further steps for CAS/web SSO, as Resolved.
Tue, May 26, 9:34 AM · CAS-SSO, Security-Team, User-jbond, Operations
jbond closed T233948: Review ticket policies as Resolved.
Tue, May 26, 9:34 AM · CAS-SSO, User-jbond, Operations
jbond moved T233947: CAS build as a deb from Backlog to in progress on the CAS-SSO board.
Tue, May 26, 9:32 AM · CAS-SSO, User-jbond, Operations
jbond added a comment to T233948: Review ticket policies.

This has been discussed and documented . Please reopen if further investigation is required

Tue, May 26, 9:31 AM · CAS-SSO, User-jbond, Operations
jbond closed T233950: Revisit Tomcat deployment of CAS, a subtask of T233921: Further steps for CAS/web SSO, as Resolved.
Tue, May 26, 9:28 AM · CAS-SSO, Security-Team, User-jbond, Operations
jbond closed T233950: Revisit Tomcat deployment of CAS as Resolved.

This is now in place, resolving

Tue, May 26, 9:28 AM · CAS-SSO, User-jbond, Operations
jbond closed T245743: Icinga check for CAS-protected web services, a subtask of T233921: Further steps for CAS/web SSO, as Resolved.
Tue, May 26, 9:28 AM · CAS-SSO, Security-Team, User-jbond, Operations
jbond closed T245743: Icinga check for CAS-protected web services as Resolved.

This is now in place, resolving

Tue, May 26, 9:28 AM · CAS-SSO, Security-Team, User-jbond, Operations
jbond closed T246010: Investigate CAS performance as Resolved.

I asked @Zbyszko to look at this and they where unable to get to the root cause however as we now use an external tomcat instance this issue no longer affects us. As such im resolving.

Tue, May 26, 9:27 AM · CAS-SSO, Operations, Performance Issue
jbond closed T246010: Investigate CAS performance, a subtask of T233921: Further steps for CAS/web SSO, as Resolved.
Tue, May 26, 9:27 AM · CAS-SSO, Security-Team, User-jbond, Operations

Mon, May 25

jbond committed rLPRI132b1756b366: wmcs::monitoring: add profile::grafana::ldap::bind_password: (authored by jbond).
wmcs::monitoring: add profile::grafana::ldap::bind_password:
Mon, May 25, 6:41 PM
jbond closed T253407: Fetching source packages for backporting as Resolved.

Thanks for raising this, i have now added the unstable src repos to the build hosts.

Mon, May 25, 10:36 AM · Packaging
jbond triaged T253407: Fetching source packages for backporting as Medium priority.
Mon, May 25, 10:03 AM · Packaging

Thu, May 21

jbond closed T251247: Add CI to the private repo as Resolved.
Thu, May 21, 3:33 PM · User-jbond, Puppet, Operations
jbond added a comment to T251247: Add CI to the private repo.

Have added yamllint checking to the private repo

Thu, May 21, 11:54 AM · User-jbond, Puppet, Operations
jbond added a comment to T252890: scrape ripe atlas data for a few anchors at other large networks.

Which measurements to you plan to scrap?

  • all measurements the anchors are performing outbound?
  • the anchoring measurements directed at thes anchor.
Thu, May 21, 11:20 AM · netops, Operations
jbond added a comment to T247972: Cloud DNS: fix inconsistent ownership of reverse domains for openstack floating ip networks.

I noticed that the ns0.openstack.codfw1dev.wikimediacloud.org. servers are configured as name servers for the 57.15.185.in-addr.arpa. zone. however you actually need to configure the 0/29.57.15.185.in-addr.arpa. zone

Thu, May 21, 9:04 AM · cloud-services-team (Kanban)

Tue, May 19

jbond created P11237 mcrouter ldd.
Tue, May 19, 3:34 PM
jbond added a comment to T251574: prepare mcrouter package for Debian Buster.

That's exactly what i have used, docker_entry.sh is just a cleaned up version of the FB packaging scripts

Tue, May 19, 3:19 PM · Patch-For-Review, serviceops, Packaging
jbond added a comment to T251574: prepare mcrouter package for Debian Buster.

(I tried to rebuild our current package for Buster, but 0.37.0 is incompatible with OpenSSL 1.1, this is likely fixed in current releases.)

Correct the 0.41 version in the docker process builds correctly

Tue, May 19, 3:15 PM · Patch-For-Review, serviceops, Packaging
jbond created P11235 yaml multiline.
Tue, May 19, 2:33 PM
jbond added a comment to T253122: Set minimum-links 2 to AMS-IX LACP.

LGMT

Tue, May 19, 2:19 PM · Operations, netops
jbond claimed T251574: prepare mcrouter package for Debian Buster.
Tue, May 19, 12:59 PM · Patch-For-Review, serviceops, Packaging
jbond added a comment to T251574: prepare mcrouter package for Debian Buster.

@Andrew I have actully made a bit of progress on this here

Tue, May 19, 9:07 AM · Patch-For-Review, serviceops, Packaging