Page MenuHomePhabricator

jbond (John Bond)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Jan 7 2019, 1:06 PM (32 w, 7 h)
Availability
Available
IRC Nick
jbond42
LDAP User
Jbond
MediaWiki User
JBond (WMF) [ Global Accounts ]

Recent Activity

Fri, Aug 16

jbond created T230600: Investigate the potential benefits of BGPalerter .
Fri, Aug 16, 9:04 AM · netops, Operations

Thu, Aug 15

jbond added a comment to T230204: Add Clara Andrew-Wani to wmf ldap group.

When i started I believe someone from OIT created an office wiki page based of a standard template. There is also an Ops specific page which i think is referenced in the first wiki.

Thu, Aug 15, 11:19 AM · LDAP-Access-Requests
jbond added a comment to T229621: Icinga check defined from LVS configuration for cloudelastic are borked.
Thu, Aug 15, 11:18 AM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Operations, Traffic

Wed, Aug 14

jbond committed rLPRI96bd5c67e0dc: apereo_cas: add some content to the keystore (authored by jbond).
apereo_cas: add some content to the keystore
Wed, Aug 14, 12:04 PM
jbond committed rLPRId6a9dc18d69f: apereo_cas: add empty keystore file (authored by jbond).
apereo_cas: add empty keystore file
Wed, Aug 14, 12:00 PM

Fri, Aug 9

jbond added a comment to T229621: Icinga check defined from LVS configuration for cloudelastic are borked.

When this alerted today and ended up going right down the rabbit hole. Anyway i think i have found out some information.

Fri, Aug 9, 2:00 PM · Patch-For-Review, Discovery-Search (Current work), Elasticsearch, Operations, Traffic

Thu, Aug 8

jbond added a comment to T230002: puppetdb queue size went up since July 30.

The lua hack seems to have worked. I have again updated the config to send some canary servers to puppetmaster1003. So far i have seen no errors in the puppetdb log. Further via puppetdb i can see that the puppet reports and facts are all refreshing.

Thu, Aug 8, 4:00 PM · Patch-For-Review, Operations

Wed, Aug 7

jbond added a comment to T230002: puppetdb queue size went up since July 30.

I have disabled puppetmaster1003 for now, unfortunately from reading the PUP-8901 It seems the advice from puppetlabs is to always upgrade all puppetmasters and the puppetdb in tandem

Wed, Aug 7, 10:35 AM · Patch-For-Review, Operations
jbond added a subtask for T228657: Upgrade Puppet Masters and Puppet DB servers: T230002: puppetdb queue size went up since July 30.
Wed, Aug 7, 10:24 AM · Patch-For-Review, Puppet
jbond added a parent task for T230002: puppetdb queue size went up since July 30: T228657: Upgrade Puppet Masters and Puppet DB servers.
Wed, Aug 7, 10:24 AM · Patch-For-Review, Operations
jbond triaged T228657: Upgrade Puppet Masters and Puppet DB servers as Normal priority.
Wed, Aug 7, 10:23 AM · Patch-For-Review, Puppet
jbond added a comment to T230002: puppetdb queue size went up since July 30.

It seems when that the severs using the new puppet master cause the following stack trace when they try reach the 'store report' phase

Wed, Aug 7, 10:11 AM · Patch-For-Review, Operations
jbond added a comment to T230002: puppetdb queue size went up since July 30.

Seems to correlate well with when i enabled the canary puppet master and when i started adding more canary hosts

Wed, Aug 7, 9:51 AM · Patch-For-Review, Operations

Tue, Aug 6

jbond created P8872 apache.
Tue, Aug 6, 3:19 PM
jbond created T229916: Create a cassandra.service which subsumes casandra-{a,b,c} services using PartsOf=cassandra.service .
Tue, Aug 6, 11:15 AM · Cassandra, Operations

Mon, Aug 5

jbond added a comment to T229861: Can't reach cloudelastic.wikimedia.org via IPv6.

@EBernhardson im not sure where the data above came from but i suspect a typo somewhere 2620:0:860::ed1a::1 dose not exist anywhere in the puppet repo. however 2620:0:860:ed1a::1 (notice the missing ':') is valid and is routed to the lvs eqiad servers. That said it does look like most lvs services are in the 2620:0:860:ed1a:: prefix an the servers i have found in the 2620:0:861:1:: prefix seem to be real machines so i suspect that the ipv6 address may be incorrect

Mon, Aug 5, 6:21 PM · Operations, Traffic, Discovery-Search (Current work)
jbond created T229807: cookbook sre.elasticsearch.rolling-restart failed with cluster relforge.
Mon, Aug 5, 11:49 AM · Discovery-Search (Current work), Operations, SRE-tools, Elasticsearch
jbond added a comment to T102099: Fix IPv6 autoconf issues once and for all, across the fleet..

Thanks brandon, Ill take a look at removing the ones SLAAC addresses from puppet this week. One of them, at least, was added by me and was what led me down this rabbit hole :)

Mon, Aug 5, 10:58 AM · Traffic, netops, Operations, IPv6

Fri, Aug 2

jbond added a comment to T102099: Fix IPv6 autoconf issues once and for all, across the fleet..

Our current set up seems to use SLACC without any of the privacy extensions as such the lower 64 bits of the ipv6 address are composed of the mac address of the interface. It seems this would be fairly simple to script the AAAA records and then im not sure if we need to do much configuration. Although the conversion then turns to what do we do with all the hosts that currently have interface::add_ip6_mapped

Answering my own question, as NIC's can and do change in a server lifetime using SLAAC like this is undesirable as we dont want an IP change if we change a NIC

Fri, Aug 2, 10:09 AM · Traffic, netops, Operations, IPv6

Thu, Aug 1

jbond added a comment to T228854: Use git commit id as "configuration version" for puppet.

I think this is a really good idea. further after a bit of investigation i think any arbitrary string can be used. Later versions of the puppet documentation even use git rev-parse as an example command. also r10k has a script in there repo which has the sha1, git server and last git log message as the version

Thu, Aug 1, 5:00 PM · Operations, observability, Puppet
jbond added a comment to T102099: Fix IPv6 autoconf issues once and for all, across the fleet..

Hi all,

Thu, Aug 1, 3:55 PM · Traffic, netops, Operations, IPv6
jbond closed T229571: labpuppetmaster1001: puppet catalog error related to canary_host as Resolved.

Sorry for the interuption, this issues ws caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/527064 when i removed the config entry at the common level. I have now added a default of 0 canary_hosts in labs.yaml and comon/profile/puppetmaster/frontend.yaml. Please reopen if you still see issues

Thu, Aug 1, 12:01 PM · cloud-services-team (Kanban)
jbond claimed T229571: labpuppetmaster1001: puppet catalog error related to canary_host.
Thu, Aug 1, 11:58 AM · cloud-services-team (Kanban)

Wed, Jul 31

jbond added a comment to T229397: Puppet: get row/rack info from Netbox.

Have a script that periodically generates a hiera file with that information for each host, to be merged by the puppetmasters with the public tree as we already do with the private repo

Wed, Jul 31, 1:12 PM · Patch-For-Review, Puppet, Operations

Tue, Jul 30

jbond created P8823 (An Untitled Masterwork).
Tue, Jul 30, 10:39 AM
jbond created P8822 (An Untitled Masterwork).
Tue, Jul 30, 10:30 AM

Thu, Jul 25

jbond added a comment to T208566: puppet.git rake fails with ruby 2.5.

In case this needs rolling back the the issue can be fixed in 4.8 with the following patch https://phabricator.wikimedia.org/P8772

Thu, Jul 25, 4:33 PM · Patch-For-Review, Continuous-Integration-Config, Operations, Puppet

Mon, Jul 22

jbond closed T226508: Icinga custom checks should follow our HTTP User-Agent policy as Resolved.

yes, closing thanks

Mon, Jul 22, 3:14 PM · observability, Operations
jbond added a comment to T227587: upgrade puppet master servers.

puppetmaster1003 has now been build and is running puppet 5.5

Mon, Jul 22, 1:06 PM · Patch-For-Review, Packaging, Puppet, Operations
jbond created T228657: Upgrade Puppet Masters and Puppet DB servers.
Mon, Jul 22, 1:05 PM · Patch-For-Review, Puppet

Jul 19 2019

jbond added a comment to T224977: puppet-catalog-compiler: compilation result randomly places servers in the 'failed' section.

Here is another example. puppetmaster1003.eqiad.wmnet is in both 'Hosts that have no differences' and 'Hosts that fail to compile when the change is applied'

Jul 19 2019, 2:30 PM · Operations, puppet-compiler
jbond edited P8775 example output.
Jul 19 2019, 11:34 AM
jbond created P8775 example output.
Jul 19 2019, 11:25 AM
jbond added a comment to T224977: puppet-catalog-compiler: compilation result randomly places servers in the 'failed' section.

After checking https://puppet-compiler.wmflabs.org/compiler1001/16855/ change error/warning logs for hosts marked as "fail to compile when the change is applied" it looks like two warnings are being interpreted as errors:

Warning: Unknown variable: '::restricted_to'. at /srv/jenkins-workspace/puppet-compiler/16855/change/src/modules/profile/manifests/ldap/client/labs.pp:5:72
Warning: Unknown variable: '::restricted_from'. at /srv/jenkins-workspace/puppet-compiler/16855/change/src/modules/profile/manifests/ldap/client/labs.pp:6:76
Jul 19 2019, 10:09 AM · Operations, puppet-compiler
jbond added a subtask for T224977: puppet-catalog-compiler: compilation result randomly places servers in the 'failed' section: T228266: PCC always has an ERROR when compiling for servers with profile::redis::slave.
Jul 19 2019, 10:09 AM · Operations, puppet-compiler
jbond added a parent task for T228266: PCC always has an ERROR when compiling for servers with profile::redis::slave: T224977: puppet-catalog-compiler: compilation result randomly places servers in the 'failed' section.
Jul 19 2019, 10:09 AM · Continuous-Integration-Infrastructure, Packaging, Operations

Jul 18 2019

jbond closed T228174: Function Call, wrong number of arguments (4 for 5) when a puppet master is connected to labs puppetmaster as Resolved.
Jul 18 2019, 8:40 PM · cloud-services-team (Kanban), Cloud-Services
jbond added a comment to T225604: log spam from mtail 3.0.0~rc19 on wezen.

This error is triggered slower on some machines vs others

  • cp2026 (cache::upload): takes a bit longer to die ~ 5 minutes
  • cp1085 (cache::text): dies instantly
  • cp1075 (cache::text): dies instantly
  • cp1076 (cache::upload): dies instantly
Jul 18 2019, 2:22 PM · Patch-For-Review, observability
jbond added a comment to T225604: log spam from mtail 3.0.0~rc19 on wezen.

Unfortunately this had to be downgraded again. We saw the following error with the new version

Jul 18 2019, 1:32 PM · Patch-For-Review, observability
jbond updated the task description for T201342: rack/setup/install puppetmaster1003.eqiad.wmnet.
Jul 18 2019, 12:00 PM · Operations
jbond added a comment to T225604: log spam from mtail 3.0.0~rc19 on wezen.

I have now pushed a patch so so that mtail uses -logs /dev/stdin instead of -logfds 0. would you like me to upgrade everything to mtail_3.0.0~rc24.1-1+wmf1_amd64.deb again?

Jul 18 2019, 10:50 AM · Patch-For-Review, observability
jbond added a comment to T228395: puppetdb prometheus metrics per-host metrics.

below are all the hosts with puppetdb package installed

Jul 18 2019, 10:29 AM · User-fgiunchedi, Operations, Puppet
jbond created P8772 puppet4.8 with ruby > 2.3 - NoMethodError: undefined method `<<' for nil:NilClass.
Jul 18 2019, 10:09 AM

Jul 17 2019

jbond added a comment to T228266: PCC always has an ERROR when compiling for servers with profile::redis::slave.

Investigating further this is due to how populate_puppetdb adds entries to the database. p`opulate_puppetdb` loops through the site.pp manifest picks one host from every role and compiles its catalog locally so that the result is sent to the puppetdb. This means that the puppet DB is populated and the other calls to query_resource have enough data for them succeed.

Jul 17 2019, 3:48 PM · Continuous-Integration-Infrastructure, Packaging, Operations
jbond renamed T228266: PCC always has an ERROR when compiling for servers with profile::redis::slave from Investigate if puppetdbquery::query_resources should work on PCC to PCC always has an ERROR when compiling for servers with profile::redis::slave.
Jul 17 2019, 3:39 PM · Continuous-Integration-Infrastructure, Packaging, Operations
jbond updated the task description for T228266: PCC always has an ERROR when compiling for servers with profile::redis::slave.
Jul 17 2019, 3:23 PM · Continuous-Integration-Infrastructure, Packaging, Operations
jbond closed T227779: Hiera incompatible with newer versions of puppet, a subtask of T227587: upgrade puppet master servers, as Resolved.
Jul 17 2019, 11:23 AM · Patch-For-Review, Packaging, Puppet, Operations
jbond closed T227779: Hiera incompatible with newer versions of puppet as Resolved.
Jul 17 2019, 11:23 AM · Patch-For-Review, Packaging, Puppet, Operations
jbond updated the task description for T228266: PCC always has an ERROR when compiling for servers with profile::redis::slave.
Jul 17 2019, 11:06 AM · Continuous-Integration-Infrastructure, Packaging, Operations
jbond added a comment to T228266: PCC always has an ERROR when compiling for servers with profile::redis::slave.

during my research i noticed that puppet db was failing with the following error

Jul 17 2019, 11:00 AM · Continuous-Integration-Infrastructure, Packaging, Operations
jbond triaged T228266: PCC always has an ERROR when compiling for servers with profile::redis::slave as Normal priority.
Jul 17 2019, 10:55 AM · Continuous-Integration-Infrastructure, Packaging, Operations

Jul 16 2019

jbond edited P8754 custome type.
Jul 16 2019, 5:42 PM
jbond edited P8754 custome type.
Jul 16 2019, 5:10 PM
jbond created P8754 custome type.
Jul 16 2019, 5:08 PM
jbond added a comment to T228174: Function Call, wrong number of arguments (4 for 5) when a puppet master is connected to labs puppetmaster.

This was broken by https://gerrit.wikimedia.org/r/c/operations/puppet/+/522101 which added two new custom empty backends. the error message is coming from the fact that hiera v3 requires 5 parameters for its lookup function and hiera v1 4. I'm not sure what triggered but it is likely due in some way to the fact that the other wmcs backends use the V1 api.

Jul 16 2019, 4:42 PM · cloud-services-team (Kanban), Cloud-Services
jbond edited P8753 (An Untitled Masterwork).
Jul 16 2019, 4:15 PM
jbond created P8753 (An Untitled Masterwork).
Jul 16 2019, 4:15 PM
jbond created P8752 (An Untitled Masterwork).
Jul 16 2019, 4:11 PM

Jul 15 2019

jbond committed rLPRId8556aa0f50e: kerberos/rsync_secrets_file: add file (authored by jbond).
kerberos/rsync_secrets_file: add file
Jul 15 2019, 1:57 PM

Jul 12 2019

jbond added a comment to T227100: monitoring::check_prometheus should error on an unquoted ! in the query.

ahh ok so in that case \\! and \! are the same. As ! is not an escape character the '\' is treated treated as a literal. I created a small puppet script to test this (see below) and also added test cases to the spec file

Jul 12 2019, 3:08 PM · good first bug, Operations, observability
jbond added a comment to T220820: Add a CI check for the use of hiera() function.

the lookup function allows one to call it in multiple ways e.g.

Jul 12 2019, 2:04 PM · Puppet, Operations
jbond added a comment to T227100: monitoring::check_prometheus should error on an unquoted ! in the query.

being as its a Friday i thought i would have a go at this however i didn't read the original message correctly and before i refactor i want to double check that you want device\\! and not device\!. I would have thought the first form would fail but i don't know how many levels of encoding/escaping are needed?

Jul 12 2019, 1:53 PM · good first bug, Operations, observability

Jul 11 2019

jbond added a comment to T225604: log spam from mtail 3.0.0~rc19 on wezen.

It seems this flag was removed from upstream just over a year ago. i have pinged fgiunchedi original issue requesting this feature to ask if there is more context around why it got removed.

Jul 11 2019, 2:58 PM · Patch-For-Review, observability
jbond triaged T227779: Hiera incompatible with newer versions of puppet as Normal priority.
Jul 11 2019, 2:17 PM · Patch-For-Review, Packaging, Puppet, Operations
jbond created T227779: Hiera incompatible with newer versions of puppet.
Jul 11 2019, 2:15 PM · Patch-For-Review, Packaging, Puppet, Operations

Jul 10 2019

jbond closed T226530: Create audit script for comparing labs/private.git with production private repo as Resolved.
Jul 10 2019, 7:36 PM · Cloud-VPS
jbond added a comment to T226530: Create audit script for comparing labs/private.git with production private repo.

@Volans and I discussed some changes via IRC which have now been applied. As there has been no further activity on this im gonna close it but feel free to reopen if something has been missed or overlooked

Jul 10 2019, 7:36 PM · Cloud-VPS
jbond triaged T227587: upgrade puppet master servers as Normal priority.
Jul 10 2019, 7:32 PM · Patch-For-Review, Packaging, Puppet, Operations

Jul 9 2019

jbond added a comment to T223463: (2019-09) Create secteam groups in admin.yaml and define permissions.

@sbassett thanks for the info

Jul 9 2019, 3:11 PM · SRE-Access-Requests, Operations, Security-Team, Patch-For-Review
jbond created P8730 (An Untitled Masterwork).
Jul 9 2019, 3:07 PM
jbond created P8729 (An Untitled Masterwork).
Jul 9 2019, 3:06 PM
jbond created T227587: upgrade puppet master servers.
Jul 9 2019, 2:36 PM · Patch-For-Review, Packaging, Puppet, Operations
jbond added a comment to T223463: (2019-09) Create secteam groups in admin.yaml and define permissions.

@chasemp I have gone ahead and created the secteam-users group (renamed from secteam to match our convention) so you can go ahead an start using that one. There where some additional comments around the secteam-admins group and as this is a downgrade in your permissions it was deemed that this change was low priority and we had some time to ensure we get this right. Please let us know if we have got the priority wrong.

Jul 9 2019, 12:48 PM · SRE-Access-Requests, Operations, Security-Team, Patch-For-Review

Jul 5 2019

jbond committed rLPRId532cad973c7: autid_hiera: some minor style changes (authored by jbond).
autid_hiera: some minor style changes
Jul 5 2019, 10:27 AM

Jul 3 2019

jbond merged T226382: Hardware Request: puppet master eqiad into T201342: rack/setup/install puppetmaster1003.eqiad.wmnet.
Jul 3 2019, 10:14 AM · Operations
jbond merged task T226382: Hardware Request: puppet master eqiad into T201342: rack/setup/install puppetmaster1003.eqiad.wmnet.
Jul 3 2019, 10:14 AM · Operations, ops-eqiad, DC-Ops

Jul 2 2019

jbond added a comment to T227111: Zuul is no longer adding jobs to any jenkins pipelines.

at least one more https://gerrit.wikimedia.org/r/c/operations/puppet/+/519227

Jul 2 2019, 4:32 PM · Wikimedia-production-error (Shared Build Failure), Release-Engineering-Team, Zuul
jbond added a comment to T226431: User knissen can't access Superset.

i looked into this a bit and i *think* the debug messages which mention cn=wmf,ou=groups,dc=wikimedia,dc=org can be ignored as we see the following in the logs as well

Jul 2 2019, 4:14 PM · Analytics
jbond added a comment to T226508: Icinga custom checks should follow our HTTP User-Agent policy.

If anyone can review ttps://gerrit.wikimedia.org/r/c/operations/puppet/+/519227 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/519218 i think we should be able to merge and close this task?

Jul 2 2019, 3:42 PM · observability, Operations

Jul 1 2019

jbond added a comment to T224677: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003).

@MoritzMuehlenhoff The updated package has broken some dependencies which is causing an error on phab1003

Jul 1 2019, 2:59 PM · Patch-For-Review, Release-Engineering-Team-TODO (201907), Release-Engineering-Team (Development services), Upstream, Packaging, User-zeljkofilipin, Operations, Diffusion
jbond added a comment to T222356: facter3: Unable to parse routing table.

The update has now been rolled out.

Jul 1 2019, 12:16 PM · Packaging, Puppet, Operations
jbond added a comment to T222356: facter3: Unable to parse routing table.

One thing to consider for buster: So far we've used the facter version in buster, so I think we have two options:

Jul 1 2019, 10:56 AM · Packaging, Puppet, Operations
jbond triaged T226937: Not possible to server-side upload certain images: "An unknown error occurred in storage backend "local-swift-eqiad"" as Normal priority.
Jul 1 2019, 10:35 AM · media-storage, Operations
jbond triaged T226970: Use white version of Wikimedia logo for grafana as Normal priority.
Jul 1 2019, 10:33 AM · Operations, UI-Standardization
jbond triaged T226840: Consistent HTTP 503 Error on some urls for some logged-in users (CentralAuth Set-Cookie storm) as Normal priority.
Jul 1 2019, 10:32 AM · Core Platform Team, Patch-For-Review, TimedMediaHandler, MW-1.34-notes (1.34.0-wmf.13; 2019-07-09), Wikimedia-Incident, Performance-Team (Radar), Traffic, MediaWiki-extensions-CentralAuth, Operations
jbond triaged T224348: Global rename of Fiona B. → Fiona*: supervision needed as Normal priority.
Jul 1 2019, 10:31 AM · Operations, Wikimedia-Site-requests
jbond added a comment to T226971: Global rename of [REDACTED] → Renamed user mou89p43twvqcvm8ut9w3: supervision needed.

please go ahead

Jul 1 2019, 9:54 AM · DBA, Wikimedia-Site-requests

Jun 28 2019

jbond added a comment to T222356: facter3: Unable to parse routing table.
Jun 28 2019, 2:07 PM · Packaging, Puppet, Operations
jbond triaged T226727: Integrate mailing lists in Wikimedia Space as Normal priority.
Jun 28 2019, 10:45 AM · Space (Jan-Mar-2020), Operations, Wikimedia-Mailing-lists
jbond triaged T226754: Video in MediaWiki:Loginprompt causes login session problems as High priority.
Jun 28 2019, 10:43 AM · TimedMediaHandler, Wikimedia-Video, MediaWiki-User-login-and-signup
jbond triaged T226811: Site: eqiad/codfw 2 VMs each for pool counters as Normal priority.
Jun 28 2019, 10:42 AM · vm-requests, Operations
jbond triaged T226813: Add centrallog1001 to syslog servers in network ACLs as Normal priority.
Jun 28 2019, 10:41 AM · netops, User-fgiunchedi, Operations
jbond triaged T226815: Gather metrics on request status codes, latencies from the MediaWiki appservers as Normal priority.
Jun 28 2019, 10:40 AM · Patch-For-Review, Operations, observability, serviceops
jbond updated subscribers of T226724: Gerrit manager rights for Ottomata.

@Legoktm @hashar are either of you able to help with this?

Jun 28 2019, 10:39 AM · Release-Engineering-Team, Gerrit-Privilege-Requests
jbond triaged T226724: Gerrit manager rights for Ottomata as Normal priority.
Jun 28 2019, 10:36 AM · Release-Engineering-Team, Gerrit-Privilege-Requests

Jun 27 2019

jbond closed T226744: please re-activate LDAP access for Dzahn as Resolved.
Jun 27 2019, 5:45 PM · Operations, SRE-Access-Requests
jbond added a comment to T226744: please re-activate LDAP access for Dzahn.

this should be fixed now reopen if more is needed or ping me on irc

Jun 27 2019, 5:45 PM · Operations, SRE-Access-Requests
jbond closed T226204: Requesting access to stats machines/ores hosts hosts for Andy Craze as Resolved.

great, i think this is done now so closing please re open if there is still an issue

Jun 27 2019, 5:35 PM · Analytics, Operations, SRE-Access-Requests
jbond closed T226204: Requesting access to stats machines/ores hosts hosts for Andy Craze, a subtask of T226416: Onboard Andy Craze -- Accounts and access, as Resolved.
Jun 27 2019, 5:35 PM · Scoring-platform-team (Current)
jbond added a comment to T218367: Create MoveCom mailing list for Movement communications group.

@Varnent I have now renamed the old list and it is now available at https://lists.wikimedia.org/mailman/listinfo/MoveCom. The old URL should redirect to this one and the old email address has been configured as an alias. Please let me know if there are any issues

Jun 27 2019, 4:05 PM · Wikimedia-Mailing-lists, Operations