bd808 (Bryan Davis)Administrator
Engineering Manager, Technical Engagement

Projects (39)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:36 PM (228 w, 4 d)
Roles
Administrator
Availability
Available
IRC Nick
bd808
LDAP User
BryanDavis
MediaWiki User
BDavis (WMF) [ Global Accounts ]

I'm BDavis (WMF) on wiki, bd808 on irc, and BryanDavis on Gerrit and Wikitech.

I've got a thing for ๐Ÿฆ„s. Don't judge.

I work for or provide services to the Wikimedia Foundation, but this is my only Phabricator account. Edits, statements, or other contributions made from this account are my own, and may not reflect the views of the Foundation.

Recent Activity

Today

bd808 added a comment to T215092: Rename developer account 'DannyS712 bot' to 'DannyS712'.

@bd808 It took a few tries, but it worked (with the username "dannyS712" (lowercase d, not a big deal);

Wed, Feb 20, 12:42 AM ยท cloud-services-team (Kanban), LDAP, User-DannyS712, Toolforge
bd808 added a comment to T141396: Provide a migration path for tools running tomcat.

In my case, my Java tool replacer is prepared to be packaged as a WAR to be run in Tomcat, or as an SpringBoot standalone JAR.

I have been running the tool as a WAR so far. On the other side, when running the tool as a JAR in the lab server, I am not able to access it from the outside (https://tools.wmflabs.org/replacer/), I guess because of the port used.

Any hint about the port to use so the tool can be accessed in a regular browser ?

Wed, Feb 20, 12:37 AM ยท Toolforge, Kubernetes

Yesterday

bd808 moved T216139: Request increased quota for packaging Cloud VPS project from Inbox to Approved on the Cloud-VPS (Quota-requests) board.
Tue, Feb 19, 11:32 PM ยท cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
bd808 closed T216139: Request increased quota for packaging Cloud VPS project as Resolved.
$ openstack quota show packaging
+----------------------+-----------+
| Field                | Value     |
+----------------------+-----------+
| cores                | 32        |
| fixed-ips            | 200       |
| floating_ips         | 0         |
| floatingip           | 0         |
| injected-file-size   | 10240     |
| injected-files       | 5         |
| injected-path-size   | 255       |
| instances            | 16        |
| key-pairs            | 100       |
| network              | 10        |
| port                 | 50        |
| project              | packaging |
| properties           | 128       |
| ram                  | 32768     |
| rbac_policy          | 10        |
| router               | 10        |
| secgroup-rules       | 100       |
| secgroups            | 40        |
| server_group_members | 10        |
| server_groups        | 10        |
| subnet               | 10        |
| subnetpool           | -1        |
+----------------------+-----------+
$ openstack quota set --ram 40960 packaging
$ openstack quota show packaging
+----------------------+-----------+
| Field                | Value     |
+----------------------+-----------+
| cores                | 32        |
| fixed-ips            | 200       |
| floating_ips         | 0         |
| floatingip           | 0         |
| injected-file-size   | 10240     |
| injected-files       | 5         |
| injected-path-size   | 255       |
| instances            | 16        |
| key-pairs            | 100       |
| network              | 10        |
| port                 | 50        |
| project              | packaging |
| properties           | 128       |
| ram                  | 40960     |
| rbac_policy          | 10        |
| router               | 10        |
| secgroup-rules       | 100       |
| secgroups            | 40        |
| server_group_members | 10        |
| server_groups        | 10        |
| subnet               | 10        |
| subnetpool           | -1        |
+----------------------+-----------+
Tue, Feb 19, 11:32 PM ยท cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Mholloway awarded T215217: deployment-prep: Code stewardship request a Pterodactyl token.
Tue, Feb 19, 11:31 PM ยท Beta-Cluster-Infrastructure, Code-Stewardship-Reviews
bd808 claimed T216139: Request increased quota for packaging Cloud VPS project.

Current quota usage in eqiad1-r is

  • 4 / 16 instances
  • 12 / 32 VCPUs
  • 24.0 GB / 32.0 GB RAM
Tue, Feb 19, 11:25 PM ยท cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
bd808 added a comment to T216461: fireflytools python Kubernetes pod stuck in CrashLoopBackOff state.

From the merged task at T216346: Facing an issue related to Toolforge Trusty:

While doing those steps, the problem was that since there was no requirements.txt file, I was not able to upgrade to the latest version. Since the migration was only partially done, the tool is currently not working. I would like to know whether I can upgrade to latest version of Python without creating the requirements.txt file.

Tue, Feb 19, 11:16 PM ยท Tools
bd808 added a comment to T214921: Setup elasticsearch on cloudelastic100[1-4].

Oh, maybe it isn't...If these are nodes in production networks then it could be fine.

These nodes should be, if i understand it right, living in the same space as the cloud mariadb replicas. The servers live in the production network and have a port opened up to the cloud network somehow.

Tue, Feb 19, 11:01 PM ยท Discovery-Search (Current work), cloud-services-team, Cloud-VPS, Operations
bd808 added a member for Trusted-Contributors: WMFOffice.
Tue, Feb 19, 10:13 PM
bd808 added a member for acl*userdisable: WMFOffice.
Tue, Feb 19, 10:13 PM
bd808 added a comment to T216461: fireflytools python Kubernetes pod stuck in CrashLoopBackOff state.

From the merged task at T216346: Facing an issue related to Toolforge Trusty:

I have recently got an email stating me to migrate Fireflytools to Kubernetes cluster. By referring to the page https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation, I started the migration. First I did the steps mentioned under the heading Move a grid engine webservice which I think has been successfully completed. Then I did the steps mentioned under Move a cron job after which I tried to do the steps mentioned under Rebuild virtualenv for python users. While doing those steps, the problem was that since there was no requirements.txt file, I was not able to upgrade to the latest version. Since the migration was only partially done, the tool is currently not working. I would like to know whether I can upgrade to latest version of Python without creating the requirements.txt file. The command qstat is also not working now.

Tue, Feb 19, 7:28 PM ยท Tools
bd808 merged task T216346: Facing an issue related to Toolforge Trusty into T216461: fireflytools python Kubernetes pod stuck in CrashLoopBackOff state.
Tue, Feb 19, 7:27 PM ยท Toolforge, Kubernetes
bd808 merged T216346: Facing an issue related to Toolforge Trusty into T216461: fireflytools python Kubernetes pod stuck in CrashLoopBackOff state.
Tue, Feb 19, 7:27 PM ยท Tools
bd808 added a comment to T216461: fireflytools python Kubernetes pod stuck in CrashLoopBackOff state.

But when I select the link present in the page https://en.wikipedia.org/wiki/Special:LintErrors, it shows 404. You need to click the link for the word 'tool' in line 3 inorder to get that page. I don't know whether anything needs to be done for that link or not.

Tue, Feb 19, 7:26 PM ยท Tools
bd808 added a comment to T180179: Evaluate the possibility to add Juniper images to Openstack.

There are a few other issues beyond the Neutron constraints that still exist (which kind of boil down to a lack of ability to support VXLAN and arbitrary typologies today):

  • Does introducing base images which require proprietary licenses puts the Cloud Services team in a position of needing to police the license keys in use?
  • Does our current image->instance bootstrapping system support arbitrary base images?
  • Does the benefit to the Foundation's NetOps team outweigh the burden of support of unique features that will be placed on the Cloud Services team?
  • Does the benefit to the Foundation's NetOps team justify the needed exception to the FOSS software policy of the Cloud Services environment?
Tue, Feb 19, 6:21 PM ยท cloud-services-team (Kanban), Cloud-VPS, netops, Traffic, Operations
bd808 added a project to T215217: deployment-prep: Code stewardship request: Beta-Cluster-Infrastructure.
Tue, Feb 19, 5:22 PM ยท Beta-Cluster-Infrastructure, Code-Stewardship-Reviews
bd808 added subtasks for T215217: deployment-prep: Code stewardship request: T216404: deployment-db03.deployment-prep.eqiad.wmflabs instance can not start, T216164: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs.
Tue, Feb 19, 5:22 PM ยท Beta-Cluster-Infrastructure, Code-Stewardship-Reviews
bd808 added a parent task for T216164: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs: T215217: deployment-prep: Code stewardship request.
Tue, Feb 19, 5:22 PM ยท Patch-For-Review, Beta-Cluster-Infrastructure
bd808 added a parent task for T216404: deployment-db03.deployment-prep.eqiad.wmflabs instance can not start: T215217: deployment-prep: Code stewardship request.
Tue, Feb 19, 5:22 PM ยท Wikidata, User-Addshore, Cloud-VPS, cloud-services-team (Kanban), Beta-Cluster-Infrastructure
Addshore awarded T215217: deployment-prep: Code stewardship request a Cup of Joe token.
Tue, Feb 19, 5:16 PM ยท Beta-Cluster-Infrastructure, Code-Stewardship-Reviews
bd808 placed T216404: deployment-db03.deployment-prep.eqiad.wmflabs instance can not start up for grabs.

I'm not sure why this is only "Normal", I'm going to at least bump this to high.
We are essentially ignoring one of our QA/CI steps here which will increase the chances of incidents in production..
Perhaps this should even be UBN..?

Tue, Feb 19, 4:10 PM ยท Wikidata, User-Addshore, Cloud-VPS, cloud-services-team (Kanban), Beta-Cluster-Infrastructure
bd808 removed a project from T216513: compiler1002.puppet-diffs.eqiad.wmflabs instance is down: cloud-services-team.
Tue, Feb 19, 3:53 PM ยท Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure, puppet-compiler
bd808 removed a project from T216517: Restore Jenkins CI slave integration-slave-jessie-android 172.16.6.183: cloud-services-team.
Tue, Feb 19, 3:52 PM ยท Patch-For-Review, Continuous-Integration-Infrastructure, Release-Engineering-Team
bd808 added a comment to T216461: fireflytools python Kubernetes pod stuck in CrashLoopBackOff state.

While trying to upgrade python, I am getting the error


How can this be solved?

Tue, Feb 19, 3:37 PM ยท Tools
bd808 added a comment to T215092: Rename developer account 'DannyS712 bot' to 'DannyS712'.

Looks like my attempt to clean the data in toolsadmin (Striker) was not quite right:

2019-02-19T05:13:11Z [9f21cd67192e4fedaec7ea0ea5808f9f] django_auth_ldap WARNING: (1062, "Duplicate entry 'DannyS712' for key 'ldapname'") while authenticating DannyS712 bot

I decided to try deleting the existing cache of information there instead.

Tue, Feb 19, 5:34 AM ยท cloud-services-team (Kanban), LDAP, User-DannyS712, Toolforge
bd808 added a comment to T216164: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs.

Actually it might be related after all. The failure is from an attempt to install the niginx-full package which seems to be triggered by profile::services_proxy's nginx::site { 'upstream_proxies': .... The profile as configured now tries to ensure => absent that define, but the define itself contains include ::nginx which does not get passed the ensure => absent parameter and thus does try to install the nginx-full package and related configuration.

Tue, Feb 19, 5:12 AM ยท Patch-For-Review, Beta-Cluster-Infrastructure
bd808 added a comment to T216164: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs.

@Joe that patch is apparently not quite sufficient. New output with the dummy settings from T216164#4955058 removed:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Failed to parse template profile/services_proxy/upstream_proxies.conf.erb:
  Filepath: /etc/puppet/modules/profile/templates/services_proxy/upstream_proxies.conf.erb
  Line: 1
  Detail: undefined method `each' for nil:NilClass
 at /etc/puppet/modules/profile/manifests/services_proxy.pp:37:20 on node deployment-deploy01.deployment-prep.eqiad.wmflabs
Tue, Feb 19, 5:02 AM ยท Patch-For-Review, Beta-Cluster-Infrastructure
bd808 committed R2043:07f5f7cb093a: Add exclusion list to nag.py (authored by bd808).
Add exclusion list to nag.py
Tue, Feb 19, 4:40 AM
bd808 closed T215588: Add exclusion list to trusty-tools nag.py as Resolved.
Tue, Feb 19, 4:37 AM ยท cloud-services-team (Kanban), Toolforge
bd808 closed T215092: Rename developer account 'DannyS712 bot' to 'DannyS712' as Resolved.
Tue, Feb 19, 2:26 AM ยท cloud-services-team (Kanban), LDAP, User-DannyS712, Toolforge
bd808 closed T207221: wikimedia/wikimania-scholarships has vulnerable dependencies as Resolved.

Updates have been deployed into production

Tue, Feb 19, 1:46 AM ยท Patch-For-Review, Wikimedia-Wikimania-Scholarships, Security
GTirloni awarded T215586: Custom Kubernetes deployment fails from Stretch bastion a Yellow Medal token.
Tue, Feb 19, 1:42 AM ยท Patch-For-Review, cloud-services-team (Kanban), Toolforge
bd808 claimed T193646: Can't access /mnt from kubernetes nodejs shell.
Tue, Feb 19, 1:42 AM ยท cloud-services-team (Kanban), Patch-For-Review, Data-Services, Toolforge, Kubernetes
bd808 created T216465: webarchivebot custom Kubernetes pod stuck in CrashLoopBackOff.
Tue, Feb 19, 1:16 AM ยท Tools
bd808 moved T177914: Switch labstore servers to default SSH configuration from Backlog to Shared Storage on the Data-Services board.
Tue, Feb 19, 1:09 AM ยท cloud-services-team (Kanban), Data-Services, Operations
bd808 moved T216167: Verify checkwiki tool against excessive DB usage from Backlog to ToolsDB on the Data-Services board.
Tue, Feb 19, 1:09 AM ยท Data-Services, Toolforge, cloud-services-team (Kanban)
bd808 moved T216168: Review labsdb1005 MariaDB configuration against prod standards from Backlog to ToolsDB on the Data-Services board.
Tue, Feb 19, 1:09 AM ยท Data-Services, cloud-services-team (Kanban)
bd808 moved T216170: toolsdb - Per-user connection limits from Backlog to ToolsDB on the Data-Services board.
Tue, Feb 19, 1:09 AM ยท Patch-For-Review, cloud-services-team (Kanban), Toolforge, Data-Services
bd808 moved T216173: labsdb1005/6 - Upgrade to Stretch from Backlog to ToolsDB on the Data-Services board.
Tue, Feb 19, 1:09 AM ยท Data-Services, cloud-services-team (Kanban)
bd808 moved T216208: ToolsDB overload and cleanup from Backlog to ToolsDB on the Data-Services board.
Tue, Feb 19, 1:08 AM ยท Patch-For-Review, TCB-Team, Phragile, Data-Services, cloud-services-team (Kanban)
bd808 moved T216213: s52481__stats_global running CREATE DATABASE IF NOT EXISTS on too many queries causing locking issues from Backlog to ToolsDB on the Data-Services board.
Tue, Feb 19, 1:08 AM ยท Data-Services, Tracking, Toolforge
bd808 moved T216422: Virtualize NFS servers used exclusively by Cloud VPS tenants from Backlog to Shared Storage on the Data-Services board.
Tue, Feb 19, 1:08 AM ยท Data-Services, cloud-services-team (Kanban)
bd808 moved T216441: Evaluate transferring the non-replicated tables to the new toolsdb server from Backlog to ToolsDB on the Data-Services board.
Tue, Feb 19, 1:08 AM ยท Data-Services, cloud-services-team (Kanban)
bd808 created T216464: nsfw python Kuberntes pod stuck in CrashLoopBackOff.
Tue, Feb 19, 1:06 AM ยท Tools
bd808 created T216463: itwikiarticlebot ruby Kubernetes pod stuck in CrashLoopBackOff.
Tue, Feb 19, 1:02 AM ยท Tools
bd808 created T216462: flossbrowser php5.6 Kubernetes pod stuck in CrashLoopBackOff.
Tue, Feb 19, 12:54 AM ยท Tools
bd808 created T216461: fireflytools python Kubernetes pod stuck in CrashLoopBackOff state.
Tue, Feb 19, 12:49 AM ยท Tools
bd808 created T216460: import-freemind pods in CrashLoopBackoff.
Tue, Feb 19, 12:38 AM ยท Tools
bd808 closed T215599: Lighttpd failing to find mono binary as cgi handler for 'mbh' tool as Resolved.

Closing per comments in T215599#4941087

Tue, Feb 19, 12:26 AM ยท cloud-services-team, Toolforge
bd808 added a comment to T216167: Verify checkwiki tool against excessive DB usage.

Per the announcement at https://lists.wikimedia.org/pipermail/cloud-announce/2019-February/000137.html, ToolsDB has been migrated to new hardware and is currently operating normally. The checkwiki tool can be re-enabled, and we will try to keep an eye on issues that may arise. Hopefully this report was just a false positive that was noticed as the failing server struggled to keep up.

Tue, Feb 19, 12:15 AM ยท Data-Services, Toolforge, cloud-services-team (Kanban)

Mon, Feb 18

bd808 added a comment to T216451: Move DNS records for tools.eqiad.wmflabs to Designate.

Related, but really separate, issue: we should deprecate some of the random aliases we have for services and announce a stop date for maintaining them.

Mon, Feb 18, 10:49 PM ยท cloud-services-team (Kanban), Toolforge
Krenair awarded T216451: Move DNS records for tools.eqiad.wmflabs to Designate a Evil Spooky Haunted Tree token.
Mon, Feb 18, 10:25 PM ยท cloud-services-team (Kanban), Toolforge
bd808 created T216451: Move DNS records for tools.eqiad.wmflabs to Designate.
Mon, Feb 18, 10:23 PM ยท cloud-services-team (Kanban), Toolforge
bd808 assigned T216375: "Looks like you already have another webservice running" failure when trying to migrate webservice to mahmoud.

@mahmoud if the instructions on working around this problem that were linked in T216375#4960607 do not resolve the issue, please do re-assign to me with any error messages you can find.

Mon, Feb 18, 10:12 PM ยท Toolforge
bd808 renamed T216375: "Looks like you already have another webservice running" failure when trying to migrate webservice from Trouble upgrading toolforge python webservices from trusty to stretch to "Looks like you already have another webservice running" failure when trying to migrate webservice.
Mon, Feb 18, 10:09 PM ยท Toolforge
bd808 created T216440: Archive/delete tool bd808-test4.
Mon, Feb 18, 6:59 PM ยท Cleanup, Tools, Toolforge
bd808 created T216439: Archive/delete tool bd808-test3.
Mon, Feb 18, 6:58 PM ยท Cleanup, Tools, Toolforge
bd808 lowered the priority of T216373: CloudVPS: run maintain-dbusers inside Toolforge from Normal to Low.

I'm dropping priority too. We may be better served by working on T216422: Virtualize NFS servers used exclusively by Cloud VPS tenants instead if our main concern is network isolation, but we can talk that through when we get things working well enough to stop and have an incident retrospective.

Mon, Feb 18, 6:26 PM ยท Patch-For-Review, Toolforge, cloud-services-team (Kanban), Cloud-VPS
bd808 closed T215586: Custom Kubernetes deployment fails from Stretch bastion as Resolved.

Verified that this is working as hoped from tools-sgebastion-07.tools.eqiad.wmflabs with the jouncebot tool by stopping and starting its custom deployment.

Mon, Feb 18, 6:14 PM ยท Patch-For-Review, cloud-services-team (Kanban), Toolforge
bd808 added a comment to T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020.

I have been talking to @aborrero about the new instance on clouddb1001 - and I have been taking a general look.
While comparing the grants, I have realised that clouddb1001 is missing a grant for the following user: s52716 (that grant exists on labsdb1005) it could be a new user. I can easily copy that grant over to clouddb1001, but I want the green light from @Bstorm just in case this has something to do with maintain-dbusers or something :-)

Mon, Feb 18, 5:35 PM ยท cloud-services-team (Kanban), Patch-For-Review, Epic, Cloud-VPS
bd808 placed T216139: Request increased quota for packaging Cloud VPS project up for grabs.
Mon, Feb 18, 5:06 PM ยท cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
bd808 triaged T216422: Virtualize NFS servers used exclusively by Cloud VPS tenants as Normal priority.
Mon, Feb 18, 4:43 PM ยท Data-Services, cloud-services-team (Kanban)
bd808 added a subtask for T210704: Migrate node-based services in production to node10: T213077: Migrate Kartotherian/Tilerator to Node 10.
Mon, Feb 18, 4:19 PM ยท serviceops, Core Platform Team Backlog (Later), Patch-For-Review, Services (next), Operations
bd808 added a parent task for T213077: Migrate Kartotherian/Tilerator to Node 10: T210704: Migrate node-based services in production to node10.
Mon, Feb 18, 4:19 PM ยท Maps (Kartotherian), Epic, Reading-Infrastructure-Team-Backlog
bd808 added a comment to T216353: toolsdb: firewalling changes for new setup (temporal mysql replication).

We need to be able to have labstore100[45] and labsdb1004 talk to port 3306 on clouddb1001.clouddb-services.eqiad.wmflabs. This will allow us to leave the maintain-dbusers script on the labstores (T216373#4960664) and also to use labsdb1004 as temporary replica of clouddb1001 until we get the second cloudvirt and its giant clouddbXXXX instances online.

Mon, Feb 18, 6:18 AM ยท User-Marostegui, netops, Operations, cloud-services-team (Kanban), Cloud-VPS
bd808 changed the status of T214907: Request increased quota for Wikidocumentaries Cloud VPS project from Open to Stalled.

Marking as stalled until @Susannaanas or @TuukkaH respond to T214907#4921971

Mon, Feb 18, 5:53 AM ยท Cloud-VPS (Quota-requests)
bd808 changed the status of T216126: Requesting contentadmin access for 'Lucas Werkmeister (WMDE)' on Wikitech from Open to Stalled.
Mon, Feb 18, 5:38 AM ยท cloud-services-team (Kanban), Toolforge, wikitech.wikimedia.org
bd808 added a comment to T216373: CloudVPS: run maintain-dbusers inside Toolforge.

I started working on some refactoring and cleanup of the maintain-dbusers script and ran into a blocker for running the script from an NFS client rather than from the NFS server. That blocker is the script's use of chattr +i. The NFS protocol does not have support for manipulating file attributes via chattr or any other method. This use of the "immutable" bit is a protection against unintentional (and intentional) data corruption caused by users renaming, deleting, or otherwise modifying their $HOME/replica.my.cnf files.

Mon, Feb 18, 5:18 AM ยท Patch-For-Review, Toolforge, cloud-services-team (Kanban), Cloud-VPS
bd808 moved T216370: IP address list for grid nodes / Freenode iline request from Inbox to Needs discussion on the cloud-services-team (Kanban) board.
Mon, Feb 18, 4:55 AM ยท cloud-services-team (Kanban), wikimedia-irc-freenode, Toolforge
bd808 added a project to T216370: IP address list for grid nodes / Freenode iline request: cloud-services-team (Kanban).
Mon, Feb 18, 4:54 AM ยท cloud-services-team (Kanban), wikimedia-irc-freenode, Toolforge
bd808 added a comment to T216370: IP address list for grid nodes / Freenode iline request.

This is fundamentally a duplicate of T151704: Freenode sometimes throttles bot connections from tools, but the problem is currently worse than it has been in the past because of the lack of public IPv4 addresses on the new Stretch job grid. We should really try T151704#3467490 before we resort to abusing public IPs like we did in the past.

Mon, Feb 18, 4:54 AM ยท cloud-services-team (Kanban), wikimedia-irc-freenode, Toolforge

Sun, Feb 17

bd808 added a comment to T129936: Ensure that Terms of Use document restrictions on third-party web interactions.

There has been a small bit of work on this project in the past 6 months, but there is no resolution yet. I will provide updates if and when there is information that can be shared.

Sun, Feb 17, 11:07 PM ยท Community-Tech-Tool-Labs, WMF-Legal, Cloud-Services
bd808 added a comment to T216167: Verify checkwiki tool against excessive DB usage.

What is the exact status now? Does the project remain deactivated and thus the quality of the entire project falls by the wayside?

Sun, Feb 17, 11:02 PM ยท Data-Services, Toolforge, cloud-services-team (Kanban)
bd808 committed R2117:b6341359a99a: Update composer.lock for PHP 7.2 runtime (authored by bd808).
Update composer.lock for PHP 7.2 runtime
Sun, Feb 17, 10:24 PM
bd808 added a comment to T216373: CloudVPS: run maintain-dbusers inside Toolforge.

In addition to ToolsDB, the maintain-dbusers script currently also manages user accounts on the Wiki Replica servers (labsdb10{09,10,11}). To maintain the current functionality we need to do one of:

  • allow code running on {cloud,lab}storeXXXX to communicate with a mysql server inside Cloud VPS address space
  • expose secrets to a service running inside Cloud VPS address space which can maintain authn/z credentials on the labsdb10{09,10,11} cluster
  • rethink the replica.my.cnf management process to split responsibility for wiki replicas (production realm) and toolsdb (labs realm)
Sun, Feb 17, 8:38 PM ยท Patch-For-Review, Toolforge, cloud-services-team (Kanban), Cloud-VPS
bd808 added a comment to T216126: Requesting contentadmin access for 'Lucas Werkmeister (WMDE)' on Wikitech.

@Lucas_Werkmeister_WMDE I have changed the title here to reflect a request for contentadmin. Granting full sysop on Wikitech is unlikely without also granting Cloud-wide root. Contentadmin is a much easier right to approve, assuming that you have a desire to help maintain sensitive content on wikitech in the long term. Please do respond here an let us know if you are actively seeking this right or were just a bit annoyed to find that there was content that you could not change on wikitech.

Sun, Feb 17, 12:43 AM ยท cloud-services-team (Kanban), Toolforge, wikitech.wikimedia.org
bd808 renamed T216126: Requesting contentadmin access for 'Lucas Werkmeister (WMDE)' on Wikitech from Administrator rights on Wikitech to Requesting contentadmin access for 'Lucas Werkmeister (WMDE)' on Wikitech.
Sun, Feb 17, 12:34 AM ยท cloud-services-team (Kanban), Toolforge, wikitech.wikimedia.org
bd808 moved T216126: Requesting contentadmin access for 'Lucas Werkmeister (WMDE)' on Wikitech from Inbox to Needs discussion on the cloud-services-team (Kanban) board.
Sun, Feb 17, 12:31 AM ยท cloud-services-team (Kanban), Toolforge, wikitech.wikimedia.org
bd808 added a project to T216126: Requesting contentadmin access for 'Lucas Werkmeister (WMDE)' on Wikitech: cloud-services-team (Kanban).
Sun, Feb 17, 12:31 AM ยท cloud-services-team (Kanban), Toolforge, wikitech.wikimedia.org

Sat, Feb 16

bd808 updated subscribers of T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline.

The plan has syslog + json as formatting, since that's what we use for logstash already and preserves more information. Although we could have syslog + current formatting?

Sat, Feb 16, 6:16 PM ยท Wikimedia-Logstash, Operations
bd808 added a comment to T214278: Quickstatements, "backend is overloaded".

It worked for a while today but now I'm logged out again and cannot log back in.
All edits through the tool seem to have stalled as well (for any user), and https://tools.wmflabs.org/quickstatements/api.php gives a HTTP 500 error.

Sat, Feb 16, 1:03 AM ยท Tools, Toolforge
bd808 moved T216263: Mentoring Program / Mentor Matching Session from Backlog to Sessions on the Wikimedia-Hackathon-2019 board.
Sat, Feb 16, 12:41 AM ยท Wikimedia-Hackathon-2019-Organization, Wikimedia-Hackathon-2019
bd808 moved T216303: Introduction to Wikimedia Cloud Services session from Backlog to Sessions on the Wikimedia-Hackathon-2019 board.
Sat, Feb 16, 12:40 AM ยท Wikimedia-Hackathon-2019
bd808 created T216303: Introduction to Wikimedia Cloud Services session.
Sat, Feb 16, 12:40 AM ยท Wikimedia-Hackathon-2019
bd808 awarded T215211: cloud instance rescue tools a Goat token.
Sat, Feb 16, 12:32 AM ยท Patch-For-Review, Cloud-VPS, cloud-services-team (Kanban)

Fri, Feb 15

bd808 added a comment to T216167: Verify checkwiki tool against excessive DB usage.

I don't know what you want from the maintainer for you to unsuspend the web service for this project.

Fri, Feb 15, 6:35 PM ยท Data-Services, Toolforge, cloud-services-team (Kanban)
bd808 moved T180179: Evaluate the possibility to add Juniper images to Openstack from Inbox to Needs discussion on the cloud-services-team (Kanban) board.
Fri, Feb 15, 1:54 AM ยท cloud-services-team (Kanban), Cloud-VPS, netops, Traffic, Operations
bd808 added a comment to T215993: tools.db.svc.eqiad.wmflabs hitting it's limit?.

From what I can see none of the labsdb1005 have any connections limit, maybe we need to establish a limit of connections per user similar to what we have on the replicas. Better to "break" a tool than the whole server.
We can probably also take a look at those specific tools that might need more than X number of connections (being X the number of connections we decide to set).

Fri, Feb 15, 1:43 AM ยท Data-Services
bd808 added a project to T216218: Cloud VPS outage on cloudvirt1025 and cloudvirt1018 due to storage failure: Cloud-VPS.
Fri, Feb 15, 1:22 AM ยท Cloud-VPS, Tracking, cloud-services-team (Kanban)
bd808 created T216216: Archive/delete tool hackmd.
Fri, Feb 15, 1:07 AM ยท Cleanup, Tools, Toolforge
bd808 added a parent task for T216213: s52481__stats_global running CREATE DATABASE IF NOT EXISTS on too many queries causing locking issues: T216208: ToolsDB overload and cleanup.
Fri, Feb 15, 12:56 AM ยท Data-Services, Tracking, Toolforge
bd808 added a subtask for T216208: ToolsDB overload and cleanup: T216213: s52481__stats_global running CREATE DATABASE IF NOT EXISTS on too many queries causing locking issues.
Fri, Feb 15, 12:56 AM ยท Patch-For-Review, TCB-Team, Phragile, Data-Services, cloud-services-team (Kanban)
bd808 created T216212: Archive/delete tool test-t188680.
Fri, Feb 15, 12:53 AM ยท Cleanup, Tools, Toolforge
bd808 created T216211: Archive/delete tool strikertest2017-08-23.
Fri, Feb 15, 12:49 AM ยท Cleanup, Tools, Toolforge
bd808 updated the task description for T170355: Figure out process for deleting an unused tool.
Fri, Feb 15, 12:33 AM ยท cloud-services-team (Kanban), Toolforge
bd808 updated the task description for T216208: ToolsDB overload and cleanup.
Fri, Feb 15, 12:18 AM ยท Patch-For-Review, TCB-Team, Phragile, Data-Services, cloud-services-team (Kanban)
bd808 added a parent task for T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020: T216208: ToolsDB overload and cleanup.
Fri, Feb 15, 12:17 AM ยท cloud-services-team (Kanban), Patch-For-Review, Epic, Cloud-VPS
bd808 added subtasks for T216208: ToolsDB overload and cleanup: T216202: Disk failure on labsdb1005, T216167: Verify checkwiki tool against excessive DB usage, T216168: Review labsdb1005 MariaDB configuration against prod standards, T216170: toolsdb - Per-user connection limits, T216173: labsdb1005/6 - Upgrade to Stretch, T215993: tools.db.svc.eqiad.wmflabs hitting it's limit?, T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020, T196507: Degraded RAID on cloudvirt1019, T194855: Degraded RAID on cloudvirt1020.
Fri, Feb 15, 12:17 AM ยท Patch-For-Review, TCB-Team, Phragile, Data-Services, cloud-services-team (Kanban)
bd808 added a parent task for T194855: Degraded RAID on cloudvirt1020: T216208: ToolsDB overload and cleanup.
Fri, Feb 15, 12:17 AM ยท Patch-For-Review, cloud-services-team (Kanban), ops-eqiad, Operations
bd808 created T216208: ToolsDB overload and cleanup.
Fri, Feb 15, 12:12 AM ยท Patch-For-Review, TCB-Team, Phragile, Data-Services, cloud-services-team (Kanban)