Page MenuHomePhabricator

dcaro (David Caro)
SRE & amauteur yak shaver

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Nov 2 2020, 11:59 AM (98 w, 6 d)
Availability
Available
IRC Nick
dcaro
LDAP User
David Caro
MediaWiki User
DCaro (WMF) [ Global Accounts ]

Recent Activity

Fri, Sep 16

dcaro updated subscribers of T316436: Cloud VPS Terraform support.
Fri, Sep 16, 2:23 PM · Epic, Cloud-VPS, cloud-services-team (Kanban)

Thu, Sep 15

dcaro assigned T317838: SystemdUnitDownForLong cloudcontrol1006:9100 Unit keystone_sync_keys_to_cloudcontrol1005.wikimedia.org.service on node cloudcontrol1006 has been down for long. to Andrew.

I'm not sure if you are still thinking about solving this, we can continue trying to fix it or if not an issue we should try to avoid the alert.

Thu, Sep 15, 7:10 AM · Patch-For-Review, cloud-services-team (Kanban)

Wed, Sep 14

dcaro added a comment to T316328: [tbs.registry-admission.toolsbeta] Allow images from harbor on toolsbeta.

Yep, it does :), so that part is done, the next one is deploying it on toolsbeta.

Wed, Sep 14, 5:05 PM · Toolforge Build Service (Iteration 00), User-Raymond_Ndibe, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro added a comment to T317643: 2022-09-13 labstore1005 down.

btw. backups are also failing as they pull from the secondary (labstore1005)

Wed, Sep 14, 9:04 AM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro

Tue, Sep 13

dcaro added a parent task for T317651: hw troubleshooting: network cards shutting down for lasbtore1005.eqiad.wmnet: T317643: 2022-09-13 labstore1005 down.
Tue, Sep 13, 2:53 PM · cloud-services-team (Kanban), DC-Ops
dcaro added a subtask for T317643: 2022-09-13 labstore1005 down: T317651: hw troubleshooting: network cards shutting down for lasbtore1005.eqiad.wmnet.
Tue, Sep 13, 2:53 PM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro created T317651: hw troubleshooting: network cards shutting down for lasbtore1005.eqiad.wmnet.
Tue, Sep 13, 2:52 PM · cloud-services-team (Kanban), DC-Ops
dcaro added a comment to T317643: 2022-09-13 labstore1005 down.

It seems that the network cards on labstore1005 overheated and decided to turn themselves off:

[10704668.955975] bnx2x: [bnx2x_attn_int_deasserted0:4170(enp133s0f0)]SPIO5 hw attention
[10704668.964747] bnx2x 0000:85:00.0 enp133s0f0: Fan Failure on Network Controller has caused the driver to shutdown the card to prevent permanent damage.
                  Please contact OEM Support for assistance
[10704669.040066] bnx2x: [bnx2x_attn_int_deasserted0:4170(enp133s0f1)]SPIO5 hw attention
[10704669.048824] bnx2x 0000:85:00.1 enp133s0f1: Fan Failure on Network Controller has caused the driver to shutdown the card to prevent permanent damage.
                  Please contact OEM Support for assistance
Tue, Sep 13, 2:25 PM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro changed the status of T317643: 2022-09-13 labstore1005 down from Open to In Progress.
Tue, Sep 13, 2:17 PM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro moved T317643: 2022-09-13 labstore1005 down from To refine to Doing on the User-dcaro board.
Tue, Sep 13, 2:17 PM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro triaged T317643: 2022-09-13 labstore1005 down as High priority.
Tue, Sep 13, 2:17 PM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro merged T317571: SystemdUnitDownForLong labstore1007:9100 Unit ferm.service on node labstore1007 has been down for long. into T317596: SystemdUnitDownForLong labstore1007:9100 - ferm.service.
Tue, Sep 13, 7:46 AM · cloud-services-team (Kanban)
dcaro merged task T317571: SystemdUnitDownForLong labstore1007:9100 Unit ferm.service on node labstore1007 has been down for long. into T317596: SystemdUnitDownForLong labstore1007:9100 - ferm.service.
Tue, Sep 13, 7:46 AM · cloud-services-team (Kanban)
dcaro renamed T317596: SystemdUnitDownForLong labstore1007:9100 - ferm.service from SystemdUnitDownForLong labstore1007:9100 to SystemdUnitDownForLong labstore1007:9100 - ferm.service.
Tue, Sep 13, 7:27 AM · cloud-services-team (Kanban)
dcaro closed T317596: SystemdUnitDownForLong labstore1007:9100 - ferm.service as Resolved.

This actually was not related (I think).

Tue, Sep 13, 7:27 AM · cloud-services-team (Kanban)
dcaro assigned T317593: SystemdUnitDownForLong clouddumps1002:9100 Unit kiwix-mirror-update.service on node clouddumps1002 has been down for long. to Andrew.
Tue, Sep 13, 7:19 AM · cloud-services-team (Kanban)
dcaro assigned T317596: SystemdUnitDownForLong labstore1007:9100 - ferm.service to Andrew.
Tue, Sep 13, 7:19 AM · cloud-services-team (Kanban)
dcaro assigned T317597: SystemdUnitDownForLong clouddumps1001:9100 Unit kiwix-mirror-update.service on node clouddumps1001 has been down for long. to Andrew.

@Andrew I'm not sure if you are still working on these, let me know otherwise.

Tue, Sep 13, 7:19 AM · cloud-services-team (Kanban)

Fri, Sep 9

dcaro changed the status of T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions from In Progress to Stalled.
Fri, Sep 9, 1:31 PM · Toolforge Build Service (Iteration 00), Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro moved T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions from Doing to Blocked on the User-dcaro board.
Fri, Sep 9, 1:31 PM · Toolforge Build Service (Iteration 00), Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro changed the status of T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions, a subtask of T267374: [tbs.beta] Create a toolforge build service beta release, from In Progress to Stalled.
Fri, Sep 9, 1:31 PM · Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, Toolforge Build Service, User-dcaro, cloud-services-team (Kanban)
dcaro moved T316854: [tbs.buildpacks] The uwsgi buildpack is failing to run due to missing Python.h library from Doing to Blocked on the User-dcaro board.
Fri, Sep 9, 1:30 PM · Patch-For-Review, Toolforge Build Service (Iteration 00), Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro changed the status of T316854: [tbs.buildpacks] The uwsgi buildpack is failing to run due to missing Python.h library from In Progress to Stalled.
Fri, Sep 9, 1:30 PM · Patch-For-Review, Toolforge Build Service (Iteration 00), Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro closed T316919: [novafullstack.alerts] stop paging and instead create tasks for any nova-fullstack related alert as Resolved.
Fri, Sep 9, 1:15 PM · Patch-For-Review, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro moved T316919: [novafullstack.alerts] stop paging and instead create tasks for any nova-fullstack related alert from Doing to Done on the User-dcaro board.
Fri, Sep 9, 1:15 PM · Patch-For-Review, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro

Wed, Sep 7

dcaro added a comment to T316222: [tbs.cli]: Create reasonable defaults for the build command (so the users have to add only the minimum needed info).

Done, you should have access

Wed, Sep 7, 4:02 PM · Toolforge Build Service (Iteration 00), User-Slst2020, User-dcaro, cloud-services-team (Kanban)
dcaro added a comment to T316222: [tbs.cli]: Create reasonable defaults for the build command (so the users have to add only the minimum needed info).

Oh! let's fix that then!

Wed, Sep 7, 4:00 PM · Toolforge Build Service (Iteration 00), User-Slst2020, User-dcaro, cloud-services-team (Kanban)
dcaro moved T316854: [tbs.buildpacks] The uwsgi buildpack is failing to run due to missing Python.h library from In Progress to In Review on the Toolforge Build Service (Iteration 00) board.
Wed, Sep 7, 3:55 PM · Patch-For-Review, Toolforge Build Service (Iteration 00), Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro added a comment to T316222: [tbs.cli]: Create reasonable defaults for the build command (so the users have to add only the minimum needed info).

Nothing, the patch just needs merging if it looks good to everyone.

Wed, Sep 7, 3:47 PM · Toolforge Build Service (Iteration 00), User-Slst2020, User-dcaro, cloud-services-team (Kanban)
dcaro added a comment to T316232: [tbs.harbor.toolsbeta] Create a postgres DB with trove and plug the current instance there.

Document also where it needs to be configured in the README for the buildservice repository:
https://github.com/toolforge/buildservice

@dcaro do you mean how to configure the docker-compose setup to work with the external db?

Wed, Sep 7, 2:38 PM · User-Slst2020, Toolforge Build Service (Iteration 00), Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro added a comment to T316222: [tbs.cli]: Create reasonable defaults for the build command (so the users have to add only the minimum needed info).

What is left here? What is blocking you?

Wed, Sep 7, 12:39 PM · Toolforge Build Service (Iteration 00), User-Slst2020, User-dcaro, cloud-services-team (Kanban)
dcaro added a comment to T317143: [tbs] Update the tekton cli package.

Just created https://gitlab.wikimedia.org/repos/cloud/toolforge/packaging-pack/-/tree/main/ to package the pack cli, the tekton cli is just a binary too so you might be able to reuse all the scripts there (it was surprisingly complicated :/)

Wed, Sep 7, 10:17 AM · Toolforge Build Service, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro

Tue, Sep 6

dcaro added a subtask for T265684: Figure out CI solution for building buildpack-based images for Toolforge: T317143: [tbs] Update the tekton cli package.
Tue, Sep 6, 4:44 PM · cloud-services-team (Kanban), Toolforge
dcaro added a parent task for T317143: [tbs] Update the tekton cli package: T265684: Figure out CI solution for building buildpack-based images for Toolforge.
Tue, Sep 6, 4:44 PM · Toolforge Build Service, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro removed a project from T317143: [tbs] Update the tekton cli package: Toolforge Build Service (Iteration 00).
Tue, Sep 6, 4:42 PM · Toolforge Build Service, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro edited projects for T317143: [tbs] Update the tekton cli package, added: Toolforge Build Service (Iteration 00); removed Toolforge Build Service.
Tue, Sep 6, 4:42 PM · Toolforge Build Service, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro added a project to T317143: [tbs] Update the tekton cli package: Toolforge Build Service.
Tue, Sep 6, 4:42 PM · Toolforge Build Service, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro triaged T317143: [tbs] Update the tekton cli package as Medium priority.
Tue, Sep 6, 4:42 PM · Toolforge Build Service, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro assigned T316984: SystemdUnitDownForLong cloudbackup1003:9100 Unit backup_glance_images.service on node cloudbackup1003 has been down for long. to Andrew.
Tue, Sep 6, 9:56 AM · cloud-services-team (Kanban)
dcaro closed T316989: SystemdUnitDownForLong cloudcontrol1005:9100 Unit keystone_sync_keys_to_cloudcontrol1007.wikimedia.org.service on node cloudcontrol1005 has been down for long. as Resolved.

Closing as it's solved, but pinging @Andrew as he was playing with this at some point.

Tue, Sep 6, 8:30 AM · cloud-services-team (Kanban)
dcaro committed rODIT8cc49512ba23: build: use the standard path to get the docker binary (authored by dcaro).
build: use the standard path to get the docker binary
Tue, Sep 6, 8:22 AM
dcaro committed rODITd9cf23a220ae: bullseye0: Improve the install-packages script (authored by dcaro).
bullseye0: Improve the install-packages script
Tue, Sep 6, 8:22 AM
dcaro committed rODIT801ea3f7cfe4: Remove buster0 buildpacks images (authored by dcaro).
Remove buster0 buildpacks images
Tue, Sep 6, 8:22 AM
dcaro committed rODIT37a95b92db31: bullseye0: Add bullseye buildpack build/run images (authored by dcaro).
bullseye0: Add bullseye buildpack build/run images
Tue, Sep 6, 8:22 AM
dcaro moved T317071: PuppetSyncFailure on project-proxy, toolsbeta and cloudinfra cloudvps projects from Doing to Blocked on the User-dcaro board.
Tue, Sep 6, 7:43 AM · Patch-For-Review, Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro changed the status of T317071: PuppetSyncFailure on project-proxy, toolsbeta and cloudinfra cloudvps projects from In Progress to Stalled.
Tue, Sep 6, 7:43 AM · Patch-For-Review, Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro added a comment to T317071: PuppetSyncFailure on project-proxy, toolsbeta and cloudinfra cloudvps projects.

Manually rebased the /var/lib/git/labs/private repositories to latest master, there was a conflict in hieradata/common.yaml due to having added an entry at the bottom of it (and there's a local patch that does the same).

Tue, Sep 6, 7:43 AM · Patch-For-Review, Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro changed the status of T317071: PuppetSyncFailure on project-proxy, toolsbeta and cloudinfra cloudvps projects from Open to In Progress.
Tue, Sep 6, 7:16 AM · Patch-For-Review, Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro moved T317071: PuppetSyncFailure on project-proxy, toolsbeta and cloudinfra cloudvps projects from To refine to Doing on the User-dcaro board.
Tue, Sep 6, 7:16 AM · Patch-For-Review, Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro triaged T317071: PuppetSyncFailure on project-proxy, toolsbeta and cloudinfra cloudvps projects as High priority.
Tue, Sep 6, 7:15 AM · Patch-For-Review, Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro

Mon, Sep 5

dcaro committed rCTTC274c5f175d1d: package: Added script to build a debian package locally (authored by dcaro).
package: Added script to build a debian package locally
Mon, Sep 5, 5:03 PM
dcaro closed T316973: labstore1006: Persistent high iowait - 2022-09-03 as Resolved.
Mon, Sep 5, 4:25 PM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro added a comment to T316973: labstore1006: Persistent high iowait - 2022-09-03.

This might be related to T317001 and would be solved/relieved by T306550

Mon, Sep 5, 4:24 PM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro committed rLTRAb8cd21d31ce4: Move to helmfile instead of kustomize (authored by dcaro).
Move to helmfile instead of kustomize
Mon, Sep 5, 8:13 AM
dcaro committed rLTRA887567012dde: deploy: add interactive and first deploy flags (authored by dcaro).
deploy: add interactive and first deploy flags
Mon, Sep 5, 8:13 AM

Sat, Sep 3

dcaro added a comment to T316973: labstore1006: Persistent high iowait - 2022-09-03.

It's going down already, started at ~8:00UTC until ~9:00UTC

Sat, Sep 3, 8:51 AM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro changed the status of T316973: labstore1006: Persistent high iowait - 2022-09-03 from Open to In Progress.
Sat, Sep 3, 8:48 AM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro moved T316973: labstore1006: Persistent high iowait - 2022-09-03 from To refine to Doing on the User-dcaro board.
Sat, Sep 3, 8:48 AM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro
dcaro triaged T316973: labstore1006: Persistent high iowait - 2022-09-03 as High priority.
Sat, Sep 3, 8:44 AM · Cloud-Services-Worktype-Unplanned, Cloud-Services-Origin-Alert, cloud-services-team (Kanban), User-dcaro

Fri, Sep 2

dcaro updated subscribers of T316285: decommission cloudservices1003.wikimedia..org.

It seems that the runbook did not cleanup puppetdb or it was repopulated right after, as the host still shows there:

Fri, Sep 2, 8:54 AM · SRE, ops-eqiad, cloud-services-team (Kanban), decommission-hardware
dcaro claimed T316021: cookbooks.ceph: add pre-flight checks when adding a new osd node.
Fri, Sep 2, 7:49 AM · Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro changed the status of T316021: cookbooks.ceph: add pre-flight checks when adding a new osd node from Open to In Progress.
Fri, Sep 2, 7:49 AM · Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro moved T316021: cookbooks.ceph: add pre-flight checks when adding a new osd node from Refined to Doing on the User-dcaro board.
Fri, Sep 2, 7:49 AM · Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro moved T315420: cookbooks.ceph.bootstrap_and_add: fix the osd class (hdd->ssd) if needed when adding a new osd from Doing to Done on the User-dcaro board.
Fri, Sep 2, 7:49 AM · Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro closed T315420: cookbooks.ceph.bootstrap_and_add: fix the osd class (hdd->ssd) if needed when adding a new osd as Resolved.
Fri, Sep 2, 7:49 AM · Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro added a comment to T315420: cookbooks.ceph.bootstrap_and_add: fix the osd class (hdd->ssd) if needed when adding a new osd.

Merged https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/824153.

Fri, Sep 2, 7:49 AM · Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro changed the status of T316919: [novafullstack.alerts] stop paging and instead create tasks for any nova-fullstack related alert from Open to In Progress.
Fri, Sep 2, 7:31 AM · Patch-For-Review, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro moved T316919: [novafullstack.alerts] stop paging and instead create tasks for any nova-fullstack related alert from Today to Doing on the User-dcaro board.
Fri, Sep 2, 7:31 AM · Patch-For-Review, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro moved T316919: [novafullstack.alerts] stop paging and instead create tasks for any nova-fullstack related alert from To refine to Today on the User-dcaro board.
Fri, Sep 2, 7:30 AM · Patch-For-Review, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro triaged T316919: [novafullstack.alerts] stop paging and instead create tasks for any nova-fullstack related alert as High priority.
Fri, Sep 2, 7:30 AM · Patch-For-Review, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro

Thu, Sep 1

dcaro changed the status of T316854: [tbs.buildpacks] The uwsgi buildpack is failing to run due to missing Python.h library from Open to In Progress.
Thu, Sep 1, 7:36 AM · Patch-For-Review, Toolforge Build Service (Iteration 00), Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro moved T316854: [tbs.buildpacks] The uwsgi buildpack is failing to run due to missing Python.h library from To refine to Doing on the User-dcaro board.
Thu, Sep 1, 7:36 AM · Patch-For-Review, Toolforge Build Service (Iteration 00), Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro moved T316854: [tbs.buildpacks] The uwsgi buildpack is failing to run due to missing Python.h library from Next Up to In Progress on the Toolforge Build Service (Iteration 00) board.
Thu, Sep 1, 7:36 AM · Patch-For-Review, Toolforge Build Service (Iteration 00), Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro edited projects for T316854: [tbs.buildpacks] The uwsgi buildpack is failing to run due to missing Python.h library, added: Toolforge Build Service (Iteration 00); removed Toolforge Build Service.
Thu, Sep 1, 7:35 AM · Patch-For-Review, Toolforge Build Service (Iteration 00), Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro added a project to T316854: [tbs.buildpacks] The uwsgi buildpack is failing to run due to missing Python.h library: Toolforge Build Service.
Thu, Sep 1, 7:35 AM · Patch-For-Review, Toolforge Build Service (Iteration 00), Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro triaged T316854: [tbs.buildpacks] The uwsgi buildpack is failing to run due to missing Python.h library as High priority.
Thu, Sep 1, 7:35 AM · Patch-For-Review, Toolforge Build Service (Iteration 00), Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro added a hashtag to Toolforge Build Service (Iteration 00): #tbs00.
Thu, Sep 1, 7:32 AM
dcaro moved T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions from In Progress to In Review on the Toolforge Build Service (Iteration 00) board.
Thu, Sep 1, 7:31 AM · Toolforge Build Service (Iteration 00), Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro moved T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions from Next Up to In Progress on the Toolforge Build Service (Iteration 00) board.
Thu, Sep 1, 7:31 AM · Toolforge Build Service (Iteration 00), Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro edited projects for T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions, added: Toolforge Build Service (Iteration 00); removed Toolforge Build Service.
Thu, Sep 1, 7:31 AM · Toolforge Build Service (Iteration 00), Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro

Wed, Aug 31

dcaro added a member for Trusted-Contributors: KHernandez-WMF.
Wed, Aug 31, 5:21 PM
dcaro assigned T316429: SystemdUnitDownForLong labstore1006:9100 Unit kiwix-mirror-update.service on node labstore1006 has been down for long. to Andrew.
Wed, Aug 31, 10:10 AM · cloud-services-team (Kanban)
dcaro assigned T316445: SystemdUnitDownForLong labstore1007:9100 Unit kiwix-mirror-update.service on node labstore1007 has been down for long. to Andrew.
Wed, Aug 31, 10:10 AM · cloud-services-team (Kanban)
dcaro assigned T316731: SystemdUnitDownForLong cloudvirt1022:9100 Unit purge_vm_backup.service on node cloudvirt1022 has been down for long. to Andrew.
Wed, Aug 31, 10:02 AM · Patch-For-Review, cloud-services-team (Kanban)
dcaro assigned T316732: SystemdUnitDownForLong cloudvirt1024:9100 Unit purge_vm_backup.service on node cloudvirt1024 has been down for long. to Andrew.
Wed, Aug 31, 10:02 AM · Patch-For-Review, cloud-services-team (Kanban)
dcaro assigned T316733: SystemdUnitDownForLong cloudvirt1025:9100 Unit purge_vm_backup.service on node cloudvirt1025 has been down for long. to Andrew.
Wed, Aug 31, 10:02 AM · Patch-For-Review, cloud-services-team (Kanban)
dcaro assigned T316735: SystemdUnitDownForLong cloudvirt1026:9100 Unit purge_vm_backup.service on node cloudvirt1026 has been down for long. to Andrew.
Wed, Aug 31, 10:02 AM · Patch-For-Review, cloud-services-team (Kanban)
dcaro assigned T316734: SystemdUnitDownForLong cloudvirt1021:9100 Unit purge_vm_backup.service on node cloudvirt1021 has been down for long. to Andrew.
Wed, Aug 31, 10:02 AM · Patch-For-Review, cloud-services-team (Kanban)
dcaro assigned T316736: SystemdUnitDownForLong cloudvirt1027:9100 Unit purge_vm_backup.service on node cloudvirt1027 has been down for long. to Andrew.
Wed, Aug 31, 10:02 AM · Patch-For-Review, cloud-services-team (Kanban)
dcaro assigned T316737: SystemdUnitDownForLong cloudvirt1023:9100 Unit purge_vm_backup.service on node cloudvirt1023 has been down for long. to Andrew.
Wed, Aug 31, 10:02 AM · Patch-For-Review, cloud-services-team (Kanban)
dcaro added a comment to T316738: SystemdUnitDownForLong cloudcontrol1005:9100 Unit purge_vm_backup.service on node cloudcontrol1005 has been down for long..

It seems to be an issue with sqlalchemy>1.4 and backy2 when cleaning up: https://github.com/wamdam/backy2/pull/93

Wed, Aug 31, 10:00 AM · Patch-For-Review, cloud-services-team (Kanban)
dcaro assigned T316738: SystemdUnitDownForLong cloudcontrol1005:9100 Unit purge_vm_backup.service on node cloudcontrol1005 has been down for long. to Andrew.
Wed, Aug 31, 9:59 AM · Patch-For-Review, cloud-services-team (Kanban)
dcaro moved T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions from Backlog to To be Discussed on the Toolforge Build Service board.
Wed, Aug 31, 8:55 AM · Toolforge Build Service (Iteration 00), Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro changed the status of T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions, a subtask of T267374: [tbs.beta] Create a toolforge build service beta release, from Open to In Progress.
Wed, Aug 31, 8:55 AM · Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, Toolforge Build Service, User-dcaro, cloud-services-team (Kanban)
dcaro changed the status of T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions from Open to In Progress.
Wed, Aug 31, 8:55 AM · Toolforge Build Service (Iteration 00), Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro moved T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions from To refine to Doing on the User-dcaro board.
Wed, Aug 31, 8:55 AM · Toolforge Build Service (Iteration 00), Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro claimed T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions.
Wed, Aug 31, 8:55 AM · Toolforge Build Service (Iteration 00), Patch-For-Review, Cloud-Services-Worktype-Project, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro

Tue, Aug 30

dcaro assigned T316685: [openstack.neutron] Policies reference a rule that is not defined to Andrew.
Tue, Aug 30, 4:56 PM · Patch-For-Review, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro updated subscribers of T316685: [openstack.neutron] Policies reference a rule that is not defined.
Tue, Aug 30, 4:55 PM · Patch-For-Review, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro
dcaro triaged T316685: [openstack.neutron] Policies reference a rule that is not defined as Medium priority.
Tue, Aug 30, 4:54 PM · Patch-For-Review, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, cloud-services-team (Kanban), User-dcaro