Page MenuHomePhabricator
Create Task
Maniphest T353550

Cassandra (logstash) logging broken
Closed, ResolvedPublic
Actions

Description

Logstash logging no longer works for the RESTBase & AQS clusters. The timing corresponds with the upgrades to Bullseye; Logstash logs stop for eaach node immediately after their upgrade. Interestingly, logging still works for the Sessionstore cluster (the first to be upgraded to Bullseye).

The logstash logback encoder hands off JSON messages to rsyslog on localhost:11514 (UDP). A tcpdump shows that Cassandra is not sending packets to rsyslog on the effected nodes.

See also:

T328490: restbase cluster: decommission end-of-life hosts
T331713: Migrate restbase servers to Bullseye
T347738: Upgrade AQS cluster to Bullseye

Details

SubjectRepoBranchLines +/-
cassandra: install git-fat to satisfy scap requirementoperations/puppetproduction+21 -0
Updated deployement targetsoperations/software/logstash-logback-encodermaster+28 -22
Fix canary name typooperations/software/logstash-logback-encodermaster+1 -1
Bring restbase & aqs targets up to currentoperations/software/logstash-logback-encodermaster+46 -28
Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Eevans triaged this task as Medium priority.Dec 15 2023, 8:21 PM
Eevans updated the task description. (Show Details)Dec 15 2023, 8:24 PM
Eevans updated the task description. (Show Details)Dec 15 2023, 8:26 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Dec 15 2023, 11:41 PM
Eevans added a comment.Dec 16 2023, 1:03 AM

From the journalctl output:

Dec 15 15:08:17 restbase2032 cassandra[1809747]: 15:08:17,879 |-ERROR in ch.qos.logback.core.joran.action.AppenderAction - Could not create an Appender of type [net.logstash.logback.appender.LogstashSocketAppender]. ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type net.logstash.logback.appender.LogstashSocketAppender
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type net.logstash.logback.appender.LogstashSocketAppender
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.util.OptionHelper.instantiateByClassNameAndParameter(OptionHelper.java:69)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.util.OptionHelper.instantiateByClassName(OptionHelper.java:45)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.util.OptionHelper.instantiateByClassName(OptionHelper.java:34)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.joran.action.AppenderAction.begin(AppenderAction.java:52)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.joran.spi.Interpreter.callBeginAction(Interpreter.java:269)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.joran.spi.Interpreter.startElement(Interpreter.java:145)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.joran.spi.Interpreter.startElement(Interpreter.java:128)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:50)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:165)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:152)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:110)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:53)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:65)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:140)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:84)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at org.apache.cassandra.service.CassandraDaemon.<clinit>(CassandraDaemon.java:135)
Dec 15 15:08:17 restbase2032 cassandra[1809747]: Caused by: java.lang.ClassNotFoundException: net.logstash.logback.appender.LogstashSocketAppender
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         at ch.qos.logback.core.util.OptionHelper.instantiateByClassNameAndParameter(OptionHelper.java:56)
Dec 15 15:08:17 restbase2032 cassandra[1809747]:         at         ... 21 common frames omitted
[ ... ]
Dec 15 15:08:17 restbase2032 cassandra[1809747]: 15:08:17,883 |-ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - Could not find an appender named [UDP]. Did you define it below instead of above in the configuration file?
Dec 15 15:08:17 restbase2032 cassandra[1809747]: 15:08:17,883 |-ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - See http://logback.qos.ch/codes.html#appender_order for more details.
Eevans added a comment.Dec 16 2023, 1:06 AM

And finally:

eevans@restbase2032:~$ file /srv/deployment/cassandra/logstash-logback-encoder/lib/logstash-logback-encoder-4.2.jar
/srv/deployment/cassandra/logstash-logback-encoder/lib/logstash-logback-encoder-4.2.jar: ASCII text
eevans@restbase2032:~$

So it would appear that the git-fat jar isn't "hydrated", and the jar file is essentially missing.

Eevans moved this task from Backlog to In-Progress on the Cassandra board.Dec 18 2023, 7:55 PM
Eevans moved this task from In-Progress to Next on the Cassandra board.Feb 6 2024, 12:50 AM
gerritbot added a comment.Feb 14 2024, 7:23 PM

Change 1003509 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/software/logstash-logback-encoder@master] Bring restbase & aqs targets up to current

https://gerrit.wikimedia.org/r/1003509

gerritbot added a project: Patch-For-Review.Feb 14 2024, 7:23 PM

Change 1003509 merged by Eevans:

[operations/software/logstash-logback-encoder@master] Bring restbase & aqs targets up to current

https://gerrit.wikimedia.org/r/1003509

Stashbot added a comment.Feb 14 2024, 7:27 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:27:31Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@5c2dd00]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:28:12Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@5c2dd00]: Deploying to updated target list — T353550 (duration: 00m 41s)

Stashbot added a comment.Feb 14 2024, 7:30 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:30:57Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@5c2dd00]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:31 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:31:17Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@5c2dd00]: Deploying to updated target list — T353550 (duration: 00m 20s)

Maintenance_bot removed a project: Patch-For-Review.Feb 14 2024, 7:31 PM
gerritbot added a comment.Feb 14 2024, 7:33 PM

Change 1003512 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/software/logstash-logback-encoder@master] Fix canary name typo

https://gerrit.wikimedia.org/r/1003512

gerritbot added a project: Patch-For-Review.Feb 14 2024, 7:33 PM

Change 1003512 merged by Eevans:

[operations/software/logstash-logback-encoder@master] Fix canary name typo

https://gerrit.wikimedia.org/r/1003512

Stashbot added a comment.Feb 14 2024, 7:35 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:35:59Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:36 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:36:13Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 00m 16s)

Stashbot added a comment.Feb 14 2024, 7:38 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:38:06Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:39:23Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 01m 17s)

Stashbot added a comment.Feb 14 2024, 7:41 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:41:23Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:42 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:42:08Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 00m 45s)

Stashbot added a comment.Feb 14 2024, 7:43 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:43:14Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:43 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:43:29Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 00m 14s)

Stashbot added a comment.Feb 14 2024, 7:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:46:08Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:46:42Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 00m 34s)

Stashbot added a comment.Feb 14 2024, 7:50 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:50:24Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:50 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:50:29Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 00m 05s)

Stashbot added a comment.Feb 14 2024, 7:50 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:50:54Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:50 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:50:58Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 00m 04s)

Stashbot added a comment.Feb 14 2024, 7:51 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:51:07Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:51 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:51:11Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 00m 03s)

Stashbot added a comment.Feb 14 2024, 7:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:53:06Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:53:13Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 00m 07s)

Stashbot added a comment.Feb 14 2024, 7:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:53:46Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:53:52Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 00m 06s)

Stashbot added a comment.Feb 14 2024, 7:54 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:53:59Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550

Stashbot added a comment.Feb 14 2024, 7:54 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:54:05Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@0521449]: Deploying to updated target list — T353550 (duration: 00m 05s)

Stashbot added a comment.Feb 14 2024, 7:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T19:59:14Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "a"} and A:restbase and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Feb 14 2024, 8:30 PM
gerritbot added a comment.Feb 14 2024, 8:44 PM

Change 1003526 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra: install git-fat to satisfy scap requirement

https://gerrit.wikimedia.org/r/1003526

gerritbot added a project: Patch-For-Review.Feb 14 2024, 8:44 PM
Stashbot added a comment.Feb 14 2024, 8:52 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T20:52:28Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching P{P:cassandra%rack = "a"} and A:restbase and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002

Eevans added a subscriber: MoritzMuehlenhoff.Feb 14 2024, 8:54 PM

I attempted a scap deploy for the restbase cluster (after updating the list of targets), but that failed because git-fat was missing. Installing git-fat and rerunning the deploy (and in some cases re-rerunning it with -f after some jars still failed to hydrate), ultimately worked. After a Cassandra restart, logging messages are now showing in OpenSearch.

Ostensibly the solution then would be to have Puppet ensure it's installed, but I was surprised to see that it still requires python2:

$ apt-cache depends git-fat
git-fat
  Depends: <python:any>
    python-is-python2
  Depends: <git-core>
    git
  Depends: rsync
$

@MoritzMuehlenhoff I know we're trying to be done with python2 (and that python-is-python2 is something of a hack), is there an alternative I should be aware of? With debmonitor showing 421 installs I'm guessing not, but thought I should ask. :)

Stashbot added a comment.Feb 14 2024, 8:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T20:57:03Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "b"} and A:restbase and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 14 2024, 9:36 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T21:36:42Z] <eevans@cumin1002> END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching P{P:cassandra%rack = "b"} and A:restbase and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 14 2024, 9:41 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T21:41:21Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase1032.eqiad.wmnet: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 14 2024, 9:52 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T21:51:59Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase1032.eqiad.wmnet: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 14 2024, 10:13 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T22:13:16Z] <urandom> restarting Cassandra: restbase/codfw, row b — T353550

Stashbot added a comment.Feb 14 2024, 10:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T22:39:30Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "c"} and A:restbase and A:codfw: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 14 2024, 11:32 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-14T23:32:27Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching P{P:cassandra%rack = "c"} and A:restbase and A:codfw: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 12:45 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T00:45:22Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "d"} and A:restbase and A:codfw: Restart to pickup logging jars — T353550 - eevans@cumin1002

gerritbot added a comment.Feb 15 2024, 1:26 AM

Change 1003576 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/software/logstash-logback-encoder@master] Updated deployement targets

https://gerrit.wikimedia.org/r/1003576

gerritbot added a comment.Feb 15 2024, 1:32 AM

Change 1003576 merged by Eevans:

[operations/software/logstash-logback-encoder@master] Updated deployement targets

https://gerrit.wikimedia.org/r/1003576

Stashbot added a comment.Feb 15 2024, 1:37 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T01:37:24Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching P{P:cassandra%rack = "d"} and A:restbase and A:codfw: Restart to pickup logging jars — T353550 - eevans@cumin1002

MoritzMuehlenhoff added a comment.Feb 15 2024, 1:24 PM
In T353550#9544270, @Eevans wrote:

@MoritzMuehlenhoff I know we're trying to be done with python2 (and that python-is-python2 is something of a hack), is there an alternative I should be aware of? With debmonitor showing 421 installs I'm guessing not, but thought I should ask. :)

Noone has currently taken on the work to port git-fat to Python 3,we have https://phabricator.wikimedia.org/T279509 to track this. As workaround we can still enable it for now (I'll followup on the patch). But there needs to be a solution eventually, as in Bookworm Python 2 is completely gone.

gerritbot added a comment.Feb 15 2024, 4:35 PM

Change 1003526 merged by Eevans:

[operations/puppet@production] cassandra: install git-fat to satisfy scap requirement

https://gerrit.wikimedia.org/r/1003526

Stashbot added a comment.Feb 15 2024, 4:43 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T16:43:01Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@162f72f] (aqs): Deploying to updated target list — T353550

Stashbot added a comment.Feb 15 2024, 4:43 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T16:43:38Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@162f72f] (aqs): Deploying to updated target list — T353550 (duration: 00m 37s)

Stashbot added a comment.Feb 15 2024, 4:45 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T16:45:12Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@162f72f] (cassandra-dev): Deploying to updated target list — T353550

Stashbot added a comment.Feb 15 2024, 4:45 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T16:45:27Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@162f72f] (cassandra-dev): Deploying to updated target list — T353550 (duration: 00m 15s)

Stashbot added a comment.Feb 15 2024, 4:45 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T16:45:55Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@162f72f] (ml-cache): Deploying to updated target list — T353550

Stashbot added a comment.Feb 15 2024, 4:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T16:46:10Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@162f72f] (ml-cache): Deploying to updated target list — T353550 (duration: 00m 15s)

Stashbot added a comment.Feb 15 2024, 4:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T16:46:18Z] <eevans@deploy2002> Started deploy [cassandra/logstash-logback-encoder@162f72f] (sessionstore): Deploying to updated target list — T353550

Stashbot added a comment.Feb 15 2024, 4:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T16:46:33Z] <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@162f72f] (sessionstore): Deploying to updated target list — T353550 (duration: 00m 15s)

Stashbot added a comment.Feb 15 2024, 4:51 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T16:51:40Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "rack1"} and A:aqs and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 5:24 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T17:24:41Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching P{P:cassandra%rack = "rack1"} and A:aqs and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002

Eevans added a comment.Feb 15 2024, 5:27 PM

The jars are deployed everywhere, all that remains are restarts:

  • restbase
  • aqs
  • cassandra-dev
  • sessionstore
  • ml-cache
Maintenance_bot removed a project: Patch-For-Review.Feb 15 2024, 5:31 PM
Stashbot added a comment.Feb 15 2024, 7:31 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T19:31:45Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "rack2"} and A:aqs and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T20:06:43Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching P{P:cassandra%rack = "rack2"} and A:aqs and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 8:08 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T20:08:30Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "rack3"} and A:aqs and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 8:41 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T20:41:48Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching P{P:cassandra%rack = "rack3"} and A:aqs and A:eqiad: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 8:47 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T20:47:32Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "a_c"} and A:aqs and A:codfw: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 9:20 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T21:20:46Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching P{P:cassandra%rack = "a_c"} and A:aqs and A:codfw: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 9:26 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T21:26:02Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "b_e"} and A:aqs and A:codfw: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 9:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T21:59:48Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching P{P:cassandra%rack = "b_e"} and A:aqs and A:codfw: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 9:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T21:59:56Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "c_f"} and A:aqs and A:codfw: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 10:34 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T22:34:13Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching P{P:cassandra%rack = "c_f"} and A:aqs and A:codfw: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 10:47 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T22:47:18Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:sessionstore: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 15 2024, 11:26 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T23:26:55Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:sessionstore: Restart to pickup logging jars — T353550 - eevans@cumin1002

Eevans added a comment.Feb 16 2024, 12:04 AM
In T353550#9547707, @Eevans wrote:

The jars are deployed everywhere, all that remains are restarts:

  • restbase
  • aqs
  • cassandra-dev
  • sessionstore
  • ml-cache
Stashbot added a comment.Feb 16 2024, 12:06 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-16T00:06:00Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:cassandra-dev: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 16 2024, 12:28 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-16T00:27:59Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:cassandra-dev: Restart to pickup logging jars — T353550 - eevans@cumin1002

Stashbot added a comment.Feb 16 2024, 12:49 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-16T00:49:13Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache: Restart to pickup logging jars — T353550 - eevans@cumin1002

Eevans added a comment.Feb 16 2024, 1:10 AM
In T353550#9546005, @MoritzMuehlenhoff wrote:
In T353550#9544270, @Eevans wrote:

@MoritzMuehlenhoff I know we're trying to be done with python2 (and that python-is-python2 is something of a hack), is there an alternative I should be aware of? With debmonitor showing 421 installs I'm guessing not, but thought I should ask. :)

Noone has currently taken on the work to port git-fat to Python 3,we have https://phabricator.wikimedia.org/T279509 to track this. As workaround we can still enable it for now (I'll followup on the patch). But there needs to be a solution eventually, as in Bookworm Python 2 is completely gone.

I've opened T357739 to track a permanent fix for this.

Stashbot added a comment.Feb 16 2024, 1:26 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-16T01:26:17Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache: Restart to pickup logging jars — T353550 - eevans@cumin1002

Eevans closed this task as Resolved.Feb 16 2024, 2:04 AM
Eevans claimed this task.
In T353550#9549229, @Eevans wrote:
In T353550#9547707, @Eevans wrote:

The jars are deployed everywhere, all that remains are restarts:

  • restbase
  • aqs
  • cassandra-dev
  • sessionstore
  • ml-cache

Done.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits