Page MenuHomePhabricator

Setup Fix Suggester Bot on Toolforge
Closed, ResolvedPublic

Description

I'm trying to set up Fix Suggester Bot on Toolforge (for some background, see T209149: Have linters/tests results show up as comments in files on gerrit).

It functions in a similar way to the newcomer welcome suggestion bot, in that it's subscribing to gerrit stream events and then posting comments on Gerrit patchsets.

High level setup:

  • bin/console fix-suggester-bot:subscribe:gerrit connects via SSH to Gerrit using the fixsuggesterbot user and subscribes to patchset-created events. These events are placed into a Redis queue.
  • bin/console messenger:consume gerrit_async consumes the messages, and invokes the various commands that end up posting a comment in Gerrit

I'm trying to get things working on the default php version after running become fixsuggesterbot. I've created an SSH key and added it to ssh-agent so I'm able to connect to gerrit via SSH when I do bin/console fix-suggester-bot:subscribe:gerrit. But now I'm stuck at this step:

bin/console messenger:setup-transports
The redis transport requires php-redis 4.3.0

{"message":"Error thrown while running command \"messenger:setup-transports\". Message: \"The redis transport requires php-redis 4.3.0 or higher.\"","context":{"exception":{"class":"Symfony\\Component\\Messenger\\Exception\\LogicException","message":"The redis transport requires php-redis 4.3.0 or higher.","code":0,"file":"/mnt/nfs/labstore-secondary-tools-project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/redis-messenger/Transport/Connection.php:67"},"command":"messenger:setup-transports","message":"The redis transport requires php-redis 4.3.0 or higher."},"level":500,"level_name":"CRITICAL","channel":"console","datetime":"2021-08-27T09:03:31.823390+00:00","extra":{}}

Based on that, my first question is if we could update the php-redis library used with the default PHP 7.2 setup on Toolforge.

My second question is if I should be using the Kubernetes backend (probably I should be). If I use the web service start command with php7.4 (creating an unnecessary public_html directory in the root of the tool; there's no web interface/API for this tool), then I see that php-redis is a new enough version in the php7.4 setup. But, then I'm stuck because it looks like ssh and ssh-agent don't exist.

Overall, I could use some advice on which direction to go in next:

  • I should probably be using Kubernetes; should I add ssh/ssh-agent to the image?
  • Symfony Messenger says I should use something like Supervisor to ensure that bin/console messenger:consume gerrit_async is always running (well, not running forever, but that a new process is started when the old one finishes), and I probably also need something similar to use so that bin/console fix-suggester-bot:subscribe:gerrit starts up again in the event of an SSH disconnect. Are these both things that could/should be managed with Kubernetes config?

Details

TitleReferenceAuthorSource BranchDest Branch
SSH: Use private key instead of ssh-agentkharlan/fix-suggester-bot!8kharlanuse-private-key-sshmain
Customize query in GitLab

Event Timeline

taavi subscribed.

Based on that, my first question is if we could update the php-redis library used with the default PHP 7.2 setup on Toolforge.

Unlikely. We use extensions packaged by Debian, and the Stretch grid uses php-redis 3.1.1; updating to 4.3 would need suitable packages for Stretch and would likely be a breaking change for some tools which we would not like to do.

My second question is if I should be using the Kubernetes backend (probably I should be)

Ideally yes. T285944: Toolforge: beta phase for the new jobs framework might be of interest if this is not a webservice.

I should probably be using Kubernetes; should I add ssh/ssh-agent to the image?

We would like to keep the base container images as small as possible, but adding ssh might be reasonable given its widespread use cases (T258841). Custom containers are not yet supported (there is some work related to it in progress, but not ready yet).

Are these both things that could/should be managed with Kubernetes config?

Yes, or via the Toolforge jobs framework which abstracts the required k8s config.

In T289850#7314470, @Majavah wrote:

Based on that, my first question is if we could update the php-redis library used with the default PHP 7.2 setup on Toolforge.

Unlikely. We use extensions packaged by Debian, and the Stretch grid uses php-redis 3.1.1; updating to 4.3 would need suitable packages for Stretch and would likely be a breaking change for some tools which we would not like to do.

Ack

My second question is if I should be using the Kubernetes backend (probably I should be)

Ideally yes. T285944: Toolforge: beta phase for the new jobs framework might be of interest if this is not a webservice.

Oh, that looks like exactly what I need. Thanks for the pointer!

I should probably be using Kubernetes; should I add ssh/ssh-agent to the image?

We would like to keep the base container images as small as possible, but adding ssh might be reasonable given its widespread use cases (T258841). Custom containers are not yet supported (there is some work related to it in progress, but not ready yet).

Yeah I don't think I need anything custom, just ssh and ssh-agent.

Are these both things that could/should be managed with Kubernetes config?

Yes, or via the Toolforge jobs framework which abstracts the required k8s config.

Ack, jobs framework looks ideal.

OK, I removed the ssh-agent implementation in my app and used a private key, so with phpseclib (pure PHP implementation of SSH2) I don't need openssh-client in the base image.

Now I'm stuck here when I try to add a gerrit stream event into a Redis stream:

ERR unknown command 'XADD'
In Connection.php line 498:

  [Symfony\Component\Messenger\Exception\TransportException]
  ERR unknown command 'XADD'


Exception trace:
  at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/redis-messenger/Transport/Connection.php:498
 Symfony\Component\Messenger\Bridge\Redis\Transport\Connection->add() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/redis-messenger/Transport/RedisSender.php:45
 Symfony\Component\Messenger\Bridge\Redis\Transport\RedisSender->send() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/redis-messenger/Transport/RedisTransport.php:66
 Symfony\Component\Messenger\Bridge\Redis\Transport\RedisTransport->send() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/messenger/Middleware/SendMessageMiddleware.php:69
 Symfony\Component\Messenger\Middleware\SendMessageMiddleware->handle() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/messenger/Middleware/FailedMessageProcessingMiddleware.php:34
 Symfony\Component\Messenger\Middleware\FailedMessageProcessingMiddleware->handle() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/messenger/Middleware/DispatchAfterCurrentBusMiddleware.php:68
 Symfony\Component\Messenger\Middleware\DispatchAfterCurrentBusMiddleware->handle() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/messenger/Middleware/RejectRedeliveredMessageMiddleware.php:48
 Symfony\Component\Messenger\Middleware\RejectRedeliveredMessageMiddleware->handle() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/messenger/Middleware/AddBusNameStampMiddleware.php:37
 Symfony\Component\Messenger\Middleware\AddBusNameStampMiddleware->handle() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/messenger/MessageBus.php:77
 Symfony\Component\Messenger\MessageBus->dispatch() at /data/project/fixsuggesterbot/fix-suggester-bot/src/Command/GerritStreamSubscriber.php:102
 App\Command\GerritStreamSubscriber->App\Command\{closure}() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php:2690
 phpseclib3\Net\SSH2->exec() at /data/project/fixsuggesterbot/fix-suggester-bot/src/Command/GerritStreamSubscriber.php:104
 App\Command\GerritStreamSubscriber->execute() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/console/Command/Command.php:299
 Symfony\Component\Console\Command\Command->run() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/console/Application.php:996
 Symfony\Component\Console\Application->doRunCommand() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/framework-bundle/Console/Application.php:96
 Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/console/Application.php:295
 Symfony\Component\Console\Application->doRun() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/framework-bundle/Console/Application.php:82
 Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/console/Application.php:167
 Symfony\Component\Console\Application->run() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/symfony/runtime/Runner/Symfony/ConsoleApplicationRunner.php:56
 Symfony\Component\Runtime\Runner\Symfony\ConsoleApplicationRunner->run() at /data/project/fixsuggesterbot/fix-suggester-bot/vendor/autoload_runtime.php:35
 require_once() at /data/project/fixsuggesterbot/fix-suggester-bot/bin/console:11

OK, I removed the ssh-agent implementation in my app and used a private key, so with phpseclib (pure PHP implementation of SSH2) I don't need openssh-client in the base image.

Now I'm stuck here when I try to add a gerrit stream event into a Redis stream:

ERR unknown command 'XADD'

Redis streams, and XADD were added on redis >=5 (https://redis.io/commands/XADD), and we are running 3.2.6, so until it's upgraded (might take more than a quarter as it's not in the immediate tasks pile) you might want to use a different way of storing the data there.

Now I'm stuck here when I try to add a gerrit stream event into a Redis stream:

ERR unknown command 'XADD'

That's a new Redis feature added in Redis 5.0. The current tools-redis server is likely a bit outdated, and I've been working on upgrading it in multiple ways (T153810, T278541). I unfortunately don't have any accurate timelines, but I'm hoping to pick it up fairly soon again now that Debian 11 is out.

In T289850#7321030, @Majavah wrote:

Now I'm stuck here when I try to add a gerrit stream event into a Redis stream:

ERR unknown command 'XADD'

That's a new Redis feature added in Redis 5.0. The current tools-redis server is likely a bit outdated, and I've been working on upgrading it in multiple ways (T153810, T278541). I unfortunately don't have any accurate timelines, but I'm hoping to pick it up fairly soon again now that Debian 11 is out.

Thanks, I figured as much. For now, I will switch to using a database as a queue.

kostajh claimed this task.
kostajh added a project: Fix-Suggester-Bot.
In T289850#7321030, @Majavah wrote:

Now I'm stuck here when I try to add a gerrit stream event into a Redis stream:

ERR unknown command 'XADD'

That's a new Redis feature added in Redis 5.0. The current tools-redis server is likely a bit outdated, and I've been working on upgrading it in multiple ways (T153810, T278541). I unfortunately don't have any accurate timelines, but I'm hoping to pick it up fairly soon again now that Debian 11 is out.

Thanks, I figured as much. For now, I will switch to using a database as a queue.

I got it working with the database queue and with the new jobs framework (docs are here: https://wikitech.wikimedia.org/w/index.php?title=Tool:Fix_Suggester_Bot). Thanks everyone for your help!