Page MenuHomePhabricator

[IDEA] Backup bot for morebots
Closed, DeclinedPublic

Description

Earlier today banbot decided it was going to go ahead and ban morebots and kick the bot as well, to prevent missing logs in channels like operations i suggest making a bot that will take over morebots job if something similar happens again (preferably not hosted on the same server if possible).

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Zppix moved this task from Triage to Backlog on the Cloud-Services board.
Peachey88 added subscribers: Stashbot, Peachey88.

That would cause double logging.

Just ask one of the ops to restart the bot, on the rare instances this happens.

Proposing this task to be declined.

(Also, @Stashbot captures and logs them all as well)

Must of misunderstood my idea as in it wouldnt actually log until the backup detects that morebots is offline or not in channel for atleast 3-4 minutes

@Zppix: #RfC means "that a task needs input and discussion in the style of a request for comments". If you think that such a broader discussion (outside of this task) is needed in this case, could you elaborate why? Thanks :)

Ah, anyways i think @Stashbot replaced morebots, considering it is logging to sal on wikitech according to irc chat http://ur1.ca/edq22 (see around5:26 utc -5)

Per T156929, Adminbot has been transitioned to Stashbot. Stashbot no longer relies on the wiki to save its messages (they're saved to ElasticSearch and also exposed via a https://tools.wmflabs.org/sal/). However, per this task, Stashbot does still depend on being in the relevant Freenode channels. A backup bot could be interesting. E.g. it could have two connections to Freenode (stashbot and stashbot_?) and effectively join each channel twice. It could automatically try to re-join or re-connect as needed and take care not to process the same message twice.

Anyway, not sure how high the need for it is, but might be interesting regardless. Alternatively, perhaps stashbot could have a restricted web interface that allows one to feed and retroactively process part of a log if it was out of a channel for a while.

I'm not familiar with Stashbot, so perhaps my concerns are unfounded, but if we are use Elasticsearch as a primary data store I wonder if we have appropriate regular backups of the data being taken?

I'm not familiar with Stashbot, so perhaps my concerns are unfounded, but if we are use Elasticsearch as a primary data store I wonder if we have appropriate regular backups of the data being taken?

The Elasticsearch index storing SAL messages is not explicitly backed up currently. The index has 2 replicas (3 copies total) on separate Labs VMs. I'm actually not sure if all three VMs are on different labvirt hosts or not. All messages are also written to wikitech (4th copy) although in a format that is not really easy to reimport into the Elasticsearch index.

Anyway, not sure how high the need for it is, but might be interesting regardless. Alternatively, perhaps stashbot could have a restricted web interface that allows one to feed and retroactively process part of a log if it was out of a channel for a while.

I've thought about something like this before, but not actually done any work towards it. In the past I have backfilled missing !log messages from both the logs from my own bouncer and from wikitech (before Stashbot took over that role too). This has been done by processing the raw data into the json docs expected by the Elasticsearch index and then directly inserting the docs into the backing index. Backfilling the on-wiki SAL has been done with similar manual wiki edits.

Stashbot has a cloak, so it is less likely to get kicked by banbot too I think.

Just cleaning up some tasks ive authored, are we done with this task are we still discussing here?