Page MenuHomePhabricator

Upgrade event platform related VMs in deployment-prep to Debian bullsye (or buster)
Closed, ResolvedPublic

Description

deployment-prep, AKA beta, has several Event Platform related VMs that run Debian Stretch. Debian Stretch. All Stretch VMs will be shut down on May 1st.

We should spin up new VMs using recent Debian OS and change relevant puppet and hiera code to move the services.

Services include:

  • kafka main
  • kafka jumbo
  • eventgate
  • eventstreams -> bullsye
  • zookeeper -> bullsye
  • schema-beta -> bullsye

Event Timeline

EChetty triaged this task as High priority.
EChetty moved this task from Apache Iceberg Migration to Ops Week on the Data-Engineering board.
Ottomata updated the task description. (Show Details)

The hiera for the kafka jumbo and main clusters in deployment-prep seems to be in a strange state. In this change @Majavah added new kafka_clusters that duplicated the existent kafka clusters that were already declared there.

@Majavah, what was the reason for that change? Can I clean it all up so that only the actual clusters remain?

From @Majavah:

./2022-03-31.log:61:[16:44:41] <taavi> ottomata: I was probably trying to unbreak something that was broken at the time, phab activity matching the timestamps might give more insight. sorry I don't have more details :/ feel free to hack it as much as you need as long as things don't break

Change 785866 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] apt distributions - Add thirdparty/confluent componenet to bullsye-wikimedia

https://gerrit.wikimedia.org/r/785866

Change 785866 merged by Ottomata:

[operations/puppet@production] apt distributions - Add thirdparty/confluent componenet to bullsye-wikimedia

https://gerrit.wikimedia.org/r/785866

Change 785904 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] profile::docker::engine - Allow not setting version

https://gerrit.wikimedia.org/r/785904

Change 785904 merged by Ottomata:

[operations/puppet@production] profile::docker::engine - Allow not setting version

https://gerrit.wikimedia.org/r/785904

Change 785905 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] profile::docker::engine - make $version Optional

https://gerrit.wikimedia.org/r/785905

Change 785905 merged by Ottomata:

[operations/puppet@production] profile::docker::engine - make $version Optional

https://gerrit.wikimedia.org/r/785905

Change 785910 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] docker engine - pick smart default for package name

https://gerrit.wikimedia.org/r/785910

Change 785910 merged by Ottomata:

[operations/puppet@production] docker engine - pick smart default for package name

https://gerrit.wikimedia.org/r/785910

Update: all nodes have been replaced with either bullsye or buster!

Only one problem remains. The webproxes to deployment-eventgate-4 don't seem to be working. The services there work on the internal node ports, but the external https proxy url doesn't finish responding.

curl -v https://intake-analytics-beta.wmflabs.org/_info

just hangs...

Ah! I just needed to add the correct firewall security group. It works!

Change 789187 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] Update changeprop beta kafka broker hostnames

https://gerrit.wikimedia.org/r/789187

Change 789187 merged by jenkins-bot:

[operations/deployment-charts@master] Update changeprop beta kafka broker hostnames

https://gerrit.wikimedia.org/r/789187