Page MenuHomePhabricator
Create Task
Maniphest T304914

Remove the presto client for swift from the flink image
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

As a maintainer of a flink session cluster I want to stop using the presto client for swift present in the flink image so that I can migrate to newer version of flink since it was removed.

This is a followup of T302494 where we dropped this dependency from the jobs running in the flink session cluster. This task is about dropping this swift client from the image.

Existing flink session clusters rely on this swift client to store their H/A related data (e.g. job jars). This means we must migrate existing clusters to using s3 as a simple drop-in replacement is unlikely to work.

Suggested migration procedure:

  • For codfw
    • route wdqs & wcqs to eqiad only
    • adapt the wikidata maxlag to poll eqiad only
    • stop (with a savepoint) all the jobs (WDQS & WCQS) running on the codfw k8s wikikube cluster 5m
    • undeploy all the k8s deployments under the rdf-streaming-updater namespace (dropping all flink generated configmaps might be necessary by e.g. recreating the k8s namespace) 10m
    • delete the flink_ha_storage folder on the corresponding s3 bucket (rdf-streaming-updater-codfw) TODO: Good time estimate
    • drop presto-swift from https://gerrit.wikimedia.org/g/wikidata/query/flink-rdf-streaming-updater and create a new image zero m
    • adapt the patch generated by PipelineLib when merging the patch above and remove all mentions to swift from deployment-charts (possibly adapting existing patch: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/766123) 10m
    • deploy the chart to the rdf-streaming-updater namespace in codfw (which should be empty) 10m
    • deploy the flink jobs (WCQS & WDQS) from their corresponding savepoints 10m
    • repool codfw & resume polling codfw for wikidata maxlag calculation 10m
  • For eqiad (do all the above replacing eqiad with codfw and vice versa)
  • stop (with a savepoint) all the jobs (WDQS & WCQS) running on the eqiad k8s wikikube cluster 5m
  • undeploy all the k8s deployments under the rdf-streaming-updater namespace (dropping all flink generated configmaps might be necessary by e.g. recreating the k8s namespace) 10m
  • delete the flink_ha_storage folder on the corresponding s3 bucket (rdf-streaming-updater-eqiad) ???
  • Merge & deploy https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/889155 10m
  • deploy the flink jobs (WCQS & WDQS) from their corresponding savepoints 10m

Note that most of this procedure can be tested against the staging cluster (omitting the parts about the routing live traffic and wikidata maxlag)

AC:

  • none of the flink session clusters are using the presto swift client

Details

SubjectRepoBranchLines +/-
rdf-streaming-updater: Use S3 instead of Swift for bucket accessoperations/deployment-chartsmaster+1 -1
deploy: add options for different k8s staging envswikidata/query/deploymaster+5 -2
flink-rdf-streaming-updater: use S3 instead of swiftoperations/deployment-chartsmaster+1 -1
flink-rdf-streaming-updater: use S3 instead of swiftoperations/deployment-chartsmaster+1 -1
flink-rdf-streaming-updater: use S3 instead of swiftoperations/deployment-chartsmaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

• dcausse created this task.Mar 29 2022, 8:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 29 2022, 8:15 AM
• dcausse mentioned this in T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol.Mar 29 2022, 8:17 AM
Maintenance_bot added a project: Wikidata.Mar 29 2022, 8:45 AM
• MPhamWMF triaged this task as Medium priority.Apr 4 2022, 3:44 PM
• MPhamWMF moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.
• dcausse mentioned this in T314835: wdqs space usage on thanos-swift.Aug 9 2022, 2:11 PM
bking subscribed.Aug 22 2022, 4:34 PM
EBernhardson subscribed.Aug 22 2022, 4:38 PM
• dcausse moved this task from Operations/SRE to Current work on the Wikidata-Query-Service board.Dec 19 2022, 10:52 AM
• dcausse added a project: Discovery-Search (Current work).
• dcausse mentioned this in T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model.Jan 24 2023, 2:04 PM
gerritbot added a comment.Jan 31 2023, 3:29 PM

Change 885365 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] flink-rdf-streaming-updater: use S3 instead of swift

https://gerrit.wikimedia.org/r/885365

gerritbot added a project: Patch-For-Review.Jan 31 2023, 3:29 PM
gerritbot added a comment.Jan 31 2023, 3:55 PM

Change 885365 merged by Bking:

[operations/deployment-charts@master] flink-rdf-streaming-updater: use S3 instead of swift

https://gerrit.wikimedia.org/r/885365

Maintenance_bot removed a project: Patch-For-Review.Jan 31 2023, 4:33 PM
gerritbot added a comment.Jan 31 2023, 5:46 PM

Change 885392 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] flink-rdf-streaming-updater: use S3 instead of swift

https://gerrit.wikimedia.org/r/885392

gerritbot added a project: Patch-For-Review.Jan 31 2023, 5:46 PM
gerritbot added a comment.Jan 31 2023, 5:57 PM

Change 885394 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] flink-rdf-streaming-updater: use S3 instead of swift

https://gerrit.wikimedia.org/r/885394

gerritbot added a comment.Jan 31 2023, 5:57 PM

Change 885392 abandoned by Bking:

[operations/deployment-charts@master] flink-rdf-streaming-updater: use S3 instead of swift

Reason:

unplanned merge conflict, discarding

https://gerrit.wikimedia.org/r/885392

gerritbot added a comment.Jan 31 2023, 6:10 PM

Change 885394 merged by Bking:

[operations/deployment-charts@master] flink-rdf-streaming-updater: use S3 instead of swift

https://gerrit.wikimedia.org/r/885394

Maintenance_bot removed a project: Patch-For-Review.Jan 31 2023, 6:31 PM
gerritbot added a comment.Jan 31 2023, 9:21 PM

Change 885425 had a related patch set uploaded (by Bking; author: Bking):

[wikidata/query/deploy@master] deploy: add options for different k8s staging envs

https://gerrit.wikimedia.org/r/885425

gerritbot added a project: Patch-For-Review.Jan 31 2023, 9:21 PM
gerritbot added a comment.Feb 6 2023, 2:20 PM

Change 885425 abandoned by Bking:

[wikidata/query/deploy@master] deploy: add options for different k8s staging envs

Reason:

not needed

https://gerrit.wikimedia.org/r/885425

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2023, 2:31 PM
bking claimed this task.Feb 6 2023, 4:43 PM
• MPhamWMF removed bking as the assignee of this task.Feb 6 2023, 4:43 PM
• MPhamWMF assigned this task to bking.
• MPhamWMF set the point value for this task to 5.
• MPhamWMF moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.
bking updated the task description. (Show Details)Feb 7 2023, 3:56 PM

Reminder to self: better estimate on how long it would take to delete flink HA storage path from swift.

gerritbot added a comment.Feb 14 2023, 3:44 PM

Change 889155 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] rdf-streaming-updater: Use S3 instead of Swift for bucket access

https://gerrit.wikimedia.org/r/889155

gerritbot added a project: Patch-For-Review.Feb 14 2023, 3:44 PM
Stashbot added a comment.Feb 14 2023, 3:50 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-14T15:50:33Z] <inflatador> bking@deploy1002 'deploying rdf-streaming-updater prod eqiad T304914'

bking added a comment.Feb 14 2023, 4:43 PM

We attempted the procedure today, but WCQS cannot create a savepoint without hitting an OOM condition
kubectl get pods -o json | jq '.items[].status.containerStatuses[0].lastState'

{
  "terminated": {
    "containerID": "docker://bdb8d22ec09eecf26fbed2c97453a3323bd33e7fc105497f41910665aa9bbe8a",
    "exitCode": 137,
    "finishedAt": "2023-02-14T16:00:01Z",
    "reason": "OOMKilled",
    "startedAt": "2023-01-24T11:01:18Z"
  }
}
{
  "terminated": {
    "containerID": "docker://e4c60e609170c01717851a6eaf05652f2080ee83fb3eb11a33c3f87fcf293710",
    "exitCode": 137,
    "finishedAt": "2023-02-14T16:00:50Z",
    "reason": "OOMKilled",
    "startedAt": "2023-01-24T11:01:19Z"
  }
}
{
  "terminated": {
    "containerID": "docker://dc24a25acac9d32ebd80c4957c6dfe0d0e9bcd010e142dadcff846d6dc66e93b",
    "exitCode": 137,
    "finishedAt": "2023-02-14T16:03:05Z",
    "reason": "OOMKilled",
    "startedAt": "2023-01-24T11:01:19Z"
  }
}
{
  "terminated": {
    "containerID": "docker://7a229041ff346bcdf9c75a0a0813252645845cafb379a8b6af08f3809280d064",
    "exitCode": 137,
    "finishedAt": "2023-02-14T16:00:27Z",
    "reason": "OOMKilled",
    "startedAt": "2023-01-24T11:01:18Z"
  }
}
bking updated the task description. (Show Details)Feb 22 2023, 3:11 PM
gerritbot added a comment.Feb 22 2023, 3:42 PM

Change 889155 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: Use S3 instead of Swift for bucket access

https://gerrit.wikimedia.org/r/889155

Maintenance_bot removed a project: Patch-For-Review.Feb 22 2023, 4:11 PM
bking added a comment.Feb 23 2023, 3:05 PM

This is complete...moving to "needs reporting"

bking moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Feb 23 2023, 3:05 PM
Gehel moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Feb 28 2023, 7:43 PM
Gehel closed this task as Resolved.Mar 10 2023, 2:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits