Page MenuHomePhabricator
Create Task
Maniphest T302494

The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Followup of T302396.

The thanos-swift cluster is S‌3 compatible so we should use that instead of the native swift client which we customized to implement tmp auth and has been removed from the official flink distribution: https://issues.apache.org/jira/browse/FLINK-21819.

Migration plan:

  • Preflight checks: Test that s3 actually fixes T302396
    • deploy a new image with s3&swift enabled to codfw
    • save a savepoint to s3 from the updater running in yarn and stop it (requires restarting this session cluster with S3 enabled)
    • start the application from this s3 savepoint
  • Migrate jobs from swift to s3

AC:

  • W[DC]QS Streaming Updater is using thanos-swift through the s3 protocol

Details

SubjectRepoBranchLines +/-
Replace Swift native API with S3operations/deployment-chartsmaster+3 -3
dse-k8s: raise memory for rdf-streaming-updateroperations/deployment-chartsmaster+25 -0
rdf-streaming-updater: Increase memory limit from 2 to 4GiBoperations/deployment-chartsmaster+24 -2
[wdqs] adapt updateQueryServiceLag...operations/puppetproduction+1 -1
Switch to S3 scheme by default instead of swiftwikidata/query/deploymaster+7 -7
flink-session-cluster: fix swift API key for s3operations/deployment-chartsmaster+2 -2
flink-session-cluster: add thanos S3 configoperations/deployment-chartsmaster+8 -2
Add S3 support with s3-prestowikidata/query/flink-rdf-streaming-updatermaster+3 -0
Customize query in gerrit

Related Objects

Event Timeline

dcausse created this task.Feb 24 2022, 2:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 24 2022, 2:11 PM
Maintenance_bot added a project: Wikidata.Feb 24 2022, 2:45 PM
gerritbot added a comment.Feb 25 2022, 10:37 AM

Change 766072 had a related patch set uploaded (by ZPapierski; author: ZPapierski):

[wikidata/query/flink-rdf-streaming-updater@master] Replace Swift tempauth with S3

https://gerrit.wikimedia.org/r/766072

gerritbot added a project: Patch-For-Review.Feb 25 2022, 10:37 AM
gerritbot added a comment.Feb 25 2022, 3:52 PM

Change 766123 had a related patch set uploaded (by ZPapierski; author: ZPapierski):

[operations/deployment-charts@master] Replace Swift native API with S3

https://gerrit.wikimedia.org/r/766123

Gehel triaged this task as High priority.Feb 28 2022, 4:34 PM
Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board.
Gehel added a project: Discovery-Search (Current work).
TJones updated the task description. (Show Details)Feb 28 2022, 4:35 PM
Gehel mentioned this in T302396: Investigate EOFException when performing the first checkpoint after restoring from a savepoint.Feb 28 2022, 4:54 PM
Gehel moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.
Gehel set the point value for this task to 3.
Gehel assigned this task to RKemper.Feb 28 2022, 4:57 PM
Gehel moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.
dcausse updated the task description. (Show Details)Mar 7 2022, 10:50 AM
Gehel moved this task from Needs review to In Progress on the Discovery-Search (Current work) board.Mar 7 2022, 4:25 PM
gerritbot added a comment.Mar 8 2022, 4:45 PM

Change 769075 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] flink-session-cluster: add thanos S3 config

https://gerrit.wikimedia.org/r/769075

gerritbot added a comment.Mar 8 2022, 5:10 PM

Change 766072 merged by jenkins-bot:

[wikidata/query/flink-rdf-streaming-updater@master] Add S3 support with s3-presto

https://gerrit.wikimedia.org/r/766072

gerritbot added a comment.Mar 10 2022, 1:17 PM

Change 769075 merged by jenkins-bot:

[operations/deployment-charts@master] flink-session-cluster: add thanos S3 config

https://gerrit.wikimedia.org/r/769075

gerritbot added a comment.Mar 10 2022, 1:32 PM

Change 769697 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/deploy@master] Switch to S3 scheme by default instead of swift

https://gerrit.wikimedia.org/r/769697

gerritbot added a comment.Mar 10 2022, 2:08 PM

Change 769699 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] flink-session-cluster: fix swift API key for s3

https://gerrit.wikimedia.org/r/769699

gerritbot added a comment.Mar 10 2022, 2:23 PM

Change 769699 merged by jenkins-bot:

[operations/deployment-charts@master] flink-session-cluster: fix swift API key for s3

https://gerrit.wikimedia.org/r/769699

gerritbot added a comment.Mar 14 2022, 8:52 AM

Change 769697 merged by DCausse:

[wikidata/query/deploy@master] Switch to S3 scheme by default instead of swift

https://gerrit.wikimedia.org/r/769697

gerritbot added a comment.Mar 14 2022, 2:38 PM

Change 770508 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/puppet@production] [wdqs] adapt updateQueryServiceLag...

https://gerrit.wikimedia.org/r/770508

dcausse updated the task description. (Show Details)Mar 14 2022, 2:55 PM
gerritbot added a comment.Mar 14 2022, 9:24 PM

Change 770508 merged by Bking:

[operations/puppet@production] [wdqs] adapt updateQueryServiceLag...

https://gerrit.wikimedia.org/r/770508

Stashbot added a comment.Mar 14 2022, 9:38 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-14T21:38:30Z] <inflatador> T302494 bking@puppetmaster1001 conftool action : set/pooled=true; selector: dnsdisc=wdqs-internal,name=codfw

Stashbot added a comment.Mar 14 2022, 10:03 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-14T22:03:20Z] <inflatador> T302494 bking@puppetmaster1001 depooling eqiad in DNS-discovery for wdqs and wdqs-internal services

bking subscribed.Mar 15 2022, 12:06 AM

Per messages above, we have completely failed over the wdqs and wdqs-internal services from eqiad to codfw.

dcausse updated the task description. (Show Details)Mar 15 2022, 8:54 AM
dcausse updated the task description. (Show Details)Mar 15 2022, 1:14 PM
Stashbot added a comment.Mar 28 2022, 2:46 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-28T14:46:59Z] <inflatador> 'bking@cumin1001 repooling wdqs services in IAD ref T302494'

dcausse mentioned this in T304914: Remove the presto client for swift from the flink image.Mar 29 2022, 8:15 AM
dcausse updated the task description. (Show Details)
dcausse moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.

Moved remaining work in T304914.

dcausse updated the task description. (Show Details)Mar 29 2022, 8:18 AM
Gehel closed this task as Resolved.Mar 29 2022, 1:34 PM
bking mentioned this in T309648: Restore lost index in cloudelastic.Jun 6 2022, 10:32 PM
dcausse mentioned this in T314835: wdqs space usage on thanos-swift.Aug 9 2022, 2:11 PM
gerritbot added a comment.Feb 14 2023, 4:38 PM

Change 889172 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] rdf-streaming-updater: Increase memory alloc from 2 to 3GB

https://gerrit.wikimedia.org/r/889172

gerritbot added a comment.Feb 15 2023, 3:16 PM

Change 889172 merged by Bking:

[operations/deployment-charts@master] rdf-streaming-updater: Increase memory limit from 2 to 4GiB

https://gerrit.wikimedia.org/r/889172

gerritbot added a comment.Feb 23 2023, 3:51 PM

Change 891577 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] dse-k8s: raise memory for rdf-streaming-updater

https://gerrit.wikimedia.org/r/891577

gerritbot added a comment.Feb 27 2023, 2:19 PM

Change 891577 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s: raise memory for rdf-streaming-updater

https://gerrit.wikimedia.org/r/891577

dcausse mentioned this in T331283: [Event Platform] [NEEDS GROOMING] Store Flink HA metadata in Zookeeper.Apr 6 2023, 6:51 AM
gerritbot added a comment.Nov 10 2023, 2:57 PM

Change 766123 abandoned by DCausse:

[operations/deployment-charts@master] Replace Swift native API with S3

Reason:

superseded by other patches

https://gerrit.wikimedia.org/r/766123

Maintenance_bot removed a project: Patch-For-Review.Nov 10 2023, 3:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL