Page MenuHomePhabricator
Create Task
Maniphest T336380

AQS 2.0: Media Analytics Service - Deploy to staging and production
Closed, ResolvedPublic
Actions

Description

Background/Goal

Key task for Epic T288303: AQS 2.0: Media Analytics Service

Required:

  • Add blubber and pipeline configuration to geo-analytics repo (blubber.yaml and config.yaml) (review)
  • Integrate media-analytics image building in CI pipeline (review 1, review 2)
  • Add media-analytics Cassandra user (review)
  • Add k8s secrets
  • Create deployment-charts helmfile.d configuration entries (review)
  • Run smoke tests against staging and certify service is working properly
  • Add service catalogue entry for media-analytics (review)
  • Deploy media-analytics to LVS in state service_setup
  • Move media-analytics lvs_setup in LVS
  • Move media-analytics production in LVS
  • Add media-analytics to the API Gateway
  • Route public requests via ATS

Done is

QA notes

Artifacts & Resources

Templates, instructions, and other useful documentation

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
media-analytics: bump imageoperations/deployment-chartsmaster+1 -1
trafficserver: route requests to mediarequests serviceoperations/puppetproduction+1 -0
service: move geo-analytics and media-analytics to productionoperations/puppetproduction+2 -2
service: add media-analytics service entryoperations/puppetproduction+14 -0
wmnet: add geo-analytics and media-analytics ingress recordsoperations/dnsmaster+6 -0
helmfile: add entries and namespace for media-analytics serviceoperations/deployment-chartsmaster+120 -0
kubernetes: add users for media_analytics service, cassandra configoperations/puppetproduction+12 -0
zuul: add media-analytics configurationintegration/configmaster+10 -0
jjb: add media-analytics serviceintegration/configmaster+15 -0
Add deployment filesgenerated-data-platform/aqs/media-analyticsmain+48 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

SGupta-WMF created this task.May 10 2023, 1:40 PM
JArguello-WMF mentioned this in T288303: AQS 2.0: Media Analytics Service.Jun 7 2023, 7:17 PM
JArguello-WMF assigned this task to hnowlan.Jun 8 2023, 12:54 PM
JArguello-WMF updated the task description. (Show Details)
JArguello-WMF edited projects, added AQS2.0 (Sprint 10); removed API Platform (AQS 2.0 Roadmap).
JArguello-WMF added subscribers: • VirginiaPoundstone, • codebug, • FJoseph-WMF.
Sfaci moved this task from Sprint 10 to Current Data Products Sprint on the AQS2.0 board.Aug 22 2023, 2:18 PM
Sfaci edited projects, added AQS2.0, Test Kitchen (Sprint 00); removed AQS2.0 (Sprint 10).
Sfaci moved this task from Sprint Backlog to In Process on the Test Kitchen (Sprint 00) board.Aug 22 2023, 2:56 PM
Sfaci updated the task description. (Show Details)Aug 22 2023, 3:06 PM
Sfaci removed subscribers: • FJoseph-WMF, • codebug.
gerritbot added a comment.Aug 22 2023, 3:13 PM

Change 951525 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[generated-data-platform/aqs/media-analytics@main] Add deployment files

https://gerrit.wikimedia.org/r/951525

gerritbot added a project: Patch-For-Review.Aug 22 2023, 3:13 PM
Sfaci updated the task description. (Show Details)Aug 22 2023, 3:13 PM
gerritbot added a comment.Aug 22 2023, 4:16 PM

Change 951539 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[integration/config@master] jjb: add media-analytics service

https://gerrit.wikimedia.org/r/951539

gerritbot added a comment.Aug 22 2023, 4:16 PM

Change 951540 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[integration/config@master] zuul: add media-analytics configuration

https://gerrit.wikimedia.org/r/951540

hnowlan updated the task description. (Show Details)Aug 22 2023, 4:17 PM

Which tables will the media_analytics cassandra user need access to?

gerritbot added a comment.Aug 22 2023, 4:53 PM

Change 951544 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] helmfile: add entries and namespace for media-analytics service

https://gerrit.wikimedia.org/r/951544

hnowlan updated the task description. (Show Details)Aug 22 2023, 4:54 PM
gerritbot added a comment.Aug 22 2023, 5:15 PM

Change 951547 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] kubernetes: add users for media_analytics service

https://gerrit.wikimedia.org/r/951547

Sfaci subscribed.Aug 23 2023, 7:42 AM

@hnowlan the following are the tables media-analytics needs access to:

  • local_group_default_T_mediarequest_per_referer.data
  • local_group_default_T_mediarequest_per_file.data
  • local_group_default_T_mediarequest_top_files.data
hnowlan updated the task description. (Show Details)Aug 23 2023, 11:18 AM
gerritbot added a comment.Aug 23 2023, 11:30 AM

Change 951901 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] service: add media-analytics service entry

https://gerrit.wikimedia.org/r/951901

hnowlan updated the task description. (Show Details)Aug 23 2023, 11:30 AM
gerritbot added a comment.Aug 23 2023, 12:57 PM

Change 951525 merged by Santiago Faci:

[generated-data-platform/aqs/media-analytics@main] Add deployment files

https://gerrit.wikimedia.org/r/951525

Sfaci updated the task description. (Show Details)Aug 23 2023, 12:58 PM
Sfaci updated the task description. (Show Details)
gerritbot added a comment.Aug 23 2023, 1:14 PM

Change 951539 merged by jenkins-bot:

[integration/config@master] jjb: add media-analytics service

https://gerrit.wikimedia.org/r/951539

gerritbot added a comment.Aug 23 2023, 1:14 PM

Change 951540 merged by jenkins-bot:

[integration/config@master] zuul: add media-analytics configuration

https://gerrit.wikimedia.org/r/951540

Stashbot added a comment.Aug 23 2023, 1:17 PM

Mentioned in SAL (#wikimedia-releng) [2023-08-23T13:17:32Z] <hashar> Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/951540 # T336380

• VirginiaPoundstone triaged this task as High priority.Aug 23 2023, 10:49 PM
gerritbot added a comment.Aug 29 2023, 4:01 PM

Change 951547 merged by Hnowlan:

[operations/puppet@production] kubernetes: add users for media_analytics service, cassandra config

https://gerrit.wikimedia.org/r/951547

gerritbot added a comment.Aug 29 2023, 4:50 PM

Change 953311 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/dns@master] wmnet: add geo-analytics and media-analytics ingress records

https://gerrit.wikimedia.org/r/953311

gerritbot added a comment.Aug 30 2023, 10:11 AM

Change 951544 merged by jenkins-bot:

[operations/deployment-charts@master] helmfile: add entries and namespace for media-analytics service

https://gerrit.wikimedia.org/r/951544

gerritbot added a comment.Aug 30 2023, 11:31 AM

Change 953311 merged by Hnowlan:

[operations/dns@master] wmnet: add geo-analytics and media-analytics ingress records

https://gerrit.wikimedia.org/r/953311

Milimetric moved this task from In Process to Ready for Testing on the Test Kitchen (Sprint 00) board.Aug 31 2023, 12:07 PM
gerritbot added a comment.Aug 31 2023, 2:12 PM

Change 951901 merged by Hnowlan:

[operations/puppet@production] service: add media-analytics service entry

https://gerrit.wikimedia.org/r/951901

gerritbot added a comment.Aug 31 2023, 2:24 PM

Change 954067 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] service: move geo-analytics and media-analytics to production

https://gerrit.wikimedia.org/r/954067

gerritbot added a comment.Aug 31 2023, 3:39 PM

Change 954067 merged by Hnowlan:

[operations/puppet@production] service: move geo-analytics and media-analytics to production

https://gerrit.wikimedia.org/r/954067

Stashbot added a comment.Aug 31 2023, 3:44 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-31T15:44:40Z] <hnowlan@cumin1001> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1020*,lvs2014*} and A:lvs (T336380)

Stashbot added a comment.Aug 31 2023, 3:45 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-31T15:45:57Z] <hnowlan@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1020*,lvs2014*} and A:lvs (T336380)

Stashbot added a comment.Aug 31 2023, 3:48 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-31T15:48:12Z] <hnowlan@cumin1001> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1019*,lvs2013*} and A:lvs (T336380)

Stashbot added a comment.Aug 31 2023, 3:49 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-31T15:49:10Z] <hnowlan@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1019*,lvs2013*} and A:lvs (T336380)

hnowlan updated the task description. (Show Details)Aug 31 2023, 3:52 PM
hnowlan updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Aug 31 2023, 4:10 PM
hnowlan moved this task from Ready for Testing to In Testing on the Test Kitchen (Sprint 00) board.Sep 5 2023, 12:21 PM
EChukwukere-WMF updated the task description. (Show Details)Sep 7 2023, 4:21 AM
EChukwukere-WMF subscribed.Sep 7 2023, 4:24 AM

Tests for MEDIA service is successful in the dockerized environment

• WDoranWMF moved this task from In Testing to Sign Off on the Test Kitchen (Sprint 00) board.Sep 7 2023, 11:17 AM
hnowlan updated the task description. (Show Details)Sep 11 2023, 2:29 PM
Sfaci edited projects, added Test Kitchen (Sprint 01); removed Test Kitchen (Sprint 00), AQS2.0.Sep 12 2023, 2:12 PM
Sfaci moved this task from Sprint Backlog to Sign Off on the Test Kitchen (Sprint 01) board.
gerritbot added a comment.Sep 12 2023, 3:03 PM

Change 956909 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: route requests to mediarequests service

https://gerrit.wikimedia.org/r/956909

gerritbot added a project: Patch-For-Review.Sep 12 2023, 3:03 PM
cjming moved this task from Sign Off to Done on the Test Kitchen (Sprint 01) board.Sep 20 2023, 3:37 PM
gerritbot added a comment.Sep 28 2023, 4:02 PM

Change 956909 merged by Hnowlan:

[operations/puppet@production] trafficserver: route requests to mediarequests service

https://gerrit.wikimedia.org/r/956909

Maintenance_bot removed a project: Patch-For-Review.Sep 28 2023, 4:10 PM
hnowlan closed this task as Resolved.Sep 28 2023, 5:08 PM
hnowlan updated the task description. (Show Details)
gerritbot added a comment.Oct 2 2023, 11:44 AM

Change 962588 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] media-analytics: bump image

https://gerrit.wikimedia.org/r/962588

gerritbot added a project: Patch-For-Review.Oct 2 2023, 11:44 AM
gerritbot added a comment.Oct 2 2023, 11:47 AM

Change 962588 merged by jenkins-bot:

[operations/deployment-charts@master] media-analytics: bump image

https://gerrit.wikimedia.org/r/962588

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2023, 12:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits