Page MenuHomePhabricator
Create Task
Maniphest T336391

AQS 2.0: Page Analytics Service: Deploy to staging and production
Closed, ResolvedPublic
Actions

Description

Background/Goal

Key task for Epic T288296: AQS 2.0: Page Analytics Service

Required:

  • Add blubber and pipeline configuration to page-analytics repo (blubber.yaml and config.yaml) (merged)
  • Integrate page-analytics image building in CI pipeline
  • Add page-analytics Cassandra user
  • Add k8s secrets
  • Create deployment-charts helmfile.d configuration entries
  • Run smoke tests against staging and certify service is working properly
  • Add service catalogue entry for page-analytics
  • Deploy page-analytics to LVS in state service_setup
  • Move page-analytics lvs_setup in LVS
  • Move page-analytics production in LVS
  • Add page-analytics to the API Gateway
  • Route public requests via ATS

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Sfaci created this task.May 10 2023, 1:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 10 2023, 1:48 PM
Sfaci added a parent task: T288296: AQS 2.0: Page Analytics Service.May 10 2023, 1:52 PM
Sfaci edited projects, added AQS2.0; removed API Platform (AQS 2.0 Roadmap).May 11 2023, 4:15 PM
VirginiaPoundstone moved this task from Incoming to To be discussed/To be estimated on the AQS2.0 board.Aug 25 2023, 5:16 PM
gerritbot added a comment.Sep 8 2023, 2:10 PM

Change 955946 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[generated-data-platform/aqs/editor-analytics@main] Add working blubber file and config

https://gerrit.wikimedia.org/r/955946

gerritbot added a project: Patch-For-Review.Sep 8 2023, 2:10 PM
gerritbot added a comment.Sep 8 2023, 2:17 PM

Change 955948 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[integration/config@master] zuul: add editor-analytics

https://gerrit.wikimedia.org/r/955948

gerritbot added a comment.Sep 8 2023, 2:20 PM

Change 955953 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[integration/config@master] jjb: add editor-analytics

https://gerrit.wikimedia.org/r/955953

gerritbot added a comment.Sep 11 2023, 8:56 AM

Change 955953 merged by jenkins-bot:

[integration/config@master] jjb: add editor-analytics

https://gerrit.wikimedia.org/r/955953

gerritbot added a comment.Sep 11 2023, 9:01 AM

Change 955948 merged by jenkins-bot:

[integration/config@master] zuul: add editor-analytics

https://gerrit.wikimedia.org/r/955948

Stashbot added a comment.Sep 11 2023, 9:03 AM

Mentioned in SAL (#wikimedia-releng) [2023-09-11T09:03:19Z] <hashar> Reloaded Zuul for "add editor-analytics" https://gerrit.wikimedia.org/r/c/integration/config/+/955948/ | T336391

hnowlan claimed this task.Sep 11 2023, 10:44 AM
hnowlan updated the task description. (Show Details)
gerritbot added a comment.Sep 11 2023, 5:24 PM

Change 955946 merged by Santiago Faci:

[generated-data-platform/aqs/editor-analytics@main] Add working blubber file and config

https://gerrit.wikimedia.org/r/955946

Maintenance_bot removed a project: Patch-For-Review.Sep 11 2023, 5:30 PM
Sfaci edited projects, added Data Products (Sprint 01); removed AQS2.0.Sep 12 2023, 2:26 PM
Sfaci unsubscribed.
Sfaci updated the task description. (Show Details)Sep 12 2023, 2:39 PM
Sfaci moved this task from Sprint Backlog to In Process on the Data Products (Sprint 01) board.Sep 12 2023, 2:53 PM
gerritbot added a comment.Sep 12 2023, 3:29 PM

Change 956912 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[generated-data-platform/aqs/page-analytics@main] Ready to be dockerized for deployment and QA purposes

https://gerrit.wikimedia.org/r/956912

gerritbot added a project: Patch-For-Review.Sep 12 2023, 3:29 PM
Sfaci triaged this task as Medium priority.Sep 12 2023, 3:31 PM
Sfaci updated the task description. (Show Details)
Sfaci raised the priority of this task from Medium to High.Sep 12 2023, 3:40 PM
gerritbot added a comment.Sep 13 2023, 6:37 AM

Change 956912 merged by Santiago Faci:

[generated-data-platform/aqs/page-analytics@main] Ready to be dockerized for deployment and QA purposes

https://gerrit.wikimedia.org/r/956912

Sfaci updated the task description. (Show Details)Sep 13 2023, 6:52 AM
Maintenance_bot removed a project: Patch-For-Review.Sep 13 2023, 7:10 AM
VirginiaPoundstone moved this task from In Process to BLOCKED on the Data Products (Sprint 01) board.Sep 18 2023, 1:06 PM
hnowlan added a comment.Sep 28 2023, 11:32 AM

Please let me know which Cassandra tables the service will need access to

gerritbot added a comment.Sep 28 2023, 11:41 AM

Change 961774 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] k8s, cassandra: add entries for {edit,editor,page}-analytics

https://gerrit.wikimedia.org/r/961774

gerritbot added a project: Patch-For-Review.Sep 28 2023, 11:41 AM
Sfaci added a comment.Sep 28 2023, 11:46 AM
  • page-analytics cassandra tables:
    • local_group_default_T_pageviews_per_project_v2
    • local_group_default_T_lgc_pagecounts_per_project
    • local_group_default_T_pageviews_per_article_flat
    • local_group_default_T_top_bycountry
    • local_group_default_T_top_pageviews
    • local_group_default_T_top_percountry
gerritbot added a comment.Sep 28 2023, 11:54 AM

Change 961777 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[integration/config@master] jjb: add editor-analytics, edit-analytics and page-analytics

https://gerrit.wikimedia.org/r/961777

gerritbot added a comment.Sep 28 2023, 11:55 AM

Change 961778 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[integration/config@master] zuul: add edit-analytics and page-analytics

https://gerrit.wikimedia.org/r/961778

gerritbot added a comment.Sep 28 2023, 12:06 PM

Change 961782 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] admin: add namespaces for remaining aqs2 services, add config for page-analytics

https://gerrit.wikimedia.org/r/961782

gerritbot added a comment.Sep 29 2023, 11:48 AM

Change 961777 merged by jenkins-bot:

[integration/config@master] jjb: add editor-analytics, edit-analytics and page-analytics

https://gerrit.wikimedia.org/r/961777

gerritbot added a comment.Sep 29 2023, 11:57 AM

Change 961778 merged by jenkins-bot:

[integration/config@master] zuul: add edit-analytics and page-analytics

https://gerrit.wikimedia.org/r/961778

Stashbot mentioned this in T336415: Editor analytics service: Configure routing in staging and production.Sep 29 2023, 11:58 AM

Mentioned in SAL (#wikimedia-releng) [2023-09-29T11:58:47Z] <hashar> Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/961778 "add edit-analytics and page-analytics" T336391 T336385 T336415

Stashbot mentioned this in T336385: Edit Analytics: Deploy to staging and production.Sep 29 2023, 11:58 AM
Sfaci moved this task from BLOCKED to In Process on the Data Products (Sprint 01) board.Oct 2 2023, 8:27 AM
gerritbot added a comment.Oct 2 2023, 10:50 AM

Change 961774 merged by Hnowlan:

[operations/puppet@production] k8s, cassandra: add entries for {edit,editor,page}-analytics

https://gerrit.wikimedia.org/r/961774

gerritbot added a comment.Oct 2 2023, 10:50 AM

Change 962570 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] service: add {edit,editor,page}-analytics services

https://gerrit.wikimedia.org/r/962570

gerritbot added a comment.Oct 2 2023, 11:34 AM

Change 961782 merged by jenkins-bot:

[operations/deployment-charts@master] admin: add namespaces for remaining aqs2 services, add config for page-analytics

https://gerrit.wikimedia.org/r/961782

hnowlan updated the task description. (Show Details)Oct 2 2023, 11:48 AM
Sfaci merged a task: T336399: AQS 2.0: Page Analytics Service: Configure routing in staging and production.Oct 2 2023, 12:01 PM
Sfaci added subscribers: Sfaci, hnowlan, Atieno and 4 others.
Sfaci edited projects, added Data Products (Sprint 02); removed Data Products (Sprint 01).Oct 3 2023, 1:41 PM
Sfaci moved this task from Sprint Backlog to In Process on the Data Products (Sprint 02) board.
FJoseph-WMF unsubscribed.Oct 3 2023, 2:26 PM
gerritbot added a comment.Oct 3 2023, 4:54 PM

Change 963106 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/dns@master] wmnet: add records for edit-, editor- and page-analytics

https://gerrit.wikimedia.org/r/963106

gerritbot added a comment.Oct 4 2023, 11:12 AM

Change 963106 merged by Hnowlan:

[operations/dns@master] wmnet: add records for edit-, editor- and page-analytics

https://gerrit.wikimedia.org/r/963106

gerritbot added a comment.Oct 4 2023, 11:34 AM

Change 962570 merged by Hnowlan:

[operations/puppet@production] service: add {edit,editor,page}-analytics services

https://gerrit.wikimedia.org/r/962570

Maintenance_bot removed a project: Patch-For-Review.Oct 4 2023, 12:10 PM
gerritbot added a comment.Oct 6 2023, 4:58 PM

Change 964044 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: route edit-,editor- and page-analytics

https://gerrit.wikimedia.org/r/964044

gerritbot added a project: Patch-For-Review.Oct 6 2023, 4:58 PM
gerritbot added a comment.Oct 10 2023, 12:26 PM

Change 964044 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: route edit-,editor- and page-analytics

https://gerrit.wikimedia.org/r/964044

Maintenance_bot removed a project: Patch-For-Review.Oct 10 2023, 12:31 PM
gerritbot added a comment.Oct 10 2023, 2:37 PM

Change 964923 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] service: change state to production for {edit,editor,page}-analytics

https://gerrit.wikimedia.org/r/964923

gerritbot added a project: Patch-For-Review.Oct 10 2023, 2:37 PM
gerritbot added a comment.Oct 10 2023, 3:08 PM

Change 964923 merged by Hnowlan:

[operations/puppet@production] service: change state to production for {edit,editor,page}-analytics

https://gerrit.wikimedia.org/r/964923

Maintenance_bot removed a project: Patch-For-Review.Oct 10 2023, 3:10 PM
gerritbot added a comment.Oct 10 2023, 4:20 PM

Change 964946 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: route pageviews to page-analytics

https://gerrit.wikimedia.org/r/964946

gerritbot added a project: Patch-For-Review.Oct 10 2023, 4:20 PM
gerritbot added a comment.Oct 11 2023, 3:06 PM

Change 964946 merged by Hnowlan:

[operations/puppet@production] trafficserver: route pageviews to page-analytics

https://gerrit.wikimedia.org/r/964946

Maintenance_bot removed a project: Patch-For-Review.Oct 11 2023, 3:10 PM
gerritbot added a comment.Oct 11 2023, 3:12 PM

Change 965184 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: correct pageviews paths

https://gerrit.wikimedia.org/r/965184

gerritbot added a project: Patch-For-Review.Oct 11 2023, 3:12 PM
gerritbot added a comment.Oct 11 2023, 3:21 PM

Change 965184 merged by Hnowlan:

[operations/puppet@production] trafficserver: correct pageviews paths

https://gerrit.wikimedia.org/r/965184

Maintenance_bot removed a project: Patch-For-Review.Oct 11 2023, 3:30 PM
mforns moved this task from In Process to Sign Off on the Data Products (Sprint 02) board.Oct 11 2023, 4:22 PM
hnowlan closed this task as Resolved.Oct 11 2023, 4:47 PM
hnowlan updated the task description. (Show Details)

To fully migrate to page-analytics so as to be able to deprecate AQS1, it is required that changes be made to wikifeeds and Mediawiki. Currently requests are still hitting the old AQS pageviews endpoint coming from appservers and wikifeeds instances - these are outside the scope of this ticket.

WDoranWMF closed subtask T348731: Follow up on remaining requests to pageviews endpoints as Resolved.Nov 20 2023, 5:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL