Page MenuHomePhabricator
Create Task
Maniphest T336400

AQS 2.0: Geo Analytics Service Deploy to Staging and production
Closed, ResolvedPublic
Actions

Description

Background/Goal

Key task for T288305: AQS 2.0: Geo Analytics Service

Required:

  • Add blubber and pipeline configuration to geo-analytics repo (review)
  • Integrate geo-analytics image building in CI pipeline (review 1, 2)
  • Add geo-analytics Cassandra user (review)
  • Add k8s secrets
  • Create deployment-charts helmfile.d configuration entries (review)
  • Add service catalogue entry for geo-analytics (review)
  • Run smoke tests against staging and certify service is working properly
  • Deploy geo-analytics to LVS in state service_setup
  • Move geo-analytics lvs_setup in LVS
  • Move geo-analytics production in LVS
  • Add geo-analytics to the API Gateway
  • Route public requests via ATS

Done is

QA notes

Artifacts & Resources

Templates, instructions, and other useful documentation

Details

SubjectRepoBranchLines +/-
api-gateway: emit cache-control header for 404soperations/deployment-chartsmaster+6 -1
rest-gateway: add config to limit routes to a domain, limit aqs2 apis to wikimedia.orgoperations/deployment-chartsmaster+17 -1
trafficserver: route requests for geo-analytics via rest-gatewayoperations/puppetproduction+1 -0
rest-gateway: set SNI when using ingressoperations/deployment-chartsmaster+8 -2
rest-gateway: preserve cluster hostname when using ingressoperations/deployment-chartsmaster+13 -1
rest-gateway: route requests to geo-analyticsoperations/deployment-chartsmaster+24 -2
service: move geo-analytics and media-analytics to productionoperations/puppetproduction+2 -2
wmnet: add geo-analytics and media-analytics ingress recordsoperations/dnsmaster+6 -0
AQS2: Move common settings (AQS cassandra nodes) to _aqs2-common_operations/deployment-chartsmaster+260 -275
service, conftool: add base configuration for geo-analyticsoperations/puppetproduction+14 -0
geo-analytics: use ingressoperations/deployment-chartsmaster+1 -1
helmfile: add namespace and service definition for geo-analyticsoperations/deployment-chartsmaster+390 -0
Added deployment and QA test suite using docker filesgenerated-data-platform/aqs/geo-analyticsmain+113 -66
zuul: add geo-analytics configurationintegration/configmaster+10 -0
jjb: add entry for geo-analytics serviceintegration/configmaster+15 -0
aqs: enable geo_analytics useroperations/puppetproduction+1 -0
deployment_server: add new service geo-analyticsoperations/puppetproduction+9 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

JArguello-WMF created this task.May 10 2023, 1:52 PM
JArguello-WMF removed BPirkle as the assignee of this task.May 10 2023, 2:10 PM
JArguello-WMF edited projects, added AQS2.0; removed Epic, Data-Engineering, User-Eevans.
JArguello-WMF added a subscriber: BPirkle.
JArguello-WMF renamed this task from Geo Analytics Service Deploy to Staging and production to AQS 2.0: Geo Analytics Service Deploy to Staging and production.May 10 2023, 2:15 PM
JArguello-WMF removed projects: API Platform (AQS 2.0 Roadmap), Platform Engineering Roadmap.
JArguello-WMF assigned this task to hnowlan.Jun 7 2023, 7:49 PM
JArguello-WMF edited projects, added AQS2.0 (Sprint 10); removed AQS2.0.
JArguello-WMF edited subscribers, added: VirginiaPoundstone, SGupta-WMF; removed: • Clarakosi, WDoranWMF.
JArguello-WMF mentioned this in T288305: AQS 2.0: Geo Analytics Service.Jun 7 2023, 8:07 PM
SGupta-WMF added a subscriber: EChukwukere-WMF.Jul 21 2023, 4:31 AM

Hi @hnowlan We have prioritised deployment of geo-analytics post device-analytics . It therefore needs to be deployed to staging once @EChukwukere-WMF signs off local testing .

SGupta-WMF edited subscribers, added: WDoranWMF; removed: FGoodwin.Jul 21 2023, 4:31 AM
hnowlan added a comment.Jul 25 2023, 10:15 AM

Before we can deploy to staging or production we will need a docker image built from the codebase using similar methods as to the .pipeline directory in the device-analytics repo.

hnowlan updated the task description. (Show Details)Jul 25 2023, 10:18 AM
hnowlan updated the task description. (Show Details)Jul 25 2023, 10:22 AM
hnowlan updated the task description. (Show Details)
gerritbot added a comment.Jul 25 2023, 10:27 AM

Change 941374 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] WIP helmfile: add namespace and service definition for geo-analytics

https://gerrit.wikimedia.org/r/941374

gerritbot added a project: Patch-For-Review.Jul 25 2023, 10:27 AM
hnowlan added a comment.Jul 25 2023, 10:29 AM

Reserved port 4973 in service ports.

hnowlan updated the task description. (Show Details)Jul 25 2023, 10:29 AM
hnowlan updated the task description. (Show Details)
gerritbot added a comment.Aug 10 2023, 3:04 PM

Change 947860 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[integration/config@master] jjb: add entry for geo-analytics service

https://gerrit.wikimedia.org/r/947860

gerritbot added a comment.Aug 10 2023, 3:05 PM

Change 947861 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[integration/config@master] zuul: add geo-analytics configuration

https://gerrit.wikimedia.org/r/947861

hnowlan updated the task description. (Show Details)Aug 10 2023, 3:05 PM
gerritbot added a comment.Aug 10 2023, 3:13 PM

Change 947862 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] deployment_server: add new service geo-analytics

https://gerrit.wikimedia.org/r/947862

gerritbot added a comment.Aug 10 2023, 3:19 PM

Change 947864 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] service, conftool: add base configuration for geo-analytics

https://gerrit.wikimedia.org/r/947864

hnowlan updated the task description. (Show Details)Aug 10 2023, 3:51 PM
phuedx moved this task from Next Up to In Progress on the AQS2.0 (Sprint 10) board.Aug 14 2023, 1:01 PM
phuedx moved this task from In Progress to Blocked/Paused on the AQS2.0 (Sprint 10) board.
WDoranWMF moved this task from Blocked/Paused to In Progress on the AQS2.0 (Sprint 10) board.Aug 16 2023, 11:21 AM
gerritbot added a comment.Aug 16 2023, 5:27 PM

Change 949560 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[generated-data-platform/aqs/geo-analytics@main] Deployment and QA test suite using docker: Added the needed blubber file for deployment and some changes to be able to create a dockerized service and run it with the cassandra test-env to be able to run QA test suite using docker

https://gerrit.wikimedia.org/r/949560

Sfaci updated the task description. (Show Details)Aug 17 2023, 9:18 AM
gerritbot added a comment.Aug 17 2023, 10:44 AM

Change 947862 merged by Hnowlan:

[operations/puppet@production] deployment_server: add new service geo-analytics

https://gerrit.wikimedia.org/r/947862

gerritbot added a comment.Aug 17 2023, 11:16 AM

Change 949947 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] aqs: enable geo_analytics user

https://gerrit.wikimedia.org/r/949947

gerritbot added a comment.Aug 18 2023, 10:02 AM

Change 949947 merged by Hnowlan:

[operations/puppet@production] aqs: enable geo_analytics user

https://gerrit.wikimedia.org/r/949947

hnowlan updated the task description. (Show Details)Aug 18 2023, 10:28 AM
gerritbot added a comment.Aug 21 2023, 7:59 AM

Change 947860 merged by jenkins-bot:

[integration/config@master] jjb: add entry for geo-analytics service

https://gerrit.wikimedia.org/r/947860

gerritbot added a comment.Aug 21 2023, 7:59 AM

Change 947861 merged by jenkins-bot:

[integration/config@master] zuul: add geo-analytics configuration

https://gerrit.wikimedia.org/r/947861

Stashbot added a comment.Aug 21 2023, 8:01 AM

Mentioned in SAL (#wikimedia-releng) [2023-08-21T08:01:47Z] <hashar> Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/947861 # T336400

Milimetric moved this task from In Progress to Blocked/Paused on the AQS2.0 (Sprint 10) board.Aug 22 2023, 1:38 PM
Milimetric moved this task from Blocked/Paused to In Progress on the AQS2.0 (Sprint 10) board.
VirginiaPoundstone moved this task from Sprint 10 to Current Data Products Sprint on the AQS2.0 board.Aug 22 2023, 2:16 PM
VirginiaPoundstone edited projects, added AQS2.0, Data Products (Sprint 00); removed AQS2.0 (Sprint 10).
Sfaci moved this task from Sprint Backlog to In Process on the Data Products (Sprint 00) board.Aug 22 2023, 2:27 PM
Sfaci updated the task description. (Show Details)Aug 22 2023, 3:00 PM
gerritbot added a comment.Aug 22 2023, 4:41 PM

Change 951544 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] helmfile: add entries and namespace for media-analytics service

https://gerrit.wikimedia.org/r/951544

gerritbot added a comment.Aug 23 2023, 2:43 PM

Change 949560 merged by Santiago Faci:

[generated-data-platform/aqs/geo-analytics@main] Added deployment and QA test suite using docker files

https://gerrit.wikimedia.org/r/949560

Sfaci updated the task description. (Show Details)Aug 23 2023, 2:46 PM
gerritbot added a comment.Aug 23 2023, 4:02 PM

Change 941374 merged by jenkins-bot:

[operations/deployment-charts@master] helmfile: add namespace and service definition for geo-analytics

https://gerrit.wikimedia.org/r/941374

hnowlan updated the task description. (Show Details)Aug 23 2023, 5:22 PM
VirginiaPoundstone triaged this task as High priority.Aug 23 2023, 10:49 PM
VirginiaPoundstone moved this task to To be discussed/To be estimated on the AQS2.0 board.Aug 25 2023, 5:17 PM
gerritbot added a comment.Aug 29 2023, 1:43 PM

Change 953258 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] geo-analytics: use ingress

https://gerrit.wikimedia.org/r/953258

gerritbot added a comment.Aug 29 2023, 2:03 PM

Change 953258 merged by jenkins-bot:

[operations/deployment-charts@master] geo-analytics: use ingress

https://gerrit.wikimedia.org/r/953258

gerritbot added a comment.Aug 29 2023, 2:28 PM

Change 947864 merged by Hnowlan:

[operations/puppet@production] service, conftool: add base configuration for geo-analytics

https://gerrit.wikimedia.org/r/947864

gerritbot added a comment.Aug 29 2023, 2:49 PM

Change 953283 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] AQS2: Move common settings (AQS cassandra nodes) to _aqs2-common_

https://gerrit.wikimedia.org/r/953283

gerritbot added a comment.Aug 29 2023, 4:09 PM

Change 953283 merged by jenkins-bot:

[operations/deployment-charts@master] AQS2: Move common settings (AQS cassandra nodes) to _aqs2-common_

https://gerrit.wikimedia.org/r/953283

gerritbot added a comment.Aug 29 2023, 4:50 PM

Change 953311 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/dns@master] wmnet: add geo-analytics and media-analytics ingress records

https://gerrit.wikimedia.org/r/953311

WDoranWMF updated the task description. (Show Details)Aug 29 2023, 7:15 PM
gerritbot added a comment.Aug 30 2023, 11:31 AM

Change 953311 merged by Hnowlan:

[operations/dns@master] wmnet: add geo-analytics and media-analytics ingress records

https://gerrit.wikimedia.org/r/953311

Milimetric moved this task from In Process to Ready for Testing on the Data Products (Sprint 00) board.Aug 31 2023, 12:08 PM
Milimetric moved this task from Ready for Testing to Sign Off on the Data Products (Sprint 00) board.
gerritbot added a comment.Aug 31 2023, 2:24 PM

Change 954067 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] service: move geo-analytics and media-analytics to production

https://gerrit.wikimedia.org/r/954067

gerritbot added a comment.Aug 31 2023, 3:39 PM

Change 954067 merged by Hnowlan:

[operations/puppet@production] service: move geo-analytics and media-analytics to production

https://gerrit.wikimedia.org/r/954067

hnowlan updated the task description. (Show Details)Aug 31 2023, 3:51 PM
WDoranWMF updated the task description. (Show Details)Sep 1 2023, 5:05 PM

@hnowlan We sign off that this is ready to go!Thanks!

gerritbot added a comment.Sep 5 2023, 9:46 AM

Change 954888 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: route requests to geo-analytics

https://gerrit.wikimedia.org/r/954888

gerritbot added a comment.Sep 5 2023, 9:48 AM

Change 954890 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: route requests for geo-analytics via rest-gateway

https://gerrit.wikimedia.org/r/954890

hnowlan moved this task from Sign Off to In Process on the Data Products (Sprint 00) board.Sep 5 2023, 12:20 PM
hnowlan moved this task from In Process to Done on the Data Products (Sprint 00) board.
gerritbot added a comment.Sep 6 2023, 9:23 AM

Change 954888 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: route requests to geo-analytics

https://gerrit.wikimedia.org/r/954888

gerritbot added a comment.Sep 6 2023, 12:10 PM

Change 955320 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: preserve cluster hostname when using ingress

https://gerrit.wikimedia.org/r/955320

gerritbot added a comment.Sep 8 2023, 9:29 AM

Change 955320 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: preserve cluster hostname when using ingress

https://gerrit.wikimedia.org/r/955320

gerritbot added a comment.Sep 8 2023, 11:45 AM

Change 955912 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: set SNI when using ingress

https://gerrit.wikimedia.org/r/955912

gerritbot added a comment.Sep 8 2023, 12:46 PM

Change 955912 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: set SNI when using ingress

https://gerrit.wikimedia.org/r/955912

hnowlan updated the task description. (Show Details)Sep 11 2023, 10:45 AM
gerritbot added a comment.Sep 12 2023, 9:26 AM

Change 954890 merged by Hnowlan:

[operations/puppet@production] trafficserver: route requests for geo-analytics via rest-gateway

https://gerrit.wikimedia.org/r/954890

gerritbot added a comment.Sep 12 2023, 10:08 AM

Change 956826 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: add config to limit routes to a domain, limit aqs2 apis to wikimedia.org

https://gerrit.wikimedia.org/r/956826

gerritbot added a comment.Sep 12 2023, 11:17 AM

Change 956833 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] api-gateway: emit cache-control header for 404s

https://gerrit.wikimedia.org/r/956833

gerritbot added a comment.Sep 12 2023, 1:15 PM

Change 956826 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: add config to limit routes to a domain, limit aqs2 apis to wikimedia.org

https://gerrit.wikimedia.org/r/956826

VirginiaPoundstone closed this task as Resolved.Sep 19 2023, 2:34 PM
VirginiaPoundstone moved this task from Done to Sign Off on the Data Products (Sprint 00) board.
VirginiaPoundstone moved this task from Sign Off to Done on the Data Products (Sprint 00) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL