Page MenuHomePhabricator
Create Task
Maniphest T320967

[AQS 2.0] New Service Request device_analytics
Closed, ResolvedPublic
Actions

Description

point people @Atieno @FGoodwin @hnowlan

The repo is here:
Gerrit

Acceptance Criteria

  • Docker container is built
  • Service supports TLS and authenticator configuration for Cassandra (T328765)
  • Integrate container creation into the pipeline
  • Write a Helm chart
  • Deploy
  • Provide metrics gathered via Prometheus and dashboards to monitor and assess
  • Documentation updated

Details

SubjectRepoBranchLines +/-
device-analytics: use global AQS configuration filesoperations/deployment-chartsmaster+8 -67
trafficserver: add gateway routing script, route device-analyticsoperations/puppetproduction+172 -0
trafficserver: add gateway routing script, route device-analytics on cp2037operations/puppetproduction+584 -0
trafficserver: add gateway routing script, route device-analyticsoperations/puppetproduction+172 -0
device-analyics: deploy new versionoperations/deployment-chartsmaster+1 -1
service: move device-analytics to productionoperations/puppetproduction+1 -1
service: move device-analytics to lvs_setupoperations/puppetproduction+1 -1
cassandra: fix device_analytics creation syntaxoperations/puppetproduction+2 -2
device-analytics: add missing mesh portoperations/deployment-chartsmaster+1 -0
helmfile: add device-analytics configuration, namespaceoperations/deployment-chartsmaster+181 -0
service, k8s: add service configuration for AQS2 service device-analyticsoperations/puppetproduction+52 -0
cassandra: add stub secret for device_analyticslabs/privatemaster+1 -0
kubernetes: add stub values for device-analyticslabs/privatemaster+4 -0
conftool-data: add device-analytics serviceoperations/puppetproduction+1 -0
Add service records for device-analytics.operations/dnsmaster+4 -0
Add pipeline configuration for device-analyticsintegration/configmaster+26 -0
Fix docker image usedgenerated-data-platform/aqs/device_analyticsmain+3 -2
Configuration for Blubber, type/format fixesgenerated-data-platform/aqs/device_analyticsmain+96 -55
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a project: Patch-For-Review.Jan 31 2023, 11:31 AM
hnowlan added a comment.Jan 31 2023, 11:50 AM

Which tests should be run as part of the image build process for unique_devices? Currently the tests run by make test require a second container to run and are more akin to integration tests, which is fine, but this won't fit into our pipeline currently.

JArguello-WMF moved this task from Next Up to In Progress on the API Platform (Sprint 03) board.Jan 31 2023, 2:53 PM
gerritbot added a comment.Jan 31 2023, 5:58 PM

Change 885295 merged by Hnowlan:

[generated-data-platform/aqs/device_analytics@main] Configuration for Blubber, type/format fixes

https://gerrit.wikimedia.org/r/885295

hnowlan mentioned this in rGADAf7675d4d0d9e: Configuration for Blubber, type/format fixes.Jan 31 2023, 5:59 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 31 2023, 6:31 PM
JArguello-WMF edited projects, added API Platform (Sprint 05); removed API Platform (Sprint 03).Feb 1 2023, 7:28 PM
JArguello-WMF moved this task from Next Up to In Progress on the API Platform (Sprint 05) board.
gerritbot added a comment.Feb 3 2023, 12:33 PM

Change 886353 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[generated-data-platform/aqs/device_analytics@main] Fix docker image used

https://gerrit.wikimedia.org/r/886353

gerritbot added a project: Patch-For-Review.Feb 3 2023, 12:33 PM
hnowlan updated the task description. (Show Details)Feb 3 2023, 12:51 PM
gerritbot added a comment.Feb 3 2023, 1:29 PM

Change 886358 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] WIP: helmfile: add device-analytics configuration

https://gerrit.wikimedia.org/r/886358

gerritbot added a comment.Feb 9 2023, 4:36 PM

Change 886353 merged by BPirkle:

[generated-data-platform/aqs/device_analytics@main] Fix docker image used

https://gerrit.wikimedia.org/r/886353

hnowlan mentioned this in rGADAdc36bb9b83da: Fix docker image used.Feb 9 2023, 4:37 PM
gerritbot added a comment.Feb 17 2023, 11:04 AM

Change 889960 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] service, k8s: add service configuration for AQS2 service device-analytics

https://gerrit.wikimedia.org/r/889960

gerritbot added a comment.Feb 20 2023, 11:09 AM

Change 890398 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/dns@master] Add service records for device-analytics using ingress.

https://gerrit.wikimedia.org/r/890398

gerritbot added a comment.Mar 1 2023, 11:12 AM

Change 893416 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[integration/config@master] Add pipeline configuration for device-analytics

https://gerrit.wikimedia.org/r/893416

hnowlan added a comment.EditedMar 1 2023, 12:18 PM

Changes pending once that is done:
✅ Integration into pipeline https://gerrit.wikimedia.org/r/893416

gerritbot added a comment.Mar 1 2023, 3:55 PM

Change 893416 merged by jenkins-bot:

[integration/config@master] Add pipeline configuration for device-analytics

https://gerrit.wikimedia.org/r/893416

Stashbot added a comment.Mar 1 2023, 3:55 PM

Mentioned in SAL (#wikimedia-releng) [2023-03-01T15:55:52Z] <hashar> Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/893416 | T320967

Stashbot added a comment.Mar 1 2023, 5:37 PM

Mentioned in SAL (#wikimedia-releng) [2023-03-01T17:37:36Z] <hashar> Deploying device-analytics-* jobs for for https://gerrit.wikimedia.org/r/c/integration/config/+/893416 | T320967

hnowlan updated the task description. (Show Details)Mar 1 2023, 6:03 PM
hnowlan moved this task from Blocked ❌ to In review on the Platform Team Workboards (Platform Engineering Reliability) board.Mar 2 2023, 4:37 PM
gerritbot added a comment.Mar 8 2023, 11:20 AM

Change 895716 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] conftool-data: add device-analytics service

https://gerrit.wikimedia.org/r/895716

gerritbot added a comment.Mar 8 2023, 12:07 PM

Change 890398 merged by Hnowlan:

[operations/dns@master] Add service records for device-analytics.

https://gerrit.wikimedia.org/r/890398

gerritbot added a comment.Mar 8 2023, 12:26 PM

Change 895716 merged by Hnowlan:

[operations/puppet@production] conftool-data: add device-analytics service

https://gerrit.wikimedia.org/r/895716

gerritbot added a comment.Mar 8 2023, 4:00 PM

Change 895824 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[labs/private@master] kubernetes: add stub values for device-analytics

https://gerrit.wikimedia.org/r/895824

gerritbot added a comment.Mar 8 2023, 4:06 PM

Change 895824 merged by Hnowlan:

[labs/private@master] kubernetes: add stub values for device-analytics

https://gerrit.wikimedia.org/r/895824

hnowlan mentioned this in rLPRIc5ee2220a094: kubernetes: add stub values for device-analytics.Mar 8 2023, 4:07 PM
JArguello-WMF moved this task from Sprint 05 to Sprint 06 on the API Platform board.Mar 14 2023, 3:02 PM
JArguello-WMF edited projects, added API Platform (Sprint 06); removed API Platform (Sprint 05).
JArguello-WMF moved this task from Next Up to In Progress on the API Platform (Sprint 06) board.
gerritbot added a comment.Mar 14 2023, 5:14 PM

Change 898810 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[labs/private@master] cassandra: add stub secret for device_analytics

https://gerrit.wikimedia.org/r/898810

gerritbot added a comment.Mar 14 2023, 5:17 PM

Change 898810 merged by Hnowlan:

[labs/private@master] cassandra: add stub secret for device_analytics

https://gerrit.wikimedia.org/r/898810

gerritbot added a comment.Mar 14 2023, 5:35 PM

Change 889960 merged by Hnowlan:

[operations/puppet@production] service, k8s: add service configuration for AQS2 service device-analytics

https://gerrit.wikimedia.org/r/889960

hnowlan mentioned this in rLPRIf0f95c7bd43f: cassandra: add stub secret for device_analytics.Mar 14 2023, 5:35 PM
gerritbot added a comment.Mar 14 2023, 5:53 PM

Change 886358 merged by jenkins-bot:

[operations/deployment-charts@master] helmfile: add device-analytics configuration, namespace

https://gerrit.wikimedia.org/r/886358

Maintenance_bot removed a project: Patch-For-Review.Mar 14 2023, 6:11 PM
gerritbot added a comment.Mar 14 2023, 6:13 PM

Change 898820 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] device-analytics: add missing mesh port

https://gerrit.wikimedia.org/r/898820

gerritbot added a project: Patch-For-Review.Mar 14 2023, 6:13 PM
gerritbot added a comment.Mar 14 2023, 6:28 PM

Change 898824 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] cassandra: fix device_analytics creation syntax

https://gerrit.wikimedia.org/r/898824

gerritbot added a comment.Mar 15 2023, 10:27 AM

Change 898820 merged by jenkins-bot:

[operations/deployment-charts@master] device-analytics: add missing mesh port

https://gerrit.wikimedia.org/r/898820

gerritbot added a comment.Mar 15 2023, 11:17 AM

Change 898824 merged by Hnowlan:

[operations/puppet@production] cassandra: fix device_analytics creation syntax

https://gerrit.wikimedia.org/r/898824

Maintenance_bot removed a project: Patch-For-Review.Mar 15 2023, 11:30 AM
hnowlan updated the task description. (Show Details)Mar 15 2023, 12:35 PM
gerritbot added a comment.Mar 15 2023, 12:43 PM

Change 899607 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] service: move device-analytics to lvs_setup

https://gerrit.wikimedia.org/r/899607

gerritbot added a project: Patch-For-Review.Mar 15 2023, 12:43 PM

Change 899608 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] service: move device-analytics to production

https://gerrit.wikimedia.org/r/899608

JArguello-WMF moved this task from In Progress to Blocked/Paused on the API Platform (Sprint 06) board.Mar 23 2023, 6:46 PM
hnowlan added a subscriber: JArguello-WMF.Mar 29 2023, 4:31 PM

I plan on moving this along this week @JArguello-WMF

JArguello-WMF edited projects, added API Platform (Sprint 07); removed API Platform (Sprint 06).Apr 5 2023, 2:15 PM
JArguello-WMF moved this task from Next Up to Blocked/Paused on the API Platform (Sprint 07) board.
gerritbot added a comment.Apr 6 2023, 2:20 PM

Change 899607 merged by Hnowlan:

[operations/puppet@production] service: move device-analytics to lvs_setup

https://gerrit.wikimedia.org/r/899607

Stashbot added a comment.Apr 6 2023, 2:30 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-06T14:30:55Z] <hnowlan@cumin1001> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1020*,lvs2010*} and A:lvs (T320967)

Stashbot added a comment.Apr 6 2023, 2:32 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-06T14:32:14Z] <hnowlan@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1020*,lvs2010*} and A:lvs (T320967)

Stashbot added a comment.Apr 6 2023, 2:37 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-06T14:37:56Z] <hnowlan@cumin1001> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1019*,lvs2009*} and A:lvs (T320967)

Stashbot added a comment.Apr 6 2023, 2:39 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-06T14:39:14Z] <hnowlan@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1019*,lvs2009*} and A:lvs (T320967)

hnowlan moved this task from In review to Doing 🔨 on the Platform Team Workboards (Platform Engineering Reliability) board.Apr 6 2023, 2:49 PM
JArguello-WMF moved this task from Blocked/Paused to In Progress on the API Platform (Sprint 07) board.Apr 6 2023, 4:19 PM
VirginiaPoundstone edited projects, added API Platform (AQS 2.0 Roadmap); removed API Platform (Sprint 07).Apr 25 2023, 1:28 PM
gerritbot added a comment.May 8 2023, 10:42 AM

Change 899608 merged by Hnowlan:

[operations/puppet@production] service: move device-analytics to production

https://gerrit.wikimedia.org/r/899608

Stashbot added a comment.May 8 2023, 10:45 AM

Mentioned in SAL (#wikimedia-operations) [2023-05-08T10:45:55Z] <hnowlan@cumin1001> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1020*,lvs2010*} and A:lvs (T320967)

Stashbot added a comment.May 8 2023, 10:47 AM

Mentioned in SAL (#wikimedia-operations) [2023-05-08T10:47:03Z] <hnowlan@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1020*,lvs2010*} and A:lvs (T320967)

Stashbot added a comment.May 8 2023, 10:54 AM

Mentioned in SAL (#wikimedia-operations) [2023-05-08T10:54:55Z] <hnowlan@cumin1001> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1019*,lvs2009*} and A:lvs (T320967)

Stashbot added a comment.May 8 2023, 10:56 AM

Mentioned in SAL (#wikimedia-operations) [2023-05-08T10:56:13Z] <hnowlan@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1019*,lvs2009*} and A:lvs (T320967)

hnowlan closed this task as Resolved.May 8 2023, 10:58 AM
hnowlan updated the task description. (Show Details)

This is now deployed to the production cluster. Given the overlap we can do the final work in T335505.

Maintenance_bot removed a project: Patch-For-Review.May 8 2023, 11:11 AM
Atieno mentioned this in T335505: Figure out what's outstanding to have device-analytics serving 100% Production data.May 10 2023, 2:40 PM
gerritbot added a comment.Jun 1 2023, 5:38 PM

Change 925859 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] device-analyics: deploy new version

https://gerrit.wikimedia.org/r/925859

gerritbot added a project: Patch-For-Review.Jun 1 2023, 5:38 PM
gerritbot added a comment.Jun 1 2023, 5:41 PM

Change 925859 merged by jenkins-bot:

[operations/deployment-charts@master] device-analyics: deploy new version

https://gerrit.wikimedia.org/r/925859

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2023, 6:10 PM
gerritbot added a comment.Jul 4 2023, 3:23 PM

Change 935464 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: add gateway routing script, route device-analytics

https://gerrit.wikimedia.org/r/935464

gerritbot added a project: Patch-For-Review.Jul 4 2023, 3:23 PM
gerritbot added a comment.Jul 6 2023, 2:09 PM

Change 935464 merged by Hnowlan:

[operations/puppet@production] trafficserver: add gateway routing script, route device-analytics

https://gerrit.wikimedia.org/r/935464

Maintenance_bot removed a project: Patch-For-Review.Jul 6 2023, 2:10 PM
gerritbot added a comment.Jul 10 2023, 7:02 AM

Change 936509 had a related patch set uploaded (by Vgutierrez; author: Hnowlan):

[operations/puppet@production] trafficserver: add gateway routing script, route device-analytics on cp2037

https://gerrit.wikimedia.org/r/936509

gerritbot added a project: Patch-For-Review.Jul 10 2023, 7:02 AM
Stashbot added a comment.Jul 10 2023, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-10T09:14:12Z] <vgutierrez> depool cp2037 (debugging ATS cacheability issues) - T320967

gerritbot added a comment.Jul 10 2023, 9:14 AM

Change 936509 merged by Vgutierrez:

[operations/puppet@production] trafficserver: add gateway routing script, route device-analytics on cp2037

https://gerrit.wikimedia.org/r/936509

Maintenance_bot removed a project: Patch-For-Review.Jul 10 2023, 9:31 AM
Stashbot added a comment.Jul 10 2023, 11:52 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-10T11:52:12Z] <vgutierrez> repool cp2037 (debugging finished) - T320967

Vgutierrez mentioned this in T341461: Unify host evaluation criteria on cache::req_handling for Varnish and ATS.Jul 10 2023, 12:07 PM
gerritbot added a comment.Jul 25 2023, 1:57 PM

Change 941405 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: add gateway routing script, route device-analytics

https://gerrit.wikimedia.org/r/941405

gerritbot added a project: Patch-For-Review.Jul 25 2023, 1:57 PM
gerritbot added a comment.Jul 25 2023, 2:31 PM

Change 941405 merged by Hnowlan:

[operations/puppet@production] trafficserver: add gateway routing script, route device-analytics

https://gerrit.wikimedia.org/r/941405

Maintenance_bot removed a project: Patch-For-Review.Jul 25 2023, 3:11 PM
gerritbot added a comment.Aug 31 2023, 10:57 AM

Change 953982 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] device-analytics: use global AQS configuration files

https://gerrit.wikimedia.org/r/953982

gerritbot added a project: Patch-For-Review.Aug 31 2023, 10:57 AM
gerritbot added a comment.Sep 5 2023, 10:29 AM

Change 953982 merged by jenkins-bot:

[operations/deployment-charts@master] device-analytics: use global AQS configuration files

https://gerrit.wikimedia.org/r/953982

Maintenance_bot removed a project: Patch-For-Review.Sep 5 2023, 10:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL