In T352103#9374409, @fgiunchedi wrote:

Pretty sure we can nuke this, I'll let @Eevans comment though as I'm not involved anymore

In T349758#9372338, @Jhancock.wm wrote:

@Eevans Hey my bad. newbie mistake. Papaul taught me how to fix this and you should be good now.

In T349758#9369347, @Eevans wrote:

In T349758#9369257, @Papaul wrote:

[ ... ]
@Eevans if i add the other 3 IP's addresses manually you should be good or do we have to re image all the hosts?

No, we can sort it out with a reimage.

In T349758#9369257, @Papaul wrote:

[ ... ]
@Eevans if i add the other 3 IP's addresses manually you should be good or do we have to re image all the hosts?

In T349758#9365653, @Papaul wrote:

@Eevans All your's

In T288470#9366377, @elukey wrote:

@Eevans Do you think that we could migrate AQS and Session store to PKI? If so I can open new tasks and propose a plan :)

This done; Thanks!

In T349875#9286181, @RobH wrote:

In T348021#9281147, @Kappakayala wrote:

@Clement_Goubert / @Joe could one of you help with the racking details?

I've split the racking task onto its own rather than block the ordering on racking details. Would one of you be able to populate the racking details section and assign back to me for followup please?

Thanks!

In T348995#9344181, @Jgiannelos wrote:

@Eevans i cant see the file you attached in your comment.

In T351320#9338030, @Jclark-ctr wrote:

Reseated hard drives. update idrac and bios firmware

In T350882#9328259, @Sfaci wrote:

In T350882#9327876, @Eevans wrote:

In T350882#9322001, @Sfaci wrote:

... We take the opportunity to use it to populate the local test env but, what we try to do fetching this data is to understand how it's structured and which edge cases we can expect to find. In fact, most of the time we are just reacting to an unexpected situation. For example:

• With the last sample data you provided us, we have found that the top_pageviews dataset contains the information about the articles in two different fields (articles and articlesJSON). For some rows the data is in the articles field and, for others, the data is in articlesJSON. It's something really unexpected and something we didn't know. Data from 2022-01 to 2022-10 seems to be the particular case. We got a "unexpected end of JSON input" error when requesting in that date range, and it was produced because we were looking for the data in the wrong field. The fix is to have an if sentence in the code to try to look for that value in the first field and, if it doesn't exist there, to try with the other. I guess that is due to some error when ingesting data to cassandra or something similar but it's something that I never imagined without taking a look to the real dataset.
• Some days ago I reached out to you to ask for data about mediarequest_top_files dataset. Doing that we learned how filepaths are stored in cassandra. Some punctuation marks are URL-decoded and others are stored as they are (or viceversa, or something similar. I don't remember it well). That was something we didn't know and I don't know if there had been another way to know it. I mean something like this: Angkor_-_Zentrum_des_K%C3%B6nigreichs_der_Khmer_(CC_BY-SA_4.0).webm

In both of these cases, wouldn't it be better to use the code (legacy aqs and/or analytics) to suss out the contract, rather than using queries to reverse-engineer them on a reactive basis?

Sure! In these specific cases we could use the code to see some specific details. Even so, sometimes, we didn't expect these issues and we didn't realize (our fault, of course). But, at the same time, we needed the data just to confirm what we were seeing in the code. But I meant in general, the common situation in which you have to code a new project using some new/existing datasets you don't have access. How to deal with situations like those? It's not common to have the code from the previous version and just have to rewrite it using a different tech stack.

In T350882#9322001, @Sfaci wrote:

... We take the opportunity to use it to populate the local test env but, what we try to do fetching this data is to understand how it's structured and which edge cases we can expect to find. In fact, most of the time we are just reacting to an unexpected situation. For example:

• With the last sample data you provided us, we have found that the top_pageviews dataset contains the information about the articles in two different fields (articles and articlesJSON). For some rows the data is in the articles field and, for others, the data is in articlesJSON. It's something really unexpected and something we didn't know. Data from 2022-01 to 2022-10 seems to be the particular case. We got a "unexpected end of JSON input" error when requesting in that date range, and it was produced because we were looking for the data in the wrong field. The fix is to have an if sentence in the code to try to look for that value in the first field and, if it doesn't exist there, to try with the other. I guess that is due to some error when ingesting data to cassandra or something similar but it's something that I never imagined without taking a look to the real dataset.
• Some days ago I reached out to you to ask for data about mediarequest_top_files dataset. Doing that we learned how filepaths are stored in cassandra. Some punctuation marks are URL-decoded and others are stored as they are (or viceversa, or something similar. I don't remember it well). That was something we didn't know and I don't know if there had been another way to know it. I mean something like this: Angkor_-_Zentrum_des_K%C3%B6nigreichs_der_Khmer_(CC_BY-SA_4.0).webm

In T350882#9319760, @Eevans wrote:

The file is quite large, so I've placed it at https://people.wikimedia.org/~eevans/local_group_default_T_top_pageviews.csv for download. Let me know when you have it so that I can clean it up.

The file is quite large, so I've placed it at https://people.wikimedia.org/~eevans/local_group_default_T_top_pageviews.csv for download. Let me know when you have it so that I can clean it up.

In T343273#9316165, @Sfaci wrote:

Hi again @Eevans!
I'm sorry for bothering again. I'm here to ask for new data. We need some new one about pageviews to debug for a bug we have in production (it seems to be related with some specific date for specific dates). And endpoint is failing for a specific range of date and we need to add a couple of years to the script to fetch that data.

The script is https://gitlab.wikimedia.org/repos/generated-data-platform/aqs/aqs-docker-cassandra-test-env/-/blob/more-data-top-pageviews/scripts/local_group_default_T_top_pageviews.sh?ref_type=heads. It's in a new branch I have created for this change. I just added 2021 and 2022 in the year range to fetch data for these dates.

Thank you very much!

This has been pushed out; The Bullseye upgrade is unblocked

In T349228#9314266, @gerritbot wrote:

Change 972461 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: add .../aqs/deploy/src/ to Environment

https://gerrit.wikimedia.org/r/972461

I'm not sure why this is the case (i.e. what changed to require this), but it is enough to add /srv/deployment/analytics/aqs/deploy/src to NODE_PATH.

In T348993#9280937, @Eevans wrote:

[ ... ]

Remaining steps:

• Get r968717 reviewed
• Create the schema in production & dev
• Create the role, grants, and corresponding credentials
• Update the deployment charts to source the credentials
• Establish a protocol for loading the new tables (since we probably do not have the space to store this twice)

The last two items probably warrant opening separate tickets for.

[ ... ]

I added an item to the acceptance criteria about having a strategy for migration that works within the capacity constraints of the cluster. Hopefully this is the right issue for this, let me know if that's not the case and I can move it elsewhere (including to a separate ticket).

In T348993#9280937, @Eevans wrote:

[ ... ]

Remaining steps:

• Get r968717 reviewed
• Create the schema in production & dev
• Create the role, grants, and corresponding credentials
• Update the deployment charts to source the credentials
• Establish a protocol for loading the new tables (since we probably do not have the space to store this twice)

The last two items probably warrant opening separate tickets for.

[ ... ]

In T348995#9299509, @Jgiannelos wrote:

Initial patch here after bootstrapping the nodejs env and gitlab CI:
https://gitlab.wikimedia.org/repos/content-transform/nodejs-cassandra-storage/-/merge_requests/1

@Eevans will you be available to take a look at this patch, mostly for the cassandra specific changes?

In T348993#9280937, @Eevans wrote:

[ ... ]

Remaining steps:

• Get r968717 reviewed
• Create the schema in production & dev
• Create the role, grants, and corresponding credentials
• Update the deployment charts to source the credentials
• Establish a protocol for loading the new tables (since we probably do not have the space to store this twice)

The last two items probably warrant opening separate tickets for.

[ ... ]

Ok, we need to formalize/document this better on the Data-Persistence side of things, but best-practice should be to keep the schema with the implementing code; Changes to the schema should move in lock-step with the code changes that necessitated them. To that end, I've opened: r968717.