In T299871#7646679, @wiki_willy wrote:

Hi @Eevans - since the refresh for this host was just installed via T294377, are you ok if we ignore this alert and resolve the ticket? Thanks, Willy

@BTullis: Regarding which node(s) to (re)pool (presumably tomorrow, Jan 25?)...

See: Draft: Proof-of-concept Javascript integration tests

In T298516#7640651, @BTullis wrote:

Scratch that previous theory...

Now I look at it more closely, the heap exhaustion seems to correlate much more closely with when aqs1010 was pooled and receiving queries.
The strange thing is that only one host was pooled, but all hosts exhibited the behaviour.

My theory is that it is this bug which is affecting us: Memory leak in CompressedChunkReader

By way of an update:

In T298805#7622471, @MoritzMuehlenhoff wrote:

I added component/cassandradev for buster and stretch. For the import we can either pull them from the Apache Cassandra repository via Secure Apt (then we need to setup the PGP keys used to sign the repo) or alternatively given that you're part of upstream and these debs, maybe alternatively just send me the sha1 sums via mail and I'll doublecheck them before the import?

In T298516#7604802, @BTullis wrote:

I have created the heap dump file. I had to chown to the cassandra user, as even root couldn't connect to the running JVM.

root@aqs1014:~# sudo su - cassandra -s /bin/bash
cassandra@aqs1014:~$ jmap -dump:live,format=b,file=/srv/cassandra-b/tmp/aqs1014-b-dump202201071450.hprof 4468
Dumping heap to /srv/cassandra-b/tmp/aqs1014-b-dump202201071450.hprof ...
Heap dump file created

The file is 13 GB in size.

In T298516#7608491, @BTullis wrote:

[ ... ]
If anyone else has any ideas on how to get more useful information out of the dump file, that would be great.

In T298516#7605339, @BTullis wrote:

I have placed the resulting 2.6 GB bzip2 file at aqs1014.eqiad.wmnet:/home/btullis/aqs1014-b-dump202201071450.hprof.bz2 in case you'd like to take a copy off-host for analysis @Eevans.
I'm currently transferring it to my workstation for analysis. The original, uncompressed 13 GB file is also still in /srv/cassandra-b/tmp.

If this host goes into a period of sustained GC I will take another heap dump using a similar method.

In T298516#7600211, @odimitrijevic wrote:

Could we give the upgrade a try to see if it resolves the memory leak, and if not only then proceed with the profiling?

A heap dump is likely to be the best means of identifying what is holding up all of this memory. For this to be most effective, we'll want the heap from a JVM that is prominently exhibiting the problem, which means letting at least one node go for a minimum of one week, preferably two.

Looking at the last 30days of heap usage for 1010-a...

To summarize: We're projected to be at ~73% by the end of this fiscal year, and the ~15GB (~5GB * 3) of the Image Suggestions dataset is little more than a rounding error against this.

Repair is such a...complex, subject. So much so that I'm not sure how to do justice to all of the considerations in a phab comment. :/

In T281517#7577179, @JAllemandou wrote:

I assume we're looking at the latter, and will need to create database roles and corresponding credentials that match the job configuration passed to the loader, is this correct?

The loader take a user and password as parameters, and this allows both of the approaches you describe above (single or multi tenant). I think we should aim at the having a multi-tenant configuration, possibly per team owning datasets? This would prevent errors as you describe.
For the moment we use a single tenant (we were a single team loading data :) and I'm supportive of not doing multi-tenant for now as we're talking about only one dataset from a different team. I would like however that we ultimately try to implement proper multi-tenant configuration, and this would also entail changes on the loading side to provide correct credentials as secrets.

Ok, now for the image suggestions use-case:

I'm wondering how to tie this into the notion of tenancy for backing stores (Cassandra, for the time being). For example: Will we have a single tenant (read: a unique user w/ credentials et al) for the data loading process, or will we have many (presumably, one for each dataset being loaded)? In a world with a platform that permits arbitrary teams to own scheduled jobs that persist output to a backing store -in a more-or-less- self-service fashion, we would want to ensure that an aberrant change or misconfiguration of one job, cannot inadvertently step on the data of another (which separate credentials & permissions would provide).

To be clear: will these be applied on every Puppet run, or only when the file has changed?

In T280042#7503454, @WDoranWMF wrote:

@gmodena @Eevans What should happen to this task now? Should it close?

In T243544#7503448, @WDoranWMF wrote:

@Eevans @hnowlan Should this go in Data Platform's backlog... ?

In T293809#7476849, @JAllemandou wrote:

Hi folks - Sorry for the late answer I was off the past two days.
I did some work in sizing for AQS last year, you can read it here: https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS/Scaling/2020/Cluster_Expansion
I think that ~5Tb per year for AQS is a good gentle overestimate.

In T293808#7477730, @BPirkle wrote:

[ ... ]
For anyone who skipped to the bottom, the service can still support requesting suggestions by page title. But it will convert title to page_id outside the dataset, probably via the Action API. This carries with it a risk that pages may be renamed, so the page_title => page_id relationship may have changed after the dataset was generated. We'll document this consideration so callers are aware of this possibility.

In T293808#7473451, @BPirkle wrote:

In T293808#7473382, @Eevans wrote:

[ ... ]
So, we have pages, almost any of which could have recommendations suggestions, even ones that already have images. Then we have (qualifying) pages that are unillustrated, for which IMA will attempt to generate suggestions for. And finally, we have those that it was successfully able to do so for. If we are saying that this data set models unillustrated pages, and any corresponding image suggestions IMA was able to make, then we're OKish (the wisdom of distinguishing between the latter by a non-nil suggestion, notwithstanding). If however we later make refinements to IMA, or add one or more additional suggestion algorithms, any of which that is able to make suggestions for already illustrated pages, then we'll have a data model unable to make that distinction. I'm willing to cross that bridge when we come to it if everyone else, I just wanted to point it out.

To restate what you said (hopefully fairly), we have:

1. pages, almost any of which could have suggestions, even ones that already have images
2. qualifying unillustrated pages for which IMA will attempt to generate suggestions
3. pages IMA was able to generate suggestions for

I'm not sure I understand what distinction you're making between #1 and #2, so let's dig into that. The service doesn't know about pages in general, in an all-pages-on-a-wiki sense. It only knows about pages that are in the dataset provided to it. And the service doesn't know or care if these pages are unillustrated or if they already have images. I think of them as "under-illustrated" pages.

I guess what I'm not following is what issue arises with the data model if we includes pages that already have images. If the IMA decides that an existing page that already has one or more images needs more, what breaks?

In T293808#7473320, @BPirkle wrote:

[ ... ]

@Eevans , I share your discomfort with the current approach where some suggestions lack, well, suggestions. The empty image_id thing was always a bit hacky, and if we can do better as we move from an experimental prototype phase to something more resembling a real production service, I'm all in favor of it. But it does seem to me that "pages in need of an image" is a valuable and useful set of data regardless of whether we have Image Matching Algorithm suggestions for all those pages. And some clients have specifically requested pages from that broader set, so that that can use MediaSearch suggestions rather than Image Matching Algorithm suggestions.

Starting with our extant use-case(s):

In T293808#7471802, @kostajh wrote:

In T293808#7471788, @lbowmaker wrote:

@kostajh - what would happen if a page title changes between the image rec output and someone viewing the image rec then calling the API?

I should have clarified in my previous comment – we're only using page titles with the API for local development and beta wikis, where it's not (easily) possible to get the page ID in those wikis to match their production equivalents (e.g. page "Foo" on my local wiki has ID 1010 but on enwiki it is ID 9132808). So page title renames wouldn't really be a problem that anyone has to spend engineering effort on; if I want recommendations for page "Foo" in my local wiki and that's renamed to "FooBar" in production, then I'll just rename it in my local wiki.

In T294377#7466932, @Papaul wrote:

@Eevans just for curiosity, any reason we have no restbase hosts in row A ?

@JAllemandou do you have an prior art when it comes to AQS capacity planning, month-over-month growth, etc?

I would propose the following:

In T293808#7454112, @gmodena wrote:

In T293808#7449361, @Eevans wrote:

[...]

• Judging by the enwiki TSV file, only ~8.6% of the rows have an image, is this correct? If so, are we expecting to store entries without an image (read: without a valid recommendation)?

That's correct, and use case specific. For the Structured Data PoC, the API team expected a dataset with

1. a list of all unillustrated articles detected on a wiki.
2. at most three candidate images that match an unillustrated article.

An empty image_ids denotes the case of "unillustrated article with no recommendations". This semantic was required (IIRC) to compare this dataset with an Elasticsearch (MediaSearch) result sets. See https://phabricator.wikimedia.org/T274798.

Based on this Superset query, the dataset seems to consist of the following:

In T293808#7449369, @lbowmaker wrote:

[ ... ]

Second question, please see this thread here. I asked the same question and Gabriele confirmed the client team explicitly asked for these to be stored.

Based on the TSV files in imagerec_prod.tar.bz2, the dataset seems to consist of the following:

There has been a bit of out-of-band discussion between Hugh and I, and we should probably attempt to summarize that here:

The arrays are still degraded (the device state is removed); I think there is still more yet to be done

In T141541#7401719, @hnowlan wrote:

The attempted FQDN-use method appears to have failed - Cassandra claims there is an issue with the keystore format despite it being the same format/method as before:

INFO  [main] 2021-10-05 11:43:09,282 IndexSummaryManager.java:80 - Initializing index summary manager with a memory pool size of 614 MB and a resize interval of 60 minutes
ERROR [main] 2021-10-05 11:43:09,297 CassandraDaemon.java:749 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Unable to create ssl socket
        at org.apache.cassandra.net.MessagingService.getServerSockets(MessagingService.java:701) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:681) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:665) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:796) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388) [apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [apache-cassandra-3.11.4.jar:3.11.4]
Caused by: java.io.IOException: Error creating the initializing the SSL Context
        at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:201) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.security.SSLFactory.getServerSocket(SSLFactory.java:61) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.net.MessagingService.getServerSockets(MessagingService.java:697) ~[apache-cassandra-3.11.4.jar:3.11.4]
        ... 8 common frames omitted
Caused by: java.io.IOException: Invalid keystore format
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:666) ~[na:1.8.0_302]
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:57) ~[na:1.8.0_302]
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) ~[na:1.8.0_302]
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:71) ~[na:1.8.0_302]
        at java.security.KeyStore.load(KeyStore.java:1445) ~[na:1.8.0_302]
        at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:179) ~[apache-cassandra-3.11.4.jar:3.11.4]
        ... 10 common frames omitted

Could there be an issue with mismatching between hostname and FQDN? FQDN certificates still exist on the puppet master for investigation but keytool doesn't show many discrepancies between them despite the obvious change of CN. For now I am reverting.