In T307035#8436528, @Eevans wrote:

In T307035#8078353, @Cmjohnson wrote:

@Eevans take your time, I just want to make sure that we're not falling behind on-site. Let me know whenever you're ready.

@Cmjohnson, apologies for the amount of time that's passed, but we're good to go with these moves now.

In T324113#8442582, @hnowlan wrote:

The restbase instances haven't really been used for anything in a long time, but the restbase sunsetting team have just been made aware of them. It'd be nice if we could have a restbase instance running on some/one of them if possible, but I think it's reasonable to say that they are, have been and will be more useful as Cassandra test nodes rather than nominally being restbase.

@hnowlan Question: Do we still use this machine to run RESTBase on for anything, or is it purely for Cassandra these days? Can we provision the new machines as say cassandra-dev200[1-3]?

In T307035#8078353, @Cmjohnson wrote:

@Eevans take your time, I just want to make sure that we're not falling behind on-site. Let me know whenever you're ready.

This is complete with the deployment of Kask v1.0.10

This is complete with the deployment of Kask v1.0.10

Done.

This was my bad™, a misconfiguration of the sessionstore VM (profile::java::java_packages not set correctly) caused Cassandra to be down.

Complete.

Complete.

In T306895#8428977, @BTullis wrote:

Thanks for reminding me @Eevans - Yes, it's this ticket: T323692: Create puppet defined type for adding/updating/deleting secrets or other small files on HDFS

In T306895#8408387, @JAllemandou wrote:

Thank you @BTullis and @Ottomata for taking over this - this will be very useful :)

I think that I've gotten what I need from instanceof_cache & title_cache, they can be cleaned up.

In T317364#8416446, @Cparle wrote:

I assume, that whatever process loaded data into the suggestions table, also loaded into instanceof_cache and title_cache, is this assumption safe?

Yes, I think so ... can't be 100% sure because that process might not have always been working from complete or consistent datasets when we were developing the thing, but it definitely seems like an approach that has a good chance of working

Per T317364#8415200, let's hold off until we're sure this data can't be used to suss out T317364

In T317364#8412449, @Cparle wrote:

New ticket for cleaning up the other tables T323561

I'll close this if there are no objections @Eevans @kostajh @JAllemandou

In T320835#8404028, @dmaza wrote:

@Eevans you mentioned Swift expiry in T320675#8317876. Would it be acceptable to set X-Delete-At to something like 3 months in the future, and if the file gets used a week prior to the expiration date, use describe to reset X-Delete-At for another 3 months? (Date ranges are TBD and can be adjusted as we go)

Wouldn't this solve any potential leakage?

Here is what I mean https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Phonos/+/858423

This has been deployed to sessionstore (production). It still needs to be deployed to:

@Cparle are you able to post the code you used for this? Just spot checking I found some older entries.

In T306895#8395895, @Snwachukwu wrote:

We now have a custom AuthConfFactory that will be passed as a parameter to the job using spark.cassandra.auth.conf.factory. This should be deployed today. @Ottomata @Eevans The next step would be to have the file containing the Cassandra password loaded to HDFS.

In T322392#8394206, @Eevans wrote:

In T322392#8392192, @kostajh wrote:

In T322392#8378010, @Tgr wrote:

In T322392#8374232, @kostajh wrote:

@Tgr do you have concerns?

No, sounds good.

@Eevans please go ahead with adding rejected_reasons as set<text> when it's convenient for you. Thanks!

Sounds good; Let's shoot for tomorrow morning (US timezone) when we can do it during a lull in eqiad bootstraps (T307802).

In T322392#8392192, @kostajh wrote:

In T322392#8378010, @Tgr wrote:

In T322392#8374232, @kostajh wrote:

@Tgr do you have concerns?

No, sounds good.

@Eevans please go ahead with adding rejected_reasons as set<text> when it's convenient for you. Thanks!

In T305570#8375505, @Volans wrote:

I've also run the sre.dns.netbox cookbook, the DNS records are now live.

As the description notes, this is a breaking change. That said, I only see a couple of items in the feedback table, presumably test values of some sort? If there is no user-facing client code that would break as a result of a schema change, I could drop the rejected_reason attribute, and add rejected_reasons as set<text>.

Ok, so let's try to move this in a (more) constructive direction:

In T320835#8361321, @Eevans wrote:

In T320835#8361202, @dmaza wrote:

@Eevans we can't really do this "multi-hook" approach. It doesn't work.

That's disappointing; What changed?

In T320675#8364656, @MusikAnimal wrote:

I originally read T320675#8330640 as giving the go-ahead, but with the shared understanding there will be proper instrumentation and monitoring on our end moving forward. We can certainly commit to that. Our first pilot wikis (T316013) are comparatively very small, but will give us a better idea of the costs you're asking about, monetary or otherwise.

Since T320835: Minimize file leakage by expiring objects when using swift file-backend appears to be in jeopardy (see: T320835#8361202), perhaps we can revisit some of these concerns:

In T320835#8361202, @dmaza wrote:

@Eevans we can't really do this "multi-hook" approach. It doesn't work.

In T283838#8358215, @Eevans wrote:

[ ... ]

To update to the latest gocql driver release (1.2.1 as of the time of this writing), will roughly require:

• Packaging golang-github-pierrec-lz4.v4-dev (not currently in any version of Debian, but package source already exists on Salsa), and uploading it to sid
• ...

Kask's dependencies are sourced entirely from Debian, the rationale for which can be found documented here. The most current version of the gocql driver in any version of Debian is 0.0~git20191102.0.9faa4c0-4 (the version we are already using); Continuing this practice will mean creating an updated package and adding it to a repository (preferably Debian, but possibly our own in the near-term).

In T320831#8343829, @Eevans wrote:

[ ... ]

Size and Growth:

• Still being investigated as the final algorithm is worked out -> https://phabricator.wikimedia.org/T315976
• Size is not known yet but realistically would be a multiple of the existing page level image suggestions (10x?)

What is it that makes it a multiple of the existing (page-based) suggestions? Will the algorithm be somehow producing that many more results? Will we be (for example) storing N * num_sections suggestions (where N is the current per-page limit)? I guess what I'm wondering is whether this will result in a corresponding change to the size of result responses as well.

[ ... ]

• I don’t think we need to have the new section id field as part of the key. Requests would still be on wiki/page_id (requestors wouldn't likely know the section id - SD correct me if I'm wrong?) - Growth may need to add a filter on items with a section id

Yes, this is important. If there is a need to query by-section, then changes to the data model will be needed. The sooner we establish this, the better.

The existing image suggestions data pipeline suggests images at an article level. There is a new data pipeline being built that will suggest images at an article section level.

The output of the new data pipeline is expected to the be the same as the article level suggestions but with the addition of a field to contain the section identifier.

When the new data pipeline is built the existing article level data pipeline will continue to run and the output consumed as it's done currently.

Write Frequency and Method:

• Weekly bulk from Airflow > Cassandra connector job

Size and Growth:

• Still being investigated as the final algorithm is worked out -> https://phabricator.wikimedia.org/T315976
• Size is not known yet but realistically would be a multiple of the existing page level image suggestions (10x?)

By this point, things seem to have stabilized. Utilization ebbs and flows between a floor of ~50G and a ceiling of ~100G. While this is considerably more than what we assume the live set to be (the extra being TTL'd tombstones), it still only represents a worst-case of ~30% utilization. Since this is a single-purpose cluster, there is no value to be had from additional free space (and freeing that space would generate additional I/O, and shorten the lifespan of the SSDs).

To summarize what was discussed during a Data Persistence meeting earlier today (in no particular order):

In T306349#8328405, @Cparle wrote:

Actually ... now that I think about it, @VirginiaPoundstone and @BPirkle is there scope to move some of the business logic into the API, so that each client doesn't have to implement it separately? IMO it'd make a lot of sense to do that

If there is scope, then Growth and us and Android need to have a little chat to figure out what we need ...

In T320739#8320182, @kostajh wrote:

In T320739#8315289, @Eevans wrote:

[ ... ]

TL;DR If such a service is going to accept POSTs for feedback submission, it would handle them by submitting an event.

AIUI need to proxy the feedback event through MediaWiki unless we adjust the EventGate configuration for the relevant stream to allow for external event creation.

To summarize what was discussed on Slack:

In T320739#8315344, @BPirkle wrote:

I feel like ImageSuggestions is growing beyond an "experiment" at this stage, and that this sort of API falls naturally under the purview of the API platform team rather than inside Product

While I don't disagree, I'm the only developer on API Platform with MediaWiki development experience, and my time is already pretty much spoken for. If this task involves MediaWiki development (as presumably it does if we're collecting user-specific feedback), then we'd need to reshuffle some things for API Platform to take it on. I have no objections to that - I'm here to do whatever is most needed - so I'll let the various folks with "Manager" in their title sort that out. :-)

In T320739#8315041, @Cparle wrote:

So if I understand this correctly, @kostajh 's proposal is for the Growth team to

1. write and maintain an Image Suggestions http api to proxy GET requests to the existing image suggestions cassandra gateway inside the GrowthExperiments extension
2. possibly also provide a POST endpoint which would involve writing to Cassandra (or maybe just would write an event to EventGate that would eventually propagate to Cassandra)

If the growth team have the people/time to do this then it might expedite things, but I'm not sure it's a good idea. I feel like ImageSuggestions is growing beyond an "experiment" at this stage, and that this sort of API falls naturally under the purview of the API platform team rather than inside Product

In T320675#8313052, @MusikAnimal wrote:

In T320675#8313006, @Eevans wrote:

Is there no point when a pronunciation is altered or replaced, invalidating the mp3? Or removed?

I suppose that's where the deleteOldPhonosFiles.php script comes into play. Seemingly this is the only thing Extension:Score does to guard against the same concern.

In T320675#8312854, @MusikAnimal wrote:

In T320675#8312841, @Eevans wrote:

Oh, I missed these entirely! Would you rather we reopen and use one of these (T314789 presumably), or continue here?

No worries! Here is fine. Those old tasks are out of date as it is. For one, we're not using WAV anymore, and:

There is also some concern about how retention is managed, if we're treating this as a cache.

Let me be clear that this is NOT a proper caching technique (though we did envision it that way initially). What we want is persistent storage, pretty much exactly like Extension:Score. Files are shared globally, last indefinitely (but note we do have deleteOldPhonosFiles.php), etc.

In T320675#8312763, @MusikAnimal wrote:

In T317417#8312730, @Eevans wrote:

Thanks for the info; Apologies for not following up on this sooner. I asked about utilization because we (Data-Persistence) are trying to engage with projects that need storage earlier, understand the requirements, be in a position to offer feedback, and plan accordingly. Ideally, earlier would be earlier than where we are now, but it would still be great to run through your storage requirements.

Since this seems out of scope for this ticket (sorry about that) -and since I didn't find a suitable existing issue- I've stubbed out T320675 for this.

I should have linked to it in my reply at T317417#8280934, but we have T314789 and T309315 from much earlier in this project where we tried to consult with Data Persistence.

In T317417#8280934, @MusikAnimal wrote:

In T317417#8280822, @Eevans wrote:

How big are the files themselves (min, max, average)? How many do we expect; What's the anticipated total storage? Read & write rates?

We aren't setting a maximum file size currently, but we do have a maximum amount of IPA that can be passed to the parser tag. That is currently set to 300 bytes (T316641). The generated MP3 in that case is somewhere in the neighborhood of 40-50kb, but on average files are going to be maybe be 3-5kb. The number of files generated depends on how well the communities adopt this feature. If fully rolled out across all wikis (we use Phonos everywhere we show IPA), we're probably looking at many hundreds of thousands of files, but I think it will be quite a while before we reach that point. Read rates are estimated to at around ~1.8 million a month (T307625). Write rates is harder to estimate, again depending on how communities adopt this feature. Since most wikis have a Template:IPA or something similar, the Phonos parser tag will likely be put there. Thus, the initial rollout will see a very high number of writes (directly proportional to the number of transclusions of Template:IPA), but we are building our own type of job and making it go slower than the normal job queue rate (T318086). After the rollout, writes will likely be by comparison relatively rare. Note however some communities might prefer to opt-in to using Phonos on a case-by-case basis, making the rollout longer and hence lower write rates.

In T317364#8290265, @JAllemandou wrote:

Deletion has been done this morning using the following actions:

# On stat machine, in a screen, launch a spark-shell
spark3-shell  --master yarn --executor-memory 8G   --executor-cores 2   --driver-memory 8G   --driver-cores 2  --conf spark.dynamicAllocation.maxExecutors=90 --jars hdfs:///wmf/cache/artifacts/airflow/analytics/spark-cassandra-connector-assembly-3.2.0-WMF-1.jar --conf spark.cassandra.connection.host=aqs1010-a.eqiad.wmnet:9042 --conf spark.cassandra.auth.username=aqsloader --conf spark.cassandra.auth.password=cassandra

# Then execute scala code

import com.datastax.spark.connector._

val r = spark.sql("""
SELECT DISTINCT
  -- Selecting C* primary key columns (except for image, not needed)
  wiki, page_id, id
FROM analytics_platform_eng.image_suggestions_suggestions
WHERE
  -- Snapshot list extracted from  /user/hive/warehouse/analytics_platform_eng.db/image_suggestions_suggestions
  -- Keeping snapshot that are before Sept 9th as after that date data normally has TTL
  snapshot IN ('2022-05-02', '2022-05-16', '2022-06-13', '2022-06-20', '2022-06-27', '2022-07-04', '2022-07-11',
               '2022-07-18', '2022-07-25', '2022-08-01', '2022-08-08', '2022-08-15', '2022-08-22', '2022-08-29', '2022-09-05')
""").repartition(6).rdd

val f = new RDDFunctions(r)

f.deleteFromCassandra("image_suggestions", "suggestions", keyColumns = SomeColumns("wiki", "page_id", "id"))

In T317417#8260312, @TheresNoTime wrote:

In T317417#8259615, @MatthewVernon wrote:

Reading the discussion on https://gerrit.wikimedia.org/r/c/operations/puppet/+/831955/ I'm left a bit confused - is the aim to use swift as essentially a cache for these sound files? If so, how is expiring them being managed?

Correct (though we've been reminded "persistent storage" is more appropriate, as we kept referring to it as "cache"). Due to the small size of these files, and the relative inexpense of storage space, expiring is being handled by a maintenance script run infrequently (yearly?)