Host has been downtimed. Accidentally associated to wrong ticket: https://phabricator.wikimedia.org/T367825#9908323

In T367442#9903813, @VRiley-WMF wrote:

Hey @RKemper would Thursday work for you? Around 12:00 EST?

In T367442#9890151, @VRiley-WMF wrote:

@RKemper When is there a preference on when we could schedule this?

Placeholder added: https://wikitech.wikimedia.org/wiki/SLO/Search#Service_Level_Indicators_(SLIs)

Confirm or deny that we do plan on setting an availability SLO target around the metric "more than 0.1% of requests to Elastic from the MW app servers fail over a 5m period".

Here's what it looked like before the puppet patch:

@dancy Should be fixed now. Here's a previously failing patch I ran a recheck on: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1028565/4#message-8d03a9f159d1ec58172e8606d84deaf43bbcbbb0

Kicked off the second run like so:

we saw that this took about 3702 minutes, or about 2.57 hours

Typo you'll want to fix here and in the original: 2.57 days

I'm realizing I don't remember enough about how we load specific graph splits (scholarly vs main). But it's possible we won't need the above nfs patch if our previous process was to manually download the relevant dump file.

Actually we'll use wdqs1021 since we're not confident NFS will work seamlessly between eqiad and codfw (we use nsf to procure the dumps that the data-reload is run from).

We'll use wdqs2023 to compare against wdqs1023 (same hardware). 1023 is scholarly-articles so that's what we'll want to load 2023 with.

Filled out the search preview SLI info in the documentation, and also updated the existing SLIs to make it more clear what exactly is being measured (page render, etc).

Here's an example curl command we can use to verify when the service is properly supporting application/sparql-results+xml:

Patched was merged and data-transfer reran; this is done.

Looks good on our end, thanks!

Was able to get a puppet run on elastic2088, but since that run a couple hours ago the host is ssh unreachable (it hangs indefinitely). Seeing some concerning stuff in the drac via getsel on elastic2088.mgmt.codfw.wmnet:

Added further context to the SLI section of the documentation explaining what each query type actually means. I believe there's no more oustanding TODOs on this task.

In T339347#9565775, @Hannah_Bast wrote:

@RKemper Is your point that the queries should return a result? Neither DBLP nor Wikidata have the predicate foaf:name, so it's clear that both SERVICE queries return an empty result. Here is an example for a query that gives a result:

PREFIX schema: <http://schema.org/>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
SELECT DISTINCT ?editor ?editorName
WHERE {
  SERVICE <https://qlever.cs.uni-freiburg.de/api/wikidata> {
    wd:Q113544723 wdt:P179 ?editor.
    ?editor schema:name ?editorName.
  }
}

I vote that we choose the default based on the cluster being operated on. So 3 for eqiad and codfw, 2 for cloudelastic and 1 for relforge.

Created subtask for dc-ops' side of the decom. Resolving this parent ticket now.

With dc-ops having closed out the decom subtask, this should be all done.

In T353878#9664756, @Volans wrote:

elastic2088 is unreachable and reported as missing from PuppetDB by Netbox report. No host should be powered on with puppet disabled or not working for longer period of time. Please either reimage it or shut it down now and reimage it at a later stage (before powering it on).

In T358046#9670564, @VRiley-WMF wrote:

@RKemper Thanks for bringing this up! I missed running the script for this device. It's been run and decommissioned.

@VRiley-WMF In netbox I see cloudelastic1003 still listed as decommissioning, whereas the other cloudelastic hosts are marked as Offline. Is it just the step to set netbox status to Offline that we're missing or are there other steps that still need to be run on cloudelastic1003 as well?

Glancing at ryankemper@apifeatureusage1001:~$ sudo journalctl -u curator_actions_apifeatureusage_eqiad:

Had forgotten to properly assign dc-ops as well as tag for the DC. Straightened that out now, so this should be ready for dc-ops to do the decom.

With https://github.com/wikimedia/restbase/pull/1336 being merged, is this ticket now resolved?

This should be all done; @TJones can you confirm all is working as it should?

@fnegri @brouberol Yeah, Brian and I will work on getting this tested and merged. Thanks for the heads up!

@HinMar Okay, I think we've got the endpoints properly allowed. Queries appear to be working for me. Are you seeing the same?

@Hannah_Bast Okay, we figured out what was making the allowed endpoints not updated properly. https://w.wiki/6q2i doesn't get an error message anymore, although the query itself returns no results.

@Loz.ross Yes the change to allowed endpoints did not get properly deployed; we've fixed that now. However there's another issue now that we've gotten past the Service URI not being allowed:

Finished the upload process; next up is rolling restart of cluster and merge of https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-elasticsearch-image/-/merge_requests/7?commit_id=f1028a26dff38603bea67a8edda8337dab07bbfc

Added new threshold markers at 95% for the 4 SLO graphs. We may want to revise the % SLO upwards, but let's stick with 95% for now until we get another quarter of data.

In T351488#9505091, @HinMar wrote:

@RKemper : Thank you for your message. The project has ended, but we still kindly ask you to whitelist this endpoint. We at the Trier Center for Digital Humanities will continue to work with LOD beyond this one project and would be delighted to be able to run federated queries starting from Wikidata directed towards the MiMoTextBase. We have developed an approach in MiMoText that we now want to transfer and adapt for other domains in a new project (“LODinG” – Linked Open Data in the Humanities). We are still very interested in the 'wikiverse' and in gaining as much experience as possible in the area of 'federation', which we see as the absolute key to the LOD vision. We are also planning to provide a showcase and if the whitelisting could be done relatively soon, we would like to include this new 'federation direction'. Can you estimate how long it will take?

In T339347#9504488, @Hannah_Bast wrote:

Yes, https://qlever.cs.uni-freiburg.de/api/dblp is the URL for API calls, whereas https://qlever.cs.uni-freiburg.de/dblp (without the /api) is the URL of the QLever UI. Same for all the other endpoints.

For example, https://qlever.cs.uni-freiburg.de/api/dblp?query=SELECT+%2A+WHERE+%7B+%3Fs+%3Fp+%3Fo+%7D+LIMIT+10 gives you the results for SELECT * WHERE { ?s ?p ?o } LIMIT 10 as application/sparql-results+json .

@Loz.ross Sorry for the delay, we've added the endpoint. Can you confirm it's working with an example query?

In T339347#9431527, @Nikki wrote:

Could https://qlever.cs.uni-freiburg.de/api/wikimedia-commons also be added?

It looks like https://qlever.cs.uni-freiburg.de/api/wikidata is working now (https://w.wiki/8iGM) but https://qlever.cs.uni-freiburg.de/api/dblp (https://w.wiki/6m6B, from the description) is still not allowed.

@HinMar Sorry for missing this request - our bad! I see your earlier comment mentioned the project expiring by end of 2023. Is the project still ongoing and therefore we should still whitelist this new endpoint or should I instead close this ticket out?

Old masters are no longer master-eligible. They're still participating in the actual cluster; we're holding off on the physical decom until T355617 is done

Forgot to add the Bug: label but https://gerrit.wikimedia.org/r/c/operations/puppet/+/992826 is part of this ticket as well

This should be all done, with the new experimental services accessible at:

Experimental microsites are up and externally reachable:

We've rolled this out following the steps in https://wikitech.wikimedia.org/wiki/Cergen#Update_a_certificate