In T358308#9711611, @Bawolff wrote:

Looking at recent logs, it does seem like some of the failures might be caused by things other than a timeout (There was one recently that failed after 17 seconds). However i still think the larger files are failing due to some sort of timeout as they tend to fail at around 202 second mark, which is pretty suspicious. Sample size is pretty low since users know large uploads don't work so they don't try and upload large files.

In T362518#9712748, @hashar wrote:

mediawiki/vagrant (php7.4-wikidiff2)

MediaWiki-Vagrant is based on Buster, then it is no more officially supported. There is a task to update it to Bullseye T319167

In T360531#9668738, @BTullis wrote:

There is also an opportunity to remove the need to specify all of the cassandra server IPs by implementing T359423: Migrate charts to Calico Network Policies for this chart.

In T358308#9698966, @Bawolff wrote:

In T358308#9696868, @hnowlan wrote:

Kubernetes jobrunners were using the default max_execution_time of 180s, which might be the culprit for these behaviours. I've bumped it to match the metal jobrunners.

Hmm, that might have also caused problems for web requests too, as in the current configuration, MediaWiki assumes that web POST requests can last more than 200 seconds, and there is code that tries to ensure that if a request is taking too much time that it is cleaned up gracefully and doesn't die inside a critical section.

As an aside, the jobs do seem to be failing after about 200 seconds, which is the timeout for a non-job runner POST request. However, i read through the code, and don't see how that limit could be applied here, since its based on SERVERGROUP and the log entries from the jobs have the correct SERVERGROUP.

Do we have a plan for when and how we'd like to move this to production?

Kubernetes jobrunners were using the default max_execution_time of 180s, which might be the culprit for these behaviours. I've bumped it to match the metal jobrunners.

Having looked at this a little further, I have a strong suspicion the timeout is coming from somewhere other than the config that we currently have defined (perhaps a default somewhere that isn't explicit). Going to keep looking into it, there's a chance this timeout will be affecting other mw-on-k8s charts but just hasn't bitten us because they don't use the longer timeouts the jobrunners do.

In T361576#9689907, @Mvolz wrote:

We don't strictly need to support $lang/v1/data/citation?format=$format&search=$search or $lang/v1/data/citation/api?format=$format&search=$search. Incoming requests with get parameters were deprecated many years ago and everyone moved to the restbase pattern. This was more a nice to have.

What external paths should we be routing to what internal paths for this service?

After spending some time on this, I think we might have some difficulty with supporting the third citoid pattern of rewriting $site/v1/data/citation/$format/$search to /api?format=$format&search=$search. Envoy's support for this kind of mangling isn't very complete as it's something it kinda expects services to do elsewhere.

Deployed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1016794 to increase Typha memory limits

It looks like the network setup between staging and cassandra-dev is all what we would expect. Pods are allowed to connect, and the cassandra firewall allows the staging IP range. While nsentered into the mobileapps staging pod:

root@kubestage1003:/home/hnowlan# time nc -z 10.192.16.15 9042  && echo ok

In T338425#9680124, @Mvolz wrote:

So yeah, this might be something we could add to Rest Gateway. Honestly it's more nice to have than anything... alternatively could have people put in their own accept-language header instead of passing on the one in the wiki automatically. Basically affects how people are using the gui api endpoint, not how the service functions inherently.

I've merged the change to increase the envoy timeout to the jobrunners (thanks for the patches!) - looks like run times are increasing for the jobs in question but I'll continue to monitor before we take any further changes.

Declined in favour of T297222

In T360597#9657181, @Jgiannelos wrote:

From logs I think there are 2 things to investigate:

• What happened since ~10th March ?
• Why traffic was completely dropped on the 19th of March on codfw?
  • Could be related to the datacenter switchover ?
  • Meanwhile eqiad kept having traffic before and after that time.

codfw:

eqiad:

Turnilo says that this is mostly being caused by clients using the mobile apps, various versions. Logstash makes it clear that Wikifeeds itself is timing out when talking to restbase. The URL pattern in that query shows an oddity - request.url shows the featured link (or similar) but the failing internalURI is almost always always an image. I can't find reference for it but I'm fairly sure we've seen this timeout-upon-image-request behaviour in restbase before, but with changeprop.

Is this the same bug as T356369?

In T358001#9563665, @Jhancock.wm wrote:

@hnowlan I've replaced the network cable on both of these. These are both connected to a 1G switch so there is no SFP to replace in this case.

If this does not fix the issue lmk and we can upgrade the idrac and bios firmware.

In T358195#9567724, @isarantopoulos wrote:

I added some more logging so that we can get a better break down of where the time is being spent and it turns out that it is the MWAPI that is taking too long

time curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/article-descriptions:predict" -X POST -d '{"lang": "fr", "title": "Tour_Eiffel", "num_beams": 2, "debug": 1}' -H  "Host: article-descriptions.experimental.wikimedia.org" -H "Content-Type: application/json" --http1.1
{"lang":"fr","title":"Tour_Eiffel","blp":false,"num_beams":2,"groundtruth":"monument de Paris, France","latency":{"wikidata-info (s)":0.0849905014038086,"mwapi - first paragraphs (s)":6.222133636474609,"total network (s)":6.26397967338562,"model (s)":4.969624042510986,"total (s)":11.23367714881897},"features"

In the specific example 16 requests are made and all 16 extracts are returned to be processed by the model. Although the requests seem to be made asynchronously they take approx 6 seconds to complete.
Regarding these requests, I think we should also be spcifying the Host header when making requests to Rest Gateway here

Done

All (non-videoscaler) jobs migrated to Kubernetes jobrunners. Videoscaler work tracked in T355292

We saw a recurrence of this issue this morning, with a large number of jobs failing with 503 messages from eventgate for a short period. Envoy also saw failures connecting to eventgate around the same time.

Has there been a final decision on access and the scope of this ticket?

Does this ticket supersede T355170? Change created but dependent upon approval by group approvers (@odimitrijevic , @WDoranWMF, @Ahoelzl @Milimetric)

Any word on approvals for this ticket @odimitrijevic , @WDoranWMF, @Ahoelzl @Milimetric?

Done

Reimage was successful, networking survived a reboot. All done!

I tried another reimage and it currently proceeding successfully - maybe replacing the SFP did the job? This is all a bit inexplicable.

Reimaging fails still after these changes fwiw - however, a reboot has broken network connectivity again?! The host is up and rebooted in the management interface, but I can't ssh in again.

This command also fails - but interestingly the host itself appears to have lost network connectivity. ethtool reports that the link is up but I can't connect in or out and the arp table is empty, I can only get in via the management console.

In T355333#9538260, @Jhancock.wm wrote:

idk if this would help, but can we run the provisioning script with the --no-dhcp and --no-user tags. to catch any bios settings that might have changed?

To limit the impact of this rollout we'll be gradually deploying to restbase servers either in small numbers or one at a time rather than doing a full scap rollout.

In T357198#9534354, @Eevans wrote:

In T357198#9533155, @hnowlan wrote:

Could Too many eqiad mediawiki originals uploads be a red herring? The traffic jumps are all in codfw.

Honestly, that alert is what prompted me to look at the Swift dashboards where network throughput corresponds with the increase on cr2-codfw:

Looking back, I guess I didn't question why that alert had fired for eqiad (which seems pretty curious to me now that you bring it up).

In T355333#9529467, @Jhancock.wm wrote:

I reseated the NIC and it connected. when I rebooted it went down again and didn't come up. swapped it out and rebooted it. stayed up this time. should have replaced the cable as well the first time D= It -should- stay up this time. lmk if it acts up again.

also tried the racreset trick and I'm getting a straight 404 error on the idrac login.

User jaz has been added to analytics-privatedata-users, you should be able to use superset with the associated password.

Moving to stalled pending approval from an analytics-privatedata-users owner (@odimitrijevic , @WDoranWMF, @Ahoelzl @Milimetric)

Moving to stalled pending approval from an analytics-privatedata-users owner (@odimitrijevic , @WDoranWMF, @Ahoelzl @Milimetric)

Could Too many eqiad mediawiki originals uploads be a red herring? The traffic jumps are all in codfw. I'm not sure what's actionable in this ticket.

Required fields empty, no domain specified. Declining.

This is kinda verging on a UBN for us as we go into the weekend because it's causing a lot of spam and it'll hide other error prod states for jobqueues.

This is now rendering after changes to resources for thumbor workers.

This may have been a question of resources, this image is now rendering after some recent bumps.

In T355333#9529271, @Jhancock.wm wrote:

I can't seem to access the idrac remotely. Is it okay if I power down the server at this time?

The train rollout hasn't fixed this issue and we're getting alerts every hour for error spikes on jobrunners - this is beginning to hide other prod issues with the jobrunners so this is a pretty serious concern for us. Apologies, the errors in the ticket *have* been fixed, however we're still seeing spikes of the other errors reported, and they are causing alerts to fire every two hours:

Good catch! Unfortunately I'm still seeing the same PXE behaviour failing on boot

To be explicit about routing differences - we have a rule for /api/rest_v1/metrics/* that directs traffic to the rest gateway and we have a rule for *most* other things under /api/rest_v1/* that defaults to restbase. There are some exceptions for services that have been migrated out of restbase.

In T355892#9515591, @Jgiannelos wrote:

This change looks like is causing an issue. From apps team:

We seem to be getting intermittent 404s for certain urls, e.g. https://maps.wikimedia.org/static/webgl/wikisprites%402x.json

From a quick look running this request in all maps nodes only maps1009 is returning 404s for this URL. I think we should depool maps1009 and 2009 again and figure out whats wrong.
Probably its running some older version of kartotherian that doesn't expose this static file.

More specifically maps1009 is missing the static folder to serve:

jgiannelos@maps1009:/srv/deployment/kartotherian/deploy/src$ stat static
stat: cannot stat 'static': No such file or directory

Failing disk:

root@aqs1013:/home/hnowlan# udevadm info --query=all --name=/dev/sde| grep SERIAL
E: ID_SERIAL=MZ7KH1T9HAJR0D3_S4KVNA0MB04213
E: ID_SERIAL_SHORT=S4KVNA0MB04213
root@aqs1013:/home/hnowlan# dmesg | grep sde | tail
[6107332.316343] sd 6:0:0:0: [sde] tag#13 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[6107332.316359] sd 6:0:0:0: [sde] tag#13 CDB: Read(10) 28 00 02 e9 0f 80 00 00 08 00
[6107332.316365] blk_update_request: I/O error, dev sde, sector 48828288 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[6107332.327008] Buffer I/O error on dev sde1, logical block 6103280, async page read
[6107332.335094] sd 6:0:0:0: [sde] tag#1 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[6107332.335101] sd 6:0:0:0: [sde] tag#1 CDB: Read(10) 28 00 df 8f df 80 00 00 08 00
[6107332.335107] blk_update_request: I/O error, dev sde, sector 3750748032 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[6107332.345913] Buffer I/O error on dev sde2, logical block 462739952, async page read
[6107370.360633] sd 6:0:0:0: [sde] tag#16 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[6107370.360637] sd 6:0:0:0: [sde] tag#16 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00

This appears to have trailed off around 1400 on the 29th, but if there is a risk of this recurring it'd be great if we could avoid these exception spikes in future.

A surge in this error started around 0700 on the 27th of January, and seems to only occur in volume upon hewikisource (there was a brief spike on itwikivoyage on the 26th also).

In T355730#9484118, @MatthewVernon wrote:

Presumably we want to restrict access somewhat beyond "everything the cassandra user can do"? At which point a separate user to sudo to seems like a sensible idea unless it's a lot of hassle...