fgiunchedi (Filippo Giunchedi)
Awesome

Projects (18)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 8:06 AM (158 w, 5 d)
Availability
Available
IRC Nick
godog
LDAP User
Filippo Giunchedi
MediaWiki User
Filippo Giunchedi

Recent Activity

Fri, Oct 13

fgiunchedi updated the task description for T177196: Port non-deprecated Diamond collectors to Prometheus.
Fri, Oct 13, 1:43 PM · Patch-For-Review, User-fgiunchedi, Goal, Operations
fgiunchedi updated the task description for T177196: Port non-deprecated Diamond collectors to Prometheus.
Fri, Oct 13, 1:42 PM · Patch-For-Review, User-fgiunchedi, Goal, Operations
fgiunchedi closed T127762: Update Debian Package for Scap3 as Resolved.

Yup all done @thcipriani !

Fri, Oct 13, 1:02 PM · Patch-For-Review, Scap
fgiunchedi created T178151: Add UDP monitor for pybal.
Fri, Oct 13, 10:13 AM · Operations, Traffic, Pybal
fgiunchedi added a comment to T150734: Make Thumbor logs available in ELK.

Last change merged and deployed, thumbor hostname is now in logstash \o/

Fri, Oct 13, 9:37 AM · Patch-For-Review, User-fgiunchedi, Performance-Team, Thumbor
fgiunchedi added projects to T178078: RESTBase logs disappeared from logstash: Operations, Traffic.
Fri, Oct 13, 8:10 AM · Patch-For-Review, Traffic, Operations, Wikimedia-Logstash, Services (watching)
fgiunchedi added a comment to T178078: RESTBase logs disappeared from logstash.

I'm investigating this and the issue seems to lie on lvs boxes, where ipvsadm the gelf service logstash-gelf_12201_udp isn't routed anywhere on lvs1003

Fri, Oct 13, 8:08 AM · Patch-For-Review, Traffic, Operations, Wikimedia-Logstash, Services (watching)
fgiunchedi awarded T175341: Review and fix PDU settings for syslog/ntp/email servers a Love token.
Fri, Oct 13, 7:11 AM · DC-Ops, Operations

Wed, Oct 11

fgiunchedi added a comment to T177747: grafana-labs often fails to generate graphs with c.datapoints is undefined.

Nice find indeed @hashar !

Wed, Oct 11, 1:15 PM · Graphite, Cloud-VPS
fgiunchedi moved T177739: Integrate stretch 9.2 point release from Backlog to Doing on the User-fgiunchedi board.
Wed, Oct 11, 12:48 PM · User-fgiunchedi, Operations
fgiunchedi moved T177196: Port non-deprecated Diamond collectors to Prometheus from Backlog to Doing on the User-fgiunchedi board.
Wed, Oct 11, 12:48 PM · Patch-For-Review, User-fgiunchedi, Goal, Operations
fgiunchedi moved T177195: Reduce technical debt in metrics monitoring from Backlog to Doing on the User-fgiunchedi board.
Wed, Oct 11, 12:48 PM · User-fgiunchedi, Technical-Debt, Goal, Operations
fgiunchedi created T177920: unattended-upgrades not upgrading "-wikimedia" packages automatically in wmcs.
Wed, Oct 11, 10:02 AM · Cloud-VPS, cloud-services-team
fgiunchedi added a comment to T177078: Decide on casing convention for JMX metrics in Prometheus.

I've re-read the thread and I think I have a proposal to move things forward.

Wed, Oct 11, 8:59 AM · Patch-For-Review, monitoring, User-Elukey, Analytics-Kanban, Analytics-Cluster
fgiunchedi closed T152791: Improvements to Ganglia-equivalent Prometheus dashboards as Resolved.

I'm resolving this task as all major use cases have been covered.

Wed, Oct 11, 8:38 AM · User-fgiunchedi, Prometheus-metrics-monitoring, Operations
fgiunchedi added a comment to T177225: Uninstall ganglia from the fleet.

Looks like this is the relevant list https://wikitech.wikimedia.org/wiki/Prometheus#Ganglia_plugins to see which plugins have been replaced with what. Can any updates be made to that list?

Wed, Oct 11, 8:35 AM · Patch-For-Review, Operations, monitoring
fgiunchedi added a comment to T136312: Encrypt syslog traffic.

The eqiad change was reverted yesterday due to (among the problem above) labservices machines hanging and not being able to successfully talk TLS with syslog servers. I'll be conducting more tests and apply the change in eqiad more gradually.

Wed, Oct 11, 8:23 AM · Patch-For-Review, monitoring, User-fgiunchedi, Operations

Tue, Oct 10

jcrespo awarded T174932: Recurrent 'mailbox lag' critical alerts and 500s a Like token.
Tue, Oct 10, 3:59 PM · Patch-For-Review, Operations, Traffic
fgiunchedi added a project to T177739: Integrate stretch 9.2 point release: User-fgiunchedi.
Tue, Oct 10, 3:05 PM · User-fgiunchedi, Operations
fgiunchedi added a comment to T136312: Encrypt syslog traffic.

syslog-tls is deployed everywhere but esams (coming shortly)

Tue, Oct 10, 1:42 PM · Patch-For-Review, monitoring, User-fgiunchedi, Operations
fgiunchedi created P6099 https://phabricator.wikimedia.org/T136312.
Tue, Oct 10, 1:39 PM
fgiunchedi created T177821: Allow syslog-tls in analytics towards wezen/lithium.
Tue, Oct 10, 9:47 AM · Operations
fgiunchedi created T177820: Allow syslog (-tls) from both wezen and lithium in labs.
Tue, Oct 10, 9:39 AM · netops, Operations
fgiunchedi updated the task description for T177196: Port non-deprecated Diamond collectors to Prometheus.
Tue, Oct 10, 9:27 AM · Patch-For-Review, User-fgiunchedi, Goal, Operations

Sat, Oct 7

Dzahn awarded T177225: Uninstall ganglia from the fleet a Love token.
Sat, Oct 7, 12:53 AM · Patch-For-Review, Operations, monitoring

Fri, Oct 6

fgiunchedi updated the task description for T177196: Port non-deprecated Diamond collectors to Prometheus.
Fri, Oct 6, 3:28 PM · Patch-For-Review, User-fgiunchedi, Goal, Operations
fgiunchedi updated the task description for T177196: Port non-deprecated Diamond collectors to Prometheus.
Fri, Oct 6, 2:15 PM · Patch-For-Review, User-fgiunchedi, Goal, Operations
fgiunchedi updated the task description for T177196: Port non-deprecated Diamond collectors to Prometheus.
Fri, Oct 6, 1:32 PM · Patch-For-Review, User-fgiunchedi, Goal, Operations
fgiunchedi updated the task description for T177196: Port non-deprecated Diamond collectors to Prometheus.
Fri, Oct 6, 1:23 PM · Patch-For-Review, User-fgiunchedi, Goal, Operations
fgiunchedi added a comment to T177199: Add Prometheus client support for varnish/statsd metrics daemons.

IMO we could approach the problem of getting the stats above to Prometheus in at least two ways:

Fri, Oct 6, 1:01 PM · Traffic, User-fgiunchedi, Goal, Operations
fgiunchedi added a comment to T175341: Review and fix PDU settings for syslog/ntp/email servers.

Thanks @ayounsi ! Looks good to me, some things I found:

Fri, Oct 6, 8:11 AM · DC-Ops, Operations
fgiunchedi added a comment to T177484: tools-mail queue length alert from prometheus cron.

@chasemp I bet that's a side effect of T166561: Rollout prometheus-node-exporter 0.14 in labs, is it persisting or has been transient during package upgrades?

Fri, Oct 6, 7:48 AM · cloud-services-team (Kanban), Toolforge, User-bd808

Thu, Oct 5

fgiunchedi closed T145659: Port application-specific metrics from ganglia to prometheus as Resolved.
Thu, Oct 5, 3:57 PM · Patch-For-Review, Prometheus-metrics-monitoring, Operations
fgiunchedi added a comment to T145659: Port application-specific metrics from ganglia to prometheus.

Resolving as the work will be completed in T177196 by porting the missing Diamond collectors.

Thu, Oct 5, 3:57 PM · Patch-For-Review, Prometheus-metrics-monitoring, Operations
fgiunchedi closed T148637: Port redis statistics from ganglia to prometheus as Resolved.

Will do as part of T177196

Thu, Oct 5, 3:56 PM · Patch-For-Review, Prometheus-metrics-monitoring, Operations
fgiunchedi closed T148637: Port redis statistics from ganglia to prometheus, a subtask of T145659: Port application-specific metrics from ganglia to prometheus, as Resolved.
Thu, Oct 5, 3:56 PM · Patch-For-Review, Prometheus-metrics-monitoring, Operations
fgiunchedi updated the task description for T145659: Port application-specific metrics from ganglia to prometheus.
Thu, Oct 5, 3:56 PM · Patch-For-Review, Prometheus-metrics-monitoring, Operations
fgiunchedi added a comment to T175952: Split ChangeProp metrics by wiki.

(apologies about the delay, I completely missed this!)

Thu, Oct 5, 3:49 PM · Analytics, MediaWiki-JobQueue, Services (designing), ChangeProp, EventBus
fgiunchedi closed T166561: Rollout prometheus-node-exporter 0.14 in labs as Resolved.

All done! I've ran the upgrade with cumin, using the command below (see also
https://wikitech.wikimedia.org/wiki/Cumin#Upgrade_Debian_packages)

Thu, Oct 5, 3:33 PM · User-fgiunchedi, Toolforge, Cloud-Services, Cloud-VPS
fgiunchedi claimed T166561: Rollout prometheus-node-exporter 0.14 in labs.

I'll take care of this since we'll need some new collectors from node-exporter 0.14 as part of T177196

Thu, Oct 5, 12:21 PM · User-fgiunchedi, Toolforge, Cloud-Services, Cloud-VPS
fgiunchedi moved T166561: Rollout prometheus-node-exporter 0.14 in labs from Backlog to Doing on the User-fgiunchedi board.
Thu, Oct 5, 12:21 PM · User-fgiunchedi, Toolforge, Cloud-Services, Cloud-VPS
fgiunchedi updated the task description for T177196: Port non-deprecated Diamond collectors to Prometheus.
Thu, Oct 5, 10:34 AM · Patch-For-Review, User-fgiunchedi, Goal, Operations

Wed, Oct 4

fgiunchedi updated the task description for T177197: Export Prometheus-compatible JVM metrics from JVMs in production.
Wed, Oct 4, 10:59 AM · User-fgiunchedi, Goal, Operations
fgiunchedi updated the task description for T177196: Port non-deprecated Diamond collectors to Prometheus.
Wed, Oct 4, 10:18 AM · Patch-For-Review, User-fgiunchedi, Goal, Operations
fgiunchedi added a project to T177199: Add Prometheus client support for varnish/statsd metrics daemons: User-fgiunchedi.
Wed, Oct 4, 8:11 AM · Traffic, User-fgiunchedi, Goal, Operations
fgiunchedi added a project to T177197: Export Prometheus-compatible JVM metrics from JVMs in production: User-fgiunchedi.
Wed, Oct 4, 8:11 AM · User-fgiunchedi, Goal, Operations
fgiunchedi added a project to T177196: Port non-deprecated Diamond collectors to Prometheus: User-fgiunchedi.
Wed, Oct 4, 8:11 AM · Patch-For-Review, User-fgiunchedi, Goal, Operations
fgiunchedi added a project to T177195: Reduce technical debt in metrics monitoring: User-fgiunchedi.
Wed, Oct 4, 8:11 AM · User-fgiunchedi, Technical-Debt, Goal, Operations

Tue, Oct 3

fgiunchedi closed T175980: Upgrade grafana to 4.5.2 as Resolved.

All done!

Tue, Oct 3, 1:56 PM · Graphite, User-fgiunchedi, monitoring, Operations
fgiunchedi added a comment to T169969: Regularly purge old ores graphite metrics.

@awight I shared the list with Amir on T174542, happy to share it with you too (4MB gz file, 500k lines). It is in your home on tin (files generated on Aug 30th).

Tue, Oct 3, 1:13 PM · Scoring-platform-team (Current), ORES, User-fgiunchedi, Operations, Graphite

Mon, Oct 2

fgiunchedi created T177225: Uninstall ganglia from the fleet.
Mon, Oct 2, 3:38 PM · Patch-For-Review, Operations, monitoring
fgiunchedi added a project to T177078: Decide on casing convention for JMX metrics in Prometheus: monitoring.
Mon, Oct 2, 3:23 PM · Patch-For-Review, monitoring, User-Elukey, Analytics-Kanban, Analytics-Cluster
fgiunchedi updated the task description for T136312: Encrypt syslog traffic.
Mon, Oct 2, 1:35 PM · Patch-For-Review, monitoring, User-fgiunchedi, Operations
fgiunchedi renamed T175980: Upgrade grafana to 4.5.2 from Upgrade grafana to 4.5 to Upgrade grafana to 4.5.2.
Mon, Oct 2, 10:06 AM · Graphite, User-fgiunchedi, monitoring, Operations
fgiunchedi updated subscribers of T177078: Decide on casing convention for JMX metrics in Prometheus.

Thanks @elukey and @Ottomata ! That would indeed be helpful to have and more readable too.

Mon, Oct 2, 9:52 AM · Patch-For-Review, monitoring, User-Elukey, Analytics-Kanban, Analytics-Cluster
fgiunchedi created T177199: Add Prometheus client support for varnish/statsd metrics daemons.
Mon, Oct 2, 9:43 AM · Traffic, User-fgiunchedi, Goal, Operations
fgiunchedi created T177197: Export Prometheus-compatible JVM metrics from JVMs in production.
Mon, Oct 2, 9:39 AM · User-fgiunchedi, Goal, Operations
fgiunchedi created T177196: Port non-deprecated Diamond collectors to Prometheus.
Mon, Oct 2, 9:37 AM · Patch-For-Review, User-fgiunchedi, Goal, Operations
fgiunchedi created T177195: Reduce technical debt in metrics monitoring.
Mon, Oct 2, 9:34 AM · User-fgiunchedi, Technical-Debt, Goal, Operations
fgiunchedi added a comment to T173056: Import Wiki Loves Monuments photos from Flickr to Commons.

@fgiunchedi a quick note that Multichill and I did some assessments of the number of photos we can transfer from Flickr to Commons as part of this year's WLM and at this point, we're talking about couple of hundred photos at most. Correct me if I'm wrong, but this doesn't require very special attention from your team.

FYI: we're planning to start testing some transfers on Saturday and Sunday, mostly late afternoon/evening UTC. Again, these will be in the order of couple of hundred, not more.

Mon, Oct 2, 8:32 AM · Operations, Wiki-Loves-Monuments (2017)

Fri, Sep 29

fgiunchedi updated subscribers of T171339: Enable 3D on test.wikipedia.org and test2.wikipedia.org.

@fgiunchedi do you know how we can get 3d2png deployed on production servers? I believe it's been deployed to beta with scap but I don't even know how that was set up.

Not sure who deployed it, but it's there on eqiad and codfw thumbor servers now.

Fri, Sep 29, 9:59 AM · Patch-For-Review, Multimedia-Team-Working-Board, 3D, Multimedia

Tue, Sep 26

fgiunchedi accepted D793: Read EXIF orientation with pyexiv2.
Tue, Sep 26, 1:53 PM
fgiunchedi moved T175980: Upgrade grafana to 4.5.2 from Backlog to Doing on the User-fgiunchedi board.
Tue, Sep 26, 1:23 PM · Graphite, User-fgiunchedi, monitoring, Operations
fgiunchedi added a project to T175980: Upgrade grafana to 4.5.2: User-fgiunchedi.
Tue, Sep 26, 1:23 PM · Graphite, User-fgiunchedi, monitoring, Operations
fgiunchedi added a comment to T175922: Use Prometheus for Kafka JMX metrics instead of jmxtrans.

Yeah the idea is to have dedicated Prometheus instances roughly per-team, in this case "analytics" to collect e.g. hadoop, kafka, etc metrics in it. When there are useful aggregated metrics we can collect them in the global prometheus instance too.

Tue, Sep 26, 1:16 PM · monitoring, User-Elukey, Patch-For-Review, Analytics-Kanban, Analytics-Cluster
fgiunchedi closed T173731: Reduce swift frontend conntrack usage as Resolved.

We're now explicitly excluding statsite traffic and swift clients running on the proxy that talk to backend swift:

Tue, Sep 26, 9:07 AM · Patch-For-Review, User-fgiunchedi, Operations, media-storage

Mon, Sep 25

fgiunchedi added a comment to T175922: Use Prometheus for Kafka JMX metrics instead of jmxtrans.
# elukey@kafka-jumbo1001:~$ curl http://10.64.0.175:7800/metrics -s | grep -i jumbo

[..]
kafka_network_requestmetrics_requestqueuetimems{cluster="jumbo",request="Heartbeat",} 0.0
kafka_network_requestmetrics_requestqueuetimems{cluster="jumbo",request="ApiVersions",} 0.0
Mon, Sep 25, 9:12 AM · monitoring, User-Elukey, Patch-For-Review, Analytics-Kanban, Analytics-Cluster
fgiunchedi added a comment to T175922: Use Prometheus for Kafka JMX metrics instead of jmxtrans.

HM, why are we making an 'analytics' prometheus instance for this? kafka-jumbo is not in the Analytics VLAN, nor is it dedicated for Analytics purposes.

The new analytics instance should be related to all the new metrics that will come with the next quarter migration to prometheus, but it does make sense to not include kafka metrics on it. Either we use the regular operations namespace or maybe we can come up with a new instance only for kafka (like we probably do with Cassandra?).

@fgiunchedi what do you think?

Mon, Sep 25, 8:58 AM · monitoring, User-Elukey, Patch-For-Review, Analytics-Kanban, Analytics-Cluster
fgiunchedi added a comment to T176472: New package builder host.

IIRC we opened T130759 because slow IO had indeed cause some minor suffering on our part. If we can avoid migrating back to SATA disks easily I think we should. There's one more option on the table btw. A ganeti VM. We currently have the CPUs, the space, the IOPS and the memory in eqiad to support this and the spikey nature of package building fits rather ok with the virtualization idea.

Mon, Sep 25, 8:49 AM · hardware-requests, Operations

Wed, Sep 20

fgiunchedi closed T135723: Restarts of ganglia-monitor are unreliable, a subtask of T135991: Automated service restarts for common low-level system services, as Declined.
Wed, Sep 20, 10:12 AM · Patch-For-Review, Operations
fgiunchedi closed T135723: Restarts of ganglia-monitor are unreliable as Declined.

Ganglia is indeed going away

Wed, Sep 20, 10:12 AM · Operations
fgiunchedi added a comment to T176293: fundraising postfix metrics into prometheus.

FWIW we'll need to do sth similar in production too, likely by parsing the logs and extracting metrics with https://github.com/google/mtail and as mentioned in T147923: Extract metrics from logs

Wed, Sep 20, 10:06 AM · monitoring, fundraising-tech-ops
fgiunchedi added a comment to T152562: Port fundraising stats off Ganglia.

@Jgreen that's awesome news! I think we can finally shut down ganglia for good !

Wed, Sep 20, 10:01 AM · Fundraising-Backlog, Operations, fundraising-tech-ops
fgiunchedi accepted D789: Replace ImageMagick -auto-orient with manual logic.

LGTM!

Wed, Sep 20, 9:29 AM
Dzahn awarded T152562: Port fundraising stats off Ganglia a Love token.
Wed, Sep 20, 1:45 AM · Fundraising-Backlog, Operations, fundraising-tech-ops

Tue, Sep 19

fgiunchedi added inline comments to D789: Replace ImageMagick -auto-orient with manual logic.
Tue, Sep 19, 1:38 PM

Mon, Sep 18

fgiunchedi added a comment to T175636: prometheus -> grafana stats for per-numa-node meminfo.

@BBlack your patch to add meminfo_numa seems to be working! Anything left to do ?

Mon, Sep 18, 3:33 PM · Patch-For-Review, monitoring, Operations, Traffic
fgiunchedi added a comment to T147923: Extract metrics from logs.

Update: with latest upstream git of mtail things seem stable so far.

Mon, Sep 18, 3:31 PM · User-fgiunchedi, Patch-For-Review, monitoring, Operations

Sep 17 2017

Liuxinyu970226 awarded T175803: Text eqiad varnish 503 spikes a The World Burns token.
Sep 17 2017, 11:24 AM · Patch-For-Review, Traffic, Operations
APerson awarded T175803: Text eqiad varnish 503 spikes a The World Burns token.
Sep 17 2017, 5:47 AM · Patch-For-Review, Traffic, Operations

Sep 16 2017

fgiunchedi added a comment to T171490: mendelevium (otrs) running out of inodes.

The growth of used inodes since a few hours was pretty steep, I compressed and removed the older otrs versions:

Sep 16 2017, 10:09 PM · OTRS, Operations
Thibaut120094 awarded T175803: Text eqiad varnish 503 spikes a The World Burns token.
Sep 16 2017, 4:55 PM · Patch-For-Review, Traffic, Operations

Sep 15 2017

fgiunchedi created T175980: Upgrade grafana to 4.5.2.
Sep 15 2017, 8:50 AM · Graphite, User-fgiunchedi, monitoring, Operations
fgiunchedi added a comment to T150734: Make Thumbor logs available in ELK.

I poked at this some more this week but go nowhere either in beta or production, to recap:

Sep 15 2017, 8:48 AM · Patch-For-Review, User-fgiunchedi, Performance-Team, Thumbor
fgiunchedi added a comment to T171772: Prometheus metrics storage for RESTBase dev environment.

A strawman for disabling/uninstalling cassandra-metrics-collector is here (not working). Turns out, doing it properly is a little bit more disruptive than I imagined, the existing code assumes we want cmc everywhere, and configures it via cassandra::metrics in the profile shared by all Cassandra clusters. The approach in r/378100 is to set jmx_exporter_enabled for a cluster, and then use it to disable cmc. I'm not a fan of this approach though; I'd hoped to use Graphite and Prometheus side-by-side in the dev cluster, at least while we build out the new dashboards.

I wonder if the easiest thing wouldn't be to just teach cassandra-metrics-collector how to pause collection (presence of a file, an arg, env var, etc). Presumably once we've tested the exporter, and have some dashboards in place, we'll convert the rest of the production clusters to Prometheus, so we really just need something in the meantime.

Sep 15 2017, 8:41 AM · Patch-For-Review, Services (doing), Cassandra
fgiunchedi added a comment to T175850: Spike: Enumerate remaining unported stats.

One way would be to generate grafana dashboards' JSON from python and a list of metrics, namely with sth like grafanalib as outlined in T171482: Programmatic generation of grafana dashboards

Sep 15 2017, 8:36 AM · Spike, Fundraising-Backlog, Operations, fundraising-tech-ops
fgiunchedi added a comment to T175738: Long term storage for frack prometheus data.

Sounds awesome!

Sep 15 2017, 8:33 AM · Operations, fundraising-tech-ops
fgiunchedi created T175979: Puppet checks for invalid class names.
Sep 15 2017, 8:29 AM · puppet-compiler, Puppet

Sep 14 2017

fgiunchedi added a comment to T173374: Deleting file on Commons "Error deleting file: An unknown error occurred in storage backend "local-multiwrite".".

@Nick could you try again to delete both files? thanks!

Sep 14 2017, 12:58 PM · User-fgiunchedi, Operations, media-storage

Sep 13 2017

fgiunchedi moved T173374: Deleting file on Commons "Error deleting file: An unknown error occurred in storage backend "local-multiwrite"." from Backlog to Radar on the User-fgiunchedi board.
Sep 13 2017, 1:15 PM · User-fgiunchedi, Operations, media-storage
fgiunchedi added projects to T175803: Text eqiad varnish 503 spikes: Operations, Traffic.
Sep 13 2017, 9:18 AM · Patch-For-Review, Traffic, Operations
fgiunchedi created T175803: Text eqiad varnish 503 spikes.
Sep 13 2017, 9:18 AM · Patch-For-Review, Traffic, Operations
fgiunchedi created T175798: Port non-deprecated Diamond collectors to Prometheus.
Sep 13 2017, 8:13 AM · monitoring, Operations

Sep 12 2017

fgiunchedi committed rWWSC9d16bab30376: scap: fix checks.yaml syntax (authored by fgiunchedi).
scap: fix checks.yaml syntax
Sep 12 2017, 4:37 PM
fgiunchedi moved T175689: Remove X-Content-Dimensions for multipage originals from Backlog to Radar on the User-fgiunchedi board.
Sep 12 2017, 2:04 PM · MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), User-fgiunchedi, Operations, Performance-Team, Thumbor
fgiunchedi created T175689: Remove X-Content-Dimensions for multipage originals.
Sep 12 2017, 1:45 PM · MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), User-fgiunchedi, Operations, Performance-Team, Thumbor
fgiunchedi added a comment to T173374: Deleting file on Commons "Error deleting file: An unknown error occurred in storage backend "local-multiwrite".".

I've downloaded the file Literature_II,_Harutyun_Surkhatian.djvu to check for corruption just in case @Nick though it might be valid and just pathological per-page dimensions.

https://people.wikimedia.org/~filippo/Literature_II_tom%252C_Harutyun_Surkhatian.djvu

It looks to be a corrupt file - tried a couple of djvu viewers and both throw a fit and crash when trying to open this file.

Sep 12 2017, 1:38 PM · User-fgiunchedi, Operations, media-storage
fgiunchedi added a comment to T173374: Deleting file on Commons "Error deleting file: An unknown error occurred in storage backend "local-multiwrite".".

We've abandoned X-Content-Dimensions, so I think we need to look at how we can clean it up. Is there a cheap way to find swift objects that have it set, or are we condemned to look at all originals?

Sep 12 2017, 1:37 PM · User-fgiunchedi, Operations, media-storage
fgiunchedi added a comment to T173374: Deleting file on Commons "Error deleting file: An unknown error occurred in storage backend "local-multiwrite".".

I've downloaded the file Literature_II,_Harutyun_Surkhatian.djvu to check for corruption just in case @Nick though it might be valid and just pathological per-page dimensions.

Sep 12 2017, 11:18 AM · User-fgiunchedi, Operations, media-storage
fgiunchedi added a comment to T169249: /usr/local/bin/xenon-generate-svgs and flamegraph.pl cronspam.

@Gilles yeah looks like the spam does recur every now and then

Sep 12 2017, 11:10 AM · Patch-For-Review, Performance-Team, Operations