fgiunchedi (Filippo Giunchedi)
Awesome

Projects (17)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 8:06 AM (142 w, 2 h)
Availability
Available
IRC Nick
godog
LDAP User
Filippo Giunchedi
MediaWiki User
Filippo Giunchedi

Recent Activity

Today

fgiunchedi created T168705: nutcracker test config in puppet doesn't work.
Fri, Jun 23, 8:20 AM · Patch-For-Review, Operations

Yesterday

fgiunchedi added a comment to T162796: Delete non-used and/or non-requested thumbnail sizes periodically.

thanks @JAllemandou ! I've converted the tables to use parquet and dropped the old plaintext tables

Thu, Jun 22, 12:43 PM · Patch-For-Review, User-fgiunchedi, Operations
fgiunchedi added a comment to T166699: Evaluate performance of 3d2png on beta cluster.

I'm assuming you'll be storing the result in swift beta, should be ok in terms of space (i.e. around 400MB + thumbs)

Thu, Jun 22, 10:58 AM · Multimedia, 3d
fgiunchedi moved T166561: Rollout prometheus-node-exporter 0.14 in labs from Backlog to Doing on the User-fgiunchedi board.
Thu, Jun 22, 9:56 AM · User-fgiunchedi, Tool-Labs, Labs, Labs-Infrastructure
fgiunchedi moved T160986: Decommission ms-fe100[1-4] from Blocked to Radar on the User-fgiunchedi board.
Thu, Jun 22, 9:55 AM · ops-eqiad, Patch-For-Review, User-fgiunchedi, hardware-requests, Operations
fgiunchedi moved T166489: Decommission ms-be1001 - ms-be1012 from Blocked to Radar on the User-fgiunchedi board.
Thu, Jun 22, 9:55 AM · ops-eqiad, hardware-requests, User-fgiunchedi, Operations
fgiunchedi moved T162785: Decommission ms-be2001 - ms-be2012 from Blocked to Radar on the User-fgiunchedi board.
Thu, Jun 22, 9:55 AM · Patch-For-Review, ops-codfw, hardware-requests, Operations, User-fgiunchedi
fgiunchedi moved T167801: Deploy thumbor in codfw from Backlog to Doing on the User-fgiunchedi board.
Thu, Jun 22, 9:25 AM · Patch-For-Review, User-fgiunchedi, Operations, Performance-Team, Thumbor
fgiunchedi moved T150206: ms-be1016 controller cache failure from Blocked to Radar on the User-fgiunchedi board.
Thu, Jun 22, 9:25 AM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi moved T168297: Rename mw1236 / mw1237 to thumbor1003 / thumbor1004 from Blocked to Radar on the User-fgiunchedi board.
Thu, Jun 22, 9:25 AM · ops-eqiad, Operations, User-fgiunchedi, Performance-Team, Thumbor
fgiunchedi moved T168297: Rename mw1236 / mw1237 to thumbor1003 / thumbor1004 from Backlog to Blocked on the User-fgiunchedi board.
Thu, Jun 22, 9:25 AM · ops-eqiad, Operations, User-fgiunchedi, Performance-Team, Thumbor
fgiunchedi closed T162609: Swift version and distro upgrade as Resolved.

This is resolved, we're running swift 2.10 and some machines in codfw/eqiad running stretch too.

Thu, Jun 22, 9:24 AM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
fgiunchedi closed T162609: Swift version and distro upgrade, a subtask of T162792: Reduce Swift technical debt, as Resolved.
Thu, Jun 22, 9:24 AM · User-fgiunchedi, Operations
fgiunchedi added a comment to T168509: Upgrade of prometheus-node-exporter complains with: chown: invalid group: ‘prometheus:prometheus’.

prometheus-node-exporter does create the prometheus group, IIRC it fails because of what you mentioned, i.e. prometheus user exists in labs/cloud

Thu, Jun 22, 8:26 AM · Beta-Cluster-reproducible, Prometheus-metrics-monitoring

Wed, Jun 21

fgiunchedi updated the task description for T162609: Swift version and distro upgrade.
Wed, Jun 21, 4:29 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
fgiunchedi updated the task description for T162609: Swift version and distro upgrade.
Wed, Jun 21, 4:23 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
fgiunchedi added a comment to T162609: Swift version and distro upgrade.

Current status, swift in esams hasn't been touched since it is slated for decom anyway. ms-be[12] from 01 to 12 are decom'd from swift. Remaining machines are either running stretch or jessie, with swift 2.10

Wed, Jun 21, 4:23 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
fgiunchedi removed a project from T1075: Something puts many different metrics into graphite, allocating a lot of disk space: Patch-For-Review.
Wed, Jun 21, 2:50 PM · Operations, Graphite
fgiunchedi placed T160644: Eventstreams graphite disk usage up for grabs.
Wed, Jun 21, 2:21 PM · Analytics
fgiunchedi reopened T160644: Eventstreams graphite disk usage as "Open".

Reopening, beginning at around 6/6 eventstreams has been creating a lot of metrics consuming ~20% of graphite disk space in 8 days and it is now at around 400G

Wed, Jun 21, 2:21 PM · Analytics
fgiunchedi renamed T1075: Something puts many different metrics into graphite, allocating a lot of disk space from something (reqstats?) puts many different metrics into graphite, allocating a lot of disk space to Something puts many different metrics into graphite, allocating a lot of disk space.
Wed, Jun 21, 2:17 PM · Operations, Graphite
fgiunchedi moved T166489: Decommission ms-be1001 - ms-be1012 from Doing to Blocked on the User-fgiunchedi board.
Wed, Jun 21, 9:57 AM · ops-eqiad, hardware-requests, User-fgiunchedi, Operations
fgiunchedi updated subscribers of T166489: Decommission ms-be1001 - ms-be1012 .

@Dzahn machines are marked as spares now and good to be decom'd /cc @RobH

Wed, Jun 21, 9:57 AM · ops-eqiad, hardware-requests, User-fgiunchedi, Operations
fgiunchedi placed T166489: Decommission ms-be1001 - ms-be1012 up for grabs.
Wed, Jun 21, 9:56 AM · ops-eqiad, hardware-requests, User-fgiunchedi, Operations
fgiunchedi renamed T168297: Rename mw1236 / mw1237 to thumbor1003 / thumbor1004 from Reimage eqiad imagescalers to be used with thumbor to Rename mw1236 / mw1237 to thumbor1003 / thumbor1004.
Wed, Jun 21, 8:41 AM · ops-eqiad, Operations, User-fgiunchedi, Performance-Team, Thumbor
fgiunchedi closed T163777: Debug HP raid cache disabled errors on ms-be1019/20/21 as Resolved.

All done, 1019 BBU was swapped yesterday by @Cmjohnson

Wed, Jun 21, 8:33 AM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi updated the task description for T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.
Wed, Jun 21, 8:32 AM · User-fgiunchedi, ops-eqiad, Operations

Tue, Jun 20

Dzahn awarded T166489: Decommission ms-be1001 - ms-be1012 a Like token.
Tue, Jun 20, 6:47 PM · ops-eqiad, hardware-requests, User-fgiunchedi, Operations
fgiunchedi closed T168378: IPMI console not working on ms-be1014 / ms-be1015, a subtask of T150160: Remote IPMI doesn't work for ~2% of the fleet, as Resolved.
Tue, Jun 20, 2:44 PM · Patch-For-Review, Operations
fgiunchedi closed T168378: IPMI console not working on ms-be1014 / ms-be1015 as Resolved.

With help from @Cmjohnson we've restored the console on these two boxes by draining flea power

Tue, Jun 20, 2:44 PM · ops-eqiad, Operations
fgiunchedi created T168378: IPMI console not working on ms-be1014 / ms-be1015.
Tue, Jun 20, 10:26 AM · ops-eqiad, Operations
fgiunchedi edited projects for T168374: Unknown error occurred in storage backend "local-swift-eqiad" when moving a file over a deleted target, added: media-storage; removed Wikimedia-Media-storage.
Tue, Jun 20, 9:55 AM · media-storage

Mon, Jun 19

fgiunchedi added a project to T168297: Rename mw1236 / mw1237 to thumbor1003 / thumbor1004: User-fgiunchedi.
Mon, Jun 19, 2:25 PM · ops-eqiad, Operations, User-fgiunchedi, Performance-Team, Thumbor
fgiunchedi created T168297: Rename mw1236 / mw1237 to thumbor1003 / thumbor1004.
Mon, Jun 19, 1:51 PM · ops-eqiad, Operations, User-fgiunchedi, Performance-Team, Thumbor
fgiunchedi added a comment to T167400: Disable serving unpatrolled new files to Wikipedia Zero users.
Mon, Jun 19, 9:52 AM · Traffic, Operations, media-storage, Commons, Multimedia, Zero
fgiunchedi added a comment to T167877: Slow performance with image (PDF?) thumbnailer..

I catched a 500 on that file with thumbnails, looks like the imagescalers are having an hard time with some of those thumbnails:

Mon, Jun 19, 9:22 AM · media-storage, Wikisource
fgiunchedi added a comment to T166489: Decommission ms-be1001 - ms-be1012 .

@Dzahn almost, I'm running the last swift ring rebalance today. ETA is two/three days, I'll update/reassign this task once the machines are good to decom!

Mon, Jun 19, 9:14 AM · ops-eqiad, hardware-requests, User-fgiunchedi, Operations
fgiunchedi added a comment to T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.

@Cmjohnson today sounds good, ping me here or on IRC

Mon, Jun 19, 9:08 AM · User-fgiunchedi, ops-eqiad, Operations

Wed, Jun 14

fgiunchedi added a comment to T160990: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU.

There's potentially other VMs across labs/cloud doing the same expensive resolving localhost, it seems to me it'd be more useful to understand the root cause instead

Wed, Jun 14, 12:57 PM · Release-Engineering-Team (Kanban), Patch-For-Review, media-storage, Beta-Cluster-Infrastructure
fgiunchedi closed T161222: Remove graphite metrics under zuul.pipeline.check-voter., a subtask of T161205: Remove check-voter pipeline from Zuul, as Resolved.
Wed, Jun 14, 7:47 AM · Release-Engineering-Team (Kanban), Patch-For-Review, Continuous-Integration-Config
fgiunchedi closed T161222: Remove graphite metrics under zuul.pipeline.check-voter. as Resolved.

{{done}}

Wed, Jun 14, 7:47 AM · Graphite

Tue, Jun 13

fgiunchedi created T167801: Deploy thumbor in codfw.
Tue, Jun 13, 1:56 PM · Patch-For-Review, User-fgiunchedi, Operations, Performance-Team, Thumbor

Mon, Jun 12

fgiunchedi added a comment to T167689: Add RIPE atlas data to Prometheus.

@ayounsi awesome!

Mon, Jun 12, 5:02 PM · Prometheus-metrics-monitoring
fgiunchedi added a comment to T166967: Load webrequest raw data into druid so ops can use it for troubleshooting.

thanks @JAllemandou ! I give it a quick try and it looks very interesting, how often is the data loaded from webrequest? IOW how much lag we should be expecting?

Mon, Jun 12, 3:29 PM · Patch-For-Review, Analytics-Kanban
fgiunchedi created T167639: Some parsercache keys yield error when tried to save into memcache.
Mon, Jun 12, 9:05 AM · MW-1.30-release-notes (WMF-deploy-2017-06-06_(1.30.0-wmf.4)), MediaWiki-Parser
fgiunchedi moved T167034: Limit maximum x-content-dimension size to avoid hitting nginx limits from Backlog to Radar on the User-fgiunchedi board.
Mon, Jun 12, 8:41 AM · MW-1.30-release-notes, User-fgiunchedi, Patch-For-Review, Operations, Performance-Team, Thumbor
fgiunchedi added a project to T167034: Limit maximum x-content-dimension size to avoid hitting nginx limits: User-fgiunchedi.
Mon, Jun 12, 8:41 AM · MW-1.30-release-notes, User-fgiunchedi, Patch-For-Review, Operations, Performance-Team, Thumbor

Fri, Jun 9

fgiunchedi created U14 wikiscrape.
Fri, Jun 9, 10:44 AM
fgiunchedi created U12 wikiscrape.
Fri, Jun 9, 10:40 AM
fgiunchedi added a comment to T158429: Switch to predictable network interface names?.

I've finished converting ms-be stretch systems to predictable network interfaces, no problems observed so far. For reference the commands:

Fri, Jun 9, 9:33 AM · Patch-For-Review, Operations
fgiunchedi accepted D681: Add support for WebP.
Fri, Jun 9, 8:49 AM
fgiunchedi updated the task description for T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.
Fri, Jun 9, 8:41 AM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi added a comment to T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.

We were getting duplicate alerts from ms-be1019 due to its hp raid check going unknown (I think). I've disabled the handler for hp raid on ms-be1019 though it'll need to be reenabled once this is fixed.

Fri, Jun 9, 8:41 AM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi merged task T167426: Degraded RAID on ms-be1019 into T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.
Fri, Jun 9, 8:40 AM · ops-eqiad, Operations
fgiunchedi merged tasks T167434: Degraded RAID on ms-be1019, T167426: Degraded RAID on ms-be1019 into T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.
Fri, Jun 9, 8:40 AM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi merged task T167434: Degraded RAID on ms-be1019 into T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.
Fri, Jun 9, 8:40 AM · ops-eqiad, Operations

Thu, Jun 8

fgiunchedi added a comment to D668: Read X-Content-Dimensions header and apply limit.

Just nits really, LGTM

Thu, Jun 8, 3:37 PM
fgiunchedi moved T166181: rack/setup/install restbase-dev100[456] from Backlog to Radar on the User-fgiunchedi board.
Thu, Jun 8, 3:09 PM · User-fgiunchedi, DC-Ops, ops-eqiad, Services, Operations
fgiunchedi added a project to T166181: rack/setup/install restbase-dev100[456]: User-fgiunchedi.
Thu, Jun 8, 3:09 PM · User-fgiunchedi, DC-Ops, ops-eqiad, Services, Operations
fgiunchedi added a comment to T167333: Increase email log retention period for the main email relays.

FWIW if we also want to store mail logs off-host a simple solution would be to syslog exim logs too, syslog hosts already have 90d retention in place.

Thu, Jun 8, 2:43 PM · Patch-For-Review, Mail, Operations
fgiunchedi merged task T167398: Degraded RAID on ms-be1019 into T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.
Thu, Jun 8, 2:39 PM · ops-eqiad, Operations
fgiunchedi merged task T167393: Degraded RAID on ms-be1019 into T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.
Thu, Jun 8, 2:39 PM · ops-eqiad, Operations
fgiunchedi merged tasks T167393: Degraded RAID on ms-be1019, T167398: Degraded RAID on ms-be1019 into T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.
Thu, Jun 8, 2:39 PM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi closed T167286: Package latest version of Thumbor and deploy it as Resolved.

I've checked the diff and uploaded thumbor 6.3.2+git20170607-1 internally to jessie-wikimedia

Thu, Jun 8, 2:31 PM · Operations, Performance-Team, Thumbor
fgiunchedi closed T167286: Package latest version of Thumbor and deploy it, a subtask of T150741: Thumbor should reject thumbnail requests that are the same size as the original or bigger, as Resolved.
Thu, Jun 8, 2:31 PM · MW-1.30-release-notes (WMF-deploy-2017-06-06_(1.30.0-wmf.4)), Patch-For-Review, Operations, Performance-Team, Thumbor
fgiunchedi closed T167287: Backport python-schedule and add it to jessie-wikimedia as Resolved.

I've uploaded schedule 0.3.2-1~bpo8+1 to Debian jessie-backports with its maintainer approval.

Thu, Jun 8, 2:29 PM · Operations, Performance-Team, Thumbor
fgiunchedi closed T167287: Backport python-schedule and add it to jessie-wikimedia, a subtask of T167286: Package latest version of Thumbor and deploy it, as Resolved.
Thu, Jun 8, 2:29 PM · Operations, Performance-Team, Thumbor

Wed, Jun 7

fgiunchedi updated the task description for T160616: Enable HTTPS for swift clients.
Wed, Jun 7, 3:55 PM · MW-1.30-release-notes, Patch-For-Review, HTTPS, Traffic, User-fgiunchedi, Operations
fgiunchedi added a comment to T162609: Swift version and distro upgrade.

I've upgraded all ms-fe2* to swift 2.10, the trusty -> stretch conversion of ms-be2* is ongoing. Regardless of the latter I think we could test some user traffic in swift codfw next week and see how that goes

Wed, Jun 7, 3:04 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
fgiunchedi added a comment to T167264: Degraded RAID on ms-be1016.
09:43  <volans> I need to check later why we got 2 tasks though
09:44  <godog> my fault, the first is manual because I thought the disk was already failed on the controller but it wasn't
09:44  <godog> then I marked the disk failed manually on the controller too
Wed, Jun 7, 8:46 AM · ops-eqiad, Operations
fgiunchedi merged T167268: Degraded RAID on ms-be1016 into T167264: Degraded RAID on ms-be1016.
Wed, Jun 7, 8:46 AM · ops-eqiad, Operations
fgiunchedi merged task T167268: Degraded RAID on ms-be1016 into T167264: Degraded RAID on ms-be1016.
Wed, Jun 7, 8:46 AM · media-storage, ops-eqiad, Operations
fgiunchedi added a comment to T167245: prometheus-node-exporter - invalid group: ‘prometheus:prometheus'.

Indeed, the problem there I think is that prometheus user exists in labs but not the group, was node-exporter working otherwise?

Wed, Jun 7, 8:37 AM · Operations, Prometheus-metrics-monitoring
fgiunchedi accepted D678: Close memcache at the end of the request.
Wed, Jun 7, 8:30 AM
fgiunchedi assigned T167264: Degraded RAID on ms-be1016 to Cmjohnson.
Wed, Jun 7, 8:29 AM · ops-eqiad, Operations
fgiunchedi updated the task description for T167264: Degraded RAID on ms-be1016.
Wed, Jun 7, 8:28 AM · ops-eqiad, Operations
fgiunchedi closed T127762: Update Debian Package for Scap3 as Resolved.
Wed, Jun 7, 8:13 AM · Patch-For-Review, Deployment-Systems, Scap

Tue, Jun 6

fgiunchedi added a comment to T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.

ms-be1020 had its bbu swapped, error cleared:

Tue, Jun 6, 1:31 PM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi updated subscribers of T167118: Degraded RAID on ms-be2001.

@Papaul this host is scheduled for decom and has otherwise no production data, don't bother replacing the disk

Tue, Jun 6, 1:13 PM · media-storage, Operations, ops-codfw
fgiunchedi added a comment to T144169: Flake8 for python files without extension in puppet repo.

Re: naming, I think an obvious convention would be to keep the name in puppet the same as what's installed on the filesystem, especially for things expected to be in PATH.

Actually what most people do out of here is exactly the opposite: have the file on the puppet repo have an explicit extension to help with CI checkers (not just flake8, but rubocop, shellcheck, etc), and I second that choice.

It would make all of this so much simpler and faster, too.

Tue, Jun 6, 10:21 AM · Patch-For-Review, Continuous-Integration-Config, Operations, Operations-Software-Development
fgiunchedi created T167095: Support chunked transfer encoding for mw server side uploads to swift.
Tue, Jun 6, 9:57 AM · MediaWiki-Uploading, Multimedia
fgiunchedi added a comment to T166806: Server side upload for Yann.

Thanks everyone for your help in debugging this, @Yann did uploading other files with e.g. v2c worked eventually? I see @Dereckson server side uploads worked

Tue, Jun 6, 9:54 AM · Patch-For-Review, Operations, media-storage, Commons, Wikimedia-Site-requests
fgiunchedi added a project to T166561: Rollout prometheus-node-exporter 0.14 in labs: User-fgiunchedi.
Tue, Jun 6, 8:33 AM · User-fgiunchedi, Tool-Labs, Labs, Labs-Infrastructure

Mon, Jun 5

fgiunchedi renamed T167035: stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist from acct monthly cron will spam when /var/log/wtmp.1 doesn't exist to stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist.
Mon, Jun 5, 3:54 PM · Operations
fgiunchedi created T167035: stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist.
Mon, Jun 5, 3:54 PM · Operations
fgiunchedi reopened T167035: stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist, a subtask of T132324: Tracking and Reducing cron-spam from root@ , as Open.
Mon, Jun 5, 3:54 PM · Patch-For-Review, Operations
fgiunchedi created T167034: Limit maximum x-content-dimension size to avoid hitting nginx limits.
Mon, Jun 5, 3:37 PM · MW-1.30-release-notes, User-fgiunchedi, Patch-For-Review, Operations, Performance-Team, Thumbor
fgiunchedi reopened T167034: Limit maximum x-content-dimension size to avoid hitting nginx limits, a subtask of T150741: Thumbor should reject thumbnail requests that are the same size as the original or bigger, as Open.
Mon, Jun 5, 3:36 PM · MW-1.30-release-notes (WMF-deploy-2017-06-06_(1.30.0-wmf.4)), Patch-For-Review, Operations, Performance-Team, Thumbor
fgiunchedi merged task T166837: Degraded RAID on ms-be1020 into T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.
Mon, Jun 5, 3:13 PM · ops-eqiad, Operations
fgiunchedi merged T166837: Degraded RAID on ms-be1020 into T163777: Debug HP raid cache disabled errors on ms-be1019/20/21.
Mon, Jun 5, 3:13 PM · User-fgiunchedi, ops-eqiad, Operations
fgiunchedi added a comment to T166806: Server side upload for Yann.

@Dereckson could you try the uploads one more time? I've disabled spooling of files to disk in nginx

Mon, Jun 5, 2:08 PM · Patch-For-Review, Operations, media-storage, Commons, Wikimedia-Site-requests
fgiunchedi added a comment to T166806: Server side upload for Yann.

So the problem is the tmpfs on /var/lib/nginx being 1G, IOW the maximum client body that nginx will spool there.

Mon, Jun 5, 1:46 PM · Patch-For-Review, Operations, media-storage, Commons, Wikimedia-Site-requests
fgiunchedi added a comment to T166806: Server side upload for Yann.

@Dereckson I've enabled debug on nginx for connections coming from terbium, can you try the uploads again? thanks!

Mon, Jun 5, 11:25 AM · Patch-For-Review, Operations, media-storage, Commons, Wikimedia-Site-requests
fgiunchedi added a comment to T166806: Server side upload for Yann.

Focusing only on one file for now, found a 500 from swift in FileOperation, now looking on the swift side

Mon, Jun 5, 10:42 AM · Patch-For-Review, Operations, media-storage, Commons, Wikimedia-Site-requests
fgiunchedi added a comment to T166806: Server side upload for Yann.

@Dereckson I saw your importImages run has finished on terbium (?) how'd it go this time?

Mon, Jun 5, 10:08 AM · Patch-For-Review, Operations, media-storage, Commons, Wikimedia-Site-requests

Fri, Jun 2

fgiunchedi added a comment to T162609: Swift version and distro upgrade.

proxy-server 2.10 seems to be basically working on ms-fe2005. I've asked upstream about an increase in proxy-server.errors metrics that seem related to ratelimit here: https://bugs.launchpad.net/swift/+bug/1695273

Fri, Jun 2, 2:03 PM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
fgiunchedi awarded P5534 Ferm fixes for stretch a Like token.
Fri, Jun 2, 12:59 PM
fgiunchedi updated the task description for T151648: Implement storage policies for swift.
Fri, Jun 2, 10:21 AM · Patch-For-Review, User-fgiunchedi, media-storage, Operations
fgiunchedi added a comment to T166561: Rollout prometheus-node-exporter 0.14 in labs.

thanks! I think the version in aptly has been removed so we should be set for tools too, what's the best way I can run a command on all labs + tools ?

Fri, Jun 2, 8:33 AM · User-fgiunchedi, Tool-Labs, Labs, Labs-Infrastructure
fgiunchedi closed T166843: prometheus-node-exporter v. Trusty as Resolved.

@Andrew indeed the uploaded version was lacking the upstart script, and falling back to jessie's init.d script won't work as you discovered, I've fixed it now in a new internal version by shipping the upstart script. Upgrading prometheus-node-exporter should fix it!

Fri, Jun 2, 8:30 AM