Joe (Giuseppe Lavagetto)
Spy

Projects (20)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 5:57 AM (146 w, 3 d)
Availability
Available
LDAP User
Giuseppe Lavagetto
MediaWiki User
Unknown

Recent Activity

Today

Joe awarded T169605: Default to ext4 instead of ext3 a Like token.
Mon, Jul 24, 4:35 PM · Patch-For-Review, Operations
Joe closed T170886: ocg1001 is broken as Resolved.
Mon, Jul 24, 2:43 PM · User-Joe, Reading-Web-Backlog (Tracking), ops-eqiad, OCG-General, Operations

Fri, Jul 21

Joe added a comment to T170886: ocg1001 is broken.

I've re-put this server in the rotation for the load-balancer.

Fri, Jul 21, 2:56 PM · User-Joe, Reading-Web-Backlog (Tracking), ops-eqiad, OCG-General, Operations
Joe closed T150434: Build calico as Resolved.
Fri, Jul 21, 2:55 PM · Prod-Kubernetes (Experiment), User-Joe, Operations
Joe closed T166328: mw2140.codfw.wmnet unreponsive, cannot be powercycled with serial console as Resolved.
Fri, Jul 21, 2:54 PM · ops-codfw, Operations
Joe reassigned T168613: Broken disk on mw1228 from Joe to RobH.
Fri, Jul 21, 2:52 PM · Operations, ops-eqiad
Joe added a comment to T168613: Broken disk on mw1228.

I agree with @MoritzMuehlenhoff - in general assume server assignment is correct unless otherwise stated by us.

Fri, Jul 21, 2:51 PM · Operations, ops-eqiad
Joe closed T162780: ocg1003 partitions are severely misconfigured as Resolved.
Fri, Jul 21, 2:34 PM · Operations
Joe moved T170886: ocg1001 is broken from Backlog to Doing on the User-Joe board.
Fri, Jul 21, 10:17 AM · User-Joe, Reading-Web-Backlog (Tracking), ops-eqiad, OCG-General, Operations
Joe claimed T170886: ocg1001 is broken.
Fri, Jul 21, 10:15 AM · User-Joe, Reading-Web-Backlog (Tracking), ops-eqiad, OCG-General, Operations
Joe moved T166888: CI for operations/puppet is taking too long from Backlog to Doing on the User-Joe board.
Fri, Jul 21, 8:39 AM · User-Joe, Release-Engineering-Team (Kanban), Patch-For-Review, Operations, Continuous-Integration-Infrastructure

Thu, Jul 20

Joe moved T162735: Hyperthreading disabled on restbase2002.codfw.wmnet & restbase1015.codfw.wmnet from Backlog to Blocking others on the User-Joe board.
Thu, Jul 20, 3:01 PM · User-fgiunchedi, User-Joe, Cassandra, Services (blocked), Operations
Joe added a project to T162735: Hyperthreading disabled on restbase2002.codfw.wmnet & restbase1015.codfw.wmnet: User-Joe.
Thu, Jul 20, 3:00 PM · User-fgiunchedi, User-Joe, Cassandra, Services (blocked), Operations
Joe added a project to T166888: CI for operations/puppet is taking too long: User-Joe.
Thu, Jul 20, 8:20 AM · User-Joe, Release-Engineering-Team (Kanban), Patch-For-Review, Operations, Continuous-Integration-Infrastructure
Joe closed T114638: Add the puppet CA to the certification authorities trusted by our systems, on demand as Resolved.
Thu, Jul 20, 5:57 AM · Operations, Patch-For-Review, Puppet
Joe added a comment to T171075: recommendation_api module breaking beta labs puppet.

Please apply the same role/profile we use in production to beta too.

Thu, Jul 20, 5:53 AM · Recommendation-API, Beta-Cluster-Infrastructure

Wed, Jul 19

Joe closed T102109: Add python-etcd the ability to discover the etcd hosts using SRV records as Resolved.
Wed, Jul 19, 3:51 PM · Operations, discovery-system
Joe accepted D600: Create a wrapper around conftool for our pooling/depooling needs.

As far as the conftool part is concerned, this seems correct.

Wed, Jul 19, 2:13 PM · Release-Engineering-Team
Joe added a comment to T170886: ocg1001 is broken.

I ran the script and it worked fine, but we also need to add a redis slave on ocg1003, as right now we lost the replica of the redis master which is on ocg1002, according to puppet.

Wed, Jul 19, 2:02 PM · User-Joe, Reading-Web-Backlog (Tracking), ops-eqiad, OCG-General, Operations
Joe added a comment to T170886: ocg1001 is broken.

Please note that if no one ran the script that removes entries corresponding to ocg1001 from the cache, we're still serving failing requests as mediawiki will try to connect to ocg1001 directly.

Wed, Jul 19, 1:53 PM · User-Joe, Reading-Web-Backlog (Tracking), ops-eqiad, OCG-General, Operations

Tue, Jul 18

Joe closed T169546: Add results of compilation with the future parser to the puppet compiler as Resolved.
Tue, Jul 18, 9:27 AM · Patch-For-Review, User-Joe, puppet-compiler, Puppet, Operations
Joe closed T169546: Add results of compilation with the future parser to the puppet compiler, a subtask of T169485: Add support for directory environments to our puppet classes, production puppetmaster, as Resolved.
Tue, Jul 18, 9:27 AM · Patch-For-Review, User-Joe, Puppet, Operations
Joe closed T169546: Add results of compilation with the future parser to the puppet compiler, a subtask of T169548: Prepare for Puppet 4, as Resolved.
Tue, Jul 18, 9:27 AM · User-Joe, Puppet, Operations

Thu, Jul 13

Eevans awarded T137303: Cassandra Puppet cleanups a Cookie token.
Thu, Jul 13, 9:05 PM · Services (later), Cassandra
Joe added a comment to D600: Create a wrapper around conftool for our pooling/depooling needs.

I think you need to fix the logic inside the set_pooled_state function as per my comment. Apart from that, it seems correct.

Thu, Jul 13, 2:41 PM · Release-Engineering-Team
Joe added a comment to T165760: Deploy Recommendation API as a service.

I finished deploying the service, but I strongly urge you to have it respond to its root url with something different than 404.

Thu, Jul 13, 7:04 AM · Patch-For-Review, User-mobrovac, Services (doing), User-Joe, Recommendation-API
Joe closed T165760: Deploy Recommendation API as a service as Resolved.
Thu, Jul 13, 7:03 AM · Patch-For-Review, User-mobrovac, Services (doing), User-Joe, Recommendation-API
Joe closed T165760: Deploy Recommendation API as a service, a subtask of T148129: Productization of Recommendation API, as Resolved.
Thu, Jul 13, 7:03 AM · Patch-For-Review, Reading Epics (AI Based Reading Recommendations), Recommendation-API, Epic

Wed, Jul 12

Joe added a comment to T170394: Cumin: add multi-query support.

I'd add another PRO to the second option (which I prefer for backwards compatibility): it allows us to be more concise.

Wed, Jul 12, 11:03 AM · Patch-For-Review, Operations-Software-Development

Tue, Jul 11

Joe moved T165760: Deploy Recommendation API as a service from Blocking others to Doing on the User-Joe board.
Tue, Jul 11, 2:34 PM · Patch-For-Review, User-mobrovac, Services (doing), User-Joe, Recommendation-API

Mon, Jul 10

Joe added a project to T170111: Implement a pod networking policy approach: User-Joe.
Mon, Jul 10, 4:38 PM · Services (watching), User-Joe, Kubernetes, Operations, Goal
Joe added a comment to T156850: set up SSL cert monitoring for benefactorevents.wm.o.

This thing is alerting since 4 days as it's apparently using the default azure ssl cert.

Mon, Jul 10, 6:58 AM · Patch-For-Review, Operations, fundraising-tech-ops
Joe added a comment to T168881: Rename mw2148 / mw2149 / mw2259 / mw2260 to thumbor200[1234].

I did deactivate those nodes, and me and filippo already added a "puppet node deactivate" to wmf-reimage, so I guess it's all right now.

Mon, Jul 10, 6:42 AM · ops-codfw, User-fgiunchedi, Operations, Performance-Team, Thumbor
Joe closed T169485: Add support for directory environments to our puppet classes, production puppetmaster as Resolved.
Mon, Jul 10, 6:35 AM · Patch-For-Review, User-Joe, Puppet, Operations
Joe closed T169485: Add support for directory environments to our puppet classes, production puppetmaster, a subtask of T169548: Prepare for Puppet 4, as Resolved.
Mon, Jul 10, 6:35 AM · User-Joe, Puppet, Operations
Joe added a comment to T103886: Translation cache exhaustion caused by changes to PHP code in file scope.

@Krinkle sure, we can enable reusing TC in beta for now and test if the feature is stable and working as expected.

Mon, Jul 10, 6:19 AM · Release-Engineering-Team (Watching / External), Patch-For-Review, Operations, Performance-Team, Deployment-Systems, HHVM
Joe added a comment to T169972: Delete "servers" metrics in graphite older than 60d.

Aren't we collecting all server metrics via prometheus? If that's the case, shouldn't we just drop the diamond collector for those metrics?

Mon, Jul 10, 6:10 AM · Patch-For-Review, User-fgiunchedi, Operations, Graphite

Fri, Jul 7

Joe added a comment to T148129: Productization of Recommendation API.

@schana I will get into the details early next week - I just have one additional question: who will be the consumer of the service exactly? The user browser? mediawiki via an extension? another software?

Fri, Jul 7, 4:07 PM · Patch-For-Review, Reading Epics (AI Based Reading Recommendations), Recommendation-API, Epic

Thu, Jul 6

Joe renamed T169893: pybal should reset the etcdindex it's looking at after losing a connection from pybal should reset the etcdindex it's looking at after losin a connection to pybal should reset the etcdindex it's looking at after losing a connection.
Thu, Jul 6, 2:40 PM · Patch-For-Review, Pybal, Traffic, Operations
Joe created T169893: pybal should reset the etcdindex it's looking at after losing a connection.
Thu, Jul 6, 2:35 PM · Patch-For-Review, Pybal, Traffic, Operations
Joe added a comment to T169765: pybal should automatically reconnect to etcd.

One option to support reconnections and srv records and everything is to use the (blocking) python-etcd library via defer.deferToThread as etcd-mirror does.

Thu, Jul 6, 7:31 AM · Pybal, Operations, Traffic

Tue, Jul 4

Joe moved T169546: Add results of compilation with the future parser to the puppet compiler from Backlog to Doing on the User-Joe board.
Tue, Jul 4, 7:07 AM · Patch-For-Review, User-Joe, puppet-compiler, Puppet, Operations

Mon, Jul 3

Joe updated the task description for T169548: Prepare for Puppet 4.
Mon, Jul 3, 3:59 PM · User-Joe, Puppet, Operations
Joe created T169548: Prepare for Puppet 4.
Mon, Jul 3, 3:57 PM · User-Joe, Puppet, Operations
Joe updated the task description for T169546: Add results of compilation with the future parser to the puppet compiler.
Mon, Jul 3, 3:54 PM · Patch-For-Review, User-Joe, puppet-compiler, Puppet, Operations
Joe added a parent task for T169546: Add results of compilation with the future parser to the puppet compiler: T169485: Add support for directory environments to our puppet classes, production puppetmaster.
Mon, Jul 3, 3:53 PM · Patch-For-Review, User-Joe, puppet-compiler, Puppet, Operations
Joe added a subtask for T169485: Add support for directory environments to our puppet classes, production puppetmaster: T169546: Add results of compilation with the future parser to the puppet compiler.
Mon, Jul 3, 3:53 PM · Patch-For-Review, User-Joe, Puppet, Operations
Joe claimed T169546: Add results of compilation with the future parser to the puppet compiler.
Mon, Jul 3, 3:53 PM · Patch-For-Review, User-Joe, puppet-compiler, Puppet, Operations
Joe created T169546: Add results of compilation with the future parser to the puppet compiler.
Mon, Jul 3, 3:52 PM · Patch-For-Review, User-Joe, puppet-compiler, Puppet, Operations
Joe closed T169493: Test restarting puppetmaster workers for every code deploy as Declined.
Mon, Jul 3, 9:51 AM · User-Joe, Puppet, Operations
Joe closed T169493: Test restarting puppetmaster workers for every code deploy, a subtask of T169485: Add support for directory environments to our puppet classes, production puppetmaster, as Declined.
Mon, Jul 3, 9:51 AM · Patch-For-Review, User-Joe, Puppet, Operations
Joe added a comment to T169493: Test restarting puppetmaster workers for every code deploy.

So all my attempts at restarting one or multiple instances of the puppetmaster backends luckily didn't cause any puppet errors, but it takes up to 4 minutes for a restart to take effect. I'm not sure this is acceptable with our current practices.

Mon, Jul 3, 9:51 AM · User-Joe, Puppet, Operations
Joe added a comment to T169493: Test restarting puppetmaster workers for every code deploy.

The passenger docs about this are pretty clear: in the floss version of passenger, a restart is blocking - that is passenger will wait for all currently spawned workers to stop, and at least a new one to be spawned, before serving a new request. Those requests are going to be queued, but if we have some expensive catalog being built, it means ~ 30 seconds of blocking.

Mon, Jul 3, 8:51 AM · User-Joe, Puppet, Operations
Joe moved T169493: Test restarting puppetmaster workers for every code deploy from Backlog to Doing on the User-Joe board.
Mon, Jul 3, 8:42 AM · User-Joe, Puppet, Operations
Joe created T169493: Test restarting puppetmaster workers for every code deploy.
Mon, Jul 3, 8:42 AM · User-Joe, Puppet, Operations
Joe added a comment to T169485: Add support for directory environments to our puppet classes, production puppetmaster.

First hurdle: puppetlabs advises to set the variable environment_timeout to unlimited and to restart the puppetmaster at every code deploy, for performance reasons.

Mon, Jul 3, 8:39 AM · Patch-For-Review, User-Joe, Puppet, Operations
Joe moved T165760: Deploy Recommendation API as a service from Backlog to Blocking others on the User-Joe board.
Mon, Jul 3, 7:55 AM · Patch-For-Review, User-mobrovac, Services (doing), User-Joe, Recommendation-API
Joe moved T168271: Decommission mw1170-mw1179 from Backlog to Blocked on others on the User-Joe board.
Mon, Jul 3, 7:55 AM · Patch-For-Review, hardware-requests, User-Joe, ops-eqiad, Operations
Joe moved T165519: rack and setup mw1307-1348 from Blocking others to Blocked on others on the User-Joe board.
Mon, Jul 3, 7:55 AM · User-Elukey, User-Joe, Operations, ops-eqiad
Joe moved T156924: Allow integration of data from etcd into the MediaWiki configuration from Doing to Blocked on others on the User-Joe board.
Mon, Jul 3, 7:54 AM · MW-1.30-release-notes (WMF-deploy-2017-07-25_(1.30.0-wmf.11)), Patch-For-Review, Availability (Multiple-active-datacenters), MediaWiki-Platform-Team, Services (watching), Performance-Team, discovery-system, User-Joe, User-mobrovac, Operations
Joe moved T169485: Add support for directory environments to our puppet classes, production puppetmaster from Backlog to Doing on the User-Joe board.
Mon, Jul 3, 7:53 AM · Patch-For-Review, User-Joe, Puppet, Operations
Joe added a project to T169485: Add support for directory environments to our puppet classes, production puppetmaster: User-Joe.
Mon, Jul 3, 7:53 AM · Patch-For-Review, User-Joe, Puppet, Operations
Joe created T169485: Add support for directory environments to our puppet classes, production puppetmaster.
Mon, Jul 3, 7:47 AM · Patch-For-Review, User-Joe, Puppet, Operations

Fri, Jun 30

Joe closed T162042: Prepare and maintain base container images as Resolved.
Fri, Jun 30, 3:00 PM · Release-Engineering-Team (Watching / External), Patch-For-Review, User-mobrovac, Services (designing), Kubernetes, Goal, Operations
Joe closed T162042: Prepare and maintain base container images, a subtask of T162039: Prepare to service applications from kubernetes, as Resolved.
Fri, Jun 30, 3:00 PM · Kubernetes, Goal, Operations
Joe renamed T169299: Mobileapps swagger spec is broken (no pronounciation for `page/mobile-sections-lead` endpoints) from Mobileapps swagger spec is broken (no pronounciation for Altrincham) to Mobileapps swagger spec is broken (no pronounciation for `page/mobile-sections-lead` endpoints).
Fri, Jun 30, 7:49 AM · Services (done), Parsoid, Reading-Infrastructure-Team-Backlog (Kanban), Mobile-Content-Service, Operations
Joe created T169299: Mobileapps swagger spec is broken (no pronounciation for `page/mobile-sections-lead` endpoints).
Fri, Jun 30, 7:11 AM · Services (done), Parsoid, Reading-Infrastructure-Team-Backlog (Kanban), Mobile-Content-Service, Operations

Wed, Jun 28

Joe added a comment to D600: Create a wrapper around conftool for our pooling/depooling needs.
In D600#13637, @demon wrote:

@Joe Any chance you could have a look at this again? We definitely need conftool in pip or this won't ever build.

Wed, Jun 28, 6:19 AM · Release-Engineering-Team

Jun 21 2017

Joe added a comment to T97192: HHVM request timeouts not working; support lowering the API request timeout per request.

@Joe, has this been fixed with 3.18?

Jun 21 2017, 7:37 AM · User-Joe, Operations, Services (watching), Wikimedia-Incident, Incident-20150423-Commons, HHVM, RESTBase, Parsoid, Availability, Performance, MediaWiki-API
Joe added a comment to T165105: Wiley requests for DOI and some other publishers don't work in production.

Hey sorry I was on vacation last week.

Jun 21 2017, 7:29 AM · Services (watching), Operations, User-mobrovac, Citoid, VisualEditor
Joe added a project to T168462: codfw row A switch upgrade: User-Joe.
Jun 21 2017, 7:14 AM · Patch-For-Review, User-Joe, Traffic, Operations, netops
Joe updated the task description for T168462: codfw row A switch upgrade.
Jun 21 2017, 7:13 AM · Patch-For-Review, User-Joe, Traffic, Operations, netops

Jun 20 2017

Joe added a project to T163337: Job queue corruption after codfw switch over (Queue growth, duplicate runs): User-Joe.
Jun 20 2017, 1:36 PM · User-Joe, Performance-Team, User-Elukey, Wikimedia-Incident, MediaWiki-JobQueue, Scoring-platform-team, User-Ladsgroup, ORES, codfw-rollout
Joe placed T168360: Gerrit constantly throws HTTP 500 error when reviewing patches (due to "Too many open files") up for grabs.
Jun 20 2017, 8:35 AM · Release-Engineering-Team (Kanban), Patch-For-Review, Operations, Gerrit
Joe added a comment to T168360: Gerrit constantly throws HTTP 500 error when reviewing patches (due to "Too many open files").

So what I did:

Jun 20 2017, 8:35 AM · Release-Engineering-Team (Kanban), Patch-For-Review, Operations, Gerrit
Joe added a comment to T168360: Gerrit constantly throws HTTP 500 error when reviewing patches (due to "Too many open files").
Caused by: java.nio.file.FileSystemException: .../_1ut5m.nvm: Too many open files
Jun 20 2017, 8:30 AM · Release-Engineering-Team (Kanban), Patch-For-Review, Operations, Gerrit
Joe claimed T168360: Gerrit constantly throws HTTP 500 error when reviewing patches (due to "Too many open files").
Jun 20 2017, 8:29 AM · Release-Engineering-Team (Kanban), Patch-For-Review, Operations, Gerrit

Jun 19 2017

Joe moved T167130: Decom mw1170-mw1179, and replace them with new systems. from Doing to Blocked on others on the User-Joe board.
Jun 19 2017, 12:56 PM · User-Joe, ops-eqiad, Operations
Joe added a comment to T168271: Decommission mw1170-mw1179.

@Cmjohnson please proceed to decom/derack these servers and rack new ones in their place.

Jun 19 2017, 12:50 PM · Patch-For-Review, hardware-requests, User-Joe, ops-eqiad, Operations
Joe updated the task description for T168271: Decommission mw1170-mw1179.
Jun 19 2017, 12:49 PM · Patch-For-Review, hardware-requests, User-Joe, ops-eqiad, Operations
Joe updated the task description for T168271: Decommission mw1170-mw1179.
Jun 19 2017, 12:47 PM · Patch-For-Review, hardware-requests, User-Joe, ops-eqiad, Operations
Joe created T168271: Decommission mw1170-mw1179.
Jun 19 2017, 11:00 AM · Patch-For-Review, hardware-requests, User-Joe, ops-eqiad, Operations
Joe moved T167130: Decom mw1170-mw1179, and replace them with new systems. from Backlog to Doing on the User-Joe board.
Jun 19 2017, 10:27 AM · User-Joe, ops-eqiad, Operations

Jun 9 2017

Joe added a comment to T167048: Services need external monitoring.

both maps and restbase are now monitored at the load-balancers of the SSL terminators in all datacenters. Resolving.

Jun 9 2017, 3:48 PM · Patch-For-Review, User-Joe, Operations, Services (next), User-mobrovac, monitoring

Jun 8 2017

Joe added a comment to T167048: Services need external monitoring.

@faidon at first I was thinking of implementing the checks on the LVS host (in the end, the puppettization is mostly the same), but I thought the nrpe checks on caches to be better just because it would monitor each cache host and not round-robin every host in a pool. It might also help seeing problems on individual caches.

Jun 8 2017, 6:07 AM · Patch-For-Review, User-Joe, Operations, Services (next), User-mobrovac, monitoring

Jun 7 2017

Joe added a comment to T167048: Services need external monitoring.

In order to do that, I want to do a local nrpe check on the cache edge servers, calling the SSL terminator, so that we cover as many logical layers as possible. Sadly, there is a bug in service-checker that I need to fix before this can go live, but apart from that it should be pretty straightforward.

Jun 7 2017, 3:04 PM · Patch-For-Review, User-Joe, Operations, Services (next), User-mobrovac, monitoring
Joe added a project to T167269: Make security updates of docker images manageable: Operations.
Jun 7 2017, 10:17 AM · Operations, User-Joe, Prod-Kubernetes (Experiment), Kubernetes
Joe added a parent task for T167269: Make security updates of docker images manageable: T162043: Define a process to keep images up-to-date on similar standards as the rest of production.
Jun 7 2017, 10:17 AM · Operations, User-Joe, Prod-Kubernetes (Experiment), Kubernetes
Joe added a subtask for T162043: Define a process to keep images up-to-date on similar standards as the rest of production: T167269: Make security updates of docker images manageable.
Jun 7 2017, 10:17 AM · Kubernetes, Goal, Operations
Joe added a subtask for T162042: Prepare and maintain base container images: T167269: Make security updates of docker images manageable.
Jun 7 2017, 10:08 AM · Release-Engineering-Team (Watching / External), Patch-For-Review, User-mobrovac, Services (designing), Kubernetes, Goal, Operations
Joe added a parent task for T167269: Make security updates of docker images manageable: T162042: Prepare and maintain base container images.
Jun 7 2017, 10:08 AM · Operations, User-Joe, Prod-Kubernetes (Experiment), Kubernetes
Joe updated the task description for T167269: Make security updates of docker images manageable.
Jun 7 2017, 9:57 AM · Operations, User-Joe, Prod-Kubernetes (Experiment), Kubernetes
Joe updated the task description for T167269: Make security updates of docker images manageable.
Jun 7 2017, 9:50 AM · Operations, User-Joe, Prod-Kubernetes (Experiment), Kubernetes
Joe created T167269: Make security updates of docker images manageable.
Jun 7 2017, 9:45 AM · Operations, User-Joe, Prod-Kubernetes (Experiment), Kubernetes

Jun 6 2017

Joe added a comment to T167048: Services need external monitoring.

I would start monitoring restbase on text-lb and maps on text-upload.

Jun 6 2017, 2:26 PM · Patch-For-Review, User-Joe, Operations, Services (next), User-mobrovac, monitoring
Joe moved T167048: Services need external monitoring from Backlog to Doing on the User-Joe board.
Jun 6 2017, 2:24 PM · Patch-For-Review, User-Joe, Operations, Services (next), User-mobrovac, monitoring
Joe added a comment to T144169: Flake8 for python files without extension in puppet repo.

After some discussion in https://gerrit.wikimedia.org/r/#/c/323559/ I've changed my vote to "automatically discover python files".

There is a mixture of conventions in python scripts inside puppet.git: underscore vs dash, extension vs no extension but having pep/flake tests is important even without explicit opt-in, the tests can be non-voting until the violations are fixed.
Re: naming, I think an obvious convention would be to keep the name in puppet the same as what's installed on the filesystem, especially for things expected to be in PATH.

Jun 6 2017, 8:22 AM · Patch-For-Review, Continuous-Integration-Config, Operations, Operations-Software-Development

Jun 5 2017

Joe added a project to T166806: Server side upload for Yann: Operations.
Jun 5 2017, 9:04 AM · Patch-For-Review, Operations, media-storage, Commons, Wikimedia-Site-requests

Jun 1 2017

Joe added a comment to T166345: wmf/1.30.0-wmf.2 performance issue for Wikipedias.

(Continued investigation using the data we did capture in SAL and Logstash)

  • Alerts for HHVM response times taking 10+ seconds only happened in the inactive data center (codfw).
Jun 1 2017, 6:28 AM · Performance-Team-notice, Patch-For-Review, Services (watching), Operations, Performance-Team, Wikimedia-General-or-Unknown, MediaWiki-General-or-Unknown

May 31 2017

Joe added a project to T165760: Deploy Recommendation API as a service: User-Joe.
May 31 2017, 2:59 PM · Patch-For-Review, User-mobrovac, Services (doing), User-Joe, Recommendation-API