Joe (Giuseppe Lavagetto)
Spy

Projects (23)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 5:57 AM (176 w, 2 d)
Availability
Available
LDAP User
Giuseppe Lavagetto
MediaWiki User
Unknown

Recent Activity

Fri, Feb 16

Joe added a project to T178278: Performance test the service: User-Joe.
Fri, Feb 16, 3:50 PM · User-Joe, Performance, Readers-Web-Kanbanana-Board, Readers-Web-Backlog, Proton, Electron-PDFs
Joe added a comment to T97562: WANObjectCache relay daemon or mcrouter support.

I'm reopening this since the status of the FLOSS mcrouter project in the last year has been dire:

Fri, Feb 16, 7:03 AM · Availability (Multiple-active-datacenters), Analytics, User-mobrovac, EventBus, Services
Joe reopened T97562: WANObjectCache relay daemon or mcrouter support as "Open".
Fri, Feb 16, 6:53 AM · Availability (Multiple-active-datacenters), Analytics, User-mobrovac, EventBus, Services
Joe reopened T97562: WANObjectCache relay daemon or mcrouter support, a subtask of T88445: MediaWiki active/active datacenter investigation and work, as Open.
Fri, Feb 16, 6:53 AM · Availability (Multiple-active-datacenters), Performance-Team, Epic

Thu, Feb 15

Joe added projects to T187467: Decommission mw2017 and mw2099: Operations, ops-eqiad.
Thu, Feb 15, 4:54 PM · ops-codfw, Operations
Joe created T187468: Create 2 VMs in codfw for mwdebug20001 and 2002.
Thu, Feb 15, 4:53 PM · Operations
Joe created T187467: Decommission mw2017 and mw2099.
Thu, Feb 15, 4:52 PM · ops-codfw, Operations
Joe created T187466: Decommission mw1259-mw1260.
Thu, Feb 15, 4:50 PM · Operations, ops-eqiad
Joe added a comment to T184481: hardware request for tin replacement.

I initially suggested tin might make use of SSD, being the deployment server, but in fact it seems that given how scap works nowadays the only iops-intensive activity on those servers is the l10n-update, which might even go away in the future, and it's still not hitting a bottleneck in terms of disk throughput AFAICS:

Thu, Feb 15, 3:56 PM · hardware-requests, Operations
Joe added a comment to T186818: Cumin: add --limit to randomly select N hosts.

The proposed solution, given the generic nature of Cumin and that the only information it has are the hostnames, is to use the ClusterShell's NodeSet compact syntax to extract them in "groups".

Thu, Feb 15, 9:56 AM · Patch-For-Review, Operations-Software-Development

Mon, Feb 12

Joe added a comment to T108557: Review and deploy UrlShortener extension to Wikimedia wikis.

No, the outstanding TODO is having any group of people competently maintaining and monitoring the url shortener for abuse/security issues.

Mon, Feb 12, 10:41 AM · Availability (Multiple-active-datacenters), Patch-For-Review, MediaWiki-extensions-UrlShortener, Wikimedia-Extension-setup

Fri, Feb 9

Joe moved T162013: etcd cluster in codfw has raft consensus issues from Backlog to Doing on the User-Joe board.
Fri, Feb 9, 10:35 AM · Patch-For-Review, User-Joe, Operations

Thu, Feb 8

Joe added a comment to T184258: get a snapshot instance running in beta with stretch, php7.

During the SRE team meetings, we decided that a good approach for handling the php7 memcached issue would be to have a separate memcached pool for php7 mediawiki hosts. In accordance with that, I'm going to set up a memcached pool in deployment-prep that is only to be used by php7 mediawiki instances.

Thu, Feb 8, 7:51 AM · User-ArielGlenn, Dumps-Generation, MediaWiki-General-or-Unknown

Wed, Feb 7

debt awarded T186718: Add email addresses to techcom@wikimedia.org email alias a Like token.
Wed, Feb 7, 9:04 PM · Operations
Joe added a comment to T186718: Add email addresses to techcom@wikimedia.org email alias.

Both addressed added.

Wed, Feb 7, 4:21 PM · Operations
Joe closed T186718: Add email addresses to techcom@wikimedia.org email alias as Resolved.
Wed, Feb 7, 4:21 PM · Operations

Tue, Feb 6

Joe closed T175089: decommission mw1163 as Resolved.
Tue, Feb 6, 3:04 PM · hardware-requests, Operations
Joe updated subscribers of T175089: decommission mw1163 .

This server is out of production and hopefully de-racked since a long time as it was part of an older batch of servers. @Cmjohnson can confirm, but this should be decommissioned per T177387

Tue, Feb 6, 3:04 PM · hardware-requests, Operations
Joe added a comment to T186069: Icinga: page in case all MediaWiki are throwing 5xx.

We do have functional checks for all services but MediaWiki. there is an old ticket ( T136839) about enabling swagger specs for MediaWiki thus allowing us to use service-checker to perform functional checks.

Tue, Feb 6, 12:25 PM · Patch-For-Review, Wikimedia-Incident, Icinga, Operations, monitoring
Joe reopened T162013: etcd cluster in codfw has raft consensus issues as "Open".
Tue, Feb 6, 8:02 AM · Patch-For-Review, User-Joe, Operations

Sat, Feb 3

Joe created T186416: Allow selecting which images to build.
Sat, Feb 3, 3:05 PM · Operations, docker-pkg

Thu, Feb 1

Joe added a comment to T107561: MediaWiki support for Composer equivalent for JavaScript packages.

@Volans while in theory i agree with you, in practice... none of our current JS code EVER passed a sign check either. So if that's something we want to do, then we should be pushing npm, because requirements without infrastructure are just plain impossible in practicality.

Thu, Feb 1, 11:48 AM · Performance-Team, MediaWiki-ResourceLoader, Developer-Wishlist (2017), TechCom-RFC, Front-end-Standards-Group

Wed, Jan 31

Joe added a comment to T186098: Special:EditMassMessageList serves Error 500.

The error seems to be:

Fatal error: Class undefined: MediaWiki\MassMessage\MWNamespace in /srv/mediawiki/php-1.31.0-wmf.17/extensions/MassMessage/includes/SpecialEditMassMessageList.php on line 123
Wed, Jan 31, 11:35 AM · MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), User-MarcoAurelio, Wikimedia-log-errors, MassMessage

Tue, Jan 30

Joe added a comment to T185644: Switch phabricator from using apache to nginx.

So before we go to such a shotgun approach, I'd like to first:

Tue, Jan 30, 4:06 AM · Patch-For-Review, Operations, Phabricator

Thu, Jan 25

Joe added a comment to T185582: Wikipedia no longer accessible to those using some braille devices.
Thu, Jan 25, 6:43 PM · Operations, Traffic, Browser-Support-Internet-Explorer, Accessibility
Joe added a comment to T185582: Wikipedia no longer accessible to those using some braille devices.

@Cameron11598 I couldn't find on the product page for that device any information on the OS/browser it's using. I see there is a recent firmware update which is supposed to support facebook/youtube; did we ask the user if the device is fully updated?

Thu, Jan 25, 6:39 PM · Operations, Traffic, Browser-Support-Internet-Explorer, Accessibility
Joe added a comment to T185582: Wikipedia no longer accessible to those using some braille devices.

I can confirm that our current SSL settings do not work with IE8 on windows XP. This is due to inherent limitations in windows XP cryptography being dramatically outdated and making browsing inherently insecure for its users. Moreover, supporting it would expose all visitors to some attacks that have been solved in more modern cryptographic settings. I'll let someone from the Traffic team comment further on the issue, but the recommendation would be to either user Firefox, or move to a more updated (and currently supported) operating system.

Thu, Jan 25, 6:31 PM · Operations, Traffic, Browser-Support-Internet-Explorer, Accessibility

Wed, Jan 24

Joe added a comment to T185646: High database error rates on s2 and s3.

The biggest offender is one of the new jobrunners, so while I'd tend to think it has to do with connection overload, but I think this merits a better investigation for sure.

Wed, Jan 24, 5:55 PM · HHVM, DBA

Tue, Jan 23

Joe added a comment to T179696: Homepage for https://docker-registry.wikimedia.org.

Portus seems like an interesting approach to two problems: -authn+authz and UI for a docker registry

Tue, Jan 23, 10:32 PM · Operations, MediaWiki-Containers
Joe added a comment to T183313: Wikimedia Developer Summit 2018 Topic: Evolving the MediaWiki Architecture.

Also, Drupal (or Symfony etc) being on par with MediaWiki in terms of scaling is very much [citation needed]. nih.org is said to be the top ranking Drupal site, and a slightly outdated comScore report claims they 10M uniques a month. That's about 1% of what Wikimedia sites get, and then I'm sure the ratio of writes is a tiny fraction of even that (outside of social media, very few sites are seriously write-heavy).

Tue, Jan 23, 5:55 PM · MediaWiki-Platform-Team, User-Jdlrobson, Services (watching), User-mobrovac, User-Daniel, Wikimedia-Developer-Summit (2018)
Joe added a comment to T183313: Wikimedia Developer Summit 2018 Topic: Evolving the MediaWiki Architecture.

Again, I have no idea if it would actually be more effective to move to something other than MediaWiki. I am however, really discouraged at the overwhelming majority view which is completely dismissive. I think it takes a great deal of intellectual dishonesty to completely dismiss an idea based on the premise and that makes me sad. I can be completely wrong, and that's fine, but I think it's worth considering. I don't want to sacrifice the mission on the alter of MediaWiki.

Tue, Jan 23, 5:50 PM · MediaWiki-Platform-Team, User-Jdlrobson, Services (watching), User-mobrovac, User-Daniel, Wikimedia-Developer-Summit (2018)

Jan 19 2018

Joe moved T185004: Decommission mw1201-mw1220 from Backlog to Blocked on others on the User-Joe board.
Jan 19 2018, 10:33 AM · ops-eqiad, Patch-For-Review, User-Joe, Operations
Joe moved T185080: Prepare conftool for safely editing mediawiki-config values from Backlog to Doing on the User-Joe board.
Jan 19 2018, 10:33 AM · Patch-For-Review, User-Joe, discovery-system, MediaWiki-Configuration, Operations
Joe moved T166081: rack/setup/install conf1004-conf1006 from Blocked on others to Blocking others on the User-Joe board.
Jan 19 2018, 10:33 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, User-Joe, Operations
Joe added a project to T185080: Prepare conftool for safely editing mediawiki-config values: User-Joe.
Jan 19 2018, 10:32 AM · Patch-For-Review, User-Joe, discovery-system, MediaWiki-Configuration, Operations

Jan 17 2018

Joe updated the task description for T185084: Allow use of EtcdConfig to configure slave databases.
Jan 17 2018, 10:19 AM · DBA, discovery-system, MediaWiki-Configuration, Operations
Joe triaged T185084: Allow use of EtcdConfig to configure slave databases as Normal priority.
Jan 17 2018, 10:19 AM · DBA, discovery-system, MediaWiki-Configuration, Operations
Joe triaged T185080: Prepare conftool for safely editing mediawiki-config values as Normal priority.
Jan 17 2018, 10:01 AM · Patch-For-Review, User-Joe, discovery-system, MediaWiki-Configuration, Operations
Joe triaged T185078: Test EtcdConfig in different failure scenarios as Normal priority.
Jan 17 2018, 9:31 AM · discovery-system, MediaWiki-Configuration, Operations
Joe placed T185004: Decommission mw1201-mw1220 up for grabs.
Jan 17 2018, 8:40 AM · ops-eqiad, Patch-For-Review, User-Joe, Operations
Joe updated the task description for T185004: Decommission mw1201-mw1220.
Jan 17 2018, 8:40 AM · ops-eqiad, Patch-For-Review, User-Joe, Operations
Joe added a project to T185004: Decommission mw1201-mw1220: hardware-requests.
Jan 17 2018, 8:15 AM · ops-eqiad, Patch-For-Review, User-Joe, Operations

Jan 16 2018

Joe claimed T185004: Decommission mw1201-mw1220.
Jan 16 2018, 3:30 PM · ops-eqiad, Patch-For-Review, User-Joe, Operations
Joe triaged T185004: Decommission mw1201-mw1220 as Normal priority.
Jan 16 2018, 3:30 PM · ops-eqiad, Patch-For-Review, User-Joe, Operations
Joe moved T149617: Integrating MediaWiki (and other services) with dynamic configuration from Blocking others to Doing on the User-Joe board.
Jan 16 2018, 3:24 PM · Patch-For-Review, Performance-Team (Radar), Availability (Multiple-active-datacenters), Services (watching), discovery-system, User-Joe, User-mobrovac, Operations
Joe closed T165519: rack and setup mw1307-1348 , a subtask of T172459: eqiad row D switch upgrade, as Resolved.
Jan 16 2018, 1:31 PM · Patch-For-Review, Operations, netops, Traffic
Joe closed T165519: rack and setup mw1307-1348 as Resolved.
Jan 16 2018, 1:31 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Joe updated the task description for T165519: rack and setup mw1307-1348 .
Jan 16 2018, 1:27 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Joe moved T165519: rack and setup mw1307-1348 from Blocked on others to Doing on the User-Joe board.
Jan 16 2018, 1:26 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad

Jan 15 2018

Joe created T184909: labvirt1021-1022 spam the dhcp server with requests.
Jan 15 2018, 1:31 PM · Operations, cloud-services-team, Cloud-Services

Jan 12 2018

Joe added a comment to T184715: pybal's "can-depool" logic only takes downServers into account.

I don't think this is entirely correct.

Jan 12 2018, 10:03 AM · Patch-For-Review, Pybal, Traffic, Operations

Jan 8 2018

Joe renamed T182597: Use EtcdConfig in production to allow automation of a datacenter switch from [DRAFT] Use EtcdConfig in production to allow automation of a datacenter switch to Use EtcdConfig in production to allow automation of a datacenter switch.
Jan 8 2018, 2:37 PM · Patch-For-Review, discovery-system, MediaWiki-Configuration, Operations

Jan 4 2018

Joe added a comment to T184048: HHVM hangs on the API cluster.

@Imarlier no I think there isn't much we can do until we have a reproduction case. For now I'm focusing on mitigations for this issue as 1) we're not on the latest HHVM LTS anyways and it's likely this bug might be resolved (and new ones pop up), and we're not investing on the very long term on HHVM.

Jan 4 2018, 8:34 AM · Performance-Team (Radar), Patch-For-Review, HHVM, Operations

Jan 3 2018

Joe updated the task description for T183895: Decommission mw1180-1200.
Jan 3 2018, 3:03 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Joe lowered the priority of T183895: Decommission mw1180-1200 from High to Normal.
Jan 3 2018, 3:03 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Joe added a comment to T183895: Decommission mw1180-1200.

@Cmjohnson whenever you're able to complete the decommission of these systems, we will have room in row C for the remaining new appservers.

Jan 3 2018, 2:56 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Joe raised the priority of T183895: Decommission mw1180-1200 from Normal to High.
Jan 3 2018, 2:55 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Joe updated the task description for T183895: Decommission mw1180-1200.
Jan 3 2018, 2:55 PM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Joe closed T179721: puppet4: The following unknown setting(s) are being ignored: parser, a subtask of T177254: Upgrade to puppet 4 (4.8 or newer), as Resolved.
Jan 3 2018, 10:10 AM · cloud-services-team (FY2017-18), Puppet, User-Joe, Operations
Joe closed T179721: puppet4: The following unknown setting(s) are being ignored: parser as Resolved.
Jan 3 2018, 10:10 AM · Patch-For-Review, Puppet, User-Joe, Operations
Joe moved T179721: puppet4: The following unknown setting(s) are being ignored: parser from Backlog to Doing on the User-Joe board.
Jan 3 2018, 10:10 AM · Patch-For-Review, Puppet, User-Joe, Operations
Joe moved T183895: Decommission mw1180-1200 from Backlog to Doing on the User-Joe board.
Jan 3 2018, 10:08 AM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Joe triaged T184048: HHVM hangs on the API cluster as High priority.
Jan 3 2018, 8:24 AM · Performance-Team (Radar), Patch-For-Review, HHVM, Operations
Joe added a comment to T183889: 2017-12-30 save timing regression.

As I mentioned in the email thread last night, and I confirmed looking at the timings, this is related to a spike in CPU usage on the API cluster in eqiad:

Jan 3 2018, 7:49 AM · Wikimedia-General-or-Unknown, Regression, Performance-Team

Jan 2 2018

Joe updated the task description for T183895: Decommission mw1180-1200.
Jan 2 2018, 10:03 AM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad
Joe triaged T183895: Decommission mw1180-1200 as Normal priority.
Jan 2 2018, 9:36 AM · Patch-For-Review, User-Elukey, User-Joe, Operations, ops-eqiad

Dec 21 2017

Joe closed T177394: Experiment with a TLS proxy/router for pods, a subtask of T178325: Operations 2017-18 Q2 Program 6 umbrella task, as Resolved.
Dec 21 2017, 2:52 PM · Operations, Kubernetes
Joe closed T177394: Experiment with a TLS proxy/router for pods as Resolved.
Dec 21 2017, 2:52 PM · User-Joe, Prod-Kubernetes, Operations, Kubernetes
Joe added a comment to T177394: Experiment with a TLS proxy/router for pods.

Indeed, we have an implementation path, the dockerfiles for building the software, and some plans for the future. We can call this done.

Dec 21 2017, 2:52 PM · User-Joe, Prod-Kubernetes, Operations, Kubernetes
Joe added a comment to T177394: Experiment with a TLS proxy/router for pods.

There are few questions. Things like

  • Should we go with the TLS certs right from the beginning ? Maybe it makes sense to avoid them in the very first phase.
Dec 21 2017, 1:30 PM · User-Joe, Prod-Kubernetes, Operations, Kubernetes
Joe added a comment to T177394: Experiment with a TLS proxy/router for pods.

To recap my experiments:

Dec 21 2017, 9:15 AM · User-Joe, Prod-Kubernetes, Operations, Kubernetes

Dec 20 2017

Joe added a comment to T179640: mw1191 ipmi-sel cpu errors.

This might be not urgent, but almost two months before having a response on this ticket is quite a long time.

Dec 20 2017, 7:05 AM · Operations, ops-eqiad
Joe moved T177276: Unify production and CI docker image build process from Backlog to Blocked on others on the User-Joe board.
Dec 20 2017, 6:52 AM · Release-Engineering-Team (Kanban), Patch-For-Review, User-Joe, Operations, Continuous-Integration-Infrastructure (shipyard)
Joe moved T177276: Unify production and CI docker image build process from Blocked on others to Backlog on the User-Joe board.
Dec 20 2017, 6:52 AM · Release-Engineering-Team (Kanban), Patch-For-Review, User-Joe, Operations, Continuous-Integration-Infrastructure (shipyard)
Joe closed T173129: Prove helm as a potential k8s deployment tool, a subtask of T177397: Create scaffolding of services templates for deployment in production/staging, as Resolved.
Dec 20 2017, 6:52 AM · Patch-For-Review, Prod-Kubernetes, User-Joe, Operations, Kubernetes
Joe closed T173129: Prove helm as a potential k8s deployment tool as Resolved.
Dec 20 2017, 6:52 AM · User-Joe, Release-Engineering-Team (Next), Release Pipeline
Joe closed T177397: Create scaffolding of services templates for deployment in production/staging as Resolved.
Dec 20 2017, 6:51 AM · Patch-For-Review, Prod-Kubernetes, User-Joe, Operations, Kubernetes
Joe closed T177397: Create scaffolding of services templates for deployment in production/staging, a subtask of T178325: Operations 2017-18 Q2 Program 6 umbrella task, as Resolved.
Dec 20 2017, 6:51 AM · Operations, Kubernetes
Joe closed T181027: Create email alias for the TechCom as Resolved.
Dec 20 2017, 6:50 AM · TechCom, Operations
Joe added a comment to T177931: Decommission OCG from production.

Marked as resolved, I don't have much to do here.

Dec 20 2017, 6:50 AM · Patch-For-Review, Services (watching), OCG-General, Operations
Joe closed T177931: Decommission OCG from production as Resolved.
Dec 20 2017, 6:49 AM · Patch-For-Review, Services (watching), OCG-General, Operations
Joe closed T177931: Decommission OCG from production, a subtask of T150871: [EPIC] (Proposal) Replicate core OCG features and sunset OCG service, as Resolved.
Dec 20 2017, 6:49 AM · Readers-Web-Backlog (Tracking), Services (watching), OCG-General, User-mobrovac, Epic, Readers-Community-Engagement, Operations

Dec 12 2017

Joe added a comment to T169450: Redirect several wikis.

@MarcoAurelio in my tests the redirects all work now, can you confirm they're ok? If so, should we close this ticket or wait for someone to fix the mediawiki config as well?

Dec 12 2017, 1:15 PM · Operations, Wikimedia-Apache-configuration, Puppet, Wiki-Setup (Close), Wikimedia-Language-setup
Joe added a comment to T169450: Redirect several wikis.

With my latest patch, all redirects (including the mowiki and mowiktionary ones) work as expected. It will take some time for the infrastructure to catch up, but probably not too long. The code can be tested with X-Wikimedia-Debug set to mwdebug1001.

Dec 12 2017, 12:42 PM · Operations, Wikimedia-Apache-configuration, Puppet, Wiki-Setup (Close), Wikimedia-Language-setup

Dec 11 2017

Joe added a project to T180384: Turn off Trending Service: User-Joe.
Dec 11 2017, 5:22 PM · Services (done), User-Joe, Operations, Reading-Infrastructure-Team-Backlog (Kanban), Trending-Service
Joe created T182597: Use EtcdConfig in production to allow automation of a datacenter switch.
Dec 11 2017, 3:35 PM · Patch-For-Review, discovery-system, MediaWiki-Configuration, Operations
Joe updated subscribers of T146285: Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7).

@tstarling AIUI we should be able to switch mwscript to use 'php' instead of 'php5' now, correct?

Dec 11 2017, 10:30 AM · Release-Engineering-Team (Someday), Deployments, User-Joe, Beta-Cluster-reproducible, HHVM, Operations
Joe moved T177394: Experiment with a TLS proxy/router for pods from Backlog to Doing on the User-Joe board.
Dec 11 2017, 10:11 AM · User-Joe, Prod-Kubernetes, Operations, Kubernetes
Joe moved T177397: Create scaffolding of services templates for deployment in production/staging from Backlog to Doing on the User-Joe board.
Dec 11 2017, 10:11 AM · Patch-For-Review, Prod-Kubernetes, User-Joe, Operations, Kubernetes
Joe added a comment to T177394: Experiment with a TLS proxy/router for pods.

I am experimenting a bit with envoy, and while it's interesting, and probably something we want to implement on the long run, I'm not sure we're happy to delegate routing/health checking to it for service-to-service calls at the moment. It both seems like a good idea and a too large shift in how we do things. Since all of our other services are outside of kubernetes, I'd start configuring it as a simple http reverse proxy with TLS termination.

Dec 11 2017, 10:10 AM · User-Joe, Prod-Kubernetes, Operations, Kubernetes

Dec 5 2017

Joe added a comment to T182023: Migrate htmlCacheUpdate job to Kafka.

A few things to note:

Dec 5 2017, 2:52 PM · Patch-For-Review, Services (done), MediaWiki-JobQueue, ChangeProp, EventBus, Operations, User-Joe, Analytics, User-Elukey
Joe added a project to T169450: Redirect several wikis: Operations.
Dec 5 2017, 12:48 PM · Operations, Wikimedia-Apache-configuration, Puppet, Wiki-Setup (Close), Wikimedia-Language-setup
Joe added a comment to T169450: Redirect several wikis.

Just trying to understand the context - what the original ticket states is we need to redirect from those domains to others. So the expected result is that if I type 'mo.wikipedia.org' in my browser, I'm redirected to 'ro.wikipedia.org'. Correct?

Dec 5 2017, 12:47 PM · Operations, Wikimedia-Apache-configuration, Puppet, Wiki-Setup (Close), Wikimedia-Language-setup

Dec 4 2017

mmodell awarded T180023: [DRAFT][RfC] Deployment of python applications in production a Barnstar token.
Dec 4 2017, 6:05 PM · Release-Engineering-Team (Watching / External), User-Joe, Operations
Joe added a comment to T181920: etcd-mirror failure.

It was trying to cleanup. There are issues with twisted + deferToThread I never got around solving. The 'NoneType' + int is a bug in logging that I should indeed fix but is not critical.

Dec 4 2017, 5:04 PM · Operations
Joe created T181971: Disable hiera autolookups.
Dec 4 2017, 10:47 AM · Patch-For-Review, User-Joe, Puppet, Operations
Joe created T181967: Update puppet code to conform to puppet 4.x and later standards.
Dec 4 2017, 9:58 AM · User-Joe, Puppet, Operations
Joe added a comment to T181920: etcd-mirror failure.

This is a reoccurrence of an old bug (T162013), basically the issue is we have the md resync happening every first Sunday of the month on the conf2* cluster, and that sometimes causes consensus issues.

Dec 4 2017, 6:29 AM · Operations

Dec 1 2017

Joe added a comment to P6415 Puppet....
define die_puppet_die ($content){                                                                                                                                            
    @@file { $title:                                                                                                                                                         
        ensure => present,                                                                                                                                                   
        content => $content                                                                                                                                                  
    }                                                                                                                                                                        
}
Dec 1 2017, 3:05 PM
Joe created P6415 Puppet....
Dec 1 2017, 2:55 PM