Page MenuHomePhabricator

Southparkfan (Ferran Tufan)
Security

Projects (11)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Oct 12 2014, 7:12 AM (362 w, 4 d)
Availability
Available
IRC Nick
Southparkfan
LDAP User
Southparkfan
MediaWiki User
Southparkfan [ Global Accounts ]

The guy who fixes bugs. In daily life I am a Computer Science student specialising in Cyber Security & Cloud.

Macro southparkfan-approves: Approved by Wikimedia

Recent Activity

Jun 29 2021

Southparkfan updated Southparkfan.
Jun 29 2021, 1:18 PM

May 4 2021

Southparkfan added a member for Security: Southparkfan.
May 4 2021, 7:13 PM

Apr 12 2021

Southparkfan updated Southparkfan.
Apr 12 2021, 10:32 PM

Apr 2 2021

Southparkfan added a comment to T127717: Move Cloud VPS auth.logs to central logging.

Short update: using local puppet patches, I got the syslog forwarding working. Next step will be getting those patches into the production branch.

Apr 2 2021, 11:37 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services

Mar 4 2021

Southparkfan added a comment to T276291: Request creation of auditlogging VPS project.

@Bstorm thank you! I presume the project will be created shortly? Let me know if there is anything I can do.

Mar 4 2021, 11:03 PM · Cloud-VPS (Project-requests)

Mar 2 2021

Southparkfan updated the task description for T276291: Request creation of auditlogging VPS project.
Mar 2 2021, 11:38 PM · Cloud-VPS (Project-requests)
Southparkfan added a parent task for T276291: Request creation of auditlogging VPS project: T127717: Move Cloud VPS auth.logs to central logging.
Mar 2 2021, 11:38 PM · Cloud-VPS (Project-requests)
Southparkfan added a subtask for T127717: Move Cloud VPS auth.logs to central logging: T276291: Request creation of auditlogging VPS project.
Mar 2 2021, 11:38 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services
Southparkfan created T276291: Request creation of auditlogging VPS project.
Mar 2 2021, 11:37 PM · Cloud-VPS (Project-requests)

Feb 26 2021

Southparkfan removed a watcher for DC-Ops: Southparkfan.
Feb 26 2021, 11:15 PM

Feb 25 2021

Southparkfan added a comment to T127717: Move Cloud VPS auth.logs to central logging.

So central logging is a thing we've been very interested in for Cloud VPS projects. A strictly security-focused syslog server would have value even if it was just for cloud admins. If it's something more like a multitenant logging solution that somehow implements good multitenancy in Elastic's stack without their enterprise offering or a multitenant Grafana Loki system (which would be simpler on the log aggregation end but not-so-simple at the Grafana end without "enterprise" features since you'd need to have it set up as multi-org) that'd be quite a project to dig into! Loki is nice because it is not slated to stop being Apache 2 licensed, but it is also not deployed at Wikimedia at all yet (though Grafana obviously is).

Yes, Grafana Loki is one of the newer solutions. I am not experienced with Grafana Loki, but it may be the best solution for extracting metrics from logs, although I am not sure if Grafana Loki fits any high availability / scalability requirements. Elastic is more known, but with the license issues, it's better to let that slide for now (if the software is forked, we can reconsider!). What do you think?

Feb 25 2021, 1:20 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services

Feb 17 2021

Southparkfan added a comment to T127717: Move Cloud VPS auth.logs to central logging.

@Andrew / @bd808, thank you for all of your comments. I wouldn't call myself the expert of WMCS and ELK, but I do have experience with syslog and log management, albeit with other tools (unfortunately nothing OSI-approved anymore, since Graylog has the SSPL license now as well).

Feb 17 2021, 10:12 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services

Feb 15 2021

Restricted Application added a project to T127717: Move Cloud VPS auth.logs to central logging: cloud-services-team (Kanban).

As the original reporter of T127656, could you use my help here? Central logging sounds like a fun, but useful side project.

Feb 15 2021, 9:26 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-Services

Feb 6 2021

Southparkfan placed T28488: Edit notice for each namespace should default to the value of Mediawiki:Editnotice via a redirect up for grabs.
Feb 6 2021, 10:37 PM · good first task, MediaWiki-Page-editing

Aug 25 2020

Southparkfan updated Southparkfan.
Aug 25 2020, 10:49 PM
Southparkfan added a comment to T252990: nagios-nrpe-server.service: systemd unit references path below legacy directory /var/run/.

@MoritzMuehlenhoff understood. Patch set 4 will use the custom unit (with /run) on systems older than bullseye. On bullseye systems we'll be using the built-in unit.

Aug 25 2020, 3:25 PM · observability, good first task, SRE
Southparkfan added a comment to T252990: nagios-nrpe-server.service: systemd unit references path below legacy directory /var/run/.

This still uses the legacy /var/run though, hence it does not fix the issue.

Correct, see this:

Aug 25 2020, 2:25 PM · observability, good first task, SRE

Aug 24 2020

Southparkfan awarded T261159: assess and re-evaluate 'weight' settings of appservers in codfw a Like token.
Aug 24 2020, 9:35 PM · serviceops, SRE
Southparkfan updated subscribers of T252990: nagios-nrpe-server.service: systemd unit references path below legacy directory /var/run/.

I have uploaded a new patch using /run on all servers (regardless of OS). However, what about removing the unit and using the built-in one? That reduces the likelihood of units conflicting with intended behavior from upstream. An apt-get source for nagios-nrpe=3.2.1-2 shows the following unit (debian/nagios-nrpe-server.service):

[Unit]
Description=Nagios Remote Plugin Executor
Documentation=http://www.nagios.org/documentation
After=var-run.mount nss-lookup.target network.target local-fs.target remote-fs.target time-sync.target
Before=getty@tty1.service plymouth-quit.service xdm.service
Conflicts=nrpe.socket
Aug 24 2020, 9:21 PM · observability, good first task, SRE
Southparkfan claimed T28488: Edit notice for each namespace should default to the value of Mediawiki:Editnotice via a redirect.
Aug 24 2020, 12:29 AM · good first task, MediaWiki-Page-editing

Aug 23 2020

Southparkfan claimed T253726: User merge log is not visible in recentchanges.
Aug 23 2020, 11:58 PM · Growth-Team-Filtering, Patch-For-Review, good first task, MediaWiki-Recent-changes, MediaWiki-Logevents, MediaWiki-extensions-UserMerge, Growth-Team
Southparkfan claimed T183628: purgeList.php doesn't purge all URL variants provided in the TitleSquidURLs hook.

@Ciencia_Al_Poder What do you mean with internal URLs? purgeList.php fetches the URLS using Title->getCdnUrls(), which seems to work fine in 1.34.2:

Aug 23 2020, 11:48 PM · good first task, MediaWiki-Maintenance-system
Southparkfan claimed T254087: Remove unused Title::nameOf in MediaWiki Core.
Aug 23 2020, 11:36 PM · MW-1.36-notes (1.36.0-wmf.6; 2020-08-25), good first task, Platform Engineering, User-DannyS712, Technical-Debt (Deprecation process)
Southparkfan claimed T252990: nagios-nrpe-server.service: systemd unit references path below legacy directory /var/run/.
Aug 23 2020, 10:31 PM · observability, good first task, SRE
Southparkfan added a comment to T252990: nagios-nrpe-server.service: systemd unit references path below legacy directory /var/run/.

The PID directory has been changed to /run since Buster: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932353#10

Aug 23 2020, 10:31 PM · observability, good first task, SRE
Southparkfan added a comment to T260739: Migrate wikistats proxy from wmflabs.org to wmcloud.org domain.

@Southparkfan Thank you very much for this detailed ticket! Everything went fine going through the check boxes.

You're welcome.

Except I wanted to ask where you got the 308 from specifically.

I got that from T256276. However, https://gerrit.wikimedia.org/r/c/operations/puppet/+/612442/3/modules/dynamicproxy/files/domainproxy.lua says "ngx.HTTP_MOVED_PERMANENTLY", which is a 301. Either one is fine, though.

Aug 23 2020, 9:51 PM · VPS-project-Wikistats

Aug 18 2020

Southparkfan created T260739: Migrate wikistats proxy from wmflabs.org to wmcloud.org domain.
Aug 18 2020, 9:42 PM · VPS-project-Wikistats

Aug 11 2020

Southparkfan awarded Blog Post: RPKI Origin Validation a Party Time token.
Aug 11 2020, 12:39 AM · netops

Aug 8 2020

Southparkfan created T259956: StructuredDiscussions hook onDeletedContributionsLineEnding a tad slow on Special:Contributions.
Aug 8 2020, 12:21 AM · Growth-Team-Filtering, Performance Issue, User-RhinosF1, StructuredDiscussions, Growth-Team

Aug 7 2020

Southparkfan created P12202 Flow hook performance on Special:Contributions.
Aug 7 2020, 11:51 PM · Performance Issue

May 22 2020

Southparkfan updated Southparkfan.
May 22 2020, 2:32 PM
Southparkfan created T253385: Forward social media events to IRC.
May 22 2020, 2:30 PM · User-Examknow, User-RhinosF1, ZppixBot

Apr 10 2020

Southparkfan created T249967: MediaWiki installer does not transclude PLURAL in error message.
Apr 10 2020, 10:47 PM · MediaWiki-Internationalization, MediaWiki-Installer

Mar 13 2020

Southparkfan awarded T193613: RFC: Establish stable interface policy for PHP code a Like token.
Mar 13 2020, 12:43 PM · TechCom-RFC (TechCom-RFC-Closed), Platform Engineering, Discovery-Search, MediaWiki-General

Jan 11 2020

Southparkfan placed T113150: [CreateRedirect] Rewrite Special:CreateRedirect with FormSpecialPage up for grabs.

Definitely not working on this. Thanks for the help!

Jan 11 2020, 1:16 AM · MediaWiki-extensions-CreateRedirect

Dec 4 2019

Ostrzyciel awarded T235551: InstantCommons should cache remote images a Like token.
Dec 4 2019, 7:48 PM · Platform Engineering (Icebox), Performance-Team (Radar), MediaWiki-File-management, Multimedia, Commons

Nov 4 2019

Southparkfan reopened T235551: InstantCommons should cache remote images as "Open".

I feel my point was completely missed, regardless of the cause or fix: this change has a performance impact of roughly 28x. And since I have new questions, reopening.

Nov 4 2019, 11:20 PM · Platform Engineering (Icebox), Performance-Team (Radar), MediaWiki-File-management, Multimedia, Commons

Oct 17 2019

Southparkfan added a comment to T138453: Many articles containing UNIQ--postMath-00000001-QINU.

It doesn't seem to be limited to broken citations, {{Special:WhatlinksHere/{{FULLPAGENAME}}}} also seems enough to reproduce this.

Oct 17 2019, 11:11 AM · User-Ryasmeen, Cite, Math

Oct 15 2019

Southparkfan added a project to T235551: InstantCommons should cache remote images: Performance-Team.
Oct 15 2019, 7:32 PM · Platform Engineering (Icebox), Performance-Team (Radar), MediaWiki-File-management, Multimedia, Commons
Southparkfan updated subscribers of T235551: InstantCommons should cache remote images.
Oct 15 2019, 7:31 PM · Platform Engineering (Icebox), Performance-Team (Radar), MediaWiki-File-management, Multimedia, Commons
Southparkfan created T235551: InstantCommons should cache remote images.
Oct 15 2019, 7:30 PM · Platform Engineering (Icebox), Performance-Team (Radar), MediaWiki-File-management, Multimedia, Commons

Aug 8 2019

Southparkfan added a comment to T230075: Setting up static maintenance page on Foundation servers for Foundation website.

GDNSD supports HTTP health checks (or custom ones as well). Provided the DNS TTL is no longer than a few minutes, 'automatic failover' is possible.

Aug 8 2019, 12:34 AM · Traffic-Icebox, Security, wikimediafoundation.org, SRE

Jun 10 2019

Southparkfan added a comment to T225035: cp3035 PS Redundancy Lost.

The tasks regarding loss of PSU redundancy on cp303[2689] are normal priority, does this one need to be high priority?

Jun 10 2019, 10:14 PM · Traffic, SRE, ops-esams

Mar 28 2019

Southparkfan awarded Blog Post: Switching production traffic to Apache Traffic Server a Heartbreak token.
Mar 28 2019, 9:44 PM · Traffic
Southparkfan removed a member for Shinken: Southparkfan.
Mar 28 2019, 9:41 PM

Jan 25 2017

Southparkfan edited projects for T156208: reclaim multatuli, added: ops-esams; removed ops-eqiad.

(correct datacenter)

Jan 25 2017, 12:23 PM · ops-esams, Patch-For-Review, SRE

Dec 30 2016

Southparkfan added a member for DC-Ops: Southparkfan.
Dec 30 2016, 3:53 PM
Southparkfan added a watcher for DC-Ops: Southparkfan.
Dec 30 2016, 3:53 PM
Southparkfan added a watcher for ops-esams: Southparkfan.
Dec 30 2016, 3:52 PM
Southparkfan added a watcher for ops-eqiad: Southparkfan.
Dec 30 2016, 3:52 PM
Southparkfan added a watcher for ops-codfw: Southparkfan.
Dec 30 2016, 3:52 PM
Southparkfan added a watcher for ops-ulsfo: Southparkfan.
Dec 30 2016, 3:49 PM
Southparkfan added a member for ops-ulsfo: Southparkfan.
Dec 30 2016, 3:49 PM

Nov 18 2016

Southparkfan reopened T141482: CentralAuth login attempt gives "No active login attempt is in progress for your session" as "Open".

The same issue is occurring again at a high rate, and is causing problems. Also, there are users reporting they are silently being logged out of the site. I'm not sure how much they are related to each other, but I spotted this in the debug logs:

[session] Session "[50]CentralAuthSessionProvider<-:2:Southparkfan>[REDACTED]": Metadata merge failed: [Exception MediaWiki\Session\MetadataMergeException(
 /srv/mediawiki/w/includes/session/SessionProvider.php:195) Key "CentralAuthSource" changed]
#0 /srv/mediawiki/w/includes/session/SessionManager.php(629): MediaWiki\Session\SessionProvider->mergeMetadata(array, array)
#1 /srv/mediawiki/w/includes/session/SessionManager.php(498): MediaWiki\Session\SessionManager->loadSessionInfoFromStore(MediaWiki\Session\SessionInfo, WebRequest)
#2 /srv/mediawiki/w/includes/session/SessionManager.php(182): MediaWiki\Session\SessionManager->getSessionInfoForRequest(WebRequest)
#3 /srv/mediawiki/w/includes/WebRequest.php(700): MediaWiki\Session\SessionManager->getSessionForRequest(WebRequest)
#4 /srv/mediawiki/w/includes/session/SessionManager.php(121): WebRequest->getSession()
#5 /srv/mediawiki/w/includes/Setup.php(747): MediaWiki\Session\SessionManager::getGlobalSession()
#6 /srv/mediawiki/w/includes/WebStart.php(137): require_once(string)
#7 /srv/mediawiki/w/index.php(40): require(string)
#8 {main}
Nov 18 2016, 9:01 PM · User-Zppix, MediaWiki-Authentication-and-authorization, MediaWiki-extensions-CentralAuth

Nov 9 2016

Southparkfan added a project to T147309: Decommission db1019: ops-eqiad.
Nov 9 2016, 4:37 PM · ops-eqiad, hardware-requests, SRE

Sep 17 2016

Southparkfan added a comment to T133674: HHVM is leaking memory on the API appservers.

Is this the same as the recent behavior seen on various API appservers? https://ganglia.wikimedia.org/latest/?c=API%20application%20servers%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2

Sep 17 2016, 4:32 PM · MediaWiki-API, Sustainability, HHVM, SRE

Aug 24 2016

Southparkfan added a comment to T139171: Rack/setup sodium (carbon/mirror server replacement).

I wonder why @RobH and @Cmjohnson are talking about 4TB disks. The current problems are caused by 4k (6TB) disks, and the accessoires link given by Cmjohnson mentions a non-4k 6TB disk.

Aug 24 2016, 5:40 PM · SRE, ops-eqiad
Southparkfan added a comment to T139171: Rack/setup sodium (carbon/mirror server replacement).

I wonder why @RobH and @Cmjohnson are talking about 4TB disks. The current problems are caused by 4k (6TB) disks, and the accessoires link given by Cmjohnson mentions a non-4k 6TB disk.

Aug 24 2016, 5:40 PM · SRE, ops-eqiad

Aug 10 2016

Southparkfan added a comment to T127904: Have fallback communication channel when freenode has problems.

Do we prefer a fallback that cannot be impacted by a Wikimedia outage of any kind? Conpherence is an option, but it is not off-site; a network outage affecting Wikimedia will render Conpherence useless.

Aug 10 2016, 9:44 PM · Deployments, SRE

Jul 31 2016

Southparkfan added a comment to T141482: CentralAuth login attempt gives "No active login attempt is in progress for your session".

Possibly your farm is not sharing sessions, which is a requirement for CentralAuth. Session storage configuration has changed for 1.27, you might want to review that.

It could be possible that my configuration is not up-to-date. Like Wikimedia, we use redis for sessions (production 1.26 config):

$wgObjectCaches['redis'] = array(
	'class' => 'RedisBagOStuff',
	'servers' => array( '<IP>' ),
	'password' => $wmgRedisPassword,
);
$wgMainCacheType = 'redis';
$wgSessionCacheType = 'redis';
$wgSessionsInObjectCache = true;
$wgMessageCacheType = CACHE_NONE;
$wgParserCacheType = CACHE_DB;
$wgLanguageConverterCacheType = CACHE_DB;
Jul 31 2016, 10:27 AM · User-Zppix, MediaWiki-Authentication-and-authorization, MediaWiki-extensions-CentralAuth

Jul 27 2016

Southparkfan updated the task description for T141482: CentralAuth login attempt gives "No active login attempt is in progress for your session".
Jul 27 2016, 9:08 PM · User-Zppix, MediaWiki-Authentication-and-authorization, MediaWiki-extensions-CentralAuth
Southparkfan created T141482: CentralAuth login attempt gives "No active login attempt is in progress for your session".
Jul 27 2016, 9:08 PM · User-Zppix, MediaWiki-Authentication-and-authorization, MediaWiki-extensions-CentralAuth

Jul 26 2016

Southparkfan moved T141352: Decomission mw1153-mw1160 from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.
Jul 26 2016, 12:53 PM · ops-eqiad, SRE

Jul 15 2016

Southparkfan awarded T140422: madhuvishy is moving to operations on 7/18/16 a Love token.
Jul 15 2016, 9:59 PM · SRE-Access-Requests, Cloud-Services, SRE

Jul 12 2016

Southparkfan closed T138509: rack/setup/install/deploy labvirt1012 labvirt1013 labvirt1014 nodes (cloudvirt1012 cloudvirt1013 cloudvirt1014) as Resolved.

Yes, it does.

Jul 12 2016, 8:49 AM · Patch-For-Review, ops-eqiad, SRE
Southparkfan added a comment to T136575: Allow default value for PATH variable in wfShellExec.

Step 1 seems to be re-introducing the texvc package (whose deletion actually was unrelated when I created the issue, it just made this task more or less invalid, because the issue was kinda gone - since nobody (except users who still have that package installed) would be hit by it anymore). That is a blocker for this task.

Jul 12 2016, 8:08 AM · MediaWiki-extensions-General

Jul 11 2016

Southparkfan added a comment to T138509: rack/setup/install/deploy labvirt1012 labvirt1013 labvirt1014 nodes (cloudvirt1012 cloudvirt1013 cloudvirt1014).

@Andrew labvirt1012 lacks hyperthreading. Can you enable that?

Jul 11 2016, 10:08 PM · Patch-For-Review, ops-eqiad, SRE

Jul 3 2016

Southparkfan added a comment to T139154: Several mw* servers missing in conftool-data, but present in site.pp.

mw1017 and mw1099 are the eqiad MW debug hosts, so those should not be in conftool-data.
mw1161-mw1169 were previous appservers, but have been converted to jobrunners in December, per rOPUPba0a47b56ded3dc748c436c2940f114389b312f2. Jobrunners are not present in conftool-data, so I guess everything is as expected?

Jul 3 2016, 3:29 PM · SRE
Southparkfan updated subscribers of T136575: Allow default value for PATH variable in wfShellExec.

@Kghbln yes, I am aware of that, and I spoke to @MoritzMuehlenhoff, and he said that in hindsight the mediawiki-math-texvc should never have been deleted. Perhaps it will be packaged again - but I'm not sure if Debian's policies allow that, since it would mean that it has to be introduced in the jessie (stable) suite again. Otherwise we'd still need to wait until Debian 9 has been released.

Jul 3 2016, 3:15 PM · MediaWiki-extensions-General

Jun 17 2016

Southparkfan added a comment to T134309: Rack and Set up new application servers mw1284-1306.

@Joe mw1306 seems to have the same IP as mw1091 (although that one shows up as mw1091 in Ganglia, whereas mw1090 shows up as mw1305...). So I think you need to fix that one as well.

Jun 17 2016, 10:05 PM · Patch-For-Review, SRE, ops-eqiad

Jun 11 2016

Southparkfan added a comment to T137622: Closing MediaViewer on desktop results in losing the reading position on a page on Firefox.

Just tested this, this doesn't happen in Firefox 47.0 on Windows 8.1. Perhaps an upgrade to 47.0 might help, although I doubt it because I can't see in the 47.0 release notes anything related to this.

Jun 11 2016, 8:09 AM · Multimedia, Readers-Web-Backlog, MediaViewer
Southparkfan closed T137594: Miraheze: Broken Flow topics with non-existent pages and strange raw JSON data as Resolved.

Yep, that change (and the maintenance script) worked perfect. Everything is displaying correctly now.

Jun 11 2016, 8:05 AM · StructuredDiscussions

Jun 10 2016

Southparkfan updated subscribers of T137594: Miraheze: Broken Flow topics with non-existent pages and strange raw JSON data.
Jun 10 2016, 9:10 PM · StructuredDiscussions
Southparkfan created T137594: Miraheze: Broken Flow topics with non-existent pages and strange raw JSON data.
Jun 10 2016, 9:09 PM · StructuredDiscussions
Southparkfan closed T132859: Investigate cp1044's strange Ganglia graphs as Declined.

cp1044 has been decomissioned per T133614

Jun 10 2016, 8:49 PM · SRE

Jun 8 2016

Southparkfan added a comment to T130130: [Epic] Upgrade paymentswiki to MediaWiki REL_* release branch or follow the WMF train. Upgrage PHP to HHVM or PHP 5.5.

That would be great! The only additional benefit we might get from HHVM
would be to cut PHP cpu load by 50%, but that can be an optimization for
later. This card is concerned with the breaking language changes, so
Jessie is a perfect solution.

As of PHP 5.5 (Jessie has 5.6), there is an opcache extension included (only needs to be enabled in config), which caches compiled bytecode in the memory. Together with the other performance improvements since 5.4 (assuming paymentswiki runs on 5.3 now), CPU usage / response time should already decrease without using HHVM (so you get some more headroom before HHVM is really needed).

Jun 8 2016, 2:23 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Southparkfan added a comment to T136938: build payments1005 with Debian Jessie, and HHVM or PHP5.5.

Jessie has PHP 5.6 actually.

Jun 8 2016, 2:17 PM · fundraising-tech-ops

Jun 6 2016

Southparkfan added a comment to T132454: Some OAuth special pages are not recognized ('no such special page').

I'm sure Legoktm is right :-)! But I failed to make it work, so someone else should take it over honestly...

Jun 6 2016, 8:02 PM · MW-1.29-release (WMF-deploy-2017-03-14_(1.29.0-wmf.16)), Patch-For-Review, good first task, MediaWiki-extensions-OAuth
Southparkfan added a comment to T132454: Some OAuth special pages are not recognized ('no such special page').

It would be cool if this bug could be fixed (whatever the solution is).

Jun 6 2016, 4:36 PM · MW-1.29-release (WMF-deploy-2017-03-14_(1.29.0-wmf.16)), Patch-For-Review, good first task, MediaWiki-extensions-OAuth

May 31 2016

Southparkfan updated the task description for T136631: rack/setup/deploy ms-be102[2-7].
May 31 2016, 7:34 PM · Patch-For-Review, SRE-swift-storage, ops-eqiad, SRE
Southparkfan added a comment to T136634: Lost Wikitech 2FA details, recovery needed.
southparkfan@tools-exec-1407:~$ echo 'oht0ipe1Pho7aa7pohChie8eath0ogoo9Eesoh9nahc3aefoh7ie2ais6oohugoo' > reset_2fa.txt
southparkfan@tools-exec-1407:~$ cat reset_2fa.txt
oht0ipe1Pho7aa7pohChie8eath0ogoo9Eesoh9nahc3aefoh7ie2ais6oohugoo
southparkfan@tools-exec-1407:~$
May 31 2016, 6:46 PM · Cloud-Services
Southparkfan created T136634: Lost Wikitech 2FA details, recovery needed.
May 31 2016, 6:39 PM · Cloud-Services

May 30 2016

Southparkfan created T136575: Allow default value for PATH variable in wfShellExec.
May 30 2016, 6:46 PM · MediaWiki-extensions-General

May 22 2016

Southparkfan added a comment to T123536: Array to string conversion in MWTimestamp.php causes HHVM 3.9 to segfault (works in HHVM 3.12).

@Paladox I've already upgraded to HHVM 3.12, which seems to work.

May 22 2016, 4:49 PM · HHVM, MediaWiki-General
Southparkfan added a comment to T134309: Rack and Set up new application servers mw1284-1306.

@Joe did you already install one of those servers? noticed https://ganglia.wikimedia.org/latest/?c=Application%20servers%20eqiad&h=mw1305.eqiad.wmnet&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2 (I can't view which specs the new servers have, but this 'mw1305' seems to have the same cpu/ram specs as the old appservers?)

May 22 2016, 11:20 AM · Patch-For-Review, SRE, ops-eqiad

May 6 2016

Southparkfan added a comment to T134360: Decommission broken db1058.

@Cmjohnson yeah, perhaps I have been a bit too fast by already doing the DNS part (despite that's the only thing I can do it seems) :-)

May 6 2016, 1:16 PM · SRE, ops-eqiad

May 5 2016

Southparkfan added a comment to T134360: Decommission broken db1058.

Just to learn how the process works, I've submitted a patch for the DNS adjustments. I noticed db1058 is referenced in the dhcpd and manifests/role/coredb.pp files in puppet but I have no idea how the latter one works, so I'll leave the puppet work to someone else.

May 5 2016, 9:17 PM · SRE, ops-eqiad

Apr 28 2016

Southparkfan renamed T133798: Rack and Set up new application servers mw1261-1283 from Rack and Set up new application servers mw1216-1307 to Rack and Set up new application servers mw1261-1307.
Apr 28 2016, 4:05 PM · Patch-For-Review, SRE, ops-eqiad

Apr 26 2016

Southparkfan added a comment to T133700: MobileFrontend contributions display edits where username is revdel'd without having appropriate rights.

Can this patch be backported to REL1_26, REL1_25 and REL1_23 too? The current security patch is only available in master, and these three branches still receive security support.

Apr 26 2016, 7:35 PM · Security, Patch-For-Review, MW-1.27-release (WMF-deploy-2016-05-03_(1.27.0-wmf.23)), Reading-Web-Sprint-71-Matisse-Monet-Kandinsky-and-the-Departing-Painters, Unplanned-Sprint-Work, Reading-Web-Sprint-70-Lady-and-the-Trumps, Vuln-Infoleak, MediaWiki-Revision-deletion, MobileFrontend
Southparkfan updated the task description for T133717: Letsencrypt all the prod things we can - planning.
Apr 26 2016, 5:57 PM · SRE, Traffic
Southparkfan awarded T133679: Could use a coffee in Nantes a Like token.
Apr 26 2016, 1:26 PM

Apr 23 2016

Southparkfan added a comment to T133441: Seeing desktop text cache while browsing mobile sites.

Yeah, this doesn't seem to be a Varnish problem:

Apr 23 2016, 4:01 PM · Readers-Web-Backlog, MobileFrontend, SRE, Traffic
Southparkfan created T133441: Seeing desktop text cache while browsing mobile sites.
Apr 23 2016, 9:57 AM · Readers-Web-Backlog, MobileFrontend, SRE, Traffic

Apr 19 2016

Southparkfan created T133093: Investigate idle appservers in codfw.
Apr 19 2016, 7:44 PM · SRE

Apr 17 2016

Southparkfan closed T116266: Undefined property: DiffEngine::$seq as Resolved.

@ori that is T124163 actually.

Apr 17 2016, 3:09 PM · affects-translatewiki.net, MediaWiki-Page-history, MW-1.26-release, MW-1.27-release (WMF-deploy-2015-12-08_(1.27.0-wmf.8)), MW-1.27-release-notes, Patch-For-Review, MediaWiki-Page-diffs
Southparkfan closed T116266: Undefined property: DiffEngine::$seq, a subtask of T41480: [DO NOT USE] Issues affecting translatewiki.net [superseded by #affects-translatewiki.net], as Resolved.
Apr 17 2016, 3:09 PM · affects-translatewiki.net, Tracking-Neverending
Southparkfan updated subscribers of T114715: Undefined index: function in MWExceptionHandler.php on line 386.
Apr 17 2016, 3:04 PM · User-bd808, Wikimedia-production-error, MediaWiki-General
Southparkfan merged task T116311: Undefined index: function in /srv/mediawiki/php-1.27.0-wmf.3/includes/exception/MWExceptionHandler.php on line 386 into T114715: Undefined index: function in MWExceptionHandler.php on line 386.
Apr 17 2016, 3:04 PM · MediaWiki-General
Southparkfan added a comment to T132845: API apache servers OOMing: mw1134 mw1132 mw1139 mw1138.

@Andrew mw1138 is not depooled (anymore), its CPU and network graphs show it is serving traffic. Looking at http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&h=mw1132.eqiad.wmnet&m=cpu_report&s=by+name&mc=2&g=network_report&c=API+application+servers+eqiad it was idle for a few hours, but then it somehow got repooled..

Apr 17 2016, 10:42 AM · SRE

Apr 16 2016

Southparkfan created T132859: Investigate cp1044's strange Ganglia graphs.
Apr 16 2016, 9:02 PM · SRE