madhuvishy (Madhu)Disabled
Disabled

Projects (6)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Apr 13 2015, 10:09 PM (192 w, 1 d)
Roles
Disabled
Availability
Available
LDAP User
Unknown
MediaWiki User
MViswanathan (WMF) [ Global Accounts ]

Recent Activity

Jun 11 2018

Gerrit Code Review <gerrit@wikimedia.org> committed rOWPI411ba3f7cbd9: Update patch set 1 (authored by madhuvishy).
Update patch set 1
Jun 11 2018, 2:10 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rOWPI4ffd0a98f655: Update patch set 1 (authored by madhuvishy).
Update patch set 1
Jun 11 2018, 2:10 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rOWPIa693e06bb579: Create change (authored by madhuvishy).
Create change
Jun 11 2018, 2:10 PM
Gerrit Code Review <gerrit@wikimedia.org> committed rODITc995451416f6: Update patch set 1 (authored by madhuvishy).
Update patch set 1
Jun 11 2018, 11:36 AM
Gerrit Code Review <gerrit@wikimedia.org> committed rODITf380cf64d9cb: Update patch set 1 (authored by madhuvishy).
Update patch set 1
Jun 11 2018, 11:35 AM

Apr 24 2018

madhuvishy closed T168486: Migrate customer-facing Dumps endpoints to Cloud Services as Resolved.
Apr 24 2018, 11:58 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy closed T168486: Migrate customer-facing Dumps endpoints to Cloud Services, a subtask of T166402: Program 7 Outcome 3: data services, as Resolved.
Apr 24 2018, 11:58 PM · Data-Services, cloud-services-team (FY2017-18), Goal
madhuvishy closed T168486: Migrate customer-facing Dumps endpoints to Cloud Services, a subtask of T182540: get datset1001, ms1001 ready for decommission, as Resolved.
Apr 24 2018, 11:58 PM · Patch-For-Review, Dumps-Generation
madhuvishy added a comment to T168486: Migrate customer-facing Dumps endpoints to Cloud Services.

The nginx logs from the active web server (labstore1007) need to get shipped over to stat1005 somehow, as we do with the dataset1001 logs for now in profile::dumps::web::xmldumps_active. If that happens already someplace, I could not find it.

Apr 24 2018, 11:57 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal

Apr 20 2018

madhuvishy closed T188726: make sure all datasets in xmldatadumps/public/other on dataset1001 are accounted for on new labs boxes as Resolved.
Apr 20 2018, 3:28 AM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown
madhuvishy closed T188726: make sure all datasets in xmldatadumps/public/other on dataset1001 are accounted for on new labs boxes, a subtask of T171541: Setup periodic rsync jobs from dumps generation hosts to labstore1006|7, as Resolved.
Apr 20 2018, 3:28 AM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown

Apr 17 2018

madhuvishy added a comment to T189283: Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches.

@Ottomata Thanks for fixing up the rsync jobs! Can we close this task now?

Apr 17 2018, 4:25 PM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown

Apr 13 2018

madhuvishy updated the task description for T168486: Migrate customer-facing Dumps endpoints to Cloud Services.
Apr 13 2018, 3:44 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal

Apr 11 2018

madhuvishy closed T188643: Migrate Dumps WMCS NFS users from labstore1003 to labstore1006/7 as Resolved.
Apr 11 2018, 6:29 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy closed T188643: Migrate Dumps WMCS NFS users from labstore1003 to labstore1006/7, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Apr 11 2018, 6:29 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy added a comment to T191318: Stop managing nfs shares for wikidata-dev project.

@hoo, alright thanks, feel free to ping me or the team if you need to reenable nfs for some reason!

Apr 11 2018, 6:10 PM · Data-Services
madhuvishy closed T188644: Migrate the stat* mount from dataset1001 to labstore1006/7 as Resolved.
Apr 11 2018, 6:07 PM · Patch-For-Review, cloud-services-team (Kanban), Datasets-General-or-Unknown
madhuvishy closed T188644: Migrate the stat* mount from dataset1001 to labstore1006/7, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Apr 11 2018, 6:07 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy closed T188645: Get all the rsync mirror sites to to switch over to labstore1006,7, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Apr 11 2018, 5:21 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy closed T188645: Get all the rsync mirror sites to to switch over to labstore1006,7 as Resolved.

Done with https://gerrit.wikimedia.org/r/#/c/425246/1

Apr 11 2018, 5:21 PM · cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy closed T188646: Point dumps.wikimedia.org to labstore1006/7 as Resolved.

Done with https://gerrit.wikimedia.org/r/#/c/425234/1

Apr 11 2018, 5:20 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy closed T188646: Point dumps.wikimedia.org to labstore1006/7, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Apr 11 2018, 5:20 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal

Apr 10 2018

madhuvishy added a comment to T189283: Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches.

The rsync config that allows old style @ezachte to sync to labstore1006 & 7 already exist. We haven't talked about switching on the old setup for the new servers since I thought the jobs are being changed so we can sync from stat1005. I'm ready to turn on the rsync jobs whenever, but I don't see the data in /srv/dumps yet either.

Apr 10 2018, 3:37 PM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown

Apr 6 2018

madhuvishy added a comment to T176091: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet).

Fixed by running sudo exportfs -ra on the nfs servers and remounting on notebook*.

Apr 6 2018, 6:24 AM · Patch-For-Review, Analytics-Kanban, Analytics

Apr 4 2018

madhuvishy added a comment to T188646: Point dumps.wikimedia.org to labstore1006/7.

This is all done. Leaving it open until all existing connections to dataset1001 drop off and we stop the web server there.

Apr 4 2018, 6:17 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy closed T171508: Investigate and implement alternative for showmount based check at instance boot time as Resolved.
Apr 4 2018, 6:14 PM · cloud-services-team (Kanban), Patch-For-Review, Cloud-Services
madhuvishy closed T171508: Investigate and implement alternative for showmount based check at instance boot time, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Apr 4 2018, 6:14 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy removed a parent task for T176091: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet): T188644: Migrate the stat* mount from dataset1001 to labstore1006/7.
Apr 4 2018, 6:13 PM · Patch-For-Review, Analytics-Kanban, Analytics
madhuvishy removed a subtask for T188644: Migrate the stat* mount from dataset1001 to labstore1006/7: T176091: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet).
Apr 4 2018, 6:13 PM · Patch-For-Review, cloud-services-team (Kanban), Datasets-General-or-Unknown
madhuvishy closed T185101: Labstore1006/7 profile for meltdown kernel, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Apr 4 2018, 6:12 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy closed T185101: Labstore1006/7 profile for meltdown kernel as Resolved.
Apr 4 2018, 6:12 PM · cloud-services-team (Kanban), Operations
madhuvishy added a comment to T188646: Point dumps.wikimedia.org to labstore1006/7.

Notes from migration etherpad:

Apr 4 2018, 5:20 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy added a comment to T189283: Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches.

Y'all I'd like to gently point out the primary goal here - we want the rsyncs to happen on the labstores and not from stat1005. To that end, I'm just looking for a directory(ies) to pull from on stat1005. I think we've all agreed on /srv/dumps as the container at least once. The directory already exists. /srv/public-other seems even more generic to me. I'm happy to add a README to /srv/dumps that says this is the container directory for things are shipped to the dumps distribution servers.

Apr 4 2018, 3:31 PM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown
madhuvishy added a comment to T189283: Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches.

Let's just go with /srv/dumps since we already have that set up then.

Apr 4 2018, 12:03 AM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown

Apr 3 2018

madhuvishy added a comment to T189283: Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches.

Thanks @ezachte and @Ottomata for the helpful explanations! I think I'd like to get the immediate task at hand done first before talking about using different rsync mechanisms and also having other analytics datasets available on the dumps distribution servers.

Apr 3 2018, 4:59 PM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown
madhuvishy added a comment to T188644: Migrate the stat* mount from dataset1001 to labstore1006/7.

This went well. Clean up task pending: Remove the dumps NFS export from dataset1001

Apr 3 2018, 4:47 PM · Patch-For-Review, cloud-services-team (Kanban), Datasets-General-or-Unknown
madhuvishy added a comment to T188643: Migrate Dumps WMCS NFS users from labstore1003 to labstore1006/7.

This went pretty well! Todos for clean up:

  • Remove the dumps export from labstore1003
  • Clean up labstore1003 dumps mount code in nfsclient.pp
  • Stop dumps rsync jobs that sync to labstore1003
Apr 3 2018, 4:47 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy triaged T191318: Stop managing nfs shares for wikidata-dev project as Normal priority.
Apr 3 2018, 4:46 PM · Data-Services
madhuvishy created T191318: Stop managing nfs shares for wikidata-dev project.
Apr 3 2018, 4:46 PM · Data-Services

Apr 2 2018

madhuvishy added a comment to T176091: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet).

You'd need to apply class https://github.com/wikimedia/puppet/blob/production/modules/statistics/manifests/dataset_mount.pp, and add the servers to https://github.com/wikimedia/puppet/blob/production/hieradata/common/profile/dumps/distribution.yaml#L14 (nfs_clients), to get this to work.

Apr 2 2018, 10:48 PM · Patch-For-Review, Analytics-Kanban, Analytics
madhuvishy added a comment to T188645: Get all the rsync mirror sites to to switch over to labstore1006,7.

Nothing to actively do here, we've let the mirrors know that they should use the dumps.wikimedia.org url and that dataset1001 and it's associated IPs will be going away after the switch over.

Apr 2 2018, 10:32 PM · cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy added a comment to T189283: Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches.

@ezachte Hello, after chatting with Andrew a bit, here's the direction we have in mind (pretty similar to what we talked about with some naming adjustments)

Apr 2 2018, 9:02 PM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown
madhuvishy added a comment to T188644: Migrate the stat* mount from dataset1001 to labstore1006/7.

This went well. Clean up task pending: Remove the dumps NFS export from dataset1001

Apr 2 2018, 7:33 PM · Patch-For-Review, cloud-services-team (Kanban), Datasets-General-or-Unknown
madhuvishy added a comment to T188643: Migrate Dumps WMCS NFS users from labstore1003 to labstore1006/7.

This went pretty well! Todos for clean up:

Apr 2 2018, 7:32 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy added a comment to T188643: Migrate Dumps WMCS NFS users from labstore1003 to labstore1006/7.

Notes from migration plan doc:

Apr 2 2018, 6:37 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy added a comment to T188644: Migrate the stat* mount from dataset1001 to labstore1006/7.

Notes from migration plan:

Apr 2 2018, 6:34 PM · Patch-For-Review, cloud-services-team (Kanban), Datasets-General-or-Unknown
madhuvishy added a comment to T185101: Labstore1006/7 profile for meltdown kernel.

From comparing the 2 kernels for NFSd and raw disk performance, I can see that there's a small loss in performance on both reads and writes in the new Spectre kernel. Looking at the load graphs from the fio tests, there's no significant difference in how the kernels perform under heavy load. These patterns don't seem similar to what we saw when we upgraded labstore1004 & 5 to 4.9 kernels, and my suspicion is nfs isn't the issue there.

Apr 2 2018, 5:26 PM · cloud-services-team (Kanban), Operations
madhuvishy edited P6921 Dumps enabled VPS instances.
Apr 2 2018, 3:33 PM
madhuvishy added a comment to T189283: Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches.

@Ottomata Thanks so much, /srv/wikistats_1 seems fine. There's also media and pagecounts-ez, cool to have those at the top level in /srv too?

Apr 2 2018, 2:24 PM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown
madhuvishy created P6921 Dumps enabled VPS instances.
Apr 2 2018, 5:50 AM
madhuvishy updated subscribers of T189283: Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches.

Update based on my discussion with @ezachte over email:

Apr 2 2018, 5:39 AM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown
madhuvishy closed T188647: Announce/Communicate dumps migration to labstore1006|7 to stakeholders as Resolved.
Apr 2 2018, 5:12 AM · cloud-services-team (Kanban), Data-Services, User-ArielGlenn, Datasets-General-or-Unknown
madhuvishy closed T188647: Announce/Communicate dumps migration to labstore1006|7 to stakeholders, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Apr 2 2018, 5:12 AM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy updated the task description for T188647: Announce/Communicate dumps migration to labstore1006|7 to stakeholders.
Apr 2 2018, 5:12 AM · cloud-services-team (Kanban), Data-Services, User-ArielGlenn, Datasets-General-or-Unknown
madhuvishy added a comment to P6813 drafts of announcements for migration from dataset1001->labstore1006,7.

For the web service migration, broader email blast:

Apr 2 2018, 5:06 AM
madhuvishy updated the task description for T168486: Migrate customer-facing Dumps endpoints to Cloud Services.
Apr 2 2018, 4:58 AM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy closed T188641: Set up the web service that serves dumps.wikimedia.org as Resolved.
Apr 2 2018, 4:58 AM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy closed T188641: Set up the web service that serves dumps.wikimedia.org, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Apr 2 2018, 4:58 AM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy added a comment to T188641: Set up the web service that serves dumps.wikimedia.org.

Running some load/performance tests. All tests from local machine.

Apr 2 2018, 4:50 AM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy added a comment to T188641: Set up the web service that serves dumps.wikimedia.org.

To failover between the two labstores for webservice:

Apr 2 2018, 1:36 AM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy closed T188642: Set up labstore1006|7 as the source for rsync mirror sites as Resolved.
Apr 2 2018, 1:30 AM · cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy closed T188642: Set up labstore1006|7 as the source for rsync mirror sites, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Apr 2 2018, 1:30 AM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal

Mar 30 2018

madhuvishy added a comment to T185101: Labstore1006/7 profile for meltdown kernel.

I also ran various tests using fio across the 2 kernels over NFSd - https://tools.wmflabs.org/labstore-profiling/

Mar 30 2018, 12:36 AM · cloud-services-team (Kanban), Operations
madhuvishy added a comment to T185101: Labstore1006/7 profile for meltdown kernel.

Reporting back here on what I found

Mar 30 2018, 12:03 AM · cloud-services-team (Kanban), Operations

Mar 29 2018

madhuvishy closed T188073: Maintain symlinks for WMCS NFS Dumps users with new directory structure as Resolved.
Mar 29 2018, 8:26 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services
madhuvishy closed T188073: Maintain symlinks for WMCS NFS Dumps users with new directory structure, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Mar 29 2018, 8:26 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal

Mar 28 2018

madhuvishy edited P6912 Dumps migration.
Mar 28 2018, 6:38 AM
madhuvishy edited P6912 Dumps migration.
Mar 28 2018, 6:37 AM
madhuvishy created P6912 Dumps migration.
Mar 28 2018, 6:36 AM

Mar 27 2018

madhuvishy added a comment to T190638: GSoC 2018 proposal for Improvements for the Toolforge 'webservice' command.

@Nehajha Thanks, this looks good to me. Good luck!

Mar 27 2018, 5:31 PM · Google-Summer-of-Code (2018)
madhuvishy added a comment to T190696: [Gsoc 2018] Proposal for Toolforge webservice command Improvement.

@djff This looks great! Good luck :)

Mar 27 2018, 5:30 PM · Google-Summer-of-Code (2018), Toolforge

Mar 26 2018

madhuvishy closed T187321: GSoc18 - webservice microtask for <djff> as Resolved.

I've reviewed and +1-ed the microtask. Thank you! Looking forward to seeing your proposal.

Mar 26 2018, 4:22 AM · Google-Summer-of-Code (2018), Toolforge
madhuvishy closed T187321: GSoc18 - webservice microtask for <djff>, a subtask of T175768: Improvements for the Toolforge 'webservice' command, as Resolved.
Mar 26 2018, 4:22 AM · Toolforge

Mar 25 2018

madhuvishy added a comment to T187321: GSoc18 - webservice microtask for <djff>.

@djff Hello! If you are having trouble with the microtask, do ask us questions at #wikimedia-cloud on IRC. Looking forward to your patch and proposal!

Mar 25 2018, 1:44 AM · Google-Summer-of-Code (2018), Toolforge
madhuvishy added a comment to T188066: GSoC 2018 - webservice microtask for APerson.

@APerson Hello! If you are having trouble with the microtask, do ask us questions at #wikimedia-cloud on IRC. Looking forward to your patch and proposal!

Mar 25 2018, 1:44 AM · Toolforge
madhuvishy closed T189974: GSoC - webservice microtask for Neha Jha as Resolved.

I've +1-ed the patch and resolving this task! /me waves at @Nehajha, looking forward to your proposal, do hang out at #wikimedia-cloud and ask questions if anything comes up.

Mar 25 2018, 1:41 AM
madhuvishy closed T189974: GSoC - webservice microtask for Neha Jha, a subtask of T175768: Improvements for the Toolforge 'webservice' command, as Resolved.
Mar 25 2018, 1:41 AM · Toolforge

Mar 21 2018

madhuvishy awarded T190182: implement renewed *.tools.wmflabs.org cert/key pair a Mountain of Wealth token.
Mar 21 2018, 5:19 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge, Operations
madhuvishy added a comment to T189284: Stop serving slowparse logs from dumps distribution servers.

@Legoktm cool! Thanks for weighing in. Looks like we're good to continue deprecating serving these from the servers then.

Mar 21 2018, 4:32 AM · Performance-Team (Radar), Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown

Mar 20 2018

madhuvishy added a comment to T188726: make sure all datasets in xmldatadumps/public/other on dataset1001 are accounted for on new labs boxes.

Note: Slowparse in other/ logs are being deprecated (T189284)

Mar 20 2018, 2:08 AM · Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown
madhuvishy added a comment to T189284: Stop serving slowparse logs from dumps distribution servers.

Update: I've removed all rsync related jobs and code from puppet on both dumps servers and mwlog servers. To do: stop serving at https://dumps.wikimedia.org/other/slow-parse/, and cleanup existing data from other/ on the dumps servers.

Mar 20 2018, 2:05 AM · Performance-Team (Radar), Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown

Mar 19 2018

madhuvishy renamed T189284: Stop serving slowparse logs from dumps distribution servers from Replace cron that syncs archived slow-parse logs to dataset host with server side fetch job to Stop serving slowparse logs from dumps distribution servers.
Mar 19 2018, 7:52 PM · Performance-Team (Radar), Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown

Mar 16 2018

madhuvishy updated the task description for T168486: Migrate customer-facing Dumps endpoints to Cloud Services.
Mar 16 2018, 7:39 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy updated subscribers of T189284: Stop serving slowparse logs from dumps distribution servers.

Also pinging @Krinkle

Mar 16 2018, 7:23 PM · Performance-Team (Radar), Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown
madhuvishy closed T171540: Figure out how NFS failovers will work for the dumps servers - labstore1006|7 as Resolved.

Chatted with @Ottomata today in #wikimedia-analytics, and we decided to use a similar strategy for the stat/notebook mounts. We'll mount shares from labstore1006/7 in /mnt, and symlink the active NFS one to /mnt/data (which is the current access point for stat users).

Mar 16 2018, 5:44 PM · Patch-For-Review, Data-Services
madhuvishy closed T171540: Figure out how NFS failovers will work for the dumps servers - labstore1006|7, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Mar 16 2018, 5:44 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy closed T171540: Figure out how NFS failovers will work for the dumps servers - labstore1006|7, a subtask of T181431: Setup NFS on dumps servers, as Resolved.
Mar 16 2018, 5:44 PM · Data-Services, cloud-services-team (Kanban), Patch-For-Review, Datasets-General-or-Unknown
madhuvishy added a comment to T188643: Migrate Dumps WMCS NFS users from labstore1003 to labstore1006/7.

Initial PoC patch for nfsclient.pp changes https://gerrit.wikimedia.org/r/#/c/403767/1

Mar 16 2018, 5:41 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services, Datasets-General-or-Unknown
madhuvishy closed Restricted Task, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Mar 16 2018, 4:35 PM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy added a comment to T185967: Cumin: add custom backend to WMCS.

@Volans Indeed, I fixed up the script based on the comments, we can close this task when the patch is merged! Thank you

Mar 16 2018, 6:18 AM · Patch-For-Review, cloud-services-team, Operations-Software-Development
madhuvishy added a comment to T176091: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet).

I think the firewall stuff already exists now, we can do this as part of the dumps NFS migration on April 2, and have the shares available from labstore1006|7 on notebook*. I added T188644 as a parent task.

Mar 16 2018, 12:53 AM · Patch-For-Review, Analytics-Kanban, Analytics
madhuvishy added a parent task for T176091: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet): T188644: Migrate the stat* mount from dataset1001 to labstore1006/7.
Mar 16 2018, 12:52 AM · Patch-For-Review, Analytics-Kanban, Analytics
madhuvishy added a subtask for T188644: Migrate the stat* mount from dataset1001 to labstore1006/7: T176091: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet).
Mar 16 2018, 12:52 AM · Patch-For-Review, cloud-services-team (Kanban), Datasets-General-or-Unknown
madhuvishy updated subscribers of T189284: Stop serving slowparse logs from dumps distribution servers.

@Ottomata Hey! Do you know anything about these logs? :) I'd like to make it so that we can fetch from the mwlog server when we move to the new dumps set up.

Mar 16 2018, 12:48 AM · Performance-Team (Radar), Patch-For-Review, User-ArielGlenn, Data-Services, Datasets-General-or-Unknown
madhuvishy closed T181431: Setup NFS on dumps servers as Resolved.
Mar 16 2018, 12:19 AM · Data-Services, cloud-services-team (Kanban), Patch-For-Review, Datasets-General-or-Unknown
madhuvishy closed T181431: Setup NFS on dumps servers, a subtask of T168486: Migrate customer-facing Dumps endpoints to Cloud Services, as Resolved.
Mar 16 2018, 12:19 AM · Patch-For-Review, Datasets-General-or-Unknown, cloud-services-team (FY2017-18), Goal
madhuvishy closed T181431: Setup NFS on dumps servers, a subtask of T182540: get datset1001, ms1001 ready for decommission, as Resolved.
Mar 16 2018, 12:19 AM · Patch-For-Review, Dumps-Generation

Mar 14 2018

madhuvishy updated the task description for T188647: Announce/Communicate dumps migration to labstore1006|7 to stakeholders.
Mar 14 2018, 6:19 AM · cloud-services-team (Kanban), Data-Services, User-ArielGlenn, Datasets-General-or-Unknown
madhuvishy updated the task description for T188647: Announce/Communicate dumps migration to labstore1006|7 to stakeholders.
Mar 14 2018, 5:51 AM · cloud-services-team (Kanban), Data-Services, User-ArielGlenn, Datasets-General-or-Unknown

Mar 13 2018

madhuvishy closed T136192: templatetiger is using 613G in Tools out of 8T as Resolved.
Mar 13 2018, 4:39 PM · Toolforge, Cloud-Services