Page MenuHomePhabricator

Phamhi (Phamhi)
Disabled

Projects

User Details

User Since
Aug 5 2019, 1:02 PM (96 w, 6 d)
Roles
Disabled
LDAP User
Phamhi
MediaWiki User
HPham (WMF) [ Global Accounts ]
This account has been disabled.

Recent Activity

Mar 9 2021

Phamhi lowered the priority of T255787: Reconcile and/or understand differences between cloud-vps and prod hiera lookups from High to Medium.
Mar 9 2021, 5:49 PM · Cloud-VPS, User-jbond, cloud-services-team (Kanban)
Phamhi added a comment to T266777: integration instances suffer from high IO latency due to Ceph.

Hi @hashar,
Can this ticket be closed?

Mar 9 2021, 5:46 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure
Phamhi lowered the priority of T275407: Investigate why debian 10.0 image got corrupted from High to Low.
Mar 9 2021, 5:41 PM · cloud-services-team (Kanban)
Phamhi triaged T275024: Toolforge: Update go runtime as Lowest priority.
Mar 9 2021, 5:38 PM · Toolforge (Software install/update), cloud-services-team (Kanban)
Phamhi merged T275512: webservice --backend=kubernetes python3.7 restart returns requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='k8s.tools.eqiad1.wikimedia.cloud', port=6443): Read timed out. into T267966: Try to squeeze better performance out of k8s-etcd nodes.
Mar 9 2021, 5:35 PM · cloud-services-team (Kanban), Toolforge
Phamhi merged task T275512: webservice --backend=kubernetes python3.7 restart returns requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='k8s.tools.eqiad1.wikimedia.cloud', port=6443): Read timed out. into T267966: Try to squeeze better performance out of k8s-etcd nodes.
Mar 9 2021, 5:35 PM · cloud-services-team (Kanban), Toolforge
Phamhi triaged T275555: [toolforge k8s] Support Cinder volumes as Medium priority.
Mar 9 2021, 5:32 PM · cloud-services-team (Kanban), Kubernetes, Toolforge
Phamhi triaged T275556: Move ceph keyring/caps management into puppet if possible as Medium priority.
Mar 9 2021, 5:31 PM · cloud-services-team (Kanban)
Phamhi edited projects for T275605: cloudmetrics1002: mysterious issue, added: cloud-services-team (Hardware); removed cloud-services-team (Kanban).
Mar 9 2021, 5:28 PM · cloud-services-team (Hardware)
Phamhi closed T275703: [Toolforge] Reading dumps is very slow as Declined.
Mar 9 2021, 5:27 PM · cloud-services-team (Kanban), Kubernetes, Toolforge
Phamhi added a comment to T275703: [Toolforge] Reading dumps is very slow.

This is working as designed. We may revisit the throttle in the future: T218338

Mar 9 2021, 5:27 PM · cloud-services-team (Kanban), Kubernetes, Toolforge
Phamhi edited projects for T275920: icinga login case mismatch, added: SRE; removed cloud-services-team (Kanban).
Mar 9 2021, 5:23 PM · observability, SRE, Icinga
Phamhi moved T276018: Investigate new roles and policies in openstack train from Inbox to Doing on the cloud-services-team (Kanban) board.
Mar 9 2021, 5:22 PM · Cloud-VPS, cloud-services-team (Kanban)
Phamhi triaged T276060: Please detach/unlink newly created LDAP from my WM account as Medium priority.
Mar 9 2021, 5:21 PM · LDAP, cloud-services-team (Kanban)
Phamhi updated subscribers of T276060: Please detach/unlink newly created LDAP from my WM account.

Hi @bd808 ,
Do you by any chance know how to do this?

Mar 9 2021, 5:20 PM · LDAP, cloud-services-team (Kanban)
Phamhi moved T276102: broken puppet on deployment-sca01.deployment-prep.eqiad1.wikimedia.cloud and deployment-sca02.deployment-prep.eqiad1.wikimedia.cloud from Inbox to Watching on the cloud-services-team (Kanban) board.
Mar 9 2021, 5:19 PM · cloud-services-team (Kanban), Beta-Cluster-Infrastructure
Phamhi moved T276122: broken puppet on deployment-eventlog05.deployment-prep.eqiad.wmflabs from Inbox to Watching on the cloud-services-team (Kanban) board.
Mar 9 2021, 5:19 PM · Beta-Cluster-Infrastructure, cloud-services-team (Kanban)
Phamhi moved T276171: Restore access to the tool srwiki-dev on Toolforge from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.
Mar 9 2021, 5:17 PM · cloud-services-team (Kanban), Toolforge
Phamhi placed T276171: Restore access to the tool srwiki-dev on Toolforge up for grabs.
Mar 9 2021, 5:17 PM · cloud-services-team (Kanban), Toolforge
Phamhi triaged T276171: Restore access to the tool srwiki-dev on Toolforge as Low priority.
Mar 9 2021, 5:17 PM · cloud-services-team (Kanban), Toolforge
Phamhi moved T276284: Establish a working setup for PAWS with multi-instance wikireplicas from Inbox to Doing on the cloud-services-team (Kanban) board.
Mar 9 2021, 5:16 PM · PAWS, Patch-For-Review, Data-Services, cloud-services-team (Kanban)
Phamhi triaged T276284: Establish a working setup for PAWS with multi-instance wikireplicas as High priority.
Mar 9 2021, 5:15 PM · PAWS, Patch-For-Review, Data-Services, cloud-services-team (Kanban)
Phamhi added a comment to T276626: Toolforge: Automatically identify when people are trying to launch a Python webservice with a venv created using the wrong Python version.

Checked documentation and can confirm that the steps are accurate.

Mar 9 2021, 5:14 PM · cloud-services-team (Kanban), Toolforge
Phamhi triaged T276626: Toolforge: Automatically identify when people are trying to launch a Python webservice with a venv created using the wrong Python version as Low priority.
Mar 9 2021, 5:14 PM · cloud-services-team (Kanban), Toolforge
Phamhi triaged T276792: Remove cloud contacts from legacy paging as Low priority.
Mar 9 2021, 5:10 PM · cloud-services-team (Kanban), User-fgiunchedi, observability
Phamhi moved T276792: Remove cloud contacts from legacy paging from Inbox to Doing on the cloud-services-team (Kanban) board.
Mar 9 2021, 5:10 PM · cloud-services-team (Kanban), User-fgiunchedi, observability
Phamhi triaged T276963: Horizon: add doc links and discouragement to the 'server groups' UIs as Medium priority.
Mar 9 2021, 5:07 PM · Patch-For-Review, Horizon, cloud-services-team (Kanban)
Phamhi triaged T276964: cloudvirt2003-dev: neutron-linuxbridge-agent and nova-compute issues as Low priority.
Mar 9 2021, 5:07 PM · cloud-services-team (Kanban)
Phamhi moved T276964: cloudvirt2003-dev: neutron-linuxbridge-agent and nova-compute issues from Inbox to Doing on the cloud-services-team (Kanban) board.
Mar 9 2021, 5:06 PM · cloud-services-team (Kanban)

Feb 24 2021

Phamhi added a comment to T275489: Request increased quota for wikisource Cloud VPS project.

Discussed on WMCS meeting and approved

Feb 24 2021, 4:22 PM · Community-Tech, WS Export, Cloud-VPS (Quota-requests)
Phamhi added a comment to T275519: Request increased quota for monitoring Cloud VPS project.

Discussed on WMCS meeting and approved

Feb 24 2021, 4:20 PM · Cloud-VPS (Quota-requests)

Feb 19 2021

Phamhi closed T274400: Request creation of research-collaborations-api VPS project as Resolved.
Feb 19 2021, 12:50 PM · Cloud-VPS (Project-requests)
Phamhi added a comment to T274400: Request creation of research-collaborations-api VPS project.

Project research-collaborations-api has been created.

Feb 19 2021, 12:48 PM · Cloud-VPS (Project-requests)

Feb 17 2021

Phamhi claimed T265781: Upgrade Toolforge buildpacks to api 0.5.
Feb 17 2021, 11:20 AM · cloud-services-team (Kanban), Toolforge

Feb 5 2021

Phamhi triaged T274002: User reports getting 500's error on on https://k8s-status.toolforge.org as Medium priority.
Feb 5 2021, 7:34 PM · Tools, cloud-services-team (Kanban)
Phamhi created T274002: User reports getting 500's error on on https://k8s-status.toolforge.org.
Feb 5 2021, 7:33 PM · Tools, cloud-services-team (Kanban)

Feb 3 2021

Phamhi moved T273746: Toolforge: 404 handler fourohfour returns 504 from Inbox to Doing on the cloud-services-team (Kanban) board.
Feb 3 2021, 2:26 PM · cloud-services-team (Kanban)
Phamhi claimed T273746: Toolforge: 404 handler fourohfour returns 504.
Feb 3 2021, 2:00 PM · cloud-services-team (Kanban)

Dec 20 2019

Phamhi added a comment to T224585: Migrate labmon* to Buster.

Hi @Ladsgroup, noted; I apologize for the inconvenience. I have updated the docs. Please let me know if you need me on anything.

Dec 20 2019, 3:51 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Dec 20 2019, 3:46 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi added a comment to T241197: cloudmetrics1001 paged for systemd-timedated failure.

I think that was my doing. I was exploding the carbon.tar file (transferred from 02) on /srv and it ran out of space. It's been fixed though.

Dec 20 2019, 3:42 PM · cloud-services-team (Kanban)
Phamhi added a comment to T241197: cloudmetrics1001 paged for systemd-timedated failure.

I investigated a bit this morning. I didn't find anything useful regarding the kernel issue but for the systemd-timedated issue seems to be subsided. The issue lasted for about 30min ish, from 01:47:52 to 02:21:46 UTC.

Dec 20 2019, 3:34 PM · cloud-services-team (Kanban)

Dec 19 2019

Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Dec 19 2019, 5:25 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi created T241155: Relabel labmon1001.eqiad.wmnet to cloudmetrics1001eqiad.wmnet and labmon1002.eqiad.wmnet to cloudmetrics1002eqiad.wmnet.
Dec 19 2019, 5:23 PM · DC-Ops, ops-eqiad, SRE, cloud-services-team
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Dec 19 2019, 4:44 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Dec 19 2019, 4:04 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Dec 19 2019, 4:03 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE

Dec 17 2019

Phamhi added a comment to T224585: Migrate labmon* to Buster.

It looks like cloudmetrics1002 IP 10.64.4.15 is already in the ACL. We will try https://gerrit.wikimedia.org/r/c/operations/puppet/+/558506 again

Dec 17 2019, 1:20 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE

Dec 16 2019

jcrespo awarded T240456: Add cloudmetrics1002 to network devices ACL a Like token.
Dec 16 2019, 8:50 AM · SRE, netops

Dec 13 2019

Phamhi added a comment to T224585: Migrate labmon* to Buster.

The package rssh is still being used in labmon servers to restrict the type of shell and/or commands can been invoked during rsync from primary to standby server. There's a few problems with this.

Dec 13 2019, 8:28 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE

Dec 12 2019

Phamhi added a comment to T240456: Add cloudmetrics1002 to network devices ACL.

Hi @jcrespo , if he or she comes back early next week then it should be fine.

Dec 12 2019, 6:07 PM · SRE, netops

Dec 11 2019

Phamhi added a comment to T224585: Migrate labmon* to Buster.

As part of the failover test from labmon1001 to cloundmetrics1002, we have discovered that cloudmetrics1002's 10.64.4.15 IP is missing in the network devices ACL. Ticket https://phabricator.wikimedia.org/T240456 has been created to remedy this issue.

Dec 11 2019, 3:00 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi created T240456: Add cloudmetrics1002 to network devices ACL.
Dec 11 2019, 2:18 PM · SRE, netops

Dec 9 2019

Phamhi added a comment to T230961: Add Python 3.7 to Toolforge Kubernetes.

What's your tool's account?

Dec 9 2019, 6:19 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge (Software install/update)
Phamhi added a comment to T230961: Add Python 3.7 to Toolforge Kubernetes.

Hi @RoySmith , our bastion servers are on debian stretch which only has up to Python 3.5; we would have to upgrade to Buster which is not on our roadmap so I wouldn't be holding my breath if I were you.

Dec 9 2019, 3:55 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge (Software install/update)
Phamhi added a comment to T230961: Add Python 3.7 to Toolforge Kubernetes.

Hi @RoySmith. Python 3.7 can be found if you start your webservice using kubernetes backend.

Dec 9 2019, 3:26 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge (Software install/update)
Phamhi added a comment to T228866: CloudVPS: VMs with broken puppet 2019-07-14.

On 2019-12-09:

Dec 9 2019, 1:29 PM · cloud-services-team (Kanban)
Phamhi closed T230961: Add Python 3.7 to Toolforge Kubernetes as Resolved.

Python 3.7 support has been added.

Dec 9 2019, 1:24 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge (Software install/update)
Phamhi renamed T230961: Add Python 3.7 to Toolforge Kubernetes from Install a version of Python newer than 3.5.3 in Toolforge to Add Python 3.7 in Toolforge.
Dec 9 2019, 1:23 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge (Software install/update)

Dec 3 2019

Phamhi added a comment to T224585: Migrate labmon* to Buster.

Confirmed that the latest patch 554115 fixed the graphite-web issue.

Dec 3 2019, 2:59 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE

Dec 2 2019

Phamhi closed T235743: Prepare and check storage layer for mnwwiki, a subtask of T235739: Create Mon Wikipedia, as Resolved.
Dec 2 2019, 10:07 PM · User-Ladsgroup, User-Urbanecm, Wiki-Setup (Create)
Phamhi closed T235743: Prepare and check storage layer for mnwwiki as Resolved.

The views step is done. Tested successfully on toolsforge.

Dec 2 2019, 10:07 PM · cloud-services-team (Kanban), Data-Services, DBA
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Dec 2 2019, 9:03 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi added a comment to T224585: Migrate labmon* to Buster.

For new gerrit patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/554115, I have updated the following to fix the uwsgi-graphite-web.service issue:

Dec 2 2019, 6:10 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi closed T239601: Disk usage cleanup wikimaps-warper on cloudstore1008 as Invalid.

Not needed

Dec 2 2019, 3:36 PM · wikimaps-warper
Phamhi added a comment to T239601: Disk usage cleanup wikimaps-warper on cloudstore1008.

Hi wikimap-warper team

Dec 2 2019, 3:35 PM · wikimaps-warper
Phamhi renamed T239601: Disk usage cleanup wikimaps-warper on cloudstore1008 from Disk usage cleanup to Disk usage cleanup wikimaps-warper on cloudstore1008.
Dec 2 2019, 1:13 PM · wikimaps-warper
Phamhi created T239601: Disk usage cleanup wikimaps-warper on cloudstore1008.
Dec 2 2019, 1:09 PM · wikimaps-warper

Nov 28 2019

Phamhi added a comment to T224585: Migrate labmon* to Buster.

I am looking into a recently discovered issue with uwsgi-graphite-web.service

Nov 28 2019, 8:15 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Nov 28 2019, 4:39 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Nov 28 2019, 4:36 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Nov 28 2019, 4:20 AM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Nov 28 2019, 4:14 AM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Nov 28 2019, 4:13 AM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE

Nov 27 2019

Phamhi moved T228866: CloudVPS: VMs with broken puppet 2019-07-14 from Clinic Duty to Doing on the cloud-services-team (Kanban) board.
Nov 27 2019, 4:16 PM · cloud-services-team (Kanban)
Phamhi claimed T228866: CloudVPS: VMs with broken puppet 2019-07-14.

On 2019-11-27:

Nov 27 2019, 4:16 PM · cloud-services-team (Kanban)

Nov 26 2019

Phamhi added a comment to T224585: Migrate labmon* to Buster.
Nov 26 2019, 5:33 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE

Nov 25 2019

Phamhi moved T224585: Migrate labmon* to Buster from Doing to Needs discussion on the cloud-services-team (Kanban) board.
Nov 25 2019, 11:51 AM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi added a comment to T224585: Migrate labmon* to Buster.

The scope of this request has been extended to OS rename as well. New OS hostname will be determined.

Nov 25 2019, 11:50 AM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi updated the task description for T224585: Migrate labmon* to Buster.
Nov 25 2019, 11:50 AM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE

Nov 22 2019

Phamhi added a comment to T224585: Migrate labmon* to Buster.

As per suggestion, I have created different python files (no longer template) for different release.

Nov 22 2019, 5:07 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE

Nov 20 2019

Phamhi added a comment to T224585: Migrate labmon* to Buster.

The following changes have been made to be compatible with Buster:

Nov 20 2019, 6:30 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi added a comment to T224585: Migrate labmon* to Buster.

Renaming MIDDLEWARE_CLASSES to MIDDLEWARE seems to have fixed this problem.

Nov 20 2019, 11:50 AM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE

Nov 18 2019

Phamhi added a comment to T224585: Migrate labmon* to Buster.

I'm getting the following

Nov 18 2019, 9:04 PM · Cloud-VPS (Debian Jessie Deprecation), cloud-services-team (Kanban), SRE
Phamhi claimed T217294: Update maintain-kubeusers to occasionally (once per day?) verify the integrity of existing credentials.
Nov 18 2019, 1:05 PM · cloud-services-team (Kanban), Toolforge

Nov 15 2019

Phamhi closed T233347: Remove access.log generation from default lighttpd.conf generated by `webservice` as Resolved.

Completed

Nov 15 2019, 11:03 AM · cloud-services-team (Kanban), Toolforge

Nov 14 2019

Phamhi added a comment to T234656: Systems and service continuity and availability constraints.

Updated documentation again with nic teaming solution

Nov 14 2019, 10:36 PM · cloud-services-team (Kanban)
Phamhi added a comment to T234656: Systems and service continuity and availability constraints.

Asked cteam to review.

Nov 14 2019, 12:23 PM · cloud-services-team (Kanban)

Nov 13 2019

Phamhi closed T136194: wikidata-exports is using 256G in Tools, a subtask of T136212: Contact tool maintainters using large amounts of disk space, as Resolved.
Nov 13 2019, 2:20 PM · Goal, Toolforge, Cloud-Services
Phamhi closed T136194: wikidata-exports is using 256G in Tools as Resolved.
Nov 13 2019, 2:20 PM · cloud-services-team (Kanban), Tools, Wikidata
Phamhi changed the status of T136194: wikidata-exports is using 256G in Tools from Open to Stalled.

Closing this as the last update was 3 years ago.

Nov 13 2019, 2:20 PM · cloud-services-team (Kanban), Tools, Wikidata
Phamhi changed the status of T136194: wikidata-exports is using 256G in Tools, a subtask of T136212: Contact tool maintainters using large amounts of disk space, from Open to Stalled.
Nov 13 2019, 2:20 PM · Goal, Toolforge, Cloud-Services
Phamhi closed T237051: accesslog.filename makes lighttpd fail starting as Resolved.

Thanks for the confirmation.

Nov 13 2019, 1:59 PM · cloud-services-team (Kanban), Tools, Toolforge
Phamhi claimed T237051: accesslog.filename makes lighttpd fail starting.
Nov 13 2019, 1:59 PM · cloud-services-team (Kanban), Tools, Toolforge
Phamhi added a comment to T238193: WatchTranslations tool doesn't work after logged in.

Please use python3.5 as MySQLdb doesn't support python3.4 (python defaults to python3.4 if not specified) https://github.com/PyMySQL/mysqlclient-python/issues/342

Nov 13 2019, 1:01 PM · cloud-services-team (Kanban), WatchTranslations, Toolforge
Phamhi added a comment to T238193: WatchTranslations tool doesn't work after logged in.

Investigating...

Nov 13 2019, 12:34 PM · cloud-services-team (Kanban), WatchTranslations, Toolforge

Nov 12 2019

Phamhi closed T237768: Could not find dependency Package[python-yaml] error in profile::toolforge::grid::node::web as Resolved.
Nov 12 2019, 3:50 PM · cloud-services-team

Nov 8 2019

Phamhi added a comment to T237768: Could not find dependency Package[python-yaml] error in profile::toolforge::grid::node::web .

That makes sense..thanks for pointing it out Krenair.

Nov 8 2019, 9:55 PM · cloud-services-team
Phamhi created T237768: Could not find dependency Package[python-yaml] error in profile::toolforge::grid::node::web .
Nov 8 2019, 9:27 PM · cloud-services-team
Phamhi added a comment to T230961: Add Python 3.7 to Toolforge Kubernetes.

I tested these images and confirmed working.

Nov 8 2019, 3:57 PM · cloud-services-team (Kanban), Patch-For-Review, Toolforge (Software install/update)