Operations Engineer with the Wikimedia Cloud Services Team
Dec 20 2019
Hi @Ladsgroup, noted; I apologize for the inconvenience. I have updated the docs. Please let me know if you need me on anything.
I think that was my doing. I was exploding the carbon.tar file (transferred from 02) on /srv and it ran out of space. It's been fixed though.
I investigated a bit this morning. I didn't find anything useful regarding the kernel issue but for the systemd-timedated issue seems to be subsided. The issue lasted for about 30min ish, from 01:47:52 to 02:21:46 UTC.
Dec 19 2019
Dec 17 2019
It looks like cloudmetrics1002 IP 10.64.4.15 is already in the ACL. We will try https://gerrit.wikimedia.org/r/c/operations/puppet/+/558506 again
Dec 16 2019
Dec 13 2019
The package rssh is still being used in labmon servers to restrict the type of shell and/or commands can been invoked during rsync from primary to standby server. There's a few problems with this.
Dec 12 2019
Hi @jcrespo , if he or she comes back early next week then it should be fine.
Dec 11 2019
As part of the failover test from labmon1001 to cloundmetrics1002, we have discovered that cloudmetrics1002's 10.64.4.15 IP is missing in the network devices ACL. Ticket https://phabricator.wikimedia.org/T240456 has been created to remedy this issue.
Dec 9 2019
What's your tool's account?
Hi @RoySmith , our bastion servers are on debian stretch which only has up to Python 3.5; we would have to upgrade to Buster which is not on our roadmap so I wouldn't be holding my breath if I were you.
Hi @RoySmith. Python 3.7 can be found if you start your webservice using kubernetes backend.
Python 3.7 support has been added.
Dec 3 2019
Confirmed that the latest patch 554115 fixed the graphite-web issue.
Dec 2 2019
The views step is done. Tested successfully on toolsforge.
For new gerrit patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/554115, I have updated the following to fix the uwsgi-graphite-web.service issue:
Hi wikimap-warper team
Nov 28 2019
I am looking into a recently discovered issue with uwsgi-graphite-web.service
Nov 27 2019
Nov 26 2019
Nov 25 2019
The scope of this request has been extended to OS rename as well. New OS hostname will be determined.
Nov 22 2019
As per suggestion, I have created different python files (no longer template) for different release.
Nov 20 2019
The following changes have been made to be compatible with Buster:
Renaming MIDDLEWARE_CLASSES to MIDDLEWARE seems to have fixed this problem.
Nov 18 2019
I'm getting the following
Nov 15 2019
Nov 14 2019
Updated documentation again with nic teaming solution
Asked cteam to review.
Nov 13 2019
Closing this as the last update was 3 years ago.
Thanks for the confirmation.
Please use python3.5 as MySQLdb doesn't support python3.4 (python defaults to python3.4 if not specified) https://github.com/PyMySQL/mysqlclient-python/issues/342
Nov 12 2019
Nov 8 2019
That makes sense..thanks for pointing it out Krenair.
I tested these images and confirmed working.
Updating tools-webservice to include new buster images options to webservice: golang111, jdk11, php73, python37 and ruby25
Nov 6 2019
Updated documentation based on feedbacks from Arturo.
Nov 5 2019
All of the lighttpd based k8s pods have been restarted
I am looking for a sane way to restart all of the lighttpd powered webservices to force the change to take effect...
Nov 4 2019
I think our best bet is to set the max-size and max-file options for the json-file logging driver
Nov 2 2019
Nov 1 2019
Thanks for letting us know.
Oct 30 2019
Hey @Magnus ... we are just doing some ticket cleanup and noticed that the last update was back in March... and it looks like the ticket had been resolved. Is it ok if we close this ticket?
Oct 29 2019
Hey @Chicocvenancio ... I have manually force killed the jupyter--43riscod container on tools-paws-worker-1007. Let me know if you need anything else.
@Fae .... Try running it in one of the kubernetes python shell
The first draft of the documentation has been posted. Will ask the cteam for the initial review.
Oct 28 2019
This ticket is closed as part of scheduled clean-up. Please let us if this issue needs to be re-opened.
Marking as done as the documentation has met the requirement scopes of the request.
Oct 25 2019
Oct 24 2019
It's fixed.. I have admin to icinga now.
It looks like I still don't have admin access to icinga.. will attempt to update the cfg.cfg file
Oct 23 2019
The new version of toollabs-webservice package 0.47 has been pushed out to:
I am closing this issue as I have confirmed that kmlexport uses grid not kubernetes (since CGI.pm isn't available). As the service is a simple lighttp based serving a single kmlexport.pl file, this issue was most likely one-off.
Jephpaul responded to the email letting us know that they have actioned the ticket. Confirmed that /home/cosmiclattes now only occupies 16GB of disk space.
Oct 22 2019
Oct 21 2019
Hi @bd808 .. I think I got a grip on creating new debian package, uploading it to the repo and updating/push new docker images.. can we proceed with this step Merge and deploy the updated webservice code including Docker image rebuilds ?