zhuyifei1999
*Not* Serious business title.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Oct 13 2014, 10:19 AM (174 w, 6 d)
Availability
Available
IRC Nick
zhuyifei1999
LDAP User
Zhuyifei1999
MediaWiki User
Zhuyifei1999

There is currently no text in this page. You can search for this page title in other pages, or search the related logs.

Recent Activity

Today

zhuyifei1999 awarded T187659: Document vitualenv creation in Toolforge a Love token.
Sun, Feb 18, 9:12 PM · Toolforge
zhuyifei1999 placed T179986: Investigate why current es2r daemon is randomly hanging up for grabs.

Unlicking this- wasn't able to reproduce in a month's continuous running of three sseclients.

Sun, Feb 18, 7:54 PM · Wikimedia-Stream, Huggle
zhuyifei1999 added a comment to T187650: requests >= 2.9 should be installed for python.

The package is provided by APT from official ubuntu apt repository:

06:09:13 0 ✓ zhuyifei1999@tools-bastion-02: ~$ apt-cache policy python-requests
python-requests:
  Installed: 2.2.1-1ubuntu0.3
  Candidate: 2.2.1-1ubuntu0.3
  Version table:
 *** 2.2.1-1ubuntu0.3 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.2.1-1ubuntu0.2 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     2.2.1-1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

Upgrading is unlikely to happen, unless we upgrade the system itself to a newer release.

Sun, Feb 18, 6:22 PM · Toolforge
zhuyifei1999 added a comment to T187635: KeyError when log event content is hidden.

Do you have some minimum code to reproduce this issue?

Sun, Feb 18, 4:30 AM · Pywikibot-core
zhuyifei1999 added a comment to T185434: PAWS fails creating a server for new user .

However, I'm undergoing the same problems as Herzi pointed out one month ago. Any way to sort this out?

Sun, Feb 18, 12:19 AM · PAWS

Yesterday

zhuyifei1999 closed T185665: Add a way to navigate from a quarry output URL to the query as Invalid.

Marking as invalid. Feel free to reopen if someone can explain what is needed here.

Sat, Feb 17, 10:51 PM · Quarry
jcrespo awarded T175881: Redirect Toolforge Quarry page to Cloud VPS Quarry a Love token.
Sat, Feb 17, 6:40 PM · Toolforge, Quarry
zhuyifei1999 closed T175881: Redirect Toolforge Quarry page to Cloud VPS Quarry as Resolved.
zhuyifei1999@zhuyifei1999-ThinkPad-X260:~$ curl -I http://tools.wmflabs.org/quarry/
HTTP/1.1 301 Moved Permanently
Server: nginx/1.13.6
Date: Sat, 17 Feb 2018 18:33:54 GMT
Connection: keep-alive
Location: https://quarry.wmflabs.org/
X-Clacks-Overhead: GNU Terry Pratchett
Sat, Feb 17, 6:34 PM · Toolforge, Quarry
zhuyifei1999 claimed T175881: Redirect Toolforge Quarry page to Cloud VPS Quarry.
Sat, Feb 17, 6:23 PM · Toolforge, Quarry
zhuyifei1999 merged T187626: https://tools.wmflabs.org/quarry/ is accessible but broken into T175881: Redirect Toolforge Quarry page to Cloud VPS Quarry.
Sat, Feb 17, 6:23 PM · Toolforge, Quarry
zhuyifei1999 merged task T187626: https://tools.wmflabs.org/quarry/ is accessible but broken into T175881: Redirect Toolforge Quarry page to Cloud VPS Quarry.
Sat, Feb 17, 6:23 PM · Quarry

Fri, Feb 16

zhuyifei1999 added a comment to T184862: Commons Localization - multiple languages.
In T184862#3977622, Liuxinyu970226 wrote:
  1. "Browse..." button
  2. No file selected.

Both implemented by browser. A sane browser should translate the text internally. Yes it may possible to override it using some JS input overhaul like OOJS-UI, but to do so when the browser already does the i18n is reinventing the wheel.

  1. dropdown box which says "Standard"

MediaWiki:Edittools explicitly stated against translation for a very good reason. If you feel that the benefit of having a translation outweigh the costs, see T184862#3899174.

  1. Categories (+)

MediaWiki:Gadget-HotCat.js line 79. Translations welcome.

Fri, Feb 16, 4:58 AM · Commons

Thu, Feb 15

Restricted Application added a project to T29917: Keep image links alive after move: Commons.
Thu, Feb 15, 3:40 PM · Commons, Multimedia, MediaWiki-File-management

Tue, Feb 13

zhuyifei1999 added a project to T187153: BadMethodCallException (mRecord->getContent() when mRecord is null) when viewing details or examine of Abuselog of Abusefilter 131 on zh.wikipedia: Wikimedia-log-errors.
Tue, Feb 13, 4:50 AM · Anti-Harassment, Regression, Multi-Content-Revisions, User-Addshore, Wikimedia-log-errors, Chinese-Sites, AbuseFilter
zhuyifei1999 added a comment to T182604: tools-static is throwing space warnings due to cdnjs git repo size.

Is there a way to enable quick link copying instead of having users to select the uri then ctrl-c manually? I'd say even adding an <a> would be an improvement. The less mouse moves, the better.

Tue, Feb 13, 3:17 AM · Patch-For-Review, Toolforge

Mon, Feb 12

zhuyifei1999 added a comment to T186552: Create alerts for bastion hosts - Usage and latency.

Do we have a number of cores variable that we could use in the config? Apparently total_cpu is not a thing in graphite right now.

Mon, Feb 12, 10:48 PM · Cloud-VPS, Toolforge
zhuyifei1999 updated subscribers of T176894: API query action=query&prop=imageinfo&iiprop=url&iiurlwidth=120&iiurlheight=120 returns urlparamnormal error on Commons File:Mohave_County_Arizona_Test.svg.

@Perhelion See the merged task. It's not practical for VFC to trial and error on the list of images to process to get as much information as it can get. Suggestions?

Mon, Feb 12, 7:37 PM · MediaWiki-API, Commons
zhuyifei1999 merged T187016: VFS on Commons: API request failed into T176894: API query action=query&prop=imageinfo&iiprop=url&iiurlwidth=120&iiurlheight=120 returns urlparamnormal error on Commons File:Mohave_County_Arizona_Test.svg.
Mon, Feb 12, 7:31 PM · MediaWiki-API, Commons
zhuyifei1999 merged task T187016: VFS on Commons: API request failed into T176894: API query action=query&prop=imageinfo&iiprop=url&iiurlwidth=120&iiurlheight=120 returns urlparamnormal error on Commons File:Mohave_County_Arizona_Test.svg.
Mon, Feb 12, 7:31 PM · Commons
zhuyifei1999 added a comment to T187016: VFS on Commons: API request failed.

From the network debugging info, I see a failed API query. Request data:

action	query
cllimit	500
clprop	hidden
format	json
iiprop	url|size|metadata
iiurlheight	120
iiurlwidth	120
inprop	talkid
prop	imageinfo|info|revisions|categories
rvprop	timestamp|content
titles	File:Kanhaiya+-+16.jpeg|File:Kanhaiya+-+2.jpeg|File:Kanhaiya+-+13.jpeg|File:Kanhaiya+-+8.jpeg|File:Kanhaiya+-+5.jpeg|File:Kanhaiya+-+1.jpeg|File:Kanhaiya+-+18.jpeg|File:Kanhaiya+-+11.jpeg|File:Kanhaiya+-+10.jpeg|File:Giroudhpuri+2.jpg
Mon, Feb 12, 7:29 PM · Commons
zhuyifei1999 added a comment to T182604: tools-static is throwing space warnings due to cdnjs git repo size.

Waiting for https://tools.wmflabs.org/cdnjs-beta/ :)

Mon, Feb 12, 7:17 PM · Patch-For-Review, Toolforge

Sat, Feb 10

zhuyifei1999 added a comment to T185314: Toolforge crontab command needs to validate output.

Please, someone check my latest crontab file if it really BOM-less now:
https://tools.wmflabs.org/ato/scripts/crontab.v3.06.txt
I try to avoid any further troubles.

Sat, Feb 10, 11:01 PM · Patch-For-Review, Toolforge, cloud-services-team

Fri, Feb 9

zhuyifei1999 added a comment to T185314: Toolforge crontab command needs to validate output.

Sorry guys. I was it again yesterday with my f* crontab file what I updated but I can not find and kill the unicode characters in it.
Could someone help me in my actual crontab file below, please?
https://tools.wmflabs.org/ato/scripts/crontab.v3.05.txt

If it works I will delete and hopefully forget my previous versions.

Fri, Feb 9, 9:46 PM · Patch-For-Review, Toolforge, cloud-services-team
zhuyifei1999 added a comment to T186924: Network error from grid with mono bot.

After reading some relevant code of the bot, then googling 'WebException PostDataAndGetResult' I found T150099.

Fri, Feb 9, 9:34 PM · Tools, Toolforge
zhuyifei1999 added a comment to T186924: Network error from grid with mono bot.

What's the minimum amount of code that is able to reproduce this error?

Fri, Feb 9, 9:13 PM · Tools, Toolforge
zhuyifei1999 added a project to T186924: Network error from grid with mono bot: Toolforge.
Fri, Feb 9, 8:59 PM · Tools, Toolforge
zhuyifei1999 added a comment to T185314: Toolforge crontab command needs to validate output.

I ran another puppet run:

[...]
Debug: Caching connection for https://tools-puppetmaster-01.tools.eqiad.wmflabs:8140
Debug: Executing: 'diff -u /var/lib/puppet/lib/puppet/parser/functions/get_clusters.rb /tmp/puppet-file20180209-17025-1gxah7s'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/get_clusters.rb]/content: 
--- /var/lib/puppet/lib/puppet/parser/functions/get_clusters.rb	2017-11-15 15:39:41.957269456 +0000
+++ /tmp/puppet-file20180209-17025-1gxah7s	2018-02-09 17:06:50.756084740 +0000
@@ -24,7 +24,7 @@
   newfunction(:get_clusters, :type => :rvalue) do |args|
     all = {}
     # Ganglia config is the source of truth about clusters/site
-    cluster_config = call_function(:hiera, ['ganglia_clusters', {}])
+    cluster_config = call_function(:hiera, ['wikimedia_clusters', {}])
Fri, Feb 9, 5:17 PM · Patch-For-Review, Toolforge, cloud-services-team
zhuyifei1999 added a comment to T185314: Toolforge crontab command needs to validate output.

I backed up the crontab with crontab -l > crontab-T185314-backup, and had the BOM stripped:

tools.ato@tools-bastion-05:~$ crontab -l | tail -c +4 | xxd | head
0000000: 5041 5448 3d2f 7573 722f 6c6f 6361 6c2f  PATH=/usr/local/
0000010: 6269 6e3a 2f75 7372 2f62 696e 3a2f 6269  bin:/usr/bin:/bi
0000020: 6e0a 3020 2a20 2a20 2a20 2a20 6a73 7562  n.0 * * * * jsub
0000030: 202d 6c20 7265 6c65 6173 653d 7472 7573   -l release=trus
0000040: 7479 202d 6f6e 6365 202d 4e20 737a 7562  ty -once -N szub
0000050: 6373 6f6e 6b20 2d6d 656d 2035 3132 6d20  csonk -mem 512m 
0000060: 2d6f 202f 6461 7461 2f70 726f 6a65 6374  -o /data/project
0000070: 2f61 746f 2f70 7562 6c69 635f 6874 6d6c  /ato/public_html
0000080: 2f6c 6f67 2f73 7a75 6263 736f 6e6b 2e74  /log/szubcsonk.t
0000090: 7874 202d 6a20 7920 2f64 6174 612f 7072  xt -j y /data/pr
tools.ato@tools-bastion-05:~$ crontab -l | tail -c +4 | crontab
Fri, Feb 9, 5:06 PM · Patch-For-Review, Toolforge, cloud-services-team
zhuyifei1999 added a comment to T185314: Toolforge crontab command needs to validate output.

Actually, this is interesting:

tools.ato@tools-bastion-05:~$ crontab -l | xxd | head
0000000: efbb bf50 4154 483d 2f75 7372 2f6c 6f63  ...PATH=/usr/loc
0000010: 616c 2f62 696e 3a2f 7573 722f 6269 6e3a  al/bin:/usr/bin:
0000020: 2f62 696e 0a30 202a 202a 202a 202a 206a  /bin.0 * * * * j
0000030: 7375 6220 2d6c 2072 656c 6561 7365 3d74  sub -l release=t
0000040: 7275 7374 7920 2d6f 6e63 6520 2d4e 2073  rusty -once -N s
0000050: 7a75 6263 736f 6e6b 202d 6d65 6d20 3531  zubcsonk -mem 51
0000060: 326d 202d 6f20 2f64 6174 612f 7072 6f6a  2m -o /data/proj
0000070: 6563 742f 6174 6f2f 7075 626c 6963 5f68  ect/ato/public_h
0000080: 746d 6c2f 6c6f 672f 737a 7562 6373 6f6e  tml/log/szubcson
0000090: 6b2e 7478 7420 2d6a 2079 202f 6461 7461  k.txt -j y /data
>>> '\xef\xbb\xbf'.decode('utf-8')
u'\ufeff'

U+FEFF. BOM!

Fri, Feb 9, 4:47 PM · Patch-For-Review, Toolforge, cloud-services-team
zhuyifei1999 added a comment to T185314: Toolforge crontab command needs to validate output.

The environment variable setting is perfectly valid crontab and is supposed to work (the regex in puppet, source of error message), but somehow it isn't. No idea why.

Fri, Feb 9, 4:44 PM · Patch-For-Review, Toolforge, cloud-services-team
zhuyifei1999 updated subscribers of T185314: Toolforge crontab command needs to validate output.
Fri, Feb 9, 4:37 PM · Patch-For-Review, Toolforge, cloud-services-team
zhuyifei1999 added a comment to T186830: tools/meta/accounteligibility: notice of heavy use till the end of February.

FWIW, the host the webservice is running on seems to be overloaded:

02:48:24 0 ✓ zhuyifei1999@tools-bastion-02: ~$ qstat -u tools.meta -xml | grep queue_name
      <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1421.tools.eqiad.wmflabs</queue_name>
[...]
02:48:34 0 ✓ zhuyifei1999@tools-webgrid-lighttpd-1421: ~$ cat /proc/loadavg 
4.10 4.12 4.13 7/673 9263
02:48:48 0 ✓ zhuyifei1999@tools-webgrid-lighttpd-1421: ~$ cat /proc/cpuinfo | grep ^processor | wc -l
4
Fri, Feb 9, 2:50 AM · Cloud-Services, cloud-services-team

Thu, Feb 8

zhuyifei1999 added a comment to T185784: Disallow uploading unwanted files to Phabricator (WP Zero abuse?).

Maroc Telecom is one of the ISPs that the on-wiki abusers originate from. Is it zero-rated? If so, asn't they been disabled from uploading in T168142#3466615?

Thu, Feb 8, 2:23 AM · Zero, Phabricator

Wed, Feb 7

zhuyifei1999 added a comment to T185452: Write installation instructions for the tool.

$ celery -A threed2commons.backend.worker worker --loglevel=info --detach --logfile /path/to/logs/celery.log

Wed, Feb 7, 4:13 PM · 3D2Commons, User-Sebastian_Berlin-WMSE

Tue, Feb 6

zhuyifei1999 closed T122035: Need to access revision histories of wikipedia pages as Invalid.

Closing 2+-year-old task that do not explain what is requested or what is wrong. See https://www.mediawiki.org/wiki/How_to_report_a_bug

Tue, Feb 6, 9:59 PM · MediaWiki-Revision-deletion, Cloud-Services
zhuyifei1999 awarded T90509: Search or filter queries by title or summary a Love token.
Tue, Feb 6, 5:07 PM · Quarry

Mon, Feb 5

zhuyifei1999 added a comment to T186565: PHP Notice: ob_end_flush(): failed to delete and flush buffer. No buffer to delete or flush on Special:UploadStash.

ob_start is also added in the same patch in the same function. The buffer shouldn't just be gone.

Mon, Feb 5, 10:36 PM · Multimedia, MediaWiki-File-management, Commons, MW-1.30-release
zhuyifei1999 added a comment to T186552: Create alerts for bastion hosts - Usage and latency.

IO alerts

I think this might exist already? Not sure if it actually works though. When I notice a host having high NFS lag it's usually 'puppet staleness' errors instead.

Mon, Feb 5, 9:00 PM · Cloud-VPS, Toolforge
zhuyifei1999 awarded T186552: Create alerts for bastion hosts - Usage and latency a Like token.
Mon, Feb 5, 8:49 PM · Cloud-VPS, Toolforge
zhuyifei1999 added a project to T186463: Uninstall Flow from Commons: Commons.
Mon, Feb 5, 2:23 AM · Commons, Patch-For-Review, User-Zoranzoki21, Wikimedia-Site-requests, StructuredDiscussions, Collaboration-Team-Triage

Sat, Feb 3

zhuyifei1999 committed rTVTCe9ea96eeb1e2: urlextract: add sanitation rule for titleblacklist-custom-double-apostrophe (authored by zhuyifei1999).
urlextract: add sanitation rule for titleblacklist-custom-double-apostrophe
Sat, Feb 3, 10:10 PM

Fri, Feb 2

zhuyifei1999 changed the status of T185665: Add a way to navigate from a quarry output URL to the query from Open to Stalled.
Fri, Feb 2, 10:56 PM · Quarry
zhuyifei1999 awarded T186255: [RFC] Allow 100 characters per line a Pterodactyl token.
Fri, Feb 2, 7:33 PM · Patch-For-Review, Pywikibot-General, Pywikibot-core, Pywikibot-RfCs

Thu, Feb 1

zhuyifei1999 committed rTVTCde326e05542b: urlextract: delete \u200b during sanitation (authored by zhuyifei1999).
urlextract: delete \u200b during sanitation
Thu, Feb 1, 11:32 PM
zhuyifei1999 added a comment to T186247: Hebrew Wikivoyage (via Tool "wikivoyage") loads assets by default from third-party sites.

Digging the history led me to https://meta.wikimedia.org/wiki/Wikivoyage/Lounge/Archive/2014-11#Maps_loading_external_resources

Thu, Feb 1, 8:25 PM · Collaboration-Team-Triage (Collab-Team-This-Quarter), Vuln-Infoleak, Community-Liaisons, Collaboration-Feature-Rollouts (Collaboration-Maps), Discovery, Privacy, Toolforge-standards-committee, Maps, WMF-Legal, Tools
zhuyifei1999 added a comment to T186247: Hebrew Wikivoyage (via Tool "wikivoyage") loads assets by default from third-party sites.

Also, where is the source code for this?

Thu, Feb 1, 7:54 PM · Collaboration-Team-Triage (Collab-Team-This-Quarter), Vuln-Infoleak, Community-Liaisons, Collaboration-Feature-Rollouts (Collaboration-Maps), Discovery, Privacy, Toolforge-standards-committee, Maps, WMF-Legal, Tools
zhuyifei1999 added a comment to T186247: Hebrew Wikivoyage (via Tool "wikivoyage") loads assets by default from third-party sites.

I do not know what NSM means

I just checked the code, N means layer "Traffic", S means layer "Hill shading", and Mmeans base layer "Mapnik". The layer chooser at the top right shows that all of these are externally hosted.

Thu, Feb 1, 7:29 PM · Collaboration-Team-Triage (Collab-Team-This-Quarter), Vuln-Infoleak, Community-Liaisons, Collaboration-Feature-Rollouts (Collaboration-Maps), Discovery, Privacy, Toolforge-standards-committee, Maps, WMF-Legal, Tools
zhuyifei1999 removed projects from T186247: Hebrew Wikivoyage (via Tool "wikivoyage") loads assets by default from third-party sites: WMF-Legal, Maps, Toolforge-standards-committee, Privacy, Discovery.

The poimap2 code is provided by Wikivoyage eV, updated on toolforge twice a week via crontab. I haven't touched this in years, and English Wikivoyage has since moved on to using the production maps service AFAIK. Mey2008 is probably the best person to contact about this.

Thu, Feb 1, 7:24 PM · Collaboration-Team-Triage (Collab-Team-This-Quarter), Vuln-Infoleak, Community-Liaisons, Collaboration-Feature-Rollouts (Collaboration-Maps), Discovery, Privacy, Toolforge-standards-committee, Maps, WMF-Legal, Tools

Wed, Jan 31

zhuyifei1999 added a comment to T186108: Changing shell preference tools.huji account.

From what I see, on a jessie cloud instance, chsh.ldap is from the nslcd-utils package. This package seems to be installed on all jessie and stretch instances during bootstrap. That would be quite a lot of instances on toolforge alone:

root@tools-worker-1027:~# which chsh.ldap
/usr/bin/chsh.ldap
Wed, Jan 31, 3:59 PM · Striker
zhuyifei1999 added a comment to T185561: weblinkchecker.py slows down (itself, OS) to freeze after a while reaching 100% of CPU usage.

PS: I don't know why there is this INFO:pywiki:, VERBOSE:pywiki: and WARNING:pywiki:. It starts correctly, but after some time of running it starts showing broken output:

Wed, Jan 31, 3:43 PM · Pywikibot-network, Pywikibot-weblinkchecker.py, Pywikibot-core
zhuyifei1999 claimed T167026: Add bash completion to sql.

I'll work on this once T181492 is done.

Wed, Jan 31, 5:31 AM · Patch-For-Review, Toolforge
Krinkle awarded T182451: Make qacct less NFS-intensive a Orange Medal token.
Wed, Jan 31, 4:49 AM · Toolforge
zhuyifei1999 merged T165763: Adds a table with namespaces on Tools Labs DB replica into T50625: Provide namespace IDs and names in the databases similar to toolserver.namespace.
Wed, Jan 31, 4:32 AM · User-bd808, Tools, Cloud-Services, Toolforge
zhuyifei1999 merged task T165763: Adds a table with namespaces on Tools Labs DB replica into T50625: Provide namespace IDs and names in the databases similar to toolserver.namespace.
Wed, Jan 31, 4:32 AM · Cloud-Services, Toolforge
zhuyifei1999 updated the task description for T182451: Make qacct less NFS-intensive.
Wed, Jan 31, 4:28 AM · Toolforge
zhuyifei1999 added a comment to T154527: Bigbrother doesn't restart wikilinkbot.

After T183171, has this been fixed for you?

Wed, Jan 31, 4:13 AM · Toolforge, Cloud-Services

Tue, Jan 30

zhuyifei1999 added a comment to T182604: tools-static is throwing space warnings due to cdnjs git repo size.
Tue, Jan 30, 10:49 PM · Patch-For-Review, Toolforge
zhuyifei1999 added a comment to T182604: tools-static is throwing space warnings due to cdnjs git repo size.

Tested a shallow bare clone:

 zhuyifei1999@gfg01  /srv/zhuyifei1999  git clone --bare --depth 1 https://github.com/cdnjs/cdnjs.git
Cloning into bare repository 'cdnjs.git'...
remote: Counting objects: 953009, done.
remote: Compressing objects: 100% (494914/494914), done.
remote: Total 953009 (delta 446410), reused 944437 (delta 445669), pack-reused 0
Receiving objects: 100% (953009/953009), 5.34 GiB | 18.30 MiB/s, done.
Resolving deltas: 100% (446410/446410), done.
Checking connectivity... done.
 zhuyifei1999@gfg01  /srv/zhuyifei1999  du -h cdnjs.git
44K	cdnjs.git/hooks
4.0K	cdnjs.git/refs/tags
4.0K	cdnjs.git/refs/heads
12K	cdnjs.git/refs
5.4G	cdnjs.git/objects/pack
4.0K	cdnjs.git/objects/info
5.4G	cdnjs.git/objects
8.0K	cdnjs.git/info
4.0K	cdnjs.git/branches
5.4G	cdnjs.git

Considering that a shallow bare clone is only 5.4G. Is it possible and good performance-wise to somehow mount the shallow clone?

Tue, Jan 30, 7:23 PM · Patch-For-Review, Toolforge
zhuyifei1999 added a comment to T182604: tools-static is throwing space warnings due to cdnjs git repo size.

I was working on the proxy-to-upstream method, some backwards-incompatibilities found:

Tue, Jan 30, 6:38 PM · Patch-For-Review, Toolforge
zhuyifei1999 added a comment to T182604: tools-static is throwing space warnings due to cdnjs git repo size.

More than half of the directories have a modification date of July 3 2017: P6646 (which I don't find anything special going on in the repo...)

Tue, Jan 30, 6:03 PM · Patch-For-Review, Toolforge
zhuyifei1999 added a comment to T182604: tools-static is throwing space warnings due to cdnjs git repo size.

I checked the sizes:

root@tools-static-11:/srv/cdnjs/ajax/libs# du -hd 1 | tee ~/srv-cdnjs-ajax-libs-du-hd-1
[...]
root@tools-static-11:/srv/cdnjs/ajax/libs# sort -h ~/srv-cdnjs-ajax-libs-du-hd-1 | tail
2.0G	./plotly.js
2.2G	./forerunnerdb
3.1G	./hola_player
3.8G	./antd
4.0G	./blackbaud-skyux
4.1G	./mathjax
5.3G	./material-design-icons
11G	./pdf.js
18G	./browser-logos
118G	.
root@tools-static-11:/srv/cdnjs/ajax/libs# df -h /srv
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vd-cdnjs--disk  139G  124G  8.0G  94% /srv
root@tools-static-11:/srv/cdnjs/ajax/libs# du ../../.git -sh
5.5G	../../.git

The majority of disk usage is cumulated from the smaller libraries, and there isn't a single 'directory-to-nuke' to gain lots of disk space. (and .git is only 5.5G so gc may not have much gain either). Unless we can find a way to de-duplicate and compress all the files, I guess T182604#3916214 is the way to go.

Tue, Jan 30, 5:01 PM · Patch-For-Review, Toolforge
zhuyifei1999 added a comment to T182604: tools-static is throwing space warnings due to cdnjs git repo size.

Why did this take such a sharp turn at the end of june?

Tue, Jan 30, 3:11 PM · Patch-For-Review, Toolforge
zhuyifei1999 created P6646 root@tools-static-11:/srv/cdnjs/ajax/libs# ls -al.
Tue, Jan 30, 3:09 PM
zhuyifei1999 added a comment to T186001: Pywikibot sends Authorization headers to upload.wikimedia.org.

PAWS sets authenticate OAuth tokens on all domains with wildcard *, and pywikibot sets the headers accordingly.

Tue, Jan 30, 2:38 PM · Pywikibot-Commons, Pywikibot-core

Mon, Jan 29

zhuyifei1999 claimed T185825: Pywikibot does not treat titleblacklist errors as a any sort of 'page not saved' or 'spam filter' errors, crashing replace.py.
Mon, Jan 29, 8:57 PM · Patch-For-Review, Pywikibot-replace.py, Pywikibot-General, Pywikibot-core
zhuyifei1999 added a comment to T178803: `qsub sync -y` jobs failing on Grid Engine with "range_list containes no elements" error.

For some reason it started working:

tools.zhuyifei1999-test@tools-bastion-05:~$ setup-tomcat
Setting up your public_tomcat directory...
All done.
You can edit the configuration in /data/project/zhuyifei1999-test/public_tomcat/conf/server.xml as needed.
Mon, Jan 29, 7:00 PM · cloud-services-team (Kanban), Toolforge, Wikisource

Sun, Jan 28

zhuyifei1999 changed the status of T185830: Change project parameter for Zh Wikinews and Es Wikipedia from Open to Stalled.
Sun, Jan 28, 5:59 PM · User-Zoranzoki21, Spanish-Sites, Chinese-Sites, Community-consensus-needed, Wikimedia-Site-requests
zhuyifei1999 created T185825: Pywikibot does not treat titleblacklist errors as a any sort of 'page not saved' or 'spam filter' errors, crashing replace.py.
Sun, Jan 28, 5:22 AM · Patch-For-Review, Pywikibot-replace.py, Pywikibot-General, Pywikibot-core
zhuyifei1999 created P6636 (An Untitled Masterwork).
Sun, Jan 28, 5:20 AM

Sat, Jan 27

zhuyifei1999 added a project to T185797: maps-wma1.maps.eqiad.wmflabs is down (wma.wmflabs.org): VPS-Projects.
Sat, Jan 27, 5:31 AM · cloud-services-team (Kanban), VPS-Projects

Fri, Jan 26

zhuyifei1999 placed T60784: Shell LOCALE neither consistent nor sane across grid engine nodes up for grabs.
Fri, Jan 26, 8:40 PM · Cloud-Services, Toolforge
zhuyifei1999 added a comment to T60784: Shell LOCALE neither consistent nor sane across grid engine nodes.

Initial tests looks good:

tools.zhuyifei1999-test@tools-bastion-02:~$ cat unitest.sh 
#! /bin/bash
exec 2>&1
set -x
Fri, Jan 26, 8:34 PM · Cloud-Services, Toolforge
zhuyifei1999 claimed T60784: Shell LOCALE neither consistent nor sane across grid engine nodes.

This can be fixed if -v LC_ALL=C.UTF-8 is a default argument to jsub. Let be see if this can break existing scripts.

Fri, Jan 26, 8:24 PM · Cloud-Services, Toolforge
zhuyifei1999 added a comment to T184906: Install 3d2commons on Toolforge.

@Sebastian_Berlin-WMSE I'm wondering, are these processes expected to be kept alive without a TTY attached?

07:26:54 0 ✓ zhuyifei1999@tools-bastion-03: ~$ ps uf -u sebastian-berlin-wmse
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
14617    26420  0.0  0.0  32420   608 ?        SNs  14:33   0:04 tail -f -n 100 /data/project/threed2commons/celery.log
14617    23731  0.0  0.0  32420   608 ?        SNs  14:29   0:04 tail -f -n 100 /data/project/threed2commons/celery.log
14617    22503  0.0  0.0  32420   608 ?        SNs  10:22   0:08 tail -f -n 100 /data/project/threed2commons/celery.log
14617    19499  0.0  0.0  32420   612 ?        SNs  Jan24   0:48 tail -f -n 100 /data/project/threed2commons/celery.log
14617     5870  0.0  0.0  32420   612 ?        SNs  Jan24   0:54 tail -f -n 100 /data/project/threed2commons/celery.log
Fri, Jan 26, 7:29 PM · User-Sebastian_Berlin-WMSE, 3D2Commons

Thu, Jan 25

zhuyifei1999 edited P6507 naughty_detector.py.
Thu, Jan 25, 5:13 PM
zhuyifei1999 added a comment to T184906: Install 3d2commons on Toolforge.

I'm having trouble getting the socket.io bit running.

Yeah, I should really have documented it... It's a combination of the main python frontend and https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#node.js_web_services, and must live on a separate tool than the main one, due to a tool can only run a webservice and it's written in a different language. (Well, I would have written it in python if it wasn't for a few inconsistent implementations of event loops and I have to choose among them...)

Thu, Jan 25, 5:02 PM · User-Sebastian_Berlin-WMSE, 3D2Commons
zhuyifei1999 merged Restricted Task into T50811: jsub breaks whitespaces in arguments.
Thu, Jan 25, 3:54 PM · Toolforge
zhuyifei1999 edited projects for T185673: Add base36 functions to ToolForge database, added: Data-Services; removed Toolforge.
Thu, Jan 25, 4:40 AM · Data-Services, DBA
zhuyifei1999 added a comment to T185647: generate_user_files.py: Report to user_config.py not found.

Disable form

Thu, Jan 25, 1:54 AM · Patch-For-Review, Pywikibot-Other-scripts, Pywikibot-core, Pywikibot-General

Wed, Jan 24

zhuyifei1999 added a comment to T185665: Add a way to navigate from a quarry output URL to the query.

I assume by 'the query' you mean something like https://quarry.wmflabs.org/query/1. Could you clarify 'quarry output URL'?

Wed, Jan 24, 10:52 PM · Quarry
zhuyifei1999 added a comment to T185647: generate_user_files.py: Report to user_config.py not found.

... which is an 'experimental', and probably not recommended, feature. T176354

Wed, Jan 24, 8:17 PM · Patch-For-Review, Pywikibot-Other-scripts, Pywikibot-core, Pywikibot-General
zhuyifei1999 added a comment to T185647: generate_user_files.py: Report to user_config.py not found.

The expected way to run them: python pwb.py <script> or python generate_user_files.py. pwb.py loads the user files and if it is not found it calls generate_user_files; if it is found it calls the script specified in the second argument. python pwb.py generate_user_files makes no sense to me.

Wed, Jan 24, 7:09 PM · Patch-For-Review, Pywikibot-Other-scripts, Pywikibot-core, Pywikibot-General
zhuyifei1999 merged task T179787: upload.wikimedia.org reports wrong mimetype for svg into Restricted Task.
Wed, Jan 24, 3:32 PM · Operations, media-storage

Tue, Jan 23

zhuyifei1999 added a comment to T185606: wikistream.wmflabs.org 502 Varnish Error.
11:00:52 1 ✗ zhuyifei1999@tools-bastion-02: ~$ curl -v ws-web.wikistream.eqiad.wmflabs
* Rebuilt URL to: ws-web.wikistream.eqiad.wmflabs/
* Hostname was NOT found in DNS cache
*   Trying 10.68.22.255...
* connect to 10.68.22.255 port 80 failed: Connection refused
* Failed to connect to ws-web.wikistream.eqiad.wmflabs port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to ws-web.wikistream.eqiad.wmflabs port 80: Connection refused

Is the service listening on eth0?

Tue, Jan 23, 11:02 PM · Cloud-VPS
zhuyifei1999 added a comment to T185577: Add zhuyifei1999_ to Toolforge roots.
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdm+Kiet0U2AjfPLFKIZLh57pXruVzlIfSypwSt7GRlZ+0sGhExYWvXdreW5zgZRJ3aXzR84qaZOqpwgHuVFlfMszXzq3E6p9dZ/bATP/lvIkucJJ2RL5C4ABvfwd4MHs68rW9UGwPVWn1qQftvoNpeDCu4t50uLbP9vcEXzaXrVioHROPoEpqk4YofHQS9F4HIKDbfjTFwUb5YQ/lsajuK0EZXlAELoaZ7s+uBolyBmXkIjWOjyGwHCoBU1FPiufLwLYuMZz0iAqw930dgoi89VgaPhlu12j2yP+2BZ3XkEgpzNBmZ/3lwVvRPyUnWw2pBU4qvs00FUARzx/WTRSJ zhuyifei1999@zhuyifei1999-ThinkPad-X260
Tue, Jan 23, 9:57 PM · Toolforge
zhuyifei1999 added a project to T185601: Request change of maintainer for gnubotmarcoo Tool: Tools.
Tue, Jan 23, 8:21 PM · Tools, Toolforge
zhuyifei1999 added a comment to T185561: weblinkchecker.py slows down (itself, OS) to freeze after a while reaching 100% of CPU usage.

Waiting for remaining 49 threads to finish, please wait...

https://github.com/wikimedia/pywikibot/blob/4ac64d7d88ff3049e178754566e0fa9b5d639edf/scripts/weblinkchecker.py#L1019

Tue, Jan 23, 3:39 PM · Pywikibot-network, Pywikibot-weblinkchecker.py, Pywikibot-core
zhuyifei1999 added a comment to T185550: Update Python 3.

Self-compilation is not recommended.

Why is that?

And why, if the new versions are backward compatible and do not contain old bugs.

Because we do not trust any other source of packages other than those we built ourselves and those provided by the official repo which periodically receives security backports as long as the version of the distro is still supported. If you install the packages from a random source the publisher of the package effectively get an rootkit on our servers.

Tue, Jan 23, 1:51 AM · Toolforge
zhuyifei1999 added a comment to T185550: Update Python 3.

TL;DR: You can:

  • wait for the entire toolforge to upgrade to a newer version of the distros
  • run your script in kubernetes which runs debian jessie and has a slightly newer python. (Apparently it's older not newer)
  • compile your own python for your tool
  • provide a super good reason to self-build python across whole toolforge that worth the cost in maintainability.

I'd recommend #3.

Tue, Jan 23, 1:47 AM · Toolforge
zhuyifei1999 merged T185550: Update Python 3 into T182820: Please upgrade python on the newest version on cloud vps.
Tue, Jan 23, 1:36 AM · Cloud-VPS
zhuyifei1999 merged task T185550: Update Python 3 into T182820: Please upgrade python on the newest version on cloud vps.
Tue, Jan 23, 1:36 AM · Toolforge

Mon, Jan 22

zhuyifei1999 added a comment to T179386: Prometheus crontab installations via Puppet failing on Toolforge bastion hosts due to remote crontab wrapper script.

I'm not sure if under the hood Puppet is doing the equivalent of sudo crontab -u prometheus ... or sudo -u prometheus crontab ...

Looks like the former.

Mon, Jan 22, 7:46 PM · Patch-For-Review, Toolforge, cloud-services-team
zhuyifei1999 added a comment to T179386: Prometheus crontab installations via Puppet failing on Toolforge bastion hosts due to remote crontab wrapper script.

It's currently determining system users by uid < 500, because of T45795, and executing local crontab via execv if that is the case. Some ways to fix / workaround this:

  • Make prometheus uid < 500. This would be a mess to adduser; find & chown on all cloud instances? If this could be done without any outages it would be awesome IMO.
  • Whitelist id -u prometheus => 14736 in crontab command so it considers prometheus to be system user. This would be the easiest short-term solution but I don't know if any future scripts will have to distinguish between system users and cloud users, and encounter the same issue.
  • Make root always access the local crontab instead of remote. The remote ssh may not work anyways according to the Permission denied (publickey,hostbased) above, but ideally if a root wants to read a normal could user's crontab they should not have to manually ssh into the cron host, and the crontab command should do it for the root.
Mon, Jan 22, 7:33 PM · Patch-For-Review, Toolforge, cloud-services-team
zhuyifei1999 closed T185314: Toolforge crontab command needs to validate output as Resolved.

The (continued) puppet-pometheus-crontab issue is tracked in T179386. The script should not spew out unicode errors anymore.

Mon, Jan 22, 7:07 PM · Patch-For-Review, Toolforge, cloud-services-team
zhuyifei1999 closed T181948: Toolforge crontab command cannot handle unicode crontab when the locale is non-utf8 as Resolved.
Mon, Jan 22, 7:04 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
zhuyifei1999 renamed T185507: SSH into all cloud instances, including toolforge, is not working from SSH into puppet-phabricator and puppet-paladox is not working to SSH into all cloud instances, including toolforge, is not working.
Mon, Jan 22, 4:55 PM · Cloud-Services
zhuyifei1999 added a comment to T181948: Toolforge crontab command cannot handle unicode crontab when the locale is non-utf8.
tools.zhuyifei1999-test@tools-bastion-05:~$ diff `which crontab` crontab.py 
253a254,257
>     # https://bugs.python.org/issue14156
>     if args.file is sys.stdin:
>         args.file = args.file.buffer
> 
tools.zhuyifei1999-test@tools-bastion-05:~$ crontab -r; echo '# 啊' | ./crontab.py -; crontab -l
# 啊
Mon, Jan 22, 4:38 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
zhuyifei1999 added a comment to T181948: Toolforge crontab command cannot handle unicode crontab when the locale is non-utf8.

Apparently argparse.FileType does not respect whether the mode string is binary or text. This is https://bugs.python.org/issue14156

Mon, Jan 22, 4:19 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
zhuyifei1999 added a comment to T181948: Toolforge crontab command cannot handle unicode crontab when the locale is non-utf8.

I got this:

tools.zhuyifei1999-test@tools-bastion-05:~$ crontab -r; echo '# 啊' | crontab -; crontab -l
Traceback (most recent call last):
  File "/usr/local/bin/crontab", line 345, in <module>
    main()
  File "/usr/local/bin/crontab", line 284, in main
    crontab.save(args.file.read())
  File "/usr/local/bin/crontab", line 130, in save
    jsub_text = self._add_jsub(text)
  File "/usr/local/bin/crontab", line 115, in _add_jsub
    text = text.decode('ascii', 'surrogateescape').split('\n')
AttributeError: 'str' object has no attribute 'decode'
no crontab for tools.zhuyifei1999-test
Mon, Jan 22, 3:59 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
zhuyifei1999 added a comment to T181948: Toolforge crontab command cannot handle unicode crontab when the locale is non-utf8.

From what I can tell from puppet's source code, puppet uses stdin, with - as its argument to specify stdin.
https://github.com/puppetlabs/puppet/blob/4.10.x/lib/puppet/util/filetype.rb#L204

Mon, Jan 22, 3:58 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge