bd808 (Bryan Davis)Administrator
Engineering Manager, Wikimedia Cloud Services

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:36 PM (219 w, 3 d)
Roles
Administrator
Availability
Busy Busy until Dec 18.
IRC Nick
bd808
LDAP User
BryanDavis
MediaWiki User
BDavis (WMF) [ Global Accounts ]

I'm BDavis (WMF) on wiki, bd808 on irc, and BryanDavis on Gerrit and Wikitech.

I've got a thing for 🦄s. Don't judge.

I work for or provide services to the Wikimedia Foundation, but this is my only Phabricator account. Edits, statements, or other contributions made from this account are my own, and may not reflect the views of the Foundation.

Recent Activity

Sat, Dec 15

bd808 added a comment to T204502: cloudvps: design project trusty deprecation.

Nothing has changed since my last comment, even with @bd808's hints. I have good feelings about what's in there, but really relieving would be to either connect to them and have a look (which I had several issues with already in the past and the credentials are probably vanished) or to get file/db backups.

Sat, Dec 15, 12:00 AM · Cloud-VPS (Ubuntu Trusty Deprecation)

Fri, Dec 14

bd808 added a comment to T210693: Create materialized views on Wiki Replica hosts for better query performance.

I'm not sure for the cloud users use case, you'd have to look at common or slow queries against the _compat views to decide.

Fri, Dec 14, 11:20 PM · Patch-For-Review, User-Banyek, Core Platform Team Backlog (Watching / External), Analytics-Kanban, DBA, Data-Services, Analytics
bd808 added a project to T212012: Create phab tag/form/workflow/something for volunteer priv requests: cloud-services-team (Kanban).
Fri, Dec 14, 8:54 PM · cloud-services-team (Kanban)
bd808 added a comment to T211523: Request creation of <Videowiki> VPS project.

One of the issues for @Videowiki123 (Hassan?) to be aware of is that per the Cloud Services Terms of use videowiki hosted on a Cloud VPS instance will need some prominent disclaimers about any authorship data that the videowiki project collects and/or stores. It should also be clear from the user interface that this is not an active content project, but rather a proof of concept for a proposed content project (or feature set to be added to an existing project).

Fri, Dec 14, 4:29 PM · Cloud-VPS (Project-requests)
bd808 added a comment to T199003: Develop timeline for Cloud VPS wide deprecation of Trusty.

Needs discussion: the final deadline is approaching: 2018-12-18. How to handle remaining Trusty VMs?

Fri, Dec 14, 4:24 PM · Cloud-VPS (Ubuntu Trusty Deprecation), cloud-services-team (Kanban), Goal
bd808 added a subtask for T210416: Upgrade grafana to 5.x: T211880: Upgrade grafana-labs.wikimedia.org to Grafana 5.x.
Fri, Dec 14, 3:35 AM · Performance-Team (Radar), Patch-For-Review, Operations, monitoring, User-CDanis
bd808 added a parent task for T211880: Upgrade grafana-labs.wikimedia.org to Grafana 5.x: T210416: Upgrade grafana to 5.x.
Fri, Dec 14, 3:35 AM · Cloud-Services

Thu, Dec 13

bd808 created T211940: maintain-views halt with fatal error when dropping a view fails.
Thu, Dec 13, 10:38 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services
bd808 closed T211939: Drop several views from srwikinews as Resolved.
$ ssh labsdb1009
$ sudo /usr/local/sbin/maintain-views --debug --clean --replace-all --database srwikinews
...
2018-12-13 22:21:43,708 INFO cleanup is enabled
2018-12-13 22:21:43,709 INFO cleaning 12 tables
2018-12-13 22:21:43,709 INFO Dropping view srwikinews_p.flaggedpage_config
2018-12-13 22:21:43,709 DEBUG SQL: drop view srwikinews_p.flaggedpage_config
2018-12-13 22:21:43,712 INFO Dropping view srwikinews_p.flaggedpage_pending
2018-12-13 22:21:43,712 DEBUG SQL: drop view srwikinews_p.flaggedpage_pending
2018-12-13 22:21:43,714 INFO Dropping view srwikinews_p.flaggedrevs_tracking
2018-12-13 22:21:43,714 DEBUG SQL: drop view srwikinews_p.flaggedrevs_tracking
2018-12-13 22:21:43,716 INFO Dropping view srwikinews_p.flaggedrevs_promote
2018-12-13 22:21:43,716 DEBUG SQL: drop view srwikinews_p.flaggedrevs_promote
2018-12-13 22:21:43,717 INFO Dropping view srwikinews_p.flaggedrevs_stats
2018-12-13 22:21:43,718 DEBUG SQL: drop view srwikinews_p.flaggedrevs_stats
2018-12-13 22:21:43,719 INFO Dropping view srwikinews_p.flaggedtemplates
2018-12-13 22:21:43,720 DEBUG SQL: drop view srwikinews_p.flaggedtemplates
2018-12-13 22:21:43,721 INFO Dropping view srwikinews_p.flaggedrevs_stats2
2018-12-13 22:21:43,721 DEBUG SQL: drop view srwikinews_p.flaggedrevs_stats2
2018-12-13 22:21:43,723 INFO Dropping view srwikinews_p.flaggedimages
2018-12-13 22:21:43,724 DEBUG SQL: drop view srwikinews_p.flaggedimages
2018-12-13 22:21:43,725 INFO Dropping view srwikinews_p.flaggedrevs_statistics
2018-12-13 22:21:43,726 DEBUG SQL: drop view srwikinews_p.flaggedrevs_statistics
2018-12-13 22:21:43,727 INFO Dropping view srwikinews_p.flaggedrevs
2018-12-13 22:21:43,727 DEBUG SQL: drop view srwikinews_p.flaggedrevs
2018-12-13 22:21:43,729 INFO Dropping view srwikinews_p.flaggedpages
2018-12-13 22:21:43,729 DEBUG SQL: drop view srwikinews_p.flaggedpages
2018-12-13 22:21:43,732 INFO Dropping view srwikinews_p.msg_resource_links
2018-12-13 22:21:43,732 DEBUG SQL: drop view srwikinews_p.msg_resource_links

Repeated on labsdb1010 and labsdb1011.

Thu, Dec 13, 10:32 PM · cloud-services-team (Kanban), Data-Services, User-Zoranzoki21
bd808 closed T211939: Drop several views from srwikinews, a subtask of T209251: Remove FlaggedRevs and add back rights autopatrolled, patroller (with enabled RCPatrol), rollbacker on srwikinews, as Resolved.
Thu, Dec 13, 10:32 PM · Wikimedia-Site-requests, User-Zoranzoki21
bd808 added a project to T211939: Drop several views from srwikinews: DBA.

If the extension is disabled and it is decided it won't be enabled again, there is no need to keep the tables and we should clean them up.
Once this is decided for sure, create a ticket for us to get them dropped.

Thanks

Thu, Dec 13, 10:11 PM · cloud-services-team (Kanban), Data-Services, User-Zoranzoki21
bd808 added a comment to T211906: Expose PHP7/HHVM to NavTiming in a header, send with navtiming beacon so we can use it as a dimension.

During the HHVM rollout, mw.config.get( 'wgPoweredByHHVM' ) would tell you if the page was being rendered via HHVM or not: rEWMV973c7340e75f: Inject 'wgPoweredByHHVM' JS config var if powered by HHVM

Thu, Dec 13, 9:44 PM · MediaWiki-extensions-NavigationTiming, Performance-Team

Wed, Dec 12

bd808 added a subtask for T210215: Quota usage not being counted properly in new region: T211718: Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues.
Wed, Dec 12, 4:34 PM · Cloud-VPS
bd808 added a parent task for T211718: Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues: T210215: Quota usage not being counted properly in new region.
Wed, Dec 12, 4:34 PM · Cloud-VPS
aborrero awarded T179463: Create a single application to provision and manage developer (LDAP) accounts a Like token.
Wed, Dec 12, 12:17 PM · LDAP, Operations, Developer-Advocacy, Cloud-Services

Tue, Dec 11

bd808 added a comment to T210215: Quota usage not being counted properly in new region.

Found upstream at https://bugs.launchpad.net/nova/+bug/1742826/comments/4:

There is a well-known issue with quotas "going out of sync" in Nova versions Ocata and earlier and is why the 'nova-manage project quota_usage_refresh' command existed. Quotas out-of-sync means that the quota_usages do not match the actual resources being consumed. This can occur due to races while restarting nova-compute, etc.

Tue, Dec 11, 11:59 PM · Cloud-VPS
bd808 added a comment to T211718: Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues.

One more medium instance would be lovely, thank you :-)

Tue, Dec 11, 11:42 PM · Cloud-VPS
bd808 added a comment to T211718: Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues.

While we get this sorted out, is there any chance my quota could get bumped up so I can create the new instance I need? Thanks!

Tue, Dec 11, 10:19 PM · Cloud-VPS
bd808 added a comment to T211718: Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues.
Tue, Dec 11, 8:15 PM · Cloud-VPS
bd808 renamed T211718: Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues from Total calculated VCPU and RAM numbers are incorrect, causing quota issues to Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues.
Tue, Dec 11, 8:11 PM · Cloud-VPS
bd808 added a comment to T211640: Grafana, icinga, prometheus in cloud-analytics project.

Ah ha, and IIRC, I need to get Puppet exported resources to work in my project, right? I'm not using a custom self hosted puppetmaster. Will exported resources work for me?

I'll also probably need Cumin set up, in order to use get_clusters via cumin::selector (https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/462810/ and T204088: Prometheus resources in deployment-prep to create grafana graphs of EventLogging).

Tue, Dec 11, 7:23 PM · Patch-For-Review, Analytics, Analytics-Kanban
bd808 renamed T209818: Mount dumps NFS share to instances in the soweego VPS project from Request to access shared storage on the soweego VPS project to Mount dumps NFS share to instances in the soweego VPS project.
Tue, Dec 11, 7:08 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS, Wikidata
bd808 added a comment to T211640: Grafana, icinga, prometheus in cloud-analytics project.

https://grafana-labs.wikimedia.org/ (and https://grafana-labs-admin.wikimedia.org) exist. There are prometheus instances in the tools and deployment-prep projects, but no multi-tenant prometheus deployment for all projects to share today. As @Andrew mentioned in T211640#4812373 we do have a multi-tenant Shinken service that is the icinga equivalent for instances today. It is used at least by the tools and deployment-prep projects today to generate email & irc alerts for basic instance health.

Tue, Dec 11, 6:01 PM · Patch-For-Review, Analytics, Analytics-Kanban
GitHub <noreply@github.com> committed rGCMPa3f059b82f72: Merge 15aacc71d372b6768a2993dacdc2e57fc8eeeae3 into… (authored by bd808).
Merge 15aacc71d372b6768a2993dacdc2e57fc8eeeae3 into…
Tue, Dec 11, 3:25 AM
bd808 committed rGCMP15aacc71d372: travis-ci: Add PHP 7.3 & tweak PHP 5.3.x (authored by bd808).
travis-ci: Add PHP 7.3 & tweak PHP 5.3.x
Tue, Dec 11, 3:25 AM
bd808 committed rGCMP222160aa787b: Update Travis config (authored by Théo FIDRY <theo.fidry@gmail.com>).
Update Travis config
Tue, Dec 11, 3:25 AM

Fri, Dec 7

bd808 added a comment to T211124: Move mediawiki to new logging infrastructure.

Thanks @bd808 for the context/insight, I agree having the change in core is the right path. I took a stab at the patch, will need some guidance for sure on the mw core production deployment part.

Fri, Dec 7, 10:43 PM · Patch-For-Review, MediaWiki-Logging, Wikimedia-Logstash, Operations
bd808 added a comment to T195119: Create user feedback gadget for technical documentation on Wikitech pages.

we could consider enhancing the extension’s capabilities to add support for specific pages.

Fri, Dec 7, 9:50 PM · Cloud-Services, Documentation
bd808 added a comment to T210934: Toolforge: Node.js policy.

but encourage users that need greater assurances to pin their projects to the major version they target.

Fri, Dec 7, 8:50 PM · Toolforge, cloud-services-team (Kanban)
bd808 added a comment to T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms.

Can you say more what this manual fixing was about? Was this because of something I did wrong (or a missing puppet class or something transient)? Or do we need to this always? Asking because I am putting together all the steps in one place for next time.

Fri, Dec 7, 4:26 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team
bd808 assigned T210917: osmit-tre - Puppet error ("no certificate found and waitforcert is disabled") to GTirloni.

@Andrew could this be a side effect of the project moving to eqiad1?

Fri, Dec 7, 6:05 AM · Cloud-VPS, cloud-services-team (Kanban)
bd808 removed a project from T210924: Request to join mwoffliner: Cloud-VPS.
Fri, Dec 7, 5:59 AM · VPS-Projects
bd808 added a comment to T210934: Toolforge: Node.js policy.

Currently we are tracking the same versions as the Wikimedia production servers use I believe. There are collective costs for back compat breaking changes in the runtimes we provide. A non-trivial number of Tools are built once and set running with little to no changes for long periods of time. Each time a breaking change is introduced in a runtime there is a chance that any or all of these low-maintenance tools will break. This then takes time for fixing the code or often worse they are never fixed.

Fri, Dec 7, 5:58 AM · Toolforge, cloud-services-team (Kanban)
bd808 added a comment to T211146: Increase resources for the services project.

@mobrovac please put your request in the format described at https://phabricator.wikimedia.org/project/profile/2880/.

Fri, Dec 7, 5:38 AM · Cloud-VPS (Quota-requests), Core Platform Team Backlog (Watching / External), Services (watching)
bd808 edited projects for T211146: Increase resources for the services project, added: Cloud-VPS (Quota-requests); removed Cloud-VPS.
Fri, Dec 7, 5:28 AM · Cloud-VPS (Quota-requests), Core Platform Team Backlog (Watching / External), Services (watching)
bd808 moved T211271: Cron <root@labweb1001> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.
Fri, Dec 7, 5:26 AM · cloud-services-team (Kanban), Cloud-Services, Operations
bd808 added a project to T211271: Cron <root@labweb1001> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ): cloud-services-team (Kanban).
Fri, Dec 7, 5:26 AM · cloud-services-team (Kanban), Cloud-Services, Operations
bd808 added a comment to T211271: Cron <root@labweb1001> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ).
$ sudo -i systemctl status apache2.service
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset:
   Active: active (running) (Result: exit-code) since Mon 2018-11-26 15:03:02 UT
 Main PID: 23693 (apache2)
    Tasks: 94 (limit: 7372)
   CGroup: /system.slice/apache2.service
           ├─23693 /usr/sbin/apache2 -k start
           ├─29744 horizon           -k start
           ├─29745 horizon           -k start
           ├─29746 horizon           -k start
           ├─29747 /usr/sbin/apache2 -k start
           └─29865 /usr/sbin/apache2 -k start
Fri, Dec 7, 5:26 AM · cloud-services-team (Kanban), Cloud-Services, Operations
bd808 added a comment to T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms.
$ ssh wikitextexp-base-1002.wikitextexp.eqiad.wmflabs
$ cd /srv/mediawiki-vagrant
$ vim puppet/hieradata/local.yaml
... added a mediawiki::multiwiki::base_domain setting ...
$ grep mediawiki::multiwiki::base_domain puppet/hieradata/local.yaml
mediawiki::multiwiki::base_domain: "-base-wikitextexp.wmflabs.org"
$ vagrant provision
...
Failed miserably
$ vagrant ssh
$ mysql
# manually fixed root and vagrant users to use unix_socket auth
$ vagrant provision
...
$ vagrant ssh -- sudo service apache2 restart

Repeated on wikitextexp-expt-1002.wikitextexp.eqiad.wmflabs. All the wikis that I spot checked from the list are working now. See https://tools.wmflabs.org/openstack-browser/project/wikitextexp for the full list.

Fri, Dec 7, 5:06 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team
bd808 removed a project from T211271: Cron <root@labweb1001> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ): Beta-Cluster-Infrastructure.
Fri, Dec 7, 5:01 AM · cloud-services-team (Kanban), Cloud-Services, Operations
bd808 added a comment to T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms.

Proxy entries created using a python cli tool I made to make it easier to automate the process:

$ for h in $(cat hosts); do
  ./webproxy.py --project=wikitextexp add ${h}-base-wikitextexp http://10.68.16.144:8080
  ./webproxy.py --project=wikitextexp add ${h}-expt-wikitextexp http://10.68.23.251:8080
done
$ ./webproxy.py --project=wikitextexp list
domain                                           backend
================================================ ========================
ar-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ar-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
ckb-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
ckb-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
cu-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
cu-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
cv-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
cv-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
de-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
de-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
en-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
en-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
enwikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
enwikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
enwikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
enwikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
enwiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
enwiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
es-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
es-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
eswikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
eswikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
eswikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
eswikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
eswiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
eswiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
fr-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
fr-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
frwikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
frwikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
frwikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
frwikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
frwiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
frwiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
he-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
he-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
hi-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
hi-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
hy-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
hy-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
is-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
is-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
it-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
it-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
itwikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
itwikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
itwikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
itwikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
itwiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
itwiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
ja-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ja-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
ka-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ka-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
kaa-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
kaa-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
ko-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ko-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
lbe-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
lbe-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
ln-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ln-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
mw-expt-tests.wmflabs.org                        http://172.16.1.159:80
mzn-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
mzn-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
nl-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
nl-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
pl-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
pl-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
pnb-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
pnb-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
pt-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
pt-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
ru-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ru-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
sv-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
sv-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
uk-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
uk-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
uz-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
uz-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
zh-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
zh-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
Fri, Dec 7, 3:54 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team
bd808 moved T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.
Fri, Dec 7, 3:45 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team
bd808 claimed T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms.
Fri, Dec 7, 3:44 AM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team
bd808 added a project to T211391: Neutron API not properly exposed?: Upstream.

Probably a couple of issues to fix here. I'm guessing that @Krenair tested this from a host in Cloud Services. It doesn't look like the firewall rules for port 9696 on cloudcontrol1003 allow 172.16.0.0/21 or 10.196.16.0/21 like ports 9292, 5000, and 8744 do. profile::openstack::base::neutron::service has a ferm rule for prod networks, but none for public networks:

ferm::rule{'neutron-server-api':
    ensure => 'present',
    rule   => "saddr (${prod_networks}) proto tcp dport (9696) ACCEPT;",
}
Fri, Dec 7, 1:56 AM · cloud-services-team (Kanban), Patch-For-Review, Cloud-VPS

Thu, Dec 6

bd808 added a subtask for T187219: Remove support for Trusty Grid Engine exec hosts: T210954: Toolforge: Stretch migration and new software stacks.
Thu, Dec 6, 9:53 PM · Cloud-VPS, Epic
bd808 added a parent task for T210954: Toolforge: Stretch migration and new software stacks: T187219: Remove support for Trusty Grid Engine exec hosts.
Thu, Dec 6, 9:53 PM · Toolforge, Cloud-VPS (Ubuntu Trusty Deprecation), cloud-services-team (Kanban)
bd808 closed T132216: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm as Resolved.

Closing this out. 2.5 years with no updates so... yeah.

Thu, Dec 6, 9:50 PM · Patch-For-Review, Operations, Cloud-Services
bd808 added a comment to T195119: Create user feedback gadget for technical documentation on Wikitech pages.

EventLogging seems like a robust approach to log data entries, JSON schema, probably easy to update entries, but the data can only be stored for ~90 days. I am inclined towards using Extension:VoteNy, as it is functional and works. But, also wondering what would be the advantage of using EventLogging over database logging and extension over gadget for this purpose? How about the idea of using a customized version of VoteNy on Wikitech, to begin?

Thu, Dec 6, 8:12 PM · Cloud-Services, Documentation
bd808 added a comment to T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms.

I think this means that we need to:

  1. Add quota to allow 2 more floating IPs for the project
  2. Attach floating ips to the new hosts
  3. Ensure that the new hosts have the necessary locally installed nginx reverse proxy
  4. Update the *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org DNS entries to point to the new hosts
  5. Verify that things work as expected
  6. Release the old floating IPs
  7. Reduce the floating IP quota back to 2
  8. Delete the old vms
  9. Profit!
Thu, Dec 6, 7:37 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team
bd808 updated subscribers of T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms.

Assuming that the state from T132216 is the desired thing that we need to replicate with the new servers, here are relevant comments from the prior task:

Thu, Dec 6, 7:22 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team
bd808 added a subtask for T204566: cloudvps: wikitextexp project trusty deprecation: T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms.
Thu, Dec 6, 7:13 PM · Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team
bd808 added a parent task for T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms: T204566: cloudvps: wikitextexp project trusty deprecation.
Thu, Dec 6, 7:13 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team
bd808 moved T211149: Ubuntu to Debian migration for maps-warper2 instance of the wikimaps warper from Backlog to VPS Projects on the Cloud-VPS (Ubuntu Trusty Deprecation) board.
Thu, Dec 6, 7:07 PM · Cloud-VPS (Ubuntu Trusty Deprecation), wikimaps-warper
bd808 updated subscribers of T211149: Ubuntu to Debian migration for maps-warper2 instance of the wikimaps warper .
Thu, Dec 6, 7:07 PM · Cloud-VPS (Ubuntu Trusty Deprecation), wikimaps-warper
bd808 added a project to T211149: Ubuntu to Debian migration for maps-warper2 instance of the wikimaps warper : Cloud-VPS (Ubuntu Trusty Deprecation).
Thu, Dec 6, 7:04 PM · Cloud-VPS (Ubuntu Trusty Deprecation), wikimaps-warper
bd808 added a subtask for T204506: cloudvps: maps project trusty deprecation: T211149: Ubuntu to Debian migration for maps-warper2 instance of the wikimaps warper .
Thu, Dec 6, 7:04 PM · Patch-For-Review, User-TheDJ, Cloud-VPS (Ubuntu Trusty Deprecation), Maps
bd808 added a parent task for T211149: Ubuntu to Debian migration for maps-warper2 instance of the wikimaps warper : T204506: cloudvps: maps project trusty deprecation.
Thu, Dec 6, 7:04 PM · Cloud-VPS (Ubuntu Trusty Deprecation), wikimaps-warper

Tue, Dec 4

bd808 added a comment to T204506: cloudvps: maps project trusty deprecation.

What happens after the 18th of December?

Tue, Dec 4, 9:36 PM · Patch-For-Review, User-TheDJ, Cloud-VPS (Ubuntu Trusty Deprecation), Maps
bd808 added a comment to T211124: Move mediawiki to new logging infrastructure.

I've looked briefly at how to implement prefixing syslog json messages with @cee: and I'd say we could do it on the "syslog side" i.e. ./includes/debug/logger/monolog/SyslogHandler.php or "logstash side" i.e. ./includes/debug/logger/monolog/LogstashFormatter.php. I don't have strong opinions or either really!

Tue, Dec 4, 6:56 PM · Patch-For-Review, MediaWiki-Logging, Wikimedia-Logstash, Operations
bd808 added a comment to T210567: Create a way to intentionally trigger fatal errors in MediaWiki.

https://www.mediawiki.org/wiki/Extension:Buggy

Tue, Dec 4, 3:54 AM · serviceops, User-Joe, Core Platform Team Kanban (Done with CPT), Patch-For-Review, Core Platform Team (PHP7 (TEC4)), PHP 7.2 support
bd808 added a comment to T204566: cloudvps: wikitextexp project trusty deprecation.

I am running vagrant up on both VMs .. and on both VMs, I see this error message ==> default: mesg: ttyname failed: Inappropriate ioctl for device .. the vagrant up command continued beyond that. But, is that something to be concerned about?

Tue, Dec 4, 1:05 AM · Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team

Mon, Dec 3

bd808 added a comment to T204566: cloudvps: wikitextexp project trusty deprecation.

I started looking at this now. I have a couple questions.

  1. I suppose I cannot reuse the mw-base and mw-expt instance names since I will first have to spin up new instances (with new unique names), migrate over data to the new VM, and then shut down and delete the old instances. Am I right?
Mon, Dec 3, 11:08 PM · Cloud-VPS (Ubuntu Trusty Deprecation), Parsing-Team
bd808 added a project to T209530: Build user data backup service based on remote sync rather than NFS: Epic.
Mon, Dec 3, 10:14 PM · Epic, cloud-services-team (Kanban)
bd808 added a parent task for T193655: rack/setup/install cloudstore1008 & cloudstore1009: T209530: Build user data backup service based on remote sync rather than NFS.
Mon, Dec 3, 10:14 PM · cloud-services-team (Kanban), Patch-For-Review, ops-eqiad, Cloud-VPS, Operations
bd808 added a subtask for T209530: Build user data backup service based on remote sync rather than NFS: T193655: rack/setup/install cloudstore1008 & cloudstore1009.
Mon, Dec 3, 10:13 PM · Epic, cloud-services-team (Kanban)
bd808 removed a parent task for T209530: Build user data backup service based on remote sync rather than NFS: T193655: rack/setup/install cloudstore1008 & cloudstore1009.
Mon, Dec 3, 10:13 PM · Epic, cloud-services-team (Kanban)
bd808 removed a subtask for T193655: rack/setup/install cloudstore1008 & cloudstore1009: T209530: Build user data backup service based on remote sync rather than NFS.
Mon, Dec 3, 10:13 PM · cloud-services-team (Kanban), Patch-For-Review, ops-eqiad, Cloud-VPS, Operations
bd808 added a comment to T211029: Sort out only one ideal hiera mechanism for Cloud VPS.

Of the two alternate proposals offered so far by @Krenair, proposal (a) of adding some kind of version control/logging to the Horizon system is probably the least technically challenging. I believe it would be possible to use django-reversion to add history to the existing system. Performance could be greatly improved at the cost of some user friendliness by removing the existing system of showing Puppet modules and instead only exposing the raw YAML editing experience. This is a regression from the existing functionality, but it matches the user experience of the other existing competing systems (git and wikitech).

Yeah I'd like to avoid maintaining our own system on top of horizon. git handles versioning for us and is our one of our main development/operations tools, so going through gerrit would be a good idea. It may take a bit of automatic integration work though. I don't care much for the Puppet ENC thing, it'd be easier to find the class I want through grep, and add it via hiera, setting the parameters in hiera.

Mon, Dec 3, 6:02 PM · Cloud-VPS
bd808 added a comment to T211029: Sort out only one ideal hiera mechanism for Cloud VPS.

Each of the existing 3 mechanisms has problems that keep it from being ideal today:

  • Wikitech's Hiera: namespace is dependent on MediaWiki-extensions-OpenStackManager which has been proposed to be undeployed (T161553)
  • operations/puppet.git requires a +2 and manual merge step by a production root user to make a change live (unless the project is using a project local puppetmaster and maintaining local commits). This does not scale well and can take days/weeks/months to occur.
  • The Horizon puppet dashboard does not provide any audit logs about who made changes and when.
Mon, Dec 3, 5:16 PM · Cloud-VPS
bd808 closed T170716: Update Toolforge k8s nodejs images to 6.11, a subtask of T170548: nodejs 6.11, as Resolved.
Mon, Dec 3, 4:44 PM · Maps-Sprint, Maps (Kartographer), Discovery, Services (done), User-mobrovac, Operations
bd808 closed T170716: Update Toolforge k8s nodejs images to 6.11 as Resolved.

Was this resolved somehow?

Mon, Dec 3, 4:44 PM · Kubernetes, Tools-Kubernetes, Toolforge

Fri, Nov 30

bd808 committed rWIEGae5387c0478c: Update index.php for slimapp migration (authored by bd808).
Update index.php for slimapp migration
Fri, Nov 30, 10:50 PM
bd808 closed D1129: Update index.php for slimapp migration.
Fri, Nov 30, 10:50 PM
bd808 accepted D1129: Update index.php for slimapp migration.

Trivial fix

Fri, Nov 30, 10:46 PM
bd808 committed rWIEGeb55757e6ef0: Update vendor dependencies (authored by bd808).
Update vendor dependencies
Fri, Nov 30, 10:46 PM
bd808 closed D1128: Update vendor dependencies.
Fri, Nov 30, 10:46 PM
bd808 requested review of D1129: Update index.php for slimapp migration.
Fri, Nov 30, 10:38 PM
bd808 added a comment to T204509: cloudvps: math project trusty deprecation.

The cluster was upgraded today. I still could not identify a way to upgrade the nodes to a new OS. Is there a guide on how to do this?

Fri, Nov 30, 9:26 PM · User-Physikerwelt, Cloud-VPS (Ubuntu Trusty Deprecation), Math
bd808 moved T210749: Hardware for cloud db replicas for analytics usage from Backlog to Wiki replicas on the Data-Services board.
Fri, Nov 30, 6:18 PM · User-Banyek, Data-Services, User-Elukey, DBA, Analytics
bd808 edited projects for T210749: Hardware for cloud db replicas for analytics usage , added: Data-Services; removed Cloud-VPS.
Fri, Nov 30, 6:18 PM · User-Banyek, Data-Services, User-Elukey, DBA, Analytics
bd808 added a reviewer for D1128: Update vendor dependencies: Bawolff.
Fri, Nov 30, 4:17 AM
bd808 requested review of D1128: Update vendor dependencies.
Fri, Nov 30, 4:15 AM
bd808 added a comment to T210696: OATHAuth logging fails using Monolog.

Any suggestions from your previous experience with monolog how to debug/narrow it down further?

Fri, Nov 30, 4:04 AM · Wikimedia-General-or-Unknown, MediaWiki-Debug-Logger
bd808 created T210820: Migrate Differential repo rWIEG back to Gerrit wikimedia/iegreview.git.
Fri, Nov 30, 3:33 AM · Release-Engineering-Team, Gerrit, Gerrit-Migration, Wikimedia-IEG-grant-review
bd808 created T210812: Remove Generic.PHP.DeprecatedFunctions.Deprecated from 19.x branch.
Fri, Nov 30, 12:21 AM · Patch-For-Review, MediaWiki-Codesniffer

Thu, Nov 29

bd808 moved T210693: Create materialized views on Wiki Replica hosts for better query performance from Backlog to Wiki replicas on the Data-Services board.
Thu, Nov 29, 8:57 PM · Patch-For-Review, User-Banyek, Core Platform Team Backlog (Watching / External), Analytics-Kanban, DBA, Data-Services, Analytics
bd808 renamed T210693: Create materialized views on Wiki Replica hosts for better query performance from Create materialized views for performance to Create materialized views on Wiki Replica hosts for better query performance.
Thu, Nov 29, 8:57 PM · Patch-For-Review, User-Banyek, Core Platform Team Backlog (Watching / External), Analytics-Kanban, DBA, Data-Services, Analytics
bd808 added a comment to T206916: Prepare and check storage layer for shnwiki.

On my side, sql shnwiki on labs runs okay. Running @Pathoschild's command linked above does fail though.

Thu, Nov 29, 4:46 PM · cloud-services-team, Cloud-Services, DBA
bd808 added a comment to T200557: Create a stretch and Son of Grid Engine grid in toolsbeta.

Sorry about that, @JJMC89, I missed that the warning message would come through in cron spam.

Thu, Nov 29, 3:27 PM · Cloud-VPS (Ubuntu Trusty Deprecation), Patch-For-Review, Toolforge, Epic, cloud-services-team (Kanban)
bd808 added a parent task for T209527: Set up scratch and maps NFS services on cloudstore1008/9: T187456: Decommission labstore100[123] and their disk shelves.
Thu, Nov 29, 1:13 AM · Patch-For-Review, cloud-services-team (Kanban)
bd808 added a subtask for T187456: Decommission labstore100[123] and their disk shelves: T209527: Set up scratch and maps NFS services on cloudstore1008/9.
Thu, Nov 29, 1:13 AM · cloud-services-team (Kanban), Data-Services, Operations, DC-Ops, ops-eqiad
bd808 removed a parent task for T187456: Decommission labstore100[123] and their disk shelves: T209527: Set up scratch and maps NFS services on cloudstore1008/9.
Thu, Nov 29, 1:12 AM · cloud-services-team (Kanban), Data-Services, Operations, DC-Ops, ops-eqiad
bd808 removed a subtask for T209527: Set up scratch and maps NFS services on cloudstore1008/9: T187456: Decommission labstore100[123] and their disk shelves.
Thu, Nov 29, 1:12 AM · Patch-For-Review, cloud-services-team (Kanban)
bd808 moved T187456: Decommission labstore100[123] and their disk shelves from Backlog to Shared Storage on the Data-Services board.
Thu, Nov 29, 1:05 AM · cloud-services-team (Kanban), Data-Services, Operations, DC-Ops, ops-eqiad
bd808 added a parent task for T187456: Decommission labstore100[123] and their disk shelves: T209527: Set up scratch and maps NFS services on cloudstore1008/9.
Thu, Nov 29, 1:04 AM · cloud-services-team (Kanban), Data-Services, Operations, DC-Ops, ops-eqiad
bd808 added a subtask for T209527: Set up scratch and maps NFS services on cloudstore1008/9: T187456: Decommission labstore100[123] and their disk shelves.
Thu, Nov 29, 1:04 AM · Patch-For-Review, cloud-services-team (Kanban)
bd808 moved T65782: Make querycache, querycachetwo and querycache_info tables visible on labs dbs from Backlog to Wiki replicas on the Data-Services board.
Thu, Nov 29, 1:03 AM · Data-Services, WMF-Legal
bd808 closed T56401: Backups systems for userDBs hosted on replicas as Declined.

T156869: Design a method for keeping user-created tables in sync across labsDBs

Thu, Nov 29, 1:01 AM · Data-Services
bd808 closed T204422: Install OCRmyPDF dependencies on Tools as Resolved.

AFAICT, we still need at least qpdf, pngquant and unpaper.

Thu, Nov 29, 12:59 AM · Patch-For-Review, cloud-services-team (Kanban), Toolforge, Wikisource
bd808 closed T204422: Install OCRmyPDF dependencies on Tools, a subtask of T55704: Packages to be added to toollabs puppet, as Resolved.
Thu, Nov 29, 12:59 AM · Cloud-Services, Tracking, Toolforge