Page MenuHomePhabricator
Feed Advanced Search

Dec 16 2019

mobrovac lowered the priority of T240518: Some jobs are not being processed / are processed slowly from Unbreak Now! to High.

Lowering the priority as we are out of the woods for the time being. Jobs are being processed now, but it will take a while to catch up with current events (at least 24h form now).

Dec 16 2019, 8:10 PM · Sustainability (Incident Followup), SDC-Statements (Machine-vision-depicts), MachineVision, Product-Infrastructure-Team-Backlog-Deprecated, Structured-Data-Backlog, MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Platform Team Workboards (Clinic Duty Team), WMF-JobQueue, SRE

Dec 13 2019

mobrovac placed T137350: Create general guidelines & processes to ensure thorough fault testing of services up for grabs.
Dec 13 2019, 12:12 PM · Sustainability (Incident Followup), serviceops, Platform Engineering
mobrovac placed T106374: DOI redirect handler for iopscience is very slow/ potentially causes citoid to behave oddly. up for grabs.
Dec 13 2019, 12:11 PM · Platform Team Legacy (Watching / External), Services (watching), Citoid
mobrovac placed T105780: Create a doc explaining the SLA between services and the monitoring tool up for grabs.
Dec 13 2019, 12:11 PM · Observability-Alerting, Platform Team Legacy (Later), Services (later), Documentation, SRE, Service-Architecture
mobrovac placed T215220: Zlib spec.yaml example produces a bad request response up for grabs.
Dec 13 2019, 12:11 PM · Math, User-Physikerwelt, Mathoid
mobrovac placed T122825: Service Ownership and Maintenance up for grabs.
Dec 13 2019, 12:11 PM · Platform Engineering, TechCom, User-mobrovac, SRE
mobrovac placed T235437: RESTBase/RESTRouter/service-runner rate limiting plans up for grabs.
Dec 13 2019, 12:11 PM · Platform Engineering (Icebox), service-runner, User-mobrovac, Services (doing), Platform Team Initiatives (RESTBase Split (CDP2)), serviceops, Kubernetes, Service-deployment-requests, SRE
mobrovac placed T239616: scap3 rollback does not depool servers before rolling back up for grabs.
Dec 13 2019, 12:10 PM · Platform Engineering (Icebox), Scap

Dec 10 2019

mobrovac moved T218395: Create RestrictionStore service from External Code Review Needed to External Code Review In Progress on the Platform Team Workboards (Clinic Duty Team) board.
Dec 10 2019, 12:47 PM · Patch-For-Review, MW-1.37-notes (1.37.0-wmf.23; 2021-09-13), Platform Team Workboards (MW Expedition), MW-1.35-notes (1.35.0-wmf.18; 2020-02-04), Platform Team Initiatives (Decoupling (CDP2))
mobrovac moved T230800: Improve Rdbms/LoadBalance and its LoadMonitor logic from External Code Review Needed to External Code Review In Progress on the Platform Team Workboards (Clinic Duty Team) board.

FYI, I already +1'ed the patch, but @jcrespo raised some questions on the PS that I think should be addressed before merging.

Dec 10 2019, 12:41 PM · MW-1.35-notes (1.35.0-wmf.23; 2020-03-10), Platform Team Workboards (Clinic Duty Team), Patch-For-Review, Performance-Team (Radar), MediaWiki-libs-Rdbms

Dec 9 2019

mobrovac added a project to T238917: Empty reply for pages with Special characters in french REST API: Platform Team Workboards (Clinic Duty Team).
Dec 9 2019, 11:25 AM · RESTBase-API, Platform Engineering (Icebox)
mobrovac added a project to T239713: Citoid is logging all request / response headers as separate fields: Platform Team Workboards (Clinic Duty Team).
Dec 9 2019, 10:44 AM · Observability-Logging, observability, Wikimedia-Logstash, Platform Engineering (Icebox), serviceops, SRE, Citoid
mobrovac added a project to T239090: Restbase logging indexing conflict on 'res' and 'body' logging fields: Platform Team Workboards (Clinic Duty Team).
Dec 9 2019, 10:43 AM · SRE Observability, observability, Platform Team Workboards (Clinic Duty Team), Wikimedia-Logstash, RESTBase
mobrovac removed a member for WMF-JobQueue: mobrovac.
Dec 9 2019, 10:03 AM
mobrovac removed a watcher for WMF-JobQueue: mobrovac.
Dec 9 2019, 10:03 AM
mobrovac removed a watcher for Trending-Service: mobrovac.
Dec 9 2019, 10:03 AM
mobrovac removed a member for Trending-Service: mobrovac.
Dec 9 2019, 10:03 AM
mobrovac removed a member for Texvcjs: mobrovac.
Dec 9 2019, 10:03 AM
mobrovac removed a member for TechCom-RFC: mobrovac.
Dec 9 2019, 10:02 AM
mobrovac removed a member for TechCom: mobrovac.
Dec 9 2019, 10:02 AM
mobrovac removed a watcher for TechCom: mobrovac.
Dec 9 2019, 10:02 AM
mobrovac removed a member for Services: mobrovac.
Dec 9 2019, 10:01 AM
mobrovac removed a watcher for Services: mobrovac.
Dec 9 2019, 10:01 AM
mobrovac removed a member for service-template-node: mobrovac.
Dec 9 2019, 10:01 AM
mobrovac removed a watcher for service-template-node: mobrovac.
Dec 9 2019, 10:01 AM
mobrovac removed a member for service-runner: mobrovac.
Dec 9 2019, 10:00 AM
mobrovac removed a watcher for service-runner: mobrovac.
Dec 9 2019, 10:00 AM
mobrovac removed a member for Service-deployment-requests: mobrovac.
Dec 9 2019, 10:00 AM
mobrovac removed a watcher for Service-deployment-requests: mobrovac.
Dec 9 2019, 10:00 AM
mobrovac removed a member for Service-Architecture: mobrovac.
Dec 9 2019, 10:00 AM
mobrovac removed a watcher for Service-Architecture: mobrovac.
Dec 9 2019, 9:59 AM
mobrovac removed a member for SCB: mobrovac.
Dec 9 2019, 9:59 AM
mobrovac removed a watcher for SCB: mobrovac.
Dec 9 2019, 9:59 AM
mobrovac removed a member for Scap: mobrovac.
Dec 9 2019, 9:58 AM
mobrovac archived RESTBase-Usecase.
Dec 9 2019, 9:58 AM
mobrovac removed a member for RESTBase-Usecase: mobrovac.
Dec 9 2019, 9:57 AM
mobrovac removed a watcher for RESTBase-Usecase: mobrovac.
Dec 9 2019, 9:57 AM
mobrovac removed a member for RESTBase-release-1.0: mobrovac.
Dec 9 2019, 9:55 AM
mobrovac archived RESTBase-release-1.0.
Dec 9 2019, 9:55 AM
mobrovac removed a watcher for RESTBase-release-1.0: mobrovac.
Dec 9 2019, 9:54 AM
mobrovac removed a member for RESTBase-Cassandra: mobrovac.
Dec 9 2019, 9:54 AM
mobrovac removed a watcher for RESTBase-Cassandra: mobrovac.
Dec 9 2019, 9:54 AM
mobrovac removed a member for RESTBase-architecture: mobrovac.
Dec 9 2019, 9:54 AM
mobrovac removed a watcher for RESTBase-architecture: mobrovac.
Dec 9 2019, 9:54 AM
mobrovac removed a watcher for RESTBase-API: mobrovac.
Dec 9 2019, 9:53 AM
mobrovac removed a member for RESTBase-API: mobrovac.
Dec 9 2019, 9:53 AM
mobrovac removed a member for RESTBase: mobrovac.
Dec 9 2019, 9:53 AM
mobrovac removed a watcher for RESTBase: mobrovac.
Dec 9 2019, 9:53 AM
mobrovac removed a member for Push-Notification-Service: mobrovac.
Dec 9 2019, 9:52 AM
mobrovac removed a member for MediaWiki-Vagrant: mobrovac.
Dec 9 2019, 9:52 AM
mobrovac removed a member for MediaWiki-Containers: mobrovac.
Dec 9 2019, 9:51 AM
mobrovac removed a watcher for MediaWiki-Containers: mobrovac.
Dec 9 2019, 9:51 AM
mobrovac removed a member for Mathoid: mobrovac.
Dec 9 2019, 9:51 AM
mobrovac removed a watcher for Mathoid: mobrovac.
Dec 9 2019, 9:51 AM
mobrovac removed a member for HyperSwitch: mobrovac.
Dec 9 2019, 9:50 AM
mobrovac removed a watcher for HyperSwitch: mobrovac.
Dec 9 2019, 9:50 AM
mobrovac removed a member for Prod-Kubernetes: mobrovac.
Dec 9 2019, 9:49 AM
mobrovac removed a watcher for Prod-Kubernetes: mobrovac.
Dec 9 2019, 9:49 AM
mobrovac removed a member for Event-Platform: mobrovac.
Dec 9 2019, 9:49 AM
mobrovac removed a watcher for Event-Platform: mobrovac.
Dec 9 2019, 9:49 AM
mobrovac removed a member for Dumps-Rewrite: mobrovac.
Dec 9 2019, 9:48 AM
mobrovac removed a watcher for Dumps-Rewrite: mobrovac.
Dec 9 2019, 9:48 AM
mobrovac removed a member for Dependency-Tracking: mobrovac.
Dec 9 2019, 9:46 AM
mobrovac removed a watcher for Dependency-Tracking: mobrovac.
Dec 9 2019, 9:46 AM
mobrovac removed a member for Citoid: mobrovac.
Dec 9 2019, 9:46 AM
mobrovac removed a watcher for Citoid: mobrovac.
Dec 9 2019, 9:46 AM
mobrovac removed a member for ChangeProp: mobrovac.
Dec 9 2019, 9:45 AM
mobrovac removed a watcher for ChangeProp: mobrovac.
Dec 9 2019, 9:45 AM
mobrovac removed a member for Cassandra: mobrovac.
Dec 9 2019, 9:45 AM
mobrovac removed a watcher for Cassandra: mobrovac.
Dec 9 2019, 9:45 AM
mobrovac removed a member for Release Pipeline: mobrovac.
Dec 9 2019, 9:44 AM

Dec 6 2019

mobrovac assigned T239877: After deploy of 1.35.0-wmf.8 to group1, surge of "Cannot access the database: Unknown error" to Anomie.
Dec 6 2019, 11:57 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Platform Team Workboards (Clinic Duty Team), DBA, Wikimedia-production-error
mobrovac added a project to T239877: After deploy of 1.35.0-wmf.8 to group1, surge of "Cannot access the database: Unknown error": Platform Team Workboards (Clinic Duty Team).
Dec 6 2019, 11:56 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Platform Team Workboards (Clinic Duty Team), DBA, Wikimedia-production-error

Dec 5 2019

mobrovac closed T239466: Possible to circumvent title-blacklist (CVE-2019-19709) as Resolved.
Dec 5 2019, 1:27 PM · Security, Platform Team Workboards (Clinic Duty Team), MediaWiki-Action-API, TitleBlacklist
mobrovac moved T236624: Unable to restore page: IncompleteRevisionException for some revisions from Waiting for Review to Waiting for deployment on the Platform Team Workboards (Clinic Duty Team) board.
Dec 5 2019, 1:26 PM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Platform Team Workboards (Clinic Duty Team), Technical-Debt, Multi-Content-Revisions, Wikimedia-production-error, MediaWiki-Core-Revision-backend
mobrovac moved T238921: MCR: Include all slots in XML dumps per default from Waiting for Review to Ready (WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.

@daniel Gerrit 552564 which is awaiting review depends on a WIP patch, so moving back to Ready.

Dec 5 2019, 1:24 PM · MW-1.35-notes (1.35.0-wmf.28; 2020-04-14), MW-1.35-release, Platform Team Initiatives (MCR Schema Migration), Platform Team Workboards (Clinic Duty Team), Dumps-Generation
mobrovac moved T239428: API edit on page with non-resolvable redirect and redirect=1 creates page with invalid title from Waiting for Review to Waiting for deployment on the Platform Team Workboards (Clinic Duty Team) board.
Dec 5 2019, 1:18 PM · MW-1.33-notes, MW-1.31-release-notes, MW-1.32-notes, MW-1.34-notes, MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), Platform Team Workboards (Clinic Duty Team), MediaWiki-Action-API
mobrovac added a comment to T206283: Failed deferred updates should be queued as jobs if possible (Deadlock from LinksUpdate in WikiPage::updateCategoryCounts).

@Krinkle @aaron Gerrit 522234 has been merged. As for Gerrit 497537, two things are missing:
(i) Amir had a concern on the PS that went unaddressed;

OK. I've signed the Gerrit change back over to @aaron.

Dec 5 2019, 1:04 PM · MW-1.35-notes (1.35.0-wmf.24; 2020-03-17), Platform Team Workboards (Clinic Duty Team), MediaWiki-Page-derived-data, Performance-Team, Wikimedia-production-error

Dec 4 2019

mobrovac moved T232866: VirtualRESTServiceClient returns status code 0 on Docker from External Code Review Needed to External Code Review Completed on the Platform Team Workboards (Clinic Duty Team) board.
Dec 4 2019, 8:14 PM · MW-1.34-notes, MW-1.33-notes, MW-1.32-notes, MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), MW-1.31-release-notes, Patch-For-Review, MediaWiki-libs-HTTP, Platform Team Workboards (Clinic Duty Team)
mobrovac closed T239120: Services downstream from RESTBase should not separately alert due to RESTBase production monitoring timeouts as Declined.

RESTBase (and other services) are alerting because we want to be sure that we catch problems visible by external users. While it does sometimes introduce false positives (the inverse is true too - RB alerts if back-end services alert), it is better to have more alerts than none due to convoluted logic or fixtures.

Dec 4 2019, 7:53 PM · Product-Infrastructure-Team-Backlog-Deprecated, Platform Engineering, Recommendation-API, Page Content Service, Wikifeeds
mobrovac added a comment to T239806: Parsoid/PHP errors.

I should note that most of the errors appearing there originate from ChangePropagation, so they seem like genuine ones (given that Parsoid/JS don't spit out an error for 99% of these).

Dec 4 2019, 4:37 PM · User-brennen, Parsoid-PHP
mobrovac closed T239768: All RestBase mirrored html2html language conversion pages have page title set to Main Page as Resolved.
Dec 4 2019, 3:31 PM · Platform Team Workboards (Clinic Duty Team), Parsoid-PHP, RESTBase
mobrovac committed rGRBDf4b752ec99ef: Update restbase to 8a9fdd1.
Update restbase to 8a9fdd1
Dec 4 2019, 3:13 PM
mobrovac moved T239768: All RestBase mirrored html2html language conversion pages have page title set to Main Page from Inbox to Waiting for deployment on the Platform Team Workboards (Clinic Duty Team) board.
Dec 4 2019, 1:17 PM · Platform Team Workboards (Clinic Duty Team), Parsoid-PHP, RESTBase
mobrovac claimed T239768: All RestBase mirrored html2html language conversion pages have page title set to Main Page.

PR #1226 adds the title component.

Dec 4 2019, 1:07 PM · Platform Team Workboards (Clinic Duty Team), Parsoid-PHP, RESTBase
mobrovac added a subtask for T229015: Tracking: Direct live production traffic at Parsoid/PHP: T239806: Parsoid/PHP errors.
Dec 4 2019, 12:37 PM · User-notice-archive, Platform Engineering, User-WDoran, Parsoid-PHP
mobrovac added a parent task for T239806: Parsoid/PHP errors: T229015: Tracking: Direct live production traffic at Parsoid/PHP.
Dec 4 2019, 12:37 PM · User-brennen, Parsoid-PHP
mobrovac triaged T239806: Parsoid/PHP errors as High priority.
Dec 4 2019, 12:37 PM · User-brennen, Parsoid-PHP

Dec 3 2019

mobrovac updated subscribers of T239643: Bugs in PHP port of LanguageConverter.

Mirroring had to be disabled due to too many errors being emitted by Parsoid/PHP. Once the fixes for it have been deployed, RESTBase can de re-set up to mirror the traffic. I have prepared everything for it, all that needs to be done is to deploy restbase with scap deploy -f from deploy1001.

Dec 3 2019, 1:23 PM · MediaWiki-Language-converter, Chinese-Sites, Parsoid-PHP
mobrovac added a reverting change for rGRBD41bb230524c0: TEMP: Log all html2html errors coming from Parsoid/PHP: rGRBD4ec80580d883: Revert "TEMP: Log all html2html errors coming from Parsoid/PHP".
Dec 3 2019, 1:17 PM
mobrovac committed rGRBD4ec80580d883: Revert "TEMP: Log all html2html errors coming from Parsoid/PHP".
Revert "TEMP: Log all html2html errors coming from Parsoid/PHP"
Dec 3 2019, 1:17 PM
mobrovac added a comment to T239643: Bugs in PHP port of LanguageConverter.

RESTBase is now mirroring all html2html requests to Parsoid/PHP.

Dec 3 2019, 12:46 PM · MediaWiki-Language-converter, Chinese-Sites, Parsoid-PHP
mobrovac committed rGRBD41bb230524c0: TEMP: Log all html2html errors coming from Parsoid/PHP.
TEMP: Log all html2html errors coming from Parsoid/PHP
Dec 3 2019, 12:28 PM
mobrovac committed rGRBDb346ebf41951: Update restbase to 0fbd266.
Update restbase to 0fbd266
Dec 3 2019, 11:55 AM

Dec 2 2019

mobrovac committed rMSCP0185252b4faf: Support dotted JSONPath notation for partitioner key (authored by Pchelolo).
Support dotted JSONPath notation for partitioner key
Dec 2 2019, 10:28 PM
mobrovac committed rGRBD92acf1e5ae89: Switch everything to Parsoid/PHP.
Switch everything to Parsoid/PHP
Dec 2 2019, 8:01 PM
mobrovac committed rGRBDe69e2e5bb8cb: Switch everything but enwiki to Parsoid/PHP.
Switch everything but enwiki to Parsoid/PHP
Dec 2 2019, 7:16 PM
mobrovac committed rGRBD1b237d93e6ad: Update restbase to 2729d6f.
Update restbase to 2729d6f
Dec 2 2019, 6:19 PM
mobrovac added a reverting change for rGRBDff7862fc188d: Switch sr and zh wp back to Parsoid/PHP: rGRBDd25ea4749e1d: Revert "Switch sr and zh wp back to Parsoid/PHP".
Dec 2 2019, 6:19 PM
mobrovac committed rGRBDd25ea4749e1d: Revert "Switch sr and zh wp back to Parsoid/PHP".
Revert "Switch sr and zh wp back to Parsoid/PHP"
Dec 2 2019, 6:19 PM