Page MenuHomePhabricator

mobrovac (Marko Obrovac)
Spy

Projects (46)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Dec 16 2014, 7:40 PM (251 w, 6 d)
Availability
Available
IRC Nick
mobrovac
LDAP User
Mobrovac
MediaWiki User
Mobrovac [ Global Accounts ]

Recent Activity

Today

mobrovac added a comment to T235346: Edits which has been reverted and revision deleted over 40 hours ago were visible on page previews.

I looked at the page in question and I see it has the correct content matching the wikitext. What could have happened is that the purge request itself fell through, so RB did have the new version stored, but Varnish was serving the old one until it fell out of cache.

Tue, Oct 15, 11:28 AM · Readers-Web-Backlog (Tracking), RESTBase-API, User-Josve05a, Page-Previews
mobrovac added a project to T235492: [[MediaWiki:Rest-compare-page-mismatch]] translation issue: CPT Initiatives (Core REST API in PHP).
Tue, Oct 15, 11:03 AM · CPT Initiatives (Core REST API in PHP), MediaWiki-REST-API, I18n
mobrovac edited projects for T235492: [[MediaWiki:Rest-compare-page-mismatch]] translation issue, added: MediaWiki-REST-API; removed RESTBase.
Tue, Oct 15, 11:03 AM · CPT Initiatives (Core REST API in PHP), MediaWiki-REST-API, I18n
mobrovac added a comment to T235439: MW REST API, wiki titles, Apache2 and URI decoding fun.

Thank you @Joe for taking the time to investigate. I created T235478: Varnish/ATS should not decode URIs for /w/rest.php to follow up with the traffic folks.

Tue, Oct 15, 9:24 AM · TechCom, User-mobrovac, serviceops, Wikimedia-Apache-configuration, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP)
mobrovac triaged T235478: Varnish/ATS should not decode URIs for /w/rest.php as High priority.
Tue, Oct 15, 9:06 AM · CPT Initiatives (Core REST API in PHP), Parsoid-PHP, MediaWiki-REST-API, Traffic, Operations
mobrovac added a comment to T235439: MW REST API, wiki titles, Apache2 and URI decoding fun.

No, MediaWiki prefers REQUEST_URI. But your output does not coincide with mine. I altered the code in-place on parsoid10 yesterday and sent back via a header the contents of REQUEST_URI and got back the decoded version. Also, the fact that MW's router 404s on that route confirms that MW sees the decoded version.

Tue, Oct 15, 6:17 AM · TechCom, User-mobrovac, serviceops, Wikimedia-Apache-configuration, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP)
mobrovac added a comment to T233127: HTTP 404 error in VE possibly when confronted with an edit conflict.

Grrr it seems there is another bug lurking somewhere. I looked at the etags you posted and it is not clear to me how that ETag gets to VE. In all cases, the Etag contains the revision 0 but none of these pages are new - they have existed for quite a while. I tried reproducing this, but haven't managed to. I suspect the problem is somewhere in RESTBase. I will open another ticket and investigate further.

Tue, Oct 15, 5:34 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac moved T235465: Stashing: revid mismatch between URI and Etag from Inbox to Doing on the Core Platform Team Workboards (Clinic Duty Team) board.
Tue, Oct 15, 5:31 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase-API, RESTBase
mobrovac added a parent task for T235465: Stashing: revid mismatch between URI and Etag: T233127: HTTP 404 error in VE possibly when confronted with an edit conflict.
Tue, Oct 15, 5:31 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase-API, RESTBase
mobrovac added a subtask for T233127: HTTP 404 error in VE possibly when confronted with an edit conflict: T235465: Stashing: revid mismatch between URI and Etag.
Tue, Oct 15, 5:31 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac triaged T235465: Stashing: revid mismatch between URI and Etag as High priority.
Tue, Oct 15, 5:29 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase-API, RESTBase
mobrovac updated the task description for T235460: Parsoid/PHP does not emit ETags.
Tue, Oct 15, 4:59 AM · Parsing-Team, CPT Initiatives (Parsoid REST API in PHP (CDP2)), Parsoid-PHP
mobrovac added projects to T233127: HTTP 404 error in VE possibly when confronted with an edit conflict: RESTBase, Core Platform Team Workboards (Clinic Duty Team).
Tue, Oct 15, 4:57 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac added a comment to T233127: HTTP 404 error in VE possibly when confronted with an edit conflict.

@mobrovac I think I have some more questions…

Tue, Oct 15, 4:57 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac triaged T235462: Parsoid/JS / RESTBase ETag harmonisation as High priority.
Tue, Oct 15, 4:33 AM · Core Platform Team Workboards (Clinic Duty Team), Parsing-Team, RESTBase-API, RESTBase, Parsoid
mobrovac created T235460: Parsoid/PHP does not emit ETags.
Tue, Oct 15, 4:23 AM · Parsing-Team, CPT Initiatives (Parsoid REST API in PHP (CDP2)), Parsoid-PHP

Yesterday

mobrovac claimed T235437: RESTBase/RESTRouter/service-runner rate limiting plans.
Mon, Oct 14, 6:37 PM · User-mobrovac, Core Platform Team Workboards (Clinic Duty Team), Services (doing), CPT Initiatives (RESTBase Split (CDP2)), serviceops, Kubernetes, Service-deployment-requests, Operations
mobrovac updated the task description for T235439: MW REST API, wiki titles, Apache2 and URI decoding fun.
Mon, Oct 14, 5:54 PM · TechCom, User-mobrovac, serviceops, Wikimedia-Apache-configuration, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP)
mobrovac added a comment to T235439: MW REST API, wiki titles, Apache2 and URI decoding fun.

According to Apache2 documentation, setting AllowEncodedSlashes NoDecode in the config should solve the problem, but when I tried it in Beta it didn't seem to work. Either way, even if that setting does work, the question is can its effect be limited to only a specific route prefix, i.e. /w/rest.php.

Mon, Oct 14, 3:23 PM · TechCom, User-mobrovac, serviceops, Wikimedia-Apache-configuration, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP)
mobrovac updated the task description for T235439: MW REST API, wiki titles, Apache2 and URI decoding fun.
Mon, Oct 14, 3:19 PM · TechCom, User-mobrovac, serviceops, Wikimedia-Apache-configuration, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP)
mobrovac merged T235375: Parsoid/PHP responds with 404 for titles with slashes and revision into T235439: MW REST API, wiki titles, Apache2 and URI decoding fun.
Mon, Oct 14, 3:18 PM · TechCom, User-mobrovac, serviceops, Wikimedia-Apache-configuration, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP)
mobrovac merged task T235375: Parsoid/PHP responds with 404 for titles with slashes and revision into T235439: MW REST API, wiki titles, Apache2 and URI decoding fun.
Mon, Oct 14, 3:18 PM · Core Platform Team Workboards (Clinic Duty Team), Parsing-Team, RESTBase, MediaWiki-REST-API, Parsoid-PHP
mobrovac updated the task description for T235439: MW REST API, wiki titles, Apache2 and URI decoding fun.
Mon, Oct 14, 3:16 PM · TechCom, User-mobrovac, serviceops, Wikimedia-Apache-configuration, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP)
mobrovac added projects to T235439: MW REST API, wiki titles, Apache2 and URI decoding fun: User-mobrovac, TechCom.

Tagging TechCom for visibility.

Mon, Oct 14, 3:11 PM · TechCom, User-mobrovac, serviceops, Wikimedia-Apache-configuration, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP)
mobrovac created T235439: MW REST API, wiki titles, Apache2 and URI decoding fun.
Mon, Oct 14, 3:11 PM · TechCom, User-mobrovac, serviceops, Wikimedia-Apache-configuration, MediaWiki-REST-API, CPT Initiatives (Core REST API in PHP)
mobrovac added a comment to T235410: Cannot read property stored of undefined.

This is happening in production, not beta.

Mon, Oct 14, 2:44 PM · Parsoid
mobrovac added a comment to T233654: Make the parsoid cluster support parsoid/PHP.

Aaah indeed, you are right @Joe. Not sure where I was looking :/ Must have been logged into a Beta instance. Ok, we don't need to do this whole port-switching dance then, yay.

Mon, Oct 14, 1:12 PM · Patch-For-Review, Operations, serviceops
mobrovac added a comment to T235375: Parsoid/PHP responds with 404 for titles with slashes and revision.

Ok, I was partially correct. The problem is indeed that MW tries to match the decoded path, which fails because of the slash. However, it turns out that that's not on MW, but rather on Apache: it fully decodes the path before passing it on to PHP.

Mon, Oct 14, 12:25 PM · Core Platform Team Workboards (Clinic Duty Team), Parsing-Team, RESTBase, MediaWiki-REST-API, Parsoid-PHP
mobrovac added a comment to T235375: Parsoid/PHP responds with 404 for titles with slashes and revision.

I haven't looked too deep into it, but I think the problem is that the title gets decoded before looking up the appropriate handler, which then presents a challenge in this case. Parsoid declares the following (relevant) route handlers:

Mon, Oct 14, 10:38 AM · Core Platform Team Workboards (Clinic Duty Team), Parsing-Team, RESTBase, MediaWiki-REST-API, Parsoid-PHP
mobrovac removed a project from T233127: HTTP 404 error in VE possibly when confronted with an edit conflict: RESTBase.

The RESTBase bug has been fixed. However, there are still 404s being emitted during transforms. They now happen at a rate of 4-5 per day (which is great improvement compared to 150 - 200 per day before the bug was fixed). I was looking at the RESTBase logs and they are all legitimate 404s from RESTBase's point of view, in that VE never asks RB for the content in the first place, i.e. there are no equivalent /page/html/{title} requests that reach RESTBase. That may be due to a couple of reasons:

  • VE does not ask the content with ?stash=true, thus hitting Varnish; or
  • VE does not ask for the content at all, presumably because it uses a (local) cache of some sort.
Mon, Oct 14, 9:20 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac closed T234928: RESTBase sometimes not retaining stashed content?, a subtask of T233127: HTTP 404 error in VE possibly when confronted with an edit conflict, as Resolved.
Mon, Oct 14, 9:04 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac closed T234928: RESTBase sometimes not retaining stashed content? as Resolved.

The bug has been found and fixed and temporary logging removed. We are done here.

Mon, Oct 14, 9:04 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase-Cassandra, Cassandra, RESTBase
mobrovac moved T235173: Stop pregenerating /page/metadata from Inbox to Done on the Core Platform Team Workboards (Clinic Duty Team) board.
Mon, Oct 14, 9:01 AM · Core Platform Team Workboards (Clinic Duty Team), Services (done), Product-Infrastructure-Team-Backlog, Page Content Service
mobrovac closed T235173: Stop pregenerating /page/metadata, a subtask of T229286: "worker died, restarting" mobileapps issue, as Resolved.
Mon, Oct 14, 9:01 AM · Patch-For-Review, Page Content Service, Product-Infrastructure-Team-Backlog (Kanban), Mobile-Content-Service, serviceops
mobrovac closed T235173: Stop pregenerating /page/metadata as Resolved.

Pre-generation has been removed from change propagation, RESTBase is now querying PCS directly and the tables have been removed from storage. We are done here.

Mon, Oct 14, 9:01 AM · Core Platform Team Workboards (Clinic Duty Team), Services (done), Product-Infrastructure-Team-Backlog, Page Content Service
mobrovac committed rDEPLOYCHARTS0e92d884d046: RESTRouter: Use image v1.1.4 (authored by mobrovac).
RESTRouter: Use image v1.1.4
Mon, Oct 14, 8:59 AM
mobrovac created T235410: Cannot read property stored of undefined.
Mon, Oct 14, 8:04 AM · Parsoid
mobrovac created T235408: Cannot read property filter of undefined.
Mon, Oct 14, 7:58 AM · Product-Infrastructure-Team-Backlog, Mobile-Content-Service, Page Content Service
mobrovac committed rGRBD4d469a1071d1: Beta: No delay is needed during deploys there (authored by mobrovac).
Beta: No delay is needed during deploys there
Mon, Oct 14, 7:35 AM
mobrovac added a reverting change for rGRBD1798e3924b90: [TEMP] Log all VE requests: rGRBD305fc7d87064: Revert "[TEMP] Log all VE requests".
Mon, Oct 14, 7:32 AM
mobrovac committed rGRBD305fc7d87064: Revert "[TEMP] Log all VE requests" (authored by mobrovac).
Revert "[TEMP] Log all VE requests"
Mon, Oct 14, 7:32 AM
mobrovac committed rGRBDe0d071fa143e: Update restbase to bd0072a (authored by mobrovac).
Update restbase to bd0072a
Mon, Oct 14, 7:30 AM

Sun, Oct 13

mobrovac added a parent task for T235375: Parsoid/PHP responds with 404 for titles with slashes and revision: T230791: Have a Mechanism for Storing and Retrieving Parsoid HTML from JS and PHP.
Sun, Oct 13, 11:18 AM · Core Platform Team Workboards (Clinic Duty Team), Parsing-Team, RESTBase, MediaWiki-REST-API, Parsoid-PHP
mobrovac added a subtask for T230791: Have a Mechanism for Storing and Retrieving Parsoid HTML from JS and PHP: T235375: Parsoid/PHP responds with 404 for titles with slashes and revision.
Sun, Oct 13, 11:18 AM · Core Platform Team Workboards (Clinic Duty Team), CPT Initiatives (Parsoid PHP (CDP2)), RESTBase, Parsoid-PHP
mobrovac moved T230791: Have a Mechanism for Storing and Retrieving Parsoid HTML from JS and PHP from Inbox to Blocked Externally on the Core Platform Team Workboards (Clinic Duty Team) board.
Sun, Oct 13, 11:18 AM · Core Platform Team Workboards (Clinic Duty Team), CPT Initiatives (Parsoid PHP (CDP2)), RESTBase, Parsoid-PHP
mobrovac claimed T230791: Have a Mechanism for Storing and Retrieving Parsoid HTML from JS and PHP.

PR #1207 accomplishes this. However, it is blocked on Parsoid failing to handle certain routes, cf. T235375: Parsoid/PHP responds with 404 for titles with slashes and revision.

Sun, Oct 13, 11:17 AM · Core Platform Team Workboards (Clinic Duty Team), CPT Initiatives (Parsoid PHP (CDP2)), RESTBase, Parsoid-PHP
mobrovac triaged T235375: Parsoid/PHP responds with 404 for titles with slashes and revision as High priority.
Sun, Oct 13, 11:17 AM · Core Platform Team Workboards (Clinic Duty Team), Parsing-Team, RESTBase, MediaWiki-REST-API, Parsoid-PHP

Sat, Oct 12

mobrovac updated subscribers of T233654: Make the parsoid cluster support parsoid/PHP.

We have to resolve the same problem here to the one we encountered in Beta. Namely, both php-fpm and parsoid services use port 8000 to listen to incoming requests. This creates a rather problematic situation, since all the appservers have it configured on that port, and Parsoid/JS has an LVS set up on that port that is used by RESTBase and possibly others.

Sat, Oct 12, 11:45 AM · Patch-For-Review, Operations, serviceops
mobrovac edited projects for T235299: Cassandra cluster management support for multi-tenancy, added: Cassandra; removed Core Platform Team.

Instead of having full templates in roles profiles, we could have an add_user.pp manifest in modules/cassandra that compiles it for the profile, so a structure with the relevant info could be passed, something like:

Sat, Oct 12, 7:20 AM · Cassandra, Core Platform Team Workboards (Clinic Duty Team), User-Eevans

Fri, Oct 11

mobrovac moved T235179: Implement workarounds in RESTBase and Flow to hit Parsoid/PHP REST API endpoints without an oldid for titles containing "." from Inbox to Backlog on the Core Platform Team Workboards (Clinic Duty Team) board.
Fri, Oct 11, 8:54 AM · Core Platform Team Workboards (Clinic Duty Team), StructuredDiscussions, RESTBase, Parsoid-PHP, Growth-Team
mobrovac added a project to T235179: Implement workarounds in RESTBase and Flow to hit Parsoid/PHP REST API endpoints without an oldid for titles containing ".": Core Platform Team Workboards (Clinic Duty Team).

From what I can tell, RESTBase never asks for a page without revision, so it should be fine there.

Fri, Oct 11, 8:54 AM · Core Platform Team Workboards (Clinic Duty Team), StructuredDiscussions, RESTBase, Parsoid-PHP, Growth-Team

Thu, Oct 10

mobrovac added a comment to T233127: HTTP 404 error in VE possibly when confronted with an edit conflict.

It seems puzzled to me, it often happened to me editing a certain page several times the same day, certainly not at the first edit.

Thu, Oct 10, 3:39 PM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac added a comment to T233127: HTTP 404 error in VE possibly when confronted with an edit conflict.

Update: I think I found the bug in RESTBase. I deployed the fix yesterday and so far so good. I will leave RB logging on until Monday to make sure the problem is indeed fixed.

Thu, Oct 10, 2:52 PM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac added a comment to T232007: Restbase: significant increase of outbound dropped packets.

I don't have a definitive answer here. The deploys that coincide with the drop did streamline the start-up sequence and increase the allowed memory per worker to avoid worker deaths, so either of these would help:

  • the start-up sequence now does not issue a high number of queries to Cassandra on start-up
  • increasing the memory limit stopped worker deaths
Thu, Oct 10, 8:53 AM · service-runner, RESTBase, User-mobrovac, Core Platform Team Workboards (Clinic Duty Team)
mobrovac committed rDEPLOYCHARTS1449665f0ea8: RESTRouter: Use image v1.1.3 (authored by mobrovac).
RESTRouter: Use image v1.1.3
Thu, Oct 10, 8:14 AM

Wed, Oct 9

mobrovac committed rGRBDaaadd73e9755: Update restbase to 1885482 (authored by mobrovac).
Update restbase to 1885482
Wed, Oct 9, 10:26 PM
mobrovac committed rGRBD068d2edff70e: Update restbase to c8faef7 (authored by mobrovac).
Update restbase to c8faef7
Wed, Oct 9, 10:26 PM
Eevans awarded T234928: RESTBase sometimes not retaining stashed content? a Stroopwafel token.
Wed, Oct 9, 4:45 PM · Core Platform Team Workboards (Clinic Duty Team), RESTBase-Cassandra, Cassandra, RESTBase
mobrovac claimed T234928: RESTBase sometimes not retaining stashed content?.

Ha! I think I found the bug. The problem is that when a user first opens the page and starts editing with VE, then VE calls /page/html/{title}?stash=true, but RESTBase expects the revision ID to be present as well, which causes it to store the page under the key {title}:undefined:{tid} instead of {title}:{revid}:{tid} in the stash bucket. However, if the user tries to VE-edit a page that they edited before (i.e. it was already loaded, because they have already edited it either via VE or the wt editor), then VE calls /page/html/{title}/{revision}?stash=true.

Wed, Oct 9, 1:30 PM · Core Platform Team Workboards (Clinic Duty Team), RESTBase-Cassandra, Cassandra, RESTBase
mobrovac moved T170455: Extract the feed endpoints from PCS into a new wikifeeds service from Blocked Externally to Done on the Core Platform Team Workboards (Clinic Duty Team) board.

RESTBase is now using wikifeeds for the feed content \o/

Wed, Oct 9, 1:01 PM · Core Platform Team Workboards (Clinic Duty Team), Product-Infrastructure-Team-Backlog, Epic, Wikifeeds, Patch-For-Review, Page Content Service
mobrovac updated the task description for T170455: Extract the feed endpoints from PCS into a new wikifeeds service.
Wed, Oct 9, 1:00 PM · Core Platform Team Workboards (Clinic Duty Team), Product-Infrastructure-Team-Backlog, Epic, Wikifeeds, Patch-For-Review, Page Content Service
mobrovac moved T234474: Expand Gerrit Manager permissions from Inbox to Watching on the TechCom board.
Wed, Oct 9, 12:43 PM · Operations, Gerrit, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, TechCom
mobrovac triaged T235051: mobileapps.wmflabs.org down as High priority.
Wed, Oct 9, 10:41 AM · Product-Infrastructure-Team-Backlog (Kanban), Mobile-Content-Service, Beta-Cluster-Infrastructure
mobrovac added a comment to T233127: HTTP 404 error in VE possibly when confronted with an edit conflict.

In other news, I opened T234928: RESTBase sometimes not retaining stashed content? as my digging revealed it's likely a problem in RESTBase. Stay tuned as we figure out what/why/how this is happening.

Wed, Oct 9, 9:07 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac added a comment to T233127: HTTP 404 error in VE possibly when confronted with an edit conflict.

Interesting. So what is VE supposed to do in this case if we want to let the user save their edit anyway? (Consistent with MediaWiki, we display a warning about this first, but we let the user override it.) Should we retry the request without oldid and etag?
Alternatively, could RESTBase keep and return the render even if the page is deleted, as long as it's requested with the right etag?

Wed, Oct 9, 9:06 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team

Tue, Oct 8

mobrovac added a comment to T234928: RESTBase sometimes not retaining stashed content?.

What consistency level did you use in cqlsh (it defaults to ONE)?

Tue, Oct 8, 3:02 PM · Core Platform Team Workboards (Clinic Duty Team), RESTBase-Cassandra, Cassandra, RESTBase
mobrovac added a comment to T234928: RESTBase sometimes not retaining stashed content?.

@Pchelolo @Eevans could you take a look? Perhaps I've overlooked something, but to me this is starting to smell like Cassandra is losing data somehow.

Tue, Oct 8, 2:47 PM · Core Platform Team Workboards (Clinic Duty Team), RESTBase-Cassandra, Cassandra, RESTBase
mobrovac added a subtask for T233127: HTTP 404 error in VE possibly when confronted with an edit conflict: T234928: RESTBase sometimes not retaining stashed content?.
Tue, Oct 8, 2:45 PM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac added a parent task for T234928: RESTBase sometimes not retaining stashed content?: T233127: HTTP 404 error in VE possibly when confronted with an edit conflict.
Tue, Oct 8, 2:45 PM · Core Platform Team Workboards (Clinic Duty Team), RESTBase-Cassandra, Cassandra, RESTBase
mobrovac triaged T234928: RESTBase sometimes not retaining stashed content? as High priority.
Tue, Oct 8, 2:44 PM · Core Platform Team Workboards (Clinic Duty Team), RESTBase-Cassandra, Cassandra, RESTBase
mobrovac committed rGRBD00eda0bd2975: Update restbase to 2b539f2 (authored by mobrovac).
Update restbase to 2b539f2
Tue, Oct 8, 10:08 AM
mobrovac committed rGRBD83fcc0cf53f9: Update restbase to 7bbc8c3 (authored by mobrovac).
Update restbase to 7bbc8c3
Tue, Oct 8, 8:39 AM
mobrovac moved T232128: Make MultiHttpClient use CURLMOPT_MAX_HOST_CONNECTIONS and reuse connections from Blocked Externally to External Code Review Completed on the Core Platform Team Workboards (Clinic Duty Team) board.
Tue, Oct 8, 8:00 AM · MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MediaWiki-libs-HTTP, Performance-Team (Radar), Core Platform Team Workboards (Clinic Duty Team)

Mon, Oct 7

mobrovac committed rGRBD13372901a5dc: Update restbase to 5d8b9bc (authored by mobrovac).
Update restbase to 5d8b9bc
Mon, Oct 7, 1:20 PM
mobrovac committed rGRBDbf72f5c30a65: Update restbase to 244e515 (authored by mobrovac).
Update restbase to 244e515
Mon, Oct 7, 1:10 PM
mobrovac committed rGRBDfe39197b6273: Update restbase to 913ae07 (authored by mobrovac).
Update restbase to 913ae07
Mon, Oct 7, 1:02 PM
mobrovac updated the task description for T220449: Split RESTBase in two services: storage service and API router/proxy.
Mon, Oct 7, 12:23 PM · CPT Initiatives (RESTBase Split (CDP2)), User-mobrovac, serviceops, Epic, RESTBase
mobrovac added a parent task for T234816: Make internal services use RESTRouter instead of RESTBase: T220449: Split RESTBase in two services: storage service and API router/proxy.
Mon, Oct 7, 12:23 PM · Core Platform Team Workboards (Clinic Duty Team), serviceops, RESTBase
mobrovac added a subtask for T220449: Split RESTBase in two services: storage service and API router/proxy: T234816: Make internal services use RESTRouter instead of RESTBase.
Mon, Oct 7, 12:23 PM · CPT Initiatives (RESTBase Split (CDP2)), User-mobrovac, serviceops, Epic, RESTBase
mobrovac triaged T234816: Make internal services use RESTRouter instead of RESTBase as Normal priority.
Mon, Oct 7, 12:22 PM · Core Platform Team Workboards (Clinic Duty Team), serviceops, RESTBase
mobrovac updated subscribers of T234813: Google requesting Parsoid HTML with `?stash=true`, which should be avoided.
Mon, Oct 7, 12:10 PM · RESTBase-API, Core Platform Team
mobrovac triaged T234813: Google requesting Parsoid HTML with `?stash=true`, which should be avoided as High priority.
Mon, Oct 7, 12:02 PM · RESTBase-API, Core Platform Team
mobrovac closed T233584: Re-adjust cirrusSearchLinksUpdate vs cirrusSearchLinksUpdatePrioritized concurrency as Resolved.

The jobs are stable even with the higher concurrency, so I'd be inclined of keeping the current level. Resolving.

Mon, Oct 7, 11:15 AM · Services (done), Core Platform Team Workboards (Clinic Duty Team), WMF-JobQueue, Discovery-Search (Current work)
mobrovac added a comment to T233127: HTTP 404 error in VE possibly when confronted with an edit conflict.

I spent most of yesterday trying to reproduce the 404 for a legit error, but was unable to do so.

Mon, Oct 7, 11:11 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac closed T234772: Add banwiki to restbase, a subtask of T234768: Create Balinese Wikipedia, as Resolved.
Mon, Oct 7, 11:08 AM · MW-1.35-notes (1.35.0-wmf.2; 2019-10-15), Patch-For-Review, User-Ladsgroup, Wiki-Setup (Create), User-Urbanecm
mobrovac closed T234772: Add banwiki to restbase as Resolved.
Mon, Oct 7, 11:08 AM · Core Platform Team Workboards (Clinic Duty Team), RESTBase
mobrovac committed rGRBD5321aac0d0bb: Increase the delay back to 52s (authored by mobrovac).
Increase the delay back to 52s
Mon, Oct 7, 10:44 AM
mobrovac committed rGRBD273067777649: Skip creating resources on start-up (authored by mobrovac).
Skip creating resources on start-up
Mon, Oct 7, 9:30 AM
mobrovac committed rGRBD1798e3924b90: [TEMP] Log all VE requests (authored by mobrovac).
[TEMP] Log all VE requests
Mon, Oct 7, 9:30 AM
mobrovac committed rGRBDcc16d2ec51e6: Add banwiki (authored by mobrovac).
Add banwiki
Mon, Oct 7, 9:30 AM
mobrovac committed rGRBDf7fbadf59105: Update restbase to 7d6dbf7 (authored by mobrovac).
Update restbase to 7d6dbf7
Mon, Oct 7, 9:30 AM
mobrovac committed rGRBD1039f347b063: Config: Increase heap limit to 750MB (authored by mobrovac).
Config: Increase heap limit to 750MB
Mon, Oct 7, 9:30 AM
mobrovac committed rDEPLOYCHARTS0967210f6189: RESTRouter: Add banwiki (authored by mobrovac).
RESTRouter: Add banwiki
Mon, Oct 7, 8:54 AM

Fri, Oct 4

mobrovac added a comment to T233127: HTTP 404 error in VE possibly when confronted with an edit conflict.

@mobrovac If you have the time to take a look, there are ~700 instances of a HTTP 404 response from RESTBase logged now: https://logstash.wikimedia.org/goto/4817263807e9f036af940cf848effc24. I apologize for the messy format (couldn't get structured logging to cooperate, see T234564), but the logs contain the URL we requested, the If-Match header we sent, and the response body.
Some of the cases have mangled or missing If-Match (let's come back to those later, in T233320), but at a glance most of the etags we sent look perfectly valid. Do you have a way to look up if RESTBase ever had those documents, if it purged them, or what?

Fri, Oct 4, 3:03 PM · Core Platform Team Workboards (Clinic Duty Team), RESTBase, MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), VisualEditor (Current work), BetaFeatures, Two-Column-Edit-Conflict-Merge, TCB-Team
mobrovac added a comment to T223953: Deploy the RESTBase front-end service (RESTRouter) to Kubernetes.

The start-up time is now pretty good: around 3-5s per worker.

Fri, Oct 4, 2:40 PM · Core Platform Team Workboards (Clinic Duty Team), CPT Initiatives (RESTBase Split (CDP2)), Patch-For-Review, Release Pipeline, Kubernetes, serviceops, Service-deployment-requests, Operations
mobrovac committed rDEPLOYCHARTSaaf9e97d6f52: RESTRouter: Bump image tag to v1.1.2 and release v0.0.7 (authored by mobrovac).
RESTRouter: Bump image tag to v1.1.2 and release v0.0.7
Fri, Oct 4, 1:36 PM
mobrovac added a comment to T232182: Parsoid/PHP performance benchmarking on scandium.

If colocated, just start with 1% of the traffic being dual-served to the new system and ramp it up gradually? If the worry is that the new service would deplete resources on the machine.

That was the plan when we started sending live *edit* client traffic to the clusters, but that was from a correctness POV so we verify correctness before increasing traffic. But, we could possibly do that with this initial shadow deployment as well. Depends what is involved in changeprop to enable that. @Pchelolo might know.

Fri, Oct 4, 12:45 PM · Performance-Team (Radar), Performance Issue, Parsoid-PHP
mobrovac closed T188255: Upgrade swagger-ui version in mathoid, a subtask of T218217: Make services swagger specs standard compliant, as Resolved.
Fri, Oct 4, 11:48 AM · Core Platform Team, serviceops-radar, Product-Infrastructure-Team-Backlog, Proton, Graphoid, CX-cxserver, Citoid, Mathoid, Recommendation-API, Services (later), Mobile-Content-Service, RESTBase-API, Operations
mobrovac closed T188255: Upgrade swagger-ui version in mathoid as Resolved.
Fri, Oct 4, 11:48 AM · Core Platform Team Workboards (Clinic Duty Team), Mathoid
mobrovac committed rDEPLOYCHARTS15330b80fbc4: restrouter: Revert the initialDelay seconds (authored by akosiaris).
restrouter: Revert the initialDelay seconds
Fri, Oct 4, 10:51 AM