mobrovac (Marko Obrovac)
Spy

Projects (46)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Dec 16 2014, 7:40 PM (217 w, 4 d)
Availability
Available
IRC Nick
mobrovac
LDAP User
Mobrovac
MediaWiki User
Mobrovac [ Global Accounts ]

Recent Activity

Yesterday

mobrovac updated the task description for T216191: Replace Istanbul with nyc.
Sat, Feb 16, 1:49 AM · Core Platform Team Kanban, Services (next)
mobrovac added a comment to T215956: Consider stashing data-parsoid for VE .

Correct me if I'm wrong, but I would think all VE traffic would already be uncacheable at the Varnish level anyways, since it happens in the context of a session.

Sat, Feb 16, 1:24 AM · Operations, Traffic, User-Eevans, User-mobrovac, Core Platform Team Backlog (Designing), Parsoid, VisualEditor, RESTBase, Services (designing)

Fri, Feb 15

mobrovac added a comment to T215956: Consider stashing data-parsoid for VE .

I don't think complicating Varnish logic worths here, I don't see how UA is better than a query parameter regarding DDOs.

Fri, Feb 15, 8:42 PM · Operations, Traffic, User-Eevans, User-mobrovac, Core Platform Team Backlog (Designing), Parsoid, VisualEditor, RESTBase, Services (designing)
mobrovac added a project to T215956: Consider stashing data-parsoid for VE : Traffic.

A possible alternative is to differentiate VE's requests based on the UA header: VE sets both Api-User-Agent and User-Agent (sample RB log entry), so it is possible to recognise its requests by checking both headers and matching them against VisualEditor-Mediawiki in both Varnish and RB. This strikes me a slightly better approach than a query parameter since with the latter we can easily suffer (D)DoS attacks. Mind you, checking for headers is only sightly better, but better nevertheless.

Fri, Feb 15, 8:10 PM · Operations, Traffic, User-Eevans, User-mobrovac, Core Platform Team Backlog (Designing), Parsoid, VisualEditor, RESTBase, Services (designing)

Wed, Feb 13

mobrovac triaged T216069: Flaky quibble-vendor-mysql-hhvm-docker test in Jenkins as High priority.
Wed, Feb 13, 8:36 PM · Analytics, Services (watching), Core Platform Team Backlog (Watching / External), EventBus, MediaWiki-Core-Testing, Release-Engineering-Team, Jenkins, Quibble
mobrovac closed T198125: MediaWiki webdriver afterTest hook should catch mwbot issues/exceptions as Resolved.

Oh, ok, just wanted to make sure. Will file a new task. Thanks @Krinkle !

Wed, Feb 13, 8:27 PM · User-zeljkofilipin, Browser-Tests, MediaWiki-Core-Testing
mobrovac added projects to T212129: Use a multi-dc aware store for ObjectCache's MainStash if needed.: Services (doing), User-mobrovac.
Wed, Feb 13, 12:57 AM · User-mobrovac, Services (doing), User-jijiki, Core Platform Team Kanban (Doing), Core Platform Team (Security, stability, performance and scalability (TEC1)), Performance-Team (Radar), Operations, MediaWiki-Cache, serviceops
mobrovac added a project to T215960: Simplify MCS storage model: Core Platform Team Backlog (Designing).
Wed, Feb 13, 12:56 AM · User-Eevans, Core Platform Team Backlog (Designing), Reading-Infrastructure-Team-Backlog, Services (designing), RESTBase
mobrovac added projects to T215956: Consider stashing data-parsoid for VE : Core Platform Team Backlog (Designing), User-mobrovac.
Wed, Feb 13, 12:55 AM · Operations, Traffic, User-Eevans, User-mobrovac, Core Platform Team Backlog (Designing), Parsoid, VisualEditor, RESTBase, Services (designing)
mobrovac added projects to T215883: Create puppet role for session storage service: Services (doing), Core Platform Team Kanban (Doing).
Wed, Feb 13, 12:51 AM · Core Platform Team Kanban (Doing), Services (doing), User-Eevans, Patch-For-Review, User-jijiki, Operations, serviceops, Core Platform Team (Session Management Service (CDP2))
mobrovac reopened T198125: MediaWiki webdriver afterTest hook should catch mwbot issues/exceptions as "Open".

It seems these problems are coming back around. For EventBus Gerrit 490141 we are seeing unrelated failures for jenkins tests which fit this ticket: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/34851/console and https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/34856/console . Is this known?

Wed, Feb 13, 12:42 AM · User-zeljkofilipin, Browser-Tests, MediaWiki-Core-Testing

Tue, Feb 12

mobrovac added projects to T209136: python3-etcd needs python3-dnspython: Services (watching), Core Platform Team Backlog (Watching / External).
Tue, Feb 12, 7:11 PM · Core Platform Team Backlog (Watching / External), Services (watching), Patch-For-Review, Operations, Operations-Software-Development

Mon, Feb 11

mobrovac added a comment to T205911: Track and install additional npm packages for all service container images.

Assuming we go for option (1), how would we go around and install these packages? And how would we instruct the app to load them?
It seems like using NODE_PATH is discouraged these days[1] and would anyway require changes to blubber to set NODE_PATH. We used to have that variable set and have moved away from it in https://gerrit.wikimedia.org/r/#/c/blubber/+/460997/

Mon, Feb 11, 9:33 PM · Release-Engineering-Team (Watching / External), Core Platform Team Backlog (Watching / External), Services (watching), Operations, Release Pipeline
mobrovac committed rMSCPa66f632c779d: Make sure deduplication has no false-positives. (authored by Pchelolo).
Make sure deduplication has no false-positives.
Mon, Feb 11, 7:02 PM
mobrovac committed rMSCPe63f7cbebce1: Minor typo in error messages (authored by awight).
Minor typo in error messages
Mon, Feb 11, 7:02 PM

Sat, Feb 9

mobrovac closed T215631: Spelling errors in RESTBase API error message for the "morelike" function as Resolved.

Fixed in master. Will be live with the next deployment of the Recommendation API service.

Sat, Feb 9, 12:11 AM · Services (watching), Core Platform Team Backlog (Watching / External), Research, Recommendation-API, RESTBase-API

Fri, Feb 8

mobrovac added a comment to T201409: Harmonise the identification of requests across our stack.

Defaulted to "never accept the incoming header" and reconfigured in Wikimedia's configuration, I agree that should work.

Fri, Feb 8, 11:47 PM · MW-1.33-notes (1.33.0-wmf.17; 2019-02-12), Patch-For-Review, Core Platform Team Backlog (Designing), User-CDanis, TechCom-RFC (TechCom-Approved), Performance-Team (Radar), Operations, Services (designing), User-mobrovac, Traffic
mobrovac added a comment to T215631: Spelling errors in RESTBase API error message for the "morelike" function.

are should (maybe) be is (It is one feature that is not enabled, right?)

Not sure about this one. The feature maybe only one, but recommendations are many. If you remove the word 'morelike' then 'are' makes sense to me. Any native speakers?

Fri, Feb 8, 8:51 PM · Services (watching), Core Platform Team Backlog (Watching / External), Research, Recommendation-API, RESTBase-API
mobrovac assigned T215631: Spelling errors in RESTBase API error message for the "morelike" function to bmansurov.
Fri, Feb 8, 8:30 PM · Services (watching), Core Platform Team Backlog (Watching / External), Research, Recommendation-API, RESTBase-API
mobrovac added projects to T215465: Development policy amendment: Require use of storage abstraction layers: Core Platform Team Backlog (Watching / External), Services (watching).
Fri, Feb 8, 12:04 AM · Services (watching), Core Platform Team Backlog (Watching / External), Performance-Team, TechCom

Thu, Feb 7

mobrovac added projects to T88445: MediaWiki active/active datacenter investigation and work (tracking): Core Platform Team Backlog (Epic), Core Platform Team (Security, stability, performance and scalability (TEC1)), Services (designing), User-mobrovac.
Thu, Feb 7, 11:19 PM · User-mobrovac, Services (designing), Core Platform Team (Security, stability, performance and scalability (TEC1)), Core Platform Team Backlog (Epic), Performance-Team (Radar), Availability (MediaWiki-MultiDC), Epic
mobrovac added a comment to T215557: Remove electron pdf endpoint backend in RESTBase.

@Pchelolo what's different in htis task as opposed to T210651: Switch all PDF render traffic to new Proton service ? Smells like a duplicate.

Thu, Feb 7, 10:01 PM · Services (next), Proton
mobrovac added a project to T215481: Add Erika to techcom@: TechCom.
Thu, Feb 7, 1:15 AM · TechCom
mobrovac triaged T215481: Add Erika to techcom@ as Normal priority.
Thu, Feb 7, 1:14 AM · TechCom
mobrovac awarded T215465: Development policy amendment: Require use of storage abstraction layers a Mountain of Wealth token.
Thu, Feb 7, 1:12 AM · Services (watching), Core Platform Team Backlog (Watching / External), Performance-Team, TechCom
mobrovac added a comment to T215320: Generate per-repo pipeline jobs.

Bonus points if these jobs could be created by someone adding a .pipeline/config.yaml to their project.

Thu, Feb 7, 1:00 AM · Release-Engineering-Team, Release Pipeline
mobrovac moved T215046: RfC: Use Github login for mediawiki.org from Inbox to Under discussion on the TechCom-RFC board.

The TechCom will be holding an IRC meeting about this RfC next Thursday, February 14 at 07:00 UTC (08:00 CET, Feb 13 23:00 PST).

Thu, Feb 7, 12:31 AM · User-Tgr, Privacy, Security, TechCom-RFC, Wikimedia-General-or-Unknown, GitHub-Mirrors
mobrovac added a comment to T213363: Fix or clarify Proton Puppeteer sandboxing and SSL settings.

We are currently using the projects' FQDNs in MW API requests becasue this was the only way to get the CSS we needed for the printable version (not sure why that is the case, but that's a problem for another day). So, for the time being, SSL certificates are being honoured by Chromium.

Thu, Feb 7, 12:24 AM · Reading-Infrastructure-Team-Backlog (Kanban), Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q3), Proton

Wed, Feb 6

mobrovac moved T213505: RfC: OpenGraph descriptions in wiki pages from Inbox to Under discussion on the TechCom-RFC board.
Wed, Feb 6, 9:35 PM · MediaWiki-General-or-Unknown, TechCom-RFC, Reading-Infrastructure-Team-Backlog (Kanban)

Tue, Feb 5

mobrovac moved T138719: Make URL normalisation/shortening a core MediaWiki interface from Inbox to Backlog on the TechCom board.
Tue, Feb 5, 8:21 PM · TechCom, MediaWiki-extensions-UrlShortener, MediaWiki-General-or-Unknown
mobrovac updated subscribers of T213345: RFC: Spin off (Parsoid) language variants functionality as a Node.js microservice?.

@ssastry and I have briefly discussed this last week. The way the code is currently structured, it should be possible to easily separate it (and later build upon it independently). Also, no calls to MW are needed for pure lang variant conversions at the current stage, which makes this a nice and well-isolated feature. @ssastry @cscott could you confirm the above is true?

Tue, Feb 5, 8:13 PM · TechCom-RFC, serviceops, User-mobrovac, Core Platform Team Backlog (Designing), Services (designing), MediaWiki-Language-converter, Parsoid, Language-Team, Parsing-Team
mobrovac added a comment to T191231: RFC: Abstract schemas and schema changes.

Personally, option (1) seems like a clear winner here. If done correctly, it would provide the correct abstraction layer around relational data, which would allow a way to provide support for arbitrary RDBMS systems. Moreover, in the context of making cleaner and clearer interfaces in mw-core, going with option (1) would do us a lot of good in terms of reassessing internal interfaces and those exposed to extensions and to external entities.

Tue, Feb 5, 6:50 PM · User-Addshore, Core Platform Team (Code Health (TEC13)), Core Platform Team Backlog (Epic), TechCom-RFC, SQLite, Oracle Database, MSSQL, PostgreSQL, MediaWiki-Database, Epic
mobrovac added projects to T215222: Recommendation API translation endpoint stopped working: Core Platform Team Backlog (Watching / External), Core Platform Team (Security, stability, performance and scalability (TEC1)).
Tue, Feb 5, 5:21 PM · Core Platform Team (Security, stability, performance and scalability (TEC1)), Core Platform Team Backlog (Watching / External), Services (watching), Recommendation-API, Research
mobrovac edited projects for T214706: Surface link changes as a stream, added: Core Platform Team Kanban (Blocked Externally); removed Core Platform Team Kanban (Doing).
Tue, Feb 5, 5:14 PM · Core Platform Team Kanban (Done with CPT), Services (done), MW-1.33-notes (1.33.0-wmf.17; 2019-02-12), Core Platform Team (Modern Event Platform (TEC2)), EventBus, Wikidata, Analytics, Research

Mon, Feb 4

mobrovac triaged T215220: Zlib spec.yaml example produces a bad request response as Normal priority.
Mon, Feb 4, 11:15 PM · Mathoid
mobrovac added a comment to T214874: Investigate a Hyperswitch frontend for ORES.

If I understand this task correctly, it seems to be a duplicate of T107196: Set up revscoring entry points in RESTBase.

Mon, Feb 4, 10:32 PM · Services (designing), Scoring-platform-team, ORES
mobrovac merged T214873: Investigate using Cassandra for the score cache into T166161: Use RESTBase for ORES precaching.
Mon, Feb 4, 10:30 PM · Core Platform Team Backlog (Later), Services (later), RESTBase-API, RESTBase, Scoring-platform-team, ORES
mobrovac merged task T214873: Investigate using Cassandra for the score cache into T166161: Use RESTBase for ORES precaching.
Mon, Feb 4, 10:30 PM · Scoring-platform-team, ORES, Cassandra

Sun, Jan 27

mobrovac removed a project from T214802: Enable and use or merge results from zotero ISBN search to improve ISBN results: RESTBase.
Sun, Jan 27, 7:57 PM · Patch-For-Review, Citoid
mobrovac closed T211411: Citoid automated monitoring times out due to Zotero v2 as Resolved.

There have been no timeouts recorded by the automatic check scripts since the deploy, so looking good. Resolving for now, let's reopen if things change.

Sun, Jan 27, 7:49 PM · serviceops, Prod-Kubernetes, Core Platform Team Backlog (Watching / External), Services (watching), Citoid, Operations
mobrovac added a comment to T213505: RfC: OpenGraph descriptions in wiki pages.

Another alternative could be to inject the tags at the edge, before the HTML is cached. The rough workflow could be something like:

  • a page is edited
  • the parser reparses it
  • Parsoid/PCS reconstruct their parts
  • when a page is requested, if the HTML is present at the edge cache, serve it; otherwise ask for the HTML and summary and inject the needed tags.
Sun, Jan 27, 7:42 PM · MediaWiki-General-or-Unknown, TechCom-RFC, Reading-Infrastructure-Team-Backlog (Kanban)

Wed, Jan 23

mobrovac updated subscribers of T212418: Memory error on restbase1016.

All of the instances have joined the ring (thnx @fgiunchedi!) and the latest version of RESTBase is in place, so we are good. There is one problem, now, though: I can't seem to be able to pool the node back. Let's try and see what this is about before resolving the ticket.

Indeed neither can I:

restbase1016:~$ pool-restbase 
restbase1016:~$ echo $?
2

Though I'm not going to have time to investigate further this week, any help is welcome

Wed, Jan 23, 10:28 PM · Patch-For-Review, Core Platform Team Backlog (Watching / External), Services (watching), RESTBase-Cassandra, RESTBase, Operations
mobrovac added a comment to T206889: Develop a library for JSON schema backwards incompatibility detection.

Given that we are intending to check compatibility between all schema versions, *_TRANSITIVE is not really important for us.

Wed, Jan 23, 7:26 PM · Core Platform Team Backlog (Watching / External), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics

Tue, Jan 22

mobrovac added a project to T213566: Transferring data from Hadoop to production MySQL database: Core Platform Team Backlog (Watching / External).
Tue, Jan 22, 8:22 PM · serviceops, Core Platform Team Backlog (Watching / External), Services (watching), User-Marostegui, Operations, Article-Recommendation, Analytics, Research
mobrovac added a comment to T213566: Transferring data from Hadoop to production MySQL database.

I agree that the most likely solution to work here is option (2), i.e. getting a host to execute it from. Perhaps it could be done from mwmaint1002, but that would need to be checked with SREs.

Tue, Jan 22, 8:22 PM · serviceops, Core Platform Team Backlog (Watching / External), Services (watching), User-Marostegui, Operations, Article-Recommendation, Analytics, Research
mobrovac added a comment to T206889: Develop a library for JSON schema backwards incompatibility detection.
  1. Language. I propose node as we already have a little bit of code in node that does it, plus I'm more familiar with node. Since this only will run in CI for testing, performance of this doesn't matter.
Tue, Jan 22, 6:59 PM · Core Platform Team Backlog (Watching / External), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
mobrovac added projects to T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage: Services (designing), Core Platform Team Backlog (Designing).
Tue, Jan 22, 5:12 PM · Core Platform Team Backlog (Designing), Services (designing), User-mobrovac, wikidata-tech-focus, TechCom-RFC, Wikibase-Quality, Wikidata
mobrovac added a project to T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage: User-mobrovac.
Tue, Jan 22, 5:11 PM · Core Platform Team Backlog (Designing), Services (designing), User-mobrovac, wikidata-tech-focus, TechCom-RFC, Wikibase-Quality, Wikidata

Sat, Jan 19

mobrovac added a comment to T212418: Memory error on restbase1016.

All of the instances have joined the ring (thnx @fgiunchedi!) and the latest version of RESTBase is in place, so we are good. There is one problem, now, though: I can't seem to be able to pool the node back. Let's try and see what this is about before resolving the ticket.

Sat, Jan 19, 12:01 AM · Patch-For-Review, Core Platform Team Backlog (Watching / External), Services (watching), RESTBase-Cassandra, RESTBase, Operations

Fri, Jan 18

mobrovac edited projects for T214166: Improve cassandra JBOD integration post-reimage, added: Cassandra, Services (watching), Core Platform Team Backlog (Watching / External); removed Services.

That's a good idea @fgiunchedi ! +1

Fri, Jan 18, 5:09 PM · Core Platform Team Backlog (Watching / External), Services (watching), Cassandra, RESTBase-Cassandra

Jan 18 2019

mobrovac removed a project from T213900: Leaving out 'prefix' option breaks Visual Editor when Collection extension is installed: RESTBase.
Jan 18 2019, 2:31 AM · Collection, Parsoid, VisualEditor
mobrovac added a project to T214094: RESTBase should not respect Accept-Language for content version upgrades: Wikimedia-Incident.
Jan 18 2019, 2:29 AM · Wikimedia-Incident, Core Platform Team Backlog (Next), Services (next), RESTBase
mobrovac raised the priority of T214094: RESTBase should not respect Accept-Language for content version upgrades from Normal to High.
Jan 18 2019, 2:29 AM · Wikimedia-Incident, Core Platform Team Backlog (Next), Services (next), RESTBase
mobrovac triaged T214094: RESTBase should not respect Accept-Language for content version upgrades as Normal priority.
Jan 18 2019, 2:28 AM · Wikimedia-Incident, Core Platform Team Backlog (Next), Services (next), RESTBase
mobrovac added a project to T213362: Limit what URLs Proton can access: Security.
Jan 18 2019, 2:26 AM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Security, Core Platform Team Backlog (Watching / External), Services (watching), Proton
mobrovac added a comment to T204024: Store WikibaseQualityConstraint check data in persistent storage instead of in the cache.

will we store data only for the latest revision or not (implies different storage semantics and guarantees)

I agree with everything @Lucas_Werkmeister_WMDE said here, only store data for the latest revision.

Jan 18 2019, 2:21 AM · User-Addshore, Dependency-Tracking, Operations, Core Platform Team Backlog (Designing), Cassandra, Services (designing), wikidata-tech-focus, Wikidata-Campsite, Wikibase-Quality-Constraints, Wikibase-Quality, Wikidata
mobrovac added subtasks for T208087: Replace remaining Samsung SSDs: Unknown Object (Task), Unknown Object (Task).
Jan 18 2019, 2:08 AM · Core Platform Team (Security, stability, performance and scalability (TEC1)), Core Platform Team Backlog (Watching / External), User-Eevans, Cassandra

Jan 17 2019

mobrovac updated subscribers of T212418: Memory error on restbase1016.

So now we should be able to get restbase1016 back into the cluster. Since we need to re-bootstrap the instances in, we can either:

Jan 17 2019, 9:57 PM · Patch-For-Review, Core Platform Team Backlog (Watching / External), Services (watching), RESTBase-Cassandra, RESTBase, Operations
mobrovac moved T213345: RFC: Spin off (Parsoid) language variants functionality as a Node.js microservice? from Inbox to Radar on the TechCom board.
Jan 17 2019, 7:40 PM · TechCom-RFC, serviceops, User-mobrovac, Core Platform Team Backlog (Designing), Services (designing), MediaWiki-Language-converter, Parsoid, Language-Team, Parsing-Team
mobrovac edited projects for T213345: RFC: Spin off (Parsoid) language variants functionality as a Node.js microservice?, added: Services (designing), Core Platform Team Backlog (Designing), TechCom, User-mobrovac; removed Services (watching).
Jan 17 2019, 7:40 PM · TechCom-RFC, serviceops, User-mobrovac, Core Platform Team Backlog (Designing), Services (designing), MediaWiki-Language-converter, Parsoid, Language-Team, Parsing-Team

Jan 16 2019

mobrovac added a comment to T212424: restbase cassandra driver excessive logging when cassandra hosts are down.

As I understand, the driver does not recognize a node being marked as DOWN by Cassandra. I assume, 01-09 was when restbase1016 was force-removed from the cluster, but it was picked up by the driver only partially.

Jan 16 2019, 12:59 AM · Services (next), RESTBase-Cassandra, Operations

Jan 15 2019

mobrovac added a project to T213857: Evaluate cassandra node driver 4.0.0 for upgrade: RESTBase-Cassandra.
Jan 15 2019, 8:23 PM · RESTBase-Cassandra, RESTBase, Core Platform Team Backlog (Later), Services (later)
mobrovac edited projects for T213362: Limit what URLs Proton can access, added: Services (watching), Core Platform Team Backlog (Watching / External); removed Services.

As discussed in T177765#4867361, Proton should not have access to the internal Wikimedia network

Jan 15 2019, 5:52 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Security, Core Platform Team Backlog (Watching / External), Services (watching), Proton

Jan 9 2019

mobrovac added a comment to T213269: Requesting access to Citoid/Zotero production servers for MVOLZ.

Also, to clarify, are the groups being requested zotero-admin and citoid-admin?

Jan 9 2019, 3:00 PM · SRE-Access-Requests, Operations, Citoid

Jan 8 2019

mobrovac updated subscribers of T212945: Add Baha as a deployer for Recommendation API.

@herron manager approval received. Has the request been discussed during the SRE weekly meeting?

Jan 8 2019, 4:58 PM · Patch-For-Review, Research, SRE-Access-Requests, Core Platform Team Backlog (Watching / External), Services (watching), Recommendation-API, Operations
mobrovac updated the task description for T212945: Add Baha as a deployer for Recommendation API.
Jan 8 2019, 4:57 PM · Patch-For-Review, Research, SRE-Access-Requests, Core Platform Team Backlog (Watching / External), Services (watching), Recommendation-API, Operations
mobrovac added a comment to T212945: Add Baha as a deployer for Recommendation API.

@mobrovac @DarTar is on vacation. @leila is a stand-in manager. Would her approval be good?

Jan 8 2019, 4:52 PM · Patch-For-Review, Research, SRE-Access-Requests, Core Platform Team Backlog (Watching / External), Services (watching), Recommendation-API, Operations
mobrovac added a comment to T212418: Memory error on restbase1016.

ping @Cmjohnson

Jan 8 2019, 11:50 AM · Patch-For-Review, Core Platform Team Backlog (Watching / External), Services (watching), RESTBase-Cassandra, RESTBase, Operations
mobrovac triaged T213154: /api/rest_v1/page/pdf/* service unstable as Low priority.
Jan 8 2019, 11:37 AM · Core Platform Team Backlog (Attic), Services (attic), Electron-PDFs, Readers-Web-Backlog, RESTBase-API
mobrovac edited projects for T213154: /api/rest_v1/page/pdf/* service unstable, added: Electron-PDFs, Services (attic), Core Platform Team Backlog (Attic); removed OCG-PDFRenderer, Core Platform Team, Services, Proton.

This is a known and recurring issue where the electron service fails to respond to requests in time. I have restarted it as this usually helps, but in the long run we will be replacing in with Proton (which should happen this Q).

Jan 8 2019, 11:36 AM · Core Platform Team Backlog (Attic), Services (attic), Electron-PDFs, Readers-Web-Backlog, RESTBase-API
mobrovac edited projects for T192691: [Commons] A new image added to a category is not shown in Watchlist, added: Services (doing), Core Platform Team Kanban (Doing); removed Patch-For-Review, Core Platform Team Backlog (Next), Services (next).
Jan 8 2019, 9:14 AM · Services (done), TCB-Team, CatWatch, Growth-Team, MediaWiki-Categories, MediaWiki-Watchlist, Commons

Jan 7 2019

mobrovac updated the task description for T212945: Add Baha as a deployer for Recommendation API.
Jan 7 2019, 4:43 PM · Patch-For-Review, Research, SRE-Access-Requests, Core Platform Team Backlog (Watching / External), Services (watching), Recommendation-API, Operations

Jan 4 2019

mobrovac updated the task description for T212945: Add Baha as a deployer for Recommendation API.
Jan 4 2019, 3:47 PM · Patch-For-Review, Research, SRE-Access-Requests, Core Platform Team Backlog (Watching / External), Services (watching), Recommendation-API, Operations
mobrovac added a project to T212945: Add Baha as a deployer for Recommendation API: Research.
Jan 4 2019, 3:46 PM · Patch-For-Review, Research, SRE-Access-Requests, Core Platform Team Backlog (Watching / External), Services (watching), Recommendation-API, Operations
mobrovac added a comment to T212945: Add Baha as a deployer for Recommendation API.

@DarTar as Baha's manager, please review and approve this request.

Jan 4 2019, 3:45 PM · Patch-For-Review, Research, SRE-Access-Requests, Core Platform Team Backlog (Watching / External), Services (watching), Recommendation-API, Operations
mobrovac triaged T212945: Add Baha as a deployer for Recommendation API as Normal priority.
Jan 4 2019, 3:43 PM · Patch-For-Review, Research, SRE-Access-Requests, Core Platform Team Backlog (Watching / External), Services (watching), Recommendation-API, Operations

Jan 3 2019

mobrovac edited projects for T192691: [Commons] A new image added to a category is not shown in Watchlist, added: Services (next), Core Platform Team Backlog (Next); removed Services.

I took a quick look and it seems like there were quite a few categoryMembershipChange jobs that failed with the message Could not find page #<page_id>, which would suggest a race condition. However, the real surge in the number of log entries are for Old event processed for the same type of job. This could suggest a subtle bug in CP-JobQueue. Sample log entry:

Jan 3 2019, 11:26 AM · Services (done), TCB-Team, CatWatch, Growth-Team, MediaWiki-Categories, MediaWiki-Watchlist, Commons
mobrovac edited projects for T212801: TEC3:O3:O3.1:Q3 Goal - Move cxserver, citoid, changeprop, eventgate (new service) and ORES (partially) through the production CD Pipeline, added: Services (watching), Core Platform Team Backlog (Watching / External); removed Services.
Jan 3 2019, 9:05 AM · Core Platform Team Backlog (Watching / External), Services (watching), Release Pipeline, serviceops, Release-Engineering-Team

Dec 31 2018

mobrovac added projects to T212722: eslint-config-node-services has unmeet peer dependencies: Services (next), Core Platform Team Backlog (Next).
Dec 31 2018, 12:36 PM · Core Platform Team Backlog (Next), Services (next), good first bug, service-template-node

Dec 28 2018

mobrovac removed a project from T212631: Kademlia rate limiter failing unexpectedly: Patch-For-Review.
Dec 28 2018, 10:14 AM · Core Platform Team Kanban (Done with CPT), Services (done), RESTBase, service-runner

Dec 27 2018

mobrovac closed T212631: Kademlia rate limiter failing unexpectedly as Resolved.

RESTBase has been deployed. Resolving.

Dec 27 2018, 7:31 PM · Core Platform Team Kanban (Done with CPT), Services (done), RESTBase, service-runner
mobrovac added a comment to T212424: restbase cassandra driver excessive logging when cassandra hosts are down.

This is indeed a problem, causing issues also during deploys. I have had to increase the delay as well as the Scap deploy timeout because of the driver being too eager.

Dec 27 2018, 6:08 PM · Services (next), RESTBase-Cassandra, Operations
mobrovac added projects to T212418: Memory error on restbase1016: RESTBase, RESTBase-Cassandra, Services (watching), Core Platform Team Backlog (Watching / External).
Dec 27 2018, 5:33 PM · Patch-For-Review, Core Platform Team Backlog (Watching / External), Services (watching), RESTBase-Cassandra, RESTBase, Operations

Dec 26 2018

mobrovac claimed T212631: Kademlia rate limiter failing unexpectedly.
Dec 26 2018, 7:34 PM · Core Platform Team Kanban (Done with CPT), Services (done), RESTBase, service-runner
mobrovac edited projects for T212631: Kademlia rate limiter failing unexpectedly, added: Services (doing), Core Platform Team Kanban (Doing); removed Core Platform Team Backlog (Next), Services (next).

Ok, I tracked down the problem to the kad library. In order to have the fix in place, the following patches need to be merged (in this order):

Dec 26 2018, 7:33 PM · Core Platform Team Kanban (Done with CPT), Services (done), RESTBase, service-runner
mobrovac triaged T212631: Kademlia rate limiter failing unexpectedly as High priority.
Dec 26 2018, 6:17 PM · Core Platform Team Kanban (Done with CPT), Services (done), RESTBase, service-runner

Dec 21 2018

mobrovac assigned T212463: [REST API] unable to set accept-language for /page/summary/ endpoint to Pchelolo.
Dec 21 2018, 7:28 AM · Services (done), RESTBase-API

Dec 20 2018

mobrovac added a comment to T202001: Don't include requests from automated checks in stats.

Since all CItoid requests come from extrenal clients, we can perhaps separate the metrics into internal and external ones based on IPs. That way we could separate them in the graphs.

I'm not sure I like this solution because IPs can change, it seems like it would require some degree of upkeep?

Dec 20 2018, 4:27 PM · Core Platform Team Backlog (Later), Services (later), Citoid
mobrovac added a comment to T212189: New Service Request: Wikidata Termbox SSR.

That works, but defies the purpose. The idea is to present a default rendering to clients that don't have JS enabled (or no sufficiently current JS support). That rendering should be generated by the same vue.js code that does the rendering on the client.

Dec 20 2018, 1:39 PM · Core Platform Team Backlog (Later), User-Addshore, serviceops, Services (next), Wikidata-Termbox-Hike, Wikidata, Service-deployment-requests, Operations
mobrovac added a comment to T212189: New Service Request: Wikidata Termbox SSR.

But for the case at hand, there might be a workaround: the PHP code that renders the (Wikibase Entity) page content already knows what data will be needed for the rendering. It can send it to the rendering service along with the request to render. The vue.js code would then need to have a "fake repo request" facility that would just use data that was passed in with the original request, and would fail (or at least warn) when trying to load any additional content by calling the actual MediaWiki API. I think that solution would still be fairly clean, and would perform better than calling back to the MediaWiki API all the time.

Dec 20 2018, 1:18 PM · Core Platform Team Backlog (Later), User-Addshore, serviceops, Services (next), Wikidata-Termbox-Hike, Wikidata, Service-deployment-requests, Operations
mobrovac added a project to T144431: RESTBase k-r-v as Cassandra anti-pattern: Core Platform Team Backlog (Attic).
Dec 20 2018, 1:01 PM · Core Platform Team Backlog (Attic), Services (attic), Operations, Cassandra, RESTBase
mobrovac added a project to T121240: Network isolation for production and semi-production services: Core Platform Team Backlog (Attic).
Dec 20 2018, 1:01 PM · Core Platform Team Backlog (Attic), Security, Services (attic), Security-General, Operations
mobrovac added a project to T159998: Investigate "no content returned" errors: Core Platform Team Backlog (Attic).
Dec 20 2018, 1:01 PM · Core Platform Team Backlog (Attic), Services (attic), RESTBase
mobrovac added a project to T100099: Meeting: Automatic deployment of backend services on beta cluster: Core Platform Team Backlog (Attic).
Dec 20 2018, 1:01 PM · Core Platform Team Backlog (Attic), Services (attic), Release-Engineering-Team, Epic, Wikimedia-Hackathon-2015, Beta-Cluster-Infrastructure
mobrovac added a project to T134461: Evaluate increased memtable_cleanup_threshold values: Core Platform Team Backlog (Attic).
Dec 20 2018, 1:01 PM · Core Platform Team Backlog (Attic), User-Eevans, Services (attic), Cassandra
mobrovac added a project to T140049: Maximum number of redirects reached / error message returned even when doi is available: Core Platform Team Backlog (Attic).
Dec 20 2018, 1:01 PM · Core Platform Team Backlog (Attic), Services (attic), Citoid
mobrovac added a project to T103445: Content platform discussion: use cases: Core Platform Team Backlog (Attic).
Dec 20 2018, 1:01 PM · Core Platform Team Backlog (Attic), Services (attic)
mobrovac added a project to T122825: Service Ownership and Maintenance: Core Platform Team Backlog (Attic).
Dec 20 2018, 1:01 PM · Core Platform Team Backlog (Attic), TechCom, User-mobrovac, Services (attic), Operations
mobrovac added a project to T118028: Consider using URL compression with pre-defined dictionary to shorten URLs; possibly use this instead of RESTBase POST storage: Core Platform Team Backlog (Designing).
Dec 20 2018, 12:55 PM · Core Platform Team Backlog (Designing), Services (designing), Performance, RESTBase
mobrovac added a project to T118128: RFC: Simplify stashing with a shared key-value stash table / bucket: Core Platform Team Backlog (Designing).
Dec 20 2018, 12:55 PM · Core Platform Team Backlog (Designing), Cassandra, Services (designing), RESTBase