mobrovac (Marko Obrovac)
Spy

Projects (47)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Dec 16 2014, 7:40 PM (200 w, 3 d)
Availability
Available
IRC Nick
mobrovac
LDAP User
Mobrovac
MediaWiki User
Mobrovac [ Global Accounts ]

Recent Activity

Today

mobrovac edited projects for T207416: Upgrade Puppeteer to 1.9.0, added: Services (blocked); removed Services, User-notice, Service-deployment-requests, Electron-PDFs, Operations.
Fri, Oct 19, 11:19 AM · Services (blocked), Patch-For-Review, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), Proton
mobrovac edited projects for T207329: Clear watchlist on enwiki only removes 50 items at a time, added: Services (blocked), Core Platform Team Kanban (Blocked Externally); removed Core Platform Team Backlog (Later), Services (next).

The above patch should mitigate the problem, however, we need to also account for possible clock drift between our servers. The more drift we tolerate, the less efficient our deduplication becomes, so I'm wondering whether we have any data on the possible clock drift to help make a decision on the exact number to tolerate?

Fri, Oct 19, 11:09 AM · MW-1.33-notes (1.33.0-wmf.1; 2018-10-23), Core Platform Team Kanban (Blocked Externally), Services (blocked), Patch-For-Review, Analytics, EventBus, WMF-JobQueue, Wikimedia-production-error, MediaWiki-Watchlist, Growth-Team
mobrovac edited projects for T199264: Properly setup fonts-config on Proton servers, added: Services (doing), Core Platform Team Kanban (Doing); removed Services (blocked).
Fri, Oct 19, 11:03 AM · Core Platform Team Kanban (Doing), Services (doing), Patch-For-Review, User-Jdlrobson, Proton

Yesterday

mobrovac lowered the priority of T203588: Feed checks timeout on RESTBase deploy from High to Normal.

While PR 1072 did drastically improve the performance of the onthisday end point, RB deployments were still failing (I managed to fully deploy RB after 5 attempts). Therefore, I opted to disable the check for the time being. There were no deployment failures afterwords.

Thu, Oct 18, 11:58 AM · Core Platform Team Kanban (Doing), Reading-Infrastructure-Team-Backlog, Mobile-Content-Service, Services (doing), RESTBase
mobrovac added a project to T207329: Clear watchlist on enwiki only removes 50 items at a time: Core Platform Team Backlog (Later).
Thu, Oct 18, 10:34 AM · MW-1.33-notes (1.33.0-wmf.1; 2018-10-23), Core Platform Team Kanban (Blocked Externally), Services (blocked), Patch-For-Review, Analytics, EventBus, WMF-JobQueue, Wikimedia-production-error, MediaWiki-Watchlist, Growth-Team
mobrovac added a comment to T202642: Investigate how to fix the performance problems caused by CPU bound work on the MCS services.

Just saw https://github.com/wikimedia/restbase/pull/1074, and am wondering if that is enough to cover also this task or should we continue promisifying CPU intensive tasks?

Thu, Oct 18, 10:27 AM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Readers-Web-Backlog (Tracking), Services (watching), Mobile-Content-Service

Wed, Oct 17

mobrovac closed T207263: Scap not restarting Proton as Invalid.

Ah, how did I miss this? Oh, because I was looking at my local copy of the deploy repo. Thank you @thcipriani for pointing out the obvious and sorry for wasting your time.

Wed, Oct 17, 8:20 PM · Proton, Core Platform Team Backlog (Watching / External), Services (watching), Scap
mobrovac triaged T207324: RESTBase adds duplicated `accept-language` entry in Vary header as Normal priority.
Wed, Oct 17, 8:13 PM · Core Platform Team Backlog (Later), goodfirstbug, RESTBase, Services (later)
mobrovac added a comment to T205452: Setup access from service to mysql.

Looks like this is done, @mobrovac?

Wed, Oct 17, 2:53 PM · Patch-For-Review, Core Platform Team Kanban (Doing), Services (doing), Recommendation-API, SCB, Operations, Research
mobrovac added a comment to T205911: Track and install additional npm packages for all service container images.

These are fairly static node packages which really need to be recompiled only when the Node version changes, so I would vote for option (1).

Wed, Oct 17, 9:03 AM · Core Platform Team Backlog (Watching / External), Services (watching), Release-Engineering-Team, Operations, Release Pipeline
mobrovac added a comment to T201158: [4hrs] Have a Grafana dashboard for Proton.

I did fix the latency graph: https://grafana.wikimedia.org/dashboard/db/proton?orgId=1&from=now-3h&to=now

However, it uncovered some even bigger issue - all I needed to do is to restart the service manuall via sudo service proton restart. Doing scap deploy --service-restart -f for some reason had no effect.

Wed, Oct 17, 8:51 AM · Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), monitoring, Proton, Services (watching)
mobrovac triaged T207263: Scap not restarting Proton as High priority.
Wed, Oct 17, 8:49 AM · Proton, Core Platform Team Backlog (Watching / External), Services (watching), Scap

Tue, Oct 16

mobrovac committed rMSRA3c04fd3082bb: Add MySQL connection info (authored by mobrovac).
Add MySQL connection info
Tue, Oct 16, 3:17 PM
mobrovac committed rESCDa65705909da0: Update proton to 42250bc (authored by mobrovac).
Update proton to 42250bc
Tue, Oct 16, 3:02 PM
mobrovac closed T203276: re-enable proton checks, a subtask of T186748: [EPIC] New service request: chromium-render/deploy, as Resolved.
Tue, Oct 16, 2:56 PM · Epic, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), User-notice, Services (blocked), Service-deployment-requests, Proton, Electron-PDFs, Operations
mobrovac closed T203276: re-enable proton checks as Resolved.

We have downgraded Puppeteer and now all seems good, so I re-enabled the checks.

Tue, Oct 16, 2:56 PM · Services (done), Proton, Electron-PDFs
mobrovac committed rMSRD0198cd8f00c9: Config: Add MySQL connection info (authored by mobrovac).
Config: Add MySQL connection info
Tue, Oct 16, 12:59 PM
mobrovac claimed T205452: Setup access from service to mysql.
Tue, Oct 16, 10:12 AM · Patch-For-Review, Core Platform Team Kanban (Doing), Services (doing), Recommendation-API, SCB, Operations, Research
mobrovac edited projects for T205452: Setup access from service to mysql, added: Services (doing), Core Platform Team Kanban (Doing); removed Core Platform Team Backlog (Later), Services (next).

For the firewall, I need to know the source (mysql client) ips.

Tue, Oct 16, 10:12 AM · Patch-For-Review, Core Platform Team Kanban (Doing), Services (doing), Recommendation-API, SCB, Operations, Research

Mon, Oct 15

mobrovac removed a project from T124196: Notice: Unable to unserialize job_params in some CirrusSearch jobs (when using JobQueueDB): WMF-JobQueue.

Removing WMF-JobQueue as we don't use JobQueueDB in production.

Mon, Oct 15, 5:15 PM · MediaWiki-JobQueue, User-aude, Discovery-Search, Wikidata, Discovery, CirrusSearch
mobrovac added projects to T201004: Spec out dependency engine interface, data structure, and states: Core Platform Team (Modern Event Platform (TEC2)), Core Platform Team Backlog (Designing).
Mon, Oct 15, 4:50 PM · Core Platform Team Backlog (Designing), Core Platform Team (Modern Event Platform (TEC2)), Dependency-Tracking, Services (designing), User-mobrovac, User-Daniel
mobrovac edited projects for T203278: Archive Mathoid deploy repo, added: Services (watching), Core Platform Team Backlog (Watching / External), Mathoid; removed Services.

Note that the deploy repo is actually still used as (a) it is still deployed on SCB; and (b) it is used for deployments in Beta. We should hold off removing it until we have removed it from SCB and found a way to have it in Beta.

Mon, Oct 15, 4:47 PM · Mathoid, Core Platform Team Backlog (Watching / External), Services (watching), GitHub-Mirrors, Repository-Admins, Cleanup, Release-Engineering-Team (Kanban), Release Pipeline (Blubber)
mobrovac edited projects for T203239: Create Debian packages for Node.js 10 upgrade, added: Core Platform Team Backlog (Watching / External); removed Patch-For-Review.
Mon, Oct 15, 4:43 PM · Core Platform Team Backlog (Watching / External), Readers-Web-Backlog (Tracking), Services (watching), Operations
mobrovac removed a project from T201946: Paragraph breaks returned by text API requests have changed: Services (watching).
Mon, Oct 15, 4:38 PM · Readers-Web-Backlog (Tracking), TextExtracts, MediaWiki-API
mobrovac added projects to T204754: HyperSwitch parameter validation allows empty string params: Services (later), Core Platform Team Backlog (Later).
Mon, Oct 15, 4:31 PM · Core Platform Team (Security, stability, performance and scalability (TEC1)), Core Platform Team Backlog (Later), Services (later), goodfirstbug, RESTBase, HyperSwitch
mobrovac added a project to T199189: [2.3] External links/references event stream: Core Platform Team Backlog (Watching / External).
Mon, Oct 15, 4:29 PM · Core Platform Team Backlog (Watching / External), Services (watching), Knowledge-Integrity, Epic
mobrovac added a project to T203854: Expand usage of x-triggered-by: Core Platform Team Backlog (Later).
Mon, Oct 15, 4:28 PM · Core Platform Team (Security, stability, performance and scalability (TEC1)), Core Platform Team Backlog (Later), goodfirstbug, service-template-node, Services (later), Reading-Infrastructure-Team-Backlog, ChangeProp, Mobile-Content-Service, RESTBase
mobrovac removed a project from T204713: Provide a Wikibase instance where we can import Wikitionaries materials and that can be queried from Wiktionaries: Service-Architecture.
Mon, Oct 15, 4:24 PM · Wiktionary, Wikidata
mobrovac added a project to T157132: Add ORES articlequality data to summaries?: Core Platform Team Backlog (Designing).
Mon, Oct 15, 4:20 PM · Core Platform Team Backlog (Designing), Scoring-platform-team, Reading Epics (Page Previews), Reading-Infrastructure-Team-Backlog, ORES, Services (designing), RESTBase-API, Mobile-Content-Service
mobrovac added a project to T202300: RESTBase passes through the user-agent: Core Platform Team Backlog (Later).
Mon, Oct 15, 2:14 PM · Core Platform Team (Security, stability, performance and scalability (TEC1)), Core Platform Team Backlog (Later), RESTBase, goodfirstbug, Services (later)
mobrovac edited projects for T201384: mobile-html for offline: versioned CSS and JS, added: Services (watching), Core Platform Team Backlog (Watching / External); removed Services.
Mon, Oct 15, 2:11 PM · Core Platform Team Backlog (Watching / External), Services (watching), Reading-Infrastructure-Team-Backlog (Kanban), Page Content Service, Reading Epics (Platform JS CSS and HTML consolidation)
mobrovac added a project to T67117: Hovercards: Display popups across wikis (interwiki popups): Core Platform Team Backlog (Watching / External).
Mon, Oct 15, 2:05 PM · Core Platform Team Backlog (Watching / External), Services (watching), Readers-Web-Backlog (Design), Page-Previews
mobrovac added a project to T205911: Track and install additional npm packages for all service container images: Core Platform Team Backlog (Watching / External).
Mon, Oct 15, 2:04 PM · Core Platform Team Backlog (Watching / External), Services (watching), Release-Engineering-Team, Operations, Release Pipeline
mobrovac added a project to T205919: TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline: Core Platform Team Backlog (Watching / External).
Mon, Oct 15, 2:01 PM · Core Platform Team Backlog (Watching / External), Services (watching), Release Pipeline, Operations, Release-Engineering-Team
mobrovac added a project to T206259: Open questions when deploying major version bumps to Parsoid's HTML: Core Platform Team Backlog (Watching / External).
Mon, Oct 15, 1:59 PM · Core Platform Team Backlog (Watching / External), Services (watching), RESTBase, Parsoid
mobrovac added a comment to T165105: Some requests for DOIs are failing or very slow; if we have a DOI and the request is taking too long, just use CrossRef data instead..

It takes 75 secs for Citoid in production to answer, so it seems like this is a problem either between Citoid and Zotero or Citoid and Crossref. We need to investigate more why that is.

Mon, Oct 15, 1:56 PM · User-Ryasmeen, Services (watching), Operations, User-mobrovac, Citoid, VisualEditor
mobrovac added a project to T205452: Setup access from service to mysql: Core Platform Team Backlog (Later).
Mon, Oct 15, 1:36 PM · Patch-For-Review, Core Platform Team Kanban (Doing), Services (doing), Recommendation-API, SCB, Operations, Research
mobrovac added a project to T204024: Store WikibaseQualityConstraint check data in an SQL table instead of in the cache: Core Platform Team Backlog (Designing).
Mon, Oct 15, 1:23 PM · Core Platform Team Backlog (Designing), Cassandra, Services (designing), wikidata-tech-focus, Wikidata-Campsite, Wikibase-Quality-Constraints, Wikibase-Quality, Wikidata
mobrovac added a project to T204031: Deploy regular running of wikidata constraint checks using the job queue: Core Platform Team Backlog (Watching / External).
Mon, Oct 15, 1:17 PM · Core Platform Team Backlog (Watching / External), Services (watching), WMF-JobQueue, wikidata-tech-focus, Wikibase-Quality-Constraints, Wikibase-Quality, Wikidata
mobrovac added projects to T206076: Wikimedia Technical Conference 2018 Session - Identifying our storage and search use cases: Core Platform Team (Storage API Specification (CDP2)), Core Platform Team Kanban (Doing).
Mon, Oct 15, 1:10 PM · Core Platform Team Kanban (Doing), Core Platform Team (Storage API Specification (CDP2)), Services (doing), User-mobrovac, Wikimedia-Technical-Conference-2018
mobrovac added projects to T206076: Wikimedia Technical Conference 2018 Session - Identifying our storage and search use cases: User-mobrovac, Services (doing).
Mon, Oct 15, 1:09 PM · Core Platform Team Kanban (Doing), Core Platform Team (Storage API Specification (CDP2)), Services (doing), User-mobrovac, Wikimedia-Technical-Conference-2018
mobrovac added a comment to T202642: Investigate how to fix the performance problems caused by CPU bound work on the MCS services.

It is normal to wither see no changes in number or to see slightly higher numbers when the process is promise-heavier. However, we are here interested in the performance of other requests when one request holds the event loop. In this case, results for other requests should be better. It is acceptable to have slightly higher latencies for the CPU-heavy request if other requests' latencies improve.

Mon, Oct 15, 12:57 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Readers-Web-Backlog (Tracking), Services (watching), Mobile-Content-Service
mobrovac edited projects for T206824: Decide whether to use schema references in the schema registry, added: Services (designing); removed Services (watching).

I wholeheartedly agree that doing copy/pasta for shared parts of the schemae is a bad design choice.

Mon, Oct 15, 12:25 PM · Core Platform Team Backlog (Designing), Services (designing), Core Platform Team (Modern Event Platform (TEC2)), Analytics-EventLogging, EventBus, Analytics
mobrovac added a comment to T206889: Develop a library for JSON schema backwards incompatibility detection.

We could perhaps hack around it in the following way. Each time a new version of the schema is created, a sample message also needs to be committed. That message can then be checked with existing JSONSchema validators against both the new and the old version of the schema.

Mon, Oct 15, 11:45 AM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
mobrovac closed T203135: ThumbnailRender job fails with 429 errors as Resolved.
Mon, Oct 15, 10:54 AM · Core Platform Team Kanban (Done with CPT), Services (done), MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Thumbor, WMF-Blog-Social-Team, Multimedia
mobrovac closed T206277: Revision visibility change event sets a wrong performer as Resolved.
Mon, Oct 15, 10:45 AM · MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)), Services (done), Core Platform Team Kanban (Done with CPT), Analytics, EventBus

Sun, Oct 14

mobrovac triaged T203588: Feed checks timeout on RESTBase deploy as High priority.

Raising priority to high as this has started happening more and more often, even during normal RESTBase operation.

Sun, Oct 14, 11:01 AM · Core Platform Team Kanban (Doing), Reading-Infrastructure-Team-Backlog, Mobile-Content-Service, Services (doing), RESTBase

Mon, Oct 8

mobrovac edited projects for T206314: Modify scap::target to define sudo rules for multiple services, added: Services (watching), Release-Engineering-Team; removed Release, Services.
Mon, Oct 8, 7:48 AM · Release-Engineering-Team, Services (watching), Patch-For-Review, Discovery-Search (Current work), Scap, Operations
mobrovac closed T206303: Add sudo rules for wdqs-updater in puppet as Resolved.
Mon, Oct 8, 7:36 AM · Services (done), Discovery-Search (Current work), Wikidata-Query-Service, Wikidata, Operations

Fri, Oct 5

mobrovac awarded T206268: Evaluate using TypeScript on node projects a 100 token.
Fri, Oct 5, 9:21 AM · Services (watching), Analytics
mobrovac removed a project from T195392: Switch cronjobs on maintenance hosts to PHP7: WMF-JobQueue.
Fri, Oct 5, 9:19 AM · Core Platform Team Backlog (Watching / External), User-ArielGlenn, Operations
mobrovac triaged T206303: Add sudo rules for wdqs-updater in puppet as Normal priority.
Fri, Oct 5, 8:16 AM · Services (done), Discovery-Search (Current work), Wikidata-Query-Service, Wikidata, Operations
mobrovac renamed T206303: Add sudo rules for wdqs-updater in puppet from Add wdqs-updater to scap target in puppet to Add sudo rules for wdqs-updater in puppet.
Fri, Oct 5, 8:16 AM · Services (done), Discovery-Search (Current work), Wikidata-Query-Service, Wikidata, Operations

Thu, Oct 4

mobrovac committed rMSCD5d004482e674: Update change-propagation to 47a554e (authored by mobrovac).
Update change-propagation to 47a554e
Thu, Oct 4, 4:56 PM
mobrovac committed rMSCP47a554ede02d: Don't attempt to commit if we are disconnected (authored by Pchelolo).
Don't attempt to commit if we are disconnected
Thu, Oct 4, 3:40 PM
mobrovac updated subscribers of T206222: Make "about" attribute IDs deterministic.
Thu, Oct 4, 2:21 PM · Parsoid
mobrovac edited projects for T206003: Beta Cluster: Parsoid config request failures from the MediaWiki API, added: Services (done); removed Patch-For-Review, Services (blocked).
Thu, Oct 4, 2:03 PM · Services (done), Parsoid, Beta-Cluster-Infrastructure
mobrovac closed T206003: Beta Cluster: Parsoid config request failures from the MediaWiki API as Resolved.

Yup, that did the trick! Thank you, @Arlolra !

Thu, Oct 4, 1:14 PM · Services (done), Parsoid, Beta-Cluster-Infrastructure
mobrovac added a comment to T205981: Add Mathew.onipe(onimisionipe) to deployment group.

I can confirm that @Mathew.onipe needs to be able to deploy wikidata query service. the deploy-service group seems to be the right one, but I'd like someone who understand better how scap3 deployments could confirm.

Thu, Oct 4, 1:02 PM · Patch-For-Review, Product-Analytics, SRE-Access-Requests, Operations, Discovery-Analysis (Current work)
mobrovac reopened T206003: Beta Cluster: Parsoid config request failures from the MediaWiki API as "Open".

The config still doesn't seem to be working for the edge case of en.wp.org:

Thu, Oct 4, 11:56 AM · Services (done), Parsoid, Beta-Cluster-Infrastructure

Wed, Oct 3

mobrovac committed rMSCP41ff54c849bd: Properly resubscribe when the list of topics has changed. (authored by Pchelolo).
Properly resubscribe when the list of topics has changed.
Wed, Oct 3, 10:04 PM
mobrovac committed rMSCPc03bff332761: Respect readonly mode (authored by Pchelolo).
Respect readonly mode
Wed, Oct 3, 9:24 PM
mobrovac merged T206107: Parsoid misconfigured in Beta into T206003: Beta Cluster: Parsoid config request failures from the MediaWiki API.
Wed, Oct 3, 2:27 PM · Services (done), Parsoid, Beta-Cluster-Infrastructure
mobrovac merged task T206107: Parsoid misconfigured in Beta into T206003: Beta Cluster: Parsoid config request failures from the MediaWiki API.
Wed, Oct 3, 2:27 PM · Beta-Cluster-Infrastructure, RESTBase, Services (watching), Parsoid
mobrovac edited projects for T206003: Beta Cluster: Parsoid config request failures from the MediaWiki API, added: Services (blocked); removed Services (watching).

The error, as seen by RESTBase:

Wed, Oct 3, 2:27 PM · Services (done), Parsoid, Beta-Cluster-Infrastructure
mobrovac added a comment to T205497: [Regression pre-wmf.23] REST API on Beta cluster returns content of different pages than requested (breaks VE).

@Ryasmeen try now

The second page is loading now with VE, but the first page still doesn't for some reason.

Wed, Oct 3, 11:23 AM · User-Ryasmeen, Services (done), VisualEditor, RESTBase, Parsoid
mobrovac added a comment to T197242: Transition citoid to use Zotero's translation-server-v2.

I noticed that a recent translators update was abandoned. From a comment to that patch, it sounded like there will be no more updates from Zotero's translator repo until this task is done. If that's correct, is there any estimate of when that will be? I have just gotten a few more translators for Swedish news sites added to Zotero's repo and would like to get them working with Citoid (T204467).

I'm afraid I can't give you much better than "soon" but if you wish to follow along, this is dependant on T201611 being done first.

Wed, Oct 3, 11:20 AM · VisualEditor (Current work), Patch-For-Review, Citoid, Services (watching), Operations
mobrovac added a comment to T206048: Eliminate Parsoid section.offsets table from Cassandra.

I agree. On the one hand, section offsets are not nearly enough requested to warrant their own table, and on the other, RB workers are currently under-utilised so we can handle the extra CPU needed to extract them from data-parsoid.

Wed, Oct 3, 11:18 AM · goodfirstbug, Services (later), RESTBase
mobrovac added a project to T205943: RESTBase hacks for compatibility to Parsoid: Services (watching).
Wed, Oct 3, 10:54 AM · Services (watching), Parsoid, RESTBase
mobrovac awarded T205943: RESTBase hacks for compatibility to Parsoid a Mountain of Wealth token.
Wed, Oct 3, 10:54 AM · Services (watching), Parsoid, RESTBase
mobrovac added a comment to T201158: [4hrs] Have a Grafana dashboard for Proton.

@mobrovac @Pchelolo could you update the proton to the latest version (include the html2pdf changes)

Wed, Oct 3, 10:46 AM · Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), monitoring, Proton, Services (watching)
mobrovac added a comment to T199264: Properly setup fonts-config on Proton servers.
<?xml version='1.0'?>
<!DOCTYPE fontconfig SYSTEM 'fonts.dtd'>
<fontconfig>
[...]
</fontconfig>
Wed, Oct 3, 10:40 AM · Core Platform Team Kanban (Doing), Services (doing), Patch-For-Review, User-Jdlrobson, Proton
mobrovac added a project to T199444: ChangeProp logging KafkaConsumer is not connected: WMF-JobQueue.
Wed, Oct 3, 10:24 AM · Services (done), WMF-JobQueue, ChangeProp, Wikimedia-Incident, EventBus, Analytics
mobrovac edited projects for T94522: Some requests fail with UIDGenerator error "Process clock is outdated or drifted", added: Services (designing), WMF-JobQueue; removed Services (doing).

Time drifting is a known occurrence. From what I see, we have two options:

  • allow it to happen and risk duplicate v1 UUIDs: the collision rate would vary with the drift, but in practice that means that some request IDs might end up with the same UUID
  • use PECL's uuid library which is a wrapper around libuuid: the library does not suffer from the time drifting problem, but requires extra packages to be installed and PECL enabled.
Wed, Oct 3, 10:23 AM · MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Performance-Team, Performance-Team-notice, WMF-JobQueue, Services (designing), Wikimedia-production-error, MediaWiki-General-or-Unknown
mobrovac added a subtask for T205935: MCS should request 2.0 Parsoid HTML: T128040: Document and implement the REST API format versioning and negotiation policy.
Wed, Oct 3, 9:33 AM · Reading-Infrastructure-Team-Backlog (Kanban), Services (watching), Mobile-Content-Service
mobrovac added a parent task for T128040: Document and implement the REST API format versioning and negotiation policy: T205935: MCS should request 2.0 Parsoid HTML.
Wed, Oct 3, 9:33 AM · Services (next), Documentation, Patch-For-Review, Services-next, RESTBase
mobrovac triaged T206107: Parsoid misconfigured in Beta as High priority.
Wed, Oct 3, 9:26 AM · Beta-Cluster-Infrastructure, RESTBase, Services (watching), Parsoid

Tue, Oct 2

mobrovac edited projects for T199444: ChangeProp logging KafkaConsumer is not connected, added: ChangeProp; removed WMF-JobQueue.

The interesting part is that this doesn't happen for CP-JQ, only CP.

Tue, Oct 2, 3:08 PM · Services (done), WMF-JobQueue, ChangeProp, Wikimedia-Incident, EventBus, Analytics

Fri, Sep 28

mobrovac added a comment to T201611: Deploy translation-server-v2.

Great, I'll test it today/Monday with citoid and hopefully get a patch up soon. That repo points to and uses two submodules hosted on gerrit - does anything need to be done for that? My inclination is to leave it if that's okay with SRE.

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/zotero/+/refs/heads/master/.gitmodules

Fri, Sep 28, 12:00 PM · Patch-For-Review, Services, User-mobrovac, Service-deployment-requests, VisualEditor (Current work), Citoid, Operations

Wed, Sep 26

mobrovac closed T205497: [Regression pre-wmf.23] REST API on Beta cluster returns content of different pages than requested (breaks VE) as Resolved.

You were right, @Arlolra . The problem was in downgrading the minor versions. I have deployed PR 1069 and now the correct content is returned. Resolving.

Wed, Sep 26, 3:42 PM · User-Ryasmeen, Services (done), VisualEditor, RESTBase, Parsoid
mobrovac added a comment to T205497: [Regression pre-wmf.23] REST API on Beta cluster returns content of different pages than requested (breaks VE).

I've tried various things in RB but I simply can't get RB to respond with 377057.

Wed, Sep 26, 2:38 PM · User-Ryasmeen, Services (done), VisualEditor, RESTBase, Parsoid
mobrovac edited projects for T204154: Kafka JobQueue should respect DB readonly mode, added: Services (doing); removed Services (designing).
Wed, Sep 26, 1:45 PM · Services (done), MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Performance-Team (Radar), ChangeProp, WMF-JobQueue
mobrovac moved T204154: Kafka JobQueue should respect DB readonly mode from General to ChangeProp-JobQueue on the WMF-JobQueue board.
Wed, Sep 26, 1:44 PM · Services (done), MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Performance-Team (Radar), ChangeProp, WMF-JobQueue
mobrovac added a project to T205497: [Regression pre-wmf.23] REST API on Beta cluster returns content of different pages than requested (breaks VE): Services (attic).
Wed, Sep 26, 11:51 AM · User-Ryasmeen, Services (done), VisualEditor, RESTBase, Parsoid
mobrovac added a comment to T205497: [Regression pre-wmf.23] REST API on Beta cluster returns content of different pages than requested (breaks VE).

The REST API is returning the correct, latest revision, which is 384154. It seems to me that 377057 is somehow being set by VE, as asking the REST API for a revision of a title that doesn't match the title would result in a client error (which is not the case here).

Wed, Sep 26, 11:50 AM · User-Ryasmeen, Services (done), VisualEditor, RESTBase, Parsoid

Tue, Sep 25

mobrovac triaged T205452: Setup access from service to mysql as Normal priority.
Tue, Sep 25, 4:22 PM · Patch-For-Review, Core Platform Team Kanban (Doing), Services (doing), Recommendation-API, SCB, Operations, Research
mobrovac edited projects for T205452: Setup access from service to mysql, added: Services (next), Operations, SCB, Recommendation-API; removed Services.

The first step would be to add the user/pass combo to the private puppet repo (ping @Joe @fgiunchedi cna you help out for this step?). After that we can include it in the service's public puppet profile module (I can help with that). We also need to set up the appropriate firewall rules on the x2 hosts so to make them accessible from SCB (ping @jcrespo - let's coordinate this?).

Tue, Sep 25, 4:22 PM · Patch-For-Review, Core Platform Team Kanban (Doing), Services (doing), Recommendation-API, SCB, Operations, Research
mobrovac closed T204880: RESTBase should 404 if it cannot satisfy requested TID as Resolved.

Merged and deployed.

Tue, Sep 25, 8:51 AM · User-Ryasmeen, Services (done), Reading-Infrastructure-Team-Backlog, VisualEditor, RESTBase

Mon, Sep 24

Tgr awarded T201409: Harmonise the identification of requests across our stack a Love token.
Mon, Sep 24, 12:09 AM · TechCom-RFC (TechCom-Approved), Performance-Team (Radar), Patch-For-Review, Operations, Services (designing), User-mobrovac, Traffic

Thu, Sep 20

mobrovac edited projects for T94522: Some requests fail with UIDGenerator error "Process clock is outdated or drifted", added: Services (doing); removed Services (watching).
Thu, Sep 20, 5:07 PM · MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Performance-Team, Performance-Team-notice, WMF-JobQueue, Services (designing), Wikimedia-production-error, MediaWiki-General-or-Unknown
mobrovac added a project to T204055: Consistently represent asynchronous code execution: Services (watching).
Thu, Sep 20, 10:33 AM · Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), Patch-For-Review, Services (watching), Technical-Debt, Proton
mobrovac added a project to T94522: Some requests fail with UIDGenerator error "Process clock is outdated or drifted": Services (watching).
Thu, Sep 20, 6:11 AM · MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Performance-Team, Performance-Team-notice, WMF-JobQueue, Services (designing), Wikimedia-production-error, MediaWiki-General-or-Unknown
mobrovac triaged T204880: RESTBase should 404 if it cannot satisfy requested TID as High priority.
Thu, Sep 20, 6:04 AM · User-Ryasmeen, Services (done), Reading-Infrastructure-Team-Backlog, VisualEditor, RESTBase
mobrovac updated subscribers of T204880: RESTBase should 404 if it cannot satisfy requested TID.

In my mind, asking for a non-existent tid is the same as asking for a non-existent revision, and should, hence, result in a 404. You do have a good point that it might affect Visual Editor, but in practice the 24h window is more than ample time to finish an edit. We should also take into account that VE edits are commonly based off of the latest revisions which we have in the latest bucket rather than the stash one. But I agree, let's see what @Esanders thinks.

Thu, Sep 20, 6:04 AM · User-Ryasmeen, Services (done), Reading-Infrastructure-Team-Backlog, VisualEditor, RESTBase

Sep 19 2018

mobrovac added a comment to T186748: [EPIC] New service request: chromium-render/deploy.

I think (I didn't verify it yet) that it fails because of introduced restbase checks:

Sep 19 2018, 9:17 AM · Epic, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), User-notice, Services (blocked), Service-deployment-requests, Proton, Electron-PDFs, Operations

Sep 18 2018

mobrovac committed rMSCPdd65e2c6d883: Decrease log level to info for HTTPError (authored by Pchelolo).
Decrease log level to info for HTTPError
Sep 18 2018, 10:27 PM
mobrovac committed rMSCP533ab2ef5bde: Log the failed event execution with warn level. (authored by Pchelolo).
Log the failed event execution with warn level.
Sep 18 2018, 10:27 PM

Sep 17 2018

mobrovac added a comment to T181623: Chromium-render doesn't handle browser connection abort well.

[...]
Presumably that's because I forcefully killed the chromium process, and the onlyOnce method was doing it's job in preventing the callback from being called multiple times. Afterwards, the chromium worker restarted successfully.

Sep 17 2018, 2:28 PM · Patch-For-Review, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q1), User-Jdlrobson, Readers-Web-Kanbanana-Board-Old, Proton, Unplanned-Sprint-Work
mobrovac closed T202682: Improve Accept header normalization in VCL for REST API as Resolved.

Since node's semver package does not deal with .x versions well, we settled for forcing the patch component of the version to be .0.

Sep 17 2018, 2:19 PM · Services (done), RESTBase-API, Parsing-Team, Traffic, Operations, RESTBase
mobrovac closed T202682: Improve Accept header normalization in VCL for REST API, a subtask of T128040: Document and implement the REST API format versioning and negotiation policy, as Resolved.
Sep 17 2018, 2:19 PM · Services (next), Documentation, Patch-For-Review, Services-next, RESTBase