mobrovac (Marko Obrovac)
Spy

Projects (41)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Dec 16 2014, 7:40 PM (144 w, 4 d)
Availability
Available
IRC Nick
mobrovac
LDAP User
Mobrovac
MediaWiki User
Mobrovac

Recent Activity

Fri, Sep 22

mobrovac edited projects for T176233: Reduce TTL for the feed end point to 10 minutes , added: Services (done); removed Services (doing).
Fri, Sep 22, 2:23 PM · Services (done), RESTBase-API, Reading-Infrastructure-Team-Backlog
mobrovac created P6035 mtr.
Fri, Sep 22, 8:30 AM · User-mobrovac
mobrovac edited projects for T176432: Document difference between spec.yaml and spec.template.yaml, added: service-template-node, Services (attic); removed Services.

Correct. spec.template.yaml is a spec template that should be used to create the actual spec.yaml for your service.

Fri, Sep 22, 7:23 AM · Discovery, Services (attic), service-template-node, Maps (Kartotherian), Maps-Sprint

Thu, Sep 21

mobrovac edited projects for T176263: Certain image URL schemes are being (re)written to 'http' in production, added: Services (done); removed Services (doing).

The fix for the summary end point in RB has been merged and deployed.

Thu, Sep 21, 8:10 AM · Services (done), Reading-Infrastructure-Team-Backlog (Kanban), Patch-For-Review, Mobile-Content-Service, RESTBase, Wikipedia-Android-App-Backlog
mobrovac added a comment to T171772: Prometheus metrics storage for RESTBase dev environment.

v4.1.0 of cassandra-metrics-collector is deployed to nodes running 3.x, and the restbase-ng nodes are paused (cmcd is running, but it is not collecting metrics). I found the services re-enabled before getting this pushed out, so I suspect that there are metrics in graphite that need cleaning up.

Thu, Sep 21, 7:38 AM · Patch-For-Review, Services (doing), Cassandra

Wed, Sep 20

mobrovac added a comment to T176335: logs sent to logstash are lost when the elasticsearch cirrus cluster is unavailable.

FTR, all of the aforementioned services use logstash1001 directly. That ought to change soon(TM) with T175242: all log producers need to use the logstash LVS endpoint.

Wed, Sep 20, 5:12 PM · Discovery-Search (Current work), Services (watching), Operations, Elasticsearch, Discovery, Wikimedia-Logstash
mobrovac added a project to T176335: logs sent to logstash are lost when the elasticsearch cirrus cluster is unavailable: Services (watching).

This is currently occurring on RESTBase and Parsoid hosts and SCB, impacting most of the Node.JS services, leaving them without logs in logstash.

Wed, Sep 20, 5:00 PM · Discovery-Search (Current work), Services (watching), Operations, Elasticsearch, Discovery, Wikimedia-Logstash
mobrovac closed T172224: Consider partitioning local_group_wikipedia even more as Resolved.

Patchset merged, deploy about to happen. Resolving.

Wed, Sep 20, 11:41 AM · User-mobrovac, Services (done), Cassandra, RESTBase
mobrovac closed T172224: Consider partitioning local_group_wikipedia even more, a subtask of T169936: Services 2017/18 Q1 goal: Start gradual roll-out of Cassandra 3 & new schema to resolve storage scaling issues and OOM errors., as Resolved.
Wed, Sep 20, 11:41 AM · Cassandra, Goal, Epic, Services (doing)
mobrovac added a comment to T175210: Select candidate jobs for transferring to the new infrastucture.

I honestly don't have a strong preference between the other "hearted" tasks. Given that all of them are fairly low volume, would it make sense to just deploy all of the hearted ones in the next wave?

Wed, Sep 20, 9:15 AM · Services (doing), MediaWiki-JobQueue, ChangeProp, Analytics, EventBus, Operations, User-Joe, User-Elukey
mobrovac added a project to T176263: Certain image URL schemes are being (re)written to 'http' in production: Services (doing).

The main reason for this is that the MW API returns http origins for images. For the RESTBase side, I have created PR #865 to encompass the original field as well, which should fix the problem in the summary end point, but for mobile-sections a similar hack will need to be placed in the MCS.

Wed, Sep 20, 9:12 AM · Services (done), Reading-Infrastructure-Team-Backlog (Kanban), Patch-For-Review, Mobile-Content-Service, RESTBase, Wikipedia-Android-App-Backlog

Tue, Sep 19

mobrovac added a comment to T175146: [RfC] Move RunJobs.php to the mediawiki (core) repository.

TL;DR: This proposal seems to be based on false premises.

Instead, we think the rpc/RunJobs.php and rpc/RunSingleJob.php scripts should be moved into the main rMW MediaWiki repository under the /rpc hierarchy (as this is the current location they are being synced to during deployment).

This does not seem to be correct. There is no /rpc hierarchy in the WMF-deployed version of the MediaWiki codebase, i.e. at /srv/mediawiki/php-$VERSION/rpc or anywhere under /srv/mediawiki/w or /srv/mediawiki/docroot.

Tue, Sep 19, 11:15 AM · TechCom-RfC, MediaWiki-JobQueue, MediaWiki-Configuration, Services (designing)
mobrovac closed T176184: Check 'depool' failed while deploying as Resolved.

Confirmed to have fixed deployments on SCB, resolving. Thank you @Joe for the quick fix!

Tue, Sep 19, 9:06 AM · Services (watching), Operations, Release-Engineering-Team (Backlog), Scap, Parsoid

Mon, Sep 18

mobrovac added a project to T173821: Investigate exposing content styles needed via API vs as HTML tags: RESTBase-API.
Mon, Sep 18, 4:22 PM · RESTBase-API, Services (watching), Reading-Infrastructure-Team-Backlog (Kanban), Page Content Service
mobrovac added a comment to T173821: Investigate exposing content styles needed via API vs as HTML tags.

Wrt caching, the URI structure and the variants it can have are also important. What were you thinking here?

@mobrovac Nothing concrete… if you have some thoughts here to help guide the design it would be helpful.

Mon, Sep 18, 4:22 PM · RESTBase-API, Services (watching), Reading-Infrastructure-Team-Backlog (Kanban), Page Content Service
mobrovac added projects to T176126: Update node-rdkafka version to 2.0: EventBus, Services (next).
Mon, Sep 18, 2:26 PM · Services (next), Analytics, EventBus, Trending-Service, ChangeProp, Reading-Infrastructure-Team-Backlog
mobrovac added a comment to T176126: Update node-rdkafka version to 2.0.

Note that we are talking about librdkafka v0.9.5 here ;) node-rdkafka does not support the v0.11 version (yet).

Mon, Sep 18, 2:25 PM · Services (next), Analytics, EventBus, Trending-Service, ChangeProp, Reading-Infrastructure-Team-Backlog

Fri, Sep 15

mobrovac added a comment to T172224: Consider partitioning local_group_wikipedia even more.
  • phase0 wikis
  • enwiki
  • all other wikipedias
  • globaldomain (wikimedia.org)
  • commons
  • remaining projects
Fri, Sep 15, 9:25 AM · User-mobrovac, Services (done), Cassandra, RESTBase
mobrovac added a project to T173821: Investigate exposing content styles needed via API vs as HTML tags: Services (watching).
Fri, Sep 15, 9:17 AM · RESTBase-API, Services (watching), Reading-Infrastructure-Team-Backlog (Kanban), Page Content Service
mobrovac added a comment to T173821: Investigate exposing content styles needed via API vs as HTML tags.

CSS files are considered to be part of the codebase. That means that they (can) change with each code deploy. Consequently, there are no events happening in the system when a particular source file changes (and given the amount of files we have in all of the repositories combined, it's not even feasible to do so). In order words, CSS files are not considered content per se. If we wanted to promote them to that status, option #2 is probably the most pragmatic and easiest to implement in the short term. To implement #1 reliably, we could explore setting up some hooks during the deployment process that could/would check the CSS files' last modified time. However, this assumes that (i) the CSS files are all part of the same service codebase; (ii) all of them are in the same directory (for setting up the checks more easily); and (iii) we can actually pull it off with Scap.

Fri, Sep 15, 9:16 AM · RESTBase-API, Services (watching), Reading-Infrastructure-Team-Backlog (Kanban), Page Content Service

Thu, Sep 14

mobrovac added a comment to T175316: Very large jobs posted by Wikidata.

@mobrovac how about a very large number of very small jobs? e.g. a million jobs to purge a million pages from cdn?

Note that we introduced batching only a few weeks ago, at the explicit request of the performance folks. We had one job per purge before. It caused problems.

Thu, Sep 14, 4:52 PM · MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), Patch-For-Review, EventBus, Services (designing), Analytics, Wikidata
mobrovac added a comment to T175316: Very large jobs posted by Wikidata.

We can tweak the chunk size - more jobs, or larger jobs, your pick.

Thu, Sep 14, 4:36 PM · MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), Patch-For-Review, EventBus, Services (designing), Analytics, Wikidata
mobrovac triaged T175146: [RfC] Move RunJobs.php to the mediawiki (core) repository as Normal priority.
Thu, Sep 14, 3:30 PM · TechCom-RfC, MediaWiki-JobQueue, MediaWiki-Configuration, Services (designing)
mobrovac updated the task description for T175146: [RfC] Move RunJobs.php to the mediawiki (core) repository.
Thu, Sep 14, 3:30 PM · TechCom-RfC, MediaWiki-JobQueue, MediaWiki-Configuration, Services (designing)
mobrovac updated subscribers of T175146: [RfC] Move RunJobs.php to the mediawiki (core) repository.

@Legoktm @Anomie @aaron @Krinkle @tstarling any thoughts/objects/suggestions on this?

Thu, Sep 14, 3:29 PM · TechCom-RfC, MediaWiki-JobQueue, MediaWiki-Configuration, Services (designing)
mobrovac added a comment to T169998: RFC: Container path conventions.

To me it is not, in fact this is the biggest problem of containers.

Thu, Sep 14, 3:07 PM · Release-Engineering-Team (Watching / External), MediaWiki-Containers, User-mobrovac, Kubernetes, Services (designing)
mobrovac closed T151934: Check & possibly improve request id handling in changeprop as Resolved.

Merged and deployed, resolving.

Thu, Sep 14, 2:07 PM · Services (done), ChangeProp
mobrovac added a project to T175780: Requests for new JobQueue monitoring capabilities: MediaWiki-JobQueue.
Thu, Sep 14, 1:40 PM · MediaWiki-JobQueue, Services (designing), ChangeProp, Analytics, EventBus
mobrovac added a project to T175758: Update MCS to new service template version: service-template-node.
Thu, Sep 14, 1:37 PM · service-template-node, Reading-Infrastructure-Team-Backlog, Mobile-Content-Service
mobrovac added a comment to T175874: RESTBase's and Parsoid's configs in MW-Vagrant don't contain additional domains.

We should find/list all of such domains and simply declare them in the configs regardless of whether they would ever be used or not.

Thu, Sep 14, 1:35 PM · Patch-For-Review, Parsoid, MediaWiki-Vagrant, Services (doing)
mobrovac renamed T175874: RESTBase's and Parsoid's configs in MW-Vagrant don't contain additional domains from RESTBase config in vagrant doesn't contain additional domains to RESTBase's and Parsoid's configs in MW-Vagrant don't contain additional domains.
Thu, Sep 14, 1:35 PM · Patch-For-Review, Parsoid, MediaWiki-Vagrant, Services (doing)
mobrovac added a comment to T169998: RFC: Container path conventions.

FTR, I would really prefer us to stay close to FHS. I 've had to debug containers that did not follow it already and the time lost trying to figure out where things were was pretty significant. We can make the argument that we would get used to a non FHS solution, but if we add the time it would take every new person to get accustomed to it I am willing to bet it would end up being considerable. Given we also want to create an environment that would be welcoming to new developers we should probably avoid extra surprises. They would already have enough new things to get accustomed to, let's not add more.

Thu, Sep 14, 1:32 PM · Release-Engineering-Team (Watching / External), MediaWiki-Containers, User-mobrovac, Kubernetes, Services (designing)
mobrovac reopened T175210: Select candidate jobs for transferring to the new infrastucture as "Open".

Sure.

Thu, Sep 14, 1:18 PM · Services (doing), MediaWiki-JobQueue, ChangeProp, Analytics, EventBus, Operations, User-Joe, User-Elukey
mobrovac reopened T175210: Select candidate jobs for transferring to the new infrastucture, a subtask of T169937: Services Q1 2017/18 goal: Begin migrating job queue processing to multi-DC enabled eventbus infrastructure., as Open.
Thu, Sep 14, 1:18 PM · Operations, User-Joe, User-Elukey, Goal, Epic, Services (doing)
mobrovac closed T172610: Invalid Cassandra seeds list is spamming the debug logs, a subtask of T160570: Cassandra 3.x Tracking, as Resolved.
Thu, Sep 14, 1:11 PM · Services (doing), Patch-For-Review, Wikimedia-Incident, Cassandra
mobrovac closed T172610: Invalid Cassandra seeds list is spamming the debug logs as Resolved.

Ok, the above patch truly fixed the issue. There were problems in the seed list in both labs and staging, and they have now been remedied.

Thu, Sep 14, 1:11 PM · Services (done), Cassandra
mobrovac created P6005 (An Untitled Masterwork).
Thu, Sep 14, 8:51 AM
mobrovac created P6004 (An Untitled Masterwork).
Thu, Sep 14, 8:50 AM

Wed, Sep 13

mobrovac closed T175210: Select candidate jobs for transferring to the new infrastucture as Resolved.

The job is being double-produced now, so resolving.

Wed, Sep 13, 2:53 PM · Services (doing), MediaWiki-JobQueue, ChangeProp, Analytics, EventBus, Operations, User-Joe, User-Elukey
mobrovac closed T175210: Select candidate jobs for transferring to the new infrastucture, a subtask of T169937: Services Q1 2017/18 goal: Begin migrating job queue processing to multi-DC enabled eventbus infrastructure., as Resolved.
Wed, Sep 13, 2:53 PM · Operations, User-Joe, User-Elukey, Goal, Epic, Services (doing)
mobrovac added a comment to T169116: Support flagged revisions in RESTBase.

An intermediary solution which consults the MW API is available as PR 863, but as of the time of writing needs some improvements, so comments/suggestions are welcomed on the PR.

Wed, Sep 13, 2:50 PM · RESTBase-API, RESTBase, VisualEditor, Reading-Infrastructure-Team-Backlog, Mobile-Content-Service, Services (designing)
mobrovac committed rMSCP6a6740f6e7c7: Separate kafka broker metrics by consumer group id (authored by Pchelolo).
Separate kafka broker metrics by consumer group id
Wed, Sep 13, 12:11 PM
mobrovac committed rMSCP7da9e8892a17: Don't reuse request_id (authored by Pchelolo).
Don't reuse request_id
Wed, Sep 13, 12:11 PM
mobrovac committed rMSCP51b165cfea57: Reduce the TTL for individual message deduplication (authored by Pchelolo).
Reduce the TTL for individual message deduplication
Wed, Sep 13, 12:11 PM
mobrovac added a project to T175735: [Spike] Investigate developing a Share-a-fact card service in MCS: RESTBase-API.

From the description, it looks like this would just perform reformatting? If so, and given the fact that clients already have all the needed info, what would be the exact benefit of doing this server-side?

Wed, Sep 13, 11:48 AM · RESTBase-API, Services (watching), Wikipedia-iOS-App-Backlog, Mobile-Content-Service, Reading-Infrastructure-Team-Backlog (Kanban)

Tue, Sep 12

mobrovac raised the priority of T172224: Consider partitioning local_group_wikipedia even more from Normal to High.

Raising the priority as we should settle on this before migrating to the new storage scheme.

Tue, Sep 12, 10:04 AM · User-mobrovac, Services (done), Cassandra, RESTBase
mobrovac closed T172610: Invalid Cassandra seeds list is spamming the debug logs as Resolved.
Tue, Sep 12, 8:49 AM · Services (done), Cassandra
mobrovac closed T172610: Invalid Cassandra seeds list is spamming the debug logs, a subtask of T160570: Cassandra 3.x Tracking, as Resolved.
Tue, Sep 12, 8:49 AM · Services (doing), Patch-For-Review, Wikimedia-Incident, Cassandra
mobrovac lowered the priority of T99883: Make "Response Messages" collapsed by default in API help from Normal to Lowest.
Tue, Sep 12, 8:48 AM · Services (later)
mobrovac moved T174993: Vandalism in "In the news" articles persisting in the app' ? from Backlog to watching on the Services board.
Tue, Sep 12, 8:39 AM · Reading-Infrastructure-Team-Backlog, Services (watching), Mobile, Wikipedia-iOS-App-Backlog, Wikipedia-Android-App-Backlog, iOS-app-Bugs, Android-app-Bugs
mobrovac moved T175284: Create parent directories for JBOD data_directories and e.g. commitlog directories from Backlog to next on the Services board.
Tue, Sep 12, 8:39 AM · Services (next), Cassandra
mobrovac moved T175281: Separate off ChangePropagation for JobQueue as a new deployment from Backlog to production on the Service-deployment-requests board.
Tue, Sep 12, 8:12 AM · Services (done), User-mobrovac, Service-deployment-requests, Operations, MediaWiki-JobQueue, EventBus, Analytics, ChangeProp

Mon, Sep 11

mobrovac added a comment to T175210: Select candidate jobs for transferring to the new infrastucture.

IMHO, updateBetaFeaturesUserCounts is the perfect candidate here. It's very lightweight (one SELECT, one UPDATE), it's idempotent and low-volume.

Mon, Sep 11, 3:06 PM · Services (doing), MediaWiki-JobQueue, ChangeProp, Analytics, EventBus, Operations, User-Joe, User-Elukey
mobrovac added a comment to T175527: Build a slim container for fluentd.

we might want to improve upon that.

Mon, Sep 11, 11:44 AM · Patch-For-Review, User-Joe, Services (watching), Kubernetes, Operations, Goal
mobrovac closed T175281: Separate off ChangePropagation for JobQueue as a new deployment as Resolved.

Everything is set up now, and the cpjobqueue service is live in production on the SCB cluster (currently idling pending the resolution of T175210: Select candidate jobs for transferring to the new infrastucture). Calling this done!

Mon, Sep 11, 9:46 AM · Services (done), User-mobrovac, Service-deployment-requests, Operations, MediaWiki-JobQueue, EventBus, Analytics, ChangeProp
mobrovac closed T175281: Separate off ChangePropagation for JobQueue as a new deployment, a subtask of T157088: [EPIC] Develop a JobQueue backend based on EventBus, as Resolved.
Mon, Sep 11, 9:46 AM · MediaWiki-JobQueue, Epic, Services (doing), User-mobrovac, Analytics, ChangeProp, EventBus
mobrovac closed T174992: Update zotero translators as Resolved.

Deployed, resolving.

Mon, Sep 11, 8:45 AM · User-Ryasmeen, Services (watching), VisualEditor, Citoid

Fri, Sep 8

mobrovac added a comment to T175281: Separate off ChangePropagation for JobQueue as a new deployment.

The repo has been set up and cloned on tin and the ops/puppet profile created and merged. Left to do is to add the profile to SCB's role, which is scheduled to happen on Monday, 2017-09-08.

Fri, Sep 8, 2:54 PM · Services (done), User-mobrovac, Service-deployment-requests, Operations, MediaWiki-JobQueue, EventBus, Analytics, ChangeProp
mobrovac updated the task description for T175281: Separate off ChangePropagation for JobQueue as a new deployment.
Fri, Sep 8, 2:50 PM · Services (done), User-mobrovac, Service-deployment-requests, Operations, MediaWiki-JobQueue, EventBus, Analytics, ChangeProp
mobrovac added a comment to T174993: Vandalism in "In the news" articles persisting in the app' ?.

WRT the 10h lag, one theory could be that it is connected to T173710: Job queue is increasing non-stop, where the backlog of refreshLinks jobs (used to trigger updates to page properties as well) has been very high lately, especially on commons. If that is indeed the case (this is yet TBD), then no caching setting would help us.

Fri, Sep 8, 1:02 PM · Reading-Infrastructure-Team-Backlog, Services (watching), Mobile, Wikipedia-iOS-App-Backlog, Wikipedia-Android-App-Backlog, iOS-app-Bugs, Android-app-Bugs
mobrovac edited projects for T175316: Very large jobs posted by Wikidata, added: Services (designing), EventBus; removed Services (doing).
Fri, Sep 8, 12:09 PM · MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), Patch-For-Review, EventBus, Services (designing), Analytics, Wikidata
mobrovac updated the task description for T175210: Select candidate jobs for transferring to the new infrastucture.
Fri, Sep 8, 12:08 PM · Services (doing), MediaWiki-JobQueue, ChangeProp, Analytics, EventBus, Operations, User-Joe, User-Elukey
mobrovac added a comment to T174993: Vandalism in "In the news" articles persisting in the app' ?.

@mobrovac by "properly" do you mean monitoring changeprop for all the page changes and purging the cache like that?

Fri, Sep 8, 8:44 AM · Reading-Infrastructure-Team-Backlog, Services (watching), Mobile, Wikipedia-iOS-App-Backlog, Wikipedia-Android-App-Backlog, iOS-app-Bugs, Android-app-Bugs

Thu, Sep 7

mobrovac added a comment to T174993: Vandalism in "In the news" articles persisting in the app' ?.

My 2 cents as an Android app user: there is no difference to me if the time frame is 10 mins or 1h since the typical workflow is (i)) spot something is wrong; (ii) refresh 2, 3 times (takes way less than a second); and then either complain or ignore. As somebody that on the back-end side of this story, I opt for ignoring it (if I'm not in the capacity to purge it from Varnish right away), but I can relate to people that complain about it. I think that informing users about this edge case would go a long way. Posting something somewhere where people complain most often would greatly help, given the fact that solving this problem properly is not a small endeavour in technical terms.

Thu, Sep 7, 8:19 PM · Reading-Infrastructure-Team-Backlog, Services (watching), Mobile, Wikipedia-iOS-App-Backlog, Wikipedia-Android-App-Backlog, iOS-app-Bugs, Android-app-Bugs
mobrovac added a comment to T168848: Bootstrap an initial version of the Page Summary API in MCS.

RB PR has been undeployed.

Thu, Sep 7, 3:33 PM · Reading-Infrastructure-Team-Backlog, Mobile-Content-Service, Readers-Web-Backlog, Page-Previews
mobrovac added a project to T175281: Separate off ChangePropagation for JobQueue as a new deployment: User-mobrovac.
Thu, Sep 7, 3:16 PM · Services (done), User-mobrovac, Service-deployment-requests, Operations, MediaWiki-JobQueue, EventBus, Analytics, ChangeProp
mobrovac updated the task description for T175281: Separate off ChangePropagation for JobQueue as a new deployment.
Thu, Sep 7, 2:49 PM · Services (done), User-mobrovac, Service-deployment-requests, Operations, MediaWiki-JobQueue, EventBus, Analytics, ChangeProp
mobrovac added a subtask for T157088: [EPIC] Develop a JobQueue backend based on EventBus: T175281: Separate off ChangePropagation for JobQueue as a new deployment.
Thu, Sep 7, 2:44 PM · MediaWiki-JobQueue, Epic, Services (doing), User-mobrovac, Analytics, ChangeProp, EventBus
mobrovac added a parent task for T175281: Separate off ChangePropagation for JobQueue as a new deployment: T157088: [EPIC] Develop a JobQueue backend based on EventBus.
Thu, Sep 7, 2:44 PM · Services (done), User-mobrovac, Service-deployment-requests, Operations, MediaWiki-JobQueue, EventBus, Analytics, ChangeProp
mobrovac created T175281: Separate off ChangePropagation for JobQueue as a new deployment.
Thu, Sep 7, 2:44 PM · Services (done), User-mobrovac, Service-deployment-requests, Operations, MediaWiki-JobQueue, EventBus, Analytics, ChangeProp
mobrovac added a comment to T168848: Bootstrap an initial version of the Page Summary API in MCS.

With the deploy to production of the MCS part I assumed that the tests had been carried out in Beta (seeing that you don't need RESTBase for that there in order to compare the outputs). I can undeploy this from the RB side from production if needed.

Thu, Sep 7, 2:38 PM · Reading-Infrastructure-Team-Backlog, Mobile-Content-Service, Readers-Web-Backlog, Page-Previews
mobrovac added a comment to T173710: Job queue is increasing non-stop.

Wikibase refreshlinks jobs might benefit from being in smaller batches

Thu, Sep 7, 2:17 PM · Patch-For-Review, Services (watching), Performance-Team (Radar), Discovery-Search, Discovery, CirrusSearch, Wikidata-Sprint, Wikidata, Operations, MediaWiki-JobQueue
mobrovac added projects to T175212: Services Q2 2017/18 goal: Migrate a subset of jobs to multi-DC enabled event processing infrastructure.: EventBus, ChangeProp, MediaWiki-JobQueue.
Thu, Sep 7, 9:53 AM · MediaWiki-JobQueue, ChangeProp, Analytics, EventBus, Goal, Services (next)
mobrovac triaged T175224: Unclear expectations around path/module in $wgVirtualRestConfig as Low priority.
Thu, Sep 7, 9:52 AM · Services (later), Documentation, RESTBase
mobrovac added a comment to T168848: Bootstrap an initial version of the Page Summary API in MCS.

@mobrovac: When was making that switch discussed? @ovasileva: Were you aware of this?

Thu, Sep 7, 9:32 AM · Reading-Infrastructure-Team-Backlog, Mobile-Content-Service, Readers-Web-Backlog, Page-Previews
mobrovac updated the task description for T175210: Select candidate jobs for transferring to the new infrastucture.
Thu, Sep 7, 9:19 AM · Services (doing), MediaWiki-JobQueue, ChangeProp, Analytics, EventBus, Operations, User-Joe, User-Elukey
mobrovac raised the priority of T175210: Select candidate jobs for transferring to the new infrastucture from Normal to High.
Thu, Sep 7, 9:14 AM · Services (doing), MediaWiki-JobQueue, ChangeProp, Analytics, EventBus, Operations, User-Joe, User-Elukey
mobrovac accepted D770: Add shared lib node module bins to PATH.
Thu, Sep 7, 9:00 AM · Release-Engineering-Team
mobrovac accepted D769: Smarter copies/sharedvolume/default behavior.
Thu, Sep 7, 9:00 AM · Release-Engineering-Team
mobrovac added a comment to T168848: Bootstrap an initial version of the Page Summary API in MCS.

The RESTBase side of things has been deployed, so that it now contacts MCS for getting the info needed for the summary end point. Note, however, that due to the fact that the summary end point is being used by Page Previews (high volume), I haven't dropped the old data from storage. This means that the new format will replace the old one gradually as pages need to be re-rendered.

Thu, Sep 7, 8:47 AM · Reading-Infrastructure-Team-Backlog, Mobile-Content-Service, Readers-Web-Backlog, Page-Previews
mobrovac added a project to T170120: Standardize on the "default" pod setup: Services (watching).

+1 on decoupling these concerns from the the running services. This model would allow developers to concentrate solely on their service's functionality and would also decouple the configuration of the service itself from auxiliary facilities (like where to send logs, metrics, handle auth(n|z), etc).

Thu, Sep 7, 8:44 AM · User-Joe, Services (watching), Kubernetes, Operations, Goal

Wed, Sep 6

mobrovac created P5964 Jobs in EventBus.
Wed, Sep 6, 4:53 PM · ChangeProp, MediaWiki-JobQueue, EventBus
mobrovac added a subtask for T157088: [EPIC] Develop a JobQueue backend based on EventBus: T175146: [RfC] Move RunJobs.php to the mediawiki (core) repository.
Wed, Sep 6, 12:03 PM · MediaWiki-JobQueue, Epic, Services (doing), User-mobrovac, Analytics, ChangeProp, EventBus
mobrovac added a parent task for T175146: [RfC] Move RunJobs.php to the mediawiki (core) repository: T157088: [EPIC] Develop a JobQueue backend based on EventBus.
Wed, Sep 6, 12:03 PM · TechCom-RfC, MediaWiki-JobQueue, MediaWiki-Configuration, Services (designing)
mobrovac created T175146: [RfC] Move RunJobs.php to the mediawiki (core) repository.
Wed, Sep 6, 12:01 PM · TechCom-RfC, MediaWiki-JobQueue, MediaWiki-Configuration, Services (designing)

Tue, Sep 5

mobrovac committed rMSTEDe3bc0de29b43: Scap: Depool the service before performing the checks (authored by mobrovac).
Scap: Depool the service before performing the checks
Tue, Sep 5, 2:43 PM
mobrovac committed rMSRDbebcec9f6014: Scap: Check the endpoints' health after depooling the service (authored by mobrovac).
Scap: Check the endpoints' health after depooling the service
Tue, Sep 5, 12:59 PM
mobrovac reopened T119094: Expose pageview data in each project's REST API as "Open".

@Nuria there seems to some confusion in this conversation. I am not proposing to migrate the existing end-points to project-specific domains, I am proposing to add these there so that the API is more easily discoverable.

Tue, Sep 5, 10:13 AM · RESTBase-API, Analytics-Kanban, Services (later), User-mobrovac
mobrovac added a comment to T174983: Parsing Swagger spec for MCS fails in Vagrant.

Doesn't http://dev.wiki.local.wmftest.net:8888/?doc give you MCS' help page?

Tue, Sep 5, 10:06 AM · Reading-Infrastructure-Team-Backlog, MediaWiki-Vagrant, Mobile-Content-Service
mobrovac added a comment to T174983: Parsing Swagger spec for MCS fails in Vagrant.

Doesn't http://dev.wiki.local.wmftest.net:8888/?doc give you MCS' help page?

Tue, Sep 5, 9:53 AM · Reading-Infrastructure-Team-Backlog, MediaWiki-Vagrant, Mobile-Content-Service
mobrovac triaged T174982: Sourcemap is incorrect in RESTBase help page as Low priority.

cause an error by putting the wrong URL in the input field

Tue, Sep 5, 9:52 AM · Services (later), RESTBase-API, RESTBase
mobrovac committed rMSCD492b1501eef7: [Config] Respect the fetch.message.max.bytes from puppet. (authored by Pchelolo).
[Config] Respect the fetch.message.max.bytes from puppet.
Tue, Sep 5, 9:19 AM
mobrovac added a comment to T173038: /v1/translate/{from}/{to}{/provider} endpoint fails while deploying cxserver.

I looked over the patches, and I don't think they will solve this problem. One thing to note is that in the code you sometimes use the GET method, but the template enforces POST, which is the correct way: the MW API should be used with the POST method only.

Tue, Sep 5, 8:30 AM · Patch-For-Review, Language-2017-July-Sept, ContentTranslation, ContentTranslation-Deployments, ContentTranslation-CXserver

Mon, Sep 4

mobrovac claimed T172610: Invalid Cassandra seeds list is spamming the debug logs.

This has been merged and Puppet has been run. The main IPs are no longer in the seeds lists, so resolving.

Mon, Sep 4, 4:37 PM · Services (done), Cassandra
mobrovac edited projects for T174960: Varnish does not vary elasticsearch query by request body, added: Traffic; removed Services.
Mon, Sep 4, 4:33 PM · Traffic, Operations, Wikimedia-Logstash
mobrovac edited projects for T159922: pdfrender fails to serve requests since Mar 8 00:30:32 UTC on scb1003, added: Services (done); removed Services (blocked), Patch-For-Review.
Mon, Sep 4, 11:06 AM · Services (done), Readers-Web-Backlog (Tracking), Operations, Electron-PDFs
mobrovac closed T159922: pdfrender fails to serve requests since Mar 8 00:30:32 UTC on scb1003 as Resolved.

Agreed, this task has become confusing. As the start-up issue has been worked around, I am closing this task. I have created T174916: electron/pdfrender hangs where we can track the service's hangs in production.

Mon, Sep 4, 11:06 AM · Services (done), Readers-Web-Backlog (Tracking), Operations, Electron-PDFs
mobrovac created T174916: electron/pdfrender hangs.
Mon, Sep 4, 11:04 AM · Readers-Web-Backlog (Tracking), Electron-PDFs, Operations, Services (blocked)
mobrovac placed T174392: Disk errors: restbase1010.eqiad.wmnet up for grabs.
Mon, Sep 4, 10:57 AM · Services (watching), Operations
mobrovac added a parent task for T174392: Disk errors: restbase1010.eqiad.wmnet: T169939: End of August milestone: Cassandra 3 cluster in production.
Mon, Sep 4, 10:57 AM · Services (watching), Operations