mobrovac (Marko Obrovac)
Spy

Projects (47)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Dec 16 2014, 7:40 PM (191 w, 3 d)
Availability
Available
IRC Nick
mobrovac
LDAP User
Mobrovac
MediaWiki User
Mobrovac [ Global Accounts ]

Recent Activity

Fri, Aug 10

mobrovac added a comment to T197242: Transition citoid to use Zotero's translation-server-v2.

Hopefully this is the right place for my questions (sorry if not): So I'd like to get rid of the one remaining deployment-prep trusty host, deployment-zotero01, but I've also been told prod is running it on sca+scb hosts which are (well, at least one is) jessie (on a related note what's the deal there, why are there no scb hosts in deployment-prep)? Is setting this up on jessie/stretch just a case of making a new instance and applying role::zotero? Can I then just update references to point at the new host and kill the ol done? Is that even the appropriate action to take or should I put it on an sca host like prod?

Fri, Aug 10, 11:02 AM · VisualEditor (Current work), Patch-For-Review, Citoid, Services (watching), Operations
mobrovac reopened T201618: Remove bibtex from Citoid's public API, a subtask of T197242: Transition citoid to use Zotero's translation-server-v2, as Open.
Fri, Aug 10, 10:23 AM · VisualEditor (Current work), Patch-For-Review, Citoid, Services (watching), Operations
mobrovac reopened T201618: Remove bibtex from Citoid's public API as "Open".

If this graph is accurate, it seems to suggest that 1/4 of our traffic is bibtex :/ https://grafana.wikimedia.org/dashboard/db/service-citoid?orgId=1

Fri, Aug 10, 10:23 AM · VisualEditor, RESTBase-API, Citoid, Services (doing)

Thu, Aug 9

mobrovac triaged T201628: Cannot edit subpages, probably due to percent-encoding of URIs as Normal priority.

The symptoms described by the user seems to suggest there is a problem in the Apache config of the site. Asked them to paste their configs here.

Thu, Aug 9, 5:26 PM · Services (next), RESTBase, VisualEditor
mobrovac added a comment to T201103: Reconsider use of RESTBase k-r-v storage for mobileapps.

I thought we were not storing media and feed before anyways, as was mentioned in the Platform/Audiences sync a few minutes ago.

Thu, Aug 9, 5:20 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog, Services (designing), RESTBase, Cassandra, User-Eevans
mobrovac added a comment to T197000: Modify revision-score schema so that model probabilities won't conflict.

From our point of view, you're asking for us to implement and maintain confusing inconsistencies inside of ORES to support your specific use-case. Having one endpoint have non-standard schema is inconsistent. Adding a new endpoint with non-standard schema is inconsistent. Adding a new version where the standard schema is simplified (normalized is not the right term) is consistent. Calling it v4 would be confusing. In this case, we would be "simplifying" out a key piece of information that is necessary for batch requests (rev_id), so we'd need to remove functionality for that version of the API. "v4" would give the impression that this simplified version is the new, best version of the API. So I'd rather call it something else.

Thu, Aug 9, 5:15 PM · Scoring-platform-team, Analytics-Kanban, Patch-For-Review, User-Ladsgroup, Services (watching), ORES, Analytics, EventBus
mobrovac added a comment to T201618: Remove bibtex from Citoid's public API.

PR #1046 removes the support for it.

Thu, Aug 9, 4:50 PM · VisualEditor, RESTBase-API, Citoid, Services (doing)
mobrovac added a comment to T201103: Reconsider use of RESTBase k-r-v storage for mobileapps.

When you say mobile endpoints you mean the mobile-sections* endpoints, right? I hope that others, like the definitions endpoint on Wiktionary, still stay the same.

Thu, Aug 9, 4:43 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog, Services (designing), RESTBase, Cassandra, User-Eevans
mobrovac added a subtask for T197242: Transition citoid to use Zotero's translation-server-v2: T201618: Remove bibtex from Citoid's public API.
Thu, Aug 9, 3:24 PM · VisualEditor (Current work), Patch-For-Review, Citoid, Services (watching), Operations
mobrovac added a parent task for T201618: Remove bibtex from Citoid's public API: T197242: Transition citoid to use Zotero's translation-server-v2.
Thu, Aug 9, 3:24 PM · VisualEditor, RESTBase-API, Citoid, Services (doing)
mobrovac triaged T201618: Remove bibtex from Citoid's public API as Normal priority.
Thu, Aug 9, 3:23 PM · VisualEditor, RESTBase-API, Citoid, Services (doing)
mobrovac added a comment to T197242: Transition citoid to use Zotero's translation-server-v2.

Is there any progress and/or timeline for this? Thanks!

Thu, Aug 9, 3:21 PM · VisualEditor (Current work), Patch-For-Review, Citoid, Services (watching), Operations
mobrovac edited projects for T201611: Deploy translation-server-v2, added: Services (next), Service-deployment-requests, User-mobrovac; removed Services (watching).
Thu, Aug 9, 3:08 PM · Services, User-mobrovac, Service-deployment-requests, VisualEditor (Current work), Citoid, Operations
mobrovac added a comment to T197000: Modify revision-score schema so that model probabilities won't conflict.

After a quick h-o with @Ottomata and @JAllemandou we've understood that the /precache endpoint used to produce these is kinda private, used only by change-prop, so we're free to change the format it emits in whatever way we want.

Thu, Aug 9, 3:07 PM · Scoring-platform-team, Analytics-Kanban, Patch-For-Review, User-Ladsgroup, Services (watching), ORES, Analytics, EventBus
mobrovac added a comment to T201103: Reconsider use of RESTBase k-r-v storage for mobileapps.

Current status:

  • pre-generation for mobile end points is enabled only for WPs
  • the mobile end points have been removed from the public API for non-WP projects
  • the data tables for the others and commons storage groups have been truncated~[1]
Thu, Aug 9, 11:11 AM · Patch-For-Review, Reading-Infrastructure-Team-Backlog, Services (designing), RESTBase, Cassandra, User-Eevans
mobrovac added a comment to T201409: Harmonise the identification of requests across our stack.

I've been investigating the use of an OpenTracing implementation, such as Jaeger or Zipkin, to provide distributed tracing. The OpenTracing standard provides for the use of headers to pass a couple of different IDs around. The goal of these IDs is to be able to build a flame graph of a request, as it transits different systems -- something like https://performance.wikimedia.org/xenon/svgs/daily/2018-08-06.index.svgz but with the trace beginning when the request was first received by Varnish, and ending when the last byte is sent (see http://opentracing.io/documentation/pages/instrumentation/instrumenting-large-systems.html for a quick overview and example of how this is useful.)

Thu, Aug 9, 10:08 AM · Performance-Team (Radar), Patch-For-Review, Operations, Services (designing), TechCom-RFC, User-mobrovac, Traffic
mobrovac added a comment to T201409: Harmonise the identification of requests across our stack.

In addition to that, do we have a task for "every service should include the request ID in its logstash entries"? It seems to me like we'd need that for this to be useful.

Thu, Aug 9, 9:43 AM · Performance-Team (Radar), Patch-For-Review, Operations, Services (designing), TechCom-RFC, User-mobrovac, Traffic
mobrovac updated the task description for T201409: Harmonise the identification of requests across our stack.
Thu, Aug 9, 9:42 AM · Performance-Team (Radar), Patch-For-Review, Operations, Services (designing), TechCom-RFC, User-mobrovac, Traffic

Wed, Aug 8

mobrovac added a comment to T201103: Reconsider use of RESTBase k-r-v storage for mobileapps.

According to @bearND the mobile-html is a direct replacement for mobile-sections, so created for apps. Although there are long-long term plans to use it for mobile web, this is not going to happen at within this fiscal year. When those plans get a bit more materialized, we can reconsider and order hardware accordingly.

Wed, Aug 8, 3:02 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog, Services (designing), RESTBase, Cassandra, User-Eevans
mobrovac merged task T201514: Auto-resolve merge conflicts in RELEASE-NOTES into T200392: RfC: Release notes automation.
Wed, Aug 8, 1:43 PM · Core-Platform-Team
mobrovac merged T201514: Auto-resolve merge conflicts in RELEASE-NOTES into T200392: RfC: Release notes automation.
Wed, Aug 8, 1:43 PM · MediaWiki-Documentation, TechCom-RFC
mobrovac updated the task description for T201409: Harmonise the identification of requests across our stack.
Wed, Aug 8, 11:53 AM · Performance-Team (Radar), Patch-For-Review, Operations, Services (designing), TechCom-RFC, User-mobrovac, Traffic
mobrovac added a comment to T201158: Have a Grafana dashboard for Proton.

Thank you, @Pchelolo !

Wed, Aug 8, 7:42 AM · monitoring, Proton, Services (watching)
mobrovac added a comment to T201068: Modern Event Platform: Scalable Event Intake.

I like this idea, with the exception of making these classes JSON-serialisable. These objects may (and probably will) deliver more information than needed for our events, so we are really looking for a subset here, i.e. we should require they be EventBus-serialisable. Given this year's Platform Evolution programme's aim at rethinking interfaces inside MW, this can be part of that.

Wed, Aug 8, 7:26 AM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
mobrovac added a comment to T201103: Reconsider use of RESTBase k-r-v storage for mobileapps.
  • Have MCS manage its own object cache
Wed, Aug 8, 6:55 AM · Patch-For-Review, Reading-Infrastructure-Team-Backlog, Services (designing), RESTBase, Cassandra, User-Eevans

Tue, Aug 7

mobrovac added a comment to T201103: Reconsider use of RESTBase k-r-v storage for mobileapps.

Is this for the native apps, or for the mobile web version (i.e. are they growing at the same rate)? Which are the right dashboards for this?

Tue, Aug 7, 5:33 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog, Services (designing), RESTBase, Cassandra, User-Eevans
mobrovac added projects to T201062: Load change tags into the Analytics Data Lake on a daily basis: EventBus, Services (watching).

+1 on creating such an event, it sounds like a useful piece of information for clients to have/be able to react to.

Tue, Aug 7, 1:30 PM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, Services (watching), EventBus, Product-Analytics, Analytics
mobrovac added a comment to T201305: ThumbnailRender jobs not including the width in the fetch URL.

One more data point: most of these failures come from wmf-15 code (less than 1% of the messages come from wmf-14 and wmf-13 combined), which seems to suggest a change in wmf-15 is causing this. Alas, after going through the diff, I couldn't find any (obvious) candidates.

Tue, Aug 7, 1:15 PM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, MediaWiki-File-management, WMF-JobQueue, Multimedia, Commons, Services (watching)
mobrovac triaged T201409: Harmonise the identification of requests across our stack as Normal priority.
Tue, Aug 7, 12:42 PM · Performance-Team (Radar), Patch-For-Review, Operations, Services (designing), TechCom-RFC, User-mobrovac, Traffic
mobrovac added a comment to T201305: ThumbnailRender jobs not including the width in the fetch URL.

[...] //ms-fe.svc.eqiad.wmnet/wikipedia/commons/thumb/1/1c/Foo.tif/-Foo.tif.png

Tue, Aug 7, 10:42 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, MediaWiki-File-management, WMF-JobQueue, Multimedia, Commons, Services (watching)
mobrovac added a comment to T201103: Reconsider use of RESTBase k-r-v storage for mobileapps.

So, it's pretty obvious that for 'commons' and 'others' we are clearly just melting the icecaps, but actually the fact they are being rerendered is a bug - normal edit-related rerendering is only for Wikipedia, here I guess we're reacting to null edits and mediawiki purges - needs to be fixed and the mobile-sections endpoints has to be removed for everything other then wikipedia.

Tue, Aug 7, 10:35 AM · Patch-For-Review, Reading-Infrastructure-Team-Backlog, Services (designing), RESTBase, Cassandra, User-Eevans

Mon, Aug 6

mobrovac closed T162357: Add support for worldcat search api xml results, a subtask of T165321: New param/pattern in service for requesting a single citation based on a unique identifier , as Resolved.
Mon, Aug 6, 5:40 PM · User-mobrovac, RESTBase-API, Services (watching), VisualEditor, Citoid
mobrovac closed T162357: Add support for worldcat search api xml results as Resolved.

Deployed, resolving.

Mon, Aug 6, 5:40 PM · User-Ryasmeen, VisualEditor, Citoid
mobrovac added a comment to T201068: Modern Event Platform: Scalable Event Intake.

IMHO, relying on client libraries for validation is not really an option if we want to ensure the well-functioning of the platform, given its stated openness. In EventBus we currently have server-side validation which is an aspect that I think we should keep (whether in the current form or a different one).

Mon, Aug 6, 5:11 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
mobrovac added a project to T172480: Add jobrunner servers to Scap canary process: Services (watching).
Mon, Aug 6, 3:21 PM · Services (watching), Core-Platform-Team, WMF-JobQueue, Wikimedia-Incident, Release-Engineering-Team (Next), Scap
mobrovac updated subscribers of T201305: ThumbnailRender jobs not including the width in the fetch URL.
Mon, Aug 6, 11:19 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, MediaWiki-File-management, WMF-JobQueue, Multimedia, Commons, Services (watching)
mobrovac added a comment to T201305: ThumbnailRender jobs not including the width in the fetch URL.

A quick investigation of merged commits for includes/jobqueue/jobs/ThumbnailRender.php, includes/file/File.php and related includes/media/* files (which are used in the generation of the URL did not turn up any recent changes.

Mon, Aug 6, 11:19 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, MediaWiki-File-management, WMF-JobQueue, Multimedia, Commons, Services (watching)
mobrovac triaged T201305: ThumbnailRender jobs not including the width in the fetch URL as High priority.
Mon, Aug 6, 11:05 AM · MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), Patch-For-Review, MediaWiki-File-management, WMF-JobQueue, Multimedia, Commons, Services (watching)

Fri, Aug 3

mobrovac added a subtask for T186748: New service request: chromium-render/deploy: T181623: Chromium-render doesn't handle browser connection abort well.
Fri, Aug 3, 6:13 PM · Patch-For-Review, Readers-Web-Kanbanana-Board, Services (blocked), Service-deployment-requests, Readers-Web-Backlog, Proton, Operations, Electron-PDFs
mobrovac edited parent tasks for T181623: Chromium-render doesn't handle browser connection abort well, added: T186748: New service request: chromium-render/deploy; removed: T181118: [EPIC] Build a Chromium-based PDF renderer service.
Fri, Aug 3, 6:13 PM · User-Jdlrobson, Patch-For-Review, Readers-Web-Kanbanana-Board, Readers-Web-Backlog, Proton, Unplanned-Sprint-Work
mobrovac removed a subtask for T181118: [EPIC] Build a Chromium-based PDF renderer service: T181623: Chromium-render doesn't handle browser connection abort well.
Fri, Aug 3, 6:13 PM · Readers-Web-Kanbanana-Board, Readers-Web-Backlog, Proton, Epic
mobrovac added a comment to T201103: Reconsider use of RESTBase k-r-v storage for mobileapps.

I would second the idea of switching the MCS' storage to key-value, at least in the short term, in this way reducing the storage capacity needs.

Fri, Aug 3, 3:50 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog, Services (designing), RESTBase, Cassandra, User-Eevans
mobrovac raised the priority of T105887: CirrusSearch generates PHP notices when server is down from Low to High.

Raising the priority since there have been more than 20k such messages in the past 24h.

Fri, Aug 3, 10:48 AM · Patch-For-Review, Discovery-Search (Current work), Services (watching), CirrusSearch, Discovery
mobrovac added a comment to T200215: Create kafka topic for mjolinr bulk daemon and decide on cluster.

+1, this rate looks acceptable to me. Thank you, @EBernhardson ! @Ottomata do we have to also tweak something in MirrorMaker to ensure the double rate will be supported (just checking to make sure) ?

Fri, Aug 3, 10:27 AM · Patch-For-Review, EventBus, Services (watching), Operations, Analytics, Discovery-Search (Current work)
mobrovac added a comment to T201068: Modern Event Platform: Scalable Event Intake.

As an engineer, I want good client libraries to produce events so that I don't have to write them myself.

Could this be extended to include client-side validation of events as an explicit requirement or would you prefer a separate AC?

Fri, Aug 3, 10:17 AM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
mobrovac added a comment to T181623: Chromium-render doesn't handle browser connection abort well.

@mobrovac We submitted an Issue in Puppeteer library: https://github.com/GoogleChrome/puppeteer/issues/1505 but they said it's a low-priority for them. We can try to fix the puppeteer lib by ourselves but I'm not sure if chromium itself allows stopping the PDF rendering. The second approach (the cruel one) would be just to kill the given chromium instance, but I'm not sure if this approach has no side effects.

Fri, Aug 3, 10:12 AM · User-Jdlrobson, Patch-For-Review, Readers-Web-Kanbanana-Board, Readers-Web-Backlog, Proton, Unplanned-Sprint-Work
mobrovac added a comment to T186748: New service request: chromium-render/deploy.

Thank you @Pchelolo ! Before we can merge and deploy the above PR, though, we need to be able to monitor the service in production. I created T201158: Have a Grafana dashboard for Proton to that end.

Fri, Aug 3, 9:59 AM · Patch-For-Review, Readers-Web-Kanbanana-Board, Services (blocked), Service-deployment-requests, Readers-Web-Backlog, Proton, Operations, Electron-PDFs
mobrovac added a subtask for T186748: New service request: chromium-render/deploy: T201158: Have a Grafana dashboard for Proton.
Fri, Aug 3, 9:58 AM · Patch-For-Review, Readers-Web-Kanbanana-Board, Services (blocked), Service-deployment-requests, Readers-Web-Backlog, Proton, Operations, Electron-PDFs
mobrovac added a parent task for T201158: Have a Grafana dashboard for Proton: T186748: New service request: chromium-render/deploy.
Fri, Aug 3, 9:58 AM · monitoring, Proton, Services (watching)
mobrovac triaged T201158: Have a Grafana dashboard for Proton as High priority.
Fri, Aug 3, 9:57 AM · monitoring, Proton, Services (watching)
mobrovac added a comment to T177765: Security review of mediawiki-services-chromium-render.

@phuedx @Bawolff is there any progress on this? Has the security review been scheduled? Let me know if I can help/assist somehow.

Fri, Aug 3, 9:54 AM · Services (watching), Security-Reviews
mobrovac added a comment to T181623: Chromium-render doesn't handle browser connection abort well.

There should be progress on this before we enter the full production stage, since keeping Chromium instances working while Proton thinks the resources are free can quickly lead to resource starvation scenarios on our scale.

Fri, Aug 3, 9:52 AM · User-Jdlrobson, Patch-For-Review, Readers-Web-Kanbanana-Board, Readers-Web-Backlog, Proton, Unplanned-Sprint-Work

Thu, Aug 2

mobrovac added a comment to T200215: Create kafka topic for mjolinr bulk daemon and decide on cluster.

Indeed, the discussion is probably out of the scope of this ticket.

Thu, Aug 2, 6:16 PM · Patch-For-Review, EventBus, Services (watching), Operations, Analytics, Discovery-Search (Current work)
mobrovac added a comment to T200717: Scan npm dependencies for vulnerabilities.

We currently have nsp run as part of npm test which automatically makes Jenkins run the test. When Jenkins gets npm v6+, we can then have npm audit run as part of the test.

Thu, Aug 2, 3:41 PM · Analytics
mobrovac added a comment to T200717: Scan npm dependencies for vulnerabilities.

We have been using nsp for a while now for Node.JS services, but it will be discontinued at the end of September and replaced by the npm audit command, which is available for npm >= v6.0.0.

Thu, Aug 2, 3:35 PM · Analytics
mobrovac added projects to T201004: Spec out dependency engine interface, data structure, and states: User-mobrovac, Services (designing), Dependency-Tracking.
Thu, Aug 2, 3:22 PM · Dependency-Tracking, Services (designing), User-mobrovac, User-Daniel
mobrovac added a watcher for Dependency-Tracking: mobrovac.
Thu, Aug 2, 3:21 PM
mobrovac created Dependency-Tracking.
Thu, Aug 2, 3:21 PM
mobrovac added a comment to T200215: Create kafka topic for mjolinr bulk daemon and decide on cluster.

Eric can correct me if I'm wrong, but I believe the reason he requested just an eqiad.* prefixed topic is that these messages are generated from the Hadoop cluster which only exists in eqiad. Even during a datacenter switch, there will never be any messages written directly to main-codfw, since there are no codfw producers of this data.

Thu, Aug 2, 3:17 PM · Patch-For-Review, EventBus, Services (watching), Operations, Analytics, Discovery-Search (Current work)
mobrovac edited projects for T200215: Create kafka topic for mjolinr bulk daemon and decide on cluster, added: EventBus; removed Patch-For-Review.

I assume the task description implies the topic would get multiple messages every week, and that the total data size would be ~3GB (as opposed to one 3GB message). If so, LGTM. Note that we have snappy compression enabled in main, so the producer can simply send plain messages and they would be compressed on the fly. One question here: instead of burst 3GB of data into Kafka in one go, is there a possibility of spacing the messages out a bit to ensure the normal functioning of the Kafka cluster?

Thu, Aug 2, 10:27 AM · Patch-For-Review, EventBus, Services (watching), Operations, Analytics, Discovery-Search (Current work)

Wed, Aug 1

mobrovac added a comment to T200594: Add client identifier to requests sent from Kartotherian to WDQS.

Aren't clients making requests to Kartotherian? If they are, then X-Client-IP will be set to the external (to our prod environment) client issuing the request.

Yes. But when Kartotherian calls to WDQS, it calls external Varnish endpoint (now), which might replace X-Client-IP - not 100% sure about it, needs checking, but I think judging from the logs that's what is happening since the IP I see in the logs is one of Kartotherian.

Wed, Aug 1, 6:35 PM · Services (watching), Maps-Sprint, Reading-Infrastructure-Team-Backlog, Wikidata, Maps (Kartotherian), Wikidata-Query-Service
mobrovac added a comment to T200594: Add client identifier to requests sent from Kartotherian to WDQS.

Yes, but we need it to be the IP of the client of Karthoterian, not the IP of the server running Karthoterian.

Wed, Aug 1, 6:01 PM · Services (watching), Maps-Sprint, Reading-Infrastructure-Team-Backlog, Wikidata, Maps (Kartotherian), Wikidata-Query-Service
mobrovac added a comment to T199432: Consider disabling automatic topic creation in main-kafka.

The caveat with the maximum number of topics is that Kafka has no hard limit on it because it depends on zookeeper, so effectively it can support as many topics as zk can support znodes.

Wed, Aug 1, 8:46 AM · ChangeProp, EventBus, WMF-JobQueue, Services (designing), Analytics
mobrovac added a comment to T200594: Add client identifier to requests sent from Kartotherian to WDQS.

I would also suggest using X-Request-ID which uniquely identifies a single request.

Wed, Aug 1, 8:42 AM · Services (watching), Maps-Sprint, Reading-Infrastructure-Team-Backlog, Wikidata, Maps (Kartotherian), Wikidata-Query-Service
mobrovac added a comment to T200594: Add client identifier to requests sent from Kartotherian to WDQS.

Nginx and Varnish already attach the X-Client-IP header to incoming requests (cf. this sample request), so all you have to do is actually use the provided header.

Wed, Aug 1, 8:41 AM · Services (watching), Maps-Sprint, Reading-Infrastructure-Team-Backlog, Wikidata, Maps (Kartotherian), Wikidata-Query-Service
mobrovac added projects to T200832: remove mathoid from scb: SCB, Services (watching).

I'd hold off with this for the time being. @akosiaris what do you think?

Wed, Aug 1, 8:25 AM · Services (watching), SCB, Mathoid, Operations

Tue, Jul 31

mobrovac closed T142255: Move mathoid to deployment-sca* hosts in Beta Cluster, a subtask of T87220: Minimize differences between beta and production (Tracking), as Declined.
Tue, Jul 31, 8:04 AM · Technical-Debt, Tracking, Operations, Puppet, Beta-Cluster-Infrastructure
mobrovac closed T142255: Move mathoid to deployment-sca* hosts in Beta Cluster as Declined.

This is no longer an issue: Mathoid has been moved to our k8s infrastructure (nominally it still exists on SCB, but it's not used there at all).

Tue, Jul 31, 8:04 AM · Beta-Cluster-Infrastructure, Mathoid

Mon, Jul 30

mobrovac closed T199594: Exception "Job queue is read-only" as Resolved.

The errors have completely disappeared as of this morning UTC.

Mon, Jul 30, 8:46 PM · Services (done), MW-1.32-release-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)), User-Joe, Operations, Wikimedia-log-errors, Core-Platform-Team, WMF-JobQueue
mobrovac added a watcher for WMF-JobQueue: mobrovac.
Mon, Jul 30, 8:42 PM
mobrovac added a member for WMF-JobQueue: mobrovac.
Mon, Jul 30, 8:42 PM
mobrovac closed T195709: "invalid locale" warning on deployment-restbase02.deployment-prep.eqiad.wmflabs as Resolved.

Not happening any longer after fixing T186750: Reset RESTBase deployment-prep environment.

Mon, Jul 30, 7:17 PM · Services (done), Beta-Cluster-Infrastructure
mobrovac closed T192539: deployment-cassandra3-0{1,2}: Contact point 0 () is not a valid host name, the following values are valid contact points: ipAddress, hostName or ipAddress:port as Resolved.

The deployment-cassandra3-0x nodes have been removed and with the resolution of T186750: Reset RESTBase deployment-prep environment these errors will not appear again.

Mon, Jul 30, 7:16 PM · Services (done), RESTBase, Cassandra, Beta-Cluster-Infrastructure
mobrovac edited projects for T186750: Reset RESTBase deployment-prep environment, added: RESTBase; removed Patch-For-Review.
Mon, Jul 30, 7:14 PM · RESTBase, Services (done), Beta-Cluster-Infrastructure, User-Eevans
mobrovac closed T186994: RESTBase errors on logstash-beta as Resolved.

Resolved via T186750: Reset RESTBase deployment-prep environment

Mon, Jul 30, 7:14 PM · RESTBase, Services (done), Beta-Cluster-Infrastructure
mobrovac closed T186750: Reset RESTBase deployment-prep environment as Resolved.
Mon, Jul 30, 7:13 PM · RESTBase, Services (done), Beta-Cluster-Infrastructure, User-Eevans
mobrovac claimed T186750: Reset RESTBase deployment-prep environment.
Mon, Jul 30, 5:51 PM · RESTBase, Services (done), Beta-Cluster-Infrastructure, User-Eevans
mobrovac added a comment to T200437: Clean up wikimediafoundation.org RESTBase storage.

Alternatively, perhaps we could simply rename the domain in the records?

Do you mean to rename them in storage, or ... ?

Mon, Jul 30, 5:07 PM · wikimediafoundation.org, User-Eevans, Services (next), Cassandra, RESTBase-Cassandra, RESTBase
mobrovac triaged T200374: Update indirect dependency on github.com/gwicke/kad.git as Low priority.
Mon, Jul 30, 10:31 AM · Services (later), service-runner

Fri, Jul 27

mobrovac closed T199813: EventStreams accumulates too much memory on SCB nodes in CODFW as Resolved.

This has finally been resolved for good. Here's a summary/post-mortem for clarity and posterity.

Fri, Jul 27, 4:27 PM · Patch-For-Review, User-mobrovac, Services (done), Analytics-Kanban, ops-codfw, Wikimedia-Stream, EventBus, Analytics, Operations, SCB
mobrovac merged T191207: EventStreams consumer backpressure for slow HTTP clients into T199813: EventStreams accumulates too much memory on SCB nodes in CODFW.
Fri, Jul 27, 11:44 AM · Patch-For-Review, User-mobrovac, Services (done), Analytics-Kanban, ops-codfw, Wikimedia-Stream, EventBus, Analytics, Operations, SCB
mobrovac merged task T191207: EventStreams consumer backpressure for slow HTTP clients into T199813: EventStreams accumulates too much memory on SCB nodes in CODFW.
Fri, Jul 27, 11:44 AM · Wikimedia-Stream, EventBus, Analytics
mobrovac committed rWKSEfe0df0e409a3: Make the main processing loop async wrt the main entry point (authored by mobrovac).
Make the main processing loop async wrt the main entry point
Fri, Jul 27, 11:22 AM
mobrovac closed D1089: Make the main processing loop async wrt the main entry point.
Fri, Jul 27, 11:22 AM
mobrovac accepted D1089: Make the main processing loop async wrt the main entry point.
Fri, Jul 27, 11:17 AM
mobrovac added reviewers for D1089: Make the main processing loop async wrt the main entry point: Ottomata, Pchelolo.
Fri, Jul 27, 11:17 AM
mobrovac requested review of D1089: Make the main processing loop async wrt the main entry point.
Fri, Jul 27, 11:17 AM

Thu, Jul 26

mobrovac committed rESCDba5a15e5db04: Update proton to 0d76a15 (authored by mobrovac).
Update proton to 0d76a15
Thu, Jul 26, 10:19 PM
mobrovac committed rESCD883cacd20762: Config: Improve the MW request template (authored by mobrovac).
Config: Improve the MW request template
Thu, Jul 26, 10:19 PM
mobrovac closed T198461: Proton cannot assume the requests are for {lang}.wikipedia.org as Resolved.

Deployed, resolving.

Thu, Jul 26, 8:00 PM · Services (done), Proton
mobrovac closed T198461: Proton cannot assume the requests are for {lang}.wikipedia.org, a subtask of T186748: New service request: chromium-render/deploy, as Resolved.
Thu, Jul 26, 8:00 PM · Patch-For-Review, Readers-Web-Kanbanana-Board, Services (blocked), Service-deployment-requests, Readers-Web-Backlog, Proton, Operations, Electron-PDFs
mobrovac committed rWKSE302e6ffb3a1d: Provide better stream and error handling (authored by mobrovac).
Provide better stream and error handling
Thu, Jul 26, 7:17 PM
mobrovac closed D1088: Provide better stream and error handling.
Thu, Jul 26, 7:17 PM
mobrovac accepted D1088: Provide better stream and error handling.

Self-merging to deploy.

Thu, Jul 26, 7:13 PM
mobrovac added reviewers for D1088: Provide better stream and error handling: Ottomata, Pchelolo.
Thu, Jul 26, 7:13 PM
mobrovac requested review of D1088: Provide better stream and error handling.
Thu, Jul 26, 7:11 PM
mobrovac closed T199491: Expose /page/mobile-html endpoint via RESTBase as Resolved.

The public API is now exposed. Resolving.

Thu, Jul 26, 5:59 PM · Services (done), Reading-Infrastructure-Team-Backlog (Kanban), Page Content Service, Reading Epics (Platform JS CSS and HTML consolidation)
mobrovac closed T199491: Expose /page/mobile-html endpoint via RESTBase, a subtask of T177433: Develop HTML Content API to be used with companion structured JSON APIs, as Resolved.
Thu, Jul 26, 5:59 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Page Content Service, Reading Epics (Platform JS CSS and HTML consolidation)
mobrovac added a parent task for T199491: Expose /page/mobile-html endpoint via RESTBase: T199527: Storage for mobile-html endpoint.
Thu, Jul 26, 2:59 PM · Services (done), Reading-Infrastructure-Team-Backlog (Kanban), Page Content Service, Reading Epics (Platform JS CSS and HTML consolidation)
mobrovac added a subtask for T199527: Storage for mobile-html endpoint: T199491: Expose /page/mobile-html endpoint via RESTBase.
Thu, Jul 26, 2:59 PM · Reading-Infrastructure-Team-Backlog, Services (designing), Page Content Service, Reading Epics (Platform JS CSS and HTML consolidation)