Pchelolo
User

Projects (7)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Jun 24 2015, 10:23 AM (173 w, 3 d)
Availability
Available
LDAP User
Unknown
MediaWiki User
Pchelolo [ Global Accounts ]

Recent Activity

Fri, Oct 19

Pchelolo added a comment to T207329: Clear watchlist on enwiki only removes 50 items at a time.

@kostajh If you have time for that it would be perfect. I admit, I don't have any idea how to test this.

Fri, Oct 19, 4:32 PM · MW-1.33-notes (1.33.0-wmf.1; 2018-10-23), Core Platform Team Kanban (Blocked Externally), Services (blocked), Patch-For-Review, Analytics, EventBus, WMF-JobQueue, Wikimedia-production-error, MediaWiki-Watchlist, Growth-Team

Thu, Oct 18

Pchelolo added a comment to T207329: Clear watchlist on enwiki only removes 50 items at a time.

Just checked, python jsonschema validates milliseconds ISO-8601s with date-time format just fine. :)

Thu, Oct 18, 10:03 PM · MW-1.33-notes (1.33.0-wmf.1; 2018-10-23), Core Platform Team Kanban (Blocked Externally), Services (blocked), Patch-For-Review, Analytics, EventBus, WMF-JobQueue, Wikimedia-production-error, MediaWiki-Watchlist, Growth-Team
Pchelolo added a comment to T207329: Clear watchlist on enwiki only removes 50 items at a time.

The above patch should mitigate the problem, however, we need to also account for possible clock drift between our servers. The more drift we tolerate, the less efficient our deduplication becomes, so I'm wondering whether we have any data on the possible clock drift to help make a decision on the exact number to tolerate?

Thu, Oct 18, 9:56 PM · MW-1.33-notes (1.33.0-wmf.1; 2018-10-23), Core Platform Team Kanban (Blocked Externally), Services (blocked), Patch-For-Review, Analytics, EventBus, WMF-JobQueue, Wikimedia-production-error, MediaWiki-Watchlist, Growth-Team
Pchelolo added a comment to T202642: Investigate how to fix the performance problems caused by CPU bound work on the MCS services.

we've switched from requesting all the onthisday content at once to requesting every portion separately

Thu, Oct 18, 3:50 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Readers-Web-Backlog (Tracking), Services (watching), Mobile-Content-Service
Pchelolo added a comment to T207324: RESTBase adds duplicated `accept-language` entry in Vary header.

Wow, in the example one Accept-Language value is capitalized, the other all lower-case. Which way do we go with?

Thu, Oct 18, 3:36 PM · Core Platform Team Backlog (Later), goodfirstbug, RESTBase, Services (later)
Pchelolo added a comment to T202642: Investigate how to fix the performance problems caused by CPU bound work on the MCS services.

And the fact it did increase the perf of /ontishday 2x suggests there is a lot of room for improvement in MCS itself - we're making 5 times more requests RB->MCS for that endpoint now, so more CPU time for making the request, parsing and routing it in MCS, combining the results, but just by splitting the actual processing into different nodes we get a lot of improvement.

Thu, Oct 18, 3:34 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Readers-Web-Backlog (Tracking), Services (watching), Mobile-Content-Service

Wed, Oct 17

Pchelolo added projects to T207329: Clear watchlist on enwiki only removes 50 items at a time: WMF-JobQueue, EventBus, Services (next).

I know what's happening. If there's more items in the list then the bach size, the code re-enqueues exactly the same job again here. That means the deduplication info is exactly the same as for the previous job, but the Kafka queue is so quick that the jobs are executed with < 1 second delay, so the dt for the second job ends up being exactly the same as the dt for the previous job and they get deduplicated.

Wed, Oct 17, 10:09 PM · MW-1.33-notes (1.33.0-wmf.1; 2018-10-23), Core Platform Team Kanban (Blocked Externally), Services (blocked), Patch-For-Review, Analytics, EventBus, WMF-JobQueue, Wikimedia-production-error, MediaWiki-Watchlist, Growth-Team
Pchelolo closed T206701: MCS must not pass through `vary: accept` from parsoid as Resolved.

Thank you!

Wed, Oct 17, 8:05 PM · Services (done), Reading-Infrastructure-Team-Backlog (Kanban), Mobile-Content-Service
Pchelolo created T207324: RESTBase adds duplicated `accept-language` entry in Vary header.
Wed, Oct 17, 8:05 PM · Core Platform Team Backlog (Later), goodfirstbug, RESTBase, Services (later)
Pchelolo closed T201158: [4hrs] Have a Grafana dashboard for Proton, a subtask of T186748: [EPIC] New service request: chromium-render/deploy, as Resolved.
Wed, Oct 17, 6:12 PM · Epic, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), User-notice, Services (blocked), Service-deployment-requests, Proton, Electron-PDFs, Operations
Pchelolo closed T201158: [4hrs] Have a Grafana dashboard for Proton as Resolved.

Ye. We can tune and tweak it indefinitely, but for now I think we're in a good state

Wed, Oct 17, 6:12 PM · Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), monitoring, Proton, Services (watching)
Pchelolo updated the task description for T204082: Cassandra-uuid improvements.
Wed, Oct 17, 5:45 PM · goodfirstbug, RESTBase, Services (later)
Pchelolo assigned T204082: Cassandra-uuid improvements to Clarakosi.
Wed, Oct 17, 5:39 PM · goodfirstbug, RESTBase, Services (later)
Pchelolo added a comment to T204082: Cassandra-uuid improvements.

However, there has been a lot of improvements to the upstream code even without the aformentioned PR. those improvements need to be brought into our fork.

Wed, Oct 17, 12:06 AM · goodfirstbug, RESTBase, Services (later)

Tue, Oct 16

Pchelolo added a comment to T201158: [4hrs] Have a Grafana dashboard for Proton.

I did fix the latency graph: https://grafana.wikimedia.org/dashboard/db/proton?orgId=1&from=now-3h&to=now

Tue, Oct 16, 6:58 PM · Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), monitoring, Proton, Services (watching)

Fri, Oct 12

Pchelolo triaged T206889: Develop a library for JSON schema backwards incompatibility detection as Normal priority.
Fri, Oct 12, 11:18 PM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Pchelolo added a comment to T206789: Modern Event Platform: Event Schema Registry: Implementation.

How're we satisfying the requirement of

Fri, Oct 12, 3:53 AM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Pchelolo added a comment to T206824: Decide whether to use schema references in the schema registry.

If we do adopt the policy to have the latest schema using the references and then getting rendered into full schemas in versioned files, so that clients are not required to support all the fancy features, we'd need to make the pre-commit hook an executable script to satisfy development requirements like:

Fri, Oct 12, 3:48 AM · Core Platform Team Backlog (Designing), Services (designing), Core Platform Team (Modern Event Platform (TEC2)), Analytics-EventLogging, EventBus, Analytics
Pchelolo added a comment to T206789: Modern Event Platform: Event Schema Registry: Implementation.

Q: should we use the term 'repository' or 'registry' here? I'm considering retitling the tickets to 'repository' since we will be using git repositories. However, there may be some extra features on a potential HTTP service that serves schemas. If we have that, would we call that the 'registry'?

Fri, Oct 12, 3:37 AM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Pchelolo added a comment to T206789: Modern Event Platform: Event Schema Registry: Implementation.

Up to date JSONSchema support (Draft 7?)

Fri, Oct 12, 3:33 AM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Neil_P._Quinn_WMF awarded T206824: Decide whether to use schema references in the schema registry a Yellow Medal token.
Fri, Oct 12, 12:36 AM · Core Platform Team Backlog (Designing), Services (designing), Core Platform Team (Modern Event Platform (TEC2)), Analytics-EventLogging, EventBus, Analytics

Thu, Oct 11

Pchelolo triaged T206824: Decide whether to use schema references in the schema registry as Normal priority.
Thu, Oct 11, 11:12 PM · Core Platform Team Backlog (Designing), Services (designing), Core Platform Team (Modern Event Platform (TEC2)), Analytics-EventLogging, EventBus, Analytics
Pchelolo added a comment to T206814: CI Support for Schema Registry.

We actually have tests for all of these in the current event-schemas repo, but they're not perfect. What would I like to change:

Thu, Oct 11, 10:21 PM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Pchelolo added projects to T206789: Modern Event Platform: Event Schema Registry: Implementation: Core Platform Team (Modern Event Platform (TEC2)), Core Platform Team Backlog (Later).

Do we need to use a custom meta JSONSchema for this, or can we just add type information outside of the JSONSchema spec in the schemas?

Thu, Oct 11, 9:20 PM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Pchelolo updated the task description for T206789: Modern Event Platform: Event Schema Registry: Implementation.
Thu, Oct 11, 9:12 PM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Pchelolo added a comment to T202642: Investigate how to fix the performance problems caused by CPU bound work on the MCS services.

I've repeated your experiments but cached the parsoid and mw api results in memory to eliminate networking. The variant without promisifying the long sync processing is faster on average, but the promisified version have way more flat distribution of the request latencies.

Thu, Oct 11, 5:34 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Readers-Web-Backlog (Tracking), Services (watching), Mobile-Content-Service
Pchelolo added a comment to T202642: Investigate how to fix the performance problems caused by CPU bound work on the MCS services.

The -c 10 numbers for the promise chain version do look a bit better in this sample, but I think that only reflects a transient network connection improvement. After running multiple times, I haven't seen the numbers consistently better or worse.

Thu, Oct 11, 5:14 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Readers-Web-Backlog (Tracking), Services (watching), Mobile-Content-Service
Pchelolo added a comment to T197000: Modify revision-score schema so that model probabilities won't conflict.

https://github.com/wikimedia/change-propagation/pull/295

Thu, Oct 11, 4:29 PM · Scoring-platform-team, Analytics-Kanban, Patch-For-Review, User-Ladsgroup, Services (watching), ORES, Analytics, EventBus
Pchelolo committed rMSCDb97e8eb14277: TEMP: Stop emitting revision-score event for schema change. (authored by Pchelolo).
TEMP: Stop emitting revision-score event for schema change.
Thu, Oct 11, 4:23 PM
Pchelolo added a comment to T197000: Modify revision-score schema so that model probabilities won't conflict.

@Ottomata I think there should be 2 steps here

  • first we just stop emitting events completely
  • second we deploy this
  • third we re-enable even emitting in the new format
Thu, Oct 11, 3:37 PM · Scoring-platform-team, Analytics-Kanban, Patch-For-Review, User-Ladsgroup, Services (watching), ORES, Analytics, EventBus
Pchelolo added a comment to T203135: ThumbnailRender job fails with 429 errors.

The patch above will make the retries stop, but will preserve all the logging.

Thu, Oct 11, 3:27 PM · Core Platform Team Kanban (Done with CPT), Services (done), MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Thumbor, WMF-Blog-Social-Team, Multimedia
Pchelolo added a comment to T203135: ThumbnailRender job fails with 429 errors.

Can this job be configured to not retry at all? I think that would be ideal. At the end of the day it's a warmup script and it's fine for it to fail.

Thu, Oct 11, 1:53 PM · Core Platform Team Kanban (Done with CPT), Services (done), MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Thumbor, WMF-Blog-Social-Team, Multimedia
Pchelolo added a comment to T202642: Investigate how to fix the performance problems caused by CPU bound work on the MCS services.

@Pchelolo would adding more workers to MCS be a reasonable course of action?

Thu, Oct 11, 12:41 AM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Readers-Web-Backlog (Tracking), Services (watching), Mobile-Content-Service
Pchelolo added a comment to T202642: Investigate how to fix the performance problems caused by CPU bound work on the MCS services.

That said, I doubt we'd see gains this dramatic in production, for a few reasons. First, of course, we wouldn't just be hammering away at a single page and enjoying a 100% cache hit rate; also, we'd expect that many of the requests that reach MCS are in response to page content changes, meaning that cached Document objects from previous renders wouldn't help us. OTOH, most MCS endpoints for both page and feed content include domino.createDocument as a processing step, so at least for popular pages we could expect cached Document objects to be reused at least several times.

Thu, Oct 11, 12:24 AM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Readers-Web-Backlog (Tracking), Services (watching), Mobile-Content-Service

Wed, Oct 10

Pchelolo updated the task description for T206701: MCS must not pass through `vary: accept` from parsoid.
Wed, Oct 10, 9:03 PM · Services (done), Reading-Infrastructure-Team-Backlog (Kanban), Mobile-Content-Service
Pchelolo created T206701: MCS must not pass through `vary: accept` from parsoid.
Wed, Oct 10, 9:00 PM · Services (done), Reading-Infrastructure-Team-Backlog (Kanban), Mobile-Content-Service

Mon, Oct 8

Liuxinyu970226 awarded T159985: Implement language variant support in the REST API a Party Time token.
Mon, Oct 8, 2:27 AM · Readers-Web-Backlog (Tracking), Patch-For-Review, MediaWiki-Language-converter, Parsing-Team, Reading Epics (Platform JS CSS and HTML consolidation), Services (blocked), RESTBase-API, Chinese-Sites, Wikipedia-Android-App-Backlog

Fri, Oct 5

Pchelolo moved T206277: Revision visibility change event sets a wrong performer from Inbox to Doing on the Core Platform Team Backlog (Designing) board.
Fri, Oct 5, 12:17 AM · MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)), Services (done), Core Platform Team Kanban (Done with CPT), Analytics, EventBus
Pchelolo created T206277: Revision visibility change event sets a wrong performer.
Fri, Oct 5, 12:16 AM · MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)), Services (done), Core Platform Team Kanban (Done with CPT), Analytics, EventBus

Thu, Oct 4

Pchelolo moved T206268: Evaluate using TypeScript on node projects from Backlog to watching on the Services board.
Thu, Oct 4, 9:59 PM · Services (watching), Analytics
Pchelolo edited projects for T204154: Kafka JobQueue should respect DB readonly mode, added: Services (done); removed Services (doing), Patch-For-Review.
Thu, Oct 4, 9:59 PM · Services (done), MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Performance-Team (Radar), ChangeProp, WMF-JobQueue
Pchelolo closed T204154: Kafka JobQueue should respect DB readonly mode as Resolved.

I think all the pieces were deployed, so I'm resolving the task. Let's see next week how it goes, will reopen in case of an issue

Thu, Oct 4, 9:59 PM · Services (done), MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Performance-Team (Radar), ChangeProp, WMF-JobQueue
Pchelolo updated the task description for T206259: Open questions when deploying major version bumps to Parsoid's HTML.
Thu, Oct 4, 9:15 PM · Core Platform Team Backlog (Watching / External), Services (watching), RESTBase, Parsoid
Pchelolo updated the task description for T206259: Open questions when deploying major version bumps to Parsoid's HTML.
Thu, Oct 4, 8:19 PM · Core Platform Team Backlog (Watching / External), Services (watching), RESTBase, Parsoid
Pchelolo added a comment to T206259: Open questions when deploying major version bumps to Parsoid's HTML.

Parsoid deploys aren't exactly fast, so there's a period where some nodes are producing the latest version which other nodes don't know how to handle.

Thu, Oct 4, 8:19 PM · Core Platform Team Backlog (Watching / External), Services (watching), RESTBase, Parsoid
Pchelolo added a project to T206259: Open questions when deploying major version bumps to Parsoid's HTML: Services (watching).
Thu, Oct 4, 8:08 PM · Core Platform Team Backlog (Watching / External), Services (watching), RESTBase, Parsoid
Pchelolo closed T199444: ChangeProp logging KafkaConsumer is not connected as Resolved.

Both CP and CPJQ were deployed with the fix. This bug happens quite rarely, so I will close the ticket now. If it happens again we will reopen it.

Thu, Oct 4, 5:40 PM · Services (done), WMF-JobQueue, ChangeProp, Wikimedia-Incident, EventBus, Analytics
Pchelolo added a comment to T203135: ThumbnailRender job fails with 429 errors.

@Gilles this sounds like legit errors. Should we ignore 429 errors and not retry the job then?

Thu, Oct 4, 2:48 PM · Core Platform Team Kanban (Done with CPT), Services (done), MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Thumbor, WMF-Blog-Social-Team, Multimedia

Wed, Oct 3

Pchelolo added a comment to T206186: Create LimitedConcurrencyKafkaConsumer.

Not to forget https://gist.github.com/Pchelolo/0203c271fe97e270e40195bc415cd22d

Wed, Oct 3, 11:49 PM · Services (designing), WMF-JobQueue, ChangeProp
Pchelolo created T206186: Create LimitedConcurrencyKafkaConsumer.
Wed, Oct 3, 11:28 PM · Services (designing), WMF-JobQueue, ChangeProp
Pchelolo added a comment to T199444: ChangeProp logging KafkaConsumer is not connected.

The previous comment also explains why we started seeing the errors after DC switchover. Topics are created on demand and while codfw was not a primary DC a lot of job types did not exist there, so when we switched and new job types started being emitted in codfw the faulty codepath was executed and we run into the race condition.

Wed, Oct 3, 7:17 PM · Services (done), WMF-JobQueue, ChangeProp, Wikimedia-Incident, EventBus, Analytics
Pchelolo edited projects for T199444: ChangeProp logging KafkaConsumer is not connected, added: Services (doing); removed Services (later).
Wed, Oct 3, 7:12 PM · Services (done), WMF-JobQueue, ChangeProp, Wikimedia-Incident, EventBus, Analytics
Pchelolo added a comment to T199444: ChangeProp logging KafkaConsumer is not connected.

I found the reason for this to be happening. At least, one possible reason. And to be honest, I'm embarrassed.

Wed, Oct 3, 7:06 PM · Services (done), WMF-JobQueue, ChangeProp, Wikimedia-Incident, EventBus, Analytics
Pchelolo added a comment to T206003: Beta Cluster: Parsoid config request failures from the MediaWiki API.

09:37 Pchelolo: arlolra: I think I found why parsoid is failing in beta
09:39 arlolra: Pchelolo: /me perks up ... I haven't actually looked yet
09:40 arlolra: I just assumed it was a configuration change
09:41 Pchelolo: when using ApiRequest, https://en.wikipedia.beta.wmflabs.org/w/api.php is provided as a uri which is conf.wiki.apiURI. http.Agent is selected based on it and https agent is assigned. then on line 291 the uri is assigned to mwApiServer===conf.parsoid.mwApiServer which is http://deployment-mediawiki-07.deployment-prep.eqiad.wmflabs/w/api.php
09:41 Pchelolo: so the protocol starts to mismatch the agent
09:41 Pchelolo: we just need to move the agent selection code way to the bottom of ApiRequest.prototype.request method and we should be fine
09:41 Pchelolo: verifying
09:43 Pchelolo: vuala, parsoid works in beta
09:44 Pchelolo: I'm not sure my approach is entirely correct, but parsoid is life-hacked on deployment-parsoid09 now and it works

Wed, Oct 3, 4:49 PM · Services (done), Parsoid, Beta-Cluster-Infrastructure
Pchelolo merged task T113728: VE should send If-Match header to the VRS to send on to RESTbase into T206029: VE does not provide If-Match for html/to/wikitext transforms sometimes.
Wed, Oct 3, 3:42 PM · Services (attic), VisualEditor-MediaWiki, VisualEditor, RESTBase-API, RESTBase
Pchelolo merged T113728: VE should send If-Match header to the VRS to send on to RESTbase into T206029: VE does not provide If-Match for html/to/wikitext transforms sometimes.
Wed, Oct 3, 3:42 PM · VisualEditor, Services (blocked), RESTBase
Pchelolo closed T205935: MCS should request 2.0 Parsoid HTML as Resolved.

Beautiful. Look at the update latency impact!

Wed, Oct 3, 2:52 PM · Reading-Infrastructure-Team-Backlog (Kanban), Services (watching), Mobile-Content-Service
Pchelolo added a project to T206048: Eliminate Parsoid section.offsets table from Cassandra: goodfirstbug.

This might be a bit too advanced to award the goodfirstbug tag, but at least it's very straightforward and makes good exposure to the convoluted storage semantics™ and the hell hole of parsoid.js module, so I will tag it.

Wed, Oct 3, 2:47 PM · goodfirstbug, Services (later), RESTBase
Pchelolo added a comment to T204752: Clean up restrictions tables in cassandra.

hm... on restbase1007:

Wed, Oct 3, 2:41 PM · Services (next), RESTBase
Eevans awarded T206048: Eliminate Parsoid section.offsets table from Cassandra a Goat token.
Wed, Oct 3, 1:06 PM · goodfirstbug, Services (later), RESTBase
mobrovac awarded T205943: RESTBase hacks for compatibility to Parsoid a Mountain of Wealth token.
Wed, Oct 3, 10:54 AM · Services (watching), Parsoid, RESTBase
phuedx awarded T170606: Add Accept header to webrequest logs a Orange Medal token.
Wed, Oct 3, 8:31 AM · Patch-For-Review, Analytics-Kanban, Operations, Traffic, Services (blocked), Analytics
Pchelolo added a comment to T170606: Add Accept header to webrequest logs.

Ok, @Pchelolo gets the persistence award!

Wed, Oct 3, 5:54 AM · Patch-For-Review, Analytics-Kanban, Operations, Traffic, Services (blocked), Analytics

Tue, Oct 2

Pchelolo added a comment to T170606: Add Accept header to webrequest logs.

We did enable the feature after all by looking at requests reaching RESTBase, but that's not very convenient.

Tue, Oct 2, 10:09 PM · Patch-For-Review, Analytics-Kanban, Operations, Traffic, Services (blocked), Analytics
Pchelolo created T206048: Eliminate Parsoid section.offsets table from Cassandra.
Tue, Oct 2, 9:39 PM · goodfirstbug, Services (later), RESTBase
Pchelolo created T206032: api_path log property doesn't show the correct domain.
Tue, Oct 2, 8:47 PM · goodfirstbug, Services (later), HyperSwitch
Pchelolo edited projects for T128525: Strip old metadata from old Parsoid content <head>: mw:TimeUuid, user, comment, added: goodfirstbug; removed Services-next, RESTBase-release-1.0.

Tagging as a good onboarding bug as once the subtask is resolved, it will be easy to fix in code and it provides a great glimpse into what a render is, how Parsoid, RESTBase and VE interacts and what constraints we need to maintain in order to make the 3 works together correctly.

Tue, Oct 2, 8:41 PM · goodfirstbug, Services (next), RESTBase
Pchelolo triaged T206029: VE does not provide If-Match for html/to/wikitext transforms sometimes as Normal priority.
Tue, Oct 2, 8:38 PM · VisualEditor, Services (blocked), RESTBase
Pchelolo added a comment to T128525: Strip old metadata from old Parsoid content <head>: mw:TimeUuid, user, comment.

There's been 700 cases when the If-Match was not supplied over the last month and only 2 user agents:

Tue, Oct 2, 8:23 PM · goodfirstbug, Services (next), RESTBase
Pchelolo removed a parent task for T128525: Strip old metadata from old Parsoid content <head>: mw:TimeUuid, user, comment: T120409: RESTBase should honor wiki-wide deletion/suppression of users.
Tue, Oct 2, 8:17 PM · goodfirstbug, Services (next), RESTBase
Pchelolo removed a subtask for T120409: RESTBase should honor wiki-wide deletion/suppression of users: T128525: Strip old metadata from old Parsoid content <head>: mw:TimeUuid, user, comment.
Tue, Oct 2, 8:17 PM · Analytics, EventBus, Services, RESTBase-release-1.0, RESTBase
Pchelolo added a comment to T160748: Create /v2/schema/:schema_uri endpoint for eventstreams that proxies schemas from eventbus.

So was this done or not after all?

Tue, Oct 2, 8:11 PM · Analytics, Patch-For-Review, Wikimedia-Stream
Pchelolo added a comment to T205319: Modern Event Platform: Stream Configuration Service.

Related: https://phabricator.wikimedia.org/T161027

Tue, Oct 2, 8:10 PM · Core Platform Team (Modern Event Platform (TEC2)), Goal, Services (watching), Analytics-EventLogging, EventBus, Analytics
Pchelolo closed T194376: GlobalRename stuck again at Beta as Resolved.

I believe that's not an issue any more?

Tue, Oct 2, 8:04 PM · Analytics, EventBus, MediaWiki-JobQueue, Beta-Cluster-Infrastructure
Pchelolo added a project to T206003: Beta Cluster: Parsoid config request failures from the MediaWiki API: Services (watching).

The problem is that MW API is configured as 'http://', but for some reason request uses the https Agent - thus the failure.

Tue, Oct 2, 6:40 PM · Services (done), Parsoid, Beta-Cluster-Infrastructure
Pchelolo closed T203929: cpjobqueue should log a warning when there is an HTTP error as Resolved.
Tue, Oct 2, 4:36 PM · Services (doing), ChangeProp
Pchelolo added a comment to T204752: Clean up restrictions tables in cassandra.

LGTM

Tue, Oct 2, 4:17 PM · Services (next), RESTBase
Pchelolo added a comment to T199444: ChangeProp logging KafkaConsumer is not connected.

It does. Yesterday I've restarted JobQueue for that.

Tue, Oct 2, 3:18 PM · Services (done), WMF-JobQueue, ChangeProp, Wikimedia-Incident, EventBus, Analytics
Pchelolo added a comment to T205935: MCS should request 2.0 Parsoid HTML.

@Mholloway Everything is deployed in both RB and Parsoid, but let's wait till tomorrow for MCS deployment in case we need to rollback?

Tue, Oct 2, 4:13 AM · Reading-Infrastructure-Team-Backlog (Kanban), Services (watching), Mobile-Content-Service
Pchelolo created T205943: RESTBase hacks for compatibility to Parsoid.
Tue, Oct 2, 1:28 AM · Services (watching), Parsoid, RESTBase

Mon, Oct 1

Pchelolo created T205935: MCS should request 2.0 Parsoid HTML.
Mon, Oct 1, 10:41 PM · Reading-Infrastructure-Team-Backlog (Kanban), Services (watching), Mobile-Content-Service
Pchelolo moved T205919: TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline from Backlog to watching on the Services board.
Mon, Oct 1, 10:38 PM · Core Platform Team Backlog (Watching / External), Services (watching), Release Pipeline, Operations, Release-Engineering-Team
Pchelolo edited projects for T205911: Track and install additional npm packages for all service container images, added: Services (watching); removed Services.
Mon, Oct 1, 6:59 PM · Core Platform Team Backlog (Watching / External), Services (watching), Release-Engineering-Team, Operations, Release Pipeline
Pchelolo added a comment to T205911: Track and install additional npm packages for all service container images.

There is a base set of npm packages that are used by all services. Currently, server.js installs heapdump and gc-stats (possibly among others).

Mon, Oct 1, 6:59 PM · Core Platform Team Backlog (Watching / External), Services (watching), Release-Engineering-Team, Operations, Release Pipeline
Pchelolo updated the task description for T205911: Track and install additional npm packages for all service container images.
Mon, Oct 1, 6:54 PM · Core Platform Team Backlog (Watching / External), Services (watching), Release-Engineering-Team, Operations, Release Pipeline
Pchelolo added a comment to T204154: Kafka JobQueue should respect DB readonly mode.

Related patch for change-prop https://github.com/wikimedia/change-propagation/pull/292

Mon, Oct 1, 6:30 PM · Services (done), MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Performance-Team (Radar), ChangeProp, WMF-JobQueue
Pchelolo renamed T199444: ChangeProp logging KafkaConsumer is not connected from After switchover to codfw Job queue ChangeProp logged KafkaConsumer is not connected to ChangeProp logging KafkaConsumer is not connected.
Mon, Oct 1, 4:51 PM · Services (done), WMF-JobQueue, ChangeProp, Wikimedia-Incident, EventBus, Analytics
Pchelolo edited projects for T67117: Hovercards: Display popups across wikis (interwiki popups), added: Services (watching); removed Services.

Adding services as enabling this would probably require access via CORs or locally to other projects. e.g. wikivoyage would need to be able to access summaries from wikipedia. I've not looked into whether this is possible yet :)

Mon, Oct 1, 2:05 PM · Core Platform Team Backlog (Watching / External), Services (watching), Readers-Web-Backlog (Design), Page-Previews

Wed, Sep 26

Pchelolo added a comment to T205497: [Regression pre-wmf.23] REST API on Beta cluster returns content of different pages than requested (breaks VE).

@Ryasmeen try now

Wed, Sep 26, 7:38 PM · User-Ryasmeen, Services (done), VisualEditor, RESTBase, Parsoid
Pchelolo added a comment to T205497: [Regression pre-wmf.23] REST API on Beta cluster returns content of different pages than requested (breaks VE).

@Ryasmeen can you list the titles you are trying now? I bet my right hand it's Varnish cached something wrong.

Wed, Sep 26, 7:24 PM · User-Ryasmeen, Services (done), VisualEditor, RESTBase, Parsoid
Pchelolo added projects to T205572: Optimize lang conversion and content negotiation combo: Services (designing), Parsoid, RESTBase.
Wed, Sep 26, 7:16 PM · RESTBase, Parsoid, Services (designing)
Pchelolo created T205572: Optimize lang conversion and content negotiation combo.
Wed, Sep 26, 7:15 PM · RESTBase, Parsoid, Services (designing)

Tue, Sep 25

Pchelolo added a comment to T184545: Finish tests for reading lists service.

In my opinion - yes. Thank you.

Tue, Sep 25, 11:06 PM · Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban), Technical-Debt, Reading List Service

Mon, Sep 24

Pchelolo added a comment to T204880: RESTBase should 404 if it cannot satisfy requested TID.

Ok, I guess you're right. For the sake of correctness here's a PR https://github.com/wikimedia/restbase/pull/1066

Mon, Sep 24, 7:26 PM · User-Ryasmeen, Services (done), Reading-Infrastructure-Team-Backlog, VisualEditor, RESTBase
Pchelolo added a comment to T205319: Modern Event Platform: Stream Configuration Service.

As an engineer, I want to specify concrete settings for different topics like the number of partitions or the retention interval. T157092

Mon, Sep 24, 6:32 PM · Core Platform Team (Modern Event Platform (TEC2)), Goal, Services (watching), Analytics-EventLogging, EventBus, Analytics
Pchelolo added a comment to T203039: Storage of data for recommendation API.

@Pchelolo where would database settings live? Would it be the service codebase itself or do we have a separate repository for that?

Mon, Sep 24, 4:23 PM · Operations, DBA, Services (designing), Research

Fri, Sep 21

Pchelolo added a comment to T203854: Expand usage of x-triggered-by.

These are different things designed for different purpose, so you should do both.

Fri, Sep 21, 8:24 PM · Core Platform Team (Security, stability, performance and scalability (TEC1)), Core Platform Team Backlog (Later), goodfirstbug, service-template-node, Services (later), Reading-Infrastructure-Team-Backlog, ChangeProp, Mobile-Content-Service, RESTBase
Pchelolo added a comment to T203854: Expand usage of x-triggered-by.

Should Parsoid be propagating x-triggered-by from request to response?

Fri, Sep 21, 3:57 PM · Core Platform Team (Security, stability, performance and scalability (TEC1)), Core Platform Team Backlog (Later), goodfirstbug, service-template-node, Services (later), Reading-Infrastructure-Team-Backlog, ChangeProp, Mobile-Content-Service, RESTBase
Pchelolo edited projects for T204981: Keeping Node services documentation in sync, added: Services (designing); removed Services.

This is a big and interesting question that I've been thinking about myself for a long long time, so it needs a lot of discussions.

Fri, Sep 21, 3:56 PM · Services (designing), Reading-Infrastructure-Team-Backlog

Sep 19 2018

Pchelolo added a project to T204880: RESTBase should 404 if it cannot satisfy requested TID: VisualEditor.
Sep 19 2018, 8:35 PM · Services (done), User-Ryasmeen, Reading-Infrastructure-Team-Backlog, VisualEditor, RESTBase