Change #1113458 had a related patch set uploaded (by Mvolz; author: Mvolz):

[mediawiki/services/citoid@master] Include headers required for api gateway

https://gerrit.wikimedia.org/r/1113458

In T361576#10483801, @hnowlan wrote:

A test rollout of the routing for testwiki was successful when testing internally from a routing perspective, but during the process it was noted that citoid lacks two things we require for routing via the rest-gateway - a cache-control header and either an etag or a last-modified header. These need to be added before we can proceed. Previously these were supplied for free by restbase, but now need to be emitted by the service itself.

I've put in a change with just ETag and cache-control for now.

Here's what I'm getting from citoid native:

access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-webkit-csp: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
content-type: application/json; charset=utf-8
content-length: 446
vary: Accept-Encoding
date: Wed, 22 Jan 2025 13:22:09 GMT
x-envoy-upstream-service-time: 685
server: production-tls

And this from restbase:

access-control-allow-origin: *
access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
access-control-expose-headers: etag
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'none'; frame-ancestors 'none'
x-content-security-policy: default-src 'none'; frame-ancestors 'none'
x-webkit-csp: default-src 'none'; frame-ancestors 'none'
content-type: application/json; charset=utf-8
vary: Accept-Encoding
date: Wed, 22 Jan 2025 13:16:31 GMT
server: restbase1038
content-location: https://en.wikipedia.org/api/rest_v1/data/citation/mediawiki/https%3A%2F%2Fwww.tandfonline.com%2Fdoi%2Ffull%2F10.1080%2F02724634.2024.2381505%3Fsrc%3Dtrending
access-control-allow-methods: GET,HEAD
referrer-policy: origin-when-cross-origin
cache-control: private, max-age=0, s-maxage=0, must-revalidate
age: 3
x-cache: cp3066 miss, cp3066 pass
x-cache-status: pass
server-timing: cache;desc="pass", host;desc="cp3066"
strict-transport-security: max-age=106384710; includeSubDomains; preload
report-to: { "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
nel: { "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}
set-cookie: 
x-client-ip: 
set-cookie: GeoIP=; Path=/; secure; Domain=.wikipedia.org
set-cookie: NetworkProbeLimit=0.001;Path=/;Secure;SameSite=Lax;Max-Age=3600
accept-ranges: bytes

I see cache-control is missing in citoid native only, but am not seeing etag or last-modified in either response, not even the restbase one.... also are there any more of these you think we should be setting or will api gateway handle the rest?

In T361576#10483956, @Mvolz wrote:

In T361576#10483801, @hnowlan wrote:

A test rollout of the routing for testwiki was successful when testing internally from a routing perspective, but during the process it was noted that citoid lacks two things we require for routing via the rest-gateway - a cache-control header and either an etag or a last-modified header. These need to be added before we can proceed. Previously these were supplied for free by restbase, but now need to be emitted by the service itself.

I've put in a change with just ETag and cache-control for now.

Here's what I'm getting from citoid native:

access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-content-security-policy: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
x-webkit-csp: default-src 'self'; object-src 'none'; media-src *; img-src *; style-src *; frame-ancestors 'self'
content-type: application/json; charset=utf-8
content-length: 446
vary: Accept-Encoding
date: Wed, 22 Jan 2025 13:22:09 GMT
x-envoy-upstream-service-time: 685
server: production-tls

And this from restbase:

access-control-allow-origin: *
access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
access-control-expose-headers: etag
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'none'; frame-ancestors 'none'
x-content-security-policy: default-src 'none'; frame-ancestors 'none'
x-webkit-csp: default-src 'none'; frame-ancestors 'none'
content-type: application/json; charset=utf-8
vary: Accept-Encoding
date: Wed, 22 Jan 2025 13:16:31 GMT
server: restbase1038
content-location: https://en.wikipedia.org/api/rest_v1/data/citation/mediawiki/https%3A%2F%2Fwww.tandfonline.com%2Fdoi%2Ffull%2F10.1080%2F02724634.2024.2381505%3Fsrc%3Dtrending
access-control-allow-methods: GET,HEAD
referrer-policy: origin-when-cross-origin
cache-control: private, max-age=0, s-maxage=0, must-revalidate
age: 3
x-cache: cp3066 miss, cp3066 pass
x-cache-status: pass
server-timing: cache;desc="pass", host;desc="cp3066"
strict-transport-security: max-age=106384710; includeSubDomains; preload
report-to: { "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
nel: { "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}
set-cookie: 
x-client-ip: 
set-cookie: GeoIP=; Path=/; secure; Domain=.wikipedia.org
set-cookie: NetworkProbeLimit=0.001;Path=/;Secure;SameSite=Lax;Max-Age=3600
accept-ranges: bytes

I see cache-control is missing in citoid native only, but am not seeing etag or last-modified in either response, not even the restbase one.... also are there any more of these you think we should be setting or will api gateway handle the rest?

I think that should be all we need to satisfy - I hadn't realised that restbase doesn't give us etags for citoid, I was assuming it did it for all endpoints as adding it to other services has been required previously. The change you've supplied looks good. The gateway should pass most other headers as needed and it will pass on the CSP headers as provided

Change #1113458 merged by jenkins-bot:

[mediawiki/services/citoid@master] Include headers required for api gateway

https://gerrit.wikimedia.org/r/1113458

In T361576#10488265, @gerritbot wrote:

Change #1113458 merged by jenkins-bot:

[mediawiki/services/citoid@master] Include headers required for api gateway

https://gerrit.wikimedia.org/r/1113458

This is now deployed and I confirmed both the native restbase shim and restbase installs are returning cache-control and etag.

Change #1115056 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: directly route to citoid on testwiki

https://gerrit.wikimedia.org/r/1115056

Change #1115056 merged by Hnowlan:

[operations/puppet@production] trafficserver: directly route to citoid on testwiki

https://gerrit.wikimedia.org/r/1115056

This is now available to test on test Wikipedia:

https://test.wikipedia.org/wiki/Main_Page

It generally seems to be working.

In T361576#10515182, @Mvolz wrote:

This is now available to test on test Wikipedia:

https://test.wikipedia.org/wiki/Main_Page

It generally seems to be working.

@Mvolz : It's not clear to me what needs to be tested here from QA side. This is what I am seeing:

Could you provide some guidance with test instructions, if I need to test in a different method.

In T361576#10519300, @Ryasmeen wrote:

In T361576#10515182, @Mvolz wrote:

This is now available to test on test Wikipedia:

https://test.wikipedia.org/wiki/Main_Page

It generally seems to be working.

@Mvolz : It's not clear to me what needs to be tested here from QA side. This is what I am seeing:

Could you provide some guidance with test instructions, if I need to test in a different method.

Yeah, I guess just that the behaviour of adding citations in test Wikipedia is not broken / works as in other Wikipedias. I tried to configure it as similarly as possible so it should be mostly the same. If you're happy it works, we can sign off on moving to group 0 wikis.

Change #1117926 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: add media-list to supported mobileapps pages

https://gerrit.wikimedia.org/r/1117926

Change #1117926 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: add media-list to supported mobileapps pages

https://gerrit.wikimedia.org/r/1117926

Change #1118204 had a related patch set uploaded (by Mvolz; author: Mvolz):

[mediawiki/services/citoid@master] Add headers to access-control-allow-headers

https://gerrit.wikimedia.org/r/1118204

Change #1122542 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: roll restbaseless citoid out to group0 wikis

https://gerrit.wikimedia.org/r/1122542

Change #1122542 merged by Hnowlan:

[operations/puppet@production] trafficserver: roll restbaseless citoid out to group0 wikis

https://gerrit.wikimedia.org/r/1122542

Change #1118204 merged by Mvolz:

[mediawiki/services/citoid@master] Add headers to access-control-allow-headers

https://gerrit.wikimedia.org/r/1118204

Change #1122973 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] citoid: migrate group1 wikis to use rest-gateway instead of restbase

https://gerrit.wikimedia.org/r/1122973

Change #1122973 merged by Hnowlan:

[operations/puppet@production] citoid: migrate group1 wikis to use rest-gateway instead of restbase

https://gerrit.wikimedia.org/r/1122973

Change #1124418 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: remove restbase from citoid request path everywhere

https://gerrit.wikimedia.org/r/1124418

Change #1124766 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: fix hostnames for citoid requests

https://gerrit.wikimedia.org/r/1124766

Change #1124766 merged by Hnowlan:

[operations/puppet@production] trafficserver: fix hostnames for citoid requests

https://gerrit.wikimedia.org/r/1124766

In T361576#10520404, @Mvolz wrote:

In T361576#10519300, @Ryasmeen wrote:

In T361576#10515182, @Mvolz wrote:

This is now available to test on test Wikipedia:

https://test.wikipedia.org/wiki/Main_Page

It generally seems to be working.

@Mvolz : It's not clear to me what needs to be tested here from QA side. This is what I am seeing:

Could you provide some guidance with test instructions, if I need to test in a different method.

Yeah, I guess just that the behaviour of adding citations in test Wikipedia is not broken / works as in other Wikipedias. I tried to configure it as similarly as possible so it should be mostly the same. If you're happy it works, we can sign off on moving to group 0 wikis.

@Ryasmeen This is now available for testing on group 1 wikis i.e. it.wikipedia.org, he.wikipedia.org, and ca.wikipedia.org

Change #1125461 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: route citoid via rest-gateway for all sites

https://gerrit.wikimedia.org/r/1125461

Change #1113182 abandoned by Hnowlan:

[operations/puppet@production] trafficserver: route citoid via rest-gateway for all sites

https://gerrit.wikimedia.org/r/1113182

Change #1125461 merged by Hnowlan:

[operations/puppet@production] trafficserver: route citoid via rest-gateway for all sites

https://gerrit.wikimedia.org/r/1125461

Citoid is now fully routed via the rest gateway for all wikis.

Should we resolve this? Or is there more work to do?

Change #1131008 had a related patch set uploaded (by Mvolz; author: Mvolz):

[mediawiki/extensions/Citoid@master] [WIP] Use restful api pattern for all configs

https://gerrit.wikimedia.org/r/1131008

Change #1131008 merged by jenkins-bot:

[mediawiki/extensions/Citoid@master] Use restful api pattern for all configs

https://gerrit.wikimedia.org/r/1131008

In T361576#10721517, @gerritbot wrote:

Change #1131008 merged by jenkins-bot:

[mediawiki/extensions/Citoid@master] Use restful api pattern for all configs

https://gerrit.wikimedia.org/r/1131008

Adding Editing QA to test that this change does not cause any side effects. It was designed to be backwards compatible so it in theory *shouldn't* change behaviour in prod (or locally.)

The progress here is awesome! @Mvolz please let us know when it's safe to remove the logic from the RESTBase codebase.

In T361576#10760706, @MSantos wrote:

The progress here is awesome! @Mvolz please let us know when it's safe to remove the logic from the RESTBase codebase.

The only think we're currently using *in* restbase right now is the swagger documentation; I don't think we've decided where to host that, ultimately. @hnowlan ideas?

We could host it from the root, or potentially move it to Mediawiki Rest API spot (since they are trialling swagger even though it's not actually a php service, but maybe we could write a shell for it and put it in there? cc: @HCoplin-WMF )

I think from the root would be /simpler/, but it doesn't make it super findable if it's not in one of the two main "lists".

Change #1124418 abandoned by Hnowlan:

[operations/puppet@production] trafficserver: remove restbase from citoid request path everywhere

Reason:

obsoleted by other changes

https://gerrit.wikimedia.org/r/1124418

Change #1139808 had a related patch set uploaded (by Mvolz; author: Mvolz):

[operations/mediawiki-config@master] Change citoid config for test wiki

https://gerrit.wikimedia.org/r/1139808

Change #1154773 had a related patch set uploaded (by Mvolz; author: Mvolz):

[mediawiki/services/citoid@master] Switch most tests to use restful implementation

https://gerrit.wikimedia.org/r/1154773

Change #1154773 merged by jenkins-bot:

[mediawiki/services/citoid@master] Switch most tests to use restful implementation

https://gerrit.wikimedia.org/r/1154773

Change #1139808 merged by jenkins-bot:

[operations/mediawiki-config@master] Change citoid config for test wiki

https://gerrit.wikimedia.org/r/1139808

Mentioned in SAL (#wikimedia-operations) [2025-06-16T13:08:46Z] <phuedx@deploy1003> Started scap sync-world: Backport for [[gerrit:1156872|ext-EventStreamConfig: Update product_metrics.web_base stream (T395692)]], [[gerrit:1127960|Set $wgCentralAuthAutomaticGlobalGroups for global IP reveal group (T376315)]], [[gerrit:1153307|Enable temporary accounts onboarding dialog on WMF wikis (T395933)]], [[gerrit:1139808|Change citoid config for test wiki (T361576)]]

Mentioned in SAL (#wikimedia-operations) [2025-06-16T13:10:40Z] <phuedx@deploy1003> phuedx, mvolz, dreamyjazz, tchanders: Backport for [[gerrit:1156872|ext-EventStreamConfig: Update product_metrics.web_base stream (T395692)]], [[gerrit:1127960|Set $wgCentralAuthAutomaticGlobalGroups for global IP reveal group (T376315)]], [[gerrit:1153307|Enable temporary accounts onboarding dialog on WMF wikis (T395933)]], [[gerrit:1139808|Change citoid config for test wiki (T361576)]] synced to t

Change #1164179 had a related patch set uploaded (by Mvolz; author: Mvolz):

[operations/mediawiki-config@master] Redo "Change citoid config for test wiki"

https://gerrit.wikimedia.org/r/1164179

Change #1166186 had a related patch set uploaded (by Mvolz; author: Mvolz):

[operations/mediawiki-config@master] Update parameter name for wikidata/citoid integration

https://gerrit.wikimedia.org/r/1166186

Change #1164179 merged by jenkins-bot:

[operations/mediawiki-config@master] Redo "Change citoid config for test wiki"

https://gerrit.wikimedia.org/r/1164179

Change #1180103 had a related patch set uploaded (by Mvolz; author: Mvolz):

[operations/mediawiki-config@master] Remove all references to deprecated parameter

https://gerrit.wikimedia.org/r/1180103

Change #1180112 had a related patch set uploaded (by Mvolz; author: Mvolz):

[mediawiki/extensions/Citoid@master] Add soft deprecation warnings for configuration variable

https://gerrit.wikimedia.org/r/1180112

Change #1180112 merged by jenkins-bot:

[mediawiki/extensions/Citoid@master] Add soft deprecation warnings for configuration variable

https://gerrit.wikimedia.org/r/1180112

Change #1180103 merged by jenkins-bot:

[operations/mediawiki-config@master] Remove all references to deprecated parameter

https://gerrit.wikimedia.org/r/1180103

Mentioned in SAL (#wikimedia-operations) [2025-09-03T13:30:11Z] <mvolz@deploy1003> Started scap sync-world: Backport for [[gerrit:1180103|Remove all references to deprecated parameter (T361576)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-03T13:36:49Z] <mvolz@deploy1003> mvolz: Backport for [[gerrit:1180103|Remove all references to deprecated parameter (T361576)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-09-03T13:47:24Z] <mvolz@deploy1003> Finished scap sync-world: Backport for [[gerrit:1180103|Remove all references to deprecated parameter (T361576)]] (duration: 17m 13s)

Change #1166186 abandoned by Mvolz:

[operations/mediawiki-config@master] Update parameter name for wikidata/citoid integration

Reason:

Different approach

https://gerrit.wikimedia.org/r/1166186

Change #1190618 had a related patch set uploaded (by Mvolz; author: Mvolz):

[mediawiki/extensions/Citoid@master] Deprecate restbase config for wikibase module

https://gerrit.wikimedia.org/r/1190618

Change #1190618 merged by jenkins-bot:

[mediawiki/extensions/Citoid@master] Deprecate restbase config for wikibase module

https://gerrit.wikimedia.org/r/1190618

Change #1204813 had a related patch set uploaded (by Mvolz; author: Mvolz):

[operations/mediawiki-config@master] Remove deprecated parameter

https://gerrit.wikimedia.org/r/1204813

Change #1218229 had a related patch set uploaded (by Thiemo Kreuz (WMDE); author: Thiemo Kreuz (WMDE)):

[mediawiki/extensions/Citoid@master] Fix inconsistent behavior when auto citations are not setup

https://gerrit.wikimedia.org/r/1218229

Change #1218229 merged by jenkins-bot:

[mediawiki/extensions/Citoid@master] Fix inconsistent behavior when auto citations are not setup

https://gerrit.wikimedia.org/r/1218229

Change #1204813 merged by jenkins-bot:

[operations/mediawiki-config@master] Remove deprecated parameter

https://gerrit.wikimedia.org/r/1204813

Mentioned in SAL (#wikimedia-operations) [2026-01-28T14:12:14Z] <mvolz@deploy2002> Started scap sync-world: Backport for [[gerrit:1204813|Remove deprecated parameter (T361576)]]

Mentioned in SAL (#wikimedia-operations) [2026-01-28T14:17:00Z] <mvolz@deploy2002> mvolz: Backport for [[gerrit:1204813|Remove deprecated parameter (T361576)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-01-28T14:35:47Z] <mvolz@deploy2002> Finished scap sync-world: Backport for [[gerrit:1204813|Remove deprecated parameter (T361576)]] (duration: 23m 33s)

Following up on this comment (a few months late, sorry):

The only think we're currently using *in* restbase right now is the swagger documentation; I don't think we've decided where to host that, ultimately.

We actually recently moved the Swagger docs to the new REST Sandbox and implemented a redirect from the RESTBase docs. Announcement was in Tech News, for more info.

The citation section is currently in the docs; is there still outstanding work here that needs to happen on that front? Or are we good from a docs migration perspective?

In T361576#11562523, @HCoplin-WMF wrote:

Following up on this comment (a few months late, sorry):

The only think we're currently using *in* restbase right now is the swagger documentation; I don't think we've decided where to host that, ultimately.

We actually recently moved the Swagger docs to the new REST Sandbox and implemented a redirect from the RESTBase docs. Announcement was in Tech News, for more info.

The citation section is currently in the docs; is there still outstanding work here that needs to happen on that front? Or are we good from a docs migration perspective?

A redirect to the restbase docs still requires restbase be running. The plan is to kill restbase entirely.

We don't currently serve the swagger docs from api-gateway, but we could. And then we could redirect to that.

Or we could just duplicate the docs and put it in the Extension or something, and then have it fetch from the extension. That's less clean in some ways but cleaner in others.

Change #1235002 had a related patch set uploaded (by Mvolz; author: Mvolz):

[mediawiki/extensions/Citoid@master] [WIP] Hard deprecate $wgWBCitoidFullRestbaseURL

https://gerrit.wikimedia.org/r/1235002