Intermittent internal API errors with Flow
Closed, ResolvedPublic
Actions

Description

Pywikibot tests are failing lately with the following error:

======================================================================

ERROR: test_topic_moderation (tests.flow_tests.TestFlowTopic)

Test Topic functions about moderation.

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/build/wikimedia/pywikibot/tests/flow_tests.py", line 245, in test_topic_moderation

    self.assertFalse(topic_hidden.is_locked)

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/flow.py", line 220, in is_locked

    return self.root._current_revision['isLocked']

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/flow.py", line 214, in root

    self._root = Post.fromJSON(self, self.uuid, self._data)

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/flow.py", line 65, in uuid

    self._load_uuid()

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/flow.py", line 55, in _load_uuid

    self._uuid = self._load()['workflowId']

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/flow.py", line 160, in _load

    self._data = self.site.load_topic(self, format)

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/site/__init__.py", line 1390, in callee

    return fn(self, *args, **kwargs)

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/site/__init__.py", line 7306, in load_topic

    data = req.submit()

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/data/api.py", line 2040, in submit

    if self._internal_api_error(code, error, result):

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/data/api.py", line 1910, in _internal_api_error

    raise e

APIMWException: internal_api_error_Flow\Exception\NoParserException: [d92f3ec5-f670-4abb-b814-f07f5c9f62b7] Exception caught: Request to parsoid for "html" to "wikitext" conversion of content connected to title "Topic:U5y53rn0dp6h70nw" failed: 503

[errorclass: Flow\Exception\NoParserException;

 *: Flow\Exception\NoParserException at /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Conversion/Utils.php(174)

#0 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Conversion/Utils.php(66): Flow\Conversion\Utils::parsoid(string, string, string, Title)

#1 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Model/AbstractRevision.php(430): Flow\Conversion\Utils::convert(string, string, string, Title)

#2 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Templating.php(159): Flow\Model\AbstractRevision->getContent(string)

#3 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Formatter/RevisionFormatter.php(289): Flow\Templating->getContent(Flow\Model\PostRevision, string)

#4 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Formatter/TopicFormatter.php(45): Flow\Formatter\RevisionFormatter->formatApi(Flow\Formatter\TopicRow, DerivativeContext)

#5 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Block/Topic.php(663): Flow\Formatter\TopicFormatter->formatApi(Flow\Model\Workflow, array, DerivativeContext)

#6 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Block/Topic.php(565): Flow\Block\TopicBlock->renderTopicApi(array)

#7 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Api/ApiFlowBaseGet.php(33): Flow\Block\TopicBlock->renderApi(array)

#8 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Api/ApiFlow.php(105): Flow\Api\ApiFlowBaseGet->execute()

#9 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(1580): Flow\Api\ApiFlow->execute()

#10 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(523): ApiMain->executeAction()

#11 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(494): ApiMain->executeActionWithErrorHandling()

#12 /srv/mediawiki/php-1.35.0-wmf.27/api.php(84): ApiMain->execute()

#13 /srv/mediawiki/w/api.php(3): require(string)

#14 {main};

 servedby: mw1317]

https://travis-ci.org/github/wikimedia/pywikibot/jobs/672473143#L11364

UPDATE 2020/04/10: see T249909 and discussion below; happens on mediawiki.org and from web UI (not just pywikibot).

Details

Subject	Repo	Branch	Lines +/-
parsoid: allow retries for connection resets in envoy	operations/puppet	production	+1 -2
[test] Ignore flow_tests until upstream issue solved	pywikibot/core	master	+6 -0
services_proxy: temp mitigation for intermittent parsoid requests failures	operations/puppet	production	+2 -1
services_proxy: retry on connect failures for parsoid	operations/puppet	production	+3 -0

Customize query in gerrit

Related Objects

Mentioned In: T273111: Flow\Exception\NoParserException when reading or editing flow board
T255105: Intermittent internal API error MWUnknownContentModelException
T249997: Flow failing with Error contacting the Parsoid/RESTBase server (HTTP 400)
rPWBCbccf4f1c8d0c: [test] Ignore flow_tests until upstream issue solved
T249909: Flow/Parsoid permanently delete the entered text after errors
T249828: test_topic of flow_tests.TestFlowTopic due to additional edit
T249745: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable"
Mentioned Here: T249997: Flow failing with Error contacting the Parsoid/RESTBase server (HTTP 400)
T249909: Flow/Parsoid permanently delete the entered text after errors
T249745: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable"
T249535: VE and Flow fail with "Error contacting the Parsoid/RESTBase server (HTTP 404)" / "…(HTTP 411)" on officewiki

Event Timeline

Dvorapa created this task.Apr 8 2020, 12:28 PM

Restricted Application added projects: Pywikibot, Platform Engineering. · View Herald TranscriptApr 8 2020, 12:28 PM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald Transcript

Dvorapa moved this task from Backlog to Upstream issues on the Pywikibot-tests board.Apr 8 2020, 12:28 PM

Dvorapa moved this task from Backlog to Upstream issues on the Pywikibot board.

Dvorapa updated the task description. (Show Details)

Anomie removed projects: Platform Engineering, MediaWiki-Action-API.Apr 8 2020, 12:59 PM

Dvorapa added projects: StructuredDiscussions, Pywikibot-Flow.Apr 8 2020, 1:01 PM

Restricted Application added a project: Growth-Team. · View Herald TranscriptApr 8 2020, 1:01 PM

This looks like a configuration error with Parsoid, probably related to the Parsoid-PHP project. You get the same error if trying to post a comment using VE on a Flow board.

Tagging @ssastry and @Jdforrester-WMF in case they know of what needs to be fixed in the config repo. Other group0 wikis (e.g. mediawiki.org) seem fine, so it seems likely to me that this is a config issue for testwiki and not something with the train.

In T249705#6039690, @kostajh wrote:

Tagging @ssastry and @Jdforrester-WMF in case they know of what needs to be fixed in the config repo. Other group0 wikis (e.g. mediawiki.org) seem fine, so it seems likely to me that this is a config issue for testwiki and not something with the train.

Is this is a new issue (since this train)? Or has this been broken on testwiki for a while?

ssastry edited projects, added Parsoid; removed Parsoid-PHP.Apr 8 2020, 2:56 PM

I've noticed this today, but it could fail for several days already (as Pywikibot tests were broken last week)

Adding as a train blocker until/unless we can prove that this isn't related to a group0 deployment.

(It's likely as @kostajh suggests that this isn't train-related, but I'm trying to err on the side of caution.)

Full config diff:

diff -d wmf-config/config-cache/conf-production-testwiki.json wmf-config/config-cache/conf-production-mediawikiwiki.json
3,8c3,4
<         "accountcreator": {
<             "override-antispoof": true,
<             "tboverride": true
<         },
<         "filemover": {
<             "movefile": true
---
>         "*": {
>             "flow-hide": false
11,18c7,11
<             "upload_by_url": true,
<             "upload": true,
<             "pagelang": true
<         },
<         "templateeditor": {
<             "templateeditor": true,
<             "tboverride": true,
<             "editcontentmodel": true
---
>             "flow-create-board": true,
>             "upload": false,
>             "reupload": false,
>             "reupload-own": false,
>             "reupload-shared": false
20,23c13,18
<         "sysop": {
<             "deleterevision": true,
<             "templateeditor": true,
<             "securepoll-create-poll": true
---
>         "autoconfirmed": {
>             "flow-hide": true,
>             "upload": false,
>             "reupload": false,
>             "reupload-own": false,
>             "reupload-shared": false
25,26c20,21
<         "reviewer": {
<             "stablesettings": true
---
>         "autopatrolled": {
>             "autopatrol": true
28,31c23,27
<         "researcher": {
<             "browsearchive": true,
<             "deletedhistory": true,
<             "apihighlimits": true
---
>         "uploader": {
>             "upload": true,
>             "reupload": true,
>             "reupload-own": true,
>             "reupload-shared": true
33,34c29,30
<         "rollbacker": {
<             "rollback": true
---
>         "translationadmin": {
>             "autopatrol": true
36,39c32,35
<         "centralnoticeadmin": {
<             "banner-protect": true,
<             "centralnotice-admin": true,
<             "editinterface": true
---
>         "sysop": {
>             "reupload": true,
>             "reupload-own": true,
>             "reupload-shared": true
114c110
<         "default": 2
---
>         "default": 25
117c113
<         "default": 0.05
---
>         "default": 0.3
122a119,123
>         "sysop": [
>             "autopatrolled",
>             "uploader",
>             "ipblock-exempt"
>         ],
124,126c125,127
<             "researcher",
<             "centralnoticeadmin",
<             "flow-bot",
---
>             "autopatrolled",
>             "transwiki",
>             "import",
133,140d133
<         ],
<         "sysop": [
<             "filemover",
<             "rollbacker",
<             "accountcreator",
<             "confirmed",
<             "templateeditor",
<             "ipblock-exempt"
143c136
<     "wgAllowCopyUploads": true,
---
>     "wgAllowCopyUploads": false,
148c141
<     "wgAllowRequiringEmailForResets": false,
---
>     "wgAllowRequiringEmailForResets": true,
152c145
<     "wgAppleTouchIcon": "/static/apple-touch/wikipedia.png",
---
>     "wgAppleTouchIcon": "/static/apple-touch/mediawiki.png",
156c149
<     "wgArticlePlaceholderImageProperty": "P47",
---
>     "wgArticlePlaceholderImageProperty": "P18",
182c175
<     "wgCanonicalServer": "https://test.wikipedia.org",
---
>     "wgCanonicalServer": "https://www.mediawiki.org",
187c180
<     "wgCategoryCollation": "uca-default-u-kn",
---
>     "wgCategoryCollation": "uca-default",
197c190
<     "wgCheckUserEnableSpecialInvestigate": true,
---
>     "wgCheckUserEnableSpecialInvestigate": false,
216c209
<     "wgCirrusSearchCompletionSettings": "fuzzy",
---
>     "wgCirrusSearchCompletionSettings": "fuzzy-subphrases",
218,219c211,212
<         "build": false,
<         "use": false,
---
>         "build": true,
>         "use": true,
239c232
<     "wgCirrusSearchEnableCrossProjectSearch": true,
---
>     "wgCirrusSearchEnableCrossProjectSearch": false,
292,351c285
<     "wgCirrusSearchLanguageToWikiMap": {
<         "ar": "ar",
<         "ay": "ay",
<         "az": "az",
<         "bg": "bg",
<         "bn": "bn",
<         "ca": "ca",
<         "cs": "cs",
<         "da": "da",
<         "de": "de",
<         "el": "el",
<         "en": "en",
<         "es": "es",
<         "et": "et",
<         "fa": "fa",
<         "fi": "fi",
<         "fil": "tl",
<         "fr": "fr",
<         "gu": "gu",
<         "he": "he",
<         "hi": "hi",
<         "hr": "hr",
<         "hu": "hu",
<         "id": "id",
<         "it": "it",
<         "ja": "ja",
<         "km": "km",
<         "ko": "ko",
<         "lt": "lt",
<         "lv": "lv",
<         "mk": "mk",
<         "ml": "ml",
<         "mr": "mr",
<         "ms": "ms",
<         "nb": "no",
<         "nl": "nl",
<         "no": "no",
<         "pa": "pa",
<         "pl": "pl",
<         "pt": "pt",
<         "ro": "ro",
<         "ru": "ru",
<         "si": "si",
<         "sk": "sk",
<         "sl": "sl",
<         "sq": "sq",
<         "sv": "sv",
<         "sw": "sw",
<         "ta": "ta",
<         "te": "te",
<         "th": "th",
<         "tl": "tl",
<         "tr": "tr",
<         "uk": "uk",
<         "ur": "ur",
<         "vi": "vi",
<         "zh": "zh",
<         "zh-cn": "zh",
<         "zh-tw": "zh"
<     },
---
>     "wgCirrusSearchLanguageToWikiMap": [],
353,354c287,288
<         "user": 0,
<         "wiki": 0
---
>         "user": 5,
>         "wiki": 2.5
363c297,303
<     "wgCirrusSearchNamespaceWeights": [],
---
>     "wgCirrusSearchNamespaceWeights": {
>         "12": 0.9,
>         "100": 0.9,
>         "102": 0.9,
>         "104": 0.9,
>         "106": 0.9
>     },
377,383c317
<     "wgCirrusSearchReplicaGroup": {
<         "type": "roundrobin",
<         "groups": [
<             "psi",
<             "omega"
<         ]
<     },
---
>     "wgCirrusSearchReplicaGroup": "chi",
510a445,448
>         100,
>         102,
>         104,
>         106,
523d460
<         "upload.wikimedia.org",
527c464
<     "wgCopyUploadsFromSpecialUpload": true,
---
>     "wgCopyUploadsFromSpecialUpload": false,
548c485
<     "wgEnableAsyncUploads": true,
---
>     "wgEnableAsyncUploads": false,
550c487
<     "wgEnableDnsBlacklist": true,
---
>     "wgEnableDnsBlacklist": false,
580c517
<     "wgExportFromNamespaces": false,
---
>     "wgExportFromNamespaces": true,
586,587c523,530
<         "118": "Draft",
<         "119": "Draft_talk"
---
>         "100": "Manual",
>         "101": "Manual_talk",
>         "102": "Extension",
>         "103": "Extension_talk",
>         "104": "API",
>         "105": "API_talk",
>         "106": "Skin",
>         "107": "Skin_talk"
589a533
>         0,
601c545
<     "wgFavicon": "/static/favicon/black-globe.ico",
---
>     "wgFavicon": "/static/favicon/mediawiki.ico",
610c554,567
<     "wgForceUIMsgAsContentMsg": [],
---
>     "wgForceUIMsgAsContentMsg": [
>         "mw-mainpage-url",
>         "mw-cat-browser-url",
>         "mw-portal-url",
>         "mw-help-url",
>         "mw-manual-url",
>         "mw-faq-url",
>         "mw-download-url",
>         "mw-extensions-url",
>         "mw-bugzilla-url",
>         "mw-downloadfromsvn-url",
>         "mw-irc-url",
>         "mw-mailing lists-url"
>     ],
612c569
<         "local"
---
>         "shared"
619,647c576,578
<     "wgGEHelpPanelEnabled": true,
<     "wgGEHelpPanelHelpDeskTitle": "Wikipedia:Requests/Help_desk",
<     "wgGEHelpPanelLinks": [
<         {
<             "title": "EditingHelp",
<             "text": "Write a Wikipedia article using the right style",
<             "id": "style"
<         },
<         {
<             "title": "Wikipedia:Creation_and_usage_of_media_files",
<             "text": "Add an image",
<             "id": "image"
<         },
<         {
<             "title": "Wikipedia:Citing_sources",
<             "text": "Add a citation",
<             "id": "citation"
<         },
<         {
<             "title": "How_to_add_an_external_link",
<             "text": "How to add an external link",
<             "id": "externallink"
<         },
<         {
<             "title": "Help:Infobox",
<             "text": "How to add/edit the infobox",
<             "id": "infobox"
<         }
<     ],
---
>     "wgGEHelpPanelEnabled": false,
>     "wgGEHelpPanelHelpDeskTitle": "",
>     "wgGEHelpPanelLinks": [],
659,660c590,591
<     "wgGEHelpPanelViewMoreTitle": "Special:MyLanguage/Help:Contents",
<     "wgGEHomepageEnabled": true,
---
>     "wgGEHelpPanelViewMoreTitle": "",
>     "wgGEHomepageEnabled": false,
662c593
<     "wgGEHomepageMentorsList": "Wikipedia:Requests/Help desk/Mentors",
---
>     "wgGEHomepageMentorsList": "",
669c600
<     "wgGEHomepageTutorialTitle": "Help:Help",
---
>     "wgGEHomepageTutorialTitle": "",
671c602
<     "wgGENewcomerTasksGuidanceEnabled": true,
---
>     "wgGENewcomerTasksGuidanceEnabled": false,
695,700d625
<         "de",
<         "en",
<         "es",
<         "fr",
<         "ja",
<         "commons",
702,705c627,628
<         "incubator",
<         "strategy",
<         "wikinews:en",
<         "wikinews:es"
---
>         "w:en",
>         "usability"
707,709d629
<     "wgInterlanguageLinkCodeMap": {
<         "no": "nb"
<     },
716c636
<     "wgKartographerUseMarkerStyle": false,
---
>     "wgKartographerUseMarkerStyle": true,
718c638
<     "wgKartographerWikivoyageMode": false,
---
>     "wgKartographerWikivoyageMode": true,
796c716,717
<         "test"
---
>         "mw",
>         "mediawikiwiki"
803c724
<     "wgMFAppPackageId": "org.wikipedia",
---
>     "wgMFAppPackageId": false,
976c897
<     "wgMaxCredits": -1,
---
>     "wgMaxCredits": 0,
993c914
<     "wgMetaNamespace": "Wikipedia",
---
>     "wgMetaNamespace": "Project",
995c916
<     "wgMinervaAlwaysShowLanguageButton": true,
---
>     "wgMinervaAlwaysShowLanguageButton": false,
997c918
<     "wgMinervaCountErrors": true,
---
>     "wgMinervaCountErrors": false,
1008c929
<     "wgMobileUrlTemplate": "%h0.m.%h1.%h2",
---
>     "wgMobileUrlTemplate": "m.%h1.%h2",
1011,1016d931
<     "wgNamespaceAliases": {
<         "WP": 4,
<         "WT": 5,
<         "Wikipedia": 4,
<         "Wikipedia_talk": 5
<     },
1022,1024c937,944
<     "wgNamespacesToBeSearchedDefault": [
<         1
<     ],
---
>     "wgNamespacesToBeSearchedDefault": {
>         "12": 1,
>         "100": 1,
>         "102": 1,
>         "104": 1,
>         "106": 1,
>         "0": 1
>     },
1028a949,951
>         "0": 1,
>         "8": 0,
>         "14": 1,
1035c958
<         "8": true,
---
>         "15": true,
1041c964
<         "15": true,
---
>         "16": true,
1086c1009
<     "wgNoticeProject": "test",
---
>     "wgNoticeProject": "mediawiki",
1092,1093c1015
<             "enabled": true,
<             "excludeBots": true
---
>             "enabled": true
1096,1097c1018
<             "enabled": true,
<             "excludeBots": true
---
>             "enabled": true
1103c1024
<             "enabled": true,
---
>             "enabled": false,
1105,1106c1026
<                 0,
<                 118
---
>                 0
1108,1109c1028
<             "cleanParent": true,
<             "keepForever": true
---
>             "cleanParent": true
1112c1031
<             "enabled": true,
---
>             "enabled": false,
1114,1115c1033
<                 0,
<                 118
---
>                 0
1119,1122c1037
<             ],
<             "excludeBots": true,
<             "cleanParent": true,
<             "keepForever": true
---
>             ]
1125c1040
<     "wgOresUiEnabled": true,
---
>     "wgOresUiEnabled": false,
1127,1129d1041
<     "wgOverrideSiteFeed": {
<         "rss": "http://www.example.org/feedstuff.rss"
<     },
1131c1043
<     "wgPageAssessmentsSubprojects": true,
---
>     "wgPageAssessmentsSubprojects": false,
1135c1047
<     "wgPageImagesLeadSectionOnly": true,
---
>     "wgPageImagesLeadSectionOnly": false,
1137,1140c1049,1052
<     "wgPageTriageDraftNamespaceId": 118,
<     "wgPageTriageEnableCopyvio": true,
<     "wgPageTriageEnableCurationToolbar": true,
<     "wgPageTriageEnableOresFilters": true,
---
>     "wgPageTriageDraftNamespaceId": false,
>     "wgPageTriageEnableCopyvio": false,
>     "wgPageTriageEnableCurationToolbar": false,
>     "wgPageTriageEnableOresFilters": false,
1148c1060
<     "wgPopupsGateway": "restbaseHTML",
---
>     "wgPopupsGateway": "mwApiPlain",
1173c1085
<     "wgPopupsVirtualPageViews": true,
---
>     "wgPopupsVirtualPageViews": false,
1330a1243,1247
>         "sysop": [
>             "autopatrolled",
>             "uploader",
>             "ipblock-exempt"
>         ],
1332,1335c1249,1251
<             "sysop",
<             "researcher",
<             "centralnoticeadmin",
<             "flow-bot",
---
>             "autopatrolled",
>             "transwiki",
>             "import",
1340,1347d1255
<         ],
<         "sysop": [
<             "filemover",
<             "rollbacker",
<             "accountcreator",
<             "confirmed",
<             "templateeditor",
<             "ipblock-exempt"
1350,1352c1258
<     "wgRestAPIAdditionalRouteFiles": [
<         "includes/Rest/coreDevelopmentRoutes.json"
<     ],
---
>     "wgRestAPIAdditionalRouteFiles": [],
1357d1262
<         "templateeditor",
1360c1265
<     "wgRevisionCacheExpiry": 3600,
---
>     "wgRevisionCacheExpiry": 604800,
1376c1281
<     "wgServer": "//test.wikipedia.org",
---
>     "wgServer": "//www.mediawiki.org",
1379c1284
<     "wgShowExceptionDetails": true,
---
>     "wgShowExceptionDetails": false,
1383c1288
<     "wgSitename": "Wikipedia",
---
>     "wgSitename": "MediaWiki",
1410c1315
<     "wgTorTagChanges": true,
---
>     "wgTorTagChanges": false,
1415,1416c1320,1321
<     "wgUploadNavigationUrl": false,
<     "wgUploadPath": "//upload.wikimedia.org/wikipedia/test",
---
>     "wgUploadNavigationUrl": "//commons.wikimedia.org/wiki/Special:UploadWizard",
>     "wgUploadPath": "//upload.wikimedia.org/wikipedia/mediawiki",
1427c1332
<     "wgUseContributionTracking": true,
---
>     "wgUseContributionTracking": false,
1439c1344
<     "wgVectorResponsive": true,
---
>     "wgVectorResponsive": false,
1518,1519c1423,1424
<     "wgWMEInukaPageViewCookiesDomain": "wikipedia.org",
<     "wgWMEInukaPageViewEnabled": true,
---
>     "wgWMEInukaPageViewCookiesDomain": "",
>     "wgWMEInukaPageViewEnabled": false,
1524c1429
<     "wgWMEUnderstandingFirstDay": true,
---
>     "wgWMEUnderstandingFirstDay": false,
1529,1531c1434
<         7,
<         118,
<         119
---
>         7
1540c1443
<     "wgWelcomeSurveyEnabled": true,
---
>     "wgWelcomeSurveyEnabled": false,
1572,1573d1474
<         "wikipedia",
<         "all",
1575,1576d1475
<         "wikidataclient-test",
<         "testwikis",
1579d1477
<         "pageassessments",
1582d1479
<         "growthexperiments",
1585c1482,1483
<         "flow-labs",
---
>         "commonsuploads",
>         "cirrussearch-big-indices",
1587c1485,1486
<         "all-labs"
---
>         "special",
>         "all"
1590c1489
<     "wmgAllowLabsAnonEdits": true,
---
>     "wmgAllowLabsAnonEdits": false,
1641c1540
<     "wmgCentralAuthLoginIcon": "/srv/mediawiki/static/images/sul/wikipedia.png",
---
>     "wmgCentralAuthLoginIcon": "/srv/mediawiki/static/images/sul/mediawiki.png",
1651c1550
<     "wmgCiteVisualEditorOtherGroup": false,
---
>     "wmgCiteVisualEditorOtherGroup": true,
1694c1593
<     "wmgContentTranslationTranslateInTarget": false,
---
>     "wmgContentTranslationTranslateInTarget": true,
1696,1697c1595,1596
<     "wmgCustomUploadDialog": true,
<     "wmgDefaultMonologHandler": "wgDebugLogFile",
---
>     "wmgCustomUploadDialog": false,
>     "wmgDefaultMonologHandler": "blackhole",
1699c1598
<     "wmgDisplayFeedsInSidebar": true,
---
>     "wmgDisplayFeedsInSidebar": false,
1701c1600
<     "wmgEchoCluster": "extension1",
---
>     "wmgEchoCluster": false,
1706c1605
<     "wmgEchoSiteNotificationIconUrl": "/static/images/project-logos/notifications/120px-Notification-icon-Wikipedia-logo.svg.png",
---
>     "wmgEchoSiteNotificationIconUrl": "/static/images/project-logos/notifications/120px-Notification-icon-MediaWiki-logo.svg.png",
1713c1612
<     "wmgEnableGeoData": true,
---
>     "wmgEnableGeoData": false,
1716,1718c1615,1617
<     "wmgEnableLandingCheck": true,
<     "wmgEnableLocalTimedText": true,
<     "wmgEnablePageTriage": true,
---
>     "wmgEnableLandingCheck": false,
>     "wmgEnableLocalTimedText": false,
>     "wmgEnablePageTriage": false,
1748c1647
<     "wmgFlowEnableOptInBetaFeature": true,
---
>     "wmgFlowEnableOptInBetaFeature": false,
1751c1650,1669
<     "wmgFlowNamespaces": [],
---
>     "wmgFlowNamespaces": [
>         1,
>         3,
>         5,
>         7,
>         9,
>         11,
>         13,
>         15,
>         101,
>         103,
>         105,
>         107,
>         487,
>         829,
>         1199,
>         2301,
>         2303,
>         5501
>     ],
2010,2011c1928,1941
<     "wmgRC2UDPPrefix": false,
<     "wmgRSSUrlWhitelist": [],
---
>     "wmgRC2UDPPrefix": "#mediawiki.wikipedia\t",
>     "wmgRSSUrlWhitelist": [
>         "https://wikimediafoundation.org/news/feed/",
>         "https://wikimediafoundation.org/category/technology/feed/",
>         "https://wikimediafoundation.org/category/technology/mediawiki/feed/",
>         "https://discourse-mediawiki.wmflabs.org/c/ask-here.rss",
>         "https://codeclimate.com/github/wikimedia/mediawiki-extensions-CentralNotice/feed.atom",
>         "https://codeclimate.com/github/wikimedia/mediawiki-extensions-DonationInterface/feed.atom",
>         "https://codeclimate.com/github/wikimedia/wikimedia-fundraising-crm/feed.atom",
>         "https://codeclimate.com/github/wikimedia/wikimedia-fundraising-dash/feed.atom",
>         "https://codeclimate.com/github/wikimedia/wikimedia-fundraising-php-queue/feed.atom",
>         "https://codeclimate.com/github/wikimedia/wikimedia-fundraising-SmashPig/feed.atom",
>         "https://codeclimate.com/github/wikimedia/wikimedia-fundraising-tools/feed.atom"
>     ],
2017c1947
<     "wmgSecurePollUseNamespace": true,
---
>     "wmgSecurePollUseNamespace": false,
2021,2029c1951,1952
<     "wmgSiteLogo1_5x": "/static/images/project-logos/testwiki-1.5x.png",
<     "wmgSiteLogo1x": "/static/images/project-logos/testwiki.png",
<     "wmgSiteLogo2x": "/static/images/project-logos/testwiki-2x.png",
<     "wmgSiteLogoIcon": "/static/images/mobile/W.png",
<     "wmgSiteLogoWordmark": {
<         "src": "/static/images/mobile/copyright/wikipedia-wordmark-en.svg",
<         "width": 116,
<         "height": 18
<     },
---
>     "wmgSiteLogo1x": "/static/images/project-logos/mediawikiwiki.png",
>     "wmgSiteLogoIcon": "/static/images/mobile/mediawiki.png",
2033c1956
<     "wmgTranslateESIndex": "ttmserver-test",
---
>     "wmgTranslateESIndex": "ttmserver",
2041c1964
<     "wmgULSPosition": "interlanguage",
---
>     "wmgULSPosition": "personal",
2043c1966
<     "wmgUpload3d": true,
---
>     "wmgUpload3d": false,
2047,2048c1970,1971
<     "wmgUseArticleCreationWorkflow": true,
<     "wmgUseArticlePlaceholder": true,
---
>     "wmgUseArticleCreationWorkflow": false,
>     "wmgUseArticlePlaceholder": false,
2059c1982
<     "wmgUseCapiunto": true,
---
>     "wmgUseCapiunto": false,
2069c1992
<     "wmgUseCodeReview": false,
---
>     "wmgUseCodeReview": true,
2071c1994
<     "wmgUseCollaborationKit": true,
---
>     "wmgUseCollaborationKit": false,
2075,2076c1998,1999
<     "wmgUseContactPage": true,
<     "wmgUseContentTranslation": true,
---
>     "wmgUseContactPage": false,
>     "wmgUseContentTranslation": false,
2080,2081c2003,2004
<     "wmgUseDoubleWiki": true,
<     "wmgUseDynamicPageList": false,
---
>     "wmgUseDoubleWiki": false,
>     "wmgUseDynamicPageList": true,
2090c2013
<     "wmgUseExternalGuidance": true,
---
>     "wmgUseExternalGuidance": false,
2093c2016
<     "wmgUseFileImporter": "FileImporter-WikimediaSitesTableSite",
---
>     "wmgUseFileImporter": false,
2097c2020
<     "wmgUseFooterCodeOfConductLink": false,
---
>     "wmgUseFooterCodeOfConductLink": true,
2104c2027
<     "wmgUseGettingStarted": true,
---
>     "wmgUseGettingStarted": false,
2112c2035
<     "wmgUseGoogleNewsSitemap": true,
---
>     "wmgUseGoogleNewsSitemap": false,
2114,2115c2037,2038
<     "wmgUseGraphWithJsonNamespace": false,
<     "wmgUseGrowthExperiments": true,
---
>     "wmgUseGraphWithJsonNamespace": true,
>     "wmgUseGrowthExperiments": false,
2129c2052
<     "wmgUseLivePreview": true,
---
>     "wmgUseLivePreview": false,
2142,2143c2065,2066
<     "wmgUseORES": true,
<     "wmgUsePageAssessments": true,
---
>     "wmgUseORES": false,
>     "wmgUsePageAssessments": false,
2153c2076
<     "wmgUsePopups": true,
---
>     "wmgUsePopups": false,
2158c2081
<     "wmgUseRSSExtension": false,
---
>     "wmgUseRSSExtension": true,
2160c2083
<     "wmgUseRelatedArticles": true,
---
>     "wmgUseRelatedArticles": false,
2163c2086
<     "wmgUseSandboxLink": true,
---
>     "wmgUseSandboxLink": false,
2173c2096
<     "wmgUseSubPageList3": true,
---
>     "wmgUseSubPageList3": false,
2188c2111
<     "wmgUseUploadWizard": true,
---
>     "wmgUseUploadWizard": false,
2207c2130
<     "wmgUseWikidataPageBanner": true,
---
>     "wmgUseWikidataPageBanner": false,
2210c2133
<     "wmgUseWikimediaShopLink": true,
---
>     "wmgUseWikimediaShopLink": false,
2214a2138,2142
>         "Project": true,
>         "Manual": true,
>         "Extension": true,
>         "API": true,
>         "Skin": true,
2220c2148
<     "wmgVisualEditorConsolidateFeedback": true,
---
>     "wmgVisualEditorConsolidateFeedback": false,
2249,2250c2177,2178
<     "wmgWikibaseAllowLocalShortDesc": true,
<     "wmgWikibaseCachePrefix": "testwikidatawiki0",
---
>     "wmgWikibaseAllowLocalShortDesc": false,
>     "wmgWikibaseCachePrefix": "wikidatawiki",
2265c2193
<     "wmgWikibaseClientChangesDatabase": "testwikidatawiki",
---
>     "wmgWikibaseClientChangesDatabase": "wikidatawiki",
2280,2281c2208,2209
<     "wmgWikibaseClientRepoConceptBaseUri": "http://test.wikidata.org/entity/",
<     "wmgWikibaseClientRepoDatabase": "testwikidatawiki",
---
>     "wmgWikibaseClientRepoConceptBaseUri": "http://www.wikidata.org/entity/",
>     "wmgWikibaseClientRepoDatabase": "wikidatawiki",
2286c2214
<     "wmgWikibaseClientRepoUrl": "https://test.wikidata.org",
---
>     "wmgWikibaseClientRepoUrl": "https://www.wikidata.org",
2289,2290c2217,2218
<             "repoDatabase": "testwikidatawiki",
<             "baseUri": "http://test.wikidata.org/entity/",
---
>             "repoDatabase": "wikidatawiki",
>             "baseUri": "http://www.wikidata.org/entity/",
2306,2308c2234,2235
<         "test",
<         "test2",
<         "testwikidata"
---
>         "wikidata",
>         "wikimania"
2338,2351c2265
<     "wmgWikibaseEntitySources": {
<         "testwikidata": {
<             "entityNamespaces": {
<                 "item": 0,
<                 "property": 120,
<                 "lexeme": 146
<             },
<             "repoDatabase": "testwikidatawiki",
<             "baseUri": "http://test.wikidata.org/entity/",
<             "rdfNodeNamespacePrefix": "wd",
<             "rdfPredicateNamespacePrefix": "",
<             "interwikiPrefix": "testwikidata"
<         }
<     },
---
>     "wmgWikibaseEntitySources": [],
2375d2288
<     "wmgWikibaseLocalEntitySourceName": "testwikidata",
2387c2300
<     "wmgWikibaseSiteGroup": "wikipedia",
---
>     "wmgWikibaseSiteGroup": "mediawiki",

Nothing jumps out at me; the Flow config changes are around user accounts and defaults, nothing about services integration. No changes on the Parsoid/VRS/RESTbase/services integration front AFAICS.

In T249705#6040487, @cscott wrote:

Adding as a train blocker until/unless we can prove that this isn't related to a group0 deployment.

(It's likely as @kostajh suggests that this isn't train-related, but I'm trying to err on the side of caution.)

Please note that this is a pywikibot breakage not Flow breakage .. see https://test.wikipedia.org/wiki/Topic:Vk4my4ths5dq3qej .. I just did that flow test. So, this is not a train blocker.

Confirmed.

Jdforrester-WMF renamed this task from Internal API error with Flow on testwiki to pywikibot encounters an internal API error with Flow on testwiki (but not other wikis).Apr 8 2020, 4:42 PM

Jdforrester-WMF removed a parent task: T247774: 1.35.0-wmf.27 deployment blockers.

https://logstash.wikimedia.org/goto/2697afe9d80ae6c138127f0a8febd9df is the logstash board showing the 503 errors.

https://logstash.wikimedia.org/goto/cc6d91314e1475059ea06a3a1d422aa4 is the logstash board for errors over a 7-day period. it shows:

this is NOT limited to testwiki
also shows a bunch of errors on April 6th around 1500 UTC (before the current train rollout to group 0)
the current spurt of errors seem to have started around 7 hours back
unclear if this is necessarily limited to pywikibot since logstash isn't showing us the user-agent string.

HTTP 503 is "Service Unavailable" -- I wonder if this is actually a rate limit or something similar which is being hit?

The full URL requested for the original 503 error should have been:

http://localhost:6002/w/rest.php/test.wikimedia.org/v3/transform/html/to/wikitext/Topic%3AU5y53rn0dp6h70nw

(Based on an internal VRS request of /restbase/local/v1/transform/html/to/wikitext/Topic%3AU5y53rn0dp6h70nw)

That localhost request is then routed to the parsoid cluster; @Joe thinks that the recent envoy changes (see T249535) might be related.

<cscott> i assume the localhost:6002 goes to envoy or some other router which actually brings it to the parsoid cluster
<cscott> ?
<_joe_> yes
<_joe_> so interestingly, I find no trace of 503 errors in the parsoid logs, but lemme check the mediawiki appservers
<_joe_> actually, it's almost 8 pm
<_joe_> I'll do it  tomorrow
<_joe_> this is not a complete breakage AFAICT

@Joe, if this turns out to be envoy-related, T249745: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" could be as well.

Joe triaged this task as Low priority.Apr 9 2020, 10:35 AM

In T249705#6040712, @ssastry wrote:

also shows a bunch of errors on April 6th around 1500 UTC (before the current train rollout to group 0)

Also I see it continues to occur on 1.35.0-wmf.26, so again likely not the wmf.27 train.

unclear if this is necessarily limited to pywikibot since logstash isn't showing us the user-agent string.

I just ran into it trying to submit a comment on officewiki with Firefox. Out of annoyance I clicked "save" several more times, and it did finally go through after erroring the first half-dozen times.

In T249705#6042810, @Anomie wrote:

In T249705#6040712, @ssastry wrote:

also shows a bunch of errors on April 6th around 1500 UTC (before the current train rollout to group 0)

Also I see it continues to occur on 1.35.0-wmf.26, so again likely not the wmf.27 train.

unclear if this is necessarily limited to pywikibot since logstash isn't showing us the user-agent string.

I just ran into it trying to submit a comment on officewiki with Firefox. Out of annoyance I clicked "save" several more times, and it did finally go through after erroring the first half-dozen times.

This seems a very different thing, though. The errors reported by pywikibot seemed sporadic, and I could trace them to a specific failure on a backend. What you are describing seems like a persistent behaviour that would possibly not be related to what I was investigating at the moment.

I just found your errors in the logs, and this thickens the cloud of mistery around this bug. I am 99% convinced this has to do with my work, rather than anything else, so I'll re-tag the task appropriately.

Joe added a project: serviceops.Apr 9 2020, 1:17 PM

Change 587732 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] parsoid: allow retries for connection resets in envoy

https://gerrit.wikimedia.org/r/587732

gerritbot added a project: Patch-For-Review.Apr 9 2020, 1:23 PM

I just found your errors in the logs, and this thickens the cloud of mistery around this bug. I am 99% convinced this has to do with my work, rather than anything else, so I'll re-tag the task appropriately.

FWIW, just a little while ago I got the same error (NoParserException) the first time I tried posting a comment on that same Flow topic on officewiki, the second time it succeeded.

Change 587787 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] services_proxy: retry on connect failures for parsoid

https://gerrit.wikimedia.org/r/587787

Dvorapa mentioned this in T249828: test_topic of flow_tests.TestFlowTopic due to additional edit.Apr 9 2020, 3:37 PM

Just for the record, here is the error after @ssastry (? I guess) edited that particular topic:

======================================================================
ERROR: test_topic_moderation (tests.flow_tests.TestFlowTopic)
Test Topic functions about moderation.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\pywikibot-g4xqx\tests\flow_tests.py", line 245, in test_topic_moderation
    self.assertFalse(topic_hidden.is_locked)
  File "C:\projects\pywikibot-g4xqx\pywikibot\flow.py", line 220, in is_locked
    return self.root._current_revision['isLocked']
  File "C:\projects\pywikibot-g4xqx\pywikibot\flow.py", line 214, in root
    self._root = Post.fromJSON(self, self.uuid, self._data)
  File "C:\projects\pywikibot-g4xqx\pywikibot\flow.py", line 65, in uuid
    self._load_uuid()
  File "C:\projects\pywikibot-g4xqx\pywikibot\flow.py", line 55, in _load_uuid
    self._uuid = self._load()['workflowId']
  File "C:\projects\pywikibot-g4xqx\pywikibot\flow.py", line 160, in _load
    self._data = self.site.load_topic(self, format)
  File "C:\projects\pywikibot-g4xqx\pywikibot\site\__init__.py", line 1390, in callee
    return fn(self, *args, **kwargs)
  File "C:\projects\pywikibot-g4xqx\pywikibot\site\__init__.py", line 7306, in load_topic
    data = req.submit()
  File "C:\projects\pywikibot-g4xqx\pywikibot\data\api.py", line 2040, in submit
    if self._internal_api_error(code, error, result):
  File "C:\projects\pywikibot-g4xqx\pywikibot\data\api.py", line 1910, in _internal_api_error
    raise e
pywikibot.data.api.APIMWException: internal_api_error_Flow\Exception\NoParserException: [3d207406-aefb-4869-8e88-041eaab0ddd1] Exception caught: Request to parsoid for "html" to "wikitext" conversion of content connected to title "Topic:U5y53rn0dp6h70nw" failed: 503
[*: Flow\Exception\NoParserException at /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Conversion/Utils.php(174)
#0 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Conversion/Utils.php(66): Flow\Conversion\Utils::parsoid(string, string, string, Title)
#1 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Model/AbstractRevision.php(430): Flow\Conversion\Utils::convert(string, string, string, Title)
#2 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Templating.php(159): Flow\Model\AbstractRevision->getContent(string)
#3 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Formatter/RevisionFormatter.php(289): Flow\Templating->getContent(Flow\Model\PostRevision, string)
#4 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Formatter/TopicFormatter.php(45): Flow\Formatter\RevisionFormatter->formatApi(Flow\Formatter\TopicRow, DerivativeContext)
#5 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Block/Topic.php(663): Flow\Formatter\TopicFormatter->formatApi(Flow\Model\Workflow, array, DerivativeContext)
#6 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Block/Topic.php(565): Flow\Block\TopicBlock->renderTopicApi(array)
#7 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Api/ApiFlowBaseGet.php(33): Flow\Block\TopicBlock->renderApi(array)
#8 /srv/mediawiki/php-1.35.0-wmf.27/extensions/Flow/includes/Api/ApiFlow.php(105): Flow\Api\ApiFlowBaseGet->execute()
#9 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(1580): Flow\Api\ApiFlow->execute()
#10 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(523): ApiMain->executeAction()
#11 /srv/mediawiki/php-1.35.0-wmf.27/includes/api/ApiMain.php(494): ApiMain->executeActionWithErrorHandling()
#12 /srv/mediawiki/php-1.35.0-wmf.27/api.php(84): ApiMain->execute()
#13 /srv/mediawiki/w/api.php(3): require(string)
#14 {main};
 errorclass: Flow\Exception\NoParserException;
 servedby: mw1408]

Not sure if it changed, but it fails more often now.

ssastry moved this task from Needs Triage to Non-Parsing-Team Tasks on the Parsoid board.Apr 9 2020, 6:20 PM

Occasionally also fails on mediawiki.org (https://travis-ci.org/github/wikimedia/pywikibot/jobs/673204826#L11391)

In T249705#6045160, @Dvorapa wrote:

Occasionally also fails on mediawiki.org (https://travis-ci.org/github/wikimedia/pywikibot/jobs/673204826#L11391)

Yes, T249705#6040712 summarizes this.

Change 587787 merged by Giuseppe Lavagetto:
[operations/puppet@production] services_proxy: retry on connect failures for parsoid

https://gerrit.wikimedia.org/r/587787

@Xqt Could we skip the whole flow_tests.py somehow in Pywikibot tests? Or is there an option to skip tests from flow_tests.py failing with this internal API error?

The whole script can be skipped by adding code like the following in flow_tests.py:

def setUpModule():  # noqa: N802
    """Always skip."""
    raise unittest.SkipTest('flow_tests.py tests are disabled due to T249705.')

Or skip the whole flow tests if an environ variable is set like in l10n_tests.py

cscott renamed this task from pywikibot encounters an internal API error with Flow on testwiki (but not other wikis) to Intermittent internal API errors with Flow.Apr 10 2020, 4:05 PM

cscott mentioned this in T249909: Flow/Parsoid permanently delete the entered text after errors.

cscott updated the task description. (Show Details)

In T249705#6045978, @Xqt wrote:
The whole script can be skipped by adding code like the following in flow_tests.py:
def setUpModule():  # noqa: N802
    """Always skip."""
    raise unittest.SkipTest('flow_tests.py tests are disabled due to T249705.')
Or skip the whole flow tests if an environ variable is set like in l10n_tests.py

This would hopefully be temporary, I would prefer the first option then.

Change 588144 had a related patch set uploaded (by Dvorapa; owner: Dvorapa):
[pywikibot/core@master] [test] Ignore flow_tests until upstream issue solved

https://gerrit.wikimedia.org/r/588144

Change 588144 merged by jenkins-bot:
[pywikibot/core@master] [test] Ignore flow_tests until upstream issue solved

https://gerrit.wikimedia.org/r/588144

Dvorapa mentioned this in rPWBCbccf4f1c8d0c: [test] Ignore flow_tests until upstream issue solved.Apr 12 2020, 10:25 AM

kostajh mentioned this in T249997: Flow failing with Error contacting the Parsoid/RESTBase server (HTTP 400).Apr 13 2020, 12:41 PM

@Joe I see your patch was merged on Friday but it seems like there is still an instance of this problem occurring as of today (T249997), do you have any thoughts on what might be going on?

ssastry merged a task: T250091: HTTP 503 error: Request to parsoid for "html" to "wikitext" conversion of content connected to title "Topic:Vkfmw9rpr9kbucul" failed.Apr 13 2020, 3:54 PM

Aklapper merged a task: T250091: HTTP 503 error: Request to parsoid for "html" to "wikitext" conversion of content connected to title "Topic:Vkfmw9rpr9kbucul" failed.Apr 13 2020, 4:35 PM

In T249705#6051589, @kostajh wrote:

@Joe I see your patch was merged on Friday but it seems like there is still an instance of this problem occurring as of today (T249997), do you have any thoughts on what might be going on?

That bug talks about an error 400, it seems a different problem. I'm going to try to do some more aggressive changes today to contrast what's happening. I'm surprised by the number of reports for something that seems to be happening once every minute or so.

If this doesn't work either, I'll probably revert my latest changes.

Changing priority as this seems to be highly user visible.

RhinosF1 subscribed.Apr 14 2020, 7:38 AM

Change 588692 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] services_proxy: temp mitigation for intermittent parsoid requests failures

https://gerrit.wikimedia.org/r/588692

Change 588692 merged by Giuseppe Lavagetto:
[operations/puppet@production] services_proxy: temp mitigation for intermittent parsoid requests failures

https://gerrit.wikimedia.org/r/588692