Page MenuHomePhabricator
Create Task
Maniphest T211871

TFA missing from MCS response
Closed, ResolvedPublic
Actions

Description

In the last 7 days, there have been over 6000 instances where the MCS result for the feed did not contain the TFA portion in the body. A sample log entry:

TypeError: Cannot set property 'thumbnail' of undefined
    at P.join.then (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/restbase/v1/feed.js:95:50)
    at tryCatcher (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:638:18)
    at PromiseArray._resolve (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise_array.js:126:19)
    at PromiseArray._promiseFulfilled (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise_array.js:144:14)
    at Promise._settlePromise (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:638:18)
    at Promise._resolveCallback (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:432:57)
    at Promise._settlePromiseFromHandler (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:524:17)
    at Promise._settlePromise (/srv/deployment/restbase/deploy-cache/revs/44e09555bc88aca274384cddaa267a25deabfc97/node_modules/bluebird/js/release/promise.js:569:18)

Open questions:

  • Does the Android app expect TFA to be present in the response?
  • Is not having the TFA in the response legitimate?

Related Objects

Event Timeline

• mobrovac triaged this task as High priority.Dec 13 2018, 11:19 AM
• mobrovac created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 13 2018, 11:19 AM
• mobrovac added a comment.Dec 13 2018, 11:21 AM

Since this happens quite a lot (~1300 failed requests in the last 24h), I am assuming the answer to the first question in the task description is yes and will move forward on PR #1088 which ensures result[0].body.tfa exists.

Stashbot mentioned this in T211070: decommission of restbase200[1-6] (lease return in December 2018).Dec 13 2018, 11:31 AM
Stashbot subscribed.

Mentioned in SAL (#wikimedia-operations) [2018-12-13T11:31:52Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871

Stashbot added a comment.Dec 13 2018, 11:39 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-13T11:39:00Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871 (duration: 07m 08s)

Stashbot added a comment.Dec 13 2018, 11:39 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-13T11:39:18Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871

Stashbot added a comment.Dec 13 2018, 11:45 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-13T11:45:26Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871 (duration: 06m 08s)

Stashbot added a comment.Dec 13 2018, 11:53 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-13T11:53:49Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@55fcd4b]: Remove restbase200[1-6], ensure body.tfa exists for feed responses and disable Citoid check - T211070 T211871 T211411

Stashbot mentioned this in T211411: Citoid automated monitoring times out due to Zotero v2.Dec 13 2018, 11:53 AM
• mobrovac added a project: Mobile-Content-Service.Dec 13 2018, 12:04 PM
Restricted Application added a project: Product-Infrastructure-Team-Backlog-Deprecated. · View Herald TranscriptDec 13 2018, 12:04 PM
Stashbot added a comment.Dec 13 2018, 12:12 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-13T12:12:47Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@55fcd4b]: Remove restbase200[1-6], ensure body.tfa exists for feed responses and disable Citoid check - T211070 T211871 T211411 (duration: 18m 59s)

• Mholloway added a subscriber: Dbrant.Dec 13 2018, 3:43 PM

Open questions:

  • Does the Android app expect TFA to be present in the response?

Yes, for supported languages, but I believe the app gracefully handles both missing feed response properties and malformed properties (ping @Dbrant to verify).

  • Is not having the TFA in the response legitimate?

See above.

I'm confused about what's really going on here, though. Won't MCS necessarily have responded with a well-formed tfa property to reach this point due to the check on https://github.com/wikimedia/restbase/blob/master/v1/feed.js#L72? Could something be going wrong in the intervening _hydrateResponse?

• Pchelolo added a comment.Dec 13 2018, 3:54 PM

I'm confused about what's really going on here, though. Won't MCS necessarily have responded with a well-formed tfa property to reach this point due to the check on https://github.com/wikimedia/restbase/blob/master/v1/feed.js#L72? Could something be going wrong in the intervening _hydrateResponse?

Indeed. _hydrateResponse would delete the property if it could not fetch the summary for the article for some reason and it fetches it from RESTBase. We have been experiencing some latency penalties and elevated error rates due to cluster reshaping, maybe that's the cause.

• Pchelolo closed this task as Resolved.Dec 17 2018, 10:59 PM
• Pchelolo claimed this task.
• Pchelolo edited projects, added Services (done), Platform Team Workboards (Done with CPT); removed Platform Team Workboards (Doing), Services (doing).

Fixed and deployed.

• mobrovac claimed this task.Dec 18 2018, 10:06 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL