Page MenuHomePhabricator

iPadOS 14's Scribble text input issues
Open, Needs TriagePublic

Description

iPadOS 14 introduces a new input method where you can use a stylus to do handwriting recognition in any text input. This works in VisualEditor, but with issues.

The main issue is that the first input stroke seems to be eaten by VE and not used for the handwriting recognition. E.g. if I write "To do", the first stroke of the T isn't used for input. This seems to happen more if the caret is at the start of a paragraph.

A secondary issue is that the undo stack behaves strangely. If I handwrite and then use the system keyboard controls to add newlines and then undo, it seems to squash the newlines and handwriting-recognition together. I'd theorize this is the IME-undo not being canceled and so we're undoing in VE at the same time it's undoing.

iPadOS 14 isn't released yet, so this may or may not need to be acted upon.

Event Timeline

DLynch created this task.Jul 9 2020, 4:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 9 2020, 4:42 PM
DLynch added a subscriber: dchan.Jul 9 2020, 4:42 PM

IME event logger output for some simple input:

Krinkle added a subscriber: Krinkle.
DLynch added a comment.Jul 9 2020, 4:51 PM

Here's an example of the stroke vanishing. I wrote "Now", and got "ow".

I've tried it on a plain contenteditable area, and not seen the issue, so it's presumably something to do with VE's extra processing.

JTannerWMF moved this task from To Triage to Triaged on the VisualEditor board.Jul 14 2020, 3:29 PM
JTannerWMF added a subscriber: JTannerWMF.

We will look at this once a production build is released for iOS 14.

We will look at this once a production build is released for iOS 14.

Sorry, clarification because I can't remember exactly what we said in meetings -- did we mean when there's a public beta, or when iOS14 is released?

I'm getting test hardware so will take a look when I get a spare moment.

GM build of iOS 14 was released today, with the final release scheduled for tomorrow (16th).

Trying this again, having tried the Scribble introduction in the iOS settings, I think that it's maybe behaving as-intended given the content. I think my first stroke is being interpreted by Scribble as text-navigation (or space-insertion)... and then once its done the cursor is left in a valid place to insert text and so it works for the remaining strokes.

The real MVP of having it work reliably in densely packed text like an article page turned out to be learning the "hold pencil in place for a second to expand some space to write in" trick.

I suspect we can close this task. People who try using Scribble in VE may have to adapt to how Scribble works, but it's not inherently broken.