Page MenuHomePhabricator

Google Keyboard input events can leave VisualEditor DM out of sync
Closed, ResolvedPublicBUG REPORT

Description

List of steps to reproduce (step by step, including full links if applicable):

  • Open the VisualEditor using an Android device with the Google Keyboard. The issue is specific to Google Keyboard on Chrome; other custom Android keyboards such as Samsung do not exhibit the behavior, nor does Mobile Firefox.
  • Input some text using the keyboard. (If using an emulator, use the Google Keyboard in the emulator rather than your physical keyboard)
  • Press Enter using the keyboard.

What happens?:
The cursor position remains unchanged, but a newline is inserted below the cursor position. The VisualEditor DM does not become aware of the newly inserted newline / paragraph.

What should have happened instead?:
The cursor should move to the newly inserted line and the DM should be updated accordingly.

Software version (if not a Wikimedia wiki), browser information, screenshots, other information, etc.:
Reproduced on an emulated Pixel 5 using Google Chrome.

Event Timeline

Interestingly, it seems that the input event is emitted twice for each keystroke. Note also how the event text contains a trailing newline that then gets inserted into the CE surface without an appropriate DM update:

Screenshot 2022-07-07 at 16.59.05.png (575×623 px, 155 KB)

We're currently experimenting with a rough patch that (a) debounces the apparent double-send and (b) ensures that any trailing newline insertions get reflected in the DM and selection state, something like this:

FandomVisualEditorSurface.prototype.onDocumentInput = function (event) {
	var surface = this;
	// The android keyboard seems to be sending the input event twice on enter, We're debouncing it here
	var elapsed = event.timeStamp - ( surface.lastEventTimestamp ? surface.lastEventTimestamp : 0 );
	surface.lastEventTimestamp = event.timeStamp;
	if ( event.originalEvent.data && event.originalEvent.data.endsWith( '\n' ) && elapsed > 30 ) {
		// The input events sent by adroid keyboard do update content editable layer, but they fail to update underlying data model, we're doing that here
		setTimeout( function () {
			const fragment = surface.getModel().getFragment();
			fragment
				.insertContent( [{ type: 'paragraph' }, { type: '/paragraph' }] )
				.collapseToEnd()
				.select();
		} );
	}
	FandomVisualEditorSurface.super.prototype.onDocumentInput.call( this, event );
};
Esanders triaged this task as High priority.Jul 7 2022, 3:51 PM
Esanders moved this task from To Triage to Triaged on the VisualEditor board.

This looks like the same issue as T217223 and T292538, both of which mention Gboard. You seem to know more about the problem than I do, so if you can confirm this, please merge them together.

It also may or may not be the same as T303109 and T302083, but these do not mention enough details to say for sure.

@matmarex Thanks, I missed those in my search. T217223 seems probably the most similar, as backspacing was explicitly mentioned in one of the user reports that got clobbered into our original ticket that prompted this one. Not immediately sure about the other ones, though.

n.b. String.prototype.endsWith is ES6, so will throw in some older browsers that we still support (IE11)

A better way to simulate the codepath of an enter event would be to call

ve.ce.keyDownHandlerFactory.lookup( 'linearEnter' ).static.execute( this, e )

Change 812093 had a related patch set uploaded (by TK-999; author: TK-999):

[VisualEditor/VisualEditor@master] [WIP] Resolve some out-of-sync issues on Android with Google Keyboard

https://gerrit.wikimedia.org/r/812093

A better way to simulate the codepath of an enter event would be to call

ve.ce.keyDownHandlerFactory.lookup( 'linearEnter' ).static.execute( this, e )

My investigation suggests that one of the problems working with GBoard is the fact that keyup/keydown events it emits don't contain a reference to what key was pressed. From my understanding it's done by design (https://bugs.chromium.org/p/chromium/issues/detail?id=118639) so we can't it to change any time soon.

Ok, change 812093 triggers the VE Enter handler — which is a big improvement! As it stands, it still leaves a spurious <div><br></div>, into which we can drag the cursor and type text which then doesn't get put into the model (see screenshot). We should check for this spurious div and clean it up — and as a bonus, we can condition triggering the VE Enter handler on finding this div, so that if another IME behaves as we want it to already, then it won't be affected.

gboard-enter1.png (924×1 px, 401 KB)

Change 812093 Patch set 5 handles the <div><br></div> but fails in other cases (e.g. when the native action adds a list item).

Change 812093 merged by jenkins-bot:

[VisualEditor/VisualEditor@master] Fixup Chromium's native Enter behaviour on Android Gboard

https://gerrit.wikimedia.org/r/812093

Probably not the same as T303109, that one is FF+SwiftKey. You can try reproducing this one with SwiftKey to confirm.

Change 836313 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/VisualEditor@master] Update VE core submodule to master (6b9937c19)

https://gerrit.wikimedia.org/r/836313

Change 836313 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Update VE core submodule to master (6b9937c19)

https://gerrit.wikimedia.org/r/836313

EAkinloose added a subscriber: EAkinloose.

✅ The cursor moves to the newly inserted line the DM is updated accordingly.

Tested on Pixel 4: See https://photos.app.goo.gl/3z6ccbt4SLexFgR6A