There's some discussion of this in T124305 as well.

So in the initial design in T124305#3971292 the context is suppressed on click. The main problem with that is the context will reappear on the next contextChange event, which could happen within the link (if there are other annotations, for example) which will happen when the users starts typing, as the selection goes from expanded to collapsed. We would have to somehow suppress the inspector until the link was unfocused. That would then feel a little hacky, as the user would have to know they need to click out of the link, then back in to it if they wanted to the edit the target, but that may be the lesser of two evils.

A simpler approach might be to hide the "Change label" bar iff the link is fully selected. (This has the opposite problem in that contextChange doesn't fire often enough, so we would have to listen to surface model select events to know if the link was selected).

In addition to T199617 I think the change label button still needs some work.

In T201232#4484982, @Esanders wrote:

A simpler approach might be to hide the "Change label" bar iff the link is fully selected.

You can paste this into your console to experiment with this functionality:

ve.ui.LinkContextItem.prototype.renderBody = function () {
	var surfaceModel = this.context.getSurface().getModel(),
		htmlDoc = surfaceModel.getDocument().getHtmlDocument(),
		annotationSelection = surfaceModel.getFragment().expandLinearSelection( 'annotation', this.model ).getSelection();
	this.$body.empty().append(
		$( '<a>' )
			.text( this.getDescription() )
			.attr( {
				href: ve.resolveUrl( this.model.getHref(), htmlDoc ),
				target: '_blank',
				rel: 'noopener'
			} )
	);
	if ( !annotationSelection.equals( surfaceModel.getSelection() ) ) {
		this.$body.append( this.$labelLayout );
		this.updateLabelPreview();
	}
};

Change 454520 had a related patch set uploaded (by Esanders; owner: Esanders):
[VisualEditor/VisualEditor@master] WIP Only show change label button when link not fully selected

https://gerrit.wikimedia.org/r/454520

I agree that this is problematic. Especially the part of auto-selecting the whole text link. It seems more sensible to me to put the cursor there and let the user change it. Currently, just writing something deletes the whole text.

I actually don't use the "Change label" button much myself. Instead, I just click the text and change it. But for people who do use the "Change label" this could be better. (How many people use it? I don't know, and I'm curious. I remember some editors being excited when this button was added, but this says nothing about actual usage.)

In T201232#5316698, @Amire80 wrote:

...Especially the part of auto-selecting the whole text link. It seems more sensible to me to put the cursor there and let the user change it. Currently, just writing something deletes the whole text.

This is a great spot, @Amire80. For Edit Cards v2, a tap on the edit pencil to edit an existing link's label will default to the cursor being placed within the existing label rather than entire label being highlighted. This change is represented by this task: T228227

This change (T228227) will apply to mobile only. Desktop functionality will be unchanged.

In T201232#5343770, @Esanders wrote:

This change (T228227) will apply to mobile only. Desktop functionality will be unchanged.

Oh, yes. Mobile only. Thank you for clarifying, Ed.