Nice! It looks good, and the framework you have is reasonable as a human reader (and presumably tractable on the computing side).

I have a few comments, questions, and suggestions:

• Assuming the sample diagram is based on some real data... people really don't go from fulltext SERP pages to target pages very often. Autocomplete really is pretty awesome!
• Looks like you are coalescing autocomplete -> autocomplete into autocomplete, which I agree with.. otherwise it'd be 95%+ looping back on itself (letter-by-letter-by-letter).
• Is the client-time-stamp-sorted data cleaner? Server-side-sorted had a few cases of a spurious post-SERP autocomplete.
• Does random page mean "random wiki page" or "random anywhere page/source"?
  • The logs I looked at for abandoned queries include people just appearing on the SERP page out of nowhere. I think I do that, too, with a keyword in my browser bar that takes me right to the fulltext results page.
  • Would it make sense to have random wiki page (or just wiki page if that's too long) and non-wiki source or off-wiki or whatever? (naming is hard)
  • I don't think I saw any autocomplete queries out of nowhere, though. Other than using the API, is that possible? Seems okay to ignore.
• I think the reformulate label could be potentially misleading. In the abandoned query data I saw people just issue a brand new query (not necessarily entirely unrelated, but clearly not a reformulation). I don't know if we want to dig into trying to try to identify reformulation with edit distance or Jaccard index or whatever. If not, then maybe requery or something similar that doesn't imply we believe all subsequent queries are reformulations.
• Assuming I'm not the only person who does it—I bet there are dozens of us!—how does hitting "search" on the Go bar with an empty query just to get to the SERP page get represented? Would it be autocomplete -> serp? I'd be okay with that if that's what happens; I'd like to call it out, but it isn't critical. It could also be random page -> serp or it's own thing. (It feels really weird submitting a keyword/regex query though the Go box.)
• Is there no serp -> autocomplete? In the abandonded queries I saw serp -> autocomplete -> target page (presumably using the Go box on the SERP page, rather than the fulltext search box on the SERP page). Would that just be serp -> target page here?
• More generally, does your aggregation code catch paths that don't fit the model here, or does everything end up getting wedged in some way or other? If there are paths that don't fit the model, seeing a count would be interesting. Something like "...and 0.00012% other paths." (If it ever got above some threshold we might want to look into it...)
• Also, if you are handing out ponies, puppies, and rainbows...
  • How hard would it be to put percentages on the flows? With the necessary curves of the paths it's even harder to estimate by eye.. like is the serp -> end session 50%? 55%? I can't quite tell...
  • You know as soon as you make a thing, I'm going to ask if I can slice it by language or by wiki. (When as I see cool stats I want to compare—I can't help it!)

This really is encouraging—I'm glad it seems tractable. It's complex, but readable, and it will give a good sense of what is happening throughout the whole system. I expect it'll put fulltext abandonment in a much clearer context, for example. Cool stuff!

In T392525#10818319, @TJones wrote:

Nice! It looks good, and the framework you have is reasonable as a human reader (and presumably tractable on the computing side).

I have a few comments, questions, and suggestions:

• Assuming the sample diagram is based on some real data... people really don't go from fulltext SERP pages to target pages very often. Autocomplete really is pretty awesome!

Sadly no, i invented those numbers out of thin air. But real numbers coming soon!

• Looks like you are coalescing autocomplete -> autocomplete into autocomplete, which I agree with.. otherwise it'd be 95%+ looping back on itself (letter-by-letter-by-letter).

Indeed, I'm actually coalescing at the page level. I essentially compress all the events submitted for a single page load into a series of booleans that describe if that page load showed ac/ft results, had a click through, etc. The paths are then calculated from the sorted per-page data. The paths do miss some of the nuance, for example if you trigger autocomplete serps but don't go through on multiple pages that won't even mark as a transition, the user will simply stay at the AutoComplete stage, but it seemed reasonable.

• Is the client-time-stamp-sorted data cleaner? Server-side-sorted had a few cases of a spurious post-SERP autocomplete.

It seems reasonable enough, looking at the post-page-aggregation data i haven't noticed any timestamp issues

• Does random page mean "random wiki page" or "random anywhere page/source"?
  • The logs I looked at for abandoned queries include people just appearing on the SERP page out of nowhere. I think I do that, too, with a keyword in my browser bar that takes me right to the fulltext results page.
  • Would it make sense to have random wiki page (or just wiki page if that's too long) and non-wiki source or off-wiki or whatever? (naming is hard)
  • I don't think I saw any autocomplete queries out of nowhere, though. Other than using the API, is that possible? Seems okay to ignore.

To better capture the intent I've renamed that to Session Start now. The intent was that they are somewhere on the wiki when the session starts. But that very well can be the SERP page directly. I hadn't considered trying to classify where people were starting the session from, but i suppose it would probably fit in the graph.

• I think the reformulate label could be potentially misleading. In the abandoned query data I saw people just issue a brand new query (not necessarily entirely unrelated, but clearly not a reformulation). I don't know if we want to dig into trying to try to identify reformulation with edit distance or Jaccard index or whatever. If not, then maybe requery or something similar that doesn't imply we believe all subsequent queries are reformulations.

I agree. I've renamed this to requery to be clearer. I also need to do a little more work here, right now I'm counting all SERP views as a requery, but if the user has already performed that exact query it should be counted as the 'back' button (from my initial graphic).

• Assuming I'm not the only person who does it—I bet there are dozens of us!—how does hitting "search" on the Go bar with an empty query just to get to the SERP page get represented? Would it be autocomplete -> serp? I'd be okay with that if that's what happens; I'd like to call it out, but it isn't critical. It could also be random page -> serp or it's own thing. (It feels really weird submitting a keyword/regex query though the Go box.)

It will likely be Session Start -> SERP, to get the AutoComplete step the user would need to see autocomplete results.

• Is there no serp -> autocomplete? In the abandonded queries I saw serp -> autocomplete -> target page (presumably using the Go box on the SERP page, rather than the fulltext search box on the SERP page). Would that just be serp -> target page here?

I suppose there could be, but typically i throw out autocomplete events on the fulltext serp page. The general idea is most pages autocomplete is navigational title completion, where autocomplete on the SERP is probably query completion. I suppose I'm slightly narrowing the idea of autocomplete to refer specifically to the navigational component. They can of course navigate away from the fulltext SERP using the skin autocomplete, but that isn't being tracked here either.

There can be, but i'm not counting autocomplete that happens on the fulltext search page. I suppose i've historically done this because autocompleting a fulltext query is different from autocompleting a title to navigate to.

• More generally, does your aggregation code catch paths that don't fit the model here, or does everything end up getting wedged in some way or other? If there are paths that don't fit the model, seeing a count would be interesting. Something like "...and 0.00012% other paths." (If it ever got above some threshold we might want to look into it...)

Everything gets wedged in one way or another. We end up with a few curious edges that way, such as SessionStart -> TargetPage, but it's not clear what to do with them. Currently I throw away some of the odd edges that don't really make sense.

• Also, if you are handing out ponies, puppies, and rainbows...
  • How hard would it be to put percentages on the flows? With the necessary curves of the paths it's even harder to estimate by eye.. like is the serp -> end session 50%? 55%? I can't quite tell...

For the current graphing solution, plotly, i'm not seeing any options to label the flows with %. This does have hover effects in the jupyter notebook, and i can add custom text to the hover flyout, but it's not clear if that would make it through publishing. I agree it would be nice to have some % if i can figure out how.

• You know as soon as you make a thing, I'm going to ask if I can slice it by language or by wiki. (When as I see cool stats I want to compare—I can't help it!)

Yes this currently reuses the same dimensions we have available in the search satisfaction metrics. It can be sliced by wiki/geoip/edit count bucket. I'm not sure how that will fit into a reporting bit though.

This really is encouraging—I'm glad it seems tractable. It's complex, but readable, and it will give a good sense of what is happening throughout the whole system. I expect it'll put fulltext abandonment in a much clearer context, for example. Cool stuff!

They can of course navigate away from the fulltext SERP using the skin autocomplete, but that isn't being tracked here either.

That was the case I had in mind. Fulltext autocomplete is like you said, just query completion (though I have at times wished clicking on a sugestion would take me to that page).

ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/notebooks/-/merge_requests/8

Analysis of user flows through desktop search

Attached patch includes the notebook and instructions for how to re-run the notebook in the future to generate an html report. We don't have a great answer around where to publish html reports, so I've put this on people.wikimedia.org for now:

• https://people.wikimedia.org/~ebernhardson/T392525-Sankey-Exploration-2025-04.html

The report looks awesome!

I have a couple of minor suggestions, none of which are necessary to call this done:

• It would be cool if the Most Common Paths could also include another column with their path classifications.
• Lots of tables could benefit from a column with percentages: Most Common Paths & Path Classification, Vertex Input Counts, Most Common State Transitions, various session dimensions.

Questions:

• Are the path classifications mutually exclusive? I thought they were, but AutoComplete Navigation and Search Refinement seem like they overlap with Direct Success. OTOH, the total from the Path Classification table is 60,237,081 (vs total sessions of 60,173,294).. which implies some overlap, but much less (~0.1%) than I'd expect.

Some other observations:

• The fact that the javascript highlighting of the Sankey diagrams themselves is there is really great!!
• It's funny that four of the top 10 paths are "autocomplete to a page", "autocomplete to a page" x2, ...x3, and ...x4. If it ain't broke don't fix it!
• It's less funny that four of the top 10 look like abandonment.. though the skew in size is pretty extreme, with the #1 path being >50x more common than #10.

Overall this is just fantastic!

In T392525#10853973, @TJones wrote:

The report looks awesome!

I have a couple of minor suggestions, none of which are necessary to call this done:

• It would be cool if the Most Common Paths could also include another column with their path classifications.
• Lots of tables could benefit from a column with percentages: Most Common Paths & Path Classification, Vertex Input Counts, Most Common State Transitions, various session dimensions.

Questions:

• Are the path classifications mutually exclusive? I thought they were, but AutoComplete Navigation and Search Refinement seem like they overlap with Direct Success. OTOH, the total from the Path Classification table is 60,237,081 (vs total sessions of 60,173,294).. which implies some overlap, but much less (~0.1%) than I'd expect.

They should be mutually exclusive. I'll try to update the wording, it was awkward to state. Auto Complete Navigation means they used autocomplete 2+ times, Search Refinement means they performed two different full text queries, where direct success is capturing the simplest possible paths of autocomplete -> ac targetpage and autocomplete -> serp -> ft targetpage (and some variants).

The variance in counts is curious. The primary analysis and the path classification do run two separate aggregations, but the definition of a session should have been the same in both. It turned out the difference was that i was only throwing out empty sessions (SessionStart -> SessionEnd) in one of them. I'm now throwing away the empty sessions on both sides and the counts line up.

I've also updated the report to have 2 variants, the full graph and a simplified graph. Found here as well in the README.md:

• https://people.wikimedia.org/~ebernhardson/T392525-Sankey-Exploration-2025-04.html
• https://people.wikimedia.org/~ebernhardson/T392525-Sankey-Exploration-Simplified-2025-04.html

ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/notebooks/-/merge_requests/8

Analysis of user flows through desktop search