When listing a subset of pages in a category using list=categorymembers and the cmstartsortkeyprefix and cmendsortkeyprefix parameters behaviour is as expected if all the requested pages are returned without a continuation call. e.g. this sandbox request for all the Gastropods described in 1999 that have a sortkey prefix of H.
If cmlimit is reduced below the number of pages that should be returned, so that a continue loop is required, the API returns pages and a continue value in the response until the end of the category is reached.
e.g. this sandbox request https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&list=categorymembers&cmtitle=Category%3AGastropods_described_in_1999&cmtype=page&cmlimit=20&cmstartsortkeyprefix=H&cmendsortkeyprefix=I
As expected, the sandbox shows a continue button, but when it is pressed results are returned that go beyond the requested range.
The attached python file shows that same test case in python code.
I have found this issue also affects the gcmendsortkeyprefix parameter when categorymembers is used as a generator.