Change Details

The UNIX time in milliseconds is used as a key part of generating our UUIDs. For current UUIDs, we use the actual such time (from microtime). However, for imports we don't have access to millisecond-level (database timestamps in MW don't include milliseconds). So we randomly generate them. There can be collisions, which are more likely if there are multiple events at the same second (quite possible with normal activity, but also bots). I've been investigating this, and so far found https://gerrit.wikimedia.org/r/#/c/251265/ and https://gerrit.wikimedia.org/r/#/c/251256/ . The Flow one is merged. The difficulty here is that there should be ~52 bits of entropy (32-bit random node ID changed each call plus log2(999) random millisecond bits plus 10-bit counter), *not* counting the second level. (Counter is incremented, not random). Will add calculations of expected number in a second before collisions, then script testing actual behavior. We don't use the same random number generator for all these calls, but not sure that is a problem.

The UNIX time in seconds + milliseconds is used as a key part of generating our UUIDs. For current UUIDs, we use the actual such time (from microtime). However, for imports we don't have access to millisecond-level (database timestamps in MW don't include milliseconds). So we randomly generate the millisecond part of the timestamp. There can be collisions, which may be more likely if there are multiple events at the same second (quite possible with normal activity, but also bots). I've been investigating this, and so far found https://gerrit.wikimedia.org/r/#/c/251265/ and https://gerrit.wikimedia.org/r/#/c/251256/ . The Flow one is merged. The difficulty here is that there should be ~52 bits of entropy (32-bit random node ID changed each call plus log2(999) random millisecond bits plus 10-bit counter), *not* counting the second level. (Counter is randomly initialized, then incremented). Will add calculations of expected number in a second before collisions, then script testing actual behavior. We don't use the same random number generator for all these calls, but not sure that is a problem (the real issue is whether there is some artifact).

The UNIX time in seconds + milliseconds is used as a key part of generating our UUIDs. For current UUIDs, we use the actual such time (from microtime). However, for imports we don't have access to millisecond-level (database timestamps in MW don't include milliseconds). So we randomly generate themthe millisecond part of the timestamp. There can be collisions, which aremay be more likely if there are multiple events at the same second (quite possible with normal activity, but also bots). I've been investigating this, and so far found https://gerrit.wikimedia.org/r/#/c/251265/ and https://gerrit.wikimedia.org/r/#/c/251256/ . The Flow one is merged. The difficulty here is that there should be ~52 bits of entropy (32-bit random node ID changed each call plus log2(999) random millisecond bits plus 10-bit counter), *not* counting the second level. (Counter is incrementedrandomly initialized, not randomthen incremented). Will add calculations of expected number in a second before collisions, then script testing actual behavior. We don't use the same random number generator for all these calls, but not sure that is a problem (the real issue is whether there is some artifact).