-
Type: Bug
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: None
The current double precision truncation code has a mathematically unnecessary multiplication and division. By removing these, we can simply the code to eliminate the loss of precision when the integer domain size is between 2^53 and 2^54.
New algorithm
int64_t v_prime = (int64_t)(trunc(args.value * exp10Double(args.precision.value))); int64_t scaled_min = (int64_t)(args.min.value * exp10Double(args.precision.value)); int64_t v_prime2 = v_prime - scaled_min;
1. Avoids an extra division in v_prime and then multiplication on v_prime
2. Uses integer math to compute the domain size which has more bits then doubles for tracking integers. This algorithm only works when numbers of < 64 bits.
Credit to algorithm goes to erwin.pe@mongodb.com and shreyas.kalyan@mongodb.com.
- depends on
-
MONGOCRYPT-702 FLE2 range floating point correctness issue when using precision mode
- Closed