Uploaded image for project: 'Libmongocrypt'
  1. Libmongocrypt
  2. MONGOCRYPT-712

Improve double handling with precision

      The current double precision truncation code has a mathematically unnecessary multiplication and division. By removing these, we can simply the code to eliminate the loss of precision when the integer domain size is between 2^53 and 2^54.

      New algorithm

              int64_t v_prime = (int64_t)(trunc(args.value * exp10Double(args.precision.value)));
              int64_t scaled_min = (int64_t)(args.min.value * exp10Double(args.precision.value));
              int64_t v_prime2 = v_prime - scaled_min;
      

      1. Avoids an extra division in v_prime and then multiplication on v_prime
      2. Uses integer math to compute the domain size which has more bits then doubles for tracking integers. This algorithm only works when numbers of < 64 bits.

      Credit to algorithm goes to erwin.pe@mongodb.com and shreyas.kalyan@mongodb.com.

            Assignee:
            mark.benvenuto@mongodb.com Mark Benvenuto
            Reporter:
            mark.benvenuto@mongodb.com Mark Benvenuto
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: