Repository navigation
MATH-1696: Order enumerated real masses when computing quantiles - #353
Conversation
Sort the copied PMF in inverseCumulativeProbability, without changing stored input order or seeded sampling. Cover unsorted values from both constructors, duplicate masses, zero weights, and support endpoints.
aherbert
left a comment
There was a problem hiding this comment.
Thanks for the big report. I suggested a different approach. Please verify.
Note the EnumeratedIntegerDistribution follows the same pattern. It does not have this bug as it does not override inverseCumulativeProbability. It delegates to a binary search within a bracket using repeat calls to cumulative probability which will be slow due to repeat passes over the entire data. A change to add a single pass inverseCumulativeProbability would require a sorted inner distribution and could be supported the same way.
| double x = getSupportLowerBound(); | ||
| for (final Pair<Double, Double> sample : innerDistribution.getPmf()) { | ||
| final List<Pair<Double, Double>> samples = innerDistribution.getPmf(); | ||
| samples.sort((a, b) -> Double.compare(a.getKey(), b.getKey())); |
There was a problem hiding this comment.
Sorting the PMF on each call to inverseCumulativeProbability is a large performance overhead. The sorted list should be computed once.
The innerDistribution is private and constructed by iterating over a Map of value -> count pairs. This is a LinkedHashMap. If this is changed to TreeMap then the inner distribution will be created with a sorted set of values. This fixes the case where the values constructor is called (equal weights).
public EnumeratedRealDistribution(final double[] data) {
final Map<Double, Integer> dataMap = new TreeMap<>();
// ...For the other case (values and counts), the List of samples can be sorted:
public EnumeratedRealDistribution(final double[] singletons,
final double[] probabilities)
throws /* lots of exceptions */ {
final List<Pair<Double, Double>> samples = createDistribution(singletons, probabilities);
samples.sort(Comparator.comparingDouble(Pair::getKey));
innerDistribution = new EnumeratedDistribution<>(samples);
}The modifications above fix your new tests.
|
Thanks, I verified the constructor-sorting approach and agree it is better here. It avoids sorting on every quantile call without an extra cached PMF, and keeps CDF and inverse-CDF accumulation in the same order. I've added a regression for that consistency case. The legacy module tests and verification pass. Sorting changes seeded sample sequences and floating-point accumulation order; I have left EnumeratedIntegerDistribution unchanged. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #353 +/- ##
============================================
+ Coverage 86.54% 87.23% +0.68%
+ Complexity 9787 91 -9696
============================================
Files 532 499 -33
Lines 35516 33534 -1982
Branches 6194 5853 -341
============================================
- Hits 30738 29252 -1486
+ Misses 3518 3160 -358
+ Partials 1260 1122 -138 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Summary
Fixes MATH-1696.
EnumeratedRealDistributionallows unsorted values and duplicate masses, but its quantile method currently accumulates probabilities in input order. For values[7, 3, -1, 3]with equal weights,inverseCumulativeProbability(0)returns7instead of-1, andinverseCumulativeProbability(1)returns3instead of7.Sort the copied PMF only inside
inverseCumulativeProbability. Stored input order and seeded sampling remain unchanged. No constructor, public API or sampling change.Three regression tests cover both constructors, duplicate masses, zero-weight outliers, support endpoints, and unchanged seeded sampling after a quantile query. The two unsorted-input tests fail against unchanged production code. Also reproduced on the published 3.3 jar.
Validation
Linux, JDK 17:
git diff --checkpasses.The initial uninterrupted
mvn clean verifyexceeded the execution time limit. Tests and verification were completed in separate module/phase runs. The default legacy test execution completed, then the separate configuredflaky-testsexecution was rerun to completion usingmvn -f commons-math-legacy/pom.xml surefire:test@flaky-tests. This is split validation, not a claim that one uninterrupted default build completed. No test, assertion, retry setting or check configuration was changed.AI disclosure: This PR description and the fix/tests were generated with AI assistance (Instinct) and checked with the validation above.