This page provides the Reproduction Angular Errors of several illuminant estimation algorithms for different datasets. Reproduction Angular Error was proposed by Graham D. Finlayson and Roshanak Zakizadeh in British Machine Vision Conference (BMVC) 2014. The paper can be downloaded from here.
Data sets:
SFU Laboratory Images
Note that illuminant estimates are available for a wide range of parameter-settings (i.e. Minkowski-norm is varied between 1 and 15, and the smoothing value ? is varied between 1 and 12).
| Method (applied to 321 images) | Mean error | Median error | Trimean error | 95% quantile error | |
|---|---|---|---|---|---|
| Grey-World: | 10.1o | 7.5o | 8.3o | 28o | |
| White-Patch: | 9.7o | 7.4o | 8.2o | 27.3o | |
| Shades-of-Grey: (p = 7) | 6.9o | 3.9o | 4.8o | 22.5o | |
| general Grey-World (p = 10, ? = 4) | 6.0o | 3.9o | 4.3o | 20.2o | |
| 1st-order Grey-Edge (p = 14, ? = 4) | 6.3o | 3.6o | 4.2o | 21.8o | |
| 2nd-order Grey-Edge (p = 15, ? = 10) | 5.8o | 3.0o | 3.8o | 24.7o | |
| Pixel-based Gamut (? = 4) | 4.2o | 2.8o | 3.0o | 14.8o | |
| Edge-based Gamut (? = 2) | 4.5o | 2.7o | 3.2o | 14.3o | |
| Union-based Gamut (? = 2) | 5.2o | 3.4o | 3.9o | 15.3o | |
| Intersection-based Gamut (? = 3) | 4.2o | 2.5o | 2.8o | 17.6o | |
| Inv. Intensity Chrom. Space | 15.1o | 9.3o | 11.5o | 43.2o | |
| Spatial Correlations: — HeavyTailed-based |
6.2o | 4.1o | 5.0o | 16.6o | |
| Weighted Grey-Edge: — p = 2, ? = 1, ? = 8 |
6.1o | 3.6o | 4.3o | 19.3o |
SFU Grey-ball Set (Original)
The original Grey-ball set consists of 11,346 images. However, because of the large correlation among some consecutive frames, various researchers decided to use a subset of this set. Feel free to use the following data to extract any results appropriate, since they are computed by processing the full set:
| Method (applied to 11346 images) | Mean error | Median error | Trimean error | 95% quantile error | |
|---|---|---|---|---|---|
| Grey-World | 8.7o | 7.6o | 7.9o | 20.9o | |
| White-Patch | 7.1o | 5.5o | 6.0o | 18.0o | |
| Shades-of-Grey (p = 14) | 6.5o | 5.6o | 5.8o | 15.2o | |
| general Grey-World (p = 12, ? = 0) | 7.1o | 6.2o | 6.4o | 16.6o | |
| 1st-order Grey-Edge (p = 2, ? = 1) | 6.3o | 4.8o | 5.4o | 15.7o | |
| 2nd-order Grey-Edge (p = 1, ? = 2) | 6.5o | 5.0o | 5.6o | 15.7o | |
| Pixel-based Gamut (? = 2) | 7.5o | 5.9o | 6.3o | 19.6o | |
| Edge-based Gamut (? = 1) | 7.3o | 5.8o | 6.3o | 18.1o | |
| Intersection-based Gamut (? = 2) | 7.5o | 5.9o | 6.4o | 19.6o | |
| Inv. Intensity Chrom. Space | 7.0o | 6.0o | 6.2o | 15.9o | |
| Using Natural Image Statistics | 5.5o | 4.3o | 4.7o | 13.7o | |
| Exemplar-Based Color Constancy | 4.8o | 3.7o | 4.0o | 12.5o | |
| Color Cat | 4.6o | 3.4o | 3.8o | 12.6o | |
| ColorDog: — Using WP / GW |
4.9o | 3.2o | 3.8o | 13.4o | |
| — Using ColorCat | 5.7o | 4.3o | 4.7o | 15.8o |
Color-checker (by Shi)
Since the original color checker set wat generated from RAW data using automatic settings, Shi reprocessed the original RAW data and generated 12-bit PNG-images (with lossless compression).
| Method (applied to 568 images) | Mean error | Median error | Trimean error | 95% quantile error | |
|---|---|---|---|---|---|
| Grey-World | 7.0o | 6.8o | 6.9o | 12.4o | |
| White-Patch | 8.1o | 6.5o | 7.1o | 20.0o | |
| Shades-of-Grey (p = 5) | 5.8o | 4.4o | 4.9o | 14.7o | |
| General Grey-World (p = 9, ? = 9) | 5.3o | 4.0o | 4.4o | 14.0o | |
| 1st-order Grey-Edge (p = 1, ? = 9) | 6.4o | 4.9o | 5.3o | 15.8o | |
| 2nd-order Grey-Edge (p = 1, ? = 1) | 6.0o | 4.8o | 5.2o | 13.7o | |
| Pixel-based Gamut (? = 4) | 4.8o | 2.7o | 3.4o | 15.8o | |
| Edge-based Gamut (? = 3, edge type = 3) | 8.0o | 5.9o | 6.6o | 20.5o | |
| Intersection-based Gamut (? = 4) | 4.8o | 2.7o | 3.4o | 16.4o | |
| Inv. Intensity Chrom. Space | 14.3o | 13.6o | 13.6o | 22.6o | |
| Regression (SVR) | 8.8o | 7.4o | 7.9o | 18.9o | |
| Bayesian | 5.6o | 3.9o | 4.4o | 15.4o | |
| Spatial Correlations: — Gaussian-based |
o | o | o | o | |
| — HeavyTailed-based | 4.2o | 3.5o | 3.6o | 9.9o | |
| Bottom-up (see here) | 4.0o | 3.0o | 3.2o | 11.6o | |
| Top-down (see here) | 4.3o | 3.1o | 3.2o | 13.8o | |
| Bottom-up+Top-down (see here) | 4.0o | 2.8o | 3.0o | 13.6o | |
| Using Natural Image Statistics | 4.8o | 3.5o | 3.9o | 12.9o | |
| CART-based Selection | 5.2o | 4.0o | 4.1o | 14.6o | |
| CART-based Combination | 4.5o | 3.5o | 3.8o | 11.4o | |
| Exemplar-Based Color Constancy | 3.4o | 2.6o | 2.9o | 8.3o | |
| Using CNNs: — AlexNet+SVR |
5.5o | 3.6o | 4.1o | 16.3o | |
| — Deep color constancy | 3.1o | 2.2o | 2.5o | 9.0o |
ColorChecker (RECommended)
The table below was updated on March 24, 2018.
We re-processed the Gehler data to address the problem raised in G. D. Finlayson, G. Hemrit, A. Gijsenij, and P. Gehler, “A Curious Problem with Using the Colour Checker Dataset for Illuminant Estimation,” in Color and Imaging Conference, 2017, pp. 64–69.
If you use the data and results below then please cite G. Hemrit et al., “Rehabilitating the ColorChecker Dataset for Illuminant Estimation,” in Color and Imaging Conference, 2018. see paper here
Finlayson et al. demonstrated that the previous rankings and performance evaluation of algorithms for the ColorChecker dataset are ill-founded, because of mixing different ground-truths. The following results are obtained using the RECommended ground-truth, calculated on the re-processed images from Gheler’s raw images, according to the calculation methodology described by Shi and Funt.
Information on the re-processed dataset as well as the images, are accessible on the Datasets page. If you have any contribution to bring to this work, feel free to send a message to the contact person of this web-site.
| Method (applied to 568 images) | Mean error | Median error | Trimean error | 95% quantile error | Max error | |
|---|---|---|---|---|---|---|
| Grey-World | 10.7o | 10.6o | 10.7o | 17.4o | 27.0o | |
| White-Patch | 9.8o | 8.0o | 8.9o | 22.3o | 32.6o | |
| Shades-of-Grey (p = 5) | 8.3o | 7.5o | 7.8o | 17.0o | 23.0o | |
| General Grey-World (p = 9, ? = 9) | 7.6o | 6.7o | 7.0o | 16.5o | 22.9o | |
| 1st-order Grey-Edge (p = 1, ? = 9) | 5.0o | 3.7o | 4.1o | 13.3o | 28.8o | |
| 2nd-order Grey-Edge (p = 1, ? = 1) | 5.4o | 4.5o | 4.8o | 12.8o | 22.6o | |
| Pixel-based Gamut (? = 4) | 6.9o | 5.2o | 5.7o | 18.2o | 28.9o | |
| Edge-based Gamut (? = 3, edge type = 3) | 6.9o | 4.6o | 5.2o | 20.6o | 40.1o | |
| Intersection-based Gamut (? = 4) | 6.8o | 5.2o | 5.6o | 18.3o | 30.1o | |
| Inv. Intensity Chrom. Space | 16.3o | 12.8o | 14.3o | 34.0o | 38.3o | |
| Regression (SVR) | 11.8o | 10.3o | 10.9o | 22.1o | 29.8o | |
| Bayesian | 6.6o | 4.6o | 5.2o | 18.4o | 30.4o | |
| Spatial Correlations: — HeavyTailed-based | 6.8o | 5.8o | 6.1o | 15.9o | 21.9o | |
| Bottom-up | 6.7o | 5.9o | 6.1o | 14.7o | 22.4o | |
| Top-down | 7.0o | 5.4o | 5.7o | 17.7o | 31.1o | |
| Bottom-up+Top-down | 6.6o | 5.4o | 5.6o | 15.8o | 31.1o | |
| Using Natural Image Statistics | 6.5o | 5.5o | 5.7o | 16.2o | 22.7o | |
| CART-based Selection | 7.1o | 6.1o | 6.4o | 16.2o | 25.3o | |
| CART-based Combination | 7.2o | 6.6o | 6.8o | 14.0o | 21.3o | |
| Exemplar-Based Color Constancy | 6.0o | 5.2o | 5.4o | 13.2o | 19.5o | |
| Using CNNs: — AlexNet+SVR |
8.2o | 6.3o | 6.8o | 20.4o | 31.6o | |
| — Deep color constancy | 5.7o | 4.7o | 5.0o | 12.4o | 18.9o | |
| Fast Fourier Color Constancy (model Q) | 2.5o | 1.4o | 1.8o | 8.8o | 27.8o |