Until now, only a limited number of algorithms on a limited number of data sets are available. If you have or know of any additions, please let me know.
You can browse the results per data set or per method. The first shows the results of all methods that are available for a particular data set. The second shows the results of one particular method on all data sets.
Data sets:
— SFU Grey-ball Set (Original)
— SFU Grey-ball Set (Linear)
— Color-checker (Original)
— Color-checker (by Shi)
SFU Laboratory Images
Results on this full set are available for the following low-level methods. Note that illuminant estimates are available for a wide range of parameter-settings (i.e. Minkowski-norm is varied between 1 and 15, and the smoothing value ? is varied between 1 and 12).
| Method (applied to 321 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Grey-World: | 9.8o | 7.0o | 7.6o | 37.3o | |
| White-Patch: | 9.1o | 6.5o | 7.5o | 36.2o | |
| Shades-of-Grey: (p = 7) | 6.4o | 3.7o | 4.6o | 29.6o |  |
| general Grey-World (p = 10, ? = 4) | 5.4o | 3.3o | 3.8o | 28.9o |  |
| 1st-order Grey-Edge (p = 7, ? = 4) | 5.6o | 3.2o | 3.7o | 31.6o |  |
| 2nd-order Grey-Edge (p = 7, ? = 5) | 5.2o | 2.7o | 3.3o | 26.7o |  |
Results of the Gamut mapping are also available, but this method requires training data. The following results are obtained using 31 images (all images recorded under the “syl-50MR16Q”-illuminant) to construct the canonical gamut. This canonical gamut is then used to estimate the illuminant for all images, so including the training images. This is done to be able to make a comparison with other methods; if you would like to compute the performance of merely the test image, simply download the results and exclude the results of the 31 training images.
| Method (applied to 321 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Pixel-based Gamut (? = 4) | 3.7o | 2.3o | 2.5o | 27.1o |  |
| Edge-based Gamut (? = 2) | 3.9o | 2.3o | 2.7o | 29.7o |  |
| Union-based Gamut (? = 2) | 4.7o | 3.0o | 3.4o | 27.7o |  |
| Intersection-based Gamut (? = 4) | 3.6o | 2.1o | 2.4o | 27.1o |  |
Various other methods have been applied to this set. If you would like to have your results reported here, feel free to send a message to the contact person of this web site.
| Method (applied to 321 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Inv. Intensity Chrom. Space | 15.5o | 8.2o | 10.7o | 80.9o | |
| Spatial Correlations: — HeavyTailed-based |
5.6o | 3.5o | 4.3o | 21.6o | |
| Weighted Grey-Edge: — p = 7, ? = 5, ? = 8 |
5.6o | 2.4o | 2.9o | 43.8o |
The results of Spatial Correlations denoted “HeavyTailed-cbased” is using the newest implementation and can be found in “Color Constancy with Spatio-spectral Statistics”, in IEEE TPAMI. More elaborate results by Chakrabarti et al. can be downloaded here. These results also contain slightly different results for the Grey-Edge-type methods.
SFU Grey-ball Set (Original)
The original Grey-ball set consists of 11,346 images. However, because of the large correlation among some consecutive frames, various researchers decided to use a subset of this set. Feel free to use the following data to extract any results appropriate, since they are computed by processing the full set:
| Method (applied to 11346 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Grey-World | 7.9o | 7.0o | 7.1o | 48.1o | |
| White-Patch | 6.8o | 5.3o | 5.8o | 38.7o | |
| Shades-of-Grey (p = 12) | 6.1o | 5.3o | 5.5o | 41.2o |  |
| general Grey-World (p = 12, ? = 0) | 6.1o | 5.3o | 5.5o | 41.2o |  |
| 1st-order Grey-Edge (p = 1, ? = 1) | 5.9o | 4.7o | 5.1o | 41.2o |  |
| 2nd-order Grey-Edge (p = 1, ? = 2) | 6.1o | 4.9o | 5.3o | 41.7o |  |
When obtaining results of the Gamut Mapping (or any other learning-based method) on the Grey-ball Set, one should carefully take the correlation between consecutive frames into account. We propose to use 15-fold cross-validation. The 15 different folds are constructed by grouping images of the same scene, ensuring the correlated images to be either all in the test set or all in the training set. (Alternatively, one can choose to create a subset where the correlation is eliminated). Results of the 15-fold cross-validation are shown here:
| Method (applied to 11346 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Pixel-based Gamut (? = 5) | 7.1o | 5.8o | 6.1o | 41.9o |  |
| Edge-based Gamut (? = 3) | 6.8o | 5.8o | 6.0o | 40.3o |  |
| Intersection-based Gamut (? = 9) | 6.9o | 5.8o | 6.1o | 41.9o |  |
Various other methods have been applied to this set. If you would like to have your results reported here, feel free to send a message to the contact person of this web site.
— SmartColorCat4.6o3.5o3.8o46.2o
| Method (applied to 11346 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Inv. Intensity Chrom. Space | 6.6o | 5.6o | 5.8o | 76.2o | |
| Using Natural Image Statistics | 5.2o | 3.9o | 4.3o | 44.5o | |
| Exemplar-Based Color Constancy | 4.4o | 3.4o | 3.7o | 45.6o | |
| Various methods (Uni. Zagreb): — ColorCat |
4.2o | 3.2o | 3.5o | 43.7o | |
| — ColorDog Using WP / GW | 5.3o | 3.7o | 4.2o | 46.8o | |
| — ColorDog Using ColorCat | 4.5o | 2.9o | 3.5o | 46.2o | |
| — ColorDog Using Sm.ColorCat | 4.8o | 3.1o | 3.7o | 48.7o | |
| — ColorAnts | 5.0o | 3.7o | 4.1o | 46.5o |
SFU Grey-ball Set (Linear)
Since the original Grey-ball Set was stored in a non-linear device-RGB color space (NTSC=RGB), we created a modification of this set by applying gamma-correction (with ? = 2.2). For consistency, we recomputed the ground truth on the linear images (click here to download this new ground truth). Results on this modified data set show all methods obtain considerably higher angular errors:
| Method (applied to 11346 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Grey-World | 13.0o | 11.0o | 11.5o | 63.0o | |
| White-Patch | 12.7o | 10.5o | 11.3o | 46.5o | |
| Shades-of-Grey (p = 4) | 11.6o | 9.7o | 10.2o | 58.1o |  |
| general Grey-World (p = 4, ? = 0) | 11.6o | 9.7o | 10.2o | 58.1o |  |
| 1st-order Grey-Edge (p = 1, ? = 1) | 10.6o | 8.8o | 9.2o | 58.4o |  |
| 2nd-order Grey-Edge (p = 1, ? = 1) | 10.7o | 9.0o | 9.4o | 56.0o |  |
The same restrictions as for the original Grey-ball Set apply. Results of the Gamut mapping using 15-fold cross-validation are shown here:
| Method (applied to 11346 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Pixel-based Gamut (? = 9) | 11.8o | 8.9o | 10.0o | 49.0o |  |
| Edge-based Gamut (? = 9) | 12.8o | 10.9o | 11.4o | 58.3o |  |
| Intersection-based Gamut (? = 9) | 11.8o | 8.9o | 10.0o | 47.5o |  |
Various other methods have been applied to this set. If you would like to have your results reported here, feel free to send a message to the contact person of this web site.
Various methods (Uni. Zagreb):
— ColorCat8.7o7.1o7.4o52.4o— SmartColorCat8.2o6.3o6.7o54.5o
| Method (applied to 11346 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Inv. Intensity Chrom. Space | 14.7o | 11.0o | 11.6o | 86.8o | |
| Regression (SVR) | 13.1o | 11.2o | 11.8o | 59.6o | |
| Spatial Correlations: — Gaussian-based |
12.7o | 10.8o | 11.5o | 41.2o | |
| — HeavyTailed-based | 10.3o | 8.9o | 9.2o | 53.6o | |
| Bottom-up (see here) | 10.0o | 8.0o | 8.5o | 58.5o | |
| Top-down (see here) | 10.2o | 8.3o | 8.7o | 63.0o | |
| Bottom-up+Top-down (see here) | 9.7o | 7.7o | 8.2o | 60.0o | |
| Using Natural Image Statistics | 9.9o | 7.7o | 8.3o | 56.1o | |
| Exemplar-Based Color Constancy | 8.0o | 6.5o | 6.8o | 53.6o | |
| — ColorDog Using WP / GW | 10.3o | 7.3o | 8.2o | 53.6o | |
| — ColorDog Using ColorCat | 8.8o | 6.0o | 7.0o | 54.8o | |
| — ColorDog Using Sm.ColorCat | 8.5o | 5.6o | 6.6o | 56.2o | |
| — ColorAnts | 8.9o | 6.5o | 7.1o | 57.6o |
The results of the Spatial Correlations denoted “Gaussian-based” are computed by Gijsenij et al. using an earlier implementation of the Spatial Correlations method. The results of Spatial Correlations denoted “HeavyTailed-based” is using the newest implementation and can be found in “Color Constancy with Spatio-spectral Statistics”, in IEEE TPAMI. More elaborate results by Chakrabarti et al. can be downloaded here. These results also contain slightly different results for the Grey-Edge-type methods.
Color-checker (Original)
Using the original images (gamma-corrected from sRGB to linear-RGB), we obtain the following results for the low-level methods (do not forget to mask the color checker!):
| Method (applied to 568 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Grey-World | 9.8o | 7.4o | 8.2o | 46.0o | |
| White-Patch | 8.1o | 6.0o | 6.4o | 36.3o | |
| Shades-of-Grey (p = 6) | 7.0o | 5.3o | 5.6o | 36.6o |  |
| general Grey-World (p = 6, ? = 0) | 7.0o | 5.3o | 5.6o | 36.6o |  |
| 1st-order Grey-Edge (p = 4, ? = 1) | 7.0o | 5.2o | 5.5o | 36.3o |  |
| 2nd-order Grey-Edge (p = 4;, ? = 7) | 7.0o | 5.0o | 5.4o | 38.0o |  |
Using three-fold cross-validation (with the folds supplied along with the original data set and ground truth) to learn the canonical gamut, we obtain the following results for the Gamut mapping:
| Method (applied to 568 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Pixel-based Gamut (? = 5) | 6.9o | 4.9o | 5.2o | 37.1o |  |
| Edge-based Gamut (? = 7) | 7.7o | 5.0o | 5.7o | 44.4o |  |
| Intersection-based Gamut (? = 5) | 6.9o | 4.9o | 5.2o | 37.1o |  |
Various other methods have been applied to this set. Please note that any algorithm that requires learning is trained using the same three folds as were used for the Gamut mapping. If you would like to have your results reported here, feel free to send a message to the contact person of this web site.
| Method (applied to 568 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Inv. Intensity Chrom. Space | 9.7o | 6.0o | 6.7o | 61.3o | |
| Bayesian | 6.7o | 4.7o | 5.0o | 39.4o | |
| Bottom-up (see here) | 6.6o | 4.9o | 5.2o | 36.5o | |
| Top-down (see here) | 6.7o | 4.7o | 5.1o | 43.8o | |
| Bottom-up+Top-down (see here) | 6.4o | 4.5o | 5.0o | 33.6o | |
| Using Natural Image Statistics | 6.1o | 4.5o | 4.9o | 36.8o | |
| Weighted Grey-Edge (p=2, ? = 1, ? = 20) |
6.6o | 4.7o | 5.1o | 44.3o |
Color-checker (by Shi)
Since the original color checker set wat generated from RAW data using automatic settings, Shi reprocessed the original RAW data and generated 12-bit PNG-images (with lossless compression). Using these new images (and a new ground truth), the following results are obtained for the low-level methods:
| Method (applied to 568 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Grey-World | 6.4o | 6.3o | 6.3o | 24.8o | |
| White-Patch | 7.6o | 5.7o | 6.4o | 40.6o | |
| Shades-of-Grey (p = 4) | 4.9o | 4.0o | 4.2o | 22.4o |  |
| General Grey-World (p = 9, ? = 9) | 4.7o | 3.5o | 3.8o | 22.0o |  |
| 1st-order Grey-Edge (p = 1, ? = 6) | 5.3o | 4.5o | 4.7o | 26.4o |  |
| 2nd-order Grey-Edge (p = 1, ? = 1) | 5.1o | 4.4o | 4.6o | 23.9o |  |
Using the same three-fold cross-validation as before, we obtain the following results for the Gamut mapping:
| Method (applied to 568 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Pixel-based Gamut (? = 4) | 4.2o | 2.3o | 2.9o | 23.2o |  |
| Edge-based Gamut (? = 4, edge type = 3) | 6.5o | 5.0o | 5.4o | 29.0o |  |
| Intersection-based Gamut (? = 4) | 4.2o | 2.3o | 2.9o | 24.2o |  |
Various other methods have been applied to this set. Please note that any algorithm that requires learning is trained using the same three folds as were used for the Gamut mapping. If you would like to have your results reported here, feel free to send a message to the contact person of this web site.
| Method (applied to 568 images) | Mean error | Median error | Trimean error | Max error | |
|---|---|---|---|---|---|
| Inv. Intensity Chrom. Space | 13.6o | 13.6o | 13.5o | 56.7o | |
| Regression (SVR) | 8.1o | 6.7o | 7.2o | 32.0o | |
| Bayesian | 4.8o | 3.5o | 3.9o | 24.5o | |
| Spatial Correlations: — Gaussian-based |
4.0o | 3.1o | 3.3o | 19.8o | |
| — HeavyTailed-based | 3.6o | 3.0o | 3.0o | 21.6o | |
| Bottom-up (see here) | 3.4o | 2.6o | 2.7o | 20.6o | |
| Top-down (see here) | 3.8o | 2.6o | 2.8o | 25.2o | |
| Bottom-up+Top-down (see here) | 3.5o | 2.5o | 2.6o | 25.2o | |
| Using Natural Image Statistics | 4.2o | 3.1o | 3.5o | 26.2o | |
| CART-based Selection | 4.5o | 3.9o | 3.3o | 22.3o | |
| CART-based Combination | 3.9o | 2.9o | 3.3o | 22.3o | |
| Exemplar-Based Color Constancy | 2.9o | 2.3o | 2.4o | 19.4o | |
| Using CNNs: — AlexNet+SVR |
4.7o | 3.1o | 3.5o | 29.2o | |
| — Deep color constancy | 2.6o | 2.0o | 2.1o | 14.8o |
Notes
– The results of the Spatial Correlations denoted “Gaussian-based” are computed by Gijsenij et al. using an earlier implementation of the Spatial Correlations method. The results of Spatial Correlations denoted “HeavyTailed-based” is using the newest implementation and can be found in “Color Constancy with Spatio-spectral Statistics”, in IEEE TPAMI. More elaborate results by Chakrabarti et al. can be downloaded here. These results also contain slightly different results for the Grey-Edge-type methods.
– The results of the CART-based Selection and Combination methods are delivered by Bianco et al. They also computed results for the reprocessed Color-checker set using an alternative ground truth, these results are specified here. This alternative ground truth and the results can be downloaded here.
ColorChecker (RECommended)
The table below was updated on March 24, 2018.
We re-processed the Gehler data to address the problem raised in G. D. Finlayson, G. Hemrit, A. Gijsenij, and P. Gehler, “A Curious Problem with Using the Colour Checker Dataset for Illuminant Estimation,” in Color and Imaging Conference, 2017, pp. 64–69.
If you use the data and results below then please cite G. Hemrit et al., “Rehabilitating the ColorChecker Dataset for Illuminant Estimation,” in Color and Imaging Conference, 2018. see paper here
Finlayson et al. demonstrated that the previous rankings and performance evaluation of algorithms for the ColorChecker dataset are ill-founded, because of mixing different ground-truths. The following results are obtained using the RECommended ground-truth, calculated on the re-processed images from Gheler’s raw images, according to the calculation methodology described by Shi and Funt.
Information on the re-processed dataset as well as the images, are accessible on the Datasets page. If you have any contribution to bring to this work, feel free to send a message to the contact person of this web-site.
| Method (applied to 568 images) | Mean error | Median error | Trimean error | 95% quantile error | Max error | |
|---|---|---|---|---|---|---|
| Grey-World | 9.7o | 10.0o | 10.0o | 14.8o | 24.8o | |
| White-Patch | 9.1o | 6.7o | 7.8o | 21.8o | 43.0o | |
| Shades-of-Grey (p = 4) | 7.3o | 6.8o | 6.9o | 14.4o | 22.5o | |
| General Grey-World (p = 9, ? = 9) | 6.6o | 5.9o | 6.1o | 14.0o | 23.0o | |
| 1st-order Grey-Edge (p = 1, ? = 6) | 4.0o | 3.1o | 3.3o | 10.8o | 20.6o | |
| 2nd-order Grey-Edge (p = 1, ? = 1) | 4.4o | 3.6o | 3.8o | 10.2o | 16.9o | |
| Pixel-based Gamut (? = 4) | 6.0o | 4.4o | 4.9o | 15.9o | 25.3o | |
| Edge-based Gamut (? = 4, edge type = 3) | 5.5o | 3.3o | 3.9o | 17.3o | 29.8o | |
| Intersection-based Gamut (? = 4) | 6.0o | 4.4o | 4.9o | 15.9o | 26.3o |
Please note that any algorithm that requires learning is trained using the same data as for the ColorChecker by Shi. The results will be updated soon, with the algorithms trained on the new data.
| Method (applied to 568 images) | Mean error | Median error | Trimean error | 95% quantile error | Max error | |
|---|---|---|---|---|---|---|
| Inv. Intensity Chrom. Space | 14.6o | 11.6o | 12.4o | 34.3o | 46.9o | |
| Regression (SVR) | 11.0o | 9.6o | 10.1o | 21.1o | 32.5o | |
| Bayesian | 5.4o | 3.8o | 4.3o | 14.8o | 25.5o | |
| Spatial Correlations: — HeavyTailed-based | 5.7o | 4.8o | 5.1o | 13.2o | 18.3o | |
| Bottom-up | 5.6o | 4.9o | 5.1o | 12.5o | 17.9o | |
| Top-down | 6.0o | 4.6o | 5.0o | 15.4o | 25.2o | |
| Bottom-up+Top-down | 5.6o | 4.5o | 4.8o | 13.4o | 25.2o | |
| Using Natural Image Statistics | 5.6o | 4.7o | 4.9o | 13.7o | 30.6o | |
| CART-based Selection | 6.1o | 5.1o | 5.3o | 14.5o | 24.7o | |
| CART-based Combination | 6.0o | 5.5o | 5.7o | 12.0o | 17.7o | |
| Exemplar-Based Color Constancy | 4.9o | 4.4o | 4.6o | 9.4o | 14.5o | |
| Using CNNs: — AlexNet+SVR |
7.0o | 5.3o | 5.7o | 17.7o | 29.1o | |
| — Deep color constancy | 4.6o | 3.9o | 4.2o | 9.0o | 14.8o | |
| Fast Fourier Color Constancy (model Q) | 2.0o | 1.1o | 1.4o | 6.9o | 25.0o |