Skip to main content
No Access

Improving a dynamic ensemble selection method based on oracle information

Published Online:pp 184-200https://doi.org/10.1504/IJICA.2012.050053

This work evaluates some strategies to approximate the performance of a dynamic ensemble selection method to the oracle performance of its pool of weak classifiers. For this purpose, we evaluated different distance metrics in the K-nearest-oracles (KNORA) method, the use of statistics related to the class accuracy of each classifier in the pool and some additional information calculated by using a clustering process in the validation dataset. Moreover, different strategies are also evaluated to combine the results of the KNORA dynamic ensemble selection method with the results of its built-in K-nearest neighbour (KNN) used to define the neighbourhood of a test pattern during the ensemble creation. A strong experimental protocol based on more than 60,000 samples of handwriting digits extracted from NIST-SD19 was used to evaluate each strategy. The experiments have shown that the fusion of the KNORA results with the results of its built-in KNN is a very promising strategy.

Keywords

dynamic ensemble selection, oracle, K-nearest-oracles, KNORA

References

  • 1. Brown, G. , Wyatt, J. , Harris, R. , Yao, X. (2005). ‘Diversity creation methods: a survey and categorisation’. International Journal of Information Fusion. 6, 1, 5-20 Google Scholar
  • 2. Cao, J. , Ahmadi, M. , Shridhar, M. (1995). ‘Recognition of handwritten numerals with multiple feature and multistage classifier’. Pattern Recognition. 28, 2, 153-160 Google Scholar
  • 3. Didaci, L. , Giacinto, G. (2004). ‘Dynamic classifier selection by adaptative k-nearest-neighbourhood rule’. International Workshop on Multiple Classifier Systems (MCS 2004), 174-183 Google Scholar
  • 4. Didaci, L. , Giacinto, G. , Roli, F. , Marcialis, G.L. (2005). ‘A study on the performances of dynamic classifier selection based on local accuracy estimation’. Pattern Recognition. 38, 11, 2188-2191 Google Scholar
  • 5. Giacinto, G. , Roli, F. (1999). ‘Methods for dynamic classifier selection’. International Conference on Image Analysis and Processing (ICIAP 1999), 659-664 Google Scholar
  • 6. Hall, M. , Frank, E. , Holmes, G. , Pfahringer, B. , Reutemann, P. , Witten, I.H. (2009). ‘The WEKA data mining software: an update’. SIGKDD Explorations. 11, 1, 10-18 Google Scholar
  • 7. Hartigan, J.A. (1975). Clustering Algorithms. New York:John Wiley & Sons Google Scholar
  • 8. Huang, Y.S. , Suen, C.Y. (1995). ‘A method of combining multiple experts for the recognition of unconstrained handwritten numerals’. IEEE Transactions on Pattern Analysis and Machine Intelligence. 17, 1, 90-93 Google Scholar
  • 9. Jain, A.K. , Murthy, M.N. , Flynn, P.J. (1999). ‘Data clustering: a review’. ACM Computing Reviews. 31, 3, 265-323 Google Scholar
  • 10. Kittler, J. , Hatef, M. , Duin, R. , Matas, J. (1998). ‘On combining classifiers’. IEEE Transactions on Pattern Analysis and Machine Intelligence. 20, 3, 226-239 Google Scholar
  • 11. Ko, A.H.R. (2007). ‘Static and dynamic selection of ensemble of classifiers’. Canada:École de Technologie Supérieure, Université Du Québec , PhD thesis 246 pp Google Scholar
  • 12. Ko, A.H.R. , Sabourin, R. , Britto, A.S., Jr. (2008). ‘From dynamic classifier selection to dynamic ensemble selection’. Pattern Recognition. 41, 5, 1718-1731 Google Scholar
  • 13. Kuncheva, L.I. (2004). Combining Pattern Classifiers. Methods and Algorithms. Hoboken, NJ:Wiley-Interscience Google Scholar
  • 14. Kuncheva, L.I. , Rodrígues, J.J. (2007). ‘Classifier ensembles with a random linear oracle’. IEEE Transactions on Knowledge and Data Engineering. 19, 4, 500-508 Google Scholar
  • 15. Kuncheva, L.I. , Whitaker, C.J. (2003). ‘Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy’. Machine Learning. 51, 2, 181-207 Google Scholar
  • 16. MacQueen, J.B. , Cam, L.M.L. Neyman, J. (1967). ‘Some methods for classification and analysis of multivariate observations’. Proc. of the Fifth Berkeley Symposium on Mathematical Statistics and Probability. 1, 281-297 Google Scholar
  • 17. Nicoletti, M.C. , Bertini, J.R., Jr. (2007). ‘An empirical evaluation of constructive neural network algorithms in classification tasks’. International Journal of Innovative Computing and Applications. 1, 1, 2-13 AbstractGoogle Scholar
  • 18. Opitz, D. , Maclin, R. (1999). ‘Popular ensemble methods: an empirical study’. Journal of Artificial Intelligence Research. 11, 169-198 Google Scholar
  • 19. Pekalska, E. , Skurichina, M. , Duin, R.P.W. (2004). ‘Combining dissimilarity-based one-class classifiers’. International Workshop on Multiple Classifier Systems (MCS 2004), 122-133 Google Scholar
  • 20. Ranawana, R. (2006). ‘Multi-classifier system – review and a roadmap for developers’. International Journal of Hybrid Intelligent Systems. 3, 2, 35-61 Google Scholar
  • 21. Santana, A. , Soares, R.G.F. , Canuto, A.M.P. , Souto, M.C.P. (2006). ‘A dynamic classifier selection method to build ensembles using accuracy and diversity’. in Proceedings of the Ninth Brazilian Symposium on Neural Networks (SBRN’06). 36-41 Google Scholar
  • 22. Santos, E.M.D. , Sabourin, R. , Maupin, P. (2008). ‘A dynamic overproduce-and-choose strategy for the selection of classifier ensembles’. Pattern Recognition. 41, 10, 2993-3009 Google Scholar
  • 23. Silva, L.A. , Del-Moral-Hernandez, E. , Moreno, R.A. , Furuie, S.S. (2009). ‘Cluster-based classification using self-organising maps for medical image databases’. International Journal of Innovative Computing and Applications. 2, 1, 13-22 AbstractGoogle Scholar
  • 24. Webb, G.I. , Zheng, Z. (2004). ‘Multistrategy ensemble learning: reducing error by combining ensemble learning techniques’. IEEE Transactions on Knowledge and Data Engineering. 16, 8, 980-991 Google Scholar
  • 25. Woods, K. , Kegelmeyer, W.P., Jr. , Bowyer, K. (1997). ‘Combination of multiple classifiers using local accuracy estimates’. IEEE Transactions of Pattern Analysis and Machine Intelligence. 19, 4, 405-410 Google Scholar
  • 26. Xiao, J. , He, C. (2009). ‘Dynamic classifier ensemble selection based on GMDH’. 1, 2009 International Joint Conference on Computational Sciences and Optimization, 731-734 Google Scholar
  • 27. Zouari, H. , Heutte, L. , Lecourtier, Y. , Alimi, A. (2004). ‘Building diverse classifier outputs to evaluate the behavior of combination methods: the case of two classifiers’. International Workshop on Multiple Classifier Systems (MCS 2004), 273-282 Google Scholar