Skip to main content
No Access

Guidelines for reporting productivity studies – a review of the reproducibility of data envelopment analysis in the service sector

Published Online:pp 407-425https://doi.org/10.1504/IJSOM.2013.056796

With respect to the scientific method, reproducing empirical research is a necessary step to develop generalisable knowledge. However, in service productivity measurement and managerial science in general, little attention seems to be paid to replicating empirical studies. One reason could be that many studies do not report their data and methodology in the degree of detail sufficient to reproduce them. In this paper, we provide evidence for a lack of reproducibility of empirical studies on service productivity measurement with data envelopment analysis (DEA), based on reviewing service productivity measurement literature. The major impediments identified comprise a superficial description of the measurement process and data sources, insufficient details on the retrieval methodology, and an incomplete account of the extraction methodology. To improve the reproducibility of future studies, this article offers a structured set of guidelines with which authors can report their studies more comprehensively.

Keywords

data envelopment analysis, DEA, service science, reproducibility, ethical guidelines, operations research, scientometrics, philosophy of OR, good practice

References

  • 1. Athanassopoulos, A.D. (1997). ‘Service quality and operating efficiency synergies for management control in the provision of financial services: evidence from Greek bank branches’. European Journal of Operational Research. 98, 2, 300-313 Google Scholar
  • 2. Berthon, P. , et al. (2002). ‘Potential research space in MIS: a framework for envisioning and evaluating research replication, extension, and generation’. Information Systems Research. 13, 4, 416-427 Google Scholar
  • 3. Bridgman, P. (1927). The Logic of Modern Physics. New York:MacMillan Google Scholar
  • 4. Brown, R. (2006). ‘Mismanagement or mismeasurement? Pitfalls and protocols for DEA studies in the financial services sector’. European Journal of Operational Research. 174, 2, 1100-1116 Google Scholar
  • 5. Buchanan, R.D. , Finch, S.J. (2005). History of Psychometrics. New York:John Wiley & Sons Google Scholar
  • 6. Carnine, D. (1997). ‘Bridging the research-to-practice gap’. Exceptional Children. 63, 4, 513-521 Google Scholar
  • 7. Chen, A. (2005). ‘Measurement and sources of overall and input inefficiencies: evidences and implications in hospital services’. European Journal of Operational Research. 161, 2, 447-468 Google Scholar
  • 8. Chen, Y. , et al. (2009). ‘Additive efficiency decomposition in two-stage DEA’. European Journal of Operational Research. 196, 3, 1170-1176 Google Scholar
  • 9. Cherchye, L. , et al. (2010). ‘Efficiency and equity in private and public education: a nonparametric comparison’. European Journal of Operational Research. 202, 2, 563-573 Google Scholar
  • 10. Coelli, T.J.T. , et al. (2005). An Introduction to Efficiency and Productivity Analysis. New York:Springer Google Scholar
  • 11. Cook, W.D. , Seiford, L.M. (2009). ‘Data envelopment analysis (DEA) – thirty years on’. European Journal of Operational Research. 192, 1, 1-17 Google Scholar
  • 12. Dehnokhalaji, A. , et al. (2010). ‘Efficiency analysis to incorporate interval-scale data’. European Journal of Operational Research. 207, 2, 1116-1121 Google Scholar
  • 13. Drekker, D. (2001). ‘A quasi-concave DEA model with an application for bank branch performance evaluation’. European Journal of Operational Research. 132, 2, 296-311 Google Scholar
  • 14. Du, J. , et al. (2011). ‘A bargaining game model for measuring performance of two-stage network structures’. European Journal of Operational Research. 210, 2, 390-397 Google Scholar
  • 15. Dyson, R.G. (2001). ‘Pitfalls and protocols in DEA’. European Journal of Operational Research. 132, 2, 245-259 Google Scholar
  • 16. Ehrenberg, A.S.C. (1990). ‘A hope for the future of statistics: MSOD’. American Statistician. 44, 3, 195-196 Google Scholar
  • 17. Einstein, A. (1908). ‘Über das Relativitätsprinzip und die aus demselben gezogenen Folgerungen’. Jahrbuch der Radioaktivität und Elektronik. 4, 11, 411-462 Google Scholar
  • 18. Emrouznejad, A. , De Witte, K. (2010). ‘COOPER-framework: a unified process for non-parametric projects’. European Journal of Operational Research. 207, 3, 1573-1586 Google Scholar
  • 19. Epure, M. , Kerstens, K. , Prior, D. (2011). ‘Bank productivity and performance groups: a decomposition approach based upon the Luenberger productivity indicator’. European Journal of Operational Research. 211, 3, 630-641 Google Scholar
  • 20. Evanschitzky, H. , et al. (2007). ‘Replication research in marketing revisited: a note on a disturbing trend’. Journal of Business Research. 60, 4, 411-415 Google Scholar
  • 21. Fandel, G. (2007). ‘On the performance of universities in North Rhine-Westphalia: government’s redistribution of funds judged using DEA efficiency measures’. European Journal of Operational Research. 176, 1, 521-533 Google Scholar
  • 22. Galton, F. (1879). ‘Psychometric experiments.’. Brain. 2, 2, 149-162 Google Scholar
  • 23. Gass, S.I. (2009). ‘Ethical guidelines and codes in operations research’. Omega. 37, 6, 1044-1050 Google Scholar
  • 24. Gonzalez-Barahona, J.M. , Robles, G. (2012). ‘On the reproducibility of empirical software engineering studies based on data retrieved from development repositories’. Empirical Software Engineering. 17, 1–2, 75-89 Google Scholar
  • 25. Grace, R.C. (2001). ‘On the failure of operationism’. Theory Psychology. 11, 1, 5-33 Google Scholar
  • 26. Grönroos, C. , Ojasalo, K. (2004). ‘Service productivity – towards a conceptualization of the transformation of inputs into economic results in services’. Journal of Business Research. 57, 4, 414-423 Google Scholar
  • 27. Hubbard, R. , Vetter, D.E. (1996). ‘An empirical comparison of published replication research in accounting, finance, management, and marketing’. Journal of Business Research. 35, 2, 153-164 Google Scholar
  • 28. Inmon, W. (2005). Building the Data Warehouse. New York:Wiley Google Scholar
  • 29. Joo, S.J , Billington, P.J. , Stoeberl, P.A. (2012). ‘Labour management for a restaurant using data envelopment analysis’. International Journal of Services and Operations Management. 11, 1, 1-12 AbstractGoogle Scholar
  • 30. Kane, E.J. (1984). ‘Why journal editors would encourage the replication of applied econometric research’. Quarterly Journal of Business and Economics. 23, 1, 3-8 Google Scholar
  • 31. Knox, L.C.A. , Pastor, J.T. (1997). ‘Target setting: an application to a bank branch network’. European Journal of Operational Research. 98, 2, 290-299 Google Scholar
  • 32. Köksalan, M. , et al. (2010). ‘A flexible approach to ranking with an application to MBA programs’. European Journal of Operational Research. 201, 2, 470-476 Google Scholar
  • 33. Kumar, K.S. , Babu, A.S. (2012). ‘An integrated method using AHP, DEA and GP for evaluating supply sources’. International Journal of Services and Operations Management. 11, 2, 123-150 AbstractGoogle Scholar
  • 34. Lee, A.S. (1989). ‘A scientific methodology for MIS case studies’. MIS Quarterly. 13, 1, 33-50 Google Scholar
  • 35. Madaus, G.F. , O’Dwyer, L.M. (1999). ‘A short history of performance assessment: lessons learned’. Phi Delta Kappan. 80, 9, 688-695 Google Scholar
  • 36. Mahajan, J. (1991). ‘A data envelopment analytic model for assessing the relative efficiency of the selling function’. European Journal of Operational Research. 53, 2, 189-205 Google Scholar
  • 37. Maniadakis, N. , Thanassoulis, E. (2004). ‘A cost Malmquist productivity index’. European Journal of Operational Research. 154, 2, 396-409 Google Scholar
  • 38. McCullough, B.D. (2009). ‘Open access economics journals and the market for reproducible economic research’. Economic Analysis and Policy. 39, 1, 117-126 Google Scholar
  • 39. Mesirov, J.P. (2010). ‘Accessible reproducible research’. Science. 327, 5964, 415-416 Google Scholar
  • 40. Mezias, S.J. , Regnier, M.O. (2007). ‘Walking the walk as well as talking the talk: replication and the normal science paradigm in strategic management research’. Strat. Organ.. 5, 3, 283-296 Google Scholar
  • 41. Michell, J. (1997). ‘Quantitative science and the definition of measurement in psychology’. British Journal of Psychology. 88, 3, 355-383 Google Scholar
  • 42. Min, H. , et al. (2008). ‘A data envelopment analysis for establishing the financial benchmark of Korean hotels’. International Journal of Services and Operations Management. 4, 2, 201-217 AbstractGoogle Scholar
  • 43. Mu, L. (2008). ‘Modelling ETL processes of data warehouses with UML activity diagrams’. Proceedings of the OTM. 44-53 Google Scholar
  • 44. Nachum, L. (1999). ‘The productivity of intangible factors of production: some measurement issues applied to Swedish management consulting firms’. Journal of Service Research. 2, 2, 123-137 Google Scholar
  • 45. Pastor, J.M. , Perez, F. , Quesada, J. (1997). ‘Efficiency analysis in banking firms?: an international comparison’. European Journal of Operational Research. 98, 96, 395-407 Google Scholar
  • 46. Peng, R.D. (2011). ‘Reproducible research in computational science’. Science. 334, 6060, 1226-1227 Google Scholar
  • 47. Piwowar, H.A. (2011). ‘Who shares? Who doesn’t? Factors associated with openly archiving raw research data’. PLoS ONE. 6, 7, 1-13 Google Scholar
  • 48. Pulina, M. , Detotto, C. , Paba, A. (2010). ‘An investigation into the relationship between size and efficiency of the Italian hospitality sector: a window DEA approach’. European Journal of Operational Research. 204, 3, 613-620 Google Scholar
  • 49. Ray, S. , Jeon, Y.J. (2008). ‘Reputation and efficiency: a non-parametric assessment of America’s top-rated MBA programs’. European Journal of Operational Research. 189, 1, 245-268 Google Scholar
  • 50. Revilla, E. , Sarkis, J. , Modrego, A. (2007). ‘Exploring public and private R&D partnership performance: a knowledge-based view of inter-organisational alliances’. International Journal of Services and Operations Management. 3, 4, 371-393 AbstractGoogle Scholar
  • 51. Ruggiero, J. (1996). ‘On the measurement of technical efficiency in the public sector’. European Journal of Operational Research. 90, 3, 553-565 Google Scholar
  • 52. Sarmiento, R. , et al. (2009). ‘An exploratory study on contextual variables and manufacturing efficiency’. International Journal of Services and Operations Management. 5, 3, 275-298 AbstractGoogle Scholar
  • 53. Schrader, U. , Hennig-Thurau, T. (2009). ‘VHB-JOURQUAL2: method, results, and implications of the German Academic Association for Business Research’s journal ranking’. BuR – Business Research. 2, 2, 180-204 Google Scholar
  • 54. Stodden, V. (2010). ‘The scientific method in practice: reproducibility in the computational sciences’. MIT Sloan Research Paper No. 4773-10. 1-33 Google Scholar
  • 55. Trujillo, J. , Luj, S. (2003). ‘A UML based approach for modeling ETL processes in data warehouses’. Proceedings of the ER. Chicago, 307-320 Google Scholar
  • 56. United States Census Bureau (2012). North American Industry Classification System 2007. (accessed 1 October 2012), [online] http://www.census.gov/eos/www/naics/ Google Scholar
  • 57. Vassiliadis, P. (2001). ‘Arktos: towards the modeling, design, control and execution of ETL processes’. Information Systems Journal. 26, 8, 537-561 Google Scholar
  • 58. Vassiliadis, P. , et al. (2005). ‘A generic and customizable framework for the design of ETL scenarios’. Information Systems Journal. 30, 7, 492-525 Google Scholar
  • 59. Vuorinen, I. , Järvinen, R. , Lehtinen, U. (1998). ‘Content and measurement of productivity in the service sector: a conceptual analysis with an illustrative case from the insurance business’. SIM. 9, 4, 377-396 Google Scholar
  • 60. Zeithaml, V.A. , Parasuraman, A. , Berry, L.L. (1985). ‘Problems and strategies in services marketing’. J. Marketing. 49, 2, 33-46 Google Scholar

Additional References