Using Meta-Analysis to Inform the Robustness of Research Findings

  • No photo
    Zhen Zhang

    Associate Professor/Dean’s Council of 100 Distinguished Scholar
    Department of Management and Entrepreneurship
    W. P. Carey School of Business
    Arizona State University
    Tempe, AZ 85287
    Phone: 480-965-5560
    Email Zhen
    Zhen’s Website

    Zhen Zhang is associate professor of management and Dean’s Council of 100 Distinguished Scholar in the Department of Management and Entrepreneurship at the W. P. Carey School of Business, Arizona State University. His research focuses on leadership process and leadership development, work teams and groups, biological basis of work behavior, start-ups and entrepreneurship, and advanced research methods. Zhang’s work has appeared in leading management journals including Academy of Management Journal, Personnel Psychology, Journal of Applied Psychology, Organization Science, Organizational Behavior and Human Decision Processes, and the Leadership Quarterly, and has been cited in media outlets such as the Wall Street Journal, New York Times, and the Globe and Mail. He currently serves as an associate editor of Personnel Psychology. Zhang has taught graduate and undergraduate courses of Organizational Behavior, Leadership, and Cross-Cultural Management.

This presentation focuses on examining and interpreting effect size heterogeneity in meta-analysis findings, and methods of incorporating heterogeneity in the two approaches of combining meta-analysis and structural equation modeling. We first review the various measures of effect sizes in organization sciences and how meta-analyses report information about effect size heterogeneity. We then compare and contrast two approaches that test structural models based on meta-analytically derived correlations. Based on recent research (Cheung, in press; Yu et al., 2016), we show how heterogeneity can be considered in these two approaches. Recommendations are provided for researchers to better use the rich information of effect size heterogeneity when they evaluate primary studies and/or conducting meta-analyses.

Digital Reader: Resources Recommended by the Speaker

  • Bergh, D. D., Aguinis, H., Heavey, C., Ketchen, D. J., Boyd, B. K., Su, P., . . . Joo, H. (2016). Using meta-analytic structural equation modeling to advance strategic management research: Guidelines and an empirical illustration via the strategic leadership-performance relationship. Strategic Management Journal, 37,477–497.
  • Carlson, K. D., & Ji, F. X. (2011). Citing and building on meta-analytic findings: A review and recommendations. Organizational Research Methods, 14, 696-717.
  • Cheung, M.W.-L. (2015). Meta-analysis: A structural equation modeling approach. West Sussex, United Kingdom: Wiley.
  • Cheung, M.W.-L. (in press). Issues in solving the problem of effect size heterogeneity in meta-analytic structural equation modeling: A commentary and simulation study on Yu, Downes, Carter, and O’Boyle (2016). Journal of Applied Psychology.
  • Cheung, M.W.-L., & Chan, W. (2005). Meta-analytic structural equation modeling: A two- stage approach. Psychological Methods, 10, 40–64.
  • Cheung, M. W.-L., & Chan, W. (2009). A two-stage approach to synthesizing covariance matrices in meta-analytic structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 16,28–53.
  • DeSimone, J. A., Köhler, T., & Schoen, J. L. (in press). If it were only that easy: The use of meta-analytic research by organizational scholars. Organizational Research Methods.
  • Edwards, J. R., & Christian, M. S. (2014). Using accumulated knowledge to calibrate theoretical propositions. Organizational Psychology Review, 4, 279–291.
  • Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.
  • Landis, R. S. (2013). Successfully combining meta-analysis and structural equation modeling: Recommendations and strategies. Journal of Business and Psychology, 28, 251–261.
  • Oswald, F. L., & Johnson, J. W. (1998). On the robustness, bias, and stability of statistics from meta-analysis of correlation coefficients: Some initial Monte Carlo findings. Journal of Applied Psychology, 83, 164-178.
  • Rosopa, P. J., & Kim, B. (2017). Robustness of statistical inferences using linear models with meta-analytic correlation matrices. Human Resource Management Review, 27, 216–236.
  • Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.
  • Sheng, Z., Kong, W., Cortina, J. M., & Hou, S. (2016). Analyzing matrices of meta-analytic correlations: Current practices and recommendations. Research Synthesis Methods, 7, 187–208.
  • Viswesvaran, C., & Ones, D. S. (1995). Theory testing: Combining psychometric meta- analysis and structural equations modeling. Personnel Psychology, 48, 865–885.
  • Whitener, E. M. (1990). Confusion of confidence intervals and credibility intervals in meta-analysis. Journal of Applied Psychology, 75, 315-321.
  • Yu, J., Downes, P. E., Carter, K. M., & O’Boyle, E. H. (2016). The problem of effect size heterogeneity in meta-analytic structural equation modeling. Journal of Applied Psychology, 101,1457-1473.