Evidence! Relevant? (Ioannidis, Slavin)
29 March, 2017
Statement: It becomes increasingly more difficult to come across evidence about educational leadership that is relevant for practice.
I will deal in this blog with this issue from several perspectives: Critics on science in general (Ionannidis); growing recognition of the relevance of context (Hallinger); broadening the meaning of evidence (USA); promise of longitudinal, multi-level research (Hallinger and Heck); trend towards self-improving schools and principals supporting each other (UK); reaching to big data (World Bank); celebrating self-evidence (the principal is the second most important factor -next to the teacher- in influencing student learning, Fullan); courageous affirmation of failure (Mulford); research reporting findings are about leadership old-school, not about leadership new-school.
Sorry, just kidding. I will do so but not all of it in this blog. I will stretch the topic of evidence over a (discontinuous) range of blogs. At times I will interject other topics.
I will limit this blog to the work of John Ioannidis (in his fundamental comments on the organisation of science nowadays), and to recent comments of Robert Slavin (in the context of the Every Student Succeeds Act (ESSA)).
November last year I had the opportunity to listen to John Ioannidis in the ‘Concertgebouw’ in Amsterdam. In his lecture 'Reliable Science’ the renowned Greek-American epidemiologist and internist addressed both the accomplishments and the shortcomings of biomedical research.
'Structural shortcomings are among other things: researchers often work in isolation and experience pressure to obtain significant results. They conduct their studies with samples that are too small, selectively report hypotheses that are confirmed, and conduct post-hoc analyses, using lenient statistical criteria. Pre-registration and replication are rare, and data sharing is unpopular. In the current literature, almost all scientific articles report statistically significant findings, and that is simply unrealistic. In science, we need other incentives than obtaining statistical significance'. (https://nrin.nl/archieven/1638)
If that happens in biomedical research it will probably also happen in our field of study. Our researchers work with the same kind of incentives. They also will suffer under the ‘file drawer problem’, the tendency to publish results that are novel, statistically significant or supportive of a particular hypothesis, while not publishing other valid but less interesting results. With as a consequence that published literature indicates stronger evidence for findings than exists in reality. (see: https://med.stanford.edu/news/all-news/2017/01/researchers-announce-master-plan-for-better-science.html)
It is not much discussed, but in our field of interest in the recent past it was better to propose a study on e.g. distributed leadership than on other topics. Donors would pay for your up-to-date research proposal and your results would be read. And given the branching out of studies on this subject many were able to discover new aspects. (And for the effects of that see my blog: Distributed Leadership: State of Art).
Since the problem indeed is not only in biomedics but of a more general nature, proposals have therefore been developed to fundamentally change the organisation of scientific work.
A major proposal is: Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021 (2017).
Ioannidis is one of the authors. The article presents proposals for changing methods, reporting and dissemination, reproducibility, evaluation and incentives.
Hopefully it will work out in the long term. Robert Slavin reports about a good start, but...
ESSA and Slavin
The Every Student Succeeds Act was signed by President Obama on December 10, 2015. This bipartisan measure reauthorizes the 50-year-old Elementary and Secondary Education Act (ESEA), the nation’s national education law and longstanding commitment to equal opportunity for all students. The previous version of the law, the No Child Left Behind (NCLB) Act, was enacted in 2002. (See https://www.ed.gov/essa?src=rn for the provisions of the law). Evidence-based innovation is a central element in the law.
Robert Slavin is Director of the Center for Research and Reform in Education (CRRE) at Johns Hopkins University School of Education.
The CRRE is a research center with as major goal to improve the quality of education for children in grades pre-K to 12 through high-quality research and evaluation studies, and the dissemination of evidence-based research.
Slavin in fact is the personification of the importance of using evidence in education.
Recently he published a blog in the Huffington Post of 28 febr. 2017: Evidence for ESSA Goes Live!
(See http://www.huffingtonpost.com/author/robertslavinphd-814 for all his blogs)
'Evidence for ESSA' is a free website (http://www.evidenceforessa.org/) providing easy access to information on programs meeting the evidence standards of the Every Student Succeeds Act.
The evidence presented is about reading and math programmes. Relevant knowledge for teachers and principals, and one might expect easier to reveal than evidence about educational leadership. However even for reading and math programmes this is not the case. In his blog of 19 jan. 2017 Slavin writes about the rapid advance of rigorous research. In the course of his work on 'Evidence for ESSA', he has noticed some interesting trends, with 'truly revolutionary implications'. It however all boils down to enthusiasm about many more studies that meet the very stringent standards CRRE uses (which indeed in itself is good), but also concluding that in large-scale randomized experiments, effect sizes are modest (the mean effect size for large, randomized experiments across all of elementary and second reading, math, and science is only +0.13, much smaller than effect sizes from smaller studies and from quasi-experiments).
So here is what I fear: With better research it will be possible to find some evidence on the effects of (several aspects of) educational leadership but so weak that that conclusions are hardly relevant. At least not for translating results into guidelines for practice of principals.
In next blogs I will deliberate on other reasons for growing difficulty to find evidence that is relevant for practice, using the perspectives I mentioned in the outset of this blog.