Have you already reached out to the people who were involved in SHIC about this idea? Iād guess theyād have interesting insights, and maybe also the ability to help set some things in motion (e.g., by reaching out to teachers they know).
Iāll give a grab-bag of other thoughts on this in a few separate comments.
Epistemic status: Some of these thoughts draw on my year doing psychology research, two years teaching, and/āor ~7 months so far doing longtermist EA research. So thatās more than nothing, but less than expertise.
Vague skepticism
I havenāt read any of the studies you cite, but I find myself quite skeptical that thereād really be effects that large, that durable, and across that many key domains. Or maybe thereād be such effects, but not when the approaches are replicated by people who arenāt as passionate and expert in the approaches as the people involved in the initial studies. (The latter patterns seems to occur sometimes with psychotherapy studies, for example.)
I base this skepticism not on analysis of these studies in particular, but on things like:
My general impressions of education research
including my impression there seem to be many āfadsā that get amazing hype and even impressive research results, and many of these conflict
(E.g., I seem to recall that some research teams repeatedly show that project based learning is clearly better than more traditional methods, and some repeatedly show the opposite. Iām almost certain Iāve seen that sort of thing, at least.)
My expectation that, if these interventions do have substantial and durable effects on things as widely prized as reading ability and logical reasoning, Iād have heard about them before seeing this post (as a result of my Masters or my independent study at that time)
This is just like a skeptical prior, so I wouldnāt want anyone to take it too seriously or not bother looking into the studies as a result. Also, even if the real effects are far less impressive than those studies suggest, this could still be an area worth investigating and perhaps prioritising. I just felt like this skepticism was worth noting.
Hi Michael, thanks for your skepticism of the empirical findings which is helpful. Khorton makes a similar point to you so I think that it is something worth taking seriously. As I mention in my epistemic status note I donāt have any experience in education and so can really benefit from insight from those that do and comments on the general state of educational research (e.g. that it struggles to come to firm conclusions on things) are useful.
In response to your surprise at not having heard of this evidence before, I donāt actually think the evidence I have cited is that incredible. For example the Trickey and Topping (2004) review finds āmoderateā effects, not large. The Trickey and Topping follow-up which finds evidence of long-term benefits does sound very impressive, but ultimately that is just one study and I take on board the comment that it might be that such a preliminary study may involve the very best, most highly-motivated teachers. Also, as I mention in the post, the research on moral development is slightly sparse and I am looking forward to seeing further evidence in this area.
There are a few relevant quotes from Trickey and Topping (2004) (which is pretty well-cited with 379 citations FYI). Perhaps I could have included some of these in the post:
āMany of the studies could be criticized on grounds of methodological rigour, but the quality and quantity of evidence nevertheless bears favourable comparison with that on many other methods in education.ā
āIt is not possible to assert that any use of the P4C process will always lead to positive outcomes, since implementation integrity may be highly variable. However, a wide range of evidence has been reported suggesting that, given certain conditions, children can gain significantly in measurable terms both academically and socially through this type of interactive process.ā
āfurther investigation is needed of wider generalization within and beyond school, and of longer term maintenance of gainsā
Overall these comments seem to imply that the evidence for P4C is highly promising building a strong case for further research. I wouldnāt say that subsequent research has definitively put the question to rest and the comments from Khorton and yourself have led me to uprate the expected value of further academic research in this area. It is worth noting that expansion of philosophical education would make such further empirical research easier, so I still see this as a worthy aim.
Thanks for that extra infoāthat all makes sense to me :)
And perhaps I interpreted your statements as suggesting stronger and more durable effects (or stronger evidence of that) than you actually intended to suggest.
Though āmoderateā effects on ālogical reasoningā and āreading abilityā still sounds like a fairly big deal (though I say this without having checked precisely whatās meant by any of those three terms, in this context). Although maybe itās the sort of pattern I and Khorton allude to, where a wide range of interventions will all cause benefits compared to business-as-usual, due to additional attention, passion, etc. when an intervention is being delivered and studied.
In any case, there could indeed be value in:
Looking into the existing evidence more closely and exhaustively (or, if youāve already done that, additional people doing that)
Gathering additional evidence on this sort of intervention
Perhaps rolling this sort of intervention out even if we get no additional, or additional and less promising, evidence about outcomes such as logical reasoning and reading ability. We might do this for the other potential benefits of this sort of intervention, e.g. for moral circle expansion or EA movement-building.
My wording may have been slightly over-positive, but I agree that the evidence seems good enough to warrant further investigation. I wouldnāt say I have done an exhaustive review by any means.
I agree with your suggestions! I probably need a little bit of time to reflect on feedback but I may reach out to some people (e.g. SHIC folks) to see if there is anyone who is interested in investigating further and who has more specialised expertise in education than I do.
(Just want to note that itās possible your wording was as positive as I initially thought and that that was appropriate, as I havenāt actually read the studies, so I canāt rule out that the effects really are very good and very well-evidenced.)
Hi Michael, thanks for your very helpful comments! I should have some time to address them tomorrow. By āaddressā I donāt necessarily mean rebut as I agree with a lot of what you have said.
I havenāt yet reached out to SHIC or anyone else for that matter, other than getting a friend to do a review of the post before I uploaded it. So the idea is pretty nascent and there is certainly much more investigation that could be carried out.
Thanks for this postāI found it clear, interesting, and potentially important.
(Iāve also added two links to this in one of the Crucial questions for longtermists Google docs, under āHow valuable are various types of moral advocacy? What are the best actions for that?ā and āValue of, and best approaches to, communication and movement-buildingā)
Have you already reached out to the people who were involved in SHIC about this idea? Iād guess theyād have interesting insights, and maybe also the ability to help set some things in motion (e.g., by reaching out to teachers they know).
Iāll give a grab-bag of other thoughts on this in a few separate comments.
Epistemic status: Some of these thoughts draw on my year doing psychology research, two years teaching, and/āor ~7 months so far doing longtermist EA research. So thatās more than nothing, but less than expertise.
Vague skepticism
I havenāt read any of the studies you cite, but I find myself quite skeptical that thereād really be effects that large, that durable, and across that many key domains. Or maybe thereād be such effects, but not when the approaches are replicated by people who arenāt as passionate and expert in the approaches as the people involved in the initial studies. (The latter patterns seems to occur sometimes with psychotherapy studies, for example.)
I base this skepticism not on analysis of these studies in particular, but on things like:
My general impressions of education research
including my impression there seem to be many āfadsā that get amazing hype and even impressive research results, and many of these conflict
(E.g., I seem to recall that some research teams repeatedly show that project based learning is clearly better than more traditional methods, and some repeatedly show the opposite. Iām almost certain Iāve seen that sort of thing, at least.)
My expectation that, if these interventions do have substantial and durable effects on things as widely prized as reading ability and logical reasoning, Iād have heard about them before seeing this post (as a result of my Masters or my independent study at that time)
This is just like a skeptical prior, so I wouldnāt want anyone to take it too seriously or not bother looking into the studies as a result. Also, even if the real effects are far less impressive than those studies suggest, this could still be an area worth investigating and perhaps prioritising. I just felt like this skepticism was worth noting.
Hi Michael, thanks for your skepticism of the empirical findings which is helpful. Khorton makes a similar point to you so I think that it is something worth taking seriously. As I mention in my epistemic status note I donāt have any experience in education and so can really benefit from insight from those that do and comments on the general state of educational research (e.g. that it struggles to come to firm conclusions on things) are useful.
In response to your surprise at not having heard of this evidence before, I donāt actually think the evidence I have cited is that incredible. For example the Trickey and Topping (2004) review finds āmoderateā effects, not large. The Trickey and Topping follow-up which finds evidence of long-term benefits does sound very impressive, but ultimately that is just one study and I take on board the comment that it might be that such a preliminary study may involve the very best, most highly-motivated teachers. Also, as I mention in the post, the research on moral development is slightly sparse and I am looking forward to seeing further evidence in this area.
There are a few relevant quotes from Trickey and Topping (2004) (which is pretty well-cited with 379 citations FYI). Perhaps I could have included some of these in the post:
āMany of the studies could be criticized on grounds of methodological rigour, but the quality and quantity of evidence nevertheless bears favourable comparison with that on many other methods in education.ā
āIt is not possible to assert that any use of the P4C process will always lead to positive outcomes, since implementation integrity may be highly variable. However, a wide range of evidence has been reported suggesting that, given certain conditions, children can gain significantly in measurable terms both academically and socially through this type of interactive process.ā
āfurther investigation is needed of wider generalization within and beyond school, and of longer term maintenance of gainsā
Overall these comments seem to imply that the evidence for P4C is highly promising building a strong case for further research. I wouldnāt say that subsequent research has definitively put the question to rest and the comments from Khorton and yourself have led me to uprate the expected value of further academic research in this area. It is worth noting that expansion of philosophical education would make such further empirical research easier, so I still see this as a worthy aim.
Thanks for that extra infoāthat all makes sense to me :)
And perhaps I interpreted your statements as suggesting stronger and more durable effects (or stronger evidence of that) than you actually intended to suggest.
Though āmoderateā effects on ālogical reasoningā and āreading abilityā still sounds like a fairly big deal (though I say this without having checked precisely whatās meant by any of those three terms, in this context). Although maybe itās the sort of pattern I and Khorton allude to, where a wide range of interventions will all cause benefits compared to business-as-usual, due to additional attention, passion, etc. when an intervention is being delivered and studied.
In any case, there could indeed be value in:
Looking into the existing evidence more closely and exhaustively (or, if youāve already done that, additional people doing that)
Gathering additional evidence on this sort of intervention
Perhaps rolling this sort of intervention out even if we get no additional, or additional and less promising, evidence about outcomes such as logical reasoning and reading ability. We might do this for the other potential benefits of this sort of intervention, e.g. for moral circle expansion or EA movement-building.
My wording may have been slightly over-positive, but I agree that the evidence seems good enough to warrant further investigation. I wouldnāt say I have done an exhaustive review by any means.
I agree with your suggestions! I probably need a little bit of time to reflect on feedback but I may reach out to some people (e.g. SHIC folks) to see if there is anyone who is interested in investigating further and who has more specialised expertise in education than I do.
(Just want to note that itās possible your wording was as positive as I initially thought and that that was appropriate, as I havenāt actually read the studies, so I canāt rule out that the effects really are very good and very well-evidenced.)
Hi Michael, thanks for your very helpful comments! I should have some time to address them tomorrow. By āaddressā I donāt necessarily mean rebut as I agree with a lot of what you have said.
I havenāt yet reached out to SHIC or anyone else for that matter, other than getting a friend to do a review of the post before I uploaded it. So the idea is pretty nascent and there is certainly much more investigation that could be carried out.