Have you already reached out to the people who were involved in SHIC about this idea? I’d guess they’d have interesting insights, and maybe also the ability to help set some things in motion (e.g., by reaching out to teachers they know).
I’ll give a grab-bag of other thoughts on this in a few separate comments.
Epistemic status: Some of these thoughts draw on my year doing psychology research, two years teaching, and/or ~7 months so far doing longtermist EA research. So that’s more than nothing, but less than expertise.
Vague skepticism
I haven’t read any of the studies you cite, but I find myself quite skeptical that there’d really be effects that large, that durable, and across that many key domains. Or maybe there’d be such effects, but not when the approaches are replicated by people who aren’t as passionate and expert in the approaches as the people involved in the initial studies. (The latter patterns seems to occur sometimes with psychotherapy studies, for example.)
I base this skepticism not on analysis of these studies in particular, but on things like:
My general impressions of education research
including my impression there seem to be many “fads” that get amazing hype and even impressive research results, and many of these conflict
(E.g., I seem to recall that some research teams repeatedly show that project based learning is clearly better than more traditional methods, and some repeatedly show the opposite. I’m almost certain I’ve seen that sort of thing, at least.)
My expectation that, if these interventions do have substantial and durable effects on things as widely prized as reading ability and logical reasoning, I’d have heard about them before seeing this post (as a result of my Masters or my independent study at that time)
This is just like a skeptical prior, so I wouldn’t want anyone to take it too seriously or not bother looking into the studies as a result. Also, even if the real effects are far less impressive than those studies suggest, this could still be an area worth investigating and perhaps prioritising. I just felt like this skepticism was worth noting.
Hi Michael, thanks for your skepticism of the empirical findings which is helpful. Khorton makes a similar point to you so I think that it is something worth taking seriously. As I mention in my epistemic status note I don’t have any experience in education and so can really benefit from insight from those that do and comments on the general state of educational research (e.g. that it struggles to come to firm conclusions on things) are useful.
In response to your surprise at not having heard of this evidence before, I don’t actually think the evidence I have cited is that incredible. For example the Trickey and Topping (2004) review finds ‘moderate’ effects, not large. The Trickey and Topping follow-up which finds evidence of long-term benefits does sound very impressive, but ultimately that is just one study and I take on board the comment that it might be that such a preliminary study may involve the very best, most highly-motivated teachers. Also, as I mention in the post, the research on moral development is slightly sparse and I am looking forward to seeing further evidence in this area.
There are a few relevant quotes from Trickey and Topping (2004) (which is pretty well-cited with 379 citations FYI). Perhaps I could have included some of these in the post:
“Many of the studies could be criticized on grounds of methodological rigour, but the quality and quantity of evidence nevertheless bears favourable comparison with that on many other methods in education.”
“It is not possible to assert that any use of the P4C process will always lead to positive outcomes, since implementation integrity may be highly variable. However, a wide range of evidence has been reported suggesting that, given certain conditions, children can gain significantly in measurable terms both academically and socially through this type of interactive process.”
“further investigation is needed of wider generalization within and beyond school, and of longer term maintenance of gains”
Overall these comments seem to imply that the evidence for P4C is highly promising building a strong case for further research. I wouldn’t say that subsequent research has definitively put the question to rest and the comments from Khorton and yourself have led me to uprate the expected value of further academic research in this area. It is worth noting that expansion of philosophical education would make such further empirical research easier, so I still see this as a worthy aim.
Thanks for that extra info—that all makes sense to me :)
And perhaps I interpreted your statements as suggesting stronger and more durable effects (or stronger evidence of that) than you actually intended to suggest.
Though “moderate” effects on “logical reasoning” and “reading ability” still sounds like a fairly big deal (though I say this without having checked precisely what’s meant by any of those three terms, in this context). Although maybe it’s the sort of pattern I and Khorton allude to, where a wide range of interventions will all cause benefits compared to business-as-usual, due to additional attention, passion, etc. when an intervention is being delivered and studied.
In any case, there could indeed be value in:
Looking into the existing evidence more closely and exhaustively (or, if you’ve already done that, additional people doing that)
Gathering additional evidence on this sort of intervention
Perhaps rolling this sort of intervention out even if we get no additional, or additional and less promising, evidence about outcomes such as logical reasoning and reading ability. We might do this for the other potential benefits of this sort of intervention, e.g. for moral circle expansion or EA movement-building.
My wording may have been slightly over-positive, but I agree that the evidence seems good enough to warrant further investigation. I wouldn’t say I have done an exhaustive review by any means.
I agree with your suggestions! I probably need a little bit of time to reflect on feedback but I may reach out to some people (e.g. SHIC folks) to see if there is anyone who is interested in investigating further and who has more specialised expertise in education than I do.
(Just want to note that it’s possible your wording was as positive as I initially thought and that that was appropriate, as I haven’t actually read the studies, so I can’t rule out that the effects really are very good and very well-evidenced.)
Hi Michael, thanks for your very helpful comments! I should have some time to address them tomorrow. By “address” I don’t necessarily mean rebut as I agree with a lot of what you have said.
I haven’t yet reached out to SHIC or anyone else for that matter, other than getting a friend to do a review of the post before I uploaded it. So the idea is pretty nascent and there is certainly much more investigation that could be carried out.
Thanks for this post—I found it clear, interesting, and potentially important.
(I’ve also added two links to this in one of the Crucial questions for longtermists Google docs, under “How valuable are various types of moral advocacy? What are the best actions for that?” and “Value of, and best approaches to, communication and movement-building”)
Have you already reached out to the people who were involved in SHIC about this idea? I’d guess they’d have interesting insights, and maybe also the ability to help set some things in motion (e.g., by reaching out to teachers they know).
I’ll give a grab-bag of other thoughts on this in a few separate comments.
Epistemic status: Some of these thoughts draw on my year doing psychology research, two years teaching, and/or ~7 months so far doing longtermist EA research. So that’s more than nothing, but less than expertise.
Vague skepticism
I haven’t read any of the studies you cite, but I find myself quite skeptical that there’d really be effects that large, that durable, and across that many key domains. Or maybe there’d be such effects, but not when the approaches are replicated by people who aren’t as passionate and expert in the approaches as the people involved in the initial studies. (The latter patterns seems to occur sometimes with psychotherapy studies, for example.)
I base this skepticism not on analysis of these studies in particular, but on things like:
My general impressions of education research
including my impression there seem to be many “fads” that get amazing hype and even impressive research results, and many of these conflict
(E.g., I seem to recall that some research teams repeatedly show that project based learning is clearly better than more traditional methods, and some repeatedly show the opposite. I’m almost certain I’ve seen that sort of thing, at least.)
My expectation that, if these interventions do have substantial and durable effects on things as widely prized as reading ability and logical reasoning, I’d have heard about them before seeing this post (as a result of my Masters or my independent study at that time)
This is just like a skeptical prior, so I wouldn’t want anyone to take it too seriously or not bother looking into the studies as a result. Also, even if the real effects are far less impressive than those studies suggest, this could still be an area worth investigating and perhaps prioritising. I just felt like this skepticism was worth noting.
Hi Michael, thanks for your skepticism of the empirical findings which is helpful. Khorton makes a similar point to you so I think that it is something worth taking seriously. As I mention in my epistemic status note I don’t have any experience in education and so can really benefit from insight from those that do and comments on the general state of educational research (e.g. that it struggles to come to firm conclusions on things) are useful.
In response to your surprise at not having heard of this evidence before, I don’t actually think the evidence I have cited is that incredible. For example the Trickey and Topping (2004) review finds ‘moderate’ effects, not large. The Trickey and Topping follow-up which finds evidence of long-term benefits does sound very impressive, but ultimately that is just one study and I take on board the comment that it might be that such a preliminary study may involve the very best, most highly-motivated teachers. Also, as I mention in the post, the research on moral development is slightly sparse and I am looking forward to seeing further evidence in this area.
There are a few relevant quotes from Trickey and Topping (2004) (which is pretty well-cited with 379 citations FYI). Perhaps I could have included some of these in the post:
“Many of the studies could be criticized on grounds of methodological rigour, but the quality and quantity of evidence nevertheless bears favourable comparison with that on many other methods in education.”
“It is not possible to assert that any use of the P4C process will always lead to positive outcomes, since implementation integrity may be highly variable. However, a wide range of evidence has been reported suggesting that, given certain conditions, children can gain significantly in measurable terms both academically and socially through this type of interactive process.”
“further investigation is needed of wider generalization within and beyond school, and of longer term maintenance of gains”
Overall these comments seem to imply that the evidence for P4C is highly promising building a strong case for further research. I wouldn’t say that subsequent research has definitively put the question to rest and the comments from Khorton and yourself have led me to uprate the expected value of further academic research in this area. It is worth noting that expansion of philosophical education would make such further empirical research easier, so I still see this as a worthy aim.
Thanks for that extra info—that all makes sense to me :)
And perhaps I interpreted your statements as suggesting stronger and more durable effects (or stronger evidence of that) than you actually intended to suggest.
Though “moderate” effects on “logical reasoning” and “reading ability” still sounds like a fairly big deal (though I say this without having checked precisely what’s meant by any of those three terms, in this context). Although maybe it’s the sort of pattern I and Khorton allude to, where a wide range of interventions will all cause benefits compared to business-as-usual, due to additional attention, passion, etc. when an intervention is being delivered and studied.
In any case, there could indeed be value in:
Looking into the existing evidence more closely and exhaustively (or, if you’ve already done that, additional people doing that)
Gathering additional evidence on this sort of intervention
Perhaps rolling this sort of intervention out even if we get no additional, or additional and less promising, evidence about outcomes such as logical reasoning and reading ability. We might do this for the other potential benefits of this sort of intervention, e.g. for moral circle expansion or EA movement-building.
My wording may have been slightly over-positive, but I agree that the evidence seems good enough to warrant further investigation. I wouldn’t say I have done an exhaustive review by any means.
I agree with your suggestions! I probably need a little bit of time to reflect on feedback but I may reach out to some people (e.g. SHIC folks) to see if there is anyone who is interested in investigating further and who has more specialised expertise in education than I do.
(Just want to note that it’s possible your wording was as positive as I initially thought and that that was appropriate, as I haven’t actually read the studies, so I can’t rule out that the effects really are very good and very well-evidenced.)
Hi Michael, thanks for your very helpful comments! I should have some time to address them tomorrow. By “address” I don’t necessarily mean rebut as I agree with a lot of what you have said.
I haven’t yet reached out to SHIC or anyone else for that matter, other than getting a friend to do a review of the post before I uploaded it. So the idea is pretty nascent and there is certainly much more investigation that could be carried out.