Research and scientific reasoning
Treatment outcome research and evidence appraisal
Strategies for testing treatments, statistical versus clinical significance, meta-analysis, empirically supported treatments and how to appraise a study.
What this module draws on:
- Textbook concept Outcome research strategies, challenges, statistical versus clinical significance, meta-analysis and empirically supported treatments (Ch. 4).
- Contemporary update Reliable change, evidence-based practice policy and modern appraisal concepts.
- Original elaboration Appraisal checklist and fictional study critique.
Learning objectives
After this module you should be able to:
- Describe the treatment package, dismantling, constructive, parametric, comparative, client–therapist variation and process strategies.
- Distinguish statistical significance from clinical significance.
- Explain what meta-analysis contributes and where it can mislead.
- Appraise a treatment study using a structured checklist.
- Explain the empirically supported treatments movement and its critics.
Introduction
Asking whether a therapy works turns out to involve many questions: Compared with what? For whom? On which outcome? For how long? And does a statistically reliable difference make a meaningful difference to someone's life?
Seven research strategies Textbook concept
| Strategy | Question it answers |
|---|---|
| Treatment package | Does the treatment as a whole work compared with no treatment or a control? |
| Dismantling | Which components are necessary? (Remove parts and compare.) |
| Constructive | Does adding a component improve outcomes? |
| Parametric | How much, how often or how long? (Vary dose or intensity.) |
| Comparative | Which of two or more treatments works better? |
| Client–therapist variation | For which clients, with which therapists, does it work best? |
| Process | What happens during therapy that leads to change? |
Challenges in outcome research Textbook concept
- Research treatments are often manualised and delivered by highly trained therapists — are they like everyday practice?
- Research participants may be screened to exclude comorbidity — are they like typical clients?
- Withholding treatment from a control group raises ethical issues; waiting lists and treatment-as-usual comparisons are common compromises.
- Outcomes can be measured by clients, clinicians or others, and at different times; results may differ.
- Different studies reach different conclusions, so reviews and meta-analyses are needed.
Statistical versus clinical significance Textbook concept
A statistically significant result is unlikely to be due to chance alone. With a large sample, a tiny difference can be statistically significant yet unimportant. Clinical significance asks whether change matters in practice — for example, whether clients move from the dysfunctional range into the functional range.
A widely used contemporary approach (Jacobson & Truax, 1991) combines a reliable change index — was the change bigger than measurement error? — with a criterion for moving into a functional range.
Meta-analysis and empirically supported treatments Textbook concept
Meta-analysis statistically combines results across studies, usually as effect sizes. Smith and Glass's 1977 meta-analysis was an early landmark suggesting psychotherapy is generally effective. Meta-analyses are only as good as the studies included: publication bias, poor-quality studies and mixing very different treatments ('apples and oranges') can mislead.
In the 1990s, criteria were proposed for labelling treatments empirically supported (for example, Chambless & Hollon, 1998). Supporters argued this protects the public; critics argued it privileges manualised, brief, easily tested therapies and neglects the therapeutic relationship and clinical judgement.
A practical appraisal checklist Original elaboration
- Question: Is the population, intervention, comparison and outcome clear?
- Allocation: Was assignment random and concealed?
- Blinding: Were outcome assessors unaware of group membership?
- Attrition: How many dropped out, and were all randomised participants analysed?
- Measures: Were outcomes reliable, valid and relevant to clients?
- Size of effect: Is the difference clinically meaningful, not just statistically significant?
- Fit: Do the participants, setting and culture resemble the people you work with?
- Conflicts: Did the developers of the treatment run the study?
Worked example: The impressive p-value Original, fictional
A fictional trial of 4,000 students finds that a smartphone app reduces stress scores by 0.8 points on a 40-point scale, p < .001. Statistically the result is very reliable. Clinically, a change of less than one point may be smaller than the measure's error and may not change anyone's daily life. Both kinds of significance must be reported and judged.
Interactive activity
Critique a fictional study
Read the fictional abstract, answer each question, then compare with the model answer.
1 Abstract: 'Thirty clients chosen by their therapists received our new Integrative Resilience Therapy. Ninety per cent improved. The developers conclude that IRT is effective for all anxiety disorders.' What is the most important design weakness?
Compare with a model answer
There is no control or comparison group, so improvement could reflect maturation, regression to the mean, expectancy or other non-specific factors. Selection by therapists also introduces bias.
2 Why is the claim 'effective for all anxiety disorders' unjustified?
Compare with a model answer
The sample is small, unspecified and not representative; generalising across all anxiety disorders far exceeds the data (external validity).
3 Who conducted the study, and why might that matter?
Compare with a model answer
The developers ran it. Allegiance effects and conflicts of interest are associated with more favourable results; independent replication is needed.
4 Suggest a better design.
Compare with a model answer
A randomised controlled trial comparing IRT with an established treatment and an attention control, with a defined diagnostic sample, blinded assessors, intention-to-treat analysis, reliable and valid outcome measures, follow-up, and reporting of clinically significant change.
Knowledge check
5 questions. Every option has an explanation. Your answers are not sent anywhere when JavaScript is available; without it, they are checked by the server and immediately discarded.
Summary
- Different research strategies answer different questions about a treatment.
- Statistical significance is not the same as clinical significance.
- Meta-analysis is powerful but vulnerable to biased or heterogeneous inputs.
- The empirically supported treatments movement and its critics shaped today's broader evidence-based practice.
- Appraising a study means checking allocation, blinding, attrition, measures, effect size, fit and conflicts.
Questions for reflection
- When research evidence and a client's preferences point in different directions, how should a clinician decide?
Key terms
Further reading
- Textbook Plante, T. G. (2005). Contemporary clinical psychology (2nd ed.). John Wiley & Sons.
- Cited in textbook Kazdin, A. E. (1994). Methodology, design, and evaluation in psychotherapy research. In A. E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behavior change (4th ed.). Wiley.
- Cited in textbook Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752–760.
- Cited in textbook Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7–18.
- Contemporary Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19.
- Contemporary APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285.