Evaluation of Switch On: Tailored Literacy


bookstackIntroduction

The project to be evaluated is a randomised controlled trial of an intervention called Switch On which has adapted a long-standing New Zealand intervention called Reading Recovery for use with Year 7. The intervention involves daily short individual sessions for 12 weeks. Here Switch On is being offered to Year 7 pupils eligible for the pupil premium who do not achieve Level 4 English at Key Stage 2 (KS2). It will take place for one group in Spring term 2012, and for a waiting-list group in Summer term 2012. There is some evidence of effectiveness but not yet in England with this age group. Therefore a trial is appropriate.

 Impact evaluation

 Design

The outline for the intervention proposes a relatively simple one-term waiting list design. All schools will receive the Switch On intervention, and all will have previously agreed for half of their relevant pupils to be randomised to immediate intervention or the intervention after one term.

 Sample size

The project involves randomisation of around 380 individual Year 7 pupils, within 19 schools, to one of two groups. Approximately 190 pupils will receive Switch On from January 2013, and 190 pupils one term later. A pseudo-random number generator will select the treatment or waiting group for each pupil, after the pre-test has been conducted with all cases.

The project outline suggests an effect size (d) of 1.3 for Reading Recovery (no data is available for Switch On itself). The What Works ClearingHouse (2008) suggests a medium to large effect size. Using Lehr’s approximation for an 80% chance of detecting a presumed effect size of 1.0 with 5% alpha, the minimum sample size required per arm is only 16 individual cases (Gorard 2013). Although this study involves individual randomisation within schools (and no randomisation of schools) 190 cases per arm should give this trial power.

Tests

The pre-test scores for both groups will be the GL New Group Reading Test A, administered individually. The test will be administered by the schools themselves conjunction with the project team. Because this will take place before randomisation, the process will be ‘blind’ as to treatment group. In addition, the evaluation will have the KS2 assessment results in literacy as a secondary pre-test score.

The post-test scores for both groups will be the GL New Group Reading Test B, administered individually. This test will be administered by schools in the presence of members of the evaluation team and their temporary employees (such as doctoral researchers) who will not know which arm each individual is in. Schools will be instructed not to disclose the fact to the evaluators. This is to help ensure that the process is ‘blind’ as to treatment group. In addition, we will have the most up-to-date teachers assessment results in literacy as a secondary post-test score.

Ideally, the test will be conducted on-line as far as possible to assist standard timing and marking.

Other data

The intervention team have been sent a template for data to be uploaded for all relevant pupils at the outset of the trial. The template includes prior attainment plus background characteristics such as FSM, sex and ethnicity.

Analysis

The primary outcome measure will be the difference in the gain score between the arms of the trial, expressed as an effect size, where the gain is the average difference between individual scores on tests A and B. A secondary outcome measure will be the average residuals between the actual scores on test B and the predicted (modelled) scores based on prior KS2 literacy assessment and pupil background characteristics.

Process evaluation

The fieldwork forming the light-touch process evaluation has the aim of providing some formative evidence on all aspects of the intervention from the selection and retention of schools, through the training of teachers to evaluating the outcomes. This can be used to help assess fidelity to treatment, and the perceptions of participants including any resentment or resistance, and to advise on improvements and issues for any future scaling up.

This will all necessitate the generation of some additional data from observation and interviews with staff, focus groups of pupils, plus observation of training, delivery and testing. These will all be as simple and integrated and non-intrusive as possible. The schedule of visits will be agree with the intervention team and the schools. Schools will agree to be part of the evaluation when agreeing to be part of the intervention.

 

Results

 

The project has completed. The report has been published by Education Endowment Foundation on 7th February 2014.

 

References

Gorard, S. (2013) Research Design: Robust approaches for the social sciences, London:Sage

What Works ClearingHouse (2008) Reading Recovery, http://ies.ed.gov/ncee/wwc/InterventionReport.aspx?sid=420