Evaluation of Anglican Schools’ Partnership: Teachers’ Use of Feedback

feedbackIntroduction

This project is a one-year pilot trial of encouraging teachers to use research evidence on effective feedback in their practice. The intention is to use the lessons learnt from this pilot to work towards a full test of effectiveness in future years. The evaluation proposal outlined below is for the pilot only, but taking into account its formative nature. It is a largely formative process evaluation, intending also to provide an estimated effect size for the intervention that could be used in future scaled up trials. The proposed intervention is interesting in being only partly defined at the outset and shaped in practice as it goes along. The intervention is therefore a template for practice at this stage.

Treatment

The Anglican Schools Partnership feedback initiative is a one-year pilot trial of encouraging teachers to use research evidence on effective feedback in their practice. The intention is to use the lessons learnt from this pilot to work towards a full test of effectiveness in future years

Sampling and recruitment

The evaluation will be conducted with a total of 9 primary schools and 1 secondary school, involving all pupils over one academic year – 2012/13. The schools are those forming the Anglican Schools Partnership in Bexley. All schools have agreed to take part in the study as full partners. The resultant sample is not likely to be easily generalisable to a wider population.

A further 10 schools from the same local authority will be matched on available measures of school organisation and intake. These will be used to provide context and pre and post-test data as a comparator group not receiving the intervention. This will assist the before and after data in estimating the likely effect size.

This means that around 3,000 Primary and Secondary pupils will receive the intervention, and a further 2,800 Primary pupils are available to act as partially matched comparator. However, it is unlikely that all year groups will contribute to the estimated effect size (see below). The proposal cites an effect size of at least 0.74. This is likely to be much less in practice, and the results from rolling out research findings about feedback into practice are not always as expected. Nevertheless, this is a substantial scale for a pilot. The inclusion of only one secondary may lead to complications for analysis, and may have to be treated separately.

Allocation to groups

Allocation has already been determined for this pilot. All of the schools in the Anglican Schools Partnership are taking part, and no others. This was how the intervention was designed when it was funded initially.

Design

The design is a before and after study with a convenience sample of 10 schools and a partly matched comparator group of 10 schools. The longitudinal design follows entire cohorts through one year of schooling, intervening, monitoring and adjusting the interventions from January to July 2013.

Outcomes will be compared with previous cohorts, what was expected in terms of progress with these cohorts, and with progress of cohorts in other schools not involved in the study. There is also a comparison between the results and progress of disadvantaged pupils (FSM or eligible for pupil premium) and the rest.

Prior measures

All of the prior background and contextual data is generated automatically by schools. No further data collection is necessary. Most of the prior background and contextual data will come from the individual pupil NPD records for pupils in both phases. This will include KS1 results (levels and points), sex, month of birth, FSM status, SEN status, ethnicity, and first language.

In addition, it would be useful to have individual attendance records, date of leaving (if during the project), and any suspensions or exclusions (where applicable). These would come from existing school records.

Outcome measures

The primary outcome measure for all pupils will be the APS in the appropriate Key Stage for all relevant year cohorts (such as year 6), and progress from the starting point assessments for all other cohorts. These would be divided into analyses covering FSM-eligible and not eligible pupils, and other groups defined by available measures of disadvantage. None of these measures or variables is additional to those collected routinely or as part of the proposed intervention. They are appropriate and standardised (as far as is possible).

Longer term, all pupils will have Key Stage results that can be compared with previous cohorts, and/or with the comparators schools. But this is beyond the scope of this project.

Analysis of outcome measures

The evaluation will compute the standardised difference between:

  • the mean APS of the treatment schools and the matched comparator group
  • the mean APS of the treatment cohorts and the prior cohorts in the same schools
  • the mean contextualised progress from starting point assessments of the treatment schools and the matched comparator group
  • the mean contextualised progress from starting point assessments of the treatment cohorts and the prior cohorts in the same schools.

And these same scores broken down into disadvantaged pupils and the rest, such as:

  • the mean APS of FSM and non-FSM pupils in the treatment schools
  • the mean APS of FSM and non-FSM pupils in the matched comparator group etc.

Differences that are robust enough to appear under all of these conditions will be considered substantial. Differences will be presented as raw-score and in standardised form such as Cohen’s d effect size.

Every attempt must be made to get complete test scores for all pupils even where they are initially absent or where they leave the schools during the trial. Where dropout, turnover or exchange between schools occurs, the results will be analysed both in terms of the original schools (intention to treat) and in terms of the eventual groups. Differences will be calculated for the post-test scores alone, and for the gain scores from pre-test to post-test. And differences will be analysed in terms of pupil prior attainment where it exists (at KS1) and background (from NPD).

However, it must be recalled that this is primarily a formative evaluation, and that these calculations are to provide an estimated effect size for a trial, and to rehearse and pilot the data requirements.

Process evaluation and fidelity to treatment

This forms the bulk of the fieldwork, with the aim of providing formative evidence on all phases and aspects of the template intervention from cascading the training to evaluating the outcomes. Uniquely for this research, the evidence can be used as part of the earlier action research cycles as appropriate and as it becomes available. For this reason also we propose an interim evaluation of the progress of a sub-set of pupils. In addition, the evaluation will assist in improving the template for a later trial, and in deciding whether the design or action research approach is useful in such circumstances. This will necessitate the generation of additional data from observation and interviews with staff and researchers, and via focus groups and a brief survey with pupils. These will all be as simple and integrated as possible. A substantial part of the evaluation fieldwork will be conducted with the aims of assessing how closely schools adhere to the intended intervention, and what the short term or intermediate impacts are (such as changes in classroom interaction). In co-operation with the teachers and trainers, it will address issues such as the reaction to training

  • the quality of training
  • the fidelity of training in cascade
  • whether the teams understand the process and purpose
  • the contents and use of the starter pack
  • starting point and subsequent assessments
  • how missing data is handled
  • changes in teacher behaviour
  • how they take control of their own ‘feedback’ loop in improving evidence-informed practice
  • the nature and use of portfolios
  • audits by classes of teacher feedback, and learner effectiveness
  • the ongoing ‘engineering’ of a pack and web resources
  • whether teachers can tell if the template is working, or can modify it accordingly
  • whether there appears to be an impact on how children are learning
  • whether teachers are providing useful and  better feedback
  • and whether pupils are responding.

The process evaluation will provide some formative evidence on all phases and aspects of the intervention from the selection and retention of schools, through the training of teachers to evaluating the outcomes. It will involve the perceptions of participants including any resentment or resistance, and lead to advice on improvements and issues for subsequent scaling up.

The evaluators will make about 20 to 24 person trips to the research sites. This will necessitate the generation of some additional data from observation and interviews with staff, pupils, and parents as well – also observation of training, delivery and testing. These will all be as simple and integrated as possible.

For project website click here 
Results

The project has completed. Education Endowment Foundation has published final reports on 7th February 2014. The reports are available here.