I CAN - Does ‘hard to reach’ mean ‘hard to evaluate’?

With a Government commitment to halving the word gap between children with poor language and their peers, there can be little doubt that language levels for children starting school is a major problem. Add that to the fact that, at 22% of pupils with SEND, speech, language and communication needs (SLCN) is now the most common primary type of need, as well as the most common type of primary need for pupils on SEN support (23% of pupils with SEND).

Language difficulties are developmental in nature. The warning signs for SLCN are well known, as is the need for early identification and intervention. There are also strong genetic and environmental risk factors: children living in areas of disadvantage are 2.3 times more likely to have poor language. Even if children attend early years education, they spend most of their time at home, and as such parents have a key role in supporting children’s early language, and in noticing these warning signs.

During 2017, I CAN worked closely with parent groups to co-produce a programme for parents of two-year-old children at risk of delayed language development, living in areas of deprivation. We listened to parents who told us what they thought would work, but also drew heavily on evidence of how children develop language through the contingent talk their parents use. The result was Tots Talking, and drawing on that advice from parents, it’s NOT a ‘programme’, ‘group sessions’ or ‘workshops’. It’s a time for parents to get together and chat about their child’s talking.

Over eight weeks, a trained facilitator meets with parents and shares information – through films and activities, they make toys and encourage discussion. Tots Talking aims to shift parents’ behaviour so that they talk with their children, focusing on the things children are interested in (contingent talk). We haven’t had 100% attendance; life gets in the way.  But there is good engagement. Feedback from both parents and practitioners has been overwhelmingly positive. But does it make a difference to children? And does it change the way parents interact with their children? All I CAN programmes and interventions are supported with evidence of impact and we were keen to have this for Tots Talking.

This was, for us, a very different evaluation. The parents attending loved Tots Talking, but still had difficulty attending all the sessions. What would this mean for its evaluation? We drew heavily on learning from others about engaging vulnerable and disadvantaged parents, but knew we would learn more ourselves. And we did!

Getting the design right

We worked closely with professor Nikki Botting from City University to plan a rigorous and yet doable design. We used a waiting control design across two geographic regions. Evaluation assistants (mainly speech and language therapy students), blind to condition, carried out assessments.

We thought long and hard about which measures to use, always keeping a focus on what we wanted to measure. Children’s language is very variable at age 2, so we needed to be as accurate as possible. We aimed for a balance: avoiding wasting parents’ time but with enough data to triangulate findings:

• the communication development inventory (CDI) - a standardised parent-report measure of children’s early vocabulary.
• a newly developed dynamic assessment of understanding: the City Assessment of Receptive Language Instrument (CARLI).
• The I CAN engagement outcome measure – a six-point scale which practitioners completed on children.
• we videoed parents playing with their children and analysed using the Parenting Interactions with Children checklist of Observations linked to outcomes (PICCOLO).
• online surveys and focus groups for both practitioners and parents.
• a short quiz for parents about what talking to expect from two-year-olds.
• the I CAN staff journey measure:  A seven-point scale to capture changes in staff knowledge and practice.

With parents involved in five of those measures, we knew we were asking a lot, but careful planning made it work. Here are our top tips:

• Being super organised was key - we planned sessions where parents took part in a range of activities, but where there was time and space to chill with a coffee.
• Using venues where parents were used to going, so a quiet room in the nursery. Involving setting practitioners so that parents saw a familiar face.
• Seeing parents individually and offering appointment times so that they weren’t hanging around or felt judged by others.
• Providing clear and accessible information so there were no surprises.
• Supporting parents with activities that required literacy skills.
• Being non-judgemental; We kept the activities fun, easy and upbeat. Exhausting but successful!
• Where possible we used dynamic assessments or ones where there is no obvious ceiling so that parents (and children) didn’t feel they had failed. 
• And yes, there were incentives. Our funding meant we could offer parents a voucher to thank them for their time.

We took measures at three time points, and now have a limited number of complete datasets. Some parents only managed one or two time points but this wasn’t because they didn’t want to come, it was because families have other – often unexpected - priorities.

We’re pleased with the data we have and are still analysing it, with support from Nikki Botting. Results are promising…..watch this space for more information.

For more information about Tots Talking, please contact Louisa Reeves, Project Lead.