Printer-friendly version

Effective implementation of Jigsaw in a community is complex, multi-faceted, and challenging. It requires simultaneous and systematic efforts conducted by a range of participants to accomplish ambitious, agreed goals. Headstrong is committed to rigorous evaluation of programme implementation and outcomes to ensure that local efforts are working, and invests considerable effort in helping communities to assess change.

The evaluation provides a method by which Headstrong and the sites can see what works, what is not working, and what must be adapted or altered.

In general, the evaluation focuses on questions such as:

• Which young people with what needs are being served?
• How are services being delivered?
• What short-term, medium-range, and long-term outcomes are attained?
• Is the programme being implemented with fidelity?

The Jigsaw evaluation is being led by Prof. Bob Illback, Deputy CEO of Headstrong.

Methodology

The methods employed for the evaluation system are both qualitative and quantitative in nature. In general, the evaluation gathers information about change for young people in five inter-connected systems:

1. Young people who receive individualised and direct mental health services and supports from the Jigsaw programme
2. Community providers and staff who work with young people and receive support from Jigsaw
3. Organisations and agencies that provide services and supports to young people
4. Young people in specific Jigsaw areas
5. The community as a whole

The Jigsaw evaluation encompasses three broad data-gathering strategies:

1. Needs assessment
2. Implementation assessment
3. Outcome assessment

Components of the Evaluation Plan

• Needs Assessment:

Each evaluation begins with an assessment of the community context, in which evaluators meet with various constituencies and stakeholders to learn about the community’s characteristics, history, needs, and strengths.

This process leads to a systematic and comprehensive assessment of community needs and resources which then leads to the design of logic models, comprehensive community goals and ultimately lays the foundations for an implementation plan. The Needs and Resource Assessment also serves as a baseline assessment of a community’s system of care and support, and is repeated bi-annually as a means of assessing change.

• Implementation Assessment:

This component measures important drivers of change, as opposed to merely accounting for the occurrence or non-occurrence of planned activities. It seeks to measure  8 domains of programme functioning deemed critical for the long-term success of Jigsaw. Interviews are conducted with key programme personnel every 6 months and coded using the Jigsaw Implementation Fidelity Scale (JIFS), yielding a profile of programme functioning that shows how the programme develops over time.

• Outcome Assessment 

 Jigsaw seeks to insure that young people receive accessible, individualised, and effective services and supports when and where they need them. To gather standardised information about service delivery, a comprehensive online data management (ODM) system has been deployed. This enables continuous measurement and tracking of programme activities, processes and outcomes. The primary tool for collation, synthesis, and graphic portrayal of the data is termed Jigsaw Analytics. Data generated by the ODM provides a basis for describing who is served, what services are delivered, and what outcomes are associated with service delivery.

The evaluation plan also incorporates a measure of inter-organisational collaboration and social network analysis (Jigsaw Service Network Survey). Administered annually, this provides a means to track the community’s progress toward developing better linkages, coordination, and collaboration between organisations that serve young people.

Jigsaw’s training evaluation focuses on questions such as the extent to which training is effective, whether what is learned is applied and supported within host organisations, and the effect its application has on young people. Training descriptions and discussions with trainers provide a framework for discerning training goals. Structured interviews of a representative sample of Jigsaw trainees and their supervisors are conducted annually throughout the course of the project to ascertain changes in how providers think about and conduct their work.

Other evaluation methods include:

• client satisfaction measure to determine what reactions service recipients have to the services they receive
• follow-up telephone interviews of service recipients
• data mining on referral patterns, engagement and service delivery characteristics
• social indicator analysis of health, mental health, education and related local data
• qualitative assessment of community engagement and discourse over time