Routine Evaluation of the Effectiveness of a University Counselling Service

Steve Potter - University of Manchester and UMIST

Introduction

In this presentation I am going to indicate why we put such store by routine evaluation in our service, the method of evaluation we use and some of the key results we have so far obtained. I want then to dwell upon some of the benefits to service delivery that arise from routine evaluation through the process of an audit cycle leading to improvements in our practice. I shall briefly highlight the impact on supervisory thinking, the idea and practice of practitioner led research and the changes in emphasis in our overall construction of the role of a counselling service in HE that has been helped by the cycle of evaluative research and audt. As a parting shot I will make a plea for funding for a practitioner led research network more generally in student counselling and briefly say what I do and don’t mean by this in terms of a minimum necessary set of data.

Background

We have been making steps towards the routine, practitioner-led evaluation of our work at the University of Manchester and UMIST Counselling Service since 1998, using the evolving resources of the Clinical Outcomes in Routine Evaluation system (CORE). CORE is being widely adopted across the full range of psychologically therapeutic services for mental health in the UK[1]. Here we report on our most recent and continuous data set of 679 clients gathered from November 2001 to August 2002 using the CORE PC[2] software. For comparison we also refer to our earlier CORE data set of 1100 gathered separately from 1997 until 2001[3] and make comparisons with CORE NHS-wide benchmarking data[4] - available to the CORE-PC Systems Benchmarking Club.

Why evaluate on a routine basis?

Routine evaluation allows comparisons. Where poor outcome is indicated we can follow it up. Where good outcome is indicated we can endorse it and copy it. Counselling and psychotherapy are the subject of a wider variety of theories and techniques and there is a growing evidence base supporting the choice of effective practice for particular needs. Counselling in a front line setting such as a student counselling service is vulnerable to either treating everything or responding to lightly and blandly or with too narrow a touch. Routine evaluation will help to create a open framework for thinking about our work. Routine evaluation can place us firmly within best practice within the filed of psychological therapies, help us challenge our methods of work, give greater value to it and understand it more deeply and clearly. It can inform our efforts at feedback and to direct resources. Counsellors should be driven by curiosity, collaboration and openness. Routine evaluation allows us to have an empirical stance which is an exact copy of the stance in the counselling relationship.

The method: using CORE

Given this argument for using a minimum data set for any counselling or psychologically therapeutic work we adopted CORE as it seemed the system with growing NHS support particular in primary care counselling and with the most credibility. The CORE system consists of three elements: a client completed questionnaire, a practitioner completed case profile, and (optional) CORE PC software offering ease of data input, management, enquiry and reporting. The three need to be used in combination for an effective process of service evaluation and development. There are, of course, two absolutely crucial additional human elements that are required to make CORE an effective tool - a computer literate administrator and a team of practitioners animated by curiosity, honesty and collaboration.

The most essential component of the CORE system is a series of 34 descriptive statements (e.g. “My problems have been impossible to put to one side”) which the client scores on a five point scale rating the frequency of their occurrence over the last week. The scale ranges from 'not at all' to 'all of the time' and the acccumulation of scores offers an indicator of emotional and psychological distress, personal functioning, problem severity and risk of harm. It is a well-validated measure, freely available and widely used. Severity scores on each dimension are easily calculated but above all, the single sheet two-sided questionnaire has a common sense feel about it, such that each item can be quickly scanned to get a sense of the person seeking help. Despite counsellors initial misgivings about doing something more ‘clinical’ to our clients, the process of form filling has not been experienced as intrusive and for many is a containing and helpful pre-therapy activity by increasing psychological thoughtfulness and focus.

The results summarised

Routine evaluation of our counselling service since 1997 (using the CORE system) has provided evidence that university counselling provision is:

· meeting a substantial mental health need (82% of our clients have pre counselling severity of functioning, distress and problem scores above the clinical cut off for patients being treated by NHS mental health services)

· responsive (53% seen for first appointment within a day, 93% within a week of approaching the service)

· effective overall (75% reliable improvement at the end of counselling)

· efficient (89% of planned sessions attended).

· effective for a wide range of problems (anxiety, depression, relationship difficulties, academic problems, self esteem, loss)

· appropriately placed in its institutional context and meeting needs not met elsewhere (the service achieves its aim of providing an easy access, low stigma, rapid ‘psychologically therapeutic’ response to those facing severe distress, whose usual forms of coping have failed and whose work and self esteem is suffering).

· approved of (it rates highly for student satisfaction and most are satisfied with frequency, number of sessions and the focus of work).

· flexibility of provision for example at the least 39% percent of sessions on a less than once a week basis indicating resourceful use of spacing of sessions in the face of pressure of demand.

· adaptive in offering a variety of approaches though mainly an integrative approach drawing extensively on cognitive analytic therapy.

· capable of empirically derived improvement by using a process of continuous, detailed audit and identifying strengths and weaknesses in service provision.

Data Attrition

In weighing the validity of the data it must be noted that the data sets, whilst substantial, are not the full data in any one year. The most recent set of data of 692 (over the past ten months) covers well over half of the clients seeking help. It includes all of the data of a substantial number of counsellors. From the whole data set only 40% offer a complete set of pre and post measures at present although this figure should improve as more end of counselling data comes in. This figure compares reasonably well with national benchmarks of 45%. We consider the data gathered to be a true mix of the cases we see with the exception of those who attend for only one session.

The benefits

Establishing a particular client profile.

For those seeking help and completing the CORE measure pre counselling the highest-scoring questions pre-counselling on average are:

Question (where minimum score is 0 and highest score is 4)	Average score
27. I have felt unhappy	2.914
2. I have felt tense, anxious or nervous	2.899
20. My problems have been impossible to put to one side	2.741
14. I have felt like crying	2.693
5. I have felt totally lacking in energy and enthusiasm	2.611
30. I have thought I am to blame for my problems and difficulties	2.535

These are the upsetting and disabling thoughts, feelings and behaviours that prompt clients to seek help. They have shaped our service goals of a prompt and ‘perspective restoring’ intervention to prevent prolonged emotional distress affecting achievement of the student’s academic or personal development goals. Whilst many students may sort problems out by themselves, with the passage of time, the high-pressured atmosphere of academic achievement and student life puts time at a premium.

For those having had counselling in the service the highest scoring change items are as illustrated in the table.

Question (where minimum score is 0 and highest score is 4)	Change
20. My problems have been impossible to put to one side	-1.702
17. I have felt overwhelmed by my problems	-1.645
14. I have felt like crying	-1.635
27. I have felt unhappy	-1.582
2. I have felt tense, anxious or nervous	-1.498
23. I have felt despairing or hopeless	-1.409

Using the CORE PC system it is possible to obtain a listing of each response to the 34 items ranked hierarchically for the degree that they have been experienced by the client in the last week. Following counselling it is possible to obtain a similar hierarchical list of the 34 items organised by the degree to which they have improved. This allows highly specific exploration and is a useful aid to supervision. Individual reports for each client, or clusters of client problems, are possible. For example when someone has generally improved in well-being, but in relation to others may feel more agitated as indicated by worsening scores on one or two functioning items, the data may be used in supervision or in collaboration with the client at follow up.

Is our work efficient?

A key marker of efficiency is the ability to offer a rapid response for anyone with such high levels of distress that it is damaging their ability to work and to the quality of their student experience. We see 73% of clients within a day of seeking help and 93% within a week. This is achieved by routinely blocking off first assessment sessions, providing a daily drop-in slot staffed by two counsellors, and filling cancelled slots with newcomers where appropriate or possible. A rapid response is our highest team priority and is affirmed as such by the outcome data scores. Resourcing a rapid response is enormously cost-effective from a clinical and educational point of view.

Is our work more than a response to the worried well?

The % of all clients with valid pre-therapy scores who scored above clinical cut-off (the benchmarked score for those accessing treatment for mental health problems in the NHS)

83 %

Counselling is a substantial contributor to student mental health: 83% of those seeking help with us present with CORE scores higher than the clinical cut-off for NHS patients seeking help for mental ill health. The percentage of clients above the cut off varies with presenting problem. For clients presenting with depression the score is 90% above the NHS cut off. For eating disorders, which as a symptomatic disorder may function to mask emotional distress, the percentage is lower with 71 % above the NHS clinical cut off. These scores for high levels of distress indicate severe impairment of functioning and, as many clients indicate they have been suffering for some time, we have no evidence that such difficulties will be spontaneously resolved. Far from it: we calculate that continuing to meet the demands of academic work and other student pressures whilst coping with this level of distress is a high risk factor for further or more chronic patterns of mental ill health and disruption to studies.

Problem frequency classified by counsellor as presenting moderate to severe difficulty for the client (frequencies from a data set of 692)

It is useful to see that practitioner completed rating of problems resonates with the mix of impaired functioning and emotional distress on the client-completed CORE scores as it affirms our assessment skills.

Risk is an important item for any frontline counselling service. There are six risk items indicating risk of harm to self or others, suicidal thoughts and plans on the CORE outcome form. Whilst students seeking help with us present with lower risk scores (though not lower distress scores) than the clinical sample, they are higher than the general population sample and wherever there is an indication of a risk it is vitally important that it is discussed. The very existence of the CORE form prompts active risk management and is valued as such by the team. The CORE outcome measure risk scores have helped increase our confidence with risk assessment and tracking.

Our earlier data set showed that risk and general well-being/distress scores improved significantly for those indicating themselves to be at risk and rated as at risk by the counsellor. However for the small number indicating severe risk the improvement was relative. We have looked at the three cases where the risk scores remain high at the end of counselling to identify actions taken by the counsellor and referral routes. It is the setting up of this kind of client tracking as a routine detailed evaluation, which improves not only our individual work but also our collective attitude of openness, curiosity and enquiry into our work.

In the most recent data set, there are 90 people (15%) with a mild to moderate suicide risk score and 4 with a severe risk score. In addition there are 125 (21%) mild to moderate risk of harm to self scores. There are only 19 with a mild to moderate risk of harm to others. These scores offer some indication of the severity of distress clients bring to the service and the importance of including risk assessment and management in the counselling process.

Is our work effective?

From the most recent set of 692 cases gathered over the past 10 months using CORE PC in-house we have obtained evidence of reliable change for the better following the brief intensive counselling help typical in student counselling.

Post-Outcome Measure - Change - Clinical and Reliable Change

The phrase “Finding better ways to cope” has become the central banner of one of our publicity sheets. Much of our work focuses on helping clients finding better ways to cope. From the CORE PC data clients report negative ways of coping as avoidance, alcohol, bottling up feelings, badly handled anger, postponing, overemotional fighting or conflict, withdrawal and hiding. In contrast, clients develop positive ways of coping, referred to repeatedly in the data as opening up, talking to others, or facing up to things. This kind of data can be analysed in relation to our theoretical assumptions and practices. For example, the coping data points to a typical dilemma in the language of cognitive analytic therapy (CAT), that of either facing up to things, but feeling anxious and exposed, or avoiding things and bottling up, feeling cut off and trapped. Our therapeutic focus is on the gentle, but clear resolution of this dilemma, which is so easy to collude with or be drawn into. To aid this we value a model such as CAT which names the problem pattern of relating to self and others, accurately weaves in ways of coping and sets up a rapid assignment to monitor and revise the pattern.[5]

The combination of exploration of problems, expression of feelings, gaining understanding and insight and developing coping strategies fits well with the aims of the service and the client completed CORE form data.

Different levels of using CORE

One of the very important uses that can be made of the CORE data is in identifying general service quality indicators against which we can benchmark our service. For example, we have used assessment profiles and outcomes to benchmark the appropriateness of what we offer, looking at pre-therapy clinical cut-off scores and numbers of clients accepted for therapy. Service ‘acceptability’ to clients can be benchmarked using data relating to planned and unplanned endings.

Whilst all of the preceding data offers service level description there is, in addition, unlimited evaluative potential to use the CORE data for detailed case study and profiling of particular counsellors, case types and problems. This is a rich aid to case formulation, case management and supervision, both overall and individually. It is important that this is done in a respectful and collaborative way and the managerial style and the sense of cohesion in the team is an important factor. There are a number of stakeholders to any audit process and none should have the dominant hand. It is easy to sabotage or spoil the authority of any audit process, as the recent shocks to the credibility of audit process in the world of high finance and company reports in the USA indicate. Whilst outcome evidence and benchmarking of effectiveness and efficiency is important it is ultimately not as important as a team of practitioners interacting on a continuous basis with their evidence of effectiveness. We are working to our own simplified audit cycle diagram. A richer version of this detailing the practitioner based research cycle can be found in Roth and Fonagy (1996)[6]

Effective audit cycles

Evidence of effectiveness is only useful if it is turned to the task of looking for improvements. See diagram of audit cycle.

The full interactive and intuitive power of the CORE PC system is hard to communicate in a linear article. One example may suffice.

Supervision using a scatter plot diagram

Uniquely the CORE PC offers us a scatter plot (see diagram 1.below) showing in the top right hand corner those clients whose score indicates deterioration in their well being post counselling and those in the bottom right hand corner who indicate reliable statistical and clinical improvement. Within the tram lines going at 45 degrees from left to right are those who have not significantly improved or deteriorated. Those in the bottom left hand corner started below the clinical cut off scores but may still have improved significantly. The software allows the practitioner alone, or in case management supervision, to review work by clicking on any particular dot in the scatter plot and call up the specific case profile including item by item rankings of change in CORE scores. Such scatter plots can be obtained for individual counsellors, for specific problem clusters and treatment types. Such sophistication in a desk top research and audit tool is unprecedented and we are currently piloting its use as an interactive aid to case management and supervision.

The process of routine evaluation has helped us understand our client population at the point they seek help much more clearly. We can see how anxiety is a key factor and how psychological problems and maladaptive or avoidant ways of coping impact on academic motivation and studies.

We put great store by a rapid response. We are increasingly offering short psycho educational workshops such as on coping better with academic pressures, confidence building and self esteem and addressing these in terms of poor ways of coping with stress and pressure. We continue to link our work to an type of psychological therapy that rapidly and collaboratively links a particular personal problem to a more general pattern of relating to self and others such that in helping someone solve a particular immediate difficulty they can learn a general lesson about themselves. We see the future direction of the service geared to a combination of counselling and psychological education. We value more following our evaluative research case management consultation within the team as much as individual or group supervision.

Routine evaluation counters the risk of ‘management by appearances’ for the one-off audit or the halo effect of a dedicated piece of research. Routine evaluation, if conducted collaboratively and sensitively, promotes a culture of constructive criticism among practitioners. We are currently engaged in a variety of small scale audit trails and practitioner research projects linked to issues such as unplanned endings, specialist interventions, gender difference effects, selection criteria for cases taken to supervision, effectiveness of groups, hard to help cases and features of the negative therapeutic reaction.

Through routine evaluation we value our work and have a greater sense of what works and what does not in our setting. We can better spot good enough practice, excellent practice and areas for improvement. We hope that this kind of practitioner led evaluation is the seedbed from which a different kind of research process into psychotherapy and counselling can grow. One very useful possibility would be a national benchmark for student counselling with an easy comparison with NHS mental health services. This would require a practitioner led research network built around the CORE PC system comprising a number of counselling services.

The potential for interactive, ongoing audit using the power and flexibility of CORE PC software offers new scope for service management and development as well as an aid to supervision and research. We have identified a set of questions specific to student counselling settings to enrich the main CORE evaluation form. The benchmarking of CORE across NHS mental health provision, and the sophistication of the CORE PC software, offers a great opportunity for a group of student counselling services to form a practitioner led research and audit network. The costs of this need not be prohibitive. Our service spent 0.5% of its annual budget on service evaluation last year.

[1] CORE Information Management Systems Ltd. http://www.coreims.co.uk (John Mellor Clarke)

[2] CORE PC, as its name implies, is a rapidly developing software support programme for the CORE system which allows individual practitioners, teams of practitioners or groups of services to analyse their data interactively and receive immediate reporting on individual cases, specific practitioner cases and overall service profiles. Data can be linked to national benchmarks for service effectiveness.

[3]Psychological Therapies Research Centre, University of Leeds

[4] CORE National Comparative Performance Indicators and Service Descriptors. A Benchmarking Guide for Services Providing Brief Psychological Therapy Services in the UK. July 2002 CORE Information Management Systems Ltd. http://www.coreims.co.uk

[5] Ryle, A & Kerr, I.B Introducing Cognitive Analytic Therapy (2002) Wiley

[6] Roth A & Fonagy P (1996) p48 Achieving Evidence Based practice in the Psychological Therapies in What Works for Whom? A Critical Review of Psychotherapy Research Guildford Press

Anxiety/stress	408
Interpersonal/relationships	327
Work/academic	305
Depression	253
Self esteem	245
Bereavement/loss	108
Physical problems	59
Trauma/Abuse	53
Personality problems	53
Living /welfare	50
Eating Disorder	24
Addictions	18
Psychosis	4

Clinical Change	No Reliable Change	Reliable Deterioration	Reliable Improvement	Totals
Totals	65 (23.466%)	3 (1.083%)	209 (75.451%)	277 (100%)

Benefit	Improved	Not addressed	Unimproved
Exploration of feelings/problems	281 (75.538%)	4 (1.075%)	12 (3.226%)
Personal insight/understanding	276 (74.194%)	11 (2.957%)	27 (7.258%)
Expression of feelings/problems	276 (74.194%)	12 (3.226%)	16 (4.301%)
Coping strategies/techniques	259 (69.624%)	20 (5.376%)	18 (4.839%)
Subjective well being	237 (63.71%)	15 (4.032%)	25 (6.72%)
Day to day functioning	222 (59.677%)	22 (5.914%)	18 (4.839%)
Relationships	205 (55.108%)	42 (11.29%)	29 (7.796%)
Symptoms	184 (49.462%)	36 (9.677%)	31 (8.333%)
Control/planning/decision making	146 (39.247%)	87 (23.387%)	23 (6.183%)
Access to practical help	95 (25.538%)	124 (33.333%)	25 (6.72%)