4.1 Introduction
In this chapter, the writer will discuss the methods of evaluation and the analysis undertaken in the change project. Evaluating the change project is essential to measure its success. Evaluation is part of the mainstreaming stage of the HSE model delineated in the previous chapter. Evaluation describes the actual intervention, its measures, and consequences. It is carried out for multiple reasons, mainly to provide information for judging the value of interventions. It is also carried out to examine causes of what has occurred for the purpose of improving, sustaining, or expanding the intervention. (Green & South, 2006) The writer has gone through the RCSI Research Ethics Committee process.
4.2 Significance of Healthcare Evaluation
In the writer’s opinion, healthcare evaluation is a study that measures the effectiveness of a specific healthcare service and assesses whether it meets its intended objectives. (Health Knowledge, 2011; World Health Organization, 2003) Evaluation of a healthcare service indicates its performance and provides areas for improvement. (Glacken & Chaney, 2004; World Health Organization, 2003) Evaluation in healthcare has been defined as a “study design for assessing effectiveness, efficiency & acceptability of services including measures of structure, process, service quality & outcome of healthcare.” (Health Knowledge, 2011, Study Design for Assessing Effectiveness, Efficiency and Acceptability of Services Including Measures of Structure, Process, Service Quality, and Outcome of Health Care section)
Donabedian’s structure-process-outcome model is one of the dominant frameworks used in today’s healthcare field to assess and improve the quality of health care. Donabedian’s model indicates that healthcare organizations are components of structures and processes to deliver outcomes, and these concepts are also evident in the definition of healthcare evaluation addressed previously. The three categories are linked together, and not one category by itself can serve as an indication of quality healthcare. Therefore, healthcare organizations need to measure all three categories and apply them in practice in order to implement improvement plans. (Donabedian, 1988) Øvretveit, another pioneer in healthcare quality, defined four different evaluation designs. Selection of the design depends upon the evaluation questions which are based on the objectives of the initiative. The descriptive case design is an observational study that describes the features of a program, policy, or the people receiving the intervention. The audit design is a compliance study that compares what people currently do to what they are supposed to be doing from a standard, policy, or procedure. On the other hand, the before and after design is a prospective study that can be single case or comparative. The single case design compares data of patient outcomes before and after an intervention has occurred. While the comparative design provides stronger evidence by comparing the data between two or more interventions. Finally, the retrospective evaluation design can be one of two approaches, theory testing or theory building. The theory testing approach is a quasi-experimental design to identify factors critical to success. In contrast, the theory building approach involves asking informants to describe the activities and effects of an intervention. (Øvretveit & Gustafson, 2003)
4.3 Evaluation Model
The evaluation process was defined through the objectives of the OD project. As previously stated, the objectives were:
- All pharmacists will have attended the required smoking cessation training by January 2015
- A smoking cessation policy that is complaint with HAAD standard for Smoking Cessation Services will be in place within the organisation by February 2015
- Within 3 months, achieve a 30-days smoking abstinence for 40% of enrolled participants
- Within 3 months, reduce the number of tobacco smoked per day by half for 30% of enrolled participants who did not abstain smoking
The model chosen to evaluate the above objectives is Kirkpatrick’ four levels of evaluation. Kirkpatick’s model is one of the most popular approaches for evaluating training. The model focuses on multiple measures of training effectiveness. It does not only measure the reaction to the training and the final results, but also the learning attributed to the training and the behavioural changes in the trainees. One of the limitations that the model has been criticised with is that it is incomplete with an oversimplified view. It is said that it does not consider individual and contextual influences in evaluating the training. Individual influences may include characteristics of the organisation of the trainee; while the contextual influences may include the organisational culture or the adequacy of material resources. (Bates, 2004; Kirkpatrick, 1994)
Although the model is intended to measure the effectiveness of training, however the writer believes that it can be applied to the OD project. The major requisite for providing smoking cessation services is to train and develop the pharmacists’ knowledge, skills, and competencies to lead the smoking cessation program. The other objectives can be easily included into one of the four levels of the model. The measures in the objectives cover the array of Donabeian’s structure-process-outcome model. These measures require different designs like those expressed in Øvretveit’s evaluation designs. Therefore, it is essential that the evaluation model can not only incorporate different types of measures but also designs and methods. The writer trusts that the Kirkpatrick model matches the needs for evaluating the objectives.
4.3.1 Level One: Reaction
Level one of the model was used to measure the pharmacists’ reaction to the CME event on basic smoking cessation. Verbal feedback was gathered from all four trained pharmacists to understand how they felt about their learning experience. The questions asked were on the overall satisfaction of the training and their perceptions of the presentation style, venue, and timing. The overall feedback received was positive stating that the training was useful and informative. There were only two general comments with regards to the pharmacists’ perceptions. One of the pharmacists mentioned the following:
Pharmacist 1: “I was really happy with the training. It had a lot of new information. Maybe it was too much to have it on one day only.”
However, none of the other pharmacists shared the same view of it being an intensive one-day training. On the other hand, two pharmacists had similar remarks on aspects of the content.
Pharmacist 2: “The course was very useful. I only preferred that they prepared us more on how to deal with difficult cases that don’t want to quit smoking.”
Pharmacist 3: “I loved the training and the speaker, but I wanted to hear more about motivational counselling.”
Although motivational counselling was one of the topics covered during the training, however it appears that the theoretical and practical learnings did not meet some of their expectations. The feedback received was very useful to plan changes for the future. This was one of the reasons for conducting the internal refresher training in a practical role-playing format.
4.3.2 Level Two: Learning
The second level of the model was used to measure the increase in learning on smoking cessation services. It is advisable to conduct before and after tests to compare the increase in knowledge gained. (Kirkpatrick, 1994) However, the writer only conducted an after test since the details of the plan and the evaluation methods were not discussed until after the pharmacists attended the training. The writer measured the learning using questions from the knowledge section of a previously structured and validated questionnaire called “Smoking Cessation in Pharmacy”. (Saba et al., 2013) The questionnaire (Appendix 8) consists of two parts. The first part included 10 questions on general smoking knowledge, common myths of smoking, NRT knowledge, specialised smoking knowledge, and nicotine dependence. The second part consisted of eight questions more specialised in pharmacotherapy and clinical case scenarios. The overall result (Table 3) was very high with a percentage of 93%. Also, the individual results were extremely high in all pharmacists with one of them scoring a full grade. It is expected that those who have received smoking cessation training or had prior experience in smoking cessation to score much higher than others especially in the second part. (Saba et al. 2013) These results are an indication of meeting the first objective and that the pharmacists have been provided with an appropriate training that increased their knowledge on smoking cessation.
Table 3: Results of Smoking Cessation in Pharmacy Questionnaire
4.3.3 Level Three: Behaviour
The third level of the model was used to measure the extent of change in behaviour based on the pharmacists’ learning. In the context of the project, the change in behaviour was not only measured based on the learning from the training received. It was also measured with regards to the approved policy and the agreed plan which were set against HAAD’s Standard for Smoking Cessation Services. Evaluating behaviour was the most challenging of all levels. It could not have been done without the support and involvement of the Pharmacy Manager, who is also the pharmacists’ line manager. An audit tool (Table 4) was used to evaluate the success of implementation of this level. An audit is a tool in which “aspects of structure, process, and outcomes of care are selected and systematically evaluated against explicit criteria.” (NICE, 2002, p.1) It is very useful not only to assess change, but also to develop and confirm improvements in the healthcare delivery. The audit was structured using four responses for meeting the criteria. They were either fully met, partially met, never met, or not applicable. The writer observed the pharmacists’ practices on a bi-weekly basis, starting with the third week after implementation commenced. Based on the observations by the writer and a discussion with the Pharmacy Manager, a response was recorded for each of the 14 criteria.
Table 4: Audit Tool
The results of the audit (Figure 9) were graphically represented using RAG (red-amber- green) coloured dots for each criterion per week. Five criteria (A, D, F, h, and L) were fully met in all audits. The rest of the criteria except for criteria J were either partially met or never met for one or two audits. However, they were fully met in the final two audits. As for criteria J, this was the only criteria that raises concern as it did not improve over time. By looking at the graph, it is quite evident that improvements have been made and sustained during the implementation. The green colours in all criteria of the final two weeks is a confirmation of improvements in the healthcare delivery.
Figure 9: Audit Results
4.3.4 Level Four: Results
The fourth level of the model was used to measure the effect of the training on the business or environment. In the context of the project, the results measured are the outcome measures from the final three objectives of the project. The writer believes that out of all the measures, these outcomes are the most important since they strongly relate to the rationale of the project. Kirkpatrick’s model also assumes that that the results of level four will provide the most useful information about the effectiveness of
the training program. (Bates, 2004) The two outcome measures selected for the project were the smoking status and satisfaction score. The participants’ smoking status was divided into four categories (lost to follow-up, reduced their tobacco intake by less than half from baseline, reduced their tobacco intake by more than half from baseline, and stopped smoking). The participants’ clinical notes were reviewed to gather the necessary data. Their characteristics and abstinence results are shown in the table below (Table 5).
| Lost to follow-up | Reduced by less than half | Reduced by more than half | Stopped | |
| n=26 % | n=13 % | n=31 % | n=43 % | |
| Gender Female Male Age 18-24 25-39 40-54 55+ Unspecified Fagerstrom Test for Nicotine Dependence 0-2 (very low) 3-4 (low) 5 (medium) 6-7 (high) 8-10 (very high) Unspecified Sessions Completed Less than four sessions Four or five sessions Six or seven sessions | 11 15 1 5 8 0 12 5 0 1 2 3 15 26 0 0 42.3% 57.7% 3.8% 19.2% 30.8% 0.0% 46.2% 19.2% 0.0% 3.8% 7.7% 11.5% 57.7% 100.0% 0.0% 0.0% | 0 0.0% 13 100.0% 0 0.0% 10 76.9% 2 15.4% 1 7.7% 0 0.0% 0 0.0% 3 23.1% 2 15.4% 7 53.8% 1 7.7% 0 0.0% 13 0 0.0% 0 0.0% 100.0% | 4 27 8 8 13 2 0 1 4 12 10 1 3 19 12 0 12.9% 87.1% 61.5% 61.5% 100.0% 15.4% 0.0% 3.2% 12.9% 38.7% 32.3% 3.2% 9.7% 61.3% 38.7% 0.0% | 7 16.3% 36 83.7% 7 16.3% 18 41.9% 17 39.5% 1 2.3% 0 0.0% 5 11.6% 8 18.6% 13 30.2% 12 27.9% 3 7.0% 2 4.7% 13 30.2% 19 44.2% 11 25.6% |
| Total | 23.0% | 11.5% | 27.4% | 38.1% |
Table 5: Participant Characteristics and Results
A total of 113 participants enrolled in the program of which 80.5% (n=91) were males. The majority of the participants were between the ages of 25 and 54. 23% (n=26) of the participants were lost to follow-up. All of them attended less than four sessions, and more than half of them did not have their FTND taken. With regards to the participants who reduced their smoking intake, 27.4% (n=31) were able to reduce by more than half. As for the participants that stopped smoking, 38.1% (n=43) were 30-days smoking abstinent with CO validation. An important notice is that 70% of these participants have attended more than four counselling sessions.
The pharmacy’s patient satisfaction questionnaire (Appendix 9) was used for the second outcome measure. This questionnaire was developed two years ago during the Pharmacy Manager’s graduate studies and was tested for reliability and validity. The questionnaire uses a 5-point Likert scale to rate the agreement with each of the statements. Two additions were made to the questionnaire to measure the views of those participating in the smoking cessation program. A question was added to measure their satisfaction in the same format of the questionnaire using a 5-point Likert scale. An open ended field was also added for any comments they had on the program. The overall satisfaction scores of the pharmacy (Figure 10) have been steadily increasing by small increments since the project started. The satisfaction score by the end of March was around 85.8%, a 2.5-3.0% increase since the end of the previous year. This was the highest monthly score received not only during the project, but also for the lifetime of the pharmacy. The satisfaction scores with the smoking cessation program (Figure 11) were also extremely satisfactory. On average, the score during the three months of implementation was 90%.
Figure 10: Overall patient satisfaction scores
Figure 11: Smoking cessation program patient satisfaction scores
4.3.5 Dissemination Plan
The purpose of the dissemination plan is twofold. The first is to communicate the results of the project with respect to its intended objectives. The second is to share the knowledge and learning earned during the project. The dissemination plan was split into three stakeholder groups: the pharmacy management, the staff, and the project sponsor. The first activity was a meeting with the Managing Director and Pharmacy Manager to share the results and decide on how to communicate the findings to the staff and sponsor. Given time constraints and unavailability for an all staff meeting, the staff and sponsor were emailed the same final report. In addition to the report, the staff were also interviewed to hear more about their views of the project and their learnings which were included in a section of the report.
4.4 Summary and Conclusion
The writer used Kirkpatrick’s model to evaluate the success of introducing a pharmacist- led smoking cessation services in a private community pharmacy. Various methods and tools were used within the model. These included verbal feedback, observations, audit, questionnaires, and participant clinical notes. The evaluation findings reflect that the aim of the project was achieved along with its stated objectives. Although targets set in the outcome measures were not attained; however the writer is confident that the project achieved an overall highly successful result. In the next chapter, the writer will discuss the project in relation to its strengths and limitations and will provide possible recommendations for the future.
