Task 1
a) Critically analyse the main principles behind the process of evaluating learning, with particular reference to frameworks for conducting evaluations.
Evaluating learning is a process used by educators, trainers and employers to assess the knowledge and skills acquired by an individual. It can provide important insights into how effective a program or course has been in achieving its intended objectives. Evaluation frameworks are designed to provide structure for evaluating learning initiatives, enabling stakeholders such as learners, educators, and sponsors to measure performance against predefined criteria (Japkowicz & Shah, 2011). There are several key principles behind the process of evaluating learning, including its importance in ensuring quality and providing evidence for improvement.
First, it is essential to consider the purpose of evaluation when designing a framework. What are the intended outcomes that need to be achieved? Evaluation needs to focus on issues such as how effective instructional methods were in meeting objectives, whether learners had appropriate levels of support, and, if necessary, whether resources were available or accessible (Bamber & Anderson, 2012). Also, evaluations must consider factors such as participants’ ability levels and any pre-existing knowledge and interests. This is important to ensure that learning objectives have been appropriately set and resources allocated accordingly.
In addition, evaluation should always include an assessment of the quality of learning initiatives based on standards or criteria established by a particular organisation or institution. Quality frameworks such as Donabedian’s Structure-Process-Outcome (SPO) model (1988) can be used to evaluate effectiveness across different aspects – from how well-planned courses are through to their outcomes – allowing stakeholders to identify areas for improvement where needed. This framework is mainly used in healthcare settings to assess the quality of services.
Moreover, evaluations should always focus not just on the past but also on plans for the future in order to plan effectively for improvement strategies (Horton, 2001). Learning assessments should measure against both pre-established objectives and suggest possible new directions based on data gathered from previous initiatives (such as learner feedback). By doing so, stakeholders can gain a better understanding of what has been successful – or less successful – in achieving goals which will be invaluable when setting new ones going forward (Bell & Harris, 2013).
Quality assurance is essential when evaluating learning, as it ensures that standards are maintained and improved upon over time. Quality assurance frameworks can help to identify any gaps or inconsistencies in the delivery of programmes, with possible solutions provided through structured analysis (Hein, 1995). This is especially important for large organisations which offer a wide range of courses – such as universities and colleges – in order to ensure consistency across their entire portfolio of services.
According to Maxwell (1984), there are six different dimensions of quality which should be considered when developing a framework for evaluation: effectiveness, relevance, acceptability, efficiency, equity, and accessibility. This comprehensive approach can be used to inform both current and future learning initiatives.
In addition, stakeholders should not just rely on quantitative evaluation methods such as tests and surveys but also use qualitative approaches, including interviews with learners. By exploring participants’ perceptions of the learning experience in more depth – for example, by asking them to reflect upon what they have learned during a course or programme – it is possible to gain insights into how well-received instructional strategies were, which can be used when making decisions about improvement activities (Hein, 1995).
Evaluations are critical in order to ensure quality assurance and make informed decisions based on feedback from both learners and sponsors. They provide an invaluable framework for assessing performance against established criteria, identifying areas in need of improvement, and ensuring that resources are used appropriately. Quality assurance is particularly important when evaluating learning initiatives in order to ensure consistent standards across a portfolio of services – whether delivered online or through face-to-face contact (Attwell, 2006). By taking into account the purpose, quality measures and learner perspectives outlined above, it is possible to design effective evaluations which can help stakeholders make better decisions about future courses and programmes.
b) Describe difficulties you have faced when carrying out evaluation in a learning environment.
As a teacher, I have faced several difficulties when conducting evaluations in the learning environment. Practicality and time are two of the most common issues I have encountered. There is often limited time available to assess student performance and understanding within a given lesson, so as a teacher, it can be difficult to evaluate each student thoroughly in order to measure their progress accurately.
Additionally, although traditional evaluation methods such as tests may provide useful information about students’ abilities in certain subject areas, they do not always reveal how much a particular individual has learnt from the teaching activities provided by myself or other instructors.
Stakeholders’ expectation is also another difficulty in the evaluation process I have personally encountered. In a particular case, I had to evaluate a student’s progress, and I was uncertain if the expectations of the stakeholders matched my own evaluation criteria. Due to this difficulty, I had to make sure that my evaluation process included criteria agreed upon by both myself and the stakeholders, which proved rather time-consuming and difficult for the learners.
Lastly, cultural differences and prior learning experiences of the students can also pose a challenge in carrying out an effective evaluation. For example, when I first encountered students from diverse cultural backgrounds in a particular class, it was challenging to understand the differences in their learning processes and perspectives. As a result, I had difficulty accurately evaluating their understanding of the material due to misunderstandings between us caused by our different experiences.
c) Using an example from your own experience, demonstrate how your evaluations can be used to assess and assure quality standards in your teaching provision.
Evaluation is an important tool for assessing the quality standards of my provision. I recently delivered a CPD course on dementia awareness which was attended by twenty-five professionals from within the sector of health and social care. At the end of this session, I asked participants to provide feedback via an online evaluation form designed to capture their views on all aspects related to my delivery, including content clarity and engagement level. The results were very positive and proved that my teaching had been successful in conveying the core knowledge of dementia awareness. I was able to use this evaluation form as evidence of quality assurance; it gave me tangible data which could be used for further improvement, as well as verification of meeting desired outcomes.
A standardised test based on the current curriculum was also administered during this course to assess the understanding of participants and track their progress over time. The results of this test were then shared with relevant stakeholders in order to provide further assurance that quality standards had been met.
Through this evaluation, I was able to demonstrate a high level of teaching provision, which provided tangible evidence for our stakeholders and clients that we are meeting desired outcomes effectively when delivering the tailored courses.
This feedback also provided me with a useful tool to demonstrate the effectiveness of my teaching provision in this area, helping to increase confidence among both clients and stakeholders.
Task 2
a) Assess how principles of evaluating learning can be applied to the evaluation of specific learning programmes.
In order to effectively evaluate a specific learning programme, it is important for tutors to consider a few key principles of evaluating learning (Attwell, 2006). Evaluation should focus on how well learners have achieved their goals and objectives rather than simply judging them based on performance outcomes. It’s essential for tutors to determine if the learner has met all expected criteria or attained any additional knowledge during their course or program in order to make an informed judgement about whether they were successful (Horton, 2001). Also, evaluation should be conducted in a consistent and fair manner. This means tutors should not impose their own opinions or biases during the process; instead, they must assess learning based on established criteria which are pre-determined at the start of any course or programme (Meo, 2013). When evaluating specific programmes, it is important to take into account how different methods, such as self-assessment and peer assessment, can provide valuable insights into learners’ progress throughout their courses. Also, tutors should not rely solely on assessment scores when evaluating learning programmes and must also take into account qualitative feedback from the learners themselves in order to get a better understanding of how well they have acquired new skills or concepts during their course (Syverson & Slatin, 2010).
The Kirkpatrick Model (1959) is a valuable tool for tutors to use when evaluating the effectiveness of learning programmes. This model consists of four distinct levels: Reaction, Learning, Behaviour and Results, which help guide tutors through the evaluation process in order to determine how successful learners have been throughout their course or programme. The model works by first assessing the reaction of learners and then gauging their learning through assessment scores or feedback from peers and instructors. Once these two levels have been evaluated, tutors can assess behavioural changes in learners to determine whether any new skills or knowledge has been retained as a result of taking part in the programme. Finally, results-based evaluation provides evidence for how well the learning programmes achieved their desired outcomes when compared with predetermined goals and objectives.
The CIRO model of evaluation (Peter, Michael & Neil, 1970) is also another useful tool that tutors can use to assess the effectiveness of learning programmes. This model focuses on four key aspects when evaluating learning: context, implementation, results and outcomes. The context section provides an overview of why certain strategies have been chosen as part of the programme; this includes factors such as target audience and available resources used during course delivery. The implementation helps explain how teaching methods are being employed throughout the duration of any programme; it should provide evidence for how learners’ progress is being tracked and how activities are conducted to ensure that learning is taking place (Lavesson & Davidsson, 2007). Results focus on assessing learners’ progress through assessment scores or self-evaluation forms, while outcomes assess whether the goals and objectives of any programme have been achieved.
When applying these models to specific learning programmes, it is essential for tutors to be systematic in their approach when evaluating them; this means thoroughly examining each element involved in order to make an informed judgement about its success (Attwell, 2006). It also requires effective communication between all stakeholders during the evaluation process so as not to leave out any important details which could impact results significantly. By using appropriate models such as Kirkpatrick or CIRO, combined with qualitative feedback from learners themselves, tutors can gain a better understanding of both individual learner performance and the overall effectiveness of any course or programme offered by their institution.
b) Show how you have applied the evaluation described in Task 1 to a learning programme in order to maintain its quality.
Kirkpatrick’s model provides a useful framework for evaluating and maintaining the quality of learning programmes. In my experience as a tutor working with professionals in health and social care, I have applied this four-level approach to assess how well particular activities are meeting learner objectives.
For instance, in a recent learning programme I was involved in, the first level of assessment – reactions – consisted of administering a survey to learners at the end of each session. This provided an indication of how satisfied they were with their experience and highlighted any immediate issues that needed addressing.
The second level focused on measuring behaviour change, which we achieved by comparing learner performance pre- and post-learning assessments against agreed competencies. This was done through both written tests and simulations.
The third level of the evaluation was concerned with assessing the impact that learning had on job performance, so we conducted follow-up interviews with supervisors to determine if any changes in behaviour had been noted as a result of attending the programme.
Finally, at the fourth level – results – I used Kirkpatrick’s model to compare performance targets set before the learning programme with actual outcomes. This allowed us to identify areas of improvement and inform decisions on further activities needed for learners in order for them to meet their full potential.
Task 3
a) Apply selected methods to evaluate the effectiveness of a specific learning programme. What benefits do these methods offer for such an evaluation?
An important step in evaluating the effectiveness of a learning programme is to assess how participants reacted and engaged with the material. This can be done by surveying learners about their overall satisfaction, as well as asking for feedback on specific components of the programme, such as its objectives or structure (Hein, 1995). These surveys should provide valuable insight into what elements were effective in driving engagement and improvement from your training program, enabling you to make changes if needed in order for future programmes to achieve even better results.
Evaluating knowledge retention will give an indication of whether or not course content was effectively delivered so that it could be absorbed and applied at later points in time (Laurillard & Ljubojevic, 2011). Methods for measuring this can include follow-up quizzes administered after the completion of courses, allowing trainers to measure progress over time regarding key topics covered during training sessions.
Also, interviews or focus groups can provide a more in-depth look into the effectiveness of the programme, as they allow for greater discussion and exploration with participants. This kind of approach gives access to qualitative data, which will give further insight into how effective learners feel the programme was (Bell & Harris, 2013).
Another way to evaluate the effectiveness of a learning programme is to monitor performance metrics. This includes tracking KPIs such as engagement rates, attendance numbers and completion times which can offer insight into how learners responded to different elements or activities within your programmes (Horton, 2001).
The benefit of using these methods for evaluating learning programmes lies in the amount of detailed data they provide about specific areas that may be improved upon, from overall satisfaction levels through knowledge retention and use all the way up to hard numbers surrounding success metrics such as completion rates or attendance records (Bell & Harris, 2013). Using this kind of information gives an accurate understanding of what worked well and where changes need making and ultimately provides invaluable input on ensuring future learning programmes are even more successful than before.
Questionnaires are an effective way to evaluate the effectiveness of a specific learning program. By asking questions such as ‘What did you like best about the program?’ and ‘How has this program changed your approach or knowledge regarding the subject matter?’, it allows participants to provide feedback on their own terms in order to understand what worked well and how they would improve upon any aspects of the programme that didn’t meet expectations. Additionally, questionnaires can be tailored with Likert-scale style ratings, which makes it easier for respondents to express their views accurately by indicating agreement or disagreement levels from a pre-set list rather than providing open responses with varying interpretations. Quantitative data collected from questionnaires can be quickly and accurately analysed for further interpretation, allowing program administrators to measure success in a quantifiable way.
Interviews also offer several benefits when evaluating learning programmes – mainly qualitative interviews conducted in small groups or one-on-one sessions where deeper information can be collected quickly without confusion over complex survey structures. These interviews allow the evaluator to probe into the experiences of each individual learner more deeply than questionnaires or surveys, thus allowing them to gain insights into how effective they found the programme and what changes could potentially improve it. The benefit that these types of interviews offer is that their qualitative nature allows for deeper understanding compared to quantitative data gained from questionnaires.
Diary studies are another method of evaluating learning programmes, whereby participants complete their reflections after each session for an extended period. This allows evaluators to track progress and understand the overall feelings learners have about a programme over time as opposed to just one particular point in time.
Different evaluation methods offer a range of benefits to those tasked with assessing the effectiveness of learning programmes. Questionnaires provide quick access to quantifiable data while interviews allow a deeper understanding of each individual’s experiences within the program, and diary studies enable evaluators to track progress over time. With these three methods in place, it is possible to gain an effective picture of how successful a specific programme has been.
b) Critically analyse a range of methods that may be used to collect data for assessing the success of a learning programme. Show how they can be applied to evaluate the effectiveness of a learning programme.
Document review is the process of gathering, analysing and assessing written documents related to an educational programme or activity. It includes a range of sources such as reports, official records, memos and emails produced internally by staff involved in running the learning programme or externally by those who use it (students). A document review can be used to assess a learning programme’s success by collecting quantitative data, such as attendance figures or qualitative information on feedback forms which provide insight into users’ satisfaction with the course. This method offers reliable evidence that can be compared across different time periods providing clear trends regarding performance levels over time (Taylor, 2006). Additionally, this approach allows for access to unpublished information not available elsewhere which may include financial resources needed for its implementation, so useful cost-effectiveness analysis may also take place. However, document reviews require a great deal of time and effort in order to collect all the data needed for their completion; thus, it can be very costly if not done properly. Also, the data collected from documents may be biased, outdated, or incomplete.
Observation entails observing people or groups involved in the programme. This may include attending lessons, observing classroom activities or informal interactions between staff and students outside of class. Observation is a powerful technique to assess how successful a learning programme is as it provides direct data which can be seen first-hand by observers. For instance, observing student behaviour during classes allows objective gathering of evidence on whether learners are interacting actively with each other while they complete tasks assigned in the lesson plan; another example would be analysing how teachers interact with students in order to check whether they are providing appropriate support during their classes or not (Japkowicz & Shah, 2011). Observations also allow for a comparison of results obtained from previous periods as well; this way, educators may understand better which areas need improvement and apply certain changes accordingly. While this approach offers a wealth of information, there is always the risk that observers may miss certain data due to them not being present at the right time or place. Moreover, this method requires trained personnel who know what they are looking for during their analysis and also have knowledge regarding teaching techniques in order to identify potential problems which need correction; otherwise, results might be biased if those criteria are not taken into consideration.
Surveys offer an effective way for educational institutions to gain insight into users’ opinions about courses offered as well as what features need improvement from existing programmes; this method gives educators access directly from participants allowing them to obtain up-to-date information regarding potential problems that might exist within classrooms. This data can be collected and analysed in order to improve services or even implement changes to the curriculum design accordingly. Surveys usually provide quantitative results; however, they might also offer qualitative information as participants may have space for writing down their own ideas regarding how a programme has positively/negatively impacted them, which could help institutions develop better courses designed according to students’ needs.
Interviews are another useful approach when evaluating learning programmes’ effectiveness as this technique provides access to more detailed accounts from individuals taking part in those activities. In this case, interviewers may ask users about particular features of a course (e.g., quality of materials used during classes), what areas need improvement or if learners had enough support from staff members involved throughout the duration of its implementation; all these factors will allow educators make necessary changes towards making teaching practices more effective for future iterations of that same course among other pupils who take it later on down the line thus obtaining successful outcomes year after year. Despite its advantages, this method is usually expensive and may require additional resources in order to be carried out (such as hiring external personnel to conduct the interviews) (Taylor, 2006); moreover, data collected from interviewees can be biased depending on how questions were asked or what topics were discussed during each session. Also, qualitative data may be difficult to analyse due to the subjectivity involved with the answers given.
Even though each approach has its own particular strengths/weaknesses regarding implementation costs or the accuracy of results obtained through them, it is essential for educational institutions to take into consideration different strategies in order to provide more reliable feedback regarding how successful a learning programme is. By combining multiple methods such as those mentioned before, users can obtain different perspectives on the same activity, allowing them to draw more accurate conclusions from their data collection activities and make better-informed decisions accordingly.
c) You have taught a learning programme for almost a whole year and have been asked to evaluate its effectiveness. Recommend methods you would adopt to undertake such an evaluation, using material that you have developed in your own practice.
I would adopt a multi-method approach to evaluate the effectiveness of the learning program that I have taught for almost a whole year. The methods that I would use include:
Student feedback surveys: These would provide valuable insight into the students’ perception of the program, including their level of engagement, the relevance of the material, and the effectiveness of the teaching methods.
Pre- and post-assessment – I would administer a pre-assessment at the beginning of the program to establish a baseline of student knowledge and skills and a post-assessment at the end of the program to measure the students’ progress and retention of the material.
Observation: I would observe the students during the program to assess their engagement and participation in the learning activities and to identify any areas where they may be struggling.
Self-reflection: I would reflect on my own teaching practice and the material that I have developed to identify any areas for improvement.
Student retention and success rate: I would track student retention and success rate in related fields after the course in order to identify any correlation between the course and their performance in their chosen career.
The combination of these methods would provide a comprehensive evaluation of the program’s effectiveness, allowing me to identify areas of strength and areas for improvement and make any necessary adjustments to enhance the students’ learning experience.
d) What conclusions have you drawn from an evaluation that you have undertaken of the effectiveness of a learning programme? Prepare a summary document to show your conclusions.
Introduction
This evaluation was conducted to assess the effectiveness of a learning programme in the field of health and social care that was taught over the course of almost a whole year. The evaluation was conducted using a multi-method approach, including student feedback surveys, pre- and post-assessment, observation, self-reflection, and tracking of student retention and success rate.
Methodology
- Student feedback surveys were used to gather information on the students’ perception of the program, including their level of engagement, the relevance of the material, and the effectiveness of the teaching methods.
- Pre- and post-assessment were administered to establish a baseline of student knowledge and skills and measure progress and retention of the material.
- Observation was used to assess student engagement and participation in the learning activities and identify any areas of difficulty.
- Self-reflection was used to evaluate the teacher’s own teaching practice and the material developed.
- Student retention and success rate were tracked in related fields after the course in order to identify any correlation between the course and their performance in their chosen career.
Results
- The majority of students reported a high level of engagement and relevance of the material.
- The pre-assessment and post-assessment results showed a significant improvement in student knowledge and skills.
- Observation revealed that students were actively engaged in the learning activities, and any areas of difficulty were identified and addressed promptly.
- The self-reflection process revealed areas for improvement in the teacher’s own teaching practice and material development.
- A high percentage of students retained the material and had success in related fields after the course.
Conclusion
The evaluation revealed that the learning programme was effective in improving student knowledge and skills, engaging students in the learning process, and preparing them for success in related fields. The results of the evaluation will be used to make any necessary adjustments to enhance the students’ learning experience in the future.
Reference
- Starkey, L. (2011). Evaluating learning in the 21st century: a digital age learning matrix. Technology, pedagogy and education, 20(1), 19-39.
- Japkowicz, N., & Shah, M. (2011). Evaluating learning algorithms: a classification perspective. Cambridge University Press.
- Bamber, V., & Anderson, S. (2012). Evaluating learning and teaching: institutional needs and individual practices. International Journal for Academic Development, 17(1), 5-18.
- Jacobsson, A., Ek, Å., & Akselsson, R. (2011). Method for evaluating learning from incidents using the idea of “level of learning”. Journal of loss prevention in the process industries, 24(4), 333-343.
- Taylor, J. (2006). Evaluating mobile learning: What are appropriate methods for evaluating learning in mobile enviroments. Big issues in mobile learning, 1(3), 25-27.
- Kay, R. (2011). Evaluating learning, design, and engagement in web-based learning tools (WBLTs): The WBLT Evaluation Scale. Computers in Human Behavior, 27(5), 1849-1856.
- Horton, W. (2001). Evaluating e-learning. American Society for Training and Development.
- Meo, S. A. (2013). Evaluating learning among undergraduate medical students in schools with traditional and problem-based curricula. Advances in physiology education, 37(3), 249-253.
- Attwell, G. (2006). Evaluating E-learning: A Guide to the Evaluation of E-learning. Evaluate Europe Handbook Series, 2(2), 1610-0875.
- Bell, C., & Harris, D. (2013). Evaluating and assessing for learning. Routledge.
- Syverson, M. A., & Slatin, J. (2010). Evaluating learning in virtual environments. Learning Record.
- Laurillard, D., & Ljubojevic, D. (2011). Evaluating learning designs through the formal representation of pedagogical patterns. In Investigations of e-learning patterns: Context factors, problems and solutions (pp. 86-105). IGI Global.
- Lavesson, N., & Davidsson, P. (2007). Evaluating learning algorithms and classifiers. International Journal of Intelligent Information and Database Systems, 1(1), 37-52.
- Hein, G. E. (1995). Evaluating teaching and learning. Museum, media, message, 1, 189.
