Section 5: Evaluating projects and reflecting on project manage print

Wed 26. Oct 16 21:30

Module 4: Evaluating projects and reflecting on project management

Overview of the unit
When you have completed your development project you should carry out two activities.

  • Evaluate the success of the project, its impact upon the school, what was achieved and what was not achieved, and what contributed to the success of the project and which forces acted as barriers.
  • Reflect upon your personal learning and the competences you have developed as a result of completing your project.
These are separate activities and you should take care not to confuse them.

No doubt by this stage you will be giving consideration to how you will go about writing the project report you will submit for assessment. In this short video, Ali provides some useful advice on how to write an effective report.

Learning outcomes
On successful completion of this unit, you will be able to:

  • demonstrate knowledge and understanding of key concepts and processes related to the evaluation of development projects
  • apply these concepts and processes to the evaluation of a development project
  • assess the development of your own professional competence through the leadership and management of the project
  • communicate your evaluation of the project and reflection on your professional development in an appropriate manner for different audiences



Closing a project

Project closure
At the end of a project it is expected that the project leader or manager will produce a report summarising what has been achieved. The title of this document may vary between organisations, but the content covered is often similar. Below is an (amended) example of the DfE's guidance to its staff on what they should include in their project closure reports


Project closure reports

The project closure report is produced at the end of the project, and provides the SLT with a review of the overall project and an assessment of how successfully the project has met its objectives.

This report will be presented at a SLT meeting, and must be signed off by them as part of the process of formally closing the project.

The areas that are usually covered in a project closure report are:

review of the overall project
an assessment of how well the project has met its objectives as defined in the Business Case
residual issues, for example areas of work that still need action or monitoring. Agreement must be sought from whichever team or individual will be taking these issues forward
lessons learnt, for example what would you do the same, what would you do differently based on your experiences


separator-2.gif(Department for Education, 2012)

Back to the beginning
The first step in the process of reviewing a project is to refer back to the business case produced at the start of the project. This should contain the success criteria that will allow a judgement to be made about the success or otherwise of the project. It also serves as a useful reminder of where the project's journey began – it is not unusual for the final outcomes from a project to vary from what may have been originally intended.





Evaluation

Project evaluation
In this and the following topic, we discuss how to approach and carry out project evaluation. The methods used for evaluating large and small projects have much in common, but you should remember that your project is small-scale and you should use the information which follows selectively. You are not expected to adhere to the standards that would be expected from a piece of university academic research.

What is evaluation?
Evaluation is measuring success in a systematic and objective way. Evaluation focuses on whether the project was effective, achieved its objectives, and the outcomes had an impact. For project outputs, evaluation might focus on whether the outputs are useful, meet user needs, and perform well.

Experts on evaluation often make a distinction between formative and summative evaluation. They are similar, and the main difference is timing:
  • Formative evaluation: performed during the project/programme to improve the work in progress and the likelihood that it will be successful.
  • Summative evaluation: performed near the end of the project/programme to provide evidence of achievements and success.
We focus here on the summative evaluation you are carrying out at the end of your project.

Aims of summative evaluation
The aims of the summative evaluation might be to:

  • assess whether the project achieved its aims and objectives
  • assess the impacts, benefits, and value of the project
  • identify achievements and stimulate discussion
  • synthesise knowledge from the project and lessons learned
  • identify future developments
You should decide which of the above items apply to your project – it could be that you want to focus on one or two items or you may want to learn about all five items.

What will you evaluate?
The factors to evaluate will depend on the project. In most cases, they will focus on how successful the project is at achieving what it set out to do. This might include:

  • achievements against aims and objectives
  • stakeholder engagement
  • outcomes and impacts
  • benefits
  • effectiveness of the project – was it value for money?
You will also wish to reflect upon the effectiveness of the processes you established for managing and delivering the project, and we consider this later.

Questions to address
It can be useful to list the specific questions your evaluation will answer. Focus on questions that really need to be answered to demonstrate success. Think about what stakeholders want to know. Make sure that the questions can be answered unambiguously. Avoid questions where the answer is likely to be 'maybe'.

Typical questions you might consider include the following:

  • Have objectives been met? For example was the project delivered on time, in budget?
  • Have outcomes been achieved? For example has a new reception facility been established?
  • What are the key findings?
  • What impact did the project have? For example do parental surveys show that they approve of the new reception facilities?
  • What benefits are there for stakeholders?
Collecting additional data
In order to answer the questions listed above, you may need to collect additional information (data).

This can be broken into two types:

  • quantitative data
  • qualitative data
Select the tabs for detail about each of these.


Qualitative data
This is sometimes referred to as 'soft' data because it is drawn from the opinions and observations of individuals and is not based on 'hard' facts. This type of data is just as valuable, however, and as well as helping us to assess the outcomes of a project it can also help us to understand the factors which contributed to the success or failure of the project.

Qualitative data can be gathered in a number of ways:

  • Interviews: these are conversations, typically with one person. They may be structured, semi-structured, or unstructured. They are useful for exploring opinions and issues in depth on a one-to-one basis.
  • Discussion groups: these are interviews conducted with a small group of people (around four or five). They allow you to get a range of views on a project.
  • Observation: observation is just that, observing what is happening and recording this. For example, following a project to improve the quality of the school's outdoor play spaces it would be useful to spend some time observing how the children are behaving in the new areas.

Activity

4.1: Evaluating impact

For this activity, refer back to the success criteria that you developed at the start of your project, then work through the following process.

Decide on a small number of quantitative and qualitative methods that you can use to gather information on the impact of your project.

Use these methods to carry out an evaluation exercise.

Describe the conclusions from your research.


Using evaluation results
Evaluation will demonstrate that you've achieved your aims and objectives, the work was useful, and there are benefits for stakeholders. Demonstrating that the work was useful and has benefits could be useful if you wish to develop a further project.

Success has been mentioned frequently, and you may wonder what happens if a project fails. In reality some projects don't achieve all their objectives. There may be circumstances beyond your control that affect what the project can achieve. This will of course be a disappointment to you and your school. However, for the purposes of this programme, we learn as much from 'failure' as we do from success.





Reflecting on your management of the project

Before you embark upon reflection, it is important to think about how we develop reflection skills. Reflection as a general skill can form part of the learning process and the ability to use this will enhance the quality of your learning.

Here are some pointers that will help you in developing and using skills of reflection. Reflection goes beyond just gaining knowledge, and challenges us to:

explore the foundations of our knowledge
strengthen our understanding
increase our self-awareness of the values and attitudes that influence our knowledge

Reflection is a dynamic process. It is not about being passive, staying where you are and looking back – but an active engagement with knowledge and experience. So, in reflecting you are able to construct new and deeper understanding and to articulate knowledge in a more meaningful way.

One of the most commonly quoted models for understanding the process of reflection is a model developed by Boud, Keogh and Walker (1985). This model highlights that experiences in learning combine behaviour, ideas and feelings and all of these aspects need to be examined in the process of reflection.

The process has three stages:

  • Returning to experience – a detailed recounting of recollection of the events. This will entail you looking back on your project and consider some of the key events and issues.
  • Attending to the feelings, both positive and negative that have been prompted by the experience. In this area you will consider the barriers and opportunities you faced and think about how well you managed them, for example did some of the barriers cause you to feel frustrated? What surprised you or pleased you about how other people reacted to key issues?
  • Re-evaluating that experience in the light of the first two stages, for example thinking about what you would do differently with the benefit of hindsight.
What are the skills and qualities needed to be reflective?
Reflection requires an approach that is:

  • curious – there has to be a willingness to ask questions, to want to find out
  • patient – not jumping to conclusions – just as there are no simple answers, there are often no instant answers either and your ideas and understanding may well change over time
  • open – to absorb what is happening
  • honest – you need to be honest with yourself and this includes being honest about doubts and uncertainly or lack of knowledge
Structuring your reflection
An important factor in the success of your project will be the way in which the project was set up, managed and monitored. It is important that you reflect on these processes to determine any lessons to be learned for you as a leader and manager.

You may find it useful to refer back to any activities you completed, and if you have kept notes on your reflections and learning during the project, these will also be useful at this point.

Approaches to structured reflection
There are a number of different ways in which you can structure your reflections.

One approach is to break your reflections down by the stages of the project:

  • identifying and negotiating the project
  • developing the project plan
  • leading and managing the plan
  • evaluating the project
You can then identify key learning points.

Another approach is to reflect on the different skills or competences you used in leading and managing the project. These might include:

Untitled84.png

When you reflect on these items, try to identify specific examples of how you have grown in competence, for example 'as a result of trusting a member of the team to take responsibility for part of the project, I now feel more confident in my capacity to delegate tasks in the future'.

Here are the types of questions you might ask yourself. The list is not exhaustive but it will provide you with a guide:

  • Initial needs analysis: How useful was this process? Was I sufficiently thorough?
  • Gaining initial buy in from stakeholders: Did I achieve this, did it help the project?
  • Risk assessment: Was I sufficiently thorough? Was it useful?
  • Project plan: Was this sufficiently detailed? Was I overambitious (or underambitious)?
  • Project resourcing: Were my estimates of costs accurate?
  • Business case: How effective was I in persuading key stakeholders? Was there anything missing?
  • Success criteria: Did I choose criteria that were easy to measure at the end of the project?
  • Team working: How effectively did I lead the team? (You may wish to complete the Team Effectiveness survey to help you form a judgement.) How well did I engage the team in decision-making?
  • Delegation: What tasks did I delegate? How successful was this? What issues arose?
  • Communication within the team and with key stakeholders: How effective were the methods I used?
Guidance on presentations
As part of the assessment for this module, you are required to prepare and deliver a 10-minute PowerPoint presentation online. Here are some tips on how you might prepare for this, and techniques to use, and avoid, when presenting online.

The focus is your evaluation of how you led and managed your development project. You need to:

  • describe the school setting
  • explain why you chose the project, and what it aimed to do
  • give an overview of planning and carrying out the project
  • explain what was achieved
  • evaluate how effective it was
  • reflect on your own professional learning
When you're getting ready for the presentation, keep it short – remember "less is more". Just because you know a lot about your school setting, it doesn't mean that your audience need to.

Try to think visually: this way of presenting puts much more emphasis on visual images, so think about what you're trying to get across and keep it focused. Use bullet points instead of blocks of text, and make sure that graphs and charts are easy to understand. You may be able to use images instead of words sometimes.

There are some things you might want to avoid:

  • animated slides
  • presenting to the computer (remember there are people at the other side, even if you can't see them)
  • reading notes: you'll sound more interesting if you are speaking naturally
Instead, think about yourself as a radio talk show host: use clear language, avoid slang and hesitation, and vary your tone and pace.

Think about using annotation tools as you speak, to highlight key information – this may help you replace the gesturing you might use for a face-to-face presentation.

Finally, practise your session to fine-tune your presentation and get used to delivering without seeing your audience. Try and persuade some friends or colleagues to let you test it on them online, and ask them to give you feedback afterwards, or you could record an online presentation and play it back.

There's an art to using powerpoint effectively. Make sure you avoid the trap of using gimmicks that detract from your key messages. Try an internet search on 'Death by powerpoint' for some examples of what we mean here.

Completing the checklist

In column 2, you should record how you have developed this particular competency during the project, and support your assessment with an example of what you did.

In column 3, you should record your future development needs. (It may be that not all of the competences are relevant to your situation.)

Once you have completed this activity, return to the professional learning portfolio (PDP) that you started in unit 2 of Development Module 1.

Update your PDP by recording progress against objectives that you set for yourself initially and extend it by adding new areas for development, based upon your learning over the last six months.

You should also reflect on your learning journey in your response to the assessed task for unit 4 of this module. Add your revised PDP as an appendix in the module assessment template




Activity

4.2: Reflecting on the competences you have developed

In order to structure your reflections on the competences you have developed as a result of leading and managing your project it is useful to work from a checklist.

One approach is to download a copy of the National Association of School Business Managers (NASBM) competency framework and to use this to assess your development (see 'Resources' below).

Another approach is to use the checklist attached below (Cometency). If you wish you could combine both approaches
Competency.docx

Enablers and blockers
When reviewing these items, you may find it useful to divide them into 'enablers' and 'blockers' which either contributed to its smooth operation or obstructed its progress. Enablers and blockers are sometimes divided into two categories – those that occur because of the attitudes and behaviours of people, and those that occur because of the situation or context in which they are working. Some examples are provided below.


Enablers
Positive support from DH who consistently argued the merits of the project.
Opportunities to regularly communicate with staff on the purpose and progress of the project.
Regular team meetings.
Sharing good practice with other schools saved time and increased project's capacity

Blockers
Project was an 'add on' to existing workload, insufficient time for project planning.
School had several other projects running at the same time, this diverted resources.
Delays and uncertainty in announcement of school budget.
Negative attitudes of small number of governors.


Activity
4.3: Enablers and blockers

What are the enablers and blockers for your project?
Record the most significant enablers and blockers for your project.

Lessons learned
Finally you could describe your learning in terms of 'Lessons learned'. This could include instances of things that were successfully managed as well as lessons deriving from problems and issues you had to deal with. Listed below are some examples of lessons learned from previous projects.

  • Ensure there is adequate resource in the project team – and that the project team's attentions are not diluted by other responsibilities.
  • Ensure external funding is secured prior to the start of the project.
  • Focus on early planning, and be realistic about what can be achieved in the available time.
  • Identify necessary skills and allow some time for training, where necessary, for project team members.
  • Establish clear channels of communication and contact points.
  • Involve appropriate key stakeholders from the earliest stage in the planning process and then consistently throughout the project.
  • Build in time for regular reviews of the risk register

Bookmark and Share