In this post, I’m going to tell you five ways in which evaluation questions can help evaluators.
Like ship
captains of yore, program evaluators rely on the stars to get where they’re
going. Well, not stars, exactly, but a few key questions. Both serve pretty
much the same purpose — they help you navigate, help you to
get where you’re going. In an ocean of data, where you can find yourself
submersed in too many choices, these key questions, commonly referred to as
“evaluation questions,” are your lodestars.
Good evaluation
questions will guide you in making decisions, ensuring that you are heading in
the right direction. After over 15 years of conducting evaluations, this has
become a truism I swear by. Now, to see how far we can stretch this metaphor
before it snaps, imagine that the ship, the crew and the navigational
instruments represent the resources and methods the evaluation team has to work
with. The evaluation questions are what guide the team.
Who determines the evaluation questions?
With most
evaluations, evaluators are hired to address a pre-determined set of queries.
These are normally provided by the client and embody what the client wants to
know about the program, the intervention, the project, the policy, or whatever needs
examining.
When the
work is done, the analysis conducted and report is submitted, what people will
want to know is “What are the answers to the questions we gave you?” Even if
you don’t like the questions, you need to find a way of answering them.
If the
client hasn’t developed the evaluation questions already, then the evaluator
can propose them, based on the client’s objectives. Sometimes the questions are
not clearly thought-out, or maybe they are difficult to answer. That applies
particularly to questions regarding a program’s sustainability. How can you
answer that in circumstances when the program is far from completed?
Evaluation questions are not the same
as interview questions
Evaluation
questions are not the same as interview
questions, which are what evaluators use when interviewing people, such as
beneficiaries, key informants, program implementers and so on. Interview questions, for example, might be
those asked by a police officer investigating a murder. The police might ask the suspect: Who is the
murderer? Why did he do it? And, Where is the weapon? Evaluators don’t ask the people involved in,
or benefiting from, the agriculture project, “Was the project effective?”
Interview questions are more specific, a way of collecting multiple data points
which will inform the body of evidence.
In
contracts, evaluation questions tend to be more along the lines of: Are
stakeholders satisfied with the program? Is it sustainable? How effective is it
at achieving its objectives?
Nevertheless,
for both cops and evaluators, it comes down to asking the right questions in
order to collect the evidence they need.
Questions should drill down from the
evaluation objectives
The evaluation
questions should not only embody the evaluation
objectives, they should drill down into those objectives. They need to be specific.
Evaluation
objectives can be broad, and open-ended. They are useful for explaining why the
evaluation needs to be conducted, but not as useful for developing a
methodology. For example, if the evaluation objectives are “To assess the
project’s effectiveness” or “To draw lessons about the project,” evaluation
questions should be much more specific. They should ask, for example, “Are
women farmers using the new technology as intended?” or “Do stakeholders
consider the hands-on technical assistance they receive to be effective?” However
the questions are formulated, evaluators almost always have an opportunity to
review them and propose modifications. I recommend doing this as early as
possible in the process.
How do evaluation questions help the
evaluator?
There is
plenty of material out there providing guidance on developing and selecting
good evaluation questions. That is not the subject of this post. Instead, I’d
like to point out that there are multiple
ways in which questions, once decided upon, can be extremely useful to the evaluator.
Evaluation questions
are crucial to keeping you on track and staying relevant to your topic. Sometimes
there is a temptation, while in the field, to go off on a tangent. For example,
the substantive or technical aspects of a project are often very interesting in
and of themselves. You or your colleagues may get caught up in discussions on different
types of irrigation water pumps, the political roots of the disparities between
northern and southern regions, or some other issue. That’s all good to know for
context, but such lines of inquiry shouldn’t distract from your main purpose.
That’s not what is being asked of the evaluator.
Regardless
of how the questions are generated, once agreed upon, they become your guide, serving
your effort in a number of valuable ways. The questions should help you with these
five aspects of your evaluation:
What issues to focus on
What evaluation methods to use
Where and whom
to collect the data from
What interview questions to ask people
How to structure and draft the final report
So, keep the
questions close at hand, and check in with them regularly. Use them to guide
you and help make decisions.
Conducting an evaluation is far from being a gentle
boat ride down the river. (If it were that boring, I would have bailed out long
ago). No, it is often difficult, sometimes treacherous, and (predictably) full
of uncertainties. Almost inevitably there is someone or something — on the client side, among the program
stakeholders, or even on your team — that
will make your life a challenge. Don’t let that distract you. Factor those
challenges into your work. With your evaluation
questions to help you navigate, you’ll know where to set your course and be
able to focus from there.
The year 2018 is over (thank goodness) and we have a chance
for a fresh start. For many of us that means time for personal stocktaking.
What did you accomplish last year? What did you learn? How can you apply those
hard-won lessons to the coming year? Should you keep striving to outdo yourself,
or should you settle for what you’ve got and ease into the comfort of routine?
Rifling through the mental files in my “2018 evaluations” folder,
I’ve come up with a few of my own lessons. The one I’ll share today is this: One
thing you can count on is that you can’t always count on people. And you need to prepare for that.
As I’ve observed in an earlier post, we live in a world
where professional failure is more common than conventional wisdom would allow.
Failure is also less interesting than is portrayed by the media and in the
self-help industry. It can become a serious headache, however, when it is your
fellow team member who is doing the failing. I can use myself as a prime
example: I don’t always live up to my own professional expectations. It won’t
come as a shock to readers that people are not always up to the task. The
question is, how do you handle it?
First, let’s get a few obvious things out of the way. Humans
are complex, multi-faceted and, not infrequently, multi-talented. This is a
marvelous thing, accounting for some truly astounding cultural, engineering,
and intellectual feats that have enriched life on this planet. Indeed, in many professions,
it is assumed that employees bring multiple talents to the table. We are not like
robots, programmed to do only one or two tasks at a time. This truism applies
very much to the evaluation field, where evaluators are called upon to deploy a
range of both soft and hard skills.
The fun starts when you suddenly discover that key talents are
missing from a team member. While it is rare that a new team member is brilliant
across the board, most bring at least basic levels of competence to the table.
Most score at least a seven on a 10-point scale across the range of necessary
competencies. But every now and then, someone doesn’t. They’re a “one” or a “two”
in some important area. That’s the thing with being human. We may be
multi-talented, or at least multi-capable, but we also come with built-in
limitations, which sometimes leads to a giant team-implosion. Oops!
What competencies are we talking about? I would offer that, in the evaluation field, you must be able to:
communicate comfortably with others;
put together words, sentences and paragraphs in
a clear and logical manner;
analyze the information you have collected;
collaborate with others like a mature and
responsible adult;
be pleasant and respectful;
do what you say you will do; and
manage your time and priorities.
On top of these soft, but necessary skills, you may also be expected to be equipped with technical skills and experience in:
the sector being evaluated, i.e. agriculture, education, environment, gender, etc.;
qualitative or quantitative evaluation methods; and, if applicable;
effectively leading a team.
Nothing listed above is rocket science, that particular
field generally not falling within the scope of international development
projects. You still find yourself surprised, however, when a fellow team member
is —
how to put this delicately? — totally incompetent.
Of course, the safest solution is to only work with people you
have worked with before, and whom you can count on. For individual consultants,
however, that is a luxury. Instead, what is more typical is that you join a new
team on almost every new assignment. Every year, for example, I end up working
on maybe half a dozen different teams, the majority of which are composed of
folks I have never laid eyes on. On the one hand, it’s a great way to meet
people, make new friends, and learn from your peers. On the other hand, you can
end up in some frustrating and stressful scenarios.
I’ve had experiences where it soon became obvious that a team
member had pretty serious deficiencies in the interpersonal skills department.
For example, Team leader Mr. A, a very plausible stand-in for Ricky Gervais in
the TV comedy series The Office, would
spend the first 10 minutes of a meeting boasting about his own experience and
often end the meeting by insulting the people on the other side of the table. Other
times you get a bad case of weak ethics and poor writing skills, as with Dr. B,
a native English speaker, who couldn’t write proper English despite her
academic pedigree. When I came across passages that were surprisingly
well-written, a quick check on Google revealed she had been happily
plagiarizing them. (Always good to find out that kind of thing sooner rather
than later.) Or someone might impress you in person, but not on paper. Local team
member Ms. C knew the sector and country very well and asked the right questions
during stakeholder interviews, but couldn’t string two sentences together in a
logical way in a report. These were all setbacks which it fell to me to remedy,
through many hours — and sometimes days — of extra work.
I have to admit that I only had the pleasure of working with
one of these people in 2018; I’d worked with the others before that. But it was
last year that it finally hit home: I needed a coping strategy for the next
time this happened.
So, what to do on occasions when capabilities are missing? For
starters, if the shortcomings are yours,
it’s a good idea to reflect and take concrete actions to perform better. If the
shortcomings belong to others,
cursing under your breath or venting to your significant other can have a wonderfully
calming effect, but may not be enough to rectify the situation. Is it possible
to overcome such defects through mentoring or teaching? Unfortunately, I have found
that it is totally unrealistic to attempt to build the capacity of someone (even
if you are in a position to do so), over the course of a single assignment. In
any case, you’d first need to spell out their failings to them. That could be
pretty awkward, right? Furthermore, you don’t really have much time for
capacity building — you need to get the bloody job done.
What you need is a back-up plan, especially if you are
ultimately responsible for the work (if you’re the team leader) or because you
were asked to pick up the slack (by the team leader). Here are three
suggestions:
Build in a
time buffer: Provide enough slack in your schedule to take into account the
extra time that you might need to address the shortfall. For example, if you
think a task will take two weeks, try to allocate three weeks.
Build in a
human resource buffer. Identify persons, either on the team or not, who could
step into the breach. Maybe the organization that put the team together (if you
are subcontracted) has the resources to bring on extra help.
Build in a
mental buffer: Prepare yourself not be surprised or upset when colleague X lets
you down. Unless you’ve worked with them before, and therefore know their
strengths and weaknesses, assume that people have a least one weakness, and
that it will impact the work at hand.
In a word, contingencies!
Let 2019 be a year of contingency planning. The plan comes
with its own reward: if you have a
decent contingency plan, you will end up with more time, energy, and even
inspiration, to focus on the interesting and fun stuff.
