In a world of intractable problems that often seem to demand complex or high-tech solutions, what is one simple way to cut through the layers and build a consensus to move forward? Start talking…face to face.
Although not a guaranteed remedy, getting people with different interests in the same room to hash out an issue is a tried and true method of dealing with all sorts of impasses. Bringing different sides to the table to sit down and talk together, or creating forums for discussion is a powerful, time-tested tool, whether the parties are in a dispute, have different interests, are communicating poorly, or are merely not paying attention to each other’s concerns. Discussing issues over a meal is even better. Apparently, sharing food has a positive impact on negotiations, according to The Economist.
It is striking how often poor communication comes up as an issue in the news, and — just as striking — how often good communication is considered part of the solution to problems large and small. From international summits to talks on climate change to committee meetings, to mediation as an alternative to court proceedings, to couples’ therapy, creating the space for conversation is at the heart of many solutions. In Eastern Europe, I once evaluated a USAID project whose purpose was to improve economic governance by promoting public private dialogue (PPD). For the uninitiated, PPD is a thing — with its own website , charter, and handbook. About $20 million in project funds were spent in convening stakeholders from government, business and civil society. These resources also covered organizing meetings and retreats on reforms in different sectors. In addition, the project complemented the dialogue forums with analytical work on individual reforms. Guess what? The public private dialogue approach was, by and large, perceived as effective by all three stakeholder groups. The PPD platform did ensure that in many cases the reforms incorporated the interests of the different actors, and, by doing so, helped move them forward. The initiative was seen as one of the most important influences on reforms.
Just getting parties to talk seems like such a simple thing, compared with all the thorny problems facing society, and the sophisticated solutions being promoted (artificial intelligence, anyone?). And talking really means talking, not texting, videoconferencing, or some other forum of digital communication. I’m not denying that these have their uses, but nothing seems to be as effective as face-to-face meetings for building rapport. How else to explain the estimated 462 million business trips taken in the U.S. (in 2017)? Think of the awkward pauses during conference calls, or the unreliable technology that shuts you off in the middle of a videoconference. But even if these issues were addressed somehow, there is a qualitative difference to talking to someone sitting in front of you.
Pay attention and you’ll see that meeting in order to talk is a solution to many, many problems. In her book Leadership in Turbulent Times , Doris Kearns Goodwin recounts how Teddy Roosevelt brought mine owners and workers together to resolve the months-long 1902 anthracite coal strike by United Mine Workers of America, which was threatening to cause major social disruptions. A recent Financial Times(behind paywall) article by Sylvaine Chassany describes French President Emmanuel Macron’s initiative to connect with citizens by holding conversations around the country — organized by En Marche party moderators. Although the word “engagement” is overused, Macron’s motive was to enable people to share their concerns directly with the government. Of course, resolutions to virtually all violent conflicts, are, at one point or another, sought through multi-party talks: North Korea, Iran, Ukraine and so on. Of course, it doesn’t always work out.
There are alternatives. One example would be the one-sided resolution delivered by crushing defeat if you’re on the losing side. There is a school of thought that this approach is actually preferred, as it is stable and final. Continued non-cooperation, or stalemate, is also very common. Or how about endless conflict? Examples of the latter that come to mind are the Israel-Palestine and Afghanistan conflicts. In fact, Wikipedia lists 55 ongoing conflicts that have lasted at least 20 years apiece. Whether these alternatives are preferred or not, they are depressingly widespread. Winston Churchill reportedly said “to jaw-jaw is always better than to war-war,” or words to that effect.
What talks can do
Talking things over with others is a good idea even when you are not at war. It can serve all kinds of ends. Beyond resolving conflicts, it is a good a way of sharing news or information (e.g. the press conference), obtaining information (e.g. interviews), exploring opportunities (e.g. business meetings); promoting ideas (e.g. during campaigning by politicians and activists), or just building trust by bringing in the personal element, when previously it was all about power or preferences. Talking is, admittedly, not necessarily a solution in and of itself. However, it seems that it has some attributes that are conducive to finding solutions. What might they be? I propose the following:
Just showing up for talks demonstrates a willingness to listen to the other side. It signals a cooperative posture, giving both sides confidence that they are not wasting their time.
By their nature, talks force the other side to listen to other views, positions, perspectives, and whatever else they need to get off their chest. Otherwise, outside of talking, the information the other side wants to share may primarily be seen via propaganda, or (possibly biased) media like the press or social media. Listening to others express themselves will probably lead to their views being considered, at a minimum.
Talks can ease tension. They allow different sides to express their positions.
A discussion forum can empower people who felt their voices weren’t being heard. This can help even the playing field
The brainstorming aspect to seek solutions can generate proposals and help chart a way forward.
It is an efficient way of sharing information. Talks focus on the essentials, compared to written reports which may contain much useful information, but much that is extraneous to the purpose of the meeting.
So the next time that you find yourself scratching your head, or worse — banging it against a wall — reach for the low-tech, low hanging fruit. Advise people to sit down together and work it out verbally. Way before Microsoft capitalized and appropriated the term, the Bible’s famous opening line was: “In the beginning was the Word.”
I’m often
asked what life as an independent consultant is like, typically by people who
have spent their working lives ensconced within an organization…while
fantasizing about breaking free.
What to say?
If you have ever hankered to play in a one-man band, put on a one-woman show, or
strike out on your own as a gun for hire, then yes,
independent consulting may
be for you. It can satisfy those urges — to a degree. It also demands
somewhat less creativity and risk-taking than the aforementioned vocations, while
allowing you to earn a pretty decent living.
…and the limits of that appeal
Independent
consulting is not, however, just about indulging in appealing work arrangements.
You still have plenty of obligations. The less that is imposed upon you by
others, e.g. managers, the more you have to impose upon yourself. You need to
substitute internal motivation for external motivation. To quote Eleanor Roosevelt,
with freedom comes responsibility. A big part of independent consulting is
about managing, or coping with, that freedom.
There are conditions
attached to being a free-wheeling consultant. This includes the need to be
self-motivated. You must also develop the discipline to manage your own
schedule, set your own internal deadlines, and find substitutes for the structure
and social interactions that a normal workplace provides. And although you will
be liberated from the yoke of the worst bosses, you’ll also want to avoid
becoming your own worst boss. In other words, there are tradeoffs.
Is there
anything in life that doesn’t involve trade-offs? No, there is not.
One hundred tasks
To return to
the one-man band analogy, consulting means doing a lot of tasks that normally would
be carried out by others. In fact, you would be delegating those chores right
now if you hadn’t turned down that plum managerial position last year.
Out of
curiosity, I recently made a list of everything I do, professionally, over the
course of a year. I came up with over 100 different tasks. (And yes, writing
this blog is one of them.) Those tasks that are normally divided among supervisors,
subordinates, and specialists — now fall in your lap as a solo
independent contractor.
You may have
a fantasy of specializing brilliantly at just one thing, of becoming, say, a
world-renowned expert in energy regulatory policy or on malaria. That’s a
luxury few consultants can afford. Going narrow means going deep, but going too
deep can become a problem if demand for that particular skill dries up, even
temporarily. I myself am a generalist. It took me years to figure out a balance
that works for me. Like Goldilocks, you want to build up capacities in the
“right” number of areas — not too many and not too few.
Many roles, one employee
As a
consultant you don’t split your identity, but you play many roles: supervisor,
researcher, data analyst, writer, administrative assistant, accountant,
business development specialist, PR person and so on. You are, to a greater or
lesser extent, going to have to internalize the various positions that
constitute, well, a small firm, while staying sane. What I mean by
“internalizing” is that, instead of different employees performing specialized
tasks, you have to do them all yourself, in effect consolidating the different
employee roles within yourself. To keep things moving, you’ll find yourself
almost continuously switching back and forth between them.
At a
minimum, you need to think of yourself as your own boss and your own employee.
A boss needs to manage the budget, make sure deadlines are met, ensure quality
control, motivate employees, and deliver results. An employee needs guidance,
instruction and direction, and to produce what is asked for. Sometimes I, in employee
mode, feel tired or lazy, or just totally bereft of inspiration. So, in manager
mode, I have to cajole, or lay down the law, to get my “employee” to complete
the darn task. Try out the carrots and sticks that work best for you.
Breaking it down
Here’s a tip:
depending on how your brain works: you may want to cut your work up into small slices,
and alternate between them. I’ve found that frequently changing tasks
throughout the day can be invigorating. When
you get tired of analyzing the data, you can revise your CV, after which you
can review that report, go to a meeting, review the background literature for
your next project, and answer emails in between. In this way, you give the different
parts of your brain a rest while still getting plenty done.
Other people
may prefer to set aside large blocks of time and complete a task in one
sitting. I really admire that. I know someone, a phlegmatic fellow, who will
sit down for eight or more hours at a time, with barely a break, staying up
into the wee hours of the morning to ram through a task to meet a deadline.
The outsourcing option
Another
option, especially if you are overwhelmed, is to consider outsourcing some of
your work. Depending on whether it makes financial sense, you can pay other
people to do some of your tasks. You probably have to absorb the costs as part
of your pay, since you are contracted as an individual. But it may be worth it,
if you find it allows you to take on more work and it improves the quality of
your outputs.
The key is
to find good people — who are reliable and deliver the
quality you want. Over the years, I have subcontracted work to research
assistants, editors, translators, and graphic designers on a short-term basis.
Find out who you are
The relative
freedom from the constraints of a nine-to-five life suits some people better
than others. You need to figure out whether you have what it takes to live
without fetters. It comes down to temperament. Do you do your best work in bed,
like Marcel Proust and Mark Twain? Be my guest. Does the ambient noise and
languid activity of a coffee shop help your neurons to fire? There are probably
a dozen choices within walking distance from where you live. Or do you find
that you do your most penetrating analysis in a beach house in Bermuda? Experiment
until you figure out an optimal routine.
And if you
discover that you are, in fact, your own worst boss, that management position
might start looking attractive. After you are back in an office, you can then hire
those freewheeling independent consultant colleagues to do the work for you…
The year is 2016. I’m sitting on the cement floor of a
home in a village outside of Tulajpur, deep in the Indian state of Maharashtra.
I’m surrounded by more than a dozen women entrepreneurs taking part in a focus
group.
My American colleague “Lisa” and I are here conducting
a USAID evaluation, as we crisscross India to assess similar projects, along
with other team members. We’ve come to this place to ask entrepreneurs to share
their experience with an innovative program that promotes the use of cheap and
affordable solar-powered lamps, as well as other solar-powered gadgets. These savvy
women, who run small shops built into the front of their houses, are a key link
in the supply chain that helps off-the-grid rural Indians light their homes. The
focus group discussion was animated and friendly, with any cultural barriers
fading to insignificance.
Two things struck me. First, all of them had
smartphones, which they were using to snap photos of us. Normally, it is the evaluators who photograph focus group
members, so you could ask, who was observing whom? Although it was a relatively
poor village, the women in our group clearly had above-average income levels.
In 2016, only about one in six Indians owned a smartphone, and most of them were
in urban areas.
The second notable thing was the latrine situation –
there was none. When our meeting ended, Lisa asked if she could use the toilet.
She didn’t get a clear answer. After she asked again (the meeting had lasted three
hours, after all), the woman hosting the meeting then reluctantly showed Lisa an
empty room at the back of the house, which had a sink, but no running water,
and a small hole in the floor. This was not a Turkish-style squat toilet with
the ceramic footholds flanking a hole, either. Lisa said the room looked
nothing like a bathroom, and she was not about to repurpose it. When she inquired
where the women did their business, the homeowner sheepishly indicated an
empty, rubbish strewn lot next door. That
was the toilet, so to speak.
The incongruity of people leapfrogging to smart
phones, while still practicing what in development circles is called open
defecation, was striking. Although I spend a good bit of every year in
developing countries, it still took me aback. The Tulajpur women were
simultaneously stuck in the past – a noxious, undignified and unhealthy past –
and yet benefiting from a high technology “future,” one which would have seemed
like science fiction to their parents.
As this insightful article in National
Geographic on the topic of latrines and open defecation makes clear,
the problem of persuading people to build or use latrines is complex. Open
defecation is still how 1.2 billion people on the planet relieve themselves,
and almost half of those live in India. The practice comes with serious health implications
as it raises the risk of infection, disease and death, especially for children.
The issue isn’t the availability of the actual
physical object. Many latrines have been built, but remain unused. A complex
array of factors come into play: tradition (historically, of course, all of
humanity lacked toilets), to the still-extant caste system which makes it a
disgrace to clean out latrine pits for anyone but the Dalits (formerly known as
untouchables), to lack of sewerage systems, to a supposed aversion to enclosed
spaces while defecating. Getting families or villages to build just a pit
latrine – or even use one that is
built for them – remains difficult. A lot of outhouses have fallen into disuse,
or have been repurposed as storage sheds.
This is not to single out India, which has made
tremendous strides on many socio-economic indicators, such as lowering its birth
rate to 2.1 children per woman – definitely a modern phenomenon. I bring up open
defecation and smartphones as examples of how governments and donors sometimes have
very different priorities from the population. Getting people to use a toilet has
major positive health impact. Almost no one argues for “cultural sensitivity”
and just letting people get on with it in this case.
The challenge of getting rural villagers to use
toilets is not that they are living in a sort of isolated, antediluvian Eden,
in harmony with nature, as the solar lamps and cell phones, not to mention
plastic trash strewn everywhere, make clear. I would argue that it encapsulates
three problems common to many development efforts.
First, the preferences of the state do not always
coincide with those of society. To paraphrase an old saying, the state
proposes, but (wo)man disposes. There may be excellent, scientifically-grounded
economic and health reasons to pull people into the modern world, but it turns
out that people have their own preferences, or priorities, such as smartphones.
Indeed, social behavior change has become a field of study in its own right. It
is one thing to build something, but getting people to use it is another
matter.
Second, having a vested interest is a powerful motivator. As every landlord and tenant has no doubt observed, a different attitude comes into play when you own something compared with just using it. That is why many programs that build infrastructure for communities focus on “ownership” as a concept. They require communities to contribute in kind or in cash. Yet often the cost is so high that the community contribution is not more than 5 percent, or less, and that feeling of ownership is concomitantly weak.
Three, when poor countries aspire to the public goods
available in richer countries, they often end up building or buying things that
they can’t afford to use. In a lot of cases, infrastructure, from the most
basic latrines, to enormous projects such as hydroelectric dams, is being built in countries where they lack the
technical and financial resources to maintain and operate them. Funds need to
be accessed and set aside, and procedures introduced.
Unfortunately, we
didn’t get into a discussion with our focus group women on these issues. That
was not why we were there. However, I’m sure they would have given us a valid
reason for their choices, at least from their perspective.
In this post, I’m going to tell you five ways in which evaluation questions can help evaluators.
Like ship
captains of yore, program evaluators rely on the stars to get where they’re
going. Well, not stars, exactly, but a few key questions. Both serve pretty
much the same purpose — they help you navigate, help you to
get where you’re going. In an ocean of data, where you can find yourself
submersed in too many choices, these key questions, commonly referred to as
“evaluation questions,” are your lodestars.
Good evaluation
questions will guide you in making decisions, ensuring that you are heading in
the right direction. After over 15 years of conducting evaluations, this has
become a truism I swear by. Now, to see how far we can stretch this metaphor
before it snaps, imagine that the ship, the crew and the navigational
instruments represent the resources and methods the evaluation team has to work
with. The evaluation questions are what guide the team.
Who determines the evaluation questions?
With most
evaluations, evaluators are hired to address a pre-determined set of queries.
These are normally provided by the client and embody what the client wants to
know about the program, the intervention, the project, the policy, or whatever needs
examining.
When the
work is done, the analysis conducted and report is submitted, what people will
want to know is “What are the answers to the questions we gave you?” Even if
you don’t like the questions, you need to find a way of answering them.
If the
client hasn’t developed the evaluation questions already, then the evaluator
can propose them, based on the client’s objectives. Sometimes the questions are
not clearly thought-out, or maybe they are difficult to answer. That applies
particularly to questions regarding a program’s sustainability. How can you
answer that in circumstances when the program is far from completed?
Evaluation questions are not the same
as interview questions
Evaluation
questions are not the same as interview
questions, which are what evaluators use when interviewing people, such as
beneficiaries, key informants, program implementers and so on. Interview questions, for example, might be
those asked by a police officer investigating a murder. The police might ask the suspect: Who is the
murderer? Why did he do it? And, Where is the weapon? Evaluators don’t ask the people involved in,
or benefiting from, the agriculture project, “Was the project effective?”
Interview questions are more specific, a way of collecting multiple data points
which will inform the body of evidence.
In
contracts, evaluation questions tend to be more along the lines of: Are
stakeholders satisfied with the program? Is it sustainable? How effective is it
at achieving its objectives?
Nevertheless,
for both cops and evaluators, it comes down to asking the right questions in
order to collect the evidence they need.
Questions should drill down from the
evaluation objectives
The evaluation
questions should not only embody the evaluation
objectives, they should drill down into those objectives. They need to be specific.
Evaluation
objectives can be broad, and open-ended. They are useful for explaining why the
evaluation needs to be conducted, but not as useful for developing a
methodology. For example, if the evaluation objectives are “To assess the
project’s effectiveness” or “To draw lessons about the project,” evaluation
questions should be much more specific. They should ask, for example, “Are
women farmers using the new technology as intended?” or “Do stakeholders
consider the hands-on technical assistance they receive to be effective?” However
the questions are formulated, evaluators almost always have an opportunity to
review them and propose modifications. I recommend doing this as early as
possible in the process.
How do evaluation questions help the
evaluator?
There is
plenty of material out there providing guidance on developing and selecting
good evaluation questions. That is not the subject of this post. Instead, I’d
like to point out that there are multiple
ways in which questions, once decided upon, can be extremely useful to the evaluator.
Evaluation questions
are crucial to keeping you on track and staying relevant to your topic. Sometimes
there is a temptation, while in the field, to go off on a tangent. For example,
the substantive or technical aspects of a project are often very interesting in
and of themselves. You or your colleagues may get caught up in discussions on different
types of irrigation water pumps, the political roots of the disparities between
northern and southern regions, or some other issue. That’s all good to know for
context, but such lines of inquiry shouldn’t distract from your main purpose.
That’s not what is being asked of the evaluator.
Regardless
of how the questions are generated, once agreed upon, they become your guide, serving
your effort in a number of valuable ways. The questions should help you with these
five aspects of your evaluation:
What issues to focus on
What evaluation methods to use
Where and whom
to collect the data from
What interview questions to ask people
How to structure and draft the final report
So, keep the
questions close at hand, and check in with them regularly. Use them to guide
you and help make decisions.
Conducting an evaluation is far from being a gentle
boat ride down the river. (If it were that boring, I would have bailed out long
ago). No, it is often difficult, sometimes treacherous, and (predictably) full
of uncertainties. Almost inevitably there is someone or something — on the client side, among the program
stakeholders, or even on your team — that
will make your life a challenge. Don’t let that distract you. Factor those
challenges into your work. With your evaluation
questions to help you navigate, you’ll know where to set your course and be
able to focus from there.
The year 2018 is over (thank goodness) and we have a chance
for a fresh start. For many of us that means time for personal stocktaking.
What did you accomplish last year? What did you learn? How can you apply those
hard-won lessons to the coming year? Should you keep striving to outdo yourself,
or should you settle for what you’ve got and ease into the comfort of routine?
Rifling through the mental files in my “2018 evaluations” folder,
I’ve come up with a few of my own lessons. The one I’ll share today is this: One
thing you can count on is that you can’t always count on people. And you need to prepare for that.
As I’ve observed in an earlier post, we live in a world
where professional failure is more common than conventional wisdom would allow.
Failure is also less interesting than is portrayed by the media and in the
self-help industry. It can become a serious headache, however, when it is your
fellow team member who is doing the failing. I can use myself as a prime
example: I don’t always live up to my own professional expectations. It won’t
come as a shock to readers that people are not always up to the task. The
question is, how do you handle it?
First, let’s get a few obvious things out of the way. Humans
are complex, multi-faceted and, not infrequently, multi-talented. This is a
marvelous thing, accounting for some truly astounding cultural, engineering,
and intellectual feats that have enriched life on this planet. Indeed, in many professions,
it is assumed that employees bring multiple talents to the table. We are not like
robots, programmed to do only one or two tasks at a time. This truism applies
very much to the evaluation field, where evaluators are called upon to deploy a
range of both soft and hard skills.
The fun starts when you suddenly discover that key talents are
missing from a team member. While it is rare that a new team member is brilliant
across the board, most bring at least basic levels of competence to the table.
Most score at least a seven on a 10-point scale across the range of necessary
competencies. But every now and then, someone doesn’t. They’re a “one” or a “two”
in some important area. That’s the thing with being human. We may be
multi-talented, or at least multi-capable, but we also come with built-in
limitations, which sometimes leads to a giant team-implosion. Oops!
What competencies are we talking about? I would offer that, in the evaluation field, you must be able to:
communicate comfortably with others;
put together words, sentences and paragraphs in
a clear and logical manner;
analyze the information you have collected;
collaborate with others like a mature and
responsible adult;
be pleasant and respectful;
do what you say you will do; and
manage your time and priorities.
On top of these soft, but necessary skills, you may also be expected to be equipped with technical skills and experience in:
the sector being evaluated, i.e. agriculture, education, environment, gender, etc.;
qualitative or quantitative evaluation methods; and, if applicable;
effectively leading a team.
Nothing listed above is rocket science, that particular
field generally not falling within the scope of international development
projects. You still find yourself surprised, however, when a fellow team member
is —
how to put this delicately? — totally incompetent.
Of course, the safest solution is to only work with people you
have worked with before, and whom you can count on. For individual consultants,
however, that is a luxury. Instead, what is more typical is that you join a new
team on almost every new assignment. Every year, for example, I end up working
on maybe half a dozen different teams, the majority of which are composed of
folks I have never laid eyes on. On the one hand, it’s a great way to meet
people, make new friends, and learn from your peers. On the other hand, you can
end up in some frustrating and stressful scenarios.
I’ve had experiences where it soon became obvious that a team
member had pretty serious deficiencies in the interpersonal skills department.
For example, Team leader Mr. A, a very plausible stand-in for Ricky Gervais in
the TV comedy series The Office, would
spend the first 10 minutes of a meeting boasting about his own experience and
often end the meeting by insulting the people on the other side of the table. Other
times you get a bad case of weak ethics and poor writing skills, as with Dr. B,
a native English speaker, who couldn’t write proper English despite her
academic pedigree. When I came across passages that were surprisingly
well-written, a quick check on Google revealed she had been happily
plagiarizing them. (Always good to find out that kind of thing sooner rather
than later.) Or someone might impress you in person, but not on paper. Local team
member Ms. C knew the sector and country very well and asked the right questions
during stakeholder interviews, but couldn’t string two sentences together in a
logical way in a report. These were all setbacks which it fell to me to remedy,
through many hours — and sometimes days — of extra work.
I have to admit that I only had the pleasure of working with
one of these people in 2018; I’d worked with the others before that. But it was
last year that it finally hit home: I needed a coping strategy for the next
time this happened.
So, what to do on occasions when capabilities are missing? For
starters, if the shortcomings are yours,
it’s a good idea to reflect and take concrete actions to perform better. If the
shortcomings belong to others,
cursing under your breath or venting to your significant other can have a wonderfully
calming effect, but may not be enough to rectify the situation. Is it possible
to overcome such defects through mentoring or teaching? Unfortunately, I have found
that it is totally unrealistic to attempt to build the capacity of someone (even
if you are in a position to do so), over the course of a single assignment. In
any case, you’d first need to spell out their failings to them. That could be
pretty awkward, right? Furthermore, you don’t really have much time for
capacity building — you need to get the bloody job done.
What you need is a back-up plan, especially if you are
ultimately responsible for the work (if you’re the team leader) or because you
were asked to pick up the slack (by the team leader). Here are three
suggestions:
Build in a
time buffer: Provide enough slack in your schedule to take into account the
extra time that you might need to address the shortfall. For example, if you
think a task will take two weeks, try to allocate three weeks.
Build in a
human resource buffer. Identify persons, either on the team or not, who could
step into the breach. Maybe the organization that put the team together (if you
are subcontracted) has the resources to bring on extra help.
Build in a
mental buffer: Prepare yourself not be surprised or upset when colleague X lets
you down. Unless you’ve worked with them before, and therefore know their
strengths and weaknesses, assume that people have a least one weakness, and
that it will impact the work at hand.
In a word, contingencies!
Let 2019 be a year of contingency planning. The plan comes
with its own reward: if you have a
decent contingency plan, you will end up with more time, energy, and even
inspiration, to focus on the interesting and fun stuff.
Like other forms of inquiry, evaluation involves sorting, filtering and distilling information in order to communicate something of importance. (Academics, journalists, attorneys, and private investigators do this, too.) When the work is done, you want to be able to present your findings in a clear, convincing, and attractive manner for easy consumption.
The problem is, there is a vast amount of information out there. It can easily overwhelm. With the Internet entering its mature phase, we swim in an information glut. I leave for another time a discussion on the differences between data, information, knowledge, intelligence, and wisdom, except to say that (from what I can tell, anyway) there is a lot less wisdom than there is data in the world…
A big part of your job, if you are an evaluator, is to know what information you need, and where and how to find it. To do this effectively, you want to be able to zero in on the essential stuff, while still being open to any interesting findings you may not have considered.
Let the purpose of the evaluation be your guide. Keep the reason you are searching at the front of your mind. Perhaps you seek to understand how well a program has built the capacity of agronomists to introduce innovative irrigation techniques to farmers? In that case, keep your focus on factors that may have a direct bearing on capacity building efforts, while limiting the amount of time you spend on learning about other things. Take note of them, but try not to let them distract you from the main question.
As an evaluator, you do not have the luxury of time that you would if, say, you worked in academia, to produce a dissertation or journal article (often years!). Evaluation, which is often about collecting and applying evidence to problems (in programs, policies, etc.) is relatively fast-paced. So even though you must review the literature, reports and data relevant to your evaluation, you simply will never have the time to read every word. You need to prioritize what you read and develop the ability to scan a document for what is essential.
When it comes to writing, you again will need to be strict with yourself. Avoid padding your reports with unessential information. Have you ever read a report (or a section of report) where you don’t understand what the point is? Have you found yourself asking, why am I reading this? Don’t put your readers through that.
A hierarchy of information
Imagine now that information and its offspring, exist in the form of a pyramid.
At the bottom of the pyramid is, let’s say, all the information in the world. Every piece of observable and non-observable phenomenon, from myriad perspectives. This amounts to untold trillions of bits of data, which are constantly accumulating and constantly changing. Think of this as an ocean of information. For all intents and purposes, this ocean is infinite and growing, much like the universe. It cannot be encompassed. All you can do is dip a sieve into these waters and try to collect what is most suitable to your purposes.
At the next level up is all the available information and knowledge. This is any information that has been processed somehow, whether printed or digital, spoken or written. Much, but far from all of it, is searchable using an Internet search engine. It is still a huge, overwhelming and unwieldy amount. But it has at least been produced by someone. It must also have some meaning, which is why I combine it with the concept of knowledge.
Next comes all the topical information that is out there. Maybe you’re writing about the electricity sector, or artificial intelligence, or breast-feeding. Depending on the area you are looking at, you will still find a plenty to review, and many experts, authors and practitioners you could talk to. If you are writing a general overview or introduction to these topics, you would synthesize all of this. Generally, however, you will not have such a broad focus.
Next comes the information that addresses your subject. It will be quite narrow in focus. For example, what is the impact on the poor of rising electricity tariffs? What does the introduction of artificial intelligence mean for workers in the fast food industry? What is the correlation between breastfeeding and the immune system? Now we are closer to where we want to be. It is still more information than you need, but the amount is manageable. You will only draw on the research and practices and reports that exist, plus any new primary data that you have distilled.
The next level in our pyramid is all the data and information collected for the specific purpose of the evaluation. This is the material you have reviewed with the aim of understanding your subject and informing your audience. It may include a database with thousands of observations and a hundred variables or more. You may have a bibliography of dozens or hundreds of sources. You may have hundreds of hours of interview or focus group discussion recordings. This is your personal store of information and it should, ideally, all be somehow relevant to the purpose of your evaluation.
Still higher up and narrower in scope are the evaluationfindings. This is where the rubber hits the road. The findings, which normally come with conclusions and recommendations, are the core information, which you have transformed into knowledge. This is what answers the evaluation questions and backs them up with evidence. In the evaluation world, reports should generally not be longer than 20-30 pages, excluding annexes. That is about the amount of detail which specialist readers who are interested in your subject can stomach.
The summary findings, which includes the executive summary of a report, and may also exist in a standalone short note or slide presentation, is how the essence of the report is presented. This is what most people will read or watch. If the findings are a distillation of all the information you have collected, the summary findings are a distillation your broader findings. As a rule, the length should be about 10 percent of the full report, from 2 to 5 written pages maximum, or no more than 10-30 slides.
Finally, the main story. This is the quick one minute story you tell your significant other or friends or colleagues who ask what you learned, without boring them with all the details. It could be in the form of a few paragraphs and bullet points that result in a one-page policy note that goes to the Minister of Energy, for example, as a policy brief. For example, “We found that most of the poor didn’t suffer as a result of electricity tariff increases because electricity expenditures fell as a share of their total expenditures. And all households now have 24 hour service;” or “We project that artificial intelligence will eliminate, on average, one job per restaurant, while customers have shown a preference for interacting with humans when they order fast food;” or “Breastfeeding was shown to reduce the incidence of illness in infants under 5 if they were weaned only after x months.”
And with that we have reached the pinnacle of our pyramid. Time for the next project.
One country after another is exiting the 2018 World Cup, packing their bags and leaving Russia. Only Croatia and France are left to play in the final on Sunday. Fans from around the world have had to face the fact that their country lost. My conclusion? Now is a good a time as any to reflect on failure.
Aside from sports, many, many other areas of human endeavor are some mix of success and failure. Superficially, and depending on how we measure it, I’d like to argue that failure is generally far more widespread than success. And that’s not only because there can only be one winner, as in sports championships like the World Cup. Let’s look at examples from completely different fields:
Every year, the US Congress introduces thousands of bills (proposed legislation), but only about 4 to 5 percent of those end up as laws. That’s a lot of effort – given that bills are hundreds and sometimes thousands of pages long – going into something that doesn’t succeed.
Now let’s take an example from the private sector. In the US, hundreds of thousands of new businesses are established every year. In 2015 that number was 679,072 according to the Bureau of Labor Statistics. Based on historic trends, 50% of all new businesses shut down within five years. Not always because they failed, of course, but probably most people start businesses that they hope will last much longer.
Returning to the subject of sports – in baseball, America’s pastime, the chances of making it to the Major Leagues from the minors, i.e. the lower professional leagues, are only about one in 10. Once they reach the Majors, there is more failure – the average batter fails to reach base by getting a hit (the primary, if not the only, reason for stepping up to the plate) almost three out of every four tries.
What about New Year’s resolutions? Studies have shown that less than 10 percent of people who make resolutions stick to them. Many fail to keep their resolutions for more than a few weeks. Let’s take a longer view. Many, if not most of us, don’t achieve the life goals we set ourselves when we were young. I certainly myself failed to earn a living as an actor (after two years of theater school in Russia and over 4 years of trying to break into the theater in New York), never ran a marathon as fast as I wanted, never wrote the novel I started (and tore up) umpteen times in my younger years. The list goes on. Even high-performing individuals and companies often don’t reach their goals, as evidenced by the sports examples from above or the losers in Presidential elections.
Do these examples, and the statistics, mean are we are simply doomed to failure in our greatest aspirations? Do they suggest that we humans are hopeless at everything we try? I would argue no, not at all.
First, what they indicate is just how much effort we put into trying. How much thinking, planning, time, and money are invested in reaching goals! Think about it: Tens of thousands of kids work hard to become professional athletes or reach the Olympics. Hundreds of thousands of people start businesses every year (even in the depths of the Great Recession over half a million new businesses were started in the US). Millions of us still make New Year’s resolutions.
I’m not saying that merely trying is good enough, or that we should excuse failure because it is the norm. But we should understand it in context – not getting all the way there is human and quite normal. And I’m not saying that we should lower the bar. You need to make a darn good effort to reach whatever it is you are aiming for. Falling short of a goal when you try is very different from not getting there because you didn’t bother. Even if most of us don’t get there, we still get something out of it, we land somewhere. And hopefully learn a few things and become a little smarter in the process.
And now for the evaluation angle: When evaluating the performance of a project, it is really important to ask why the goal was not achieved, as well as asking what was achieved. Don’t just focus on the binary succeeded/didn’t succeed parameter. Don’t pass judgement too quickly because a performance metric wasn’t achieved.
Naturally, it also comes down to how we define and measure achievement. It’s hard not to be impressed by people who start their own businesses, run for high office, train for the Olympics, or get selected to play on your country’s World Cup team. On a CV, all those things look pretty good. But there is a lot of sweat and time that goes into reaching those goals.
And remember, most people who are, or seem, successful, have gone through many failures before they got there. Failure, is after all, the norm. So buck up, and go out there and try one more time.
It can be useful to think about the words we use to make it happen, to get things done, to bring about change, to reform. So today I’ve put together a list of key terms, such as ‘goal’ to ‘policy’ to ‘evaluation.’ . They are used by government policy makers and officials, but they can also be used by just about anybody, like you!
For each term, I’ve included my definition in the form of a question. I rather like questions, they’re good for stimulating thought. Underneath, I’ve also given a dictionary definition. I’ve taken what are the most relevant Merriam-Webster online dictionary. However, quite honestly, I don’t think all of them are that clear or that good, perhaps reflecting the rich ambiguity that pervades the English language (and others). You can see that some terms are used almost interchangeably, some overlap. Feel free to propose your own definitions.
To see how this looks in the messy world out there, I’ve applied the terms to a couple of situations. First, the case of a 21st century malaise – the smartphone malaise. Second, President Trump’s position on immigration to the US.
Goal: What do you want?
Merriam-Webster: the end toward which effort is directed
Objectives: Why do you want that?
Merriam-Webster: something toward which effort is directed
Strategy: How are you going to get it?
Merriam-Webster: a careful plan or method
Policy: What conduct will you follow to get it?
Merriam-Webster: a high-level overall plan embracing the general goals and acceptable procedures especially of a governmental body
Plan: What are you going to do to get it?
Merriam-Webster: a method for achieving an end
Legislation: What incentives and disincentives will make sure the policy is followed?
Merriam-Webster: the exercise of the power and function of making rules (such as laws) that have the force of authority by virtue of their promulgation by an official organ of a state or other organization
Evaluation: Did you achieve your goal and objectives? Why or why not?
Merriam-Webster: determination of the value, nature, character, or quality of something or someone
Exhibit A: The smartphone challenge
Goal: Spend less time on my smartphone
Objectives: Get more exercise, reduce eye strain, reduce risk of getting run over, be more social
Strategy: Avoid using the phone during longer periods of time
Policy: Phone is never taken to the bedroom or taken out at the table
Plan: Leave phone downstairs; ask my spouse to remind me to not use it during meals
Legislation: (Not normally an option for individuals, but could For every quarter that I stick to my policy, I allow myself to buy a new pair of shoes
Evaluation: Did these restrictions on cell phone use help me meet my objectives? If not, what should I do differently.
(On a side note, I myself don’t own a smartphone. Have been holding out for 11 years and counting. I find that the advantages still outweigh the advantages of ownership.)
Exhibit B: US immigration
Goal: Make America Great Again
Objectives: Return the country to an earlier era, invigorate the white working class, get (re)elected,
Strategy: Demonize undocumented immigrants (Trump referring to Mexicans when announcing his candidacy: ““They’re bringing drugs. They’re bringing crime. They’re rapists. And some, I assume, are good people”) and minority would be immigrants; build a wall with Mexico
Policy: Bring up immigration issue frequently; get people to focus on crimes committed by immigrants
Plan: Have ICE conduct raids on businesses; increase deportations, pass new legislation restricting legal immigration and the rights of migrants and asylum seekers
Legislation: Border Security and Immigration Reform Act of 2018 proposed by Republicans (not passed), a bill which would have provided funding for a border wall, modified visa programs to limit legal immigration, mandated the use of a worker verification program, allowed the administration to cut funding to sanctuary cities, allow recipients of DACA to apply for legal status, and prevent families from being separated at the border.
Evaluation: None yet, but an evaluation might explore whether the Trump policy has an impact on immigration flows, whether it actually benefits Americans (in terms of jobs, income, their identity), whether it helps the Republicans win the next elections, what its affects are on immigrant families in the US,
As the title suggests, in this post I am going to try to tackle three things: meetings, the human brain, and digital intelligence. Bear with me.
I go to meetings, like most of you. They make up a small, but significant part of my work, when I’m not doing background research, writing reports, and travelling. I actually like meetings. This is not just because, in my line of work as a freelance consultant, you develop a real appreciation for periodic human contact, but because meetings – when they focus on specific goals or have a clear agenda – can be extremely productive. In the case of interviews, which is the form a lot of my meetings take (the other types are team meetings and policy discussions), they are the probably most efficient way of obtaining the information you need when I’m evaluating a program, a project, a sector, or some topic.
What are the alternatives to these meetings, to learn new things? Mostly culling information from reports, books, and, of course, the internet, mostly via a search engine or social media. A large amount of Google’s search engine activity now uses artificial intelligence (AI). (I like to think of AI as just the latest manifestation of brainless intelligence, but that’s another blog topic.) Yes, the internet has made our lives a lot easier. But we’re fooling ourselves if we think everything that’s knowable can be found at the click of a mouse.
But before I get to this, yes, I am aware of the complaints. I’ve read a lot about how office meetings are unproductive, a waste of time and money, and brain cells. That may be so in the private sector or in management, areas where I am quite happy not to work. However, I find meeting with other people extremely valuable, for two reasons. First, it is a quick and efficient way to learn the most important thing about an issue. Second, it promotes cooperation, through relationship-building. And without cooperation, things tend to fall to pieces. (I’ll try to get to that in yet another blog post). For now, I’ll focus on why holding meetings is great for information gathering and, in some important ways, much better than Google. Why? It comes down to this – humans are exposed to, immersed in, and able to reflect upon a breathtakingly large amount of real world experiences, interactions, visual stimuli and sensations. We also feel and use our judgement. This is something computers and artificial intelligence can hardly do, despite the recent advances so breathlessly talked up in the media. In fact, search engines are limited to what they can find on servers.
I am not a fan of the reductive approach, e.g. reducing the human mind, or the soul, to biological impulses to be digitally mimicked. But I think there is a useful comparison to make. Strides in computing power and artificial intelligence notwithstanding, humans still have some serious comparative advantages. You can read online about how the human brain compares to a supercomputer, with some saying it has been surpassed, and others saying, not yet, not by a long shot. There are also some interesting comparisons and discussions regarding the human brain vis á vis search engines, especially Google.
You’ll see that, in a narrow sense and along quantitative parameters, search engines may be superior: processing power to retrieve keywords, access to data, speed, etc. But here is one parameter where computer search engines don’t perform anywhere near as well as humans – they are limited to the written, numerical and recorded, information they can find online. That misses out on a huge amount of information. What might that be? Well, everything that isn’t recorded: conversations, events, personal notes, observations of others, email exchanges (that Google doesn’t have access to), and so on. An expert or stakeholder who is engaged in the field you are studying – whether it be education policy in the Maldives, the Uzbekistan irrigation sector, energy efficiency in Ukraine – will be able to draw on a depth and breadth of information that the most powerful search engine in the world can only dream of (if androids could dream of electric sheep, that is). Although the information stored on all the world’s servers is vast and growing, it is still a fraction of all the information in the world and inside the heads of its population.
Ask your interlocutor, your key informant (the term used in evaluation) a question, and he or she will be able to draw on countless, non-digital, resources in order to answer you. Google is limited to giving you what it finds on the web. Certainly useful, but limited. Humans still have some value, it seems. That is why meeting them, if you ask good questions, is so invaluable. If you are an evaluator, an investigator, a journalist, or in a similar line of work, you quickly realize that you get more from holding a few meetings with key individuals than from plowing through hundreds or thousands of pages of documents.
Caveats, caveats. There are always caveats. So yes, it is true that not everyone’s memory functions at an optimal level. And some key informants, you quickly realize, don’t have much to say. Or maybe you find yourself talking to the wrong person. And, naturally, you still need to consult the thematic literature, the reports and journal articles and so on, to complement the meetings you hold. But overall – as a professional, doing my job, I’ll keep going to those meetings. And here’s a(n open) secret: most people actually like to talk about what they do and what they know. Most are happy to share. Also they don’t show you those annoying ads before answering your questions…
Sometimes, at the tail end of another 20 or 30-hour trip, say, to conduct fieldwork for an evaluation, and after crossing as many 11 time zones and spending two nights on airplanes, I ask myself – why? What is the point of traveling, 7,000 miles? Couldn’t we have just had a few phone calls? In this day and age of instant communication, skype and other video chat applications, is it really necessary to be physically present for a meeting? Just think about the time and money spent on business travel. The Global Business Travel Association estimates that global spending on business travel topped $1 trillion about 5 years ago and has been expanding ever since. That amount represents, give or take, 1% of the entire global economy. That is a lot – roughly the size of the Mexico’s GDP. Basically, you can think of it as the marginal value, to the public and private sectors, of face-to-face meetings vs. phone, skype or videoconferencing.
I was stimulated to write this blog post by the latest issue of New Directions in Evaluation (Winter 2017). The entire volume, edited by Randi Nelson and Denise Roseland is devoted to field visits – how and why they are a key element of conducting evaluations, among other things – a somewhat neglected issue in the evaluation literature. The discussions and articles have caused me to reflect a bit on the importance of going to the field (i.e. being physically present in the area where the program is being implemented) in my own work in international development.
There are two main aspects to field visits, which in the field of evaluation, are defined as going to the place where the program or project is being implemented. There might not even be any physical thing to see, such as a new school, equipment, newly installed energy efficient boilers, or what have you. You need to be there to meet with people in their own environment. And long trips and jet lag are actually the least of it. Field trips within countries can involve long car trips to remote rural areas along terrible roads in countries where road safety is somewhat of an afterthought, if not a downright inconvenience. (Auto fatalities pose a real risk to international development professionals. A few years ago I was involved in an incident in which the car I was driving in spun and flipped over on a country road in southern Malawi. I was lucky. I escaped with minor bruises. (Use that seat belt, folks.)
Yet, if you ask me whether I’d rather to do all my analysis from the comfort of my own home, based only on phone interviews or a desk review, I would respond…well, sure, but don’t expect the same depth of analysis or the same quality of information. It is much easier to get a sense of a project when meeting people – implementers, beneficiaries, other stakeholders. Doing a focus group by phone or video? Forget it. Take a look at the photo below, from a field trip I was on in Tajikistan in 2011. Imagine talking to those people from the village by video-conference.
There is also a very practical reason for preferring face to face meetings. When you can see the other people in the room, you can avoid the awkward pauses and people speaking over each other.
Conducting many types of analysis is not just about collecting information. A key factor in getting people to open up to you, and reveal their thoughts, is trust. And when you have never met the person, you can’t look them in the eye, you can’t read their body language. When people get a measure of you (the evaluator, researcher, entrepreneur, etc.) as a person, they tend to talk more. Until you see someone in the flesh, both of you are quite literally disembodied, and not quite real. Who hasn’t in the online dating scene hasn’t had such an experience? Before you met the person they may have seemed terrific, almost perfect, but once you meet them, you have a completely different (and often disappointing) impression? The digital world, the world of two dimensions still doesn’t hold a candle to the world of flesh and blood! Whether you’re evaluating a project or a potential mate, you gotta get yourself up off the sofa, and get out there. For whatever reason, being physically present is a big step in getting closer to that ever elusive truth.
