Today dark clouds hang over Afghanistan as the Taliban takes over the country. The situation may soon deteriorate into a humanitarian crisis, leading to a mass exodus of refugees to neighboring countries and beyond.
However the story in Afghanistan unfolds, at least four critical global developments will keep migration at the top of many country’s policy agendas for decades to come.
More people crossing borders, whether as refugees or economic migrants, ensures that countries will be unable to ignore questions of inclusion and exclusion.
Conflict and terrorism. A 2019 paper estimated that global conflicts were at an all-time high since 1945. all-time high since 1945. Ongoing as well as new conflicts and terrorism will result in more refugees and economic migrants. Pressure will increase on countries to either accommodate more people from outside their borders, or they will use ever more draconian measure to keep people out.
Climate change and the devastation wrought by extreme weather events are pushing people to cross borders. Last year, in a lengthy article on the subject, New York Times declared that the “The Great Climate Migration” has begun. An unprecedented number of people will migrate to more temperate zones. Madagascar, already extremely poor, is suffering from a devastating famine. UN organizations are linking it to climate change.
Democratic backsliding. In 2021, Freedom House reported the 15th consecutive year of decline in global freedom. Lack of participation in democratic processes across the globe—Tunisia, the one Arab country where the Arab Spring did not end in failure, is the latest example—means fewer people are participating politically. If people can’t vote at the ballot box, they will vote with their feet, and get out if they can. Already thousands of people have fled Belarus to escape President Lukashenko’s violent crackdown after elections were rigged.
Falling birth rates and ageing populations in the rich world are contributing to a labor deficit. The number of births per 1,000 is now half of what it was in 1950, and fertility rates are falling well below replacement levels. Unlike the push factors above, low fertility will act as a pull factor for migration. If societies are to continue to prosper, higher workforce participation will be key. This will involve skills training and better childcare options to make it easier for those who don’t work, while making it easier for more migrants to enter.
Meanwhile, racial injustice in the US and elsewhere has been brought into relief by the killing of George Floyd in 2020, and the broader recognition that the issue can no longer be ignored. Many more people have become aware that African Americans and other minorities still don’t enjoy the rights and privileges other citizens take for granted. Hate crimes against people of Asian descent in the U.S. are up.
These global developments all but guarantee that the issue of inclusion, who gets in and who doesn’t, will only grow more acute as the century progresses.
Broader inclusion and the diversity that comes with can bring huge benefits, but it won’t be easy, either for societies or organizations.
How policy makers manage these pressures and processes, without inciting more backlash from anti-immigrant groups, will pose a major challenge for policy makers for generations to come.
In many countries, the urge to erect more barriers—physical and otherwise—will compete with calls to let more people cross.
Can we muster the empathy to recognize that anyone of us could be a migrant, excluded, and struggling to start a new life? Can enough of us muster that empathy?
CNN has reported that some people in Missouri are getting vaccinated in secret to avoid backlash from loved ones.
And thus, Covid has offered up yet another way to demonstrate tribal loyalty. First it was face masks; now it’s vaccinations.
Vaccine hesitancy has a range of underlying causes, from fears about side effects, to distrust in government, to skepticism over the risks. I have relatives who believe Covid-19 is a plot hatched either by the government or by Bill Gates to control people and reap profits.
Whatever the cause, American society is bifurcating into two groups, the vaccinated and the voluntarily unvaccinated. In a recent Washington Post-ABC News poll, 58% of Republicans said they do not intend to get vaccinated (or at least that is what they are telling pollsters), compared to 6% of Democrats.
I’m all in favor of diversity. However, dividing ourselves into “vaccination tribes” may not be the best way to celebrate our differences.
Alas, my current research on inclusion and exclusion leads me to conclude that as long as the human race exists, we will continue to self-select into tribes. Social identity theory posits that we categorize people, figure out the group that matches our identity, and then compare ourselves with other groups.
Our self-esteem is bound up in belonging to a group, especially if it is high status. This is how humans are made, and almost any excuse will do: skin color (but not eye color, for some reason, at least outside of this fascinating experiment), ethnicity, religion, party affiliation.
Pick a difference, no matter how arbitrary or trivial, and someone will be out there with a crowbar, jamming it in to widen the crack between two groups.
If not enough of the U.S. population gets vaccinated, the virus may continue to mutate, more people will die unnecessarily, the crisis will drag on, and more lockdowns may be imposed.
If vaccine hesitancy isn’t overcome soon, the early vaccination successes may yet become a case study of “divided we fall”.
The year is 2016. I’m sitting on the cement floor of a
home in a village outside of Tulajpur, deep in the Indian state of Maharashtra.
I’m surrounded by more than a dozen women entrepreneurs taking part in a focus
group.
My American colleague “Lisa” and I are here conducting
a USAID evaluation, as we crisscross India to assess similar projects, along
with other team members. We’ve come to this place to ask entrepreneurs to share
their experience with an innovative program that promotes the use of cheap and
affordable solar-powered lamps, as well as other solar-powered gadgets. These savvy
women, who run small shops built into the front of their houses, are a key link
in the supply chain that helps off-the-grid rural Indians light their homes. The
focus group discussion was animated and friendly, with any cultural barriers
fading to insignificance.
Two things struck me. First, all of them had
smartphones, which they were using to snap photos of us. Normally, it is the evaluators who photograph focus group
members, so you could ask, who was observing whom? Although it was a relatively
poor village, the women in our group clearly had above-average income levels.
In 2016, only about one in six Indians owned a smartphone, and most of them were
in urban areas.
The second notable thing was the latrine situation –
there was none. When our meeting ended, Lisa asked if she could use the toilet.
She didn’t get a clear answer. After she asked again (the meeting had lasted three
hours, after all), the woman hosting the meeting then reluctantly showed Lisa an
empty room at the back of the house, which had a sink, but no running water,
and a small hole in the floor. This was not a Turkish-style squat toilet with
the ceramic footholds flanking a hole, either. Lisa said the room looked
nothing like a bathroom, and she was not about to repurpose it. When she inquired
where the women did their business, the homeowner sheepishly indicated an
empty, rubbish strewn lot next door. That
was the toilet, so to speak.
The incongruity of people leapfrogging to smart
phones, while still practicing what in development circles is called open
defecation, was striking. Although I spend a good bit of every year in
developing countries, it still took me aback. The Tulajpur women were
simultaneously stuck in the past – a noxious, undignified and unhealthy past –
and yet benefiting from a high technology “future,” one which would have seemed
like science fiction to their parents.
As this insightful article in National
Geographic on the topic of latrines and open defecation makes clear,
the problem of persuading people to build or use latrines is complex. Open
defecation is still how 1.2 billion people on the planet relieve themselves,
and almost half of those live in India. The practice comes with serious health implications
as it raises the risk of infection, disease and death, especially for children.
The issue isn’t the availability of the actual
physical object. Many latrines have been built, but remain unused. A complex
array of factors come into play: tradition (historically, of course, all of
humanity lacked toilets), to the still-extant caste system which makes it a
disgrace to clean out latrine pits for anyone but the Dalits (formerly known as
untouchables), to lack of sewerage systems, to a supposed aversion to enclosed
spaces while defecating. Getting families or villages to build just a pit
latrine – or even use one that is
built for them – remains difficult. A lot of outhouses have fallen into disuse,
or have been repurposed as storage sheds.
This is not to single out India, which has made
tremendous strides on many socio-economic indicators, such as lowering its birth
rate to 2.1 children per woman – definitely a modern phenomenon. I bring up open
defecation and smartphones as examples of how governments and donors sometimes have
very different priorities from the population. Getting people to use a toilet has
major positive health impact. Almost no one argues for “cultural sensitivity”
and just letting people get on with it in this case.
The challenge of getting rural villagers to use
toilets is not that they are living in a sort of isolated, antediluvian Eden,
in harmony with nature, as the solar lamps and cell phones, not to mention
plastic trash strewn everywhere, make clear. I would argue that it encapsulates
three problems common to many development efforts.
First, the preferences of the state do not always
coincide with those of society. To paraphrase an old saying, the state
proposes, but (wo)man disposes. There may be excellent, scientifically-grounded
economic and health reasons to pull people into the modern world, but it turns
out that people have their own preferences, or priorities, such as smartphones.
Indeed, social behavior change has become a field of study in its own right. It
is one thing to build something, but getting people to use it is another
matter.
Second, having a vested interest is a powerful motivator. As every landlord and tenant has no doubt observed, a different attitude comes into play when you own something compared with just using it. That is why many programs that build infrastructure for communities focus on “ownership” as a concept. They require communities to contribute in kind or in cash. Yet often the cost is so high that the community contribution is not more than 5 percent, or less, and that feeling of ownership is concomitantly weak.
Three, when poor countries aspire to the public goods
available in richer countries, they often end up building or buying things that
they can’t afford to use. In a lot of cases, infrastructure, from the most
basic latrines, to enormous projects such as hydroelectric dams, is being built in countries where they lack the
technical and financial resources to maintain and operate them. Funds need to
be accessed and set aside, and procedures introduced.
Unfortunately, we
didn’t get into a discussion with our focus group women on these issues. That
was not why we were there. However, I’m sure they would have given us a valid
reason for their choices, at least from their perspective.
It can be useful to think about the words we use to make it happen, to get things done, to bring about change, to reform. So today I’ve put together a list of key terms, such as ‘goal’ to ‘policy’ to ‘evaluation.’ . They are used by government policy makers and officials, but they can also be used by just about anybody, like you!
For each term, I’ve included my definition in the form of a question. I rather like questions, they’re good for stimulating thought. Underneath, I’ve also given a dictionary definition. I’ve taken what are the most relevant Merriam-Webster online dictionary. However, quite honestly, I don’t think all of them are that clear or that good, perhaps reflecting the rich ambiguity that pervades the English language (and others). You can see that some terms are used almost interchangeably, some overlap. Feel free to propose your own definitions.
To see how this looks in the messy world out there, I’ve applied the terms to a couple of situations. First, the case of a 21st century malaise – the smartphone malaise. Second, President Trump’s position on immigration to the US.
Goal: What do you want?
Merriam-Webster: the end toward which effort is directed
Objectives: Why do you want that?
Merriam-Webster: something toward which effort is directed
Strategy: How are you going to get it?
Merriam-Webster: a careful plan or method
Policy: What conduct will you follow to get it?
Merriam-Webster: a high-level overall plan embracing the general goals and acceptable procedures especially of a governmental body
Plan: What are you going to do to get it?
Merriam-Webster: a method for achieving an end
Legislation: What incentives and disincentives will make sure the policy is followed?
Merriam-Webster: the exercise of the power and function of making rules (such as laws) that have the force of authority by virtue of their promulgation by an official organ of a state or other organization
Evaluation: Did you achieve your goal and objectives? Why or why not?
Merriam-Webster: determination of the value, nature, character, or quality of something or someone
Exhibit A: The smartphone challenge
Goal: Spend less time on my smartphone
Objectives: Get more exercise, reduce eye strain, reduce risk of getting run over, be more social
Strategy: Avoid using the phone during longer periods of time
Policy: Phone is never taken to the bedroom or taken out at the table
Plan: Leave phone downstairs; ask my spouse to remind me to not use it during meals
Legislation: (Not normally an option for individuals, but could For every quarter that I stick to my policy, I allow myself to buy a new pair of shoes
Evaluation: Did these restrictions on cell phone use help me meet my objectives? If not, what should I do differently.
(On a side note, I myself don’t own a smartphone. Have been holding out for 11 years and counting. I find that the advantages still outweigh the advantages of ownership.)
Exhibit B: US immigration
Goal: Make America Great Again
Objectives: Return the country to an earlier era, invigorate the white working class, get (re)elected,
Strategy: Demonize undocumented immigrants (Trump referring to Mexicans when announcing his candidacy: ““They’re bringing drugs. They’re bringing crime. They’re rapists. And some, I assume, are good people”) and minority would be immigrants; build a wall with Mexico
Policy: Bring up immigration issue frequently; get people to focus on crimes committed by immigrants
Plan: Have ICE conduct raids on businesses; increase deportations, pass new legislation restricting legal immigration and the rights of migrants and asylum seekers
Legislation: Border Security and Immigration Reform Act of 2018 proposed by Republicans (not passed), a bill which would have provided funding for a border wall, modified visa programs to limit legal immigration, mandated the use of a worker verification program, allowed the administration to cut funding to sanctuary cities, allow recipients of DACA to apply for legal status, and prevent families from being separated at the border.
Evaluation: None yet, but an evaluation might explore whether the Trump policy has an impact on immigration flows, whether it actually benefits Americans (in terms of jobs, income, their identity), whether it helps the Republicans win the next elections, what its affects are on immigrant families in the US,
Wherever you stand on the issue of foreign aid, the White House budget proposal has put the topic back on the table for discussion. And it definitely deserves being discussed.
On the whole, Americans are, apparently, a generous people. The vast majority say they donate money – 83% according to a 2013 Gallup survey, which also found that 65% volunteer. It is probably safe to say that most Americans believe giving to those in need is a good thing. But should the government be involved in giving to others, especially to people living in other countries? That is a different matter, and the debate over ‘foreign aid’ or ‘development aid’ has been going on for decades now.
American attitudes on this matter are a bit fuzzy. Before we even get to the principle of foreign aid, there is already plenty of confusion about how much the US spends on it. (The short answer is – vastly less than most people think.) However, one group of people has a pretty clear idea on the matter, and it is those working for the current US Administration.
The new White House budget proposal for fiscal year 2018 would see foreign aid, along with State Department funding, cut by almost one third. This is a massive reduction. Remember the budget sequestration of 2013, which cut programs across the board, and how difficult that adjustment was? The current White House budget proposal, which also includes deep cuts to domestic programs, transfers virtually all of the $52 billion in savings to the Department of Defense, increasing the latter’s budget by about 10%. (Bear in mind that this is not the actual budget, which is up to Congress to draft and pass. But it signals the President’s priorities to Congress and it will exert influence over Congress if there is pushback. It is best thought of as a negotiating position.)
So there it is – the new Administration has signaled the approach it wants to take in order to build “A New Foundation For American Greatness,” the official name of the proposal. It will be guns, not butter.
How exactly will this budget help America become great? The explicit rationale is that the President will “place America first by returning more American dollars home and ensuring foreign aid supports American interests and values.” Those values, as per the Administration, are pretty much all about security and military prowess. In evaluation, this is called a ‘theory of change’ or ‘program logic’ – how specific action will lead to a desired outcome. Naturally, there is much debate about whether the budget the White House is proposing would have the desired effect.
The proposal argues that it will make the Department of State and USAID leaner, more efficient and more effective, although it does not specify how it would do this, aside from forcing the two agencies to operate with considerably fewer resources. Furthermore, it proposes redirecting foreign aid spending primarily to security issues.
What the President’s budget proposal does is open a Pandora’s box of questions. Whatever you think of it, these questions are legitimate and worth pondering.
Definitions. How is foreign aid defined? What is the money actually spent on? Who ends up benefiting and how?
Effectiveness. How effectively is aid spent? Is it getting the intended results? How do we know?
Amounts. How much does the US actually spend on foreign aid? Interestingly, the average American believes we spend far, far more than we do. About 20 times more. Not 20% more, 20 times more.
Rationale. What is the rationale behind foreign aid? How valid are the arguments made in its favor, specifically: national security, commercial interests and humanitarian concerns?
Investment vs. cost. What is the chance that “returning more American dollars home” by cutting foreign aid and diplomacy funding will make America great? Is there a cost to spending less overseas? Is aid an investment or just a cost, an expenditure?
Relativity. Is foreign aid spent less effectively than other budget items, such as defense spending, which is targeted to be the prime beneficiary of cuts to aid and many other sectors? Are there different standards of transparency and accountability?
Measurement. How is effectiveness measured? Are random-control trials, expensive experiments which measure the impacts against what-if scenarios, the best way to measure assistance? What about short-term vs. long-term effects of aid? What about the indirect and intangible effects?
Home vs. abroad. Why should one country give to another? Even if you believe there are benefits, how do you balance giving assistance to foreigners vs. giving assistance to your own citizens?
Altruism. How strong is the moral case for giving to strangers in a strange land? Should rich country’s citizens give much more, as long as people are dying of hunger and disease, simply because they can?
Geopolitics. By drawing down its foreign assistance, does the US abdicate its leadership role and reduce the geopolitical influence?
Where it goes. How much of foreign aid does actually end up in other countries compared to how much of it finds its way back to US citizens, US companies and the US budget via contract work and purchase of US goods, services and military hardware?
Typologies of aid. Aid comes in many shapes and forms, and with different objectives. What form of aid has proven to be the most beneficial? How does aid in the form of investment in infrastructure compare with the transfer and exchange of know-how (referred to as technical assistance)? Most aid is not simply transferred to poor people in developing countries – much of it is in the form of investment and technical knowledge transfer.
Potential harm. Does aid do more harm than good? Even leaving aside aid that goes to foreign militaries with questionable records, there is evidence, and arguments, that aid is not only ineffective, but that it can actually be damaging to individuals and society. What does the record show?
Government’s role. Should the government be in the business of giving? Perhaps aid should be left to philanthropic organizations, and be based on purely voluntary donations as opposed to taxes.
Externally provided assistance. Is there something intrinsically problematic with externally-provided assistance, i.e. efforts to exert change from outside a country?
In a series of upcoming blog posts I’m going to grapple with these thorny issues one by one, not necessarily in order. Let’s see where we end up.
Numbers are used a lot to make a point or serve as evidence. When you come across a number, in a report or on the news, always remember its meaning depends almost entirely on its context.
Certainly, numbers often seem inherently more reliable than descriptions. In the social sciences, there is a distinct preference for quantitative vs. qualitative findings. There is a bias, but perhaps it’s justified?
When you are presented with a number, there seems to be less intermediation. Numbers don’t lie, they say. But then the phrase “lies, damned lies and statistics,” often attributed to 19th century British Prime Minister Disraeli, is also pretty popular. It highlights inherent distrust toward statisticians and politicians. Because even if on the face of it numbers seem to be objective and verifiable, often there is quite a bit of intermediation behind their production.
While intermediation (statistical analysis) is acceptable, indeed necessary in order to collect and organize information and ensure that the results represent what they are supposed to, manipulation is not.
Forgive me for being blindingly obvious, but we need hard numbers. They are extremely useful. They allow us to have a common basis for discussion.
On a side note, numbers even serve different purposes. There are cardinal numbers (indicators of quantity, e.g. how many cups of flour in a bread recipe), there are ordinal numbers (indicating the order of things, who came in first place), and there are nominal numbers (for identifying things, like your seat number in the theatre). I’m going to focus on cardinal numbers for now.
Remember that a number is just a number. Its meaning (good or bad, important or trivial, expected or unexpected) is imposed externally. Its value is not inherent but rather depends on context. The context is what tells the story.
It is not enough to say that the unemployment rate was low. The description “low” can mean any number of things. Your first reaction should be “how low?” and “low compared to what?” The thing is, plugging in an actual number only gets you part of the way there. Let’s say unemployment was 7%. How does that compare to last year? It may be low if last year it was 9%, or it may be high if it was 5%. Or if in similar economies is above 10%, 7% looks pretty good, whereas if everywhere else it is below 6%, it looks less good.
Not only are comparisons across time and space important, but comparisons with purely fictitious numbers also matter. What I’m referring to by this is the number in your head, which doesn’t’ actually exist, right? This is the number you were expecting.
When unemployment numbers are predicted to fall, but they actually go up, the stock market may fall. When you expect to be offered a salary of $60,000 but then get an offer of $80,000, that will make you very happy. The age of newly elected French President Emanuel Macron’s wife, Brigitte (64) is not interesting – there are many women in their sixties in France – until you know that he is only 39. Their age difference is unexpected because of the gender markers. Its unexpectedness is in inverse relation to the age difference between Donald Trump and his wife, Melania. Who wouldn’t expect a billionaire property developer to have a wife a quarter century his junior?
Placing bets is an extreme example of what we can call the arbitrage between expected and actual numbers (or current and future numbers, if you will.) So a lot of money is often at stake.
You can apply this type of thinking to pretty much everything – from the price of a new refrigerator, to your annual salary to the number of people without access to drinking water. Context is everything. What that means is that reference points are critical. Numbers floating out there on their own don’t mean much.
Of course, you want to know how a number is derived. You should ask how they are collected, what they represent (often a matter of definition), and whether they truly represent what they are meant to? Obviously, in the case of the Brexit referendum and the last US election, the numbers which the vast majority of polls gave us did not accurately represent the people who actually voted.
Statistics can be presented misleadingly even when they are accurate. One of the most common issues perpetuated by the media and researchers themselves, involves comparing differences to a (seemingly arbitrary) base, or even not revealing the actual base. For example, findings reported in the American Journal of Epidemiology indicated that drinking more coffee reduced the chances of dying from oral/pharyngeal cancer by 49%. Sounds impressive, right? But you also need to know that your chances of dying from this type of cancer are about 0.09%, i.e. less than one in a 1,000. They weren’t high to begin with. And what about all the other cancers? Did they check those? And what about negative effects? Maybe those cancel out the positive ones.
Responsible reporting of statistics, whether in academic journals or the media, involves providing the reader or viewer with meaningful context. When you see a number, ask yourself the following questions:
What does this actually represent? How is the number defined?
How does it compare with alternatives, similar situations?
How does it compare with the past?
How does it compare with what was expected? Does the number fall in line with a trend or go against the trend?
What are the possible interests of the person or organization in sharing this number? Journalists and evaluators have a code of ethics they are supposed to follow.
The tension between belief and evidence has probably been around ever since humans could sense the world and formulate thoughts. It has certainly been documented for several thousand years. In the Bible, we have the original ‘doubting Thomas’, who would not accept that Jesus had come back to life until he could put his hand on the wound in Jesus’ side. Galileo was put on trial by the powerful Church for arguing, based on scientific evidence, that the Earth circled the Sun and not the other way around.
The issue goes beyond belief. A set of beliefs tend to form an ideology, which typically concern human nature, religion, politics, or the economy.
For various reasons, people hold certain theories of how the world works, or should work. People may be influenced by tradition, tribe (the community to which they belong) or an authority they revere (like their church, or their president). Belief systems can have enormous consequences. At the extreme end, people have died for their beliefs – consider the early Christians refusing to renounce their faith and being fed to the lions, or young men volunteering to go to Syria and join ISIS. But even relatively anodyne beliefs about the world can have tremendous social impacts. For example, the American belief that this is the land of opportunity and if you don’t succeed you only have yourself to blame. In fact, there is a strong counter set of beliefs: that it is mostly those lucky enough to be born into the right (privileged) circumstances who actually enjoy those boundless opportunities…There are plenty of studies finding that children born into poverty have far fewer chances in life.
With respect to the current tensions between beliefs and evidence, there are at least two things worth noting.
Overwhelmed by too much information?
First, we live in an age saturated with information, which would make you think that evidence is readily accessible, and that virtually anyone can find at out information with a little web-browsing. Yet the rise of conspiracy theories, fake news, and the bifurcation into ideological communities in many countries defy this. The almost infinite wealth of information out there is also good cast doubting on evidence and supplying hermetic communities with a tailor-made set of facts that align with their beliefs. It has become easy to engage in selective perception. Information that doesn’t conform with what you think you already know can be ignored, devalued, or diluted with misinformation.
A bipartisan issue
Second, disregard for evidence or facts seems to be prevalent at both ends of the political spectrum. Both rightwing and the leftwing elements in US society (and possibly elsewhere) have embraced ideology at the expense of evidence. Facts that don’t fit are belittled or disregarded.
Many on the right, for example, scorn the science and statistics which show that the earth is more than roughly 6,000 years old, despite the geological evidence; that there is a near consensus among scientists that global warming is an actual, man-made phenomenon, or that gun ownership significantly increases risks to one’s life and those nearby. On the left, there is a rise of intolerance toward free speech on university campuses, which muzzles the expression of opinions deemed contrary to prevailing norms. There seems to be a prevailing wisdom that the college campus should be a protected ‘safe space’ (as if it were kindergarten) and that students need to be sheltered from ideas they disagree with or find uncomfortable – hence all those trigger warnings. Both right and left extremes seem to have become biased against any evidence which goes against their prevailing orthodoxy.
Living with belief and paying attention to evidence
Almost all humans believe things, or believe in things. This doesn’t mean we have to let beliefs cloud our judgment. If you believe in God, you generally still don’t take stupid risks because you believe He loves you and won’t let anything bad happen. Most people try to pay attention to their personal safety – staying on the sidewalk, driving carefully, not handling poisonous snakes – whether they believe in guardian angels or not.
Even evaluators believe things. We’re not robots. But as professionals we’re essentially paid to put our personal preferences aside and use our brains to objectively process large amounts of information.
Try this at home
If you are an evaluator, or just someone interested in getting closer rather than farther from the truth, you need to be aware of your beliefs and not let them interfere with your professional judgment. You need to maintain a boundary.
Here are a few things that I try to do to stay ‘conceptually sober’, i.e. not under the influence of beliefs, while on the job:
First, recognize your biases. Monitor yourself as you collect and analyze and present information. It is easy to skew findings to fit in with your hypothesis, or ignore them. Unfortunately academics do this all the time, by not publishing research findings that aren’t in line with their hypothesis, or simply interesting (publication bias)
Consider alternative causal explanations. You may have a good story that makes intuitive sense. You may have evidence (you better have evidence if you’re doing research). But always think about and look for evidence that might offer different explanations.
When you have findings that seemingly contradict each other, dig deeper to find an explanation. Maybe your data is inaccurate, or your sampling method was poor. But maybe different sources are revealing different things, or people you are interviewing are interpreting or defining events in different ways.
Double check what you hear from people when you’re doing research. If you hear pretty much the same thing from a lot of different that should give you confidence. If I hear something new or unusual from someone, I’ll ask others to corroborate.
Follow a guiding principle. It could be something like: “To present what happened as accurately and fairly as possible.”
Be curious. Find out as much as you can about a topic. Invest yourself in finding out more. Don’t be satisfied with the most obvious answers.
The non-partisan Congressional Budget Office (CBO) last week released its cost estimate of the American Health Care Act, the Republicans’ plan to replace the Affordable Care Act, colloquially known as Obamacare.
The CBO looked at a range of impacts. The headline numbers from the CBO estimate are a reduction in the federal deficit between 2017 and 2026 by $337 billion and a total of 52 million uninsured by 2026 (with 14 million losing insurance next year). There’s something to like (deficit reduction) and something to dislike (loss of health insurance for millions), depending on where you stand on these issues. Without passing judgment on the significance of the potential effects of the new bill, let’s focus on the reaction of the bill’s backers, including the White House, to the CBO and its work.
Even before the CBO report was published on March 9, potshots were being taken at the normally highly respected office. Forbes characterized the them as a “pre-emptive, coordinated attack.” Joe Barton, a Republican former House Energy and Commerce Committee Chairman, had this to say about the CBO: “I don’t know what they do, they sit around and think great thoughts and everything on the issues…One of the things we need to do is reform the CBO folks.” And Gary Cohn, director of the White House National Economic Council said on Fox News that “in the past, the CBO score has really been meaningless.”
The reactions suggest that some supporters of “repeal and replace” already sensed that the new healthcare proposal would not follow Trump’s professed goal of providing all Americans with great healthcare at lower costs than Obamacare. It is also worth remembering that the CBO director, Keith Hall, was named to his post by Republicans. This doesn’t mean that the CBO always gets its numbers right. It doesn’t. But its analysis is transparent and explained in enough detail that one can understand how it reaches its conclusions.
As an evaluator, part of whose work involves estimating the impacts of policy reforms, I can sympathize with the CBO being targeted for attack. Conducting evaluations, which is essentially what the CBO has done in tallying the costs and benefits of replacing Obamacare, is a great way to lose friends and alienate people. Evaluators are never the most popular kids on the block. We don’t control pots of money, we aren’t trumpeting success stories, our job doesn’t involve being ingratiating in order to sell stuff. We dig around and find out what worked and what didn’t, who’s winning and who’s losing. It’s necessary (and hopefully useful) work, but it’s not a popularity contest. And evaluations always turn up shortcomings. Nobody’s perfect. As the messenger, you can expect to get (metaphorically) shot at.
At a minimum, people get a bit nervous when their organization or program is evaluated. Even if the client who commissions the evaluation outlines the questions they want answered, evaluators are still being allowed ‘inside’; they’re able to ask questions of pretty much anyone connected to or benefiting from the project. Good evaluators pry through reports, extract data from whatever sources they can get their hands on, and double check everything they hear. Sometimes, the evaluation can seem a lot like an investigation.
I’ve conducted evaluations all over the world, some of them under fairly hostile circumstances. Even if the main client wants to have evidence on the impacts of a reform, that doesn’t mean everyone wants to know. There are potential winners and losers who have a stake in the outcome of your evaluation. There are vested interests. Trade union representatives, for example, can be a tough bunch. I once worked on an evaluation of the potential impacts of a mine privatization in eastern Serbia. Layoffs were expected. When I conduct this type of work, it is my policy to meet with representatives of all the affected groups. In this case, everyone knew that the restructuring was going to lead to the loss of about 2,500 jobs. It was the task of my evaluation team to estimate what would happen to their income and job prospects afterwards. The concerns of the workers were legitimate and completely understandable from their perspective, even if the mine was dependent on tens of millions of dollars of budget support annually. My approach to dealing with the trade unions was to open a line of communication with them, and keep it open throughout the study preparation, fieldwork and reporting period. This involved meeting with them periodically, listening to their concerns, and explaining what we researchers were doing.
On a similar study, this one collecting evidence on the impact of downsizing Croatia’s shipbuilding industry, we had a very different experience. There was unfortunately not enough budget or time to meet with the trade union representatives more than once. The antagonism toward the evaluation was considerable. Fieldwork included conducting an employee survey in a room on the premises of the shipyards. In fact, our survey supervisor, a young Croatian woman, was asked by a shipyard manager to turn over the list of (randomly selected) employees she was interviewing. When she refused he locked the door to the room and threatened not to let her out unless she complied with his request. She resolutely stuck to her guns however, risking her safety and wellbeing in the name of evaluation ethics. Luckily, she was able to call someone in the Ministry from the locked room with her cell phone, and secure her own release. But it left her shaken. I have even heard of survey interviewers in some countries being detained and jailed for doing their work.
In some respects, evaluators are indeed like investigative reporters. That makes the work interesting, and occasionally risky. But the evaluator as an investigator is not really the ideal association you want to create. It can sound, well, threatening. Another, and more conducive analogy is that of evaluator as a “critical friend.” This concept was proposed a quarter century ago by Costa and Kallick in a 1993 article. They noted that critical friendship must begin with building trust, and go on to highlight the qualities that such a friend provides. That includes listening well, offering value judgments only when the learner (i.e. the client) asks for them, responding with integrity to the work, and advocating for the success of the work (p. 50). As an evaluator, you are not trying to establish guilt, or attack or push an agenda. You are there to help the organization or policy maker better understand the impacts of their programs or proposals, and improve them so that their goals can be attained.
Going back to the CBO’s report, it reads like a levelheaded, thoughtful piece of analysis. If its critics have a problem with it, you might expect them (at least in a less frenzied atmosphere) to respond by questioning its assumptions, or offering counter-evidence. When critical voices fail to do this, it is probably because they don’t have good answers.
This does not mean that, as evaluators, we can be smug. We live in a world where the idea of “evidence-based” does not have a strong hold on the public’s imagination, and is anathema to many politicians. We need to work harder, and use the evidence we have to tell a more compelling tale.
