If you want to be popular, probably best not to go into evaluation. Pick another role, another profession.
However, evaluation performs a necessary and valuable function. Like street cleaning and colonoscopies, someone has to do it.
Evaluators are paid (tolerated?) to deliver sometimes unpleasant truths or hard-to-swallow advice. The role has evolved, slightly, since Medieval times when the king’s fool, among other things, had to speak truth to power, presumably in some palatable way like mixing humor and self-deprecation. Mercifully, we evaluators don’t need to dress up in funny costumes and makes fools of ourselves anymore (although sometimes we inadvertently do a bit of the latter).
In all honesty, the point of this blog post is not to deter would-be evaluators from entering the field. Rather, it is to warn you that you may not make as many friends as you would if you were, say, working in sales or marketing.
Also, when I say “unpopular” I do not mean that demand for evaluators is low. To the contrary, there is (still) lots of evaluation and evaluation-type work out there.
The fact is, auditors, tax collectors, inspectors, evaluators and their ilk – in what are sometimes called the accountability professions – are not really meant to be liked.
President Trump’s firing of six Inspectors General in the last few months notwithstanding, most people know that accountability is important. Like taking medicine or going to the gym, being evaluated or investigated can be disagreeable for the object of the evaluation, but there is also good chance it will make whatever or whoever is being evaluated better.
Why you need a thick skin
I have known of evaluators who have been threatened, fired, had their work trashed, or even been held against their will. Here are a couple of examples (one personal).
I once was leading an evaluation in Croatia on the impact of employee redundancies at the country’s shipyards. The data collection supervisor on our team, an intrepid young Croatian woman, was asked by a shipyard manager to turn over the list of (randomly selected) employees she was interviewing. When she refused, he locked her in the interview room and threatened not to let her out unless she complied with his request. She still refused, risking her safety and well-being in the name of professional integrity and respondent confidentiality. Luckily, she had a contact in the Ministry she was able to call, and the manager relented and unlocked the door. She was shaken, but able to continue with her interviews. The evaluation was completed and well received. The reforms, on the hand, did not happen.
It happens to evaluators – it has happened to me – that your findings and conclusions are rejected, even when I thought the analysis was strong.
Last year I conducted an evaluation-type study in an African country for the World Bank. It involved assessing the likely social impacts of a $100 million program. The manager who commissioned my work (in a division– referred to as “Global Practices” – at the Bank, with a focus on social issues) was happy with the analysis. However, the manager whose program I reviewed (from the Global Practice responsible for the assistance program) actually refused to speak with me. After reading my report and its conclusions, he rejected the analysis and brought in someone else to redo it. Not a happy experience, but you develop a thick skin in this line of work.
A popular way of rejecting evaluation findings is to question or attack the evaluation methods, or the evaluator’s qualifications. These are good ways of deflecting attention from the findings.
Often, you won’t know whether the client was unhappy with the quality or scope of the work, or if there were other internal politics at play.
These types of unpleasant experiences tend to be rarer, and less of an issue when the client commissioning the evaluation is not the one whose work is being evaluated. This may be the case with donors or US Government agencies such as USAID or Millennium Challenge Corporation evaluations. They are the agencies funding the work, while another organization implements it, so they are generally truly interested in whether the money is being well spent, e.g. is the program going according to plan and getting results.
You may not be popular, but you still have to be nice
With evaluators, when people are nice to you, it isn’t necessarily because they like you, or see you as a linchpin in their career progression. Of course, hopefully you’re a decent person with a disarming personality! But quite possibly, their chumminess could reflect a, shall we say, slight bias. You are evaluating their programs after all, and they most likely prefer it that you see them at their very best. If people are quite nice to you at the beginning, but when they realize that you’re serious about your job, they start cooling to you, you’ll know that the amicability was more of a tactic than anything else.
It is your job to look past the surface, dig into the data, find what what’s really happening, and report fairly. Yet, you yourself need to adopt an attitude of goodwill, and cordiality toward others, regardless of what your finding are, no matter how useless your inept or corrupt the program is. (I honestly have evaluated very few programs that fall into that category, quite possibly because everyone knows that the evaluators will be showing up.)
Why is being nice to others who might not be nice to you important?
First, being nice is simply part of being professional.
Second, you want to build relationships. You need others trust you and share information and, if all goes well, accept your findings.
Third, maintaining a pleasant demeanor is a simply good default attitude to have to contain whatever feelings you may have about the program you’re evaluating. Whether you think it is amazing or terrible, you want to keep those feelings separate from the work.
The bottom line is that being decent to others is a soft skill you want in your toolbox.
The evaluator as outsider
Closely linked to being “unpopular” is being an outsider.
Professional independent evaluators are, by dint of their position, outsiders. (This is different from internal evaluators who work within an organization.) You need to accept and embrace that role, even while building trust with the client and stakeholders you meet. You need to obtain information from them, and want them to accept your findings.
However, it is a fine line. As an evaluator, you arrive in a new place, with its own professional or work culture, maybe in a new country. You start poking around, and asking questions. That’s the job. People will be on their guard.
The outsider status is beneficial in that it can shield you from certain biases that you might bring. These biases might include if you were part of the system you are evaluating, such as belonging to one or the other political parties, ethnic groups, clans or other groupings of which you are probably not aware.
A corollary of this is that foreign governments often value non-nationals because they are outsiders, independent, not connected to a particular faction. Sometimes this is justified, sometimes not. Despite being an outsider, and thus often partially aware (or unaware) of the unwritten and unspoken codes and connections, there is value in standing outside, in not being part of the culture.
One can be less beholden, less biased, and face lower risk of consequences from producing unpopular findings, since everyone knows that when it’s over the evaluator will board the plane and leave the country.
The international consultant, very much an outsider position, also brings an international perspective to the table, based on evaluation experience in multiple countries and cultures.
The evaluator as friend?
I like the concept of critical friend, which I have found very useful in understanding and accepting my role as an evaluator. It implies that you are there to help through constructive criticism. One of the best descriptions comes from John MacBeath, a Cambridge University academic, in a 1998 article on improving school effectiveness:
The Critical Friend is a powerful idea, perhaps because it contains an inherent tension. Friends bring a high degree of unconditional positive regard. Critics are, at first sight at least, conditional, negative and intolerant of failure. Perhaps the critical friend comes closest to what might be regarded as ‘true friendship’ – a successful marrying of unconditional support and unconditional critique.
Good evaluators should be respected for their work. They are not going to be the most popular kid on the block and should not strive for that.
Sometimes evaluation findings are accepted, sometimes rejected, sometimes ignored. Sometimes you are hired again, sometimes you are not. As an independent consultant moving from one assignment to the next, one client to the next, you often never learn of the outcome of your work. It comes with the territory and you should not be disheartened by this.
I began this post by noting how evaluation is not a popular profession. In this age of online calumny and fake news, where many suspect any criticism of being driven by ulterior motives, the notion of accountability is more important than ever.
Nonetheless, if you stick to your guns, maintain your integrity and deliver credible and useful advice, you may be the best critical friend, the people who hired you have ever had.
Today we’ll look at when and how to share a piece of professional writing, specifically when it comes to producing notes or reports for international development work.
Three core elements of writing
Let me propose that writing is made up of three core elements: content, style, and management.
Content is what the writing is about, the rough material, the substance to be molded into shape, like turning a lump of clay into a sculpture. A near infinite number of topics are out there, waiting to be written about. The biggest hurdle may be deciding on which one to focus on. Think of content as the what.
Style is the form, the shape and design that the content assumes. Editing and revising for grammar, punctuation, clarity, structure; all those key style elements that Strunk and White, George Orwell and others address. The style must also be appropriate for the type of report and organization for which it is being written. Prose fiction, journalism, doctoral theses and UN reports are different animals. Each has its own logic and purpose and appropriate style. Think of style as the how.
Management is the process of producing the writing, the steps along the way, from concept to outline to draft to revisions to sharing for comment. As dull and prosaic (no pun intended) as this third element may sound, it matters quite a bit. You may have plenty of ideas in your head, and an inimitable writing style. However, if you don’t manage to write them down and prepare them for readers, it won’t matter. Think of management as the when, how much, where, to whom, etc. That is, anything that goes beyond content and style.
On a side note – if you don’t have at least basic competence in all three of the above, you will probably struggle. However, if you are part of a team responsible for the report, you can rely on each other. To a certain extent, team members can compensate for each other’s weaknesses.
Managing the written word
Perhaps for good reason, the third element, management, is the most neglected. What writer wants to think about managing their writing process? We are not all brilliant wordsmiths, such as the late Christopher Hitchens, who, astoundingly, could sit down and produce a polished, piece in just 20 minutes.
Most of us need to plan, and work rather laboriously toward the finished product. It’s not always pretty. Like any process, writing can be managed well or managed poorly, but it must be managed.
One way of improving writing is, of course, by exploiting the feedback of others. That becomes a little tricky when the feedback is from the manager, editor, or client who has commissioned you to do the work, however.
As you knit your brow and sigh over your unfinished report, questions you may ask yourself include:
You know your draft still needs work. Should you share it or continue revising it, even if it means delaying delivery?
How much detail should you include? Is it better to produce too much and pare it back, based on feedback? Or is it better to “park” extra material in reserve, and add it later, when you get the inevitable request to expand on this or that point?
Is it okay to leave in weak parts if you point out that you are aware of them and will continue to improve them?
Share now or share later?
You want to avoid sharing a draft that is too weak, since it could cause the reviewers to question your writing or analytical abilities, or commitment to quality.However, perhaps the draft is due in a few days, and you know it will not be ready. Should you deliver it on time, because that’s what was agreed, or ask for an extension? Generally, choose the latter: better to deliver late but good, than on time but poor.
However, if you do ask for an extension, let the person you report to, who will be reviewing the piece, know in advance. They may well be flexible, especially when it comes to internal deadlines.
If most sections are in good shape, but one section needs a lot of work, send the document with a note that section X needs a bit more time. Similarly, in those parts of your report where you know more information or analysis is needed, note that directly in the document, as a heads up. Better to deliver good writing that is incomplete — than weak writing that is complete.
If you have written good material but it exceeds the page limit you’ve been given, either hold it back or create an annex. The annex is your Joker card, where you can put surplus (but hopefully good) writing, and save it from oblivion.
Internal deadlines are your friends. Deadlines are often not hard, and delivering a few days, or even a week late might not make a difference. The writing process is managed by setting a series of intermediate milestones that need to be met. This is not college, where you can cram all night and deliver your paper before class in the morning.
Keep people in the loop on your progress. Share outlines, concepts, or smaller sections of the report in advance, to show that the work is progressing, and to get feedback. Or, have a few meetings or calls to keep the people who will be reviewing updated. This process is similar to how companies release “beta” versions of software programs, piloting them in the real world before upgrading them to the final release version. There is a clear advantage to sharing a work in progress, as opposed to a work you feel is close to complete. Others will see things that you don’t, making the revision process more efficient.
Unless you have an open-ended or a very long timeline, and almost no other obligations, remember that perfection is not the goal. If you are a consultant, you have been given a fixed budget or finite number of days. You need to produce competent, high quality, relevant work. That is what is asked and expected of you. Of course, you can always do more and better. But the best way to get there is through repeated practice, not fixating on this one piece.
Some reviewers will be more forgiving than others. Some focus on strengths, others on weaknesses. Your work will benefit from both. However, you will know which ones you can trust to provide constructive feedback on preliminary efforts. Reviewers are your allies. Use them.
There are no strict rules, but the better you are at managing how and when you share your work, the less stressful it will be, and the more pride you can take in the result.
For writers, the road to perfection passes through the review purgatory.
The fact that the review process is collaborative makes writing both easier and harder. Easier because the burden of improving the writing is shared. Harder, because you as the writer are exposed to the scrutiny and criticism of others.
In this blog post I will propose some ways of smoothing the rocky passage, from the perspective of both the giver of feedback and the receiver.
In the fields of international development and evaluation, sharing written drafts for comment is standard practice. While essential to good quality outputs, it is also a laborious process. When a report is developed, it can go through multiple rounds of revisions over a period lasting weeks, and even months.
A silent dialogue
Imagine, in this fast-paced world, a slow-motion conversation in which the speaker takes as much time as he or she needs to reflect and ruminate on a given subject. It is an extended back-and-forth dialogue with the audience that starts…and…stops…and…starts…and…goes…on…for…weeks. The dialogue continues until…no one has more to say, and it is finally over.
On top of that, imagine that the issues under discussion are technical in nature, and would mean little to most people listening in.
Such a conversation might sound agonizingly dull. But it mirrors the way good reports get written. Most of this dialogue is not spoken, of course. It takes place on the page. The back-and-forth is the writing, reviewing, commenting, editing, revising, and rewriting that happens in the “document space.” It is usually very effective. I have seen plenty of reports transformed from sub-par to excellent as a result.
A far from dull process
Being the responsible writer in this ‘dialogue’ is far from boring. It can, in fact, be nerve-wracking, waiting to see/hear what the other person thinks, knowing they will find and point out weaknesses (which is their job), and wondering how tough they will be.
If you are the primary team member responsible for the writing, after you have labored over your draft, there’s always a moment of trepidation after hitting the “send” button. Is the report on the right track? Is it broadly acceptable? How difficult will the comments be to address? How many comments will there be? Should I have spent another day revising before sending it in? If you are a consultant, you may even be wondering, Will I ever get hired by this client again?
In the fields of evaluation and development, you will be in the role of reviewer as well as reviewee. You need to be able to dish it out as well as take it. That is, to give and accept critiques of the reports that you and others write. These are skills most development professionals learn over the course of their career.
(I should note that the word “critique” is not normally used in these circumstances. It’s a little too loaded, perhaps. The preference is for “comments,” “feedback” or “review.” But critiquing is essentially what’s happening.)
When I first began working in this field, I was delighted to find weaknesses in a given document. Being asked to review the work of others gave me the sense that I had arrived, that I could hold my own among colleagues, most of whom were older and far more experienced. It gave me a confidence.
Unfortunately, it also occasionally caused me at times to become a bit cocky in my reviews. I may have expressed my reservations in language that was a little too harsh. I’ve learned that, on the written page at least, while being straightforward is fine, being severe is unnecessary and unhelpful.
Who comments?
Comments can be generated by the client, the manager overseeing the task, other members of the team, and by specialists from outside the team. For official publications, an editor will be hired. For certain documents, the feedback process may be formalized as a peer review process, as is done for academic journal articles.
If you are new to the field, you’ll need to get comfortable with feedback, because it will always be there. If you don’t receive feedback from someone, it is not because you are brilliant, I’m afraid. It’s because they either didn’t review your draft, or didn’t read very carefully.
On the flipside, you can always find constructive ways that work written by others can be improved. It may take a couple of read-throughs, but issues will come into focus, like those magic eye pictures or autostereograms that reveal a 3D image if you stare at them long enough.
Receiving a lot of comments means additional work, of course. But you want substantive feedback: it will make the report that much better. What you want to avoid is receiving feedback along the lines of “I don’t understand what you’re trying to do here” or “this is not what we were expecting” or “the quality is unacceptable.” That generally means you need to start over.
Then there is the pedantic reviewer, who finds fault with every minor issue or who gratuitously asks for everything to be explained ad nauseum, which can also be stressful.
Below are a few things to keep in mind:
On receiving feedback
All comments from the client or your manager will need to be addressed, either by incorporating them in the text or by making a good case for why not.
Other than the above two cases, you don’t need to address every single comment. Indeed, some comments may contradict one another.
When deciding the order in which to address comments, consider plucking the low-hanging fruit first. Addressing the easier comments first will give you an encouraging sense of progress, and additional time to reflect on how to tackle the trickier feedback.
Keep in mind that it is good if you get no feedback suggesting that the work’s approach was wrong, or if they don’t ask for a complete redo.
Before sharing your work with others, it is paramount to edit one’s own work. That is one way of reducing the amount of comments you will receive.
To avoid going down a cul-de-sac, it is a good idea to communicate with people who will be reviewing the piece before you send them the final draft to review. Share ideas and outlines with them early on, and incorporate feedback. This generates interest and buy-in for the work.
Don’t take criticism personally. It’s not you, it’s the writing.
On giving feedback
As you review, ask yourself:
Is the piece addressing the stated objectives, the questions it poses?
Is anything important missing?
Does the structure work?
Does the work flow in a natural progression?
Are certain elements underemphasized or overemphasized?
Are there any errors?
Avoid framing comments in a negative way, e.g. “this is incorrect” or “you didn’t understand.” Positive turns of phrase include “I suggest” or “think about phrasing it this way.” You don’t want to demotivate the writer with harsh criticism.
Although you probably have not been asked to copyedit, if you do come across grammatical errors or typos, it’s not inappropriate to simply make the correction. When I do that, I’ll add a note to the effect “that I took the liberty of doing some light editing” or that I “made a few edits along the way.”
There will be reports which are poor quality, or completely miss the mark. Remedial measures may be needed, including a complete rewrite, or even another person to write it. Even this situation should be handled diplomatically.
You can write comments directly into the report, and also include general comments in the body or in your email response.
Use the sandwich approach — start with what you like about the report, and end on a positive note. Highlight the strengths. That’s encouraging for the writer.
As a general rule, feedback makes everyone’s work better. It is the essence of quality control. Having more eyes poring over a report, more brains scanning it, is effective for uncovering issues before a written work is signed, sealed and delivered.
Sometimes it may feel as though you are getting hammered by critics. If your critics are insightful and forthright, what they’re really doing is helping you hammer your work into shape. And that’s a good thing.
