INTRODUCTION
Always-on surveys or annual employee engagement surveys? Qualitative or quantitative? Satisfaction or engagement? Are you more Hobbit or Harry Potter? We are daily bombarded with surveys, tests, and assessments of all sorts. But what is the difference between surveying and soothsaying? Is it possible that an employee engagement survey could cause more problems than it solves? What makes a good survey and, more importantly, should we even be running a survey?
Understanding employee engagement is critical to any organization. When employees are engaged, they throw their hearts, spirits, minds and hands into their work. This discretionary effort results in innovation, dedication, improved quality, lower turnover, increased customer service… the list goes on. Employee surveys play an important role in understanding how this level of engagement changes over time. Their primary purpose is to provide vital feedback, identifying influences for predicting and shaping future performance.
Imagine trying to kick a goal when you weren’t able to detect the distance, the direction, or even if you made contact with the ball! Trying to run a business without accurate employee feedback would be just as farcical. In this article, I will discuss a number of concerns, and how to cope with them while creating your own survey protocol. I will use an employee engagement survey as an example, although the methods detailed below can be generalized to almost any type of survey.
Here are 10 questions to consider before your next employee engagement survey:
- Should we even conduct an employee survey?
- What makes a good employee engagement survey?
- Can a good employee engagement survey still create problems?
- What counts as an employee survey?
- How often should we conduct an employee engagement survey?
- How long should an employee engagement survey be?
- How do we design a good employee engagement survey?
- Does it matter how an employee survey is administered?
- How do we interpret employee engagement survey results?
- Are we done yet?
QUESTION 1: SHOULD WE EVEN CONDUCT AN EMPLOYEE ENGAGEMENT SURVEY?
Yes! Are you still skeptical? Ok, consider this model of goal achievement. When I was designing college curricula, I broke my planning into three distinct areas: the goals I wanted to achieve, the activities that would accomplish those goals, and then some method of assessing whether the activities resulted in the goals. If not successful,
I would use the feedback from the assessment to reevaluate the activities and, sometimes, the goals.
This iterative process applies to any goal, in any industry. Furthermore, every step is necessary. Omitting the goals step leaves us directionless. Omitting the activities step leaves us adrift. And omitting the assessment step leaves us ignorant.
Without feedback, we tend to create our own realities. Often, those “realities” aren’t accurate. Yet, organizations that fail to gather feedback often make assumptions— and the results of these assumptions can be costly!—about what they think is going on in an organization.
Obviously there is some nuance to that position. What makes a good survey? Can even a good survey create problems? And what even counts as a survey?
QUESTION 2: WHAT MAKES A GOOD EMPLOYEE ENGAGEMENT SURVEY?
Speaking broadly, there are two fundamental concerns when discussing surveys: reliability and validity. The first is reliability or the ability to provide consistent measurements. While important, it can be misleading, like a body weight scale that is 10 pounds too heavy or a clock that runs 5 minutes slow. It is useful but still not accurately measuring the “truth.”
This leads us to the second component of an accurate surveying, validity, or the ability of a survey to measure what it purports to measure. A scale that is consistently off is not measuring what it claims to measure (in the case of the body weight scale, the amount of force caused by gravitational pull). It becomes even more difficult when you leave the realm of physics and attempt to validly measure people.The vast majority of problems that arise with a survey can be categorized as a violation of either reliability or validity.
QUESTION 3: CAN A GOOD EMPLOYEE ENGAGEMENT SURVEY STILL CREATE PROBLEMS?
It would seem that if you had nailed both reliability and validity that it’s time to sit back and count how big your raise will be. Unfortunately, like I’ve already alluded to, measuring opinions can be a messy business.
Because we don’t exist in a vacuum, sometimes the simple act of having an employee survey can change the results. Known by social scientists as the observer effect, sometimes it’s a good thing. Research shows that more frequent surveying can actually improve performance outcomes. Other times it’s less positive. If the last song you belted out in the shower now had to be performed in front of your board of directors, would it be an accurate approximation? Several years ago, we had a client that came to us with a problem. They had just completed a do-it-yourself survey in which they asked how nervous their employees were about potential layoffs. They created a rebellion because by asking the question, they had succeeded in making people more nervous for possible layoffs.
In the Case of the Pending Layoff mentioned above, the very act of conducting an employee survey changed the outcome of the survey, either by employees masking their true feelings (“Wow, I better change my answer or I might be out the door!”) or by actually changing attitudes (“They’re asking about layoffs—should I be worried?”), this is a validity problem. It’s not insurmountable, but it does require some expertise to minimize possible difficulties.
Another problem is dependency. This is one that arises frequently with an always- on type survey. If a manager knows she can just read the results of a weekly survey, it would be unusual if she didn’t come to depend on those results, rather than cultivating personal relationships with employees. This isn’t a problem with the survey (i.e. reliability or validity) but, rather, with human nature. Unfortunately, much as water seeks the easiest path down the mountain, people tend to conserve mental resources. The solution may be to redefine what our intuitive definition of a survey may be.
QUESTION 4: WHAT COUNTS AS AN EMPLOYEE SURVEY?
We naturally think of employee surveys in terms of formal, “fill in the bubble sheets with a No. 2 pencil” types of programs. But that ignores the surveying we do all the time. When you walk by a colleague and observe their body language, that’s a survey. When you ask about their day, that is a survey. When you interpret their words, that is a survey.
I spill a lot of digital ink decrying how capricious our intuitions are and how inaccurate our gut tends to be (i.e. reliability and validity respectively). But in trying to swing the pendulum away from instinctual decision-making, I invariably give a shout-out to its benefits. And while every interview of a serial-killer’s neighbor sounds the same for the most part (“I had no idea! They seemed so normal!”), given time, we can get a fairly good idea of who a person is and how they are feeling.
Ideally, a manager would be able to validate her or his own feelings by cross- checking with a formal survey. Much like science doesn’t ever prove anything by one dramatic study but, rather, simply attempts to predict by a boring system of replication and validation over a series of years, we can borrow those principles by using a dual-pronged approach to surveying without over-weighting any one method or result.
Again, this is easier said than done, and people train for years to be able to identify how best to survey an employee population. So let’s stand on their shoulders and explore some best practices.
QUESTION 5: HOW OFTEN SHOULD WE CONDUCT AN EMPLOYEE ENGAGEMENT SURVEY?
Ah, that’s easy! It depends. I wish I could give an easy answer for this, but it is entirely contextual. An always-on survey allows for instant feedback at the possible expense of survey burnout or biased sampling (we respond only when we’re really happy… or really ticked off). An annual attempt has the benefit of standardization and being perceived as more important, but without the immediacy of results. Consider surveying as a spectrum, one end being constant feedback being generated and the other end being feedback generated once every 10 years (like the U.S. Census). Each has its positives and negatives, and the more you move along one way of the spectrum, the more sacrifices you are going to have to contemplate.
The important point to remember here is that while it is tempting to create a survey that you can pull of quickly and simply, the real goal is to create one that provides valid, useful feedback. So, while you may be able to get all your employees to take a 100-item survey or have them fill out a short survey every week, if the employees get tired or bored doing it, their hurried answers won’t be accurate (read: not valid). Another example is how a survey which employees can fill out at their leisure (or not at all) will lend itself towards what is known as a bimodal distribution, where extreme viewpoints are overrepresented. It’s why when you read restaurant reviews online they either laud the food as ambrosia or describe the special place in hell reserved for the chef. You hardly ever see something describing the experience as a perfectly adequate expectation-meeting experience.
The ideal assessment would be a longitudinal design (several surveys over a period of time) which participants were highly motivated to complete without a great deal of effort. If there were a particular change you were interested in (such as how performance changed after a raise or change in management), then a microgenetic design (several surveys over a shortened length of time) would be appropriate. In either case, the principle is to capture several data points for the same question so that variables like “you caught me in a bad mood” or “I just came off of vacation” can be controlled for.
The highly-motivated employee is likely your sticking point from the previous paragraph. Despite what economists will tell you, offering rewards will not necessarily improve your employee survey. Offer a cash reward to fill out the survey and you better believe you’ll get employees that are highly-motivated…to complete the survey. And what’s more, you’ll probably find they’re completing them faster than ever before, almost as if they weren’t putting any thought into it! Go figure. So how do you get employees to want to put effort (and candor) into their answers?
Humans are awful at busywork. We crave meaning and the feeling that our actions have farther-reaching effects than the behavior itself. And research shows that it doesn’t take much to help us feel that. If you want employees to feel that a survey is worth their time and effort, they need to see results from their work. This doesn’t mean if everyone writes about needing a foosball table you necessarily put one in.
What it means is that you acknowledge their request and address why it can or can’t be fulfilled (money, productivity, etc.) and address the underlying real concern (why they would think they need a foosball table).
QUESTION 6: HOW LONG SHOULD AN EMPLOYEE ENGAGEMENT SURVEY BE?
Again, the short answer is simply a pithy remark that alludes to larger principles: an employee engagement survey should be just long enough to get what you want, without compromising validity. And it is very closely related to the frequency with which you administer assessments. The more frequent the assessment, the less comprehensive it can be, and vice versa.
When considering more frequent assessments, the biggest validity concern should be whether employees will fill it out to the best of their ability. Related to that is one of the biggest pragmatic concerns – how much time needs to be budgeted away from their work to do so. Remember, I discussed how making their efforts worthwhile is the biggest part of it. The next part is simply to show that level of caring with taking up as little of their time as possible. Make the survey short and convenient. If you want qualitative answers, ensure that the question is pointed enough that they don’t have to interpret it. As a rule of thumb, we recommend about 10 items for frequent pulse surveys, and between 35-50 items for a comprehensive yearly employee engagement survey. But, be careful with those numbers. You need a survey that works for your organization. There’s no single best way that fits every organizational need.
QUESTION 7: HOW DO WE DESIGN A GOOD EMPLOYEE ENGAGEMENT SURVEY?
My first bit of advice is: you don’t. Get an expert. Very seldom does a day or two go by that we don’t see a horribly designed in-house (designed by the organization) survey. But if you don’t, for whatever reason, choose to use experts trained in survey design, I’ll give you Plan B. I will attempt to condense several years worth of high-priced university study into one sentence. Come up with a question, and then try your hardest to undermine it. How many possible interpretations are there? Is it worded at higher than a sixth grade reading level? Is it so long you tuned out while reading it? Is it fishing for a particular answer?
A common rookie mistake I see is to create double-barreled questions (two questions within a question) – “Would you be happy with a raise or better equipment?” – does a yes mean to the raise, the equipment, or both? But additionally, what does happy even mean in this context? What does better equipment mean? When you say raise, what amount are they thinking of? Maybe the amount you’re thinking of would be insulting.
Another consideration in survey design is what we refer to as actionability. In other words, based on the responses from the survey, can we take action? Is the survey designed in such a way that the organization is able to delve into the strengths and areas for improvement, and create action plans based on these results? If not, you may have gone through an expensive process, with very little return.
There is an art to creating good assessment questions, but if you follow the rule of being your own harshest critic, you stand your best chance of creating valid test items.
QUESTION 8: DOES IT MATTER HOW AN EMPLOYEE SURVEY IS ADMINISTERED?
No. But also yes (are you starting to see a pattern in my responses?). In a cognitive sense, everything is about perception, and people are good about imbuing meaning onto things, regardless of how something is manifest. However, the nature of the input can affect other factors such as test fatigue. For your typical office employee, answering an online survey that came through an email is much more convenient (and subsequently more likely to be completed) than having to fill out a bubble sheet with
a pencil and then depositing that hard copy somewhere. Conversely, your warehouse workers are probably more comfortable if they meet in the break room to complete a paper survey that is submitted to a ballot box while on the clock with a cup of coffee.
There is no perfect way to administer a survey. The good news is that this lets you exercise your creative muscles to come up with what works for your situation and environment. Use emails, texts, or paper surveys (phone surveys don’t work well- catch me another time and I’ll tell you why). Have a ballot box or computer kiosk set up where your line workers can easily access them. Proctor paper surveys in a break room or environment where it’s comfortable to focus. We even saw one organization place a trashcan in the middle of the room and had people crumple up their completed questionnaires and try and shoot a basket! Unusual, and it wouldn’t work for all organizations, but it fit well with their culture.
QUESTION 9: HOW DO WE INTERPRET EMPLOYEE ENGAGEMENT SURVEY RESULTS?
I really appreciate the rise in sports analytics because it’s slowly changing how we think about interpreting data at a cultural level. For a long time we just accepted certain truisms, like baseball RBIs were indicative of a good batter, the closer a shot in basketball the better, and teams should spend a lot of money on good midfielders. We know that complex behavior requires more complex thinking, and just because something is measureable, doesn’t mean it is valid.
The first example is using frequencies and ratios as measures. I worked with a client that wanted to measure whether a manager’s ability on one measure predicted
the performance of their team. By looking at the percentage of managers who had high-performing teams, they concluded that there was, in fact, a relationship and they needed to implement a program that helped managers increase their ability in that skill. Now, the first red flag is deciding whether the manager’s score caused high-performing teams or whether high-performing teams caused the manager’s score. But leaving that aside for now, the bigger problem was that because the size of the groups was different, using a ratio weighted the scores more heavily in one group. In other words, if I have a group of 1,000 people and a group of 100 people, the scores of the 100 people are worth 10x as much if I turn both groups’ results into percentages. In fact, a correlation revealed that there was not a relationship between manager scores and team performance after all. They acted on information that didn’t exist (or was inaccurate, at best).
But correlations can be misleading as well. A simple correlation is taking one variable and comparing it to another variable. It captures relationships and gives them a score along a spectrum of 1 (perfect relationship, e.g. the relationship between me pressing the “j” key and a “j” appearing on my screen) to -1 (perfect inverse relationship, e.g. every step up I take on a staircase vs. the amount of stairs remaining). If there is no relationship, then the score is simply 0. As a general rule, anything between -.2 and .2 is considered trivial. The correlation from my previous example (manager scores) was .05. There is a stronger relationship between movies Nicholas Cage has appeared in and swimming pool drownings.
This brings me to next my point, correlations are fairly susceptible to misinterpretation as well. For example, that aforementioned relationship between Nick Cage and drowning is actually .66, a relatively strong correlation. And the correlation between how much cheese Americans eat and the amount of people who die by getting tangled in their bed sheets is .94—almost perfect! While it’s possible that cheese and bed sheets are related, it seems more likely that a third variable, for example population increase, could be a valid explanation for both.
These are just two examples (far from exhaustive) of how simple analyses can be flat out wrong and at their worst, completely misleading. The best practice here would be not to skimp on having an expert, someone who is trained in identifying the assumptions and possible pitfalls inherent to a dataset, handle the analysis and interpretation. It’s essentially the same reason I didn’t let my neighbor perform my appendectomy, despite his convenient location, positive mental attitude, and the fact that he got a new tool set for his birthday.
QUESTION 10: ARE WE DONE YET?
There are three take-home points that I want you to walk away with. The first is that assessments and surveys are crucial to growth. All learning in all human history is based on getting feedback and using that feedback to predict future behaviors and events. The second is that surveying is a legitimate skill that can be greatly improved by expertise. While everything is better served having an expert involved, there are a number of factors that will improve your employee engagement surveys starting day one. And the third is to think about surveys in terms of validity, not feasibility.
Just because a survey can be made doesn’t mean it should. The market is flooded with employee surveys and companies that produce results, just not results that mean anything. Remember, a survey does little good unless the organization can act upon it (and does so). But with validity as your lodestar, and an effective, actionable survey, you have one of the most powerful agents for affecting organizational change in your hands.
