Why the ‘Campus Climate Surveys’ That Are All the Rage These Days Are Seriously Flawed

By Jillian Kay Melchior | 11:32 am, October 7, 2016

The University of Northern Colorado has spent $72,000 on a “campus climate survey” to try to gauge from students whether racism and sexism are problems at the school. UNC will use the results as a “blueprint” for its campus-wide strategy for expand diversity and inclusion.

But there’s one big problem: Most students aren’t participating. To encourage students to take the survey, the university offers a raffle of prizes for participants, including a $300 Visa gift card and a $300 voucher for professional-development travel.

Still, less than 6 percent of the students have opted to take the survey, said Katrina Rodriguez, the vice president for campus community and climate. Beyond the low participation rate,  how the questions are phrased and even when the survey is given may be skewing the results at UNC. The net effect is a survey that at best may be highly inaccurate—and at worst is vulnerable to claims that it was rigged to fit a particular political agenda.

UNC isn’t alone. Campus climate surveys have become commonplace at colleges, from private schools like Harvard and Stanford to big state schools, including Mizzou and the University of Wisconsin and University of California systems. While UNC’s response rate was particularly low, many campus climate surveys are plagued by the same problem. Some colleges have attempted to get results that better represent the student data as a whole by providing the survey to a randomized group of students or using sample sizes that mirror the student body’s demographics.

UNC’s campus climate survey is taking place as the school struggles to balance diversity and inclusion with free speech and academic freedom. Records obtained exclusively by Heat Street this summer revealed that the university’s Bias Response Team, which was created to investigate students complaints of everything from discrimination to “microaggressions,” had asked professors to change their lesson plans to avoid controversial topics in the classroom.

After an outcry from the public and at least two Colorado state legislators, UNC President Kay Norton said she was dissolving the Bias Response Team and would take further steps to protect free speech while still promoting diversity and inclusion. Norton has called the campus climate survey “critically important to our continuing work to be an inclusive community.” She even mentioned the campus climate survey in her State of the University speech last month.

Neither Rodriguez, nor the university spokesman or two other faculty members involved in the campus climate survey responded to Heat Street’s request for an interview. The consultancy conducting the campus climate survey also didn’t respond.

But three statisticians interviewed by Heat Street raised significant questions about UNC’s methodology, pointing out ways that the survey could yield flawed results that don’t accurately represent the majority of students.

“I would hesitate to use it myself with such a low response rate,” says David Mundfrom, a statistician and professor at Eastern University of Kentucky. “If I’m going to say the responses here are representing our students as a whole, or a campus climate as a whole—I’d be skeptical if I read that in someone else’s documentation, if that was the conclusion they drew.”

Mack Shelley, a professor of statistics and political science at Iowa State University, says such surveys can be inherently slanted. “Absent some kind of really good reason to want to respond to these things, it’s difficult to gin up a whole lot of interest from students, mainly because they’re worried about getting to the next class or they have some kind of assignment due,” says Shelley, who has helped out with other campus climate surveys. “There will be some students for whom this will be a high priority, and they’re the students who are on the receiving end of some kind of untoward behavior.”

Steve Simon, an independent consultant and statistician, put it this way: “There’s a fair amount of evidence that people who respond to a voluntary survey are likely to have polarized attitudes [and to be] people who have strongly held perspectives.” The average, generally satisfied student isn’t likely to take 20 to 30 minutes to fill out the campus climate survey.

The statisticians also pointed out a problem with some of UNC’s other survey practices. In particular, Rodriguez said in a public meeting this week that some offices had asked their student to take the survey during work hours, “and that’s great.”

But if several like-minded staffers in an office take the survey—particularly if they’re an office with a specific focus or priority—it could also skew the data.

Though the climate survey is supposed to be taken only by students, staff and faculty, Heat Street was able to find the link online and complete it.

We noticed that several questions and pre-written answers on the survey may also yield confusing results. For example, participants are asked if they’ve experienced “exclusionary… intimidating, offensive and/or hostile conduct.” But that list even includes things like getting a low performance evaluation or feeling isolated and left out on campus–experiences that many people would argue have nothing to do with offensive or hostile conduct.

Similarly, the survey asks about “unwanted sexual conduct.” But there’s only one option for “sexual interaction,” ranging from cat-calling to repeated sexual advances to overt sexual harassment. Being “ridiculed” falls under “relationship violence,” under the same umbrella as “hitting.”

After asking the questions, the survey provides participants with a link for “campus climate resources, including counseling, in case “you experienced any discomfort in responding to these questions and would like to speak with someone.”

— Jillian Kay Melchior writes for Heat Street and is a fellow for the Steamboat Institute and the Independent Women’s Forum.