Garbage In Garbage Out

What follows is a case study in media polling that we should not trust, but rather treat as our enemy. It makes a few points about health care as an issue in US politics. But primarily it is episode 10,255 in my ongoing screed against garbage in garbage out polls and shitty media spin generated from them. My longstanding dissatisfaction with polling about US religion (for example here) led me to this place—triggering and informing what I will say—but that aspect of the problem is for another article on another day. 

Instead, I will describe how I tumbled down a different rabbit hole created by this general problem. It started like any other day, when my friend John, who has wide experience working with state governments and insurance companies to improve health care, sent me this article from a website called Liberal Patriot. This was in the context of Republican efforts to slash Medicaid funding, and in part it followed up on earlier conversations with John about this podcast which discusses grassroots organizing on health care as a model for fighting Republican austerity politics in red states.  

I don’t always agree with Liberal Patriot, and in this regard its willingness to trust the poll I will discuss is symptomatic. However, in this case I do agree with its key takeaway, which is that Democrats should prioritize improving US health care. Sadly, its argument appeals to a poll from the New York Times that did not pass an initial smell test. My problem began when I naively hoped I could clarify my questions about it with a quick detour through the Times article. 

As summarized in the Times headline, its pollsters found that “Many Americans Say the Democratic Party Does Not Share Their Priorities.” Supposedly Democrats prioritize abortion and LGBTQ issues over “cost of living,” so that Republicans are “more in sync with the mood of the country.” Liberal Patriot largely concurs with this thought, then uses it as a starting point to push in a different direction, zeroing in on health. It hammers on how, amid a list of possible priorities the poll ranks, only health care showed up in a sweet spot—at the same time, strong proportions of “Americans” prioritized it and Democrats were perceived to care about it. Although it wasn’t the top Democratic priority, it wasn’t a Republican priority at all. Liberal Patriot proposes to build on this in various ways—more wonky and piecemeal than Bernie Sanders’s vision of Medicare for All but headed in the same general direction. Surely there is much to say about the pragmatics of what to do during however long it takes Bernie to get the Titanic of our system turned around. John and I like to kick around such questions. But that’s a different article than I’m writing today. Rather, as already noted, I want to rage against garbage in garbage out polling and shitty media spin generated from it. 

My screed takes the form of recounting how I descended into this shit as I tried to clarify a simple question provoked by these two pieces. Exactly how did the Times come to think it knows what “many Americans” care about—regarding health care and more—amid what it thinks it knows about people’s “perceptions” of Democratic priorities? Compared to what other issues? Was the methodology sound? Did it have a passable sample? Did it ask reasonably unbiased questions that did not preselect its findings? Did it even summarize its own results well? 

In general, my answer is… just no!… it’s a pile of garbage. Liberal Patriot does useful work with parts of it, but might have done just as well starting from nothing. The kernels of gold it salvaged were from a slurry of right-wing talking points and strategic omissions.

But, Mark, Show Your Work!   

Here’s part one of my saga. Near the end of the Times piece, I read that “you can see the exact questions that were asked and the order in which they were asked here. Clicking took me to a summary of poll results. OK, great, take a look at this for a few minutes and that’s a wrap!  But spoiler alert, I gradually came to see—nearly an hour later—that this summary leaves out out questions 12 to 16 out of 46, which include most of the evidence about health care and decisive parts of the evidence for what “Americans” think Democrats care about.  

However, this link does offer revealing information about the poll’s sample, method, and lines of inquiry. The polling was done over the internet with a sample of 2,128 people—42 per state. This was down from an initial pool of 3333 recruited “to ensure representative coverage.” Afterward, as the Times summarized, “more weight was given to respondents from demographic groups that are underrepresented among survey respondents, such as people without a college degree.” When I dug for more about the underrepresented categories being smoothed out this way (or better stated, forced to conform to benchmarks based on what else but previous polls!) I found that these include gender, race, region, education, income, “English language dominance,” and “Presidential vote choice.” Supposedly the margin of error “based on the entire sample of adults” was 2.6%, but one wonders how high the margin of error rises for various subsets of these categories addressing various issues.  

Don’t get me wrong, such sample sizes and overweighting are common. I’m skeptical about the whole trend, not just this case.   

The Incredible Missing Health Care Evidence

Scanning quickly once, and then again, through this underwhelming base of data about “exact questions that were asked,” I did not notice the word “health care” mentioned. The poll mainly flags a set of hot-button issues—sometimes dumbed down to single words like “immigration,” “taxes,” or “guns”—and then asks none-too-nuanced questions about “support for" and/or “optimism about" how Trump will address them. It sorts answers into results for Republicans, Democrats, and the aggregate group. The most interesting thing I noticed here that seemed to be grist, however indirect, for the Liberal Patriot's mill was that 68% of the people agreed: “The economic system in this country unfairly favors the wealthy.” As to following up on this thought, there was little unless you are far more confident than I am about extrapolating from reported “optimism” about “taxes.” 

On my third time through, I finally slowed down enough to find the word! It was built into a pair of statements pitted against each other: “Immigrants today are a burden on our country because they take our jobs, housing and health care” (emphasis added) versus “Immigrants today strengthen our country cause of their hard work and talents.”  41% agreed with the first versus 56% for the second. 

Perhaps more relevant, I also noted questions about “puberty-blocking drugs”—narrowing to rules for doctors about prescribing them to minors—and parents’ rights not to vaccinate kids. Most importantly, there was a choice between “Government often does a better job than people give it credit for” versus “Government is almost always wasteful and inefficient.” Here a pro-government perception strongly prevailed, 60% to 39%. One wonders if this relates to perceptions of Social Security, Medicare, and Obamacare. Never mind, the poll doesn’t pursue that. 

Finally we are close to where Liberal Patriot picks up the fumble and tries to gain some yardage. But recall that it claimed to be starting from data about the priority of health care compared to other concerns. The only apparent straw available to grasp such data, so far, from the Times, was to clarify how the queries about the hot-button words related to an open-ended question posed to two sets of people—those sorted as “excited” and “optimistic” versus “worried” and “pessimistic” about Trump. Each group was asked what issues it most cared about in this connection. Then, as the pollsters put it, “open-ended responses have been coded.” This generated two ranked lists, one each for the worried and the excited. 

The results? In line with the Times’s headline, the coders conflated “economy/inflation” into one category which rose to the top. They jammed “abortion, women's rights, sexism” into one category that landed quite low. They mashed together an omnibus category of “cronyism, favoritism, abuse of power, corruption, enriching the wealthy and disregarding interests of the middle class” which landed fairly high, especially for the worried. 

We are left to query whether “health care” can be discerned as a dog just barely barking if we can somehow cross-reference these “responses that have been coded” with other questions. Also, Liberal Patriot has some of its own numbers. However, health care doesn’t appear on either ranked list in the Times.  

Finding the Needle in the Haystack! 

Increasingly baffled but determined to find the data that Liberal Patriot claimed as its take-off points, I eventually found another sentence in the Times. It offered this link alongside the following sentence: “asked to identify the Democratic Party’s most important priorities, Americans most often listed abortion, LGBTQ. rights and climate change.” The link takes us to what seems—at both first and second glances—to be the exact same report we have been discussing (the one linked at the end of the Times piece, promising specifics). However, with persistence we can discern that this second report includes questions 12 to 16 out of 46, which had been excluded before.   

Now we can see that poll-takers were offered 23 options and asked to choose the most important ones—first “to you personally,” then to Democrats, then to Republicans. One of these 23 was “health care." 

Note well, this is all of health care. This is the option you should choose if you do or don’t like any of the following: Medicare for All, Obamacare, vaccinating your kids, reproductive rights, or various implications of gutting Medicaid including rising insurance premiums and closed rural hospitals. Meanwhile, if you like or dislike abortion rights or puberty blockers or cancer research you might need to choose “abortion” or “gay/lesbian/transgender policy” or “education” as well.

Competing with health care on this list are “the economy/inflation,” “foreign policy” (once again, all of it!), “immigration,” plus a long list of apples and oranges: “taxes” (pick this if you want either to raise or lower taxes, either for billionaires or poor people), “corruption,” “racism,” “guns” (pro and con), “the state of democracy,” separate categories for “education” and “student loans,” and more.  

Not appearing as salient choices include the following: Anything directly mapping onto the 68% who worry about a rigged economy created by and for billionaires. Anything else we might say about the economy distinct from “the economy/inflation,” such as affordable housing or jobs that pay a living wage. Anything about protection of online data or social media. Anything about the role of dark money in political propaganda (except perhaps if you choose “censorship/free speech”?) The list goes on.  

Still, let’s stop to savor our victory. Finally we have details about the ball that Liberal Patriot is running with. Health care came in second (30%) on the list of concerns for “you personally.” It was tied for fifth (17%) as perceived concern of Democrats, and was far down the list (5%) of Republicans' perceived concerns. 

This makes sense. It makes me wish I could trust this tidbit of the overall data and feel OK about appealing to this poll given how much damage it does through leading us by the nose on so many other fronts.

Another Dog Not Barking! Christian Nationalism

Remember I said I didn’t want to write about religion polls today? But let’s make two quick exceptions. 

First, religion—including “Christian nationalism”—is another dog entirely failing to bark, either in this poll or the two articles spinning from it. It’s not even an afterthought, but more like a non-thought. Perhaps, in light of the overall shit show, that’s a blessing. Yet surely it’s another thing to note about this contestable framing with huge margins of error. If we want a movement for better health care, surely we will need some of our nation’s religious folks to be part of it.  

Since right-wing Christianity is a key part of Trump’s base, it must mean something that “open-ended responses were coded” with no apparent inkling that any religious values—whether pro-Trump or pro-social-safety-net or pro-common-good—had any bearing on anything. But what does it mean? Is the issue with the “many Americans” or the pollsters? Surely some questions that probed this could have turned up something to consider, however distorted or low priority.   

Second, let’s note how the Times’s most cited religion poll guru, Ryan Burge, wrote on his Substack

about how samples far larger than this one are too small to speak about key religious subgroups. A common “solution” is a variation on how the Times made weighting adjustments. Imagine, for example, that someone reasoned “since we already know (from other polls!) we want 300 people with X and Y characteristics in a target sample, but we only got 100 to respond, we’ll weigh each of the 100 three times more heavily than other people in the sample. Voila, fixed!” Correct me if I’m wrong, but this seems to triple the margin of error, at least for some of the results regarding X and Y.  To be clear, Burge wrote about the huge range of questions he has no leverage to answer due to small samples even before weighting adjustments. (Update: here's another example from Burge.) However, to say the least, this virtue is not always in evidence on religion polling fronts.  

I hate to quibble about this point when the deepest problems are baked into how pollsters frame questions. (Note, for example, how in the second Burge piece, how much of the action happens before what he primarily frets about; the entire discussion is fundamentally structured by the underlying question "are you an evangelical or a none?") Still, it’s a non-trivial part of the big picture. If the weighting tweaks by the Times increased the fishy smell you perceived in today’s case, a rule of thumb might be to double your wariness the next time it pitches you a religion poll, then double that again if it's about Christian nationalism.

In Conclusion

What shall we say about this— the original Times sample, the Times’s massaging of it, the talking points embedded in the headline, the staggering omissions (are these conscious? unconscious? which is worse?)—all before we get to the Liberal Patriot trying to build on one fragment?  

At best it’s pathetic. At worst it includes falling into a trap laid (consciously? unconsciously?) to reinforce right-wing hypotheses and/or talking points.   

As a learned habit of mind, it is degrading to trust polls like the one in focus today. Perhaps not in every case, but as a general proposition, using them to understand our world and to think about our political priorities is likely to do more harm than good. 

Treating this as the ground on which to fight for many of the policies, values, and/or clearer-sighted analyses that we might champion—for example, suppose what we really care about today is clawing back some of the outrageous wealth imbalances to fund a Green New Deal, moving toward Medicare for All, and understanding the actual balance of power between a hard core of Christofascists and various liberal-to-leftish religious folks—is like setting up for a battle on the ground chosen by the people we want to defeat. Failure is baked into this habit of mind. Trying to fix it with more polling could well be like spinning car wheels deeper into the mud.   

Circling back to where we started, I do think Medicare for All (at least smart roads in that general direction) and hammering on Republicans for slashing Medicaid (among other aspects of health care) are winning issues for Democrats. All kinds of arguments, including some based on polls, might strengthen this cause. 

Just never mind the habits of mind taught by garbage in garbage out polling. Set up your orienting analysis with arguments that go far beyond this.