Being bound by rules…
and then smashing them
In Against Method, Paul Feyerabend writes:
one of the most striking features of recent discussions in the history and philosophy of science [he’s saying this in 1975] is the realization that events and developments, such as the invention of atomism in antiquity, the Copernican Revolution, the rise of modern atomism (kinetic theory; dispersion theory; stereochemistry; quantum theory), the gradual emergence of the wave theory of light, occurred only because some thinkers either decided not to be bound by certain ‘obvious’ methodological rules, or because they unwittingly broke them.
In the past, astute people decided to ignore certain ‘obvious’ rules to discover new things. Breaking rules, sometimes, is good. Feyerabend even discovered it before Gino! In that vein, I suppose Socrates (not believing in the gods of the city) or Diogenes (rejecting everything) also ‘broke’ rules.
Today, in quantitative social science, there’s a lingering feeling that we may need to break some established rules to uncover something fresh. Quantitative social science is a bit stale; stuff doesn’t always hold up.
Some say these breaks will take the form of interdisciplinary approaches or flexible methodologies, AI, open science, changing our metaphors. I’m not sure.
Better social science could mean less social science of the type we do today. It could mean less naturalistic social science of the type where “naturalists hold that there is no intrinsic obstacle for the social sciences to do the things that the natural sciences do, where there is no reason why social scientists should be unable to explain social phenomena, predict them, and intervene to change the course of events in ways that we find desirable”.
As such, I’m sympathetic to a project like this one; one that proposes an ‘anti-naturalist’1 framework. Interpretation must come first; where we strive to enrich our lives and the contexts in which we live by giving them meaning. We read Charles Taylor on the sources of the self, or novels, instead of fitting beta-binomial regression models to datasets about US presidents.
There is no way for us to philosophically affirm naturalism as an acceptable pluralism. This is because […] naturalism and interpretivism are fundamentally philosophically incompatible. Both cannot be affirmed without contradiction (p. 14).
But that’s incorrect, there’s one single web of belief, and our best course is to adopt the perspective of the most reliable knowledge available to us at any given time. That’s what I consider naturalism, and it’s not incompatible with interpretivism. This naturalism isn’t limited to the methods of natural science; it also includes interpretivism and hermeneutics. The authors continue:
our claim that the study of human behavior must properly reckon with meanings doesn’t entail that social scientists are prohibited from the use of statistics, regression analyses, and mass polling if these prove handy for their research. We agree, therefore, with Bent Flyvbjerg’s argument that determining which methods are appropriate to a research project depends on exercising judgment within context (Flyvbjerg follows Aristotle in dubbing this “phronesis”).
So, even the authors admit that nothing is off-limits. Maybe my research needs both regression analysis and interpreting the Bible. Who knows? The perspective isn’t compatible with Feyerabend’s ‘anything goes’. If ‘anything goes’ I can’t say “naturalism and interpretivism are fundamentally philosophically incompatible”. There are no such rules.2
Take this recent paper in the American Journal of Political Science: “When a Republican Senator invokes President Biden in a policy speech, Republican respondents increase approval of that Senator and oppose political compromise.” This effect is certainly noisy. You can invoke Biden for all sorts of reasons. The context (vignette experiments have no mundane realism) can vary widely. The accumulation of several instances of such divisive rhetoric collectively might contribute to a more toxic public sphere. Maybe not. Politicians increasingly adjust their rhetoric to criticize and mock their adversaries. The rise in partisan jibes is linked to a broader cultural shift, marking a move away from consensus-building. This trend isn’t exclusive to Senators; it’s a reflection of a wider trend.
Further, we get: “lawmakers reference the president more in the out-party, and when representing constituencies that increasingly share their partisanship.” These are perhaps interesting facts. But the cues invoked could instead reference Democratic Senators, they could reference the cultural differences between Republican and Democratic Senators. These are very arbitrary facts. It doesn’t constitute very generalizable science.
One basic definition says that data analysis is the practice of working with data to glean useful information, which can then be used to make informed decisions. In contrast, quantitative social science, like in this paper, involves analyzing data to extract information, though it often yields insights that are not particularly useful, and hence, aren’t used in decision-making.
In the current social science paradigm, this is a great research paper. There’s a causal estimate. It’s published in a prestigious journal. Essentially, it measures something not very important with great precision. It then says: here is a stylized fact, you may or may not act on it, but in truth, there isn’t much to do about it. In some obvious sense, it’s edutainment.
It’s safe to say that many researchers within this tradition, and many people outside it, are dissatisfied and looking for a rule to break.
In the words of Adam Mastroianni:
We’re hungry for something that makes us feel. A few decades from now, when a wizened Bloom asks his question again [what are the major discoveries in psychology? — a question that could apply to any social science], we hope for a world where people pile into the comments with major discoveries. Or, better yet, a world where Bloom doesn’t even have to ask in the first place, because the answer is so obvious.
It’s unlikely. In my web of belief, in what we call the social world, people value different things. What matters varies from one person to another. What we ought to say about speech in the social sphere will involve arithmetic means for some, references to Nietzsche for others. Some believe in tech, some are luddites. People will try to smash rules in different ways, and people will continue to disagree on whether that constitutes a successful smashing or a pathetic failed attempt. The causal mechanisms of the effects studies are all social and mental, they aren’t in the physical realm. So these effects, they work until they don’t work anymore (e.g., a Get out the vote campaign works until people get bored and used to it and it doesn’t work - it works on some but not all in a way we fundamentally don’t understand). The effects are noisy. Again, the causal mechanisms work on some people, not on others, in a way that we fundamentally don’t understand.
Footnotes
To be clear, I’m a naturalist. Like Quine, I think philosophy should be treated as a part of natural science, rejecting any distinction between philosophical inquiry and scientific investigation. There is no external standpoint from which to evaluate or justify scientific knowledge; rather, philosophy and science work together to refine our understanding of the world based on observation and empirical methods. When philosophy and science collide, science takes precedence. But it’s very rare they do. A lot of what we think are ‘collisions’ are just people making errors or acting in bad faith. While I’m sympathetic to their project in substance, I don’t think it’s ‘anti-naturalist’ in this sense. The naturalist web of belief is very inclusive.↩︎
Not everything is acceptable. I draw the line at fraud, fabricating data. The problem with fraud is that it pulls you away from the goal. Knowledge creation is a social and collective enterprise. Fraud makes the social communication collapse, that’s a serious problem. You communicate things which are literal lies; now everyone is confused. It takes you backwards in the project. There is, however, a utilitarian argument for fraud or malpractice that I find repulsive but not entirely implausible: in some cases, p-hacking or even fraud could theoretically bring positive outcomes. It could generate hype, attract funding, spur research, help us reach out goals. However, it depends on the extent of the fraud, it’s a fine line. Too much malpractice undermines trust and risks collapsing the whole system. the fraud or the malpractice also can’t inflict significant harm on individuals, like with FTX or Theranos. But the uncomfortable idea remains: limited malpractice, with no serious negative impacts beyond perhaps wasting money, might sometimes have beneficial effects if it fits into a larger, coherent framework. Gelman has an article called “Honesty and Transparency are not Enough’. Beyond honesty and transparency, he thinks people have to do things correctly too (i.e., have to follow rules). I’m not sure about that, perhaps my view is naive, requires too much of the public sphere. My view is that, in general, if people are honest and transparent, the marketplace of idea will sort things out, will eliminate garbage. I’m also pragmatist, so what’s true to some extent is what allows us to do what we want to do - except in some limited factual cases where the correspondence theory of truth makes sense.↩︎