Progress Studies: A Discipline is a Set of Institutional Norms

In a world with Progress Studies, academic departments and degree programs would not necessarily have to be reorganized. That’s probably going to be costly and time-consuming. Instead, a new focus on progress would be more comparable to a school of thought that would prompt a decentralized shift in priorities among academics, philanthropists, and funding agencies. Over time, we’d like to see communities, journals, and conferences devoted to these questions.

– Patrick Collison and Tyler Cowen, We Need a New Science of Progress

Contrary to the article’s title, what we have now is not a “Science of Progress”. It is at best a “Subculture of Progress”, but really, more like a subculture of demanding progress.

How do we bridge the gap?

When William James defined psychology as “the science of mental life”, he did not imagine the institution of academic psychology as it exists today. Psychology is much more than James’s broad charter. It is also an established standard of rigour (e.g. p < 0.05, placebo-controled RCTs), a canonical body of established knowledge founded on that standard, and living practitioners tasked with upholding truth and banishing heresy.

The science of psychology relies also on living institutions. It is a set of journals, grant-making organizations, academic departments and conferences, all with their associated level of prestige.

And then there is folk knowledge: Taboos born from historical failures never to be questioned. Social threads that mediate relationships between practitioners. The particular cultures and subcultures that span those threads. Foundational assumptions held sacred. [1]

As Tyler once described it: “You need barely scratch the surface in our prevailing ideologies to find central questions almost completely unaddressed.”

Unless Progress Studies gains acceptance from the existing institutions, it must strive to build new ones in its name. Otherwise, it risks never ascending to become a genuine “Science of Progress”.

Be careful however, not to cross into institutional role play. It would be too easy to replicate the trappings of “real sciences” without any of the associated benefits. In a quest for legitimacy, we must be careful to ask who we’re seeking it from, lest all power stem from the same corrupted source.

Cargo Cult Science, Serious Social Science

To avoid cargo cult science, you have to first understand the purpose of the thing you’re trying to replicate. It is not enough to build something that looks like an airplane from the outside if you haven’t understood the engine itself.

As Aaron Swartz writes in Serious Social Science:

The first thing that comes is the numbers. Real science papers are filled with tables and graphs and regressions on piles of data, so the social scientists decide to do all that.

…This is not to say that there is anything intrinsically wrong with using math or jargon or making grand claims. But to adopt these habits reflexively is to put the means before the ends. Scientists do not use math because it is complicated but because, for what they are doing, it is effective.

No one has yet attempted to artificially imbue Progress Studies with mathematical complexity, but there have been other attempts to be more like a real science. Remember Jasmine Wang’s attempt to compile a canon of knowledge in the early days of Progress Studies? In the year since, that canon has gone largely untouched by today’s practitioners. [2] Though it still serves as an interesting reference, very little of Wang’s canon is actually widely cited. [3]

I’m prone to my own prescriptive behavior. This whole series is an exercise in trying to explain what Progress Studies ought to be and how it should function. But I’ll admit, these sorts of top-down efforts are unlikely to have much impact.

Instead, the shortest path to becoming a legitimate science is to:

1. Publish good research
2. Make the case that it could not exist under an existing field
3. Label it Progress Studies

This is deceptively simple. In reality, the quality of research can only be judged within a particular institutional framework. We know work is good in other fields because it’s influential, highly cited and revered by the associated scientific community. Despite our fondness of the term “independent researcher”, no such thing has ever been possible. [6]

Minimum Viable Norms

If we don’t need prestigious conferences or journals, what is required?

At a minimum, norms must:

1. Enable substantive discourse
2. Which in turn progresses the field
3. Resulting in a coherent standard of quality
4. Allowing us to publish good research and label it Progress Studies

Where are we currently in this process?

Without established standards of rigour: authors can go back and forth criticizing each other’s work without making any progress.

Without foundational assumptions: it is too easy to dismiss an entire body of research on grounds it is not even attempting to assert or contest.

Without technical jargon: complex concepts must be rehashed each time, or worse, deployed with different definitions to suit the context. [7]

We already see a bit of this happening. My exchange [8] with Noah Smith is admirable in some sense (at least we are replying to each other), but regrettable in another. I did not really engage with his arguments, but merely attempted to criticize the cultural moment he chose to partake in. In response, he dodged my meta-level objection to optimism as an inconsistent interpretation of the data, and instead chose to double down on his object-level claims.

In other cases, I’ve seen outsiders alienated by the entire concept of Progress Studies as relying on the naive assumption that “progress” is a good worth pursuing. Rather than seen as challenging complacency, we’re accused perpetuating the status quo. Though Tyler Cowen’s Stubborn Attachments attempts to serve as this foundational definition, it is a frequently misunderstood book, still lacking in proper exegesis.

Finally, our jargon is simply not well established, nor are its operationalizations. Just as Effective Altruism settled on DALY, Progress Studies has attempted to coalesce around various measures of productivity and growth. Since Cowen dodged the question in Stubborn Attachments, writing instead of the nebulous Wealth Plus, we’re turned to specific metrics like GDP and Total Factor Productivity. Unfortunately, both are poorly understood, and don’t proxy well for the kind of progress we actually care about. [9]

We may discover further along that more is required, but these three are a good place to start. In the coming months, it will be up to us to propose, experiment with, and coalesce around better norms.

As the Swartz piece concludes:

[It’s] unlikely that the existing disciplines can be reformed. Instead what is needed is a culture of serious social science built outside the existing systems of academia… there is certainly much more to do, including building structures to do the work in.

Science advances one funeral at a time, but scientific institutions merely decay. While tenured professors eventually die, the institutions do not. Without natural senescence, an immortal being can be arbitrarily dysfunctional. [10] [11]

As absurd as it sounds, it is easier to construct a new field from scratch than to reform the existing ones. Without the entrenched interests and sacred institutions, we might actually stand a chance.

---

Footnotes

[1] See also David Chapman.

[2] For that matter, it’s not clear to me that there are actually Progress Studies researchers. Mostly, it seems to be a side project for people who’s real work is in building non-profits or working at think tanks. [4] [5]

[3] Perhaps you once skimmed Vannevar Bush’s Science: The Endless Frontier (or at least read Nintil’s article about it), but I don’t know anyone who claims to have actually understood Heidegger’s The Question Concerning Technology.

[4] Jason Crawford is at least full-time, though his grant from EA Funds describes it as “Telling the story of human progress to the world, and promote [sic] progress as a moral imperative” which sounds more like propaganda than research. That’s not a bad thing, we do need science educators! But first there must be a science.

[5] I occasionally get emails from people wowed by my blog’s prominence despite not having been around very long: “One does not just show up on the internet and write/think this well out of nowhere.” My answer is that very few other people are even trying! Scott Alexander is among the most popular authors, and has been working a demanding full-time job this entire time. As has Nintil, until he quit very recently. Leopold Aschenbrenner is not employed, but only because he’s a full time student.

[6] A more accurate title is perhaps “extra-institutional researcher” which I first heard here, but that’s a mouthful.

[7] The term “jargon” evokes esoteric slang. What I really mean is technical language, consistently defined and operationalized.

[8] Noah Smith wrote Techno-optimism for the 2020s, I responded with Isolated Demands for Rigour in New Optimism, which he has since replied to in a series of posts (1, 2). This gets messy after a while, but so long as everyone is linking back to the previous posts, it’s not too hard for a reader to follow along.

[9] Certainly, they do not proxy well for the progress I think we ought to care about.

[10] I believe some version of this is attributed to Peter Thiel, but can’t find the source.

[11] Harvard is the oldest institution of higher education in the US, and still ranked #1. Oxford is the “oldest university in the English-speaking world”, and depending on who you ask, shares that #1 ranking.

---

Appendix: Abolish Peer Review

Rather than attempt to identify a set of “minimum viable norms”, should we just assume all the trappings of academia are necessary? It’s not an ideal system, but that doesn’t mean you can just pick and choose which parts you want, and just hope the whole thing still holds together.

That’s a good argument, and I think agree we will have to do the work of explaining why some norms are not worth keeping.

In machine learning, arXiv has already eroded the importance of journals and conferences. It is still very important to get accepted to NeurIPS, but that very acceptance is contingent on having your pre-print widely cited.

In Progress Studies, many of the conventional trappings are being replaced as well, for better or for worse.

Instead of citations, we have retweets, and instead of journals, anyone can publish on their own blog. There is even a grant system! Though Emergent Ventures is central to Progress Studies, Jason Crawford is funded by a variety of sources, including “Open Philanthropy, the Long-Term Future Fund, and Jaan Tallinn”.

And instead of pre-publication peer-review, we have post-publication rebuttals.

Can post-hoc review even be called a legitimate standard of knowledge production?

Remember that while peer view has been around in some form since the 18th century, the term itself only took off around 1967:

As Scientific American writes:

Science and The Journal of the American Medical Association did not use outside reviewers until after 1940, "(Spier, 2002). The Lancet did not implement peer-review until 1976 (Benos et al., 2006). After the war and into the fifties and sixties, the specialization of articles increased and so did the competition for journal space.

I wasn’t sure about this claim, but as a sanity check, Wikipedia confirms:

The present-day peer-review system evolved from this 18th-century process,[11] began to involve external reviewers in the mid-19th-century,[12] and did not become commonplace until the mid-20th-century.

So it is not quite accurate to say there was no peer-review system before 1970, but it is worth understanding that our modern system is a relatively recent invention.

And yet, so much of the legendary science we now hail as transformative pre-dates 1970. Nintil’s Peer Rejection in Science summarizes breakthrough discoveries once considered crankery. How many of these would never see the light of day in today’s system? Or from Alexey Guzey’s Peer Review is a Disaster:

Peer reviewers in your field are your competitors, who have not themselves solved the problem you claim to be able to solve. They have both personal and professional interest (especially so if funding is limited) in giving low scores to grant applications of competing teams and to recommend rejection of their journal submissions. Further, since they’re experts in the grant application topic, while rejecting your paper or grant application, they can lift your research ideas and then pursue them themselves. This happens more frequently than you would expect.

This is not a niche view held merely among outsides. Richard Smith, former editor of the British Medical Journal once published the widely cited article Peer review: a flawed process at the heart of science and journals, where he writes:

Famously, it is compared with democracy: a system full of problems but the least worst we have.

…You can steal ideas and present them as your own, or produce an unjustly harsh review to block or at least slow down the publication of the ideas of a competitor. These have all happened.

Of course, any method will have false negatives and false positives. I’m not claiming Progress Studies’s current process of post-hoc review is obviously better, merely that is bad in a different way, and thus has the opportunity to produce knowledge that would not otherwise be possible.

We have to try something new, and while meta-science tries to come up with an improved mechanism, we might as well get started experimenting.