The Long History of Bayesian and Frequentist

Bayesian

Bayesian is a statistical inference built upon the Bayes Theorem. Introduced by Reverend Thomas Bayes, it calculates the probability distribution of an outcome given a known previous outcome that happened under similar circumstances.

So if you tossed a coin 100 times sitting on a park bench in a cloudy day, you can only use that prior if you are sitting on a similar bench, and the sky is gray.

This method updates probabilities based on past knowledge, the prior, summarizing what is known about the outcome before any information is obtained. What makes this method unique is that the parameters are random variables with probability distribution, and the data is fixed.

How is a prior determined. Can someone just throw some data in the ring and hope for a coherent output?

Not exactly.

The prior is usually formulated by expert beliefs, historical data, or both.

Suppose we want to estimate the proportion of Georgia voters who will vote Republican in the 2028 presidential election. There is no need for an expert in this case. The conditions are fairly similar around every presidential election. Additionally, we have enough data to dismiss the opinion of an “expert.”

For a better part of two decades, the Peach State has been reliably red. But in the last two elections, it has shifted becoming a battleground. Using that knowledge, we are assuming that no candidate would win by a landslide. It is plausible to call it anywhere between 46% and 52%.

For good measure, let’s place the probability distribution centered around 49%. This becomes our prior. It summarizes what we believe the votes share before observing the actual results. In other words, we go into the analysis knowing that the result should hit around 49%.

Once the elections happen, it provides us with new data. Suppose the Republican candidate received 51.6%. This is much narrower than the original prior. In Bayesian analysis, we combined the prior with the observed evidence.

Frequentists loyalists have accused this method to introduce bias into the analysis. But there are some advantages to this approach. As Fornacon-Wood et al. (2022) cleverly states, inference provides direct probability to the hypothesis. Additionally, starting from somewhere is better than starting from nowhere (Schooth et at., 2014). The Bayesian method allows researchers to incorporate background knowledge into the analysis. A step into the right direction.

But here is the tricky part. What if the analysis is on a subject where determining a prior is more complicated? You would have to rely on experts where bias could very well be introduced. And if a bad prior is selected… well, your analysis will suffer for it.

Frequentist

The second school of thought and the most common among statisticians is frequentist. Unlike Bayesians, frequentist will a

nalyze from the ground up. Nothing in mind. It evaluates the likelihood of observed data under the hypothesis. It starts with the H0. It assumes nothing is happening.

One strength in this method is the objectivity. It provides a standardize way to assess the significance of results. Thus, the widespread acceptance in diverse fields. You don’t need an expert telling you what prior you should or shouldn’t consider.

For our Georgia presidential election problem, using this method, we would take a very different approach. We would begin with the estimating the proportion of GOP voters, rather than a prior knowledge. Frequentist will rely solely on the sample collected.

Let’s say that we have 2,000 replies poll survey in hand. We find that 50.7% intend to vote for the Republican candidate. The sample proportion, 50.7, becomes the point estimate of the unknown proportion of Republican voters in Georgia.

If we take a 95% confidence interval, the estimate Republican vote share between 48.5% and 52.9%. Under the frequentist interpretation, this doesn’t mean that there is a 95% probability that the true proportion is in between those numbers. It simply means that if we repeated similar polls and constructed confidence intervals using the same procedure, approximately 95% of those would contain the true proportion of GOP voters.

The one limitation, if I can call it that, is the p-value paradigm. In the scientific community, if the p-value doesn’t meet the 0.05 holy grail, the research is rejected or left unpublished. Except that the p-value doesn’t carry enough weight for an entire research to be thrown out because it didn’t hit the threshold.

As Johansen & Fjællingsdal (2026) explain in their paper the p-value is misused and misunderstood and, unfortunately, continues to dictate publish findings. This is a much deeper subject and it deserves a post of its own. I briefly wrote about this issue before.

What we should remember about this method is that it regards the population value as a fixed, unvarying quantity, without a probability distribution.

The Controversy

Some statisticians are hard core one method or the other. Bland and Altman (1998) claims it has to do with the university attended. It is my personal opinion that these are simply methods. And as such, they should be used when deemed appropriate.

A few years ago, Andrew Gelman, a professor of statistics and political science at Columbia University, wrote a controversial article claiming that Bayesians are frequentists. At first, I thought it was click bait. But it wasn’t.

It is a bold statement. To some, even fighting words.

Bayesians are not frequentists. They are fundamentally different principles with results that differ drastically depending on the prior. However, Gelman has a point; and I quote:

… any Bayesian interpretation of probability has to have some frequentist underpinning to be useful. And, conversely, any frequentist sample space corresponds to some set of problems to be averaged over.

In essence, both methods rely on long-run averages and repeated sampling concepts to evaluate how well their statistical procedures perform.