Inverse Probability

Conditional probability: $P(A\mid B)=\frac{P(A,B)}{P\left(B\right)}$

which yields $P(A,B)=P(A\mid B)P\left(B\right)$ (joint = conditional $\times$ marginal)

On the other side, $P(B\mid A)=\frac{P(B,A)}{P\left(A\right)}$

Bayes' Theorem

$$P(B\mid A)=\frac{P(A\mid B)P\left(B\right)}{P\left(A\right)}$$

Which says how can go from $P(A\mid B)$ to $P(B\mid A)$

Consider $B_i$ $(i=1,\dots ,n)$ as one of the many possible mutually exclusive events

$$\begin{array}{cc}P(B_i\mid A)& =\frac{P(A\mid B_i)P\left(B_i\right)}{P\left(A\right)}\\ & =\frac{P(A\mid B_i)P\left(B_i\right)}{{\sum }_{k=1}^{n}P(A\mid B_k)P\left(B_k\right)}\end{array}$$

Gigerenzer (2004)

$p$ value = $P$(data | hypothesis), not $P$(hypothesis | data)

• $H_0$: the person is sober (not drunk)
• data: breathalyzer result

$p$ = $P$(positive | sober) = 0.01 $\to$ reject $H_0$ at .05 level

However, as we have been, given that $P\left(H_0\right)$ is small, $P(H_0\mid \text{data})$ is still small

Bayes' Theorem in Data Analysis

• Bayesian statistics
  • more than applying Bayes's theorem
  • a way to quantify the plausibility of every possible value of some parameter $\theta$
    • E.g., population mean, regression coefficient, etc
  • Goal: update one's Belief about $\theta$ based on the observed data $D$

Example 2

shiny::runGitHub("plane_search", "marklhc")

• Try choosing different priors. How does your choice affect the posterior?
• Try adding more data. How does the number of data points affect the posterior?

Setting Priors

Prior beliefs used in data analysis must be admissible by a skeptical scientific audience (Kruschke, 2015, p. 115)

Likelihood/Model/Data $P(D\mid \theta ,M)$

Probability of observing the data as a function of the parameter(s)

Likelihood of Multiple Data Points

1. Given $D_1$, obtain posterior $P(\theta \mid D_1)$
2. Use $P(\theta \mid D_1)$ as prior, given $D_2$, obtain posterior $P(\theta \mid D_1,D_2)$

The posterior is the same as getting $D_2$ first then $D_1$, or $D_1$ and $D_2$ together, if

• data-order invariance is satisfied, which means
• $D_1$ and $D_2$ are exchangeable

Coin Flipping

Q: Estimate the probability that a coin gives a head

• $\theta$: parameter, probability of a head

Flip a coin, showing head

• $y=1$ for showing head

How do you obtain the likelihood?

Bernoulli Likelihood

The likelihood depends on the probability model chosen

• Some models are commonly used, for historical/computational/statistical reasons

One natural way is the Bernoulli model $$\begin{array}{cc}P(y=1\mid \theta )& =\theta \\ P(y=0\mid \theta )& =1-\theta \end{array}$$

The above requires separating $y=1$ and $y=0$. A more compact way is $$P(y\mid \theta )={\theta }^y(1-\theta {)}^{(1-y)}$$

Multiple Observations

Assume the flips are exchangeable given $\theta$,

$$\begin{array}{cc}P(y_1,\dots ,y_N\mid \theta )& =\prod _{i=1}^{N}P(y_i\mid \theta )\\ & ={\theta }^{{\sum }_{i=1}^N{y}_i}(1-\theta {)}^{{\sum }_{i=1}^N(1-y_i)}\\ & ={\theta }^z(1-\theta {)}^{N-z}\end{array}$$

$z$ = # of heads; $N$ = # of flips

Note: the likelihood only depends on the number of heads, not the particular sequence of observations

Posterior

Same posterior, two ways to think about it

Prior belief, weighted by the likelihood

$$P(\theta \mid y)\propto \underset{\text{weights}}{\underset{}{P(y\mid \theta )}}P\left(\theta \right)$$

Likelihood, weighted by the strength of prior belief

$$P(\theta \mid y)\propto \underset{\text{weights}}{\underset{}{P\left(\theta \right)}}P(\theta \mid y)$$

Posterior

Say $N$ = 4 and $z$ = 1

E.g., $P(\theta \mid y_1=1)\propto P(y_1=1\mid \theta )P\left(\theta \right)$

How About the Denominator?

Numerator: relative posterior plausibility of the $\theta$ values

We can avoid computing the denominator because

• The sum of the probabilities need to be 1

So, for discrete parameters:

• Posterior probabilitiy = relative plausibility / sum of relative plausibilities

However, the denominator is useful for computing the Bayes factor

Summarizing a Posterior Distribution

th <- seq(.05, .95, by = .10)
pth <- c(.01, .055, .10, .145, .19, .19, .145, .10, .055, .01)
py_th <- th^1 * (1 - th)^4
pth_y_unscaled <- pth * py_th
pth_y <- pth_y_unscaled / sum(pth_y_unscaled)
post_samples <- sample(th,
  size = 1000, replace = TRUE,
  prob = pth_y
)

>#   mean median     sd    mad   ci.1   ci.9 
>#  0.360  0.350  0.143  0.148  0.150  0.550

Prior Predictive Distribution

Bayesian models are generative

Simulate data from the prior distribution to check whether the data fit our intuition

• Clearly impossible values/patterns?

• Overly restrictive?

Criticism of "Subjectivity"

Main controversy: subjectivity in choosing a prior

• Two people with the same data can get different results because of different chosen priors

Counters to the Subjectivity Criticism

• With enough data, different priors hardly make a difference

• Prior: just a way to express the degree of ignorance

  • One can choose a weakly informative prior so that the Influence of subjective Belief is small

Counters to the Subjectivity Criticism 2

Subjectivity in choosing a prior is

• Same as in choosing a model, which is also done in frequentist statistics

• Relatively strong prior needs to be justified,

  • Open to critique from other researchers

• Inter-subjectivity $\to$ Objectivity

Counters to the Subjectivity Criticism 3

The prior is a way to incorporate previous research efforts to accumulate scientific evidence

Why should we ignore all previous literature every time we conduct a new study?