Probabilities on Two Events

The Joint Probability is defined as the probability of two events happening together.

The Marginal Distribution is defined as follows:
- This can be extended for continuous variables by evaluating the following integral
- We may interpret the marginal distribution as aggregating all probabilities conditioned on .
The Conditional Probability is defined as the probability of given or
- Conditioning on another variable still gives the same definition as above. That is

Independence

Two random variables are independent if the joint probability can be written as That is, the two variables do not depend on each other.
Two random variables are conditionally independent given if the conditional joint can be written as a product of conditional marginals, or Alternatively, it can be stated as

The Chain Rule of Probability
Bayes’ Theorem is defined as
- is the prior
  - A prior is informative if it significantly affects the posterior. It is uninformative otherwise.
- is the likelihood
- is the posterior.
  - We say that the prior is a conjugate prior if it is of the same form as the posterior. This is done for mathematical convenience.
  - In many cases, it is desirable to have distributions that are amenable to being used as a conjugate prior (i.e., Gaussian, Beta, Inverse Wishart)
- is the probability of the evidence.
Boole’s Inequality or the Union Bound. Let be a countable set of events. We have that