1.S: Summary
( \newcommand{\kernel}{\mathrm{null}\,}\)
\( \newcommand\Dalpha
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[1], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Dbeta
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[2], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Dgamma
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[3], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Ddelta
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[4], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Depsilon
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[5], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Dvarepsilon
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[6], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Dzeta
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[7], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Deta
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[8], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Dtheta
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[9], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Dvartheta
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[10], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Diota
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[11], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Dkappa
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[12], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Dlambda
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[13], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Dvarpi
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[14], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\DGamma
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[15], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\DDelta
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[16], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\DTheta
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[17], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vmu
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[18], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vnu
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[19], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vxi
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[20], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vom
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[21], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vpi
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[22], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vvarpi
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[23], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vrho
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[24], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vvarrho
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[25], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vsigma
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[26], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vvarsigma
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[27], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vtau
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[28], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vupsilon
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[29], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vphi
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[30], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vvarphi
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[31], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vchi
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[32], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vpsi
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[33], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\Vomega
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[34], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\VGamma
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[35], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\VDelta
Callstack: at (Template:MathJaxArovas), /content/body/div/p[1]/span[36], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\newcommand\BI{\mib I}}
\)
\newcommand { M}
\newcommand { m}
}
\( \newcommand\tcb{\textcolor{blue}\)
\( \newcommand\tcr{\textcolor{red}\)
1$#1_$
\newcommand\SZ{\textsf Z}} \( \newcommand\kFd{k\ns_{\RF\dar}\)
\newcommand\mutB{\tilde\mu}\ns_\ssr{B}
\( \newcommand\xhihOZ
Callstack: at (Template:MathJaxArovas), /content/body/div/span[1], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
\( \newcommand\labar
Callstack: at (Template:MathJaxArovas), /content/body/div/span[2], line 1, column 1 at template() at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.S:_Summary), /content/body/p/span, line 1, column 23
References
- C. Gardiner, Stochastic Methods (4th edition, Springer-Verlag, 2010) Very clear and complete text on stochastic methods with many applications.
- J. M. Bernardo and A. F. M. Smith, Bayesian Theory (Wiley, 2000) A thorough textbook on Bayesian methods.
- D. Williams, Weighing the Odds: A Course in Probability and Statistics (Cambridge, 2001) A good overall statistics textbook, according to a mathematician colleague.
- E. T. Jaynes, Probability Theory (Cambridge, 2007) An extensive, descriptive, and highly opinionated presentation, with a strongly Bayesian approach.
- A. N. Kolmogorov, Foundations of the Theory of Probability (Chelsea, 1956) The Urtext of mathematical probability theory.
Summary
∙ Discrete distributions: Let n label the distinct possible outcomes of a discrete random process, and let p∗n be the probability for outcome n. Let A be a quantity which takes values which depend on n, with A∗n being the value of A under the outcome n. Then the expected value of A is ⟨A⟩=∑np∗nA∗n, where the sum is over all possible allowed values of n. We must have that the distribution is normalized, ⟨1⟩=∑np∗n=1.
∙ Continuous distributions: When the random variable φ takes a continuum of values, we define the probability density P(φ) to be such that P(φ)dμ is the probability for the outcome to lie within a differential volume dμ of φ, where dμ=W(φ)∏ni=1dφ∗i, were φ is an n-component vector in the configuration space Ω, and where the function W(φ) accounts for the possibility of different configuration space measures. Then if A(φ) is any function on Ω, the expected value of A is ⟨A⟩=∫ΩdμP(φ)A(φ).
∙ Central limit theorem: If {x∗1,…,x∗N} are each independently distributed according to P(x), then the distribution of the sum X=∑Ni=1x∗i is P∗N(X)=∞∫−∞dx∗1⋯∞∫−∞dx∗NP(x∗1)⋯P(x∗N)δ(X−N∑i=1x∗i)=∞∫−∞dk2π[ˆP(k)]NeikX , where ˆP(k)=∫dxP(x)e−ikx is the Fourier transform of P(x). Assuming that the lowest moments of P(x) exist, ln[ˆP(k)]=−iμk−12σ2k2+O(k3), where μ=⟨x⟩ and σ2=⟨x2⟩−⟨x⟩2 are the mean and standard deviation. Then for N→∞, P∗N(X)=(2πNσ2)−1/2e−(X−Nμ)2/2Nσ2 , which is a Gaussian with mean ⟨X⟩=Nμ and standard deviation √⟨X2⟩−⟨X⟩2=√Nσ. Thus, X is distributed as a Gaussian, even if P(x) is not a Gaussian itself.
∙ Entropy: The entropy of a statistical distribution is {p∗n} is S=−∑np∗nlnp∗n. (Sometimes the base 2 logarithm is used, in which case the entropy is measured in bits.) This has the interpretation of the information content per element of a random sequence.
∙ Distributions from maximum entropy: Given a distribution {p∗n} subject to (K+1) constraints of the form Xa=∑nXanp∗n with a∈{0,…,K}, where X0=X0n=1 (normalization), the distribution consistent with these constraints which maximizes the entropy function is obtained by extremizing the multivariable function S∗({p∗n},{λ∗a})=−∑np∗nlnp∗n−K∑a=0λ∗a(∑nXanp∗n−Xa) , with respect to the probabilities {p∗n} and the Lagrange multipliers {λ∗a}. This results in a Gibbs distribution, p∗n=1Zexp{−K∑a=1λ∗aXan} , where Z=e1+λ∗0 is determined by normalization, ∑np∗n=1 ( the a=0 constraint) and the K remaining multipliers determined by the K additional constraints.
∙ Multidimensional Gaussian integral: ∞∫−∞dx∗1⋯∞∫−∞dx∗nexp(−12x∗iA∗ijx∗j+b∗ix∗i)=((2π)ndetA)1/2exp(12b∗iA−1ijb∗j) .
∙ Bayes’ theorem: Let the conditional probability for B given A be P(B|A). Then Bayes’ theorem says P(A|B)=P(A)⋅P(B|A)/P(B). If the ’event space’ is partitioned as {A∗i}, then we have the extended form, P(A∗i|B)=P(B|A∗i)⋅P(A∗i)∑jP(B|A∗j)⋅P(A∗j) . When the event space is a ‘binary partition’ {A,¬A}, as is often the case in fields like epidemiology ( test positive or test negative), we have P(A|B)=P(B|A)⋅P(A)P(B|A)⋅P(A)+P(B|¬A)⋅P(¬A) . Note that P(A|B)+P(¬A|B)=1 (which follows from ¬¬A=A).
∙ Updating Bayesian priors: Given data in the form of observed values x={x∗1,…,x∗N}∈X and a hypothesis in the form of parameters θ={θ∗1,…,θ∗K}∈Θ, we write the conditional probability (density) for observing x given θ as f(x|θ). Bayes’ theorem says that the corresponding distribution π(θ|x) for θ conditioned on x is π(θ|x)=f(x|θ)π(θ)∫Θdθ′f(x|θ′)π(θ′) , We call π(θ) the prior for θ, f(x|θ) the likelihood of x given θ, and π(θ|x) the posterior for θ given x. We can use the posterior to find the distribution of new data points y, called the posterior predictive distribution, f(y|x)=∫Θdθf(y|θ)π(θ|x). This is the update of the prior predictive distribution, f(x)=∫Θdθf(x|θ)π(θ) . As an example, consider coin flipping with f(x|θ)=θX(1−θ)N−X, where N is the number of flips, and X=∑Nj=1x∗j with x∗j a discrete variable which is 0 for tails and 1 for heads. The parameter θ∈[0,1] is the probability to flip heads. We choose a prior π(θ)=θα−1(1−θ)β−1/B(α,β) where \SB(\alpha,\beta)=\RGamma(\alpha)\,\RGamma(\beta)/\RGamma(\alpha+\beta) is the Beta distribution. This results in a normalized prior \int\limits_0^1 \!d\theta\,\pi(\theta)=1. The posterior distribution for \theta is then \pi(\theta|x\ns_1,\ldots,x\ns_N)={f(x\ns_1,\ldots,x\ns_N|\theta)\,\pi(\theta)\over\int_0^1\!d\theta'\, f(x\ns_1,\ldots,x\ns_N|\theta')\,\pi(\theta')} ={\theta^{X+\alpha-1} (1-\theta)^{N-X+\beta-1}\over\SB(X+\alpha,N-X+\beta)}\ . The prior predictive is f(\Bx)=\int\limits_0^1\!d\theta f(\Bx|\theta)\,\pi(\theta)=\SB(X+\alpha,N-X+\beta)/\SB(\alpha,\beta) , and the posterior predictive for the total number of heads Y in M flips is f(\By|\Bx)=\!\!\int\limits_0^1\!d\theta\,f(\By|\theta)\,\pi(\theta|\Bx)= {\SB(X+Y+\alpha,N-X+M-Y+\beta)\over\SB(X+\alpha,N-X+\beta)}\quad.