1.3: Entropy and Probability

\( \newcommand\bes{\begin{equation}\begin{split}}\)
\( \newcommand\ltwid{\propto}\)
\( \newcommand\ees{\end{split}\end{equation}}\)
\( \newcommand\mib{\mathbf}\)
\( \newcommand\Sa{\textsf a}\)
\( \newcommand\Sb{\textsf b}\)
\( \newcommand\Sc{\textsf c}\)
\( \newcommand\Sd{\textsf d}\)
\( \newcommand\Se{\textsf e}\)
\( \newcommand\Sf{\textsf f}\)
\( \newcommand\Sg{\textsf g}\)
\( \newcommand\Sh{\textsf h}\)
\( \newcommand\Si{\textsf i}\)
\( \newcommand\Sj{\textsf j}\)
\( \newcommand\Sk{\textsf k}\)
\( \newcommand\Sl{\textsf l}\)
\( \newcommand\Sm{\textsf m}\)
\( \newcommand\Sn{\textsf n}\)
\( \newcommand\So{\textsf o}\)
\( \newcommand\Sp{\textsf p}\)
\( \newcommand\Sq{\textsf q}\)
\( \newcommand\Sr{\textsf r}\)
\( \newcommand\Ss{\textsf s}\)
\( \newcommand\St{\textsf t}\)
\( \newcommand\Su{\textsf u}\)
\( \newcommand\Sv{\textsf v}\)
\( \newcommand\Sw{\textsf w}\)
\( \newcommand\Sx{\textsf x}\)
\( \newcommand\Sy{\textsf y}\)
\( \newcommand\Sz{\textsf z}\)
\( \newcommand\SA{\textsf A}\)
\( \newcommand\SB{\textsf B}\)
\( \newcommand\SC{\textsf C}\)
\( \newcommand\SD{\textsf D}\)
\( \newcommand\SE{\textsf E}\)
\( \newcommand\SF{\textsf F}\)
\( \newcommand\SG{\textsf G}\)
\( \newcommand\SH{\textsf H}\)
\( \newcommand\SI{\textsf I}\)
\( \newcommand\SJ{\textsf J}\)
\( \newcommand\SK{\textsf K}\)
\( \newcommand\SL{\textsf L}\)
\( \newcommand\SM{\textsf M}\)
\( \newcommand\SN{\textsf N}\)
\( \newcommand\SO{\textsf O}\)
\( \newcommand\SP{\textsf P}\)
\( \newcommand\SQ{\textsf Q}\)
\( \newcommand\SR{\textsf R}\)
\( \newcommand\SS{\textsf S}\)
\( \newcommand\ST{\textsf T}\)
\( \newcommand\SU{\textsf U}\)
\( \newcommand\SV{\textsf V}\)
\( \newcommand\SW{\textsf W}\)
\( \newcommand\SX{\textsf X}\)
\( \newcommand\SY{\textsf Y}\)
\( \newcommand\SZ{\textsf Z}\)
\( \newcommand\Ha{\hat a}\)
\( \newcommand\Hb{\hat b}\)
\( \newcommand\Hc{\hat c}\)
\( \newcommand\Hd{\hat d}\)
\( \newcommand\He{\hat e}\)
\( \newcommand\Hf{\hat f}\)
\( \newcommand\Hg{\hat g}\)
\( \newcommand\Hh{\hat h}\)
\( \newcommand\Hi{\hat \imath}\)
\( \newcommand\Hj{\hat \jmath}\)
\( \newcommand\Hk{\hat k}\)
\( \newcommand\Hl{\hat l}\)
\( \newcommand\Hm{\hat m}\)
\( \newcommand\Hn{\hat n}\)
\( \newcommand\Ho{\hat o}\)
\( \newcommand\Hp{\hat p}\)
\( \newcommand\Hq{\hat q}\)
\( \newcommand\Hr{\hat r}\)
\( \newcommand\Hs{\hat s}\)
\( \newcommand\Ht{\hat t}\)
\( \newcommand\Hu{\hat u}\)
\( \newcommand\Hv{\hat v}\)
\( \newcommand\Hw{\hat w}\)
\( \newcommand\Hx{\hat x}\)
\( \newcommand\Hy{\hat y}\)
\( \newcommand\Hz{\hat z}\)
\( \newcommand\HA{\hat A}\)
\( \newcommand\HB{\hat B}\)
\( \newcommand\HC{\hat C}\)
\( \newcommand\HD{\hat D}\)
\( \newcommand\HE{\hat E}\)
\( \newcommand\HF{\hat F}\)
\( \newcommand\HG{\hat G}\)
\( \newcommand\HH{\hat H}\)
\( \newcommand\HI{\hat I}\)
\( \newcommand\HJ{\hat J}\)
\( \newcommand\HK{\hat K}\)
\( \newcommand\HL{\hat L}\)
\( \newcommand\HM{\hat M}\)
\( \newcommand\HN{\hat N}\)
\( \newcommand\HO{\hat O}\)
\( \newcommand\HP{\hat P}\)
\( \newcommand\HQ{\hat Q}\)
\( \newcommand\HR{\hat R}\)
\( \newcommand\HS{\hat S}\)
\( \newcommand\HT{\hat T}\)
\( \newcommand\HU{\hat U}\)
\( \newcommand\HV{\hat V}\)
\( \newcommand\HW{\hat W}\)
\( \newcommand\HX{\hat X}\)
\( \newcommand\HY{\hat Y}\)
\( \newcommand\HZ{\hat Z}\)
\( \newcommand\Halpha{\hat\alpha}\)
\( \newcommand\Hbeta{\hat\beta}\)
\( \newcommand\Hgamma{\hat\gamma}\)
\( \newcommand\Hdelta{\hat\delta}\)
\( \newcommand\Hepsilon{\hat\epsilon}\)
\( \newcommand\Hvarepsilon{\hat\varepsilon}\)
\( \newcommand\Hzeta{\hat\zeta}\)
\( \newcommand\Heta{\hat\eta}\)
\( \newcommand\Htheta{\hat\theta}\)
\( \newcommand\Hvartheta{\hat\vartheta}\)
\( \newcommand\Hiota{\hat\iota}\)
\( \newcommand\Hkappa{\hat\kappa}\)
\( \newcommand\Hlambda{\hat\lambda}\)
\( \newcommand\Hmu{\hat\mu}\)
\( \newcommand\Hnu{\hat\nu}\)
\( \newcommand\Hxi{\hat\xi}\)
\( \newcommand\Hom{\hat\omicron}\)
\( \newcommand\Hpi{\hat\pi}\)
\( \newcommand\Hvarpi{\hat\varpi}\)
\( \newcommand\Hrho{\hat\rho}\)
\( \newcommand\Hvarrho{\hat\varrho}\)
\( \newcommand\Hsigma{\hat\sigma}\)
\( \newcommand\Hvarsigma{\hat\varsigma}\)
\( \newcommand\Htau{\var\tau}\)
\( \newcommand\Hupsilon{\hat\upsilon}\)
\( \newcommand\Hphi{\hat\phi}\)
\( \newcommand\Hvarphi{\hat\varphi}\)
\( \newcommand\Hchi{\hat\chi}\)
\( \newcommand\Hxhi{\hat\xhi}\)
\( \newcommand\Hpsi{\hat\psi}\)
\( \newcommand\Homega{\hat\omega}\)
\( \newcommand\HGamma{\hat\Gamma}\)
\( \newcommand\HDelta{\hat\Delta}\)
\( \newcommand\HTheta{\hat\Theta}\)
\( \newcommand\HLambda{\hat\Lambda}\)
\( \newcommand\HXi{\hat\Xi}\)
\( \newcommand\HPi{\hat\Pi}\)
\( \newcommand\HSigma{\hat\Sigma}\)
\( \newcommand\HUps{\hat\Upsilon}\)
\( \newcommand\HPhi{\hat\Phi}\)
\( \newcommand\HPsi{\hat\Psi}\)
\( \newcommand\HOmega{\hat\Omega}\)
\( \newcommand\xhat{\hat\Bx}\)
\( \newcommand\yhat{\hat\By}\)
\( \newcommand\zhat{\hat\Bz}\)
\( \newcommand\ehat{\hat\Be}\)
\( \newcommand\khat{\hat\Bk}\)
\( \newcommand\nhat{\hat\Bn}\)
\( \newcommand\rhat{\hat\Br}\)
\( \newcommand\phihat{\hat\Bphi}\)
\( \newcommand\thetahat{\hat\Btheta}\)
\( \newcommand\MA{\mathbb A}\)
\( \newcommand\MB{\mathbb B}\)
\( \newcommand\MC{\mathbb C}\)
\( \newcommand\MD{\mathbb D}\)
\( \newcommand\ME{\mathbb E}\)
\( \newcommand\MF{\mathbb F}\)
\( \newcommand\MG{\mathbb G}\)
\( \newcommand\MH{\mathbb H}\)
\( \newcommand\MI{\mathbb I}\)
\( \newcommand\MJ{\mathbb J}\)
\( \newcommand\MK{\mathbb K}\)
\( \newcommand\ML{\mathbb L}\)
\( \newcommand\MM{\mathbb M}\)
\( \newcommand\MN{\mathbb N}\)
\( \newcommand\MO{\mathbb O}\)
\( \newcommand\MP{\mathbb P}\)
\( \newcommand\MQ{\mathbb Q}\)
\( \newcommand\MR{\mathbb R}\)
\( \newcommand\MS{\mathbb S}\)
\( \newcommand\MT{\mathbb T}\)
\( \newcommand\MU{\mathbb U}\)
\( \newcommand\MV{\mathbb V}\)
\( \newcommand\MW{\mathbb W}\)
\( \newcommand\MX{\mathbb X}\)
\( \newcommand\MY{\mathbb Y}\)
\( \newcommand\MZ{\mathbb Z}\)
\( \newcommand\CA{\mathcal A}\)
\( \newcommand\CB{\mathcal B}\)
\( \newcommand\CC{\mathcal C}\)
\( \newcommand\CD{\mathcal D}\)
\( \newcommand\CE{\mathcal E}\)
\( \newcommand\CF{\mathcal F}\)
\( \newcommand\CG{\mathcal G}\)
\( \newcommand\CH{\mathcal H}\)
\( \newcommand\CI{\mathcal I}\)
\( \newcommand\CJ{\mathcal J}\)
\( \newcommand\CK{\mathcal K}\)
\( \newcommand\CL{\mathcal L}\)
\( \newcommand\CM{\mathcal M}\)
\( \newcommand\CN{\mathcal N}\)
\( \newcommand\CO{\mathcal O}\)
\( \newcommand\CP{\mathcal P}\)
\( \newcommand\CQ{\mathcal Q}\)
\( \newcommand\CR{\mathcal R}\)
\( \newcommand\CS{\mathcal S}\)
\( \newcommand\CT{\mathcal T}\)
\( \newcommand\CU{\mathcal U}\)
\( \newcommand\CV{\mathcal V}\)
\( \newcommand\CW{\mathcal W}\)
\( \newcommand\CX{\mathcal X}\)
\( \newcommand\CY{\mathcal Y}\)
\( \newcommand\CZ{\mathcal Z}\)
\( \newcommand\Fa{\mathfrak a}\)
\( \newcommand\Fb{\mathfrak b}\)
\( \newcommand\Fc{\mathfrak c}\)
\( \newcommand\Fd{\mathfrak d}\)
\( \newcommand\Fe{\mathfrak e}\)
\( \newcommand\Ff{\mathfrak f}\)
\( \newcommand\Fg{\mathfrak g}\)
\( \newcommand\Fh{\mathfrak h}\)
\( \newcommand\Fi{\mathfrak i}\)
\( \newcommand\Fj{\mathfrak j}\)
\( \newcommand\Fk{\mathfrak k}\)
\( \newcommand\Fl{\mathfrak l}\)
\( \newcommand\Fm{\mathfrak m}\)
\( \newcommand\Fn{\mathfrak n}\)
\( \newcommand\Fo{\mathfrak o}\)
\( \newcommand\Fp{\mathfrak p}\)
\( \newcommand\Fq{\mathfrak q}\)
\( \newcommand\Fr{\mathfrak r}\)
\( \newcommand\Fs{\mathfrak s}\)
\( \newcommand\Ft{\mathfrak t}\)
\( \newcommand\Fu{\mathfrak u}\)
\( \newcommand\Fv{\mathfrak v}\)
\( \newcommand\Fw{\mathfrak w}\)
\( \newcommand\Fx{\mathfrak x}\)
\( \newcommand\Fy{\mathfrak y}\)
\( \newcommand\Fz{\mathfrak z}\)
\( \newcommand\FA{\mathfrak A}\)
\( \newcommand\FB{\mathfrak B}\)
\( \newcommand\FC{\mathfrak C}\)
\( \newcommand\FD{\mathfrak D}\)
\( \newcommand\FE{\mathfrak E}\)
\( \newcommand\FF{\mathfrak F}\)
\( \newcommand\FG{\mathfrak G}\)
\( \newcommand\FH{\mathfrak H}\)
\( \newcommand\FI{\mathfrak I}\)
\( \newcommand\FJ{\mathfrak J}\)
\( \newcommand\FK{\mathfrak K}\)
\( \newcommand\FL{\mathfrak L}\)
\( \newcommand\FM{\mathfrak M}\)
\( \newcommand\FN{\mathfrak N}\)
\( \newcommand\FO{\mathfrak O}\)
\( \newcommand\FP{\mathfrak P}\)
\( \newcommand\FQ{\mathfrak Q}\)
\( \newcommand\FR{\mathfrak R}\)
\( \newcommand\FS{\mathfrak S}\)
\( \newcommand\FT{\mathfrak T}\)
\( \newcommand\FU{\mathfrak U}\)
\( \newcommand\FV{\mathfrak V}\)
\( \newcommand\FW{\mathfrak W}\)
\( \newcommand\FX{\mathfrak X}\)
\( \newcommand\FY{\mathfrak Y}\)
\( \newcommand\FZ{\mathfrak Z}\)
\( \newcommand\Da{\dot a}\)
\( \newcommand\Db{\dot b}\)
\( \newcommand\Dc{\dot c}\)
\( \newcommand\Dd{\dot d}\)
\( \newcommand\De{\dot e}\)
\( \newcommand\Df{\dot f}\)
\( \newcommand\Dg{\dot g}\)
\( \newcommand\Dh{\dot h}\)
\( \newcommand\Di{\dot \imath}\)
\( \newcommand\Dj{\dot \jmath}\)
\( \newcommand\Dk{\dot k}\)
\( \newcommand\Dl{\dot l}\)
\( \newcommand\Dm{\dot m}\)
\( \newcommand\Dn{\dot n}\)
\( \newcommand\Do{\dot o}\)
\( \newcommand\Dp{\dot p}\)
\( \newcommand\Dq{\dot q}\)
\( \newcommand\Dr{\dot r}\)
\( \newcommand\Ds{\dot s}\)
\( \newcommand\Dt{\dot t}\)
\( \newcommand\Du{\dot u}\)
\( \newcommand\Dv{\dot v}\)
\( \newcommand\Dw{\dot w}\)
\( \newcommand\Dx{\dot x}\)
\( \newcommand\Dy{\dot y}\)
\( \newcommand\Dz{\dot z}\)
\( \newcommand\DA{\dot A}\)
\( \newcommand\DB{\dot B}\)
\( \newcommand\DC{\dot C}\)
\( \newcommand\DD{\dot D}\)
\( \newcommand\DE{\dot E}\)
\( \newcommand\DF{\dot F}\)
\( \newcommand\DG{\dot G}\)
\( \newcommand\DH{\dot H}\)
\( \newcommand\DI{\dot I}\)
\( \newcommand\DJ{\dot J}\)
\( \newcommand\DK{\dot K}\)
\( \newcommand\DL{\dot L}\)
\( \newcommand\DM{\dot M}\)
\( \newcommand\DN{\dot N}\)
\( \newcommand\DO{\dot O}\)
\( \newcommand\DP{\dot P}\)
\( \newcommand\DQ{\dot Q}\)
\( \newcommand\DR{\dot R}\)
\( \newcommand\DS{\dot S}\)
\( \newcommand\DT{\dot T}\)
\( \newcommand\DU{\dot U}\)
\( \newcommand\DV{\dot V}\)
\( \newcommand\DW{\dot W}\)
\( \newcommand\DX{\dot X}\)
\( \newcommand\DY{\dot Y}\)
\( \newcommand\DZ{\dot Z}\)
\( \newcommand\Dalpha

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[1], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Dbeta

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[2], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Dgamma

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[3], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Ddelta

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[4], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Depsilon

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[5], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Dvarepsilon

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[6], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Dzeta

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[7], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Deta

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[8], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Dtheta

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[9], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Dvartheta

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[10], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Diota

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[11], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Dkappa

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[12], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Dlambda

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[13], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Dmu{\dot\mu}\)
\( \newcommand\Dnu{\dot\nu}\)
\( \newcommand\Dxi{\dot\xi}\)
\( \newcommand\Dom{\dot\omicron}\)
\( \newcommand\Dpi{\dot\pi}\)
\( \newcommand\Dvarpi

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[14], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Drho{\dot\rho}\)
\( \newcommand\Dvarrho{\dot\varrho}\)
\( \newcommand\Dsigma{\dot\sigma}\)
\( \newcommand\Dvarsigma{\dot\varsigma}\)
\( \newcommand\Dtau{\var\tau}\)
\( \newcommand\Dupsilon{\dot\upsilon}\)
\( \newcommand\Dphi{\dot\phi}\)
\( \newcommand\Dvarphi{\dot\varphi}\)
\( \newcommand\Dchi{\dot\chi}\)
\( \newcommand\Dpsi{\dot\psi}\)
\( \newcommand\Domega{\dot\omega}\)
\( \newcommand\DGamma

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[15], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\DDelta

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[16], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\DTheta

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[17], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\DLambda{\dot\Lambda}\)
\( \newcommand\DXi{\dot\Xi}\)
\( \newcommand\DPi{\dot\Pi}\)
\( \newcommand\DSigma{\dot\Sigma}\)
\( \newcommand\DUps{\dot\Upsilon}\)
\( \newcommand\DPhi{\dot\Phi}\)
\( \newcommand\DPsi{\dot\Psi}\)
\( \newcommand\DOmega{\dot\Omega}\)
\( \newcommand\Va{\vec a}\)
\( \newcommand\Vb{\vec b}\)
\( \newcommand\Vc{\vec c}\)
\( \newcommand\Vd{\vec d}\)
\( \newcommand\Ve{\vec e}\)
\( \newcommand\Vf{\vec f}\)
\( \newcommand\Vg{\vec g}\)
\( \newcommand\Vh{\vec h}\)
\( \newcommand\Vi{\vec \imath}\)
\( \newcommand\Vj{\vec \jmath}\)
\( \newcommand\Vk{\vec k}\)
\( \newcommand\Vl{\vec l}\)
\( \newcommand\Vm{\vec m}\)
\( \newcommand\Vn{\vec n}\)
\( \newcommand\Vo{\vec o}\)
\( \newcommand\Vp{\vec p}\)
\( \newcommand\Vq{\vec q}\)
\( \newcommand\Vr{\vec r}\)
\( \newcommand\Vs{\vec s}\)
\( \newcommand\Vt{\vec t}\)
\( \newcommand\Vu{\vec u}\)
\( \newcommand\Vv{\vec v}\)
\( \newcommand\Vw{\vec w}\)
\( \newcommand\Vx{\vec x}\)
\( \newcommand\Vy{\vec y}\)
\( \newcommand\Vz{\vec z}\)
\( \newcommand\VA{\vec A}\)
\( \newcommand\VB{\vec B}\)
\( \newcommand\VC{\vec C}\)
\( \newcommand\VD{\vec D}\)
\( \newcommand\VE{\vec E}\)
\( \newcommand\VF{\vec F}\)
\( \newcommand\VG{\vec G}\)
\( \newcommand\VH{\vec H}\)
\( \newcommand\VI{\vec I}\)
\( \newcommand\VJ{\vec J}\)
\( \newcommand\VK{\vec K}\)
\( \newcommand\VL{\vec L}\)
\( \newcommand\VM{\vec M}\)
\( \newcommand\VN{\vec N}\)
\( \newcommand\VO{\vec O}\)
\( \newcommand\VP{\vec P}\)
\( \newcommand\VQ{\vec Q}\)
\( \newcommand\VR{\vec R}\)
\( \newcommand\VS{\vec S}\)
\( \newcommand\VT{\vec T}\)
\( \newcommand\VU{\vec U}\)
\( \newcommand\VV{\vec V}\)
\( \newcommand\VW{\vec W}\)
\( \newcommand\VX{\vec X}\)
\( \newcommand\VY{\vec Y}\)
\( \newcommand\VZ{\vec Z}\)
\( \newcommand\Valpha{\vec\alpha}\)
\( \newcommand\Vbeta{\vec\beta}\)
\( \newcommand\Vgamma{\vec\gamma}\)
\( \newcommand\Vdelta{\vec\delta}\)
\( \newcommand\Vepsilon{\vec\epsilon}\)
\( \newcommand\Vvarepsilon{\vec\varepsilon}\)
\( \newcommand\Vzeta{\vec\zeta}\)
\( \newcommand\Veta{\vec\eta}\)
\( \newcommand\Vtheta{\vec\theta}\)
\( \newcommand\Vvartheta{\vec\vartheta}\)
\( \newcommand\Viota{\vec\iota}\)
\( \newcommand\Vkappa{\vec\kappa}\)
\( \newcommand\Vlambda{\vec\lambda}\)
\( \newcommand\Vmu

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[18], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vnu

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[19], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vxi

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[20], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vom

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[21], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vpi

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[22], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vvarpi

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[23], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vrho

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[24], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vvarrho

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[25], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vsigma

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[26], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vvarsigma

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[27], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vtau

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[28], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vupsilon

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[29], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vphi

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[30], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vvarphi

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[31], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vchi

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[32], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vpsi

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[33], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\Vomega

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[34], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\VGamma

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[35], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\VDelta

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/p[1]/span[36], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\VTheta{\vec\Theta}\)
\( \newcommand\VLambda{\vec\Lambda}\)
\( \newcommand\VXi{\vec\Xi}\)
\( \newcommand\VPi{\vec\Pi}\)
\( \newcommand\VSigma{\vec\Sigma}\)
\( \newcommand\VUps{\vec\Upsilon}\)
\( \newcommand\VPhi{\vec\Phi}\)
\( \newcommand\VPsi{\vec\Psi}\)
\( \newcommand\VOmega{\vec\Omega}\)
\( \newcommand\BA{\mib A}\)
\( \newcommand\BB{\mib B}\)
\( \newcommand\BC{\mib C}\)
\( \newcommand\BD{\mib D}\)
\( \newcommand\BE{\mib E}\)
\( \newcommand\BF{\mib F}\)
\( \newcommand\BG{\mib G}\)
\( \newcommand\BH{\mib H}\)
\( \newcommand\BI{\mib I}}\)
\( \newcommand\BJ{\mib J}\)
\( \newcommand\BK{\mib K}\)
\( \newcommand\BL{\mib L}\)
\( \newcommand\BM{\mib M}\)
\( \newcommand\BN{\mib N}\)
\( \newcommand\BO{\mib O}\)
\( \newcommand\BP{\mib P}\)
\( \newcommand\BQ{\mib Q}\)
\( \newcommand\BR{\mib R}\)
\( \newcommand\BS{\mib S}\)
\( \newcommand\BT{\mib T}\)
\( \newcommand\BU{\mib U}\)
\( \newcommand\BV{\mib V}\)
\( \newcommand\BW{\mib W}\)
\( \newcommand\BX{\mib X}\)
\( \newcommand\BY{\mib Y}\)
\( \newcommand\BZ{\mib Z}\)
\( \newcommand\Ba{\mib a}\)
\( \newcommand\Bb{\mib b}\)
\( \newcommand\Bc{\mib c}\)
\( \newcommand\Bd{\mib d}\)
\( \newcommand\Be{\mib e}\)
\( \newcommand\Bf{\mib f}\)
\( \newcommand\Bg{\mib g}\)
\( \newcommand\Bh{\mib h}\)
\( \newcommand\Bi{\mib i}\)
\( \newcommand\Bj{\mib j}\)
\( \newcommand\Bk{\mib k}\)
\( \newcommand\Bl{\mib l}\)
\( \newcommand\Bm{\mib m}\)
\( \newcommand\Bn{\mib n}\)
\( \newcommand\Bo{\mib o}\)
\( \newcommand\Bp{\mib p}\)
\( \newcommand\Bq{\mib q}\)
\( \newcommand\Br{\mib r}\)
\( \newcommand\Bs{\mib s}\)
\( \newcommand\Bt{\mib t}\)
\( \newcommand\Bu{\mib u}\)
\( \newcommand\Bv{\mib v}\)
\( \newcommand\Bw{\mib w}\)
\( \newcommand\Bx{\mib x}\)
\( \newcommand\By{\mib y}\)
\( \newcommand\Bz{\mib z}\)\)
\( \newcommand\vrh{\varrho}\)
\( \newcommand\vsig{\varsigma}\)
\( \newcommand\ups{\upsilon}\)
\( \newcommand\eps{\epsilon}\)
\( \newcommand\ve{\varepsilon}\)
\( \newcommand\vth{\vartheta}\)
\( \newcommand\vphi{\varphi}\)
\( \newcommand\xhi{\chi}\)
\( \newcommand\Ups{\Upsilon}\)
\( \newcommand\Balpha{\mib\alpha}\)
\( \newcommand\Bbeta{\mib\beta}\)
\( \newcommand\Bgamma{\mib\gamma}\)
\( \newcommand\Bdelta{\mib\delta}\)
\( \newcommand\Beps{\mib\epsilon}\)
\( \newcommand\Bve{\mib\varepsilon}\)
\( \newcommand\Bzeta{\mib\zeta}\)
\( \newcommand\Beta{\mib\eta}\)
\( \newcommand\Btheta{\mib\theta}\)
\( \newcommand\Bvth{\mib\vartheta}\)
\( \newcommand\Biota{\mib\iota}\)
\( \newcommand\Bkappa{\mib\kappa}\)
\( \newcommand\Blambda{\mib\lambda}\)
\( \newcommand\Bmu{\mib\mu}\)
\( \newcommand\Bnu{\mib\nu}\)
\( \newcommand\Bxi{\mib\xi}\)
\( \newcommand\Bom{\mib\omicron}\)
\( \newcommand\Bpi{\mib\pi}\)
\( \newcommand\Bvarpi{\mib\varpi}\)
\( \newcommand\Brho{\mib\rho}\)
\( \newcommand\Bvrh{\mib\varrho}\)
\( \newcommand\Bsigma{\mib\sigma}\)
\( \newcommand\Bvsig{\mib\varsigma}\)
\( \newcommand\Btau{\mib\tau}\)
\( \newcommand\Bups{\mib\upsilon}\)
\( \newcommand\Bphi{\mib\phi}\)
\( \newcommand\Bvphi{\mib\vphi}\)
\( \newcommand\Bchi{\mib\chi}\)
\( \newcommand\Bpsi{\mib\psi}\)
\( \newcommand\Bomega{\mib\omega}\)
\( \newcommand\BGamma{\mib\Gamma}\)
\( \newcommand\BDelta{\mib\Delta}\)
\( \newcommand\BTheta{\mib\Theta}\)
\( \newcommand\BLambda{\mib\Lambda}\)
\( \newcommand\BXi{\mib\Xi}\)
\( \newcommand\BPi{\mib\Pi}\)
\( \newcommand\BSigma{\mib\Sigma}\)
\( \newcommand\BUps{\mib\Upsilon}\)
\( \newcommand\BPhi{\mib\Phi}\)
\( \newcommand\BPsi{\mib\Psi}\)
\( \newcommand\BOmega{\mib\Omega}\)
\( \newcommand\Bxhi{\raise.35ex\hbox{$\Bchi$}}\)
\( \newcommand\RGamma{ \Gamma}\)
\( \newcommand\RDelta{ \Delta}\)
\( \newcommand\RTheta{ \Theta}\)
\( \newcommand\RLambda{ \Lambda}\)
\( \newcommand\RXi{ \Xi}\)
\( \newcommand\RPi{ \Pi}\)
\( \newcommand\RSigma{ \Sigma}\)
\( \newcommand\RUps{ \Upsilon}\)
\( \newcommand\RPhi{ \Phi}\)
\( \newcommand\RPsi{ \Psi}\)
\( \newcommand\ROmega{ \Omega}\)
\( \newcommand\RA{ A}\)
\( \newcommand\RB{ B}\)
\( \newcommand\RC{ C}\)
\( \newcommand\RD{ D}\)
\( \newcommand\RE{ E}\)
\( \newcommand\RF{ F}\)
\( \newcommand\RG{ G}\)
\( \newcommand\RH{ H}\)
\( \newcommand\RI{ I}\)
\( \newcommand\RJ{ J}\)
\( \newcommand\RK{ K}\)
\( \newcommand\RL{ L}\)
\( \newcommand { M}\)
\( \newcommand\RN{ N}\)
\( \newcommand\RO{ O}\)
\( \newcommand\RP{ P}\)
\( \newcommand\RQ{ Q}\)
\( \newcommand\RR{ R}\)
\( \newcommand\RS{ S}\)
\( \newcommand\RT{ T}\)
\( \newcommand\RU{ U}\)
\( \newcommand\RV{ V}\)
\( \newcommand\RW{ W}\)
\( \newcommand\RX{ X}\)
\( \newcommand\RY{ Y}\)
\( \newcommand\RZ{ Z}\)
\( \newcommand\Ra{ a}\)
\( \newcommand\Rb{ b}\)
\( \newcommand\Rc{ c}\)
\( \newcommand\Rd{ d}\)
\( \newcommand\Re{ e}\)
\( \newcommand\Rf{ f}\)
\( \newcommand\Rg{ g}\)
\( \newcommand\Rh{ h}\)
\( \newcommand\Ri{ i}\)
\( \newcommand\Rj{ j}\)
\( \newcommand\Rk{ k}\)
\( \newcommand\Rl{ l}\)
\( \newcommand { m}\)
\( \newcommand\Rn{ n}\)
\( \newcommand\Ro{ o}\)
\( \newcommand\Rp{ p}\)
\( \newcommand\Rq{ q}\)
\( \newcommand\Rr{ r}\)
\( \newcommand\Rs{ s}\)
\( \newcommand\Rt{ t}\)
\( \newcommand\Ru{ u}\)
\( \newcommand\Rv{ v}\)
\( \newcommand\Rw{ w}\)
\( \newcommand\Rx{ x}\)
\( \newcommand\Ry{ y}\)
\( \newcommand\Rz{ z}\)
\( \newcommand\BBA{\boldsymbol\RA}\)
\( \newcommand\BBB{\boldsymbol\RB}\)
\( \newcommand\BBC{\boldsymbol\RC}\)
\( \newcommand\BBD{\boldsymbol\RD}\)
\( \newcommand\BBE{\boldsymbol\RE}\)
\( \newcommand\BBF{\boldsymbol\RF}\)
\( \newcommand\BBG{\boldsymbol\RG}\)
\( \newcommand\BBH{\boldsymbol\RH}\)
\( \newcommand\BBI{\boldsymbol\RI}\)
\( \newcommand\BBJ{\boldsymbol\RJ}\)
\( \newcommand\BBK{\boldsymbol\RK}\)
\( \newcommand\BBL{\boldsymbol\RL}\)
\( \newcommand\BBM{\boldsymbol }\)
\( \newcommand\BBN{\boldsymbol\RN}\)
\( \newcommand\BBO{\boldsymbol\RO}\)
\( \newcommand\BBP{\boldsymbol\RP}\)
\( \newcommand\BBQ{\boldsymbol\RQ}\)
\( \newcommand\BBR{\boldsymbol\RR}\)
\( \newcommand\BBS{\boldsymbol\RS}\)
\( \newcommand\BBT{\boldsymbol\RT}\)
\( \newcommand\BBU{\boldsymbol\RU}\)
\( \newcommand\BBV{\boldsymbol\RV}\)
\( \newcommand\BBW{\boldsymbol\RW}\)
\( \newcommand\BBX{\boldsymbol\RX}\)
\( \newcommand\BBY{\boldsymbol\RY}\)
\( \newcommand\BBZ{\boldsymbol\RZ}\)
\( \newcommand\BBa{\boldsymbol\Ra}\)
\( \newcommand\BBb{\boldsymbol\Rb}\)
\( \newcommand\BBc{\boldsymbol\Rc}\)
\( \newcommand\BBd{\boldsymbol\Rd}\)
\( \newcommand\BBe{\boldsymbol\Re}\)
\( \newcommand\BBf{\boldsymbol\Rf}\)
\( \newcommand\BBg{\boldsymbol\Rg}\)
\( \newcommand\BBh{\boldsymbol\Rh}\}\)
\( \newcommand\BBi{\boldsymbol\Ri}\)
\( \newcommand\BBj{\boldsymbol\Rj}\)
\( \newcommand\BBk{\boldsymbol\Rk}\)
\( \newcommand\BBl{boldsymbol\Rl}\)
\( \newcommand\BBm{\boldsymbol }\)
\( \newcommand\BBn{\boldsymbol\Rn}\)
\( \newcommand\BBo{\boldsymbol\Ro}\)
\( \newcommand\BBp{\boldsymbol\Rp}\)
\( \newcommand\BBq{\boldsymbol\Rq}\)
\( \newcommand\BBr{\boldsymbol\Rr}\)
\( \newcommand\BBs{\boldsymbol\Rs}\)
\( \newcommand\BBt{\boldsymbol\Rt}\)
\( \newcommand\BBu{\boldsymbol\Ru}\)
\( \newcommand\BBv{\boldsymbol\Rv}\)
\( \newcommand\BBw{\boldsymbol\Rw}\)
\( \newcommand\BBx{\boldsymbol\Rx}\)
\( \newcommand\BBy{\boldsymbol\Ry}\)
\( \newcommand\BBz{\boldsymbol\Rz}\)
\( \newcommand\tcb{\textcolor{blue}\)
\( \newcommand\tcr{\textcolor{red}\)
\( \newcommand\bnabla{\boldsymbol{\nabla}}\)
\( \newcommand\Bell{\boldsymbol\ell}\)
\( \newcommand\dbar{\,{\mathchar'26\mkern-12mu d}} \)
\( \newcommand\ns{^\vphantom{*}}\)
\( \newcommand\uar{\uparrow}\)
\( \newcommand\dar{\downarrow}\)
\( \newcommand\impi{\int\limits_{-\infty}^{\infty}\!\!}\)
\( \newcommand\izpi{\int\limits_{0}^{\infty}\!\!}\)
\( \newcommand\etc{\it etc.\/}\)
\( \newcommand\etal{\it et al.\/}\)
\( \newcommand\opcit{\it op. cit.\/}\)
\( \newcommand\ie{\it i.e.\/}\)
\( \newcommand\Ie{\it I.e.\/}\)
\( \newcommand\viz{\it viz.\/}\)
\( \newcommand\eg{\it e.g.\/}\)
\( \newcommand\Eg{\it E.g.\/}\)
\( \newcommand\dbar{\,{\mathchar'26\mkern-12mu d}} \)
\( \def\sss#1{\scriptscriptstyle #1}\)
\( \def\ss#1{\scriptstyle #1}\)
\( \def\ssr#1{\scriptstyle #1}\)
\( \def\ssf#1{\scriptstyle #1}\)
\( \newcommand\NA{N_{\ssr{\!A}}}\)
\( \newcommand\lala{\langle\!\langle}\)
\( \newcommand\rara{\rangle\!\rangle}\)
\( \newcommand\blan{\big\langle}\)
\( \newcommand\bran{\big\rangle}\)
\( \newcommand\Blan{\Big\langle}\)
\( \newcommand\Bran{\Big\rangle}\)
\( \newcommand\intl{\int\limits}\)
\( \newcommand\half{\frac{1}{2}}\)
\( \newcommand\third{\frac{1}{3}}\)
\( \newcommand\fourth{\frac{1}{4}}\)
\( \newcommand\eighth{\frac{1}{8}}\)
\( \newcommand\uar{\uparrow}\)
\( \newcommand\dar{\downarrow}\)
\( \newcommand\undertext#1{$\underline{\hbox{#1}}$}\)
\( \newcommand\Tra{\mathop{\textsf{Tr}}\,}\)
\( \newcommand\det{\mathop{\textsf{det}}\,}\)
\( \def\tket#1{|  #1 \rangle}\)
\( \def\tbra#1{\langle #1|}\)
\( \def\tbraket#1#2{\langle #1  |   #2 \rangle}\)
\( \def\texpect#1#2#3{\langle #1 |   #2  |  #3 \rangle}\)
\( \def\sket#1{|  \, #1 \,  \rangle}\)
\( \def\sbra#1{\langle \,  #1 \, |}\)
\( \def\sbraket#1#2{\langle \, #1  \, |  \, #2 \,  \rangle}\)
\( \def\sexpect#1#2#3{\langle \, #1 \, | \,  #2  \, | \, #3 \, \rangle}\)
\(\def\ket#1{\big| \, #1\, \big\rangle}\)
\( \def\bra#1{\big\langle \, #1 \, \big|}\)
\( \def\braket#1#2{\big\langle \, #1\, \big| \,#2 \,\big\rangle}\)
\( \def\expect#1#2#3{\big\langle\, #1\, \big|\, #2\, \big| \,#3\, \big\rangle}\)
\( \newcommand\pz{\partial}\)
\( \newcommand\pzb{\bar{\partial}}\)
\( \newcommand\svph{\vphantom{\int}}\)
\( \newcommand\vph{\vphantom{\sum_i}}\)
\( \newcommand\bvph{\vphantom{\sum_N^N}}\)
\( \newcommand\nd{^{\vphantom{\dagger}}}\)
\( \newcommand\ns{^{\vphantom{*}}}\)
\( \newcommand\yd{^\dagger}\)
\( \newcommand\zb{\bar z}\)
\( \newcommand\zdot{\dot z}\)
\( \newcommand\zbdot{\dot{\bar z}}\)
\( \newcommand\kB{k_{\sss{B}}}\)
\( \newcommand\kT{k_{\sss{B}}T}\)
\( \newcommand\gtau{g_\tau}\)
\( \newcommand\Htil{\tilde H}\)
\( \newcommand\pairo{(\phi\nd_0,J\nd_0)}\)
\( \newcommand\pairm{(\phi\nd_0,J)}\)
\( \newcommand\pairob{(\Bphi\nd_0,\BJ\nd_0)}\)
\( \newcommand\pairmb{(\Bphi\nd_0,\BJ)}\)
\( \newcommand\pair{(\phi,J)}\)
\( \newcommand\Hz{H\nd_0}\)
\( \newcommand\Ho{H\nd_1}\)
\( \newcommand\Htz{\Htil\nd_0}\)
\( \newcommand\Hto{\Htil\nd_1}\)
\( \newcommand\oc{\omega_\Rc}\)

\(\newcommand \gtwid{\approx}\)

\( \newcommand\index{\textsf{ind}}\)
\( \newcommand\csch{\,{ csch\,}}\)
\( \newcommand\ctnh{\,{ ctnh\,}}\)
\( \newcommand\ctn{\,{ ctn\,}}\)
\( \newcommand\sgn{\,{ sgn\,}}\)
\( \def\tmapright#1{\xrightarrow \limits^{#1}}\)
\( \def\bmapright#1{\xrightarrow\limits_{#1}}\)
\( \newcommand\hfb{\hfill\break}\)
\( \newcommand\Rep{\textsf{Re}\,}\)
\( \newcommand\Imp{\textsf{Im}\,}\)
\( \newcommand\ncdot{\!\cdot\!}\)
\( \def\tmapright#1{ \smash{\mathop{\hbox to 35pt{\rightarrowfill}}\limits^{#1}}\ }\)
\( \def\bmapright#1{ \smash{\mathop{\hbox to 35pt{\rightarrowfill}}\limits_{#1}}\ }\)
\( \newcommand\bsqcap{\mbox{\boldmath{$\sqcap$}}}\)

\( \def\pabc#1#2#3{\left({\pz #1\over\pz #2}\right)\ns_{\!\!#3}}\)
\( \def\spabc#1#2#3{\big({\pz #1\over\pz #2}\big)\ns_{\!#3}}\)
\( \def\qabc#1#2#3{\pz^2\! #1\over\pz #2\,\pz #3}\)
\( \def\rabc#1#2#3#4{(\pz #1,\pz #2)\over (\pz #3,\pz #4)}\)
\( \newcommand\subA{\ns_\ssr{A}}\)
\( \newcommand\subB{\ns_\ssr{B}}\)
\( \newcommand\subC{\ns_\ssr{C}}\)
\( \newcommand\subD{\ns_\ssr{D}}\)
\( \newcommand\subAB{\ns_\ssr{AB}}\)
\( \newcommand\subBC{\ns_\ssr{BC}}\)
\( \newcommand\subCD{\ns_\ssr{CD}}\)
\( \newcommand\subDA{\ns_\ssr{DA}}\)
\( \def\lmapright#1{\ \ \smash{\mathop{\hbox to 55pt{\rightarrowfill}}\limits^{#1}}\ \ }\)
\( \def\enth#1{\RDelta {\textsf H}^0_\Rf[{ #1}]}\)
\( \newcommand\longrightleftharpoons{ \mathop{\vcenter{\hbox{\ooalign{\raise1pt\hbox{$\longrightharpoonup\joinrel$}\crcr  \lower1pt\hbox{$\longleftharpoondown\joinrel$}}}}}}\)
\( \newcommand\longrightharpoonup{\relbar\joinrel\rightharpoonup}\)
\( \newcommand\longleftharpoondown{\leftharpoondown\joinrel\relbar}\)
\( \newcommand\cds{\,\bullet\,}\)
\( \newcommand\ccs{\,\circ\,}\)
\( \newcommand\nsub{_{\vphantom{\dagger}}}\)
\( \newcommand\rhohat{\hat\rho}\)
\( \newcommand\vrhhat{\hat\vrh}\)
\( \newcommand\impi{\int\limits_{-\infty}^\infty\!\!\!}\)
\( \newcommand\brangle{\big\rangle}\)
\( \newcommand\blangle{\big\langle}\)
\( \newcommand\vet{\tilde\ve}\)
\( \newcommand\zbar{\bar z}\)
\( \newcommand\ftil{\tilde f}\)
\( \newcommand\XBE{\RXi\ns_\ssr{BE}}\)
\( \newcommand\XFD{\RXi\ns_\ssr{FD}}\)
\( \newcommand\OBE{\Omega\ns_\ssr{BE}}\)
\( \newcommand\OFD{\Omega\ns_\ssr{FD}}\)
\( \newcommand\veF{\ve\ns_\RF}\)
\( \newcommand\kF{k\ns_\RF}\)
\( \newcommand\kFu{k\ns_{\RF\uar}}\)
\( \newcommand\SZ{\textsf Z}}\) \( \newcommand\kFd{k\ns_{\RF\dar}\)
\( \newcommand\muB{\mu\ns_\ssr{B}}\)
\( \newcommand\mutB{\tilde\mu}\ns_\ssr{B}\)
\( \newcommand\xoN{\Bx\ns_1\,,\,\ldots\,,\,\Bx\ns_N}\)
\( \newcommand\rok{\Br\ns_1\,,\,\ldots\,,\,\Br\ns_k}\)
\( \newcommand\xhiOZ{\xhi^\ssr{OZ}}\)
\( \newcommand\xhihOZ

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/span[1], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\jhz{\HJ(0)}\)
\( \newcommand\nda{\nd_\alpha}\)
\( \newcommand\ndap{\nd_{\alpha'}}\)
\( \newcommand\labar

ParseError: invalid DekiScript (click for details)

Callstack:
    at (Template:MathJaxArovas), /content/body/div/span[2], line 1, column 1
    at template()
    at (Bookshelves/Thermodynamics_and_Statistical_Mechanics/Book:_Thermodynamics_and_Statistical_Mechanics_(Arovas)/01:_Fundamentals_of_Probability/1.03:_Entropy_and_Probability), /content/body/p/span, line 1, column 23

\)
\( \newcommand\msa{m\ns_\ssr{A}}\)
\( \newcommand\msb{m\ns_\ssr{B}}\)
\( \newcommand\mss{m\ns_\Rs}\)
\( \newcommand\HBx{\hat\Bx}\)
\( \newcommand\HBy{\hat\By}\)
\( \newcommand\HBz{\hat\Bz}\)
\( \newcommand\thm{\theta\ns_m}\)
\( \newcommand\thp{\theta\ns_\phi}\)
\( \newcommand\mtil{\widetilde m}\)
\( \newcommand\phitil{\widetilde\phi}\)
\( \newcommand\delf{\delta\! f}\)
\( \newcommand\coll{\bigg({\pz f\over\pz t}\bigg)\nd_{\! coll}}\)
\( \newcommand\stre{\bigg({\pz f\over\pz t}\bigg)\nd_{\! str}}\)
\( \newcommand\idrp{\int\!\!{d^3\!r\,d^3\!p\over h^3}\>}\)
\( \newcommand\vbar{\bar v}\)
\( \newcommand\BCE{\mbox{\boldmath{$\CE$}}\!}\)
\( \newcommand\BCR{\mbox{\boldmath{$\CR$}}\!}\)
\( \newcommand\gla{g\nd_{\RLambda\nd}}\)
\( \newcommand\TA{T\ns_\ssr{A}}\)
\( \newcommand\TB{T\ns_\ssr{B}}\)
\( \newcommand\ncdot{\!\cdot\!}\)
\( \newcommand\NS{N\ns_{\textsf S}}\)

Entropy and Information Theory

It was shown in the classic 1948 work of Claude Shannon that entropy is in fact a measure of information⁵. Suppose we observe that a particular event occurs with probability \(p\). We associate with this observation an amount of information \(I(p)\). The information \(I(p)\) should satisfy certain desiderata:

• Information is non-negative, \(I(p)\ge 0\).
• If two events occur independently so their joint probability is \(p\ns_1\,p\ns_2\), then their information is additive, \(I(p\ns_1 p\ns_2)=I(p\ns_1)+I(p\ns_2)\).
• \(I(p)\) is a continuous function of \(p\).
• There is no information content to an event which is always observed, \(I(1)=0\).

From these four properties, it is easy to show that the only possible function \(I(p)\) is \[I(p)=-A\,\ln p\ ,\] where \(A\) is an arbitrary constant that can be absorbed into the base of the logarithm, since \(\log\ns_b x = \ln x / \ln b\). We will take \(A=1\) and use \(e\) as the base, so \(I(p)=-\ln p\). Another common choice is to take the base of the logarithm to be \(2\), so \(I(p)=-\log\ns_2 p\). In this latter case, the units of information are known as bits. Note that \(I(0)=\infty\). This means that the observation of an extremely rare event carries a great deal of information⁶

Now suppose we have a set of events labeled by an integer \(n\) which occur with probabilities \(\{p\ns_n\}\). What is the expected amount of information in \(N\) observations? Since event \(n\) occurs an average of \(N p\ns_n\) times, and the information content in \(p\ns_n\) is \(-\ln p\ns_n\), we have that the average information per observation is \[S={\langle I\ns_N\rangle\over N} = -\sum_n p\ns_n\ln p\ns_n \ ,\] which is known as the entropy of the distribution. Thus, maximizing \(S\) is equivalent to maximizing the information content per observation.

Consider, for example, the information content of course grades. As we shall see, if the only constraint on the probability distribution is that of overall normalization, then \(S\) is maximized when all the probabilities \(p\ns_n\) are equal. The binary entropy is then \(S=\log\ns_2\Gamma\), since \(p\ns_n=1/\Gamma\). Thus, for pass/fail grading, the maximum average information per grade is \(-\log\ns_2 (\frac{1}{2})=\log\ns_2 2 =1\) bit. If only A, B, C, D, and F grades are assigned, then the maximum average information per grade is \(\log\ns_2 5= 2.32\) bits. If we expand the grade options to include {A+, A, A-, B+, B, B-, C+, C, C-, D, F}, then the maximum average information per grade is \(\log\ns_2 11 = 3.46\) bits.

Equivalently, consider, following the discussion in vol. 1 of Kardar, a random sequence \(\{n\ns_1,n\ns_2,\ldots,n\ns_N\}\) where each element \(n\ns_j\) takes one of \(K\) possible values. There are then \(K^N\) such possible sequences, and to specify one of them requires \(\log\ns_2 (K^N)=N\log\ns_2 K\) bits of information. However, if the value \(n\) occurs with probability \(p\ns_n\), then on average it will occur \(N\ns_n=N p\ns_n\) times in a sequence of length \(N\), and the total number of such sequences will be \[g(N)={N!\over\prod_{n=1}^K N\ns_n !}\ .\] In general, this is far less that the total possible number \(K^N\), and the number of bits necessary to specify one from among these \(g(N)\) possibilities is \[\log\ns_2 g(N)=\log\ns_2 (N!)- \sum_{n=1}^K \log\ns_2 (N\ns_n! )\approx -N\sum_{n=1}^K p\ns_n \log\ns_2 p\ns_n\ ,\] up to terms of order unity. Here we have invoked Stirling’s approximation. If the distribution is uniform, then we have \(p\ns_n={1\over K}\) for all \(n\in\{1,\ldots,K\}\), and \(\log\ns_2 g(N)=N\log\ns_2 K\).

Probability distributions from maximum entropy

We have shown how one can proceed from a probability distribution and compute various averages. We now seek to go in the other direction, and determine the full probability distribution based on a knowledge of certain averages.

At first, this seems impossible. Suppose we want to reproduce the full probability distribution for an \(N\)-step random walk from knowledge of the average \(\langle X \rangle = (2p-1)N\), where \(p\) is the probability of moving to the right at each step (see §1 above). The problem seems ridiculously underdetermined, since there are \(2^N\) possible configurations for an \(N\)-step random walk: \(\sigma\ns_j=\pm 1\) for \(j=1,\ldots,N\). Overall normalization requires \[\sum_{\{\sigma\ns_j\}}P(\sigma\ns_1,\ldots,\sigma\ns_N)=1\ ,\] but this just imposes one constraint on the \(2^N\) probabilities \(P(\sigma\ns_1,\ldots,\sigma\ns_N)\), leaving \(2^N-1\) overall parameters. What principle allows us to reconstruct the full probability distribution \[P(\sigma\ns_1,\ldots,\sigma\ns_N)=\prod_{j=1}^N \big( p\,\delta\ns_{\sigma\ns_j,1} + q \, \delta\ns_{\sigma\ns_j,-1}\big) = \prod_{j=1}^N p^{(1+\sigma\ns_j)/2}\,q^{(1-\sigma\ns_j)/2}\ ,\] corresponding to \(N\) independent steps?

The principle of maximum entropy

The entropy of a discrete probability distribution \(\{p\ns_n\}\) is defined as \[S=-\sum_n p\ns_n\ln p\ns_n\ ,\] where here we take \(e\) as the base of the logarithm. The entropy may therefore be regarded as a function of the probability distribution: \(S=S\big(\{p\ns_n\}\big)\). One special property of the entropy is the following. Suppose we have two independent normalized distributions \(\big\{p^\SA_a\big\}\) and \(\big\{p^\SB_b\big\}\). The joint probability for events \(a\) and \(b\) is then \(P\ns_{a,b}=p^\SA_a\,p^\SB_b\). The entropy of the joint distribution is then \[\begin{aligned} S&=-\sum_a\sum_b P\ns_{a,b}\ln P\ns_{a,b} = -\sum_a\sum_b p^\SA_a\, p^\SB_b \ln \big(p^\SA_a\,p^\SB_b\big) = -\sum_a\sum_b p^\SA_a\, p^\SB_b \big(\ln p^\SA_a + \ln p^\SB_b\big) \nonumber \\ &= -\sum_a p^\SA_a \ln p^\SA_a \cdot \sum_b p^\SB_b -\sum_b p^\SB_b \ln p^\SB_b \cdot \sum_a p^\SA_a =-\sum_a p^\SA_a \ln p^\SA_a - \sum_b p^\SB_b \ln p^\SB_b \nonumber \\ &= S^\SA+S^\SB\ .\nonumber\end{aligned}\] Thus, the entropy of a joint distribution formed from two independent distributions is additive.

Suppose all we knew about \(\{p\ns_n\}\) was that it was normalized. Then \(\sum_n p\ns_n=1\). This is a constraint on the values \(\{p\ns_n\}\). Let us now extremize the entropy \(S\) with respect to the distribution \(\{p\ns_n\}\), but subject to the normalization constraint. We do this using Lagrange’s method of undetermined multipliers. We define \[S^*\big(\{p\ns_n\},\lambda\big)=-\sum_n p\ns_n \ln p\ns_n - \lambda\Big(\sum_n p\ns_n -1 \Big)\] and we freely extremize \(S^*\) over all its arguments. Thus, for all \(n\) we have \[\begin{split} 0&={\pz S^*\over\pz p\ns_n}=-\big(\ln p\ns_n + 1 + \lambda\big)\\ 0&={\pz S^*\over\pz\lambda}=\sum_n p\ns_n-1 \ . \end{split}\] From the first of these equations, we obtain \(p\ns_n=e^{-(1+\lambda)}\), and from the second we obtain \[\sum_n p\ns_n=e^{-(1+\lambda)}\cdot\sum_n 1 = \Gamma\, e^{-(1+\lambda)}\ ,\] where \(\Gamma\equiv \sum_n 1\) is the total number of possible events. Thus, \(p\ns_n=1/\Gamma\), which says that all events are equally probable.

Now suppose we know one other piece of information, which is the average value \(X=\sum_n X\ns_n\,p\ns_n\) of some quantity. We now extremize \(S\) subject to two constraints, and so we define \[S^*\big(\{p\ns_n\},\lambda\ns_0,\lambda\ns_1\big)=-\sum_n p\ns_n \ln p\ns_n - \lambda\ns_0\Big(\sum_n p\ns_n -1 \Big) -\lambda\ns_1\Big(\sum_n X\ns_n \,p\ns_n - X\Big)\ .\] We then have \[{\pz S^*\over\pz p\ns_n}=-\big(\ln p\ns_n + 1 + \lambda\ns_0 + \lambda\ns_1\,X\ns_n\big) = 0\ ,\] which yields the two-parameter distribution \[p\ns_n=e^{-(1+\lambda\ns_0)}\,e^{-\lambda\ns_1 X\ns_n}\ .\] To fully determine the distribution \(\{p\ns_n\}\) we need to invoke the two equations \(\sum_n p\ns_n=1\) and \(\sum_n X\ns_n\,p\ns_n=X\), which come from extremizing \(S^*\) with respect to \(\lambda\ns_0\) and \(\lambda\ns_1\), respectively: \[\begin{split} 1&=e^{-(1+\lambda\ns_0)}\sum_n e^{-\lambda\ns_1 X\ns_n} \\ X&=e^{-(1+\lambda\ns_0)}\sum_n X\ns_n\, e^{-\lambda\ns_1 X\ns_n} \ . \end{split}\]

General formulation

The generalization to \(K\) extra pieces of information (plus normalization) is immediately apparent. We have \[X^a=\sum_n X^a_n\,p\ns_n\ , \label{Kpoc}\] and therefore we define \[S^*\big(\{p\ns_n\},\{\lambda\ns_a\}\big)=-\sum_n p\ns_n \ln p\ns_n - \sum_{a=0}^K \lambda\ns_a \Big(\sum_n X^a_n\,p\ns_n - X^a\Big) \ ,\] with \(X^{(a=0)}_n\equiv X^{(a=0)}=1\). Then the optimal distribution which extremizes \(S\) subject to the \(K+1\) constraints is \[\begin{split} p\ns_n&=\exp\Bigg\{\!-1-\sum_{a=0}^K \lambda\ns_a\,X^a_n \Bigg\}\\ &={1\over Z}\exp\Bigg\{\!-\sum_{a=1}^K \lambda\ns_a\,X^a_n \Bigg\}\ , \end{split}\] where \(Z=e^{1+\lambda\ns_0}\) is determined by normalization: \(\sum_n p\ns_n=1\). This is a \((K+1)\)-parameter distribution, with \(\{\lambda\ns_0,\lambda\ns_1,\ldots,\lambda\ns_K\}\) determined by the \(K+1\) constraints in Equation [Kpoc].

Example

As an example, consider the random walk problem. We have two pieces of information: \[\begin{split} \sum_{\sigma\ns_1}\cdots\sum_{\sigma\ns_N} P(\sigma\ns_1,\ldots,\sigma\ns_N) &= 1 \\ \sum_{\sigma\ns_1}\cdots\sum_{\sigma\ns_N} P(\sigma\ns_1,\ldots,\sigma\ns_N) \sum_{j=1}^N \sigma\ns_j &= X\ . \end{split}\] Here the discrete label \(n\) from §3.2 ranges over \(2^N\) possible values, and may be written as an \(N\) digit binary number \(r\ns_N\cdots r\ns_1\), where \(r\ns_j=\half(1+\sigma\ns_j)\) is \(0\) or \(1\). Extremizing \(S\) subject to these constraints, we obtain \[P(\sigma\ns_1,\ldots,\sigma\ns_N) = \CC\,\exp\Bigg\{\!-\lambda\sum_j \sigma\ns_j\Bigg\}= \CC\prod_{j=1}^N e^{-\lambda\,\sigma\ns_j}\ ,\] where \(\CC\equiv e^{-(1+\lambda\ns_0)}\) and \(\lambda\equiv\lambda\ns_1\). Normalization then requires \[\Tra P \equiv \sum_{\{\sigma\ns_j\}}P(\sigma\ns_1,\ldots,\sigma\ns_N) = \CC\,\big(e^\lambda+ e^{-\lambda}\big)^N\ ,\] hence \(\CC=(\cosh\lambda)^{-N}\). We then have \[P(\sigma\ns_1,\ldots,\sigma\ns_N) = \prod_{j=1}^N {e^{-\lambda\sigma\ns_j}\over e^{\lambda} + e^{-\lambda}} =\prod_{j=1}^N \big( p\,\delta\ns_{\sigma\ns_j,1} + q \, \delta\ns_{\sigma\ns_j,-1}\big) \ ,\] where \[p={e^{-\lambda}\over e^{\lambda} + e^{-\lambda}} \quad,\quad q=1-p={e^\lambda\over e^{\lambda} + e^{-\lambda}} \ .\] We then have \(X=(2p-1)N\), which determines \(p=\half(N+X)\), and we have recovered the Bernoulli distribution.

Of course there are no miracles⁷, and there are an infinite family of distributions for which \(X=(2p-1)N\) that are not Bernoulli. For example, we could have imposed another constraint, such as \(E=\sum_{j=1}^{N-1}\sigma\ns_j\,\sigma\ns_{j+1}\). This would result in the distribution \[P(\sigma\ns_1,\ldots,\sigma\ns_N)={1\over Z}\exp\Bigg\{\!-\lambda\ns_1\sum_{j=1}^N\sigma\ns_j -\lambda\ns_2\!\sum_{j=1}^{N-1} \sigma\ns_j\,\sigma\ns_{j+1}\Bigg\}\ ,\] with \(Z(\lambda\ns_1,\lambda\ns_2)\) determined by normalization: \(\sum_\Bsigma P(\Bsigma)=1\). This is the one-dimensional Ising chain of classical equilibrium statistical physics. Defining the transfer matrix \(R\ns_{ss'}= e^{-\lambda\ns_1(s+s')/2}\,e^{-\lambda\ns_2 ss'}\) with \(s,s'=\pm 1\) , \[\begin{split} R&=\begin{pmatrix} e^{-\lambda\ns_1-\lambda\ns_2} & e^{\lambda\ns_2} \\ e^{\lambda\ns_2} & e^{\lambda\ns_1-\lambda\ns_2}\end{pmatrix}\\ &=e^{-\lambda\ns_2}\cosh\lambda\ns_1 \,\MI + e^{\lambda\ns_2}\,\tau^x - e^{-\lambda\ns_2}\sinh\lambda\ns_1\,\tau^z\ , \end{split}\] where \(\tau^x\) and \(\tau^z\) are Pauli matrices, we have that \[Z\ns_{ring}=\Tra\!\big(R^N\big) \quad,\quad Z\ns_{chain}=\Tra\!\big(R^{N-1}S\big)\ ,\] where \(S\ns_{ss'}=e^{-\lambda\ns_1(s+s')/2}\) , \[\begin{split} S&=\begin{pmatrix} e^{-\lambda\ns_1} & 1 \\ 1 & e^{\lambda\ns_1}\end{pmatrix}\\ &=\cosh\lambda\ns_1 \, \MI + \tau^x - \sinh\lambda\ns_1\,\tau^z\ . \end{split}\] The appropriate case here is that of the chain, but in the thermodynamic limit \(N\to\infty\) both chain and ring yield identical results, so we will examine here the results for the ring, which are somewhat easier to obtain. Clearly \(Z\ns_{ring}=\zeta_+^N +\zeta_-^N\) , where \(\zeta\ns_\pm\) are the eigenvalues of \(R\): \[\zeta\ns_\pm=e^{-\lambda\ns_2}\cosh\lambda\ns_1\pm\sqrt{e^{-2\lambda\ns_2}\sinh^2\!\lambda\ns_1 + e^{2\lambda\ns_2}}\quad .\] In the thermodynamic limit, the \(\zeta\ns_+\) eigenvalue dominates, and \(Z\ns_{ring}\simeq \zeta_+^N\). We now have \[X=\Big\langle\sum_{j=1}^N\sigma\ns_j\Big\rangle = -{\pz\ln Z\over\pz\lambda\ns_1} = -{N\sinh\lambda\ns_1\over \sqrt{\sinh^2\!\lambda\ns_1 + e^{4\lambda\ns_2}}}\ .\] We also have \(E=-\pz\ln Z/\pz\lambda\ns_2\). These two equations determine the Lagrange multipliers \(\lambda\ns_1(X,E,N)\) and \(\lambda_2(X,E,N)\). In the thermodynamic limit, we have \(\lambda\ns_i=\lambda\ns_i(X/N,E/N)\). Thus, if we fix \(X/N=2p-1\) alone, there is a continuous one-parameter family of distributions, parametrized \(\ve=E/N\), which satisfy the constraint on \(X\).

So what is it about the maximum entropy approach that is so compelling? Maximum entropy gives us a calculable distribution which is consistent with maximum ignorance given our known constraints. In that sense, it is as unbiased as possible, from an information theoretic point of view. As a starting point, a maximum entropy distribution may be improved upon, using Bayesian methods for example (see §5.2 below).

Continuous probability distributions

Suppose we have a continuous probability density \(P(\Bvphi)\) defined over some set \(\ROmega\). We have observables \[X^a=\int\limits_\ROmega\!\!d\mu\>X^a(\Bvphi)\,P(\Bvphi)\ , \label{constcont}\] where \(d\mu\) is the appropriate integration measure. We assume \(d\mu=\prod_{j=1}^D d\vphi\ns_j\), where \(D\) is the dimension of \(\ROmega\). Then we extremize the functional \[S^*\big[P(\Bvphi),\{\lambda\ns_a\}\big]=-\int\limits_\ROmega\!\!d\mu\> P(\Bvphi)\ln P(\Bvphi) - \sum_{a=0}^K\lambda\ns_a \Bigg( \int\limits_\ROmega\!\!d\mu\> P(\Bvphi)\,X^a(\Bvphi) - X^a \Bigg)\] with respect to \(P(\Bvphi)\) and with respect to \(\{\lambda\ns_a\}\). Again, \(X^0(\Bvphi)\equiv X^0 \equiv 1\). This yields the following result: \[\ln P(\Bvphi)=-1-\sum_{a=0}^K \lambda\ns_a\, X^a(\Bvphi)\ .\] The \(K+1\) Lagrange multipliers \(\{\lambda\ns_a\}\) are then determined from the \(K+1\) constraint equations in Equation [constcont].

As an example, consider a distribution \(P(x)\) over the real numbers \(\MR\). We constrain \[\int\limits_{-\infty}^\infty\!\!dx\>P(x)=1 \quad,\quad \int\limits_{-\infty}^\infty\!\!dx\>x\,P(x)=\mu \quad,\quad \int\limits_{-\infty}^\infty\!\!dx\>x^2\,P(x)=\mu^2+\sigma^2\ .\] Extremizing the entropy, we then obtain \[P(x)=\CC\,e^{-\lambda\ns_1 x - \lambda\ns_2 x^2}\ ,\] where \(\CC=e^{-(1+\lambda\ns_0)}\). We already know the answer: \[P(x)={1\over\sqrt{2\pi\sigma^2}}\,e^{-(x-\mu)^2/2\sigma^2}\ .\] In other words, \(\lambda\ns_1=-\mu/\sigma^2\) and \(\lambda\ns_2=1/2\sigma^2\), with \(\CC=(2\pi\sigma^2)^{-1/2}\,\exp(-\mu^2/2\sigma^2)\).