Trang 1
CS 229 – Machine Learning https://stanford.edu/~shervine VIP Refresher: Probabilities and Statistics ∑n Remark: for any event B in the sample space, we have P(B) = P(B|Ai)P(Ai). i=1 Afshine Amidi and Shervine Amidi r Extended form of Bayes’ rule – Let {Ai, i ∈ [[1,n]]} be a partition of the sample space. We have: August 6, 2018 P(B|Ak)P(Ak) P(Ak|B) = ∑ n P(B|Ai)P(Ai) i=1 Introduction to Probability and Combinatorics r Sample space – The set of all possible outcomes of an experiment is known as the sample r Independence – Two events A and B are independent if and only if we have: space of the experiment and is denoted by S. P(A ∩ B) = P(A)P(B) r Event – Any subset E of the sample space is known as an event. That is, an event is a set consisting of possible outcomes of the experiment. If the outcome of the experiment is contained in E, then we say that E has occurred. Random Variables r Axioms of probability – For each event E, we denote P(E) as the probability of event E r Random variable – A random variable, often noted X, is a function that maps every element occuring. By noting E1,...,En mutually exclusive events, we have the 3 following axioms: in a sample space to a real line. ( ) ⋃n ∑n r Cumulative distribution function (CDF) – The cumulative distribution function F, (1) 0 6 P(E) 6 1 (2) P(S) = 1 (3) P Ei = P(Ei) which is monotonically non-decreasing and is such that lim F(x) = 0 and lim F(x) = 1, is x→−∞ x→+∞ i=1 i=1 defined as: F(x) = P(X 6 x) r Permutation – A permutation is an arrangement of r objects from a pool of n objects, in a given order. The number of such arrangements is given by P(n, r), defined as: Remark: we have P(a < X 6 B) = F(b) − F(a). n! P(n, r) = r Probability density function (PDF) – The probability density function f is the probability (n − r)! that X takes on values between two adjacent realizations of the random variable. r Relationships involving the PDF and CDF – Here are the important properties to know r Combination – A combination is an arrangement of r objects from a pool of n objects, where in the discrete (D) and the continuous (C) cases. the order does not matter. The number of such arrangements is given by C(n, r), defined as: P(n, r) n! Case CDF F PDF f Properties of PDF C(n, r) = = r! r!(n − r)! ∑ ∑ (D) F(x) = P(X = xi) f(xj) = P(X = xj) 0 6 f(xj) 6 1 and f(xj) = 1 Remark: we note that for 0 6 r 6 n, we have P(n,r) > C(n,r). xi6x j ˆ x ˆ +∞ Conditional Probability dF (C) F(x) = f(y)dy f(x) = f(x) > 0 and f(x)dx = 1 −∞ dx −∞ r Bayes’ rule – For events A and B such that P(B) > 0, we have: P(B|A)P(A) P(A|B) = r Variance – The variance of a random variable, often noted Var(X) or σ2, is a measure of the P(B) spread of its distribution function. It is determined as follows: Remark: we have P(A ∩ B) = P(A)P(B|A) = P(A|B)P(B). Var(X) = E[(X − E[X])2] = E[X2] − E[X]2 r Partition – Let {Ai, i ∈ [[1,n]]} be such that for all i, Ai 6= ∅. We say that {Ai} is a partition if we have: r Standard deviation – The standard deviation of a random variable, often noted σ, is a ⋃ measure of the spread of its distribution function which is compatible with the units of the n actual random variable. It is determined as follows: ∀i 6= j, Ai ∩ Aj = ∅ and Ai = S √ i=1 σ = Var(X) Stanford University 1 Fall 2018
Trang 2
CS 229 – Machine Learning https://stanford.edu/~shervine r Expectation and Moments of the Distribution – Here are the expressions of the expected r Marginal density and cumulative distribution – From the joint density probability value E[X], generalized expected value E[g(X)], kth moment E[Xk] and characteristic function function fXY , we have: ψ(ω) for the discrete and continuous cases: Case Marginal density Cumulative function ∑ ∑ ∑ Case E[X] E[g(X)] E[Xk] ψ(ω) (D) fX(xi) = fXY (xi,yj) FXY (x,y) = fXY (xi,yj) ∑n ∑n ∑n ∑n j xi6x yj6y (D) x f(x ) g(x )f(x ) xkf(x ) f(x )eiωxi ˆ ˆ ˆ i i i i i i i +∞ x y i=1 i=1 i=1 i=1 (C) f (x) = f (x,y)dy F (x,y) = f (x′,y′)dx′dy′ X XY XY XY ˆ +∞ ˆ +∞ ˆ +∞ ˆ +∞ −∞ −∞ −∞ (C) xf(x)dx g(x)f(x)dx xkf(x)dx f(x)eiωxdx −∞ −∞ −∞ −∞ r Distribution of a sum of independent random variables – Let Y = X +...+X with 1 n X1, ..., Xn independent. We have: Remark: we have eiωx = cos(ωx) + isin(ωx). ∏n ψY (ω) = ψX (ω) r Revisiting the kth moment – The kth moment can also be computed with the characteristic k function as follows: k=1 [ ] 1 ∂kψ E[Xk] = r Covariance – We define the covariance of two random variables X and Y , that we note σ2 ik ∂ωk XY ω=0 or more commonly Cov(X,Y ), as follows: Cov(X,Y ) , σ2 = E[(X − μ )(Y − μ )] = E[XY ] − μ μ XY X Y X Y r Transformation of random variables – Let the variables X and Y be linked by some function. By noting fX and fY the distribution function of X and Y respectively, we have: ∣ ∣ r Correlation – By noting σX, σY the standard deviations of X and Y , we define the correlation ∣∣dx∣∣ between the random variables X and Y , noted ρXY , as follows: fY (y) = fX(x) dy σ2 ρ = XY XY σ σ X Y r Leibniz integral rule – Let g be a function of x and potentially c, and a, b boundaries that may depend on c. We have: Remarks: For any X, Y , we have ρXY ∈ [−1,1]. If X and Y are independent, then ρXY = 0. (ˆ ) ˆ ∂ b ∂b ∂a b ∂g r Main distributions – Here are the main distributions to have in mind: g(x)dx = ·g(b) − ·g(a) + (x)dx ∂c a ∂c ∂c a ∂c Type Distribution PDF ψ(ω) E[X] Var(X) ( ) r Chebyshev’s inequality – Let X be a random variable with expected value μ and standard n X ∼ B(n, p) P(X = x) = pxqn−x (peiω + q)n np npq deviation σ. For k, σ > 0, we have the following inequality: x Binomial x ∈ [[0,n]] 1 (D) P(|X − μ| > kσ) 6 k2 μx X ∼ Po(μ) P(X = x) = e−μ eμ(eiω−1) μ μ x! Poisson x ∈ N Jointly Distributed Random Variables 1 eiωb − eiωa a + b (b − a)2 X ∼ U(a, b) f(x) = r Conditional density – The conditional density of X with respect to Y , often noted f , b − a (b − a)iω 2 12 X|Y Uniform x ∈ [a,b] is defined as follows: ( ) 1 1 x−μ 2 f (x,y) (C) X ∼ N(μ, σ) f(x) = √ e− 2 σ eiωμ− 1 ω2σ2 μ σ2 f (x) = XY 2 X|Y f (y) 2πσ Y Gaussian x ∈ R −λx 1 1 1 r Independence – Two random variables X and Y are said to be independent if we have: X ∼ Exp(λ) f(x) = λe 1 − iω λ λ2 Exponential x ∈ R λ fXY (x,y) = fX(x)fY (y) + Stanford University 2 Fall 2018
Trang 3
CS 229 – Machine Learning https://stanford.edu/~shervine Parameter estimation r Random sample – A random sample is a collection of n random variables X1, ..., Xn that are independent and identically distributed with X. r Estimator – An estimator θˆ is a function of the data that is used to infer the value of an unknown parameter θ in a statistical model. r Bias – The bias of an estimator θˆ is defined as being the difference between the expected value of the distribution of θˆ and the true value, i.e.: Bias(θˆ) = E[θˆ] − θ Remark: an estimator is said to be unbiased when we have E[θˆ] = θ. r Sample mean and variance – The sample mean and the sample variance of a random sample are used to estimate the true mean μ and the true variance σ2 of a distribution, are noted X and s2 respectively, and are such that: 1 ∑n 1 ∑n X = X and s2 = ˆσ2 = (X − X)2 n i n − 1 i i=1 i=1 r Central Limit Theorem – Let us have a random sample X1, ..., Xn following a given distribution with mean μ and variance σ2, then we have: ( ) σ X ∼ N μ, √ n→+∞ n Stanford University 3 Fall 2018
Hiển thị 3 trang đầu. Tải tài liệu để xem đầy đủ.
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!