Introduction to machine learning (I): understanding maximum likelihood estimation in supervised learning

1. Abstract

   This article decrypts the machine learning modeling process in the context of Statistics . We will show you how assumptions about data enable us to create meaningful optimization problems . in fact , We will derive common criteria , Such as cross entropy in classification and mean square error in regression .

2. likelihood VS Probability and probability density

First , Let's start with a basic question ： What is the difference between possibility and probability ？ data $x$, Passing probability $P(x,\theta )$ Or probability density function (pdf)$P(x,\theta )$ Connect to possible models $\theta$ .

In short , The probability density function gives the probability of occurrence of different possible values . The probability density function describes the infinitesimal probability of any given value . We insist on using pdf The symbol of . For any given set of parameters $\theta$ ,$P(x,\theta )$ Aims to become $x$ The probability density function of .

likelihood $P(x,\theta )$ Is defined as the joint density of observed data , As a function of model parameters . It means , For any given $x$,$p(x=\mathrm{fixed},\theta )$ Can be seen as $\theta$ Function of . therefore , Likelihood function is only a parameter $\theta$ Function of , The data remains a fixed constant .

What we will consider is , What we will consider is , We have to deal with a problem caused by $m$ Data instances $X$ aggregate $X=\{{\text{x}}^{(1)},..,{\text{x}}^{(m)}\}$, Follow the empirical training data distribution $p_{data}^{train}(\text{x})=p_{data}(\text{x})$,$p_{data}^{real}(\text{x})$ It is a good and representative sample of unknown and wider data distribution .

3. Independent and identically distributed hypothesis

This brings us ML The most basic assumption ： Independent homologous distribution (IID) data （ A random variable ）. Statistical independence means for random variables A and B, Joint distribution $P_{A,B}(A,B)$

To be continued ..... Busy recently , Come back and fill the pit when you have time