Poisson Regression - Maximum Likelihood-based Parameter Estimation

Maximum Likelihood-based Parameter Estimation

Given a set of parameters θ and an input vector x, the mean of the predicted Poisson distribution, as stated above, is given by

,

and thus, the Poisson distribution's probability mass function is given by

Now suppose we are given a data set consisting of m vectors, along with a set of m values . Then, for a given set of parameters θ, the probability of attaining this particular set of data is given by

By the method of maximum likelihood, we wish to find the set of parameters θ that makes this probability as large as possible. To do this, the equation is first rewritten as a likelihood function in terms of θ:

.

Note that the expression on the right hand side has not actually changed. A formula in this form is typically difficult to work with; instead, one uses the log-likelihood:

.

Notice that the parameters θ only appear in the first two terms of each term in the summation. Therefore, given that we are only interested in finding the best value for θ we may drop the y_i! and simply write

.

To find a maximum, we need to solve an equation which has no closed-form solution. However, the negative log-likelihood, is a convex function, and so standard convex optimization techniques such as gradient descent can be applied to find the optimal value of θ.