Protocol
NTLM is a challenge-response authentication protocol which uses three messages to authenticate a client in a connection oriented environment (connectionless is similar), and a fourth additional message if integrity is desired. First, the client establishes a network path to the server and sends a NEGOTIATE_MESSAGE advertising its capabilities. Next, the server responds with CHALLENGE_MESSAGE which is used to establish the identity of the client. Finally, the client responds to the challenge with an AUTHENTICATE_MESSAGE.
The NTLM protocol uses one or both of two hashed password values, both of which are also stored on the server (or domain controller), and which are password equivalent, meaning that if you grab the hash value from the server, you can authenticate without knowing the actual password. The two are the LM Hash (a DES-based function applied to the first 14 chars of the password converted to the traditional 8 bit PC charset for the language), and the NT Hash (MD4 of the little endian UTF-16 Unicode password). Both hash values are 16 bytes (128 bits) each.
The NTLM protocol also uses one of two one way functions, depending on the NTLM version. NT LanMan and NTLM version 1 use the DES based LanMan one way function (LMOWF), while NTLMv2 uses the NT MD4 based one way function (NTOWF).
Read more about this topic: NTLM