A Cartan subgroup of a compact connected Lie group is a maximal connected Abelian subgroup (a maximal torus). Its Lie algebra is a Cartan subalgebra.
For disconnected compact Lie groups there are several inequivalent definitions of a Cartan subgroup. The most common seems to be the one given by David Vogan, who defines a Cartan subgroup to be the group of elements that normalize a fixed maximal torus and fix the fundamental Weyl chamber. This is sometimes called the large Cartan subgroup. There is also a small Cartan subgroup, defined to be the centralizer of a maximal torus. These Cartan subgroups need not be abelian in general.
For connected algebraic groups over an algebraically closed field a Cartan subgroup is usually defined as the centralizer of a maximal torus. In this case the Cartan subgroups are connected, nilpotent, and are all conjugate.