Derivation
え and エ originate, via man'yōgana, from the kanji 衣 and 江, respectively.
The archaic kana ゑ (we), as well as many non-initial occurrences of the character へ (he), have entered the modern Japanese language as え. The directional particle へ is today pronounced "e", though not written as え. Compare this to は (ha) and を (wo), which are pronounced "wa" and "o" when used as grammatical particles.
When initial or following /i/ or /n/, many speakers give the letter an initial glide, pronouncing it, leading to Romanizations such as "yen", "Yedo", "Yebisu", and so forth.
Read more about this topic: E (kana)