The New 52
In the new continuity, it is revealed that Alfred's father, Jarvis Pennyworth was the butler of the Wayne family prior him when Bruce was still a child. Jarvis was blackmailed by the Court of Owls to set a trap for the pregnant Martha Wayne. Despite declining, the Court managed to cause a car accident that caused the child to be born prematurely and eventually to have died. Jarvis attempted to resign from his services and write a letter to his son, in which he describes the manor as a cursed place, and despite his initial wishes, he wants nothing more for Alfred than to not begin his service under the Wayne family. However, he was unable to send it as he was murdered that night.
Read more about this topic: Alfred Pennyworth, Fictional Character Biography