Pseudo-Philo
In Pseudo-Philo, one of the earliest accounts (c. AD 70) though not thought to be by Philo, the direction for the building is ascribed not only to Nimrod, who is made prince of the Hamites, but also to Joktan as prince of the Semites, and to Phenech son of Dodanim as prince of the Japhetites. Twelve men are arrested for refusing to bring bricks, including Abraham, Lot, Nahor, and several sons of Joktan. However, Joktan finally saves the twelve from the wrath of the other two princes.
Read more about this topic: Tower Of Babel, In Other Sources