When the emperor was divine

Where is the first place that the family stays after being forced to leave their homes? What is the biggest problem about living there?