If you're in an ivory tower, you live in a world of ideas separate from the realities of most people's lives. Your university philosophy department might be your ivory tower, where discussions about theory are the most important things in life. It's a privileged situation to be in, since ordinary people tend to have more mundane priorities, concerns, and struggles. The phrase most often describes academia, and it's been around since the early twentieth century, from the French tour d'ivoire.