One of the themes when Kevin White was hired by Duke was: why leave Notre Dame for Duke? Duke is a great school and the athletic program is in many ways spectacular, but Notre Dame is a pretty special place too, and football, as we've (re) learned in recent years, is the bigger deal nationally.
One reason might be pretty simple: having the authority to do your job. Notre Dame, as it turns out, doesn't allow the A.D. much control over his department. If you're a talented and bright guy, that could get pretty frustrating.