Ed Norton spoke to this issue in an interview with NPR. The very short answer is that he deeply disliked the "roadshow" aspect of marketing a major tentpole film and wasn't willing to compromise.
“My feeling was that I experimented and experienced what I wanted to. I really, really enjoyed it. And yet, I looked at the balance of time in life that one spends not only making those sorts of films but then especially putting them out, and the obligations that rightly come with that."
He also stated that he feared becoming type-cast.
"There were just a lot of things—I wanted more diversity. I sort of chose to continue on my path of having a diversity of experiences. Maybe on some unconscious level, I didn’t want to have an association with one thing in any way degrade my effectiveness as an actor, in characters. I think you can sort of do anything once, but if you do it too many times, it can become a suit that’s hard to take off, in other peoples’ eyes. And if I had continued on with it, I wouldn’t have made Moonrise Kingdom, or Grand Budapest, or Birdman, because those all overlapped with [Avengers]. And those were more the priority for me, but I continue to be a fan and I’m really, really happy I got to do it once.”