It’s not too early to go to sleep. Too early only applies to waking up.

Commentary:
"Ah, the eternal struggle of night owls everywhere. Who knew that 'too early' could be such a flexible concept? I guess it's all about perspective – early to bed, early to rise, makes a person healthy, wealthy, and…well, maybe a little grumpy if they're not a morning person!"