That's why during my rest days, I will surely sleep as long as I can, but I notice that, I'm much more looks haggard and exhausted if I sleep more than 8 hours, I feel that, I did something heavy and looks like really tired.
You will just have to get used to sleeping longer hours. For the fact that you have trained yourself to sleep for few hours have made it used to sleeping for short periods of time, it's the reason why when you try sleeping for far too long, you wake up feeling exhausted.