My husband won't let me touch him while he is asleep

My husband will let me touch him when he is awake, likewise when we are having sex.

However he won't let me touch him if he is sleeping, he becomes angry and will remove my hand or push me away. It's got worse recently, and honestly it scares me. He knows it's a problem but I've been understanding and supportive, I am unsure how I can help?