You need to have a talk with your partner regarding this. The best way to go about any relationship issue is to talk it out and unless you don't take things will never change. More importantly, unless you don't talk you will never know what is going on and what are the issues to be addressed. No one can dive into the mind of your partner, and so you should be the one to ask him, poke him, make him talk. When he does talk, you will know what the issue is.