Spending time together and sharing common interests is great, but focusing on your own interests also is healthy.
Your schedules and interests may not always work out. Pursue your interests with those who share those interests. Not instead of spending time with your spouse, rather in addition to the time you spend together.
You don’t have to go to every ball game he plays in. He doesn’t have to come with you to every gallery you visit.