Dear Jasmine,
Yes, you are indeed a family, but maybe your husband has the wrong idea about exactly what a "family" is and does. Within a healthy family, there is shared love and responsibility, between parents and children, and they do indeed share and enjoy together a number of suitable activities. But they also all need and enjoy time on their own, and activities of individual or partly hared interest. Normally, married parents enjoy doing some things togther one-one-one as a couple ; other things as a family group with their children ; and other things individually, as do the kids. Maybe he still feels insecure about the divorce, and about his ex being so close-by ; and feels, as some men do in such a situation, a sort of competitive urge to make the time the kids spend with him ( and you ) of such amazingly high quality that no-one could criticize him. He needs to relax, and enjoy his relationship with you, as well as with the children. Assuming, though, that the kids are presumably not with you every day, as they surely also spend time with their mum --- can't you two creatively enjoy your time together on the other days, when they're not with you ?