I think I should start out by saying I am not a fashion expert! However, I do know what photographs well (or doesn't!).
Your clothing choices should be a reflection of your personal style. You should feel confident in what you wear. If you're uncomfortable or worried about your clothing you will look uncomfortable and worried in your photos.
You should also consider your home decor and colors. Your photos will be hanging on your walls so you will want them to compliment your decor and color scheme.
I recommend choosing 2-3 colors and carry that color scheme throughout the outfits so everything looks nice together, but doesn't match. I find that mixing light colored neutrals with an accent color looks really nice. Adding layers and textures in your outfits always looks great.
I would start with mom's outfit first and choose everyone else's clothing to match moms. So maybe a beautiful floral dress for mom and use the colors in her floral dress to determine everyone else's clothes. Or if mom chooses a solid colored outfit then dad or one of the children could have a pattern in their outfit. (Be careful of mixing too much pattern though.)
Also, keep in mind your location. You don't want to blend in to the background.
Your clothing should be unwrinkled and fit well. Clothing that is bulky might not look flattering.
The following is some do's and don'ts:
Don't wear clothing that is too trendy and will look outdated soon. I recommend choosing classic, neutral, and pastel colors.