I think for most of my life I've thought of family as comprising those people closest to you

The umbrella usually covers romantic partners, close blood relatives and adopted children but there is room for flexibility (very close friends can become part of the family, and wayward children can be kicked out of the family). One can limit the definition of family to only include blood relations, and while I think that's a perfectly valid definition, it's usually of limited utility (if someone asks you about your family, presumably they're interested in learning about the people closest to you, so what good is it to tell them about your estranged biological brother that you've not spoken to in decades?)