A transgender man/woman (or vice versa, M.T.F or F.T.M) will somewhat still retain the same qualities of their assigned-at-birth gender.
Meaning that a transgender woman could still have the strength/speed of a male, especially if they identify as female without having taken estrogen or anything of the sort.
This can be a large issue when you consider that men biologically: Feel less physical pain than women, have stronger muscles, and are faster.

And for those reasons above, men and women's sports categories are separated.
While it could be more "fair" in terms of equality for women who were assigned male at birth to participate in women's only sports categories, it PHYSICALLY is not fair.

So I guess the answer is; Yes.