There are too many factors throughout development (from childhood and into adulthood), hormone levels present and how ones body uses (or is unable to use those hormones) to make this a clear cut yes or no.

People are naturally born with hormonal advantages and physiological advantages without any intervention of drugs/hormone therapies. These advantages someone is born with already create a disparity which honestly make sport inherently "not fair".

When it gets to the level of actual professional competitive athletes I think further categories or different ways of classifying categories needs to occur.

While splitting by gender/sex the physiological difference of males and females, it hardly accounts for other factors that will give significant competive advantage.