The amount that play may be minuscule, however, the policies that allow them to play and to access women's locker rooms are pervasive across K-12 and the University systems. In other words, there are people who are in power (School Board, Chancellors, College Presidents, Sports Governing Boards etc.) who have all agreed that this is OK.
To me that is the major issue.....I suspect over time that more trans-females will make the jump to women's sports, but that is a total assumption. However if they do, people like me with a daughter in sports have NO AVENUES to protect my daughter rights to not be exposed to a man's penis.......her rights/feelings are totally not considered even though there is only a small minority of men competing in women's sports.
Sooooo, policies have been set regardless of if their is only a small # of instances. This is not a "fake panic" as you put it, but parents concerned about school/sports policies that ignore women's rights. That is NOT a trivial thing.