The National Center for Transgender Equality (NCTE) has launched a campaign to gather information about trans identified people called the U.S. Trans Survey. The intent may be noble, but the data collection methods leave much to be desired. Here is some background on the project from its organizers (emphasis added):
[The U.S. Trans Survey] is the follow up to the groundbreaking National Transgender Discrimination Survey, which was conducted by the National Center for Transgender Equality and the National LGBTQ Task Force in 2009, and examined the lives of over 6,400 trans people in the U.S. The results were released in a 2011 report called Injustice At Every Turn. As the single-most cited study about trans people, it has changed how the public understands the challenges facing our community.
That last sentence is the crux of what bothers me. An internet based questionnaire is the single-most cited “study” about trans people? Are you serious? In what world is an anonymous survey with no sample validity considered the primary or best evidence by which “the public understands the challenges” faced by any group of people? Look, I’m no statistician but I did take Stats 101 in college. Even I can tell this “study” has holes the size of the Grand Canyon.
Before we review some of the most glaring flaws, I want to make perfectly clear that trans people are not the sole stakeholders of this survey’s results. The entire U.S. population is potentially impacted by legal and policy changes that cater to special interest groups. Revising bathroom and locker room access–from urogenital configuration (physical sex) to personal preference (gender identity)–is probably the best example of how policies designed to protect one group of people can inadvertently expose others to situations they find objectionable or even dangerous. As such, every single person in the U.S. has a stake in the conversation surrounding this project. It is every person’s business.
PLEASE VALIDATE YOUR IDENTITY, IF YOU CAN
The concept of identity is subjective by definition. I-dentity, as I like to spell it, is a self reported statement about the subject by the subject. It has no measurable criteria; it has no external reference by which we can judge whether the person claiming the I-dentity is lucid or delusional or somewhere in between. “Because I said so” is the entire standard of proof. This is not enough to accurately demarcate the boundaries of a discrete class of persons about whom conclusions can be accurately drawn and understood by others. I-dentity alone is a weak, even illusory, basis by which conduct a study that is intended to influence public policy.
More specifically, when we speak of gender I-dentity, internal feelings must be prioritized over material reality. The idea that your subjective insides don’t match your objective outsides is precisely what makes someone “trans” or “not trans.” See cisgender theory for more. Personally, I have no clue what inside/outside matching means because it reeks of essentialist and/or spiritual nonsense, but my point is that self-proclaimed gender I-dentity is wholly characterized by internal feelings. It’s all in your head, by definition.
Further, the meaning of what’s in your head is frequently contested. Being “trans” may include just about anyone or no one, depending on who you talk to. The DSM 5 requires dysphoria. But this recent article tells me that not all “trans” people experience body dysphoria. So, what’s The Truth? The problem is that there is no Truth About Trans People’s Internal Feelings. And that compounds the problems discussed above about the incoherent nature of I-dentity as a criteria for study.
Prevalence rates for transgenderism are hard to pin down, because the definitions are obscure, and the decision to identify as transgender is a subjective one. [source]
Indeed, the survey is not just for “trans” people. If you identify as “genderqueer” or “non-binary,” you are also encouraged to complete the survey. One of the emails I received from Sandy James, Survey Project Manager for the NCTE, explained that “getting the most diverse sample yet” was their goal. Hmmm, ok. Well, I might be genderqueer. Look at me, sitting here in my pants with no make-up and short hair! That’s, like, everyday baby. My sister asked me this past weekend if I had stopped shaving my legs. And shoot, I’m a lesbian too! Ok, I’m definitely genderqueer. How about you? Are you feeling “non-binary” in your mind? If so, this survey might be for you, wink wink.
Respondent feedback
VOTE EARLY AND VOTE OFTEN!
The U.S. Trans Survey has no automated participant controls or sample validity whatsoever. As you can see in this screenshot, it actually asks if you’ve taken it before. Um, no? Well, maybe. Who knows! Not you! Once again, we’re working entirely on the honor system: because I said so, dammit!
I’m really good at using The Google, so I did a little research into online polls. I quickly learned that ballot box stuffing is a common problem that serious researchers attempt to avoid.
Traditionally, ballot box stuffing applies to elections. It means to illegally fill a ballot box with more votes than the number of real voters. It also applies to surveys when a participant takes the survey more than once, or even multiple times. This immediately diminishes the quality and integrity of your survey results. That’s why it’s so important to limit survey responses to one unique response per participant. [source]
I also found that there are tools you can employ to prevent repeater trolls from messing up your game! More simplistic polls use cookies to prevent people from taking the survey multiple times. Some track IP address. There are more sophisticated methods on the market as well. None of these have been used by the NCTE’s survey.
GLOBAL ACCESS
The U.S. Trans Survey is intended to evaluate trans people’s experiences in the United States. Yet my (highly reliable) sources tell me that people outside the U.S. can not only access the survey, but depending on your “because I said so” answers, you can continue with the rest of the questions. Hey, you know how sometimes you can’t watch a video from the BBC in America? Well, the survey makers could have employed some country restrictions. But they didn’t.
NCTE isn’t using any of this technology to protect the integrity of their data. With over a million dollars pledged to the organization in 2013 (2014 report not available), I see no excuse for this failure except a blatant disregard for the accuracy of the data being collected.
BIAS IN, BIAS OUT
As the intended successor to the National Transgender Discrimination Survey, also known as Injustice At Every Turn, the survey is designed and targeted to prove the conclusion it has already come to: that “trans” people are in great danger and therefore in urgent need of political attention, sympathy, and access to public resources.
The name of the predecessor project alone is highly suggestive. Discrimination! Injustice! Its baldly stated purpose is to reach people who believe that they are or have been persecuted on the basis of being trans. This is not a random sample of the general population, or even of the trans population. The survey is specifically geared towards those who feel aggrieved or victimized because they self-I-dentify as as trans. These people’s experiences will necessarily be over-represented in the survey. This bias could be significantly mitigated by simply not telling people what the survey is about in the first place (and including more questions about everyday life).
The survey has also been marketed to people with a vested interest in the outcome of the survey. This is another form of bias inherent to the project: the targeted respondents are uniquely positioned to benefit from public policy changes and increased resource allocation that may flow from the survey’s findings. There is potential motivation, therefore, to answer questions in a particular way. Namely, the most persecutory way possible that maximizes the respondents’ perceived victimization.
The NCTE has a corresponding political agenda to prove its relevance, sustain current operations, and to solicit additional funding for expansion. From another email sent by Sandy James (emphasis in original):
Almost everything we know about trans people came from the National Transgender Discrimination Survey (NTDS). That ground-breaking study has provided the data the LGBT movement needed to help paint a clear picture of trans lives and show the world how deep our struggles really are. Because of that study, we’ve been able to transform our advocacy and fight for life-saving laws and policies.
The motives and goals of this project are crystal clear: to prove that trans people are the ultimate social victims. Both the creators and the participants have an interest in generating the most sensational results possible. And I have no doubt they will succeed at doing so.
GAUGING SUICIDALITY
On a positive note, I was pleased to see an improvement in the quality of the questions. There are more questions and more detailed questions this time. The section about suicidality is of particular interest given the irresponsible and hyperbolic reporting (<must read link) about “data” from Injustice at Every Turn. In that survey, there was only one question about suicide. In an independent analysis, the Williams Institute explains:
[T]he NTDS questionnaire included only a single item about suicidal behavior that asked, “Have you ever attempted suicide?” with dichotomized responses of Yes/No. Researchers have found that using this question alone in surveys can inflate the percentage of affirmative responses, since some respondents may use it to communicate self-harm behavior that is not a “suicide attempt,” such as seriously considering suicide, planning for suicide, or engaging in self-harm behavior without the intent to die (Bongiovi-Garcia et al., 2009). The National Comorbity Survey, a nationally representative survey, found that probing for intent to die through in-person interviews reduced the prevalence of lifetime suicide attempts from 4.6 percent to 2.7 percent of the adult sample (Kessler et al., 1999; Nock & Kessler, 2006).
By contrast, the U.S. Trans Survey features approximately 7 questions about suicidal thoughts and attempts, including whether professional after-care was sought (questions 16.1 through 16.10). The possible responses are still strictly limited to pre-populated choices, but it’s some kind of progress. There is also a section on emotional health (questions 12.2 and 12.3), which I hope will be analyzed in conjunction with information about self-harm.
Significantly, however, the survey does not ask if you been diagnosed with or self-identify as having a mental disorder. It is well known that people who who suffer from concurrent mental disorders, depression for example, are at much higher risk for fantasizing about or engaging in self-harm regardless of gender I-dentity. This is a very basic oversight that compromises the accuracy of the data and the inferences that can be legitimately drawn therefrom.
If you would like to see a(n almost) full copy of the questions, please download this pdf. USTransSurvey_questionnaire_2015
LEADING AND CONFUSING QUESTIONS
It is well established that a study’s target concepts should be clearly defined. As discussed above, even the words “trans” and “non-binary” are not defined by the U.S. Trans Survey. Additionally, questions about psychologists and counselors (section 13) are separated out from questions about health care providers (section 12). First, psychologists are medical practitioners. Full stop. Second, this doesn’t make sense to those of us who receive medical treatment, paid for by health insurance companies, from certified mental health therapists or counselors. In fact, trans people who want to medically transition require a mental disorder diagnosis (see “gender dysphoria” in the DSM-5) in order to access hormonal treatments from endocrinologists. So, should we leave answers about our experiences with psychologists and mental health counselors in section 12 or section 13? We don’t know! This confusion negatively impacts the accuracy of the data. It could be resolved by defining these concepts such that respondents will answer consistently.
Survey questions should not lead the respondent to answer in a particular way. Sandy James has made public statements about the complex design of the study:
James also pointed out that the 2015 survey “is the most dynamic and individualized of its kind. For example, if you report that you didn’t serve in the military, you won’t get questions pertaining to military service.”
This may be true about the military questions, but it is not true overall. The more politically controversial sections are extremely leading. The name-change section (questions numbered 10.1 through 10.18) asks the same series of questions multiple different ways, apparently expecting or seeking a pre-conceived response. This feels very much like leading the respondent to express a desire to change her name, even if she doesn’t want to. Same problem in the section about sex work:
Respondent feedback about the survey from an article on Advocate.com
Repeated questions can be found in many sections of the survey, including those about mistreatment by health care providers and what forms of medical intervention you are seeking for your trans identity (even if you made clear in the beginning of the survey that you identify as non-binary, not trans). The leading format of these questions is another way in which the survey uses flawed methodology to manipulates the responses it will receive.
GOOD DATA
Good public health data is based on verifiable health outcomes and can be repeated with newly generated samples. For example, the CDC uses in-person interviews and medical record reviews. Good discrimination data is based on actual complaints filed, as the EEOC has done here.
I fully understand that political groups have political agendas. I appreciate that the NCTE is sincerely interested in representing trans I-dentified people and advancing “social justice” on their behalf. That’s all well and good, but in light of the problems discussed above, the U.S. Trans Survey will artificially inflate the challenges– and therefore misrepresent the experiences of– people who identify as trans and their gender non-conforming brethren. I don’t know if this is a result of methodological negligence/ignorance or if it’s an intentional attempt at self-serving data collection. Either way, there are major weaknesses in the survey. I think the NCTE can do better by all of us– trans people and the general public alike.
We need long term studies about trans people. We need accurate and reliable data about trans people. We need to define what we mean by “trans” people in the first place. We need to validate that sample subjects are actually trans-itioning and in what ways. We need to prevent ballot box stuffing, international tampering, leading questions, confusing questions, and un/conscious bias. These problems are not strictly inevitable or unavoidable. There are well-known methodological controls that go a long way towards mitigating each of them.
The NCTE can “clean” the U.S. Trans Survey data all they want. Unfortunately, there is no way to correct a survey’s design flaws after the survey has been released. Garbage in, garbage out.
Sex matters. 
Reblogged this on GenderTrender.
I don’t understand this: “urogenital configuration (physical sex).” As a sexually dimorphic species, our physical sex is based on our gametes: sperm or ovum. Disclaimer: As I know you know but it seems it must always be said, disorders of sexual development (intersex conditions) exist and do not negate dimorphism.
Configuration sounds subject to manipulation, yet biological sex is immutable. It does not change. A reconfiguration of urogential appearance does not change someone’s sex, it only changes their appearance.
If you want a survey of people who identify as trans… then you ask them, and if they then identify themselves as trans… is that not all the evidence you need… that they identify as trans?
As for involving the rest of the population, well, my genitals are my *private* business. It’s none of their concern. Your desire to discriminate against us for our genitalia, is sexist and an infraction upon my fundamental human rights.
I used the “urogenital configuration” phrasing because I am not referring to reproductive capacity, but physical “plumbing.”
Plus, as you know, I’m ok with fully transitioned transwomen in the women’s bathroom. So yeah, they might have “reconfigured” their genitals.
You are correct, that genitals don’t *necessarily* equate to sex because sex immutable. But then again, AIS women and blah blah blah.
I didn’t say anything about DSD. I’ve covered them in other posts, I don’t think it’s relevant here.
Reblogged this on Stop Trans Chauvinism and commented:
As Elizabeth Hungerford comments, “The entire U.S. population is potentially impacted by legal and policy changes that cater to special interest groups. Revising bathroom and locker room access–from urogenital configuration (physical sex) to personal preference (gender identity)–is probably the best example of how policies designed to protect one group of people can inadvertently expose others to situations they find objectionable or even dangerous. As such, every single person in the U.S. has a stake in the conversation surrounding this project. It is every person’s business.”
The notion that a male should have access to women’s and girls’ private spaces (restrooms, showers, dressing rooms, etc.) was unthinkable even a few years ago. If these men in drag want to force society to allow them to change our long-standing culture of sex-based facilities, then the onus should be on them to PROVE that their desires don’t harm women and girls. Of course they cannot prove this – because it’s patently false. Instead, they use coercion, manipulation and bullying of females to achieve their goals. Anyone who enables them or colludes with them in this abusive behavior is absolutely unethical. Unfortunately, some of these unethical colluders are PhD’s and MD’s. Who says that being highly educated is any guarantee of basic human ethics? We’ve had PhD’s and MD’s who are rapists and murderers, after all. It’s time for women to PUSH BACK against this onslaught and to publicly shame any PhD or MD who participates in this coercive attempt to destroy our safety and privacy by misogynists in drag. Enough.
Ok, the survey questions pdf is now working!
AIS women are still technically male. They have testicles, they just never descended. They have a prostate gland. They have no uterus. Whether they have a vagina or how deep it goes varies from individual to individual. (They usually have one from what I understand, but it may not be functional in terms of PIV sex life. It’s definitely not functional for periods or childbearing.)
But we have to discern female from male by genitals most of the time since we can’t exactly run karyotypes on every stranger we meet that we’re curious about.
I’m not sure how I feel about a “fully transitioned” transwoman being in the women’s. I suppose I might be more OK with it than one who still has full twig and berries. It is not like they all prefer men sexually, and some “cis” men think it’d be hilarious to pose as trans just to get into a women’s bathroom. No, I am not kidding. At least if they’ve had the bottom surgery it tells me it’s not just a lark but they’re still considerably bigger and stronger than me on average, and come from that background of male socialization. Might be a problem, might not.
Bring on the third bathrooms already, and all the trans people and their handmaid-supporters can use THAT bathroom and leave the rest of us in peace.
and Angelica? I give a f?!k what someone identifies as. You people CLAIM that trans is real. OK, how do I identify a trans person? In no other instance of anything that’s supposed to be biological do I have to ASK the being in question who or what they are. If someone’s human, I can tell. If someone’s a cat, I can tell. OK, you’re trans. What does trans mean? How can someone even tell? Your definitions are constantly changing. It sounds more like a mental illness than a below-the-neck physical state.
It’s not good for children to have the rules constantly changing and frankly it’s not good for adults either. It’s abusive to impose that on people. Here’s a nutty idea: Stop being abusive, and put up or shut up.
I especially like how this survey speaks of people who “choose to identify as trans”. Oh I see, so you could have chosen differently, so we are arguing about bathrooms for nothing. But now someone would say “But it isn’t a choice!” See? No consistent rules.
Reblogged this on anywomans humanity.
[…] U.S. Trans Survey and methodological flaws, or “garbage in, garbage out” | Sex matters. […]
“urogenital configuration (physical sex).” …
@Elizabeth,
“Plus, as you know, I’m ok with fully transitioned transwomen in the women’s bathroom.”
The human species is sexually dimorphic and all primates reproduce sexually. No primate can change its sex. It’s getting really insane, and maybe we just need to nip it in the bud. Unless a person has an actual disorder of sexual development (trans is NOT the same as intersex), then he or she shouldn’t be able to legally change his or her sex. When it’s clear the individual is male or female with no actual intersex genetic condition, then he or she shouldn’t be able to legally change sex. Other than people with actual disorders of sexual development, no one should be able to legally change their sex. I think voters would support this because people are getting fed up. It just keeps getting crazier all the time. Many transwomen (males) are fathers, and FTM Thomas Beatie gave birth to three babies after legally changing her sex to male.
As to “fully transitioned transwomen” (males) using the women’s restroom, I tend to agree, but as more and more males “transition”, I foresee a lot of problems.
(1.) Douglas, “Donna”, Perry and Richard, “Sherri”, Masbruch are proof that any biological male can be seriously deranged and a sociopathic criminal and declare they are “women”. I don’t care if Dougla, “Donna”, Perry flew to Thailand to get his genitals surgically altered. He is a sociopathic career criminal, and I wouldn’t want him in the same restroom with me. A man can be bat crap crazy and a serial killer of women, and fly to Thailand to get his willy whacked off. Apprently, being sane is no prerequisite for sex reassignment surgery. Richard, “Sherri”, Masbruch tied two women up and sadistically tortured them with an electrical wire before raping them. I don’t care if he self-mutilated his own penis in prison and had to have it removed. He should never be allowed in a women’s restroom, and he doesn’t deserve the honor of being called a woman.
(2.) Sex reassignment surgery doesn’t actually change one’s sex. Even after SRS, unless he has had facial feminization, etc. he is still going to look male. Women notice these males, and I wish people would stop denying it. Male characteristics (facial bone structure, height, etc.) can’t entirely be surgically altered.
(3.) There is no reason why any male can’t use the men’s restroom.
“Revising bathroom and locker room access–from urogenital configuration (physical sex) to personal preference (gender identity)–is probably the best example of how policies designed to protect one group of people can inadvertently expose others to situations they find objectionable or even dangerous. As such, every single person in the U.S. has a stake in the conversation surrounding this project. It is every person’s business.”
I’m not sure men as a group mind transmen (women) in their locker rooms. And i suspect that is why such laws are passed. If men had something to lose … well, it probably would not happen.
It is closer to 60% of people having a stake in the conversation.
Sure, assuming males don’t care about females in their sex-segregated spaces, it’s less than every single person. But maybe you are the father of a young child? Or maybe you just care about women’s privacy and safety. If so, you might have a vicarious stake (as opposed to a personal stake).
[…] Continue reading… […]