Although transgender women are acknowledged as a priority population for HIV prevention, there is little knowledge on men who have sex with transgender women (MSTGWs). MSTGWs challenge conventional sexual orientation categories in public health and HIV prevention research, and warrant increased attention from the public health community. This paper used qualitative techniques to analyze how MSTGWs describe their sexual orientation identities, and to explore the correspondence between men’s identities and sexual behaviors with transgender women. We conducted in-depth semi-structured individual interviews with 46 MSTGWs in San Francisco. We observed a diversity in the ways participants identified and explained their sexual orientation, and found no consistent patterns between how men described their sexual orientation identity versus their sexual behavior and attraction to transgender women. Findings from this qualitative study question the utility of category-based approaches to HIV prevention with MSTGWs and offer insights into developing HIV interventions for these men.