Probably one of the hardest hurdles we face is our sexuality in the public space. Maybe you are gay or bisexual or straight, it doesn’t matter. That’s a separate issue. A womans sexuality is not called into question when she wears jeans. That’s because most women wear jeans today at some point or another, even mens jeans. But a guy in a skirt that is clearly not a kilt and dressed for daily life rather than specifically for the renaissance festival or something, is probably going to be viewed as gay or close to it. Maybe even a male prostitute.
It’s a sad fact we face and one I’m constantly aware of when out in public in mixed attire. Convincing the masses takes education and time. Even then it’s hard to teach old dogs new tricks. Look how many decades it took the LGBT community to gain some semblance of acceptance. Yet they are still fighting for equal rights and acceptance. Even then I would argue that only the LG part of that community has really gained ground. The B (bisexual) part is generally only accepted if you’re female. If you’re male and bisexual there is still a view that you must be a queer who just hasn’t come out of the closet yet. This mentality is even still in practice by some gays and lesbians. As for the T (transgender) there is a long way to go for acceptability there. At best most of them are seen as a fetish plaything by straight guys and couples who want to play with a chick with something extra. Ask them if they want to get into a relationship with a transperson and most would probably laugh. Sadly.
Crossdressers would probably most identify with the T section of the LGBT community. Being that there are many transgender people who start with all-out crossdressing as the oppoiste gender before proceeding onward toward hormone replace and perhaps even surgery. Whereas casual crossdressers are most analogous to the B section. Even to the same degree that female casual crossdressers are readily accepted like their bisexual female counterparts. But male casual crossdresser are seen (like their male bisexual counterparts) as just closeted gay guys.
Of course the one benefit we have as casual crossdressers over traditional crossdressers is that we aren’t as apparent many times. That really depends on what you’re wearing. But the fact that you probably aren’t in a wig and trying to act and/or speak all feminine is a good way to be more accepted. My theory is that your acceptance and unacceptance is less about what you’re wearing and more about your outward aura. Are you projecting yourself as a male or faking a female aura? Because people can tell. They aren’t stupid and when you are outwardly trying to fake being a female they’re going to see it and may well call you out on it.
Of course the best practice is just to be yourself. If you’re somebody with a deep equal masculine and feminine aspect to your personality then be that and always be that no matter what you are wearing. Don’t let your outward attire dictate your outward personality projection. Because doing that is fake and will be seen as just fetish. Nobody wants to see some guys fetish out in their daily lives. That’s creepy, not casual. Be casual.
As for how people judge your sexuality. Not much you can do there but correct them if they ask or assume. But don’t just blurt it out because that’s awkward and generally uncalled for. It’s going to take more public exposure of casual crossdressers more often before the general public will learn that sexuality isn’t linked to outward attire.