In my trip to nippon there were seasoned men in anime book and toy shops, jj is right its the norm there. No one thinks anything of it, its just a common interest.
Do not go into any basement shops with subtle signage. There is definitely questionable underground culture there.
I had completed my study and research and was in the process of writing my doctoral thesis on quantum physics.