You said it yourself. Some indigenous cultures wear little to nothing. Why is there no epidemic of rape there?
There are many many studies.
eprints.soton.ac.uk
Google for more.
This is a disappointing argument. That is like asking why brothers and sisters are not attracted to each other romantically yet they live in the same house.
It is basic conditioning and this case you have mentioned, it is cultural conditioning. For example, why would anyone find a partner more attractive once they get their lower teeth removed or the female is circumcised?
Similarly, if your default state is being naked, why would your brain be stimulated to find that irresistable?
Why must a couple continue keeping it fresh in the bedroom to keep their sexual life active after years of living together?
It really is that simple.