Why should there be a law that requires non-hunters to spend money on clothing to use public lands?
Sorry, but the easier answer, if "Moar Lawrs" are needed is to say, "No Hunting," as there are likely more non-hunters than hunters on the land in question, and, therefore this is a less-onerous-to-the-most-people solution. Putting responsibility on person A to avoid an "accident" caused by person B is how we get the crappy laws we now have. Remember this, when some criminal does a criminal act, and the law-abiding are saddled with more legal burdens.
That said, I'd wear orange in areas were hunting is going on. Because it's a good idea. Same with seat belts. When I cross the line to NH, I take off my seat belt as it's no longer required....I then put it on again, as all the cars around me are still Ma**h*** drivers.