Sounds like no one has been hurt, but the local 'officials' are forbidden from sticking their noses in. How about this, if someone is reckless (check the downrange area for impacts), or if it violates a noise statute (50BMG at 2AM isn't very nice) - deal with it that way. Otherwise, SCREW !!!
I have a range, I only shoot when no neighbors are out enjoying their yards for quiet time or having a party. Never a complaint, never a problem, I just let the local PD know when I'll be shooting so if they get a call they can tell them it's not an issue.