The ultimate fix to these situations is to take over the municipal government and ZBA by electing friendly people to the governing boards. If there are more working class, full-time residents than elites and summer only residents, then this should not be difficult. The problem is that these positions require real work, not just showing up on issues you personally think are important. If you want to actually run your town then you have to run your town all the time.
The law may be on your side now (IF the courts agree), but that only helps until the law is changed. If you want to control this you have to be the government.