How sure are you that he can't see it? For me to grab my gun from a strong side hip holster, the coat has to be brushed back.
With pocket carry, your hand is just in your pocket. When drawing a firearm from a strong-side hip holster (if that is how you are carrying), your arm is in a pretty distinctive position that many people would recognize.
A lot of it comes down to the definition of assault in MA, and that appears to be part of common law, not statute. I'm not an attorney, so I'm in over my head at this point. I thought that I'd read at least one article by MA attorney suggesting that a simple threat could be assault, but I may be misremembering it. I'll try to find the article tonight.
From a tactical perspective, if he is that close it will likely start with grappling and you might be better served having your hands in front of you, rather than one hand on a gun which you can't use.