Build the box as a unit, then cut the lid off.
If you cut dadoes in the sides for your shelves to slide in, remember to mark top and bottom on the outside, or you may end up with your cuts wrong way around.
Ask me how I know this!
I once went to couples counseling alone, similar distrust issues. The guy gave me the best advice I've ever gotten.
He told me "It's over, move on, you're a hot property, get back out there."
I did, I was, best thing I ever did.
But I'm me, you are you. Good luck to you.