I have a batch of Lapua brass that is on ~7 or 8 reloads. I have lost some due to damage attrition (someone steps on it, case mouth hits something and gets a big ding) but have yet to see one fail. I have some LC '67 NM brass that looks like I need to keep a close eye on it after the 3rd firing.
I don't anneal. I have seen some Federal cases split after multiple benchrest reloads including a very stout fire-forming load. Otherwise I'm losing more brass to misc attrition and doubt I'd recoup any expense on annealing.
I didn't like the X-dies. Too difficult for me to keep set up properly so that I didn't create 'donuts' and other sizing anomalies. They worked fine once set up, but if I changed brass manufacturer, I'd have to start all over again for adjustments. My Lapua, LC, and HXP brass needs only one trim at most and I'm getting plenty of reloads so no real need for the X-dies. Operator error, I'm sure, but I decided to just punt and use regular dies. Mostly I use RCBS competition die sets.
I think the key to long brass life is don't beat the crap out of it. Almost all of my rifle calibers are loaded to 2600 to 2700 fps. When I load something to 2900 fps and beyond, I can definitely see and measure much more movement of the brass.