Great topic. Lots of insightful answers in this thread.
The issue I found with multiple calibers is if you don't have ammo for your gun, then you have a paperweight. Ammo always has been a priority to me. I currently only have 4 calibers: 9, 223, x39, and 308.
If I absolutely had to, I would give up 308. This is due strictly to the cost and commonality of the caliber (and only 1 magazine type in my case).
Regarding diversification, I can see the benefit of having diverse rifles for barter use, but not for defense. Again, without ammo, it's a paperweight. Guns themselves are extremely valuable and if someone needs one badly enough, they'll pay for it and find the ammo elsewhere.