As the replies thus far ably illustrate, this "either/or" question runs aground on the fact that the M1 and the M4 are different tools, adapted to different missions. This leads to two conclusions:
First, as a number of folks have opined, putting aside cost and availability issues, you might consider getting both. Owning a screwdriver in no way negates or diminishes the value of also owning a hammer. If you're into tools.
Second, also as a number of folks have opined, availability of the two rifles is vastly different. M1s, particularly good shooter M1s, are much harder to find, so if you find one, there is merit on this basis alone for seizing the opportunity.
As for felt recoil, this is also undeniably a subjective issue, on which one person's opinion may have little or no value to how you might feel. For myself (not a huge fellow by any means), the recoil of an M1 is quite tolerable (far more so than O3s and other '06 bolt guns, which I also shoot).
Store-bought '06 ammo certainly is more expensive than store-bought 5.56 ammo, and if you are limited to store-bought, you have the added complication that not all factory '06 ammo is usable in an M1. (Do some research on the issue of "M1 port pressure" or study
https://www.zediker.com/downloads/m14.html.) Likewise, some of the from time to time offered '06 military surplus ammo is corrosive, and this you should avoid at all costs.
On the other hand, if you reload then the cost differential between the two rounds is manageable. My M1 round is a 165 grain SGK pushed by 46 grains of Win748, both of which were until recently both readily available and tolerably priced. My 5.56 load is a 55 grain M193 slug (FMJ BT), preferably Winchester, pushed by 23 grains of Win748. Before recent perturbations in the components market, I used to use $0.15/round as an estimate of the cost of the 5.56 reloads and $0.25 for the M1.
Perhaps the biggest difference between the two rifles in question is their amenability to optical sights: easy with the M4 and (in my opinion) impossible with the M1. If you feel you need a scope, the choice is clear. On the other hand, the M1 shares with the M14 perhaps the best iron sights ever found on a military rifle, and even with my high mileage eyesight, cleaning the reduced L target at 200 yards (prone) isn't that hard.
Hope the foregoing is helpful to the OP.