That's the rub isn't it? Anytime you pick a number, there are going to be people that both fall below and above it. I know a few extremely responsible 18 year olds and a few extremely irresponsible 40 year olds. But in general, the higher you set the age, the more people that can reach the bar.
Right now an 18 year old can vote, smoke and fight. None of those things is going to increase an 18 year olds maturity level by the time they hit 21. The freedoms we do give them (get a job, be subject to adult legal sanctions, etc) will increase their maturity. But letting 18 year olds drink legally or carry handguns is a bad idea. Teenage brains and not wired well and many teenagers do not do stupid things because they are intrinsically stupid, it's just a matter of flaky brain development and hormones. They get into their mid-20's and do many fewer stupid things. Frankly, I don't even think they should be allowed in the military until they are 21. But then you lose a lot of your combat force... 18 year olds are kids, regardless of whether they can go to adult jail or not...
