since 5.7x28 is inextricably linked to the FN five-seven, we cannot really say whether the market rejected the pistol or the cartridge. i would guess more the pistol due to price and weird ergos. Ruger has the following, influence and overall market share to dictate which cartridges get a chance. a good example is their SR1911 in 10mm. while 10mm was already undergoing a rebirth, ruger chambering a 1911 in 10mm pretty much sealed the deal that 10mm would survive for a while.
30 carbine is probably one of the most undervalued and sleeper cartridges around. if i were designing a blowback carbine i would want the option to build in 30 carbine. it is pleasant to shoot and sufficiently powerful to do some work. i would not be surprised if it makes a come-back. the ruger No 1 is an expensive fuddy rifle that will never be popular...it costs ruger nothing to chamber it in whatever cartridge so they keep it alive for their hardcore fudds. in general Ruger enjoys this level of following that i will never understand. there are so many people out their who love their products. most of us gun nerds are not ruger loyalists. it's the people who know just enough about firearms to be annoying....those types worship ruger and will buy any of their shit.
I think the public have rejected both the cartridge and the pistol. There's a huge difference between 10mm Auto and 5.7x28. What's the story of 10mm?
*Its a caliber that starts with a .4, so that's got the "stopping power" crowd happy
*Jeff Cooper developed the cartridge and the Bren Ten + Colt Delta Elite made 10mm an '80s thing
*10mm is the parent case for .40S&W and .40 is probably the third most popular handgun cartridge in the US currently
Now look at 5.7x28:
*0.224 bore diameter, so people obsessed with bore diameter rejected the 5.7 just because it doesn't fit their narrative of "muh stopping power"
*There's nothing nostalgic about the 5.7 - nostalgia sells, look at the US Milsurp market, the whole 1911 market, and the new Colt Python
*The 5.7's two main selling points are armor penetration and low recoil = accuracy, but the average CCW'er doesn't really consider armor penetration as a factor
*A lot of people will think "overpenetration" despite hollowpoints being on the market
Back when the 5.7x28 was still the new hotness, most people simply didn't want to touch it because of the cost and its a nonstandard caliber with a radically different approach to putting down targets. You can make an absolutely great pistol but if the caliber isn't acceptable to the public, the gun won't sell. But gun companies can make absolute turds of guns like Jennings and because they're in a known, accepted caliber (.22LR, the ACPs), people still bought them.
My bet is Ruger introduced the .30 Carbine Blackhawk back when people were still buying surplus M1 carbines and surplus ammo was still a thing. I'd love to play with the cartridge but my gun options are a single action revolver (not my thing) and the M1 Carbine, which is expensive and unreliable, or, not as reliable as an AR or AK.