I have not worked on Colt revolvers, but I am very familiar with the internal workings of S&W revolvers. Misfires in double action, but reliable ignition in single action, generally indicates a lack of mainspring tension. The hammer does not come quite as far back in double action as it does when it is cocked for single action. So the hammer always has less striking force in double action.
However, your statement that pulling the trigger faster makes it more reliable is a bit unusual. Most of the time, the speed of trigger pull would not make a difference. So you might want to take a close look at the timing and lock up of the revolver as well. I am wondering if the cylinder is not turning quite far enough when the trigger is pulled slowly, so the firing pin strike might be slightly off. A fast trigger pull could move the cylinder a little farther due to momentum. On a Colt revolver in perfect tune, the lockup is rock solid, and speed of trigger pull could not affect the position of the firing pin strike, but everything has tolerances and can wear, so it still might be worth a look.
Also, I sometimes hear people say "I replaced the mainspring, and now it works 100%, so that was the problem." This statement makes sense, but it is not correct. If things are a little out of tolerance, like the firing pin strike is a little out of position, or there is a little bit of friction on the hammer, or the headspace is a little large, then a stronger mainspring can overcome the problem. But the stronger mainspring is not necessarily the correct or only solution.
But with all of that said, based on the misfires happening only in double action, the mainspring does seem like the most likely issue. On S&W revolvers, I see this problem occasionally.