I don't have an Ultra, but I did add a similar speaker switch to my SESG2. Doesn't the CVO Ultra come with a speaker/cruise control switch?
Any issues with intermittent cruise operation?
My switch is the "Street Glide" version; bad-azz and bare-bones - hahahaha.

And cheap ... never forget cheap!

(I think a "standard" speaker toggle switch is ~$30?)
Anyway...
In general, if toggling a switch causes the problem to go away, then the switch is causing the issue. I would suspect a dirty contact within the switch. If you're a DIY guy then I would suggest pulling the speaker switch, spraying it with contact cleaner, then blowing the residual contact cleaner out with "mild-compressed" air. Don't get crazy with it - 50-60lbs should be enough - just enough to "blow-out" any dirt and residual contact cleaner. Let the switch dry fully then re-install.
Any more info on history of the bikes? Stored indoors, stored outdoors? Mileage? Humid/Desert environment?
Just offering a possible solution to replacing a ~$100 switch.
Might be something worth considering.
Hope this helps ... be interested in final fix action.