I realize it's hard to believe but these $25-$35K bikes are generally impulse buys. No one really needs them so we all spend a lot of time rationalizing why we pay this or do that. The dealer needs to sell the bike a lot more then you need to buy it but there WILL be another impulse buyer walking through the door shortly after you leave and CVO's do really stick to impulse buyers.
My personal view is always let someone else take the first bite and save several thousand on a lightly used bike that's 1-5 years old (usually with at least a few mods I would have made anyway that I now get for free). Driving a brand new vehicle is nice but it carries a HUGE price premium.
In your case, you definitely made the right decision. Be sure the dealer has your number. Feel free to show him a listing of similar bikes at other dealers. Let him know you are shopping and will be sure to purchase the thousands of dollars of add ons to the bike you get at his shop when you reach a deal for the bike at another dealer. Consider looking on Chopper Exchange, CraigsList, and ebay for the bike you want. There are tons of 2012's showing up and your color is bound to be out there. Lots of other impulse buyers jumped on board earlier in the year and now are regretting the purchase, found a new shiny object, or don't want to hear it from their spouse anymore so they're looking to dump their bike. It probably already has a new exhaust and a few $K in other goodies, some you might even like.
I enjoy the hunt as much as the score so our personalities may differ but if you look online you will find there is no shortage of low mileage, well maintained CVO's of all years, models, and colors.
Good luck with your hunt!