There are a couple people on this site that have first hand information about the hazards of riding.
Mikeytee came wheeling into Eureka Springs that day after his mishap and we were all shocked that he faired so well, so to speak. Unfortunately I think he was a bit too eager to share his butt raspberry with others... 
One that you may want to PM is RGlideKid about loss and getting back in the saddle.
As many have said, it's a personal choice. I think about it some times, but I try not to dwell on it. I do ride a bit more conservatively as I've gotten older. I also ride 2-up 95% of the time so I have to think about my beautiful BSR.
Coming from MI and having been in 9 or 10, lost count, car/deer mishaps, I tend to do wide road scanning and try to limit my dusk time and night time riding. As I ride for the fun of it, limiting my riding hours to sunlight is not an issue to me. I ride at night when needed, but if I have a choice, I won't.
Even though we're in a "metro" area, most of our riding it two lane roads and out in the boonies. Critters/deer are always a concern, but unlike riding defensively against caggers, animals don't follow the same rules.
I say go for it. If it makes you forget your daily problems and relaxes you, that's what it's all about.
This is pretty much where I am at on riding. When I was younger, 1 foot on the seat wheelies (slow or fast), taking every bike to redline/absolute top end, and taking risks like racing did not seem to be a big deal. The older I got, and the more friends who were dead from riding or hurt badly from riding accidents changed my perspective on a lot of stuff.
Scared to ride? Nope, love it long time. Calculate risks when riding to minimize them as best as possible? You bet. This would have made no sense to me when younger & more bulletproof, but yeah, there are times I sit it out instead of taking off into a situation which has too great a risk factor. Ride when it rains? You betcha. On a long trip with a need to get somewhere by some time, yep. But if it's really bad & I have the time & I can avoid a wet ride - you betcha. Motel rooms can be boring but living to ride another day is better than the alternative. That especially includes local rides to lunch and such. Bad viz, slick roads, cars that do not GAS, and a short stormy/wet ride to eat a local burger makes little sense to me these days. There can be exceptions, but not many.
As Dan says, on long trips I have a BSR to think about also. She is a sport & doesn't mind if I have a little fun in the twisties. On the other hand, if she gets too much "bobble head" time back there, my fun is lessened by her fun being lessened. Have taken it down a notch or two no doubt from a few years back. Too many "Mr. Toad's Wild Rides" when you hit a corner too hot and then find out there is gravel side to side as well.
Everybody should ride their own ride. More importantly, everyone should allow that everyone has their own ride & not try to push someone into something that is not for them, for whatever reason. I have quit riding with some who drank too much to be safe or just rode like hell and you never knew where they were going to be or what they were about to do on the road. Not many poker runs these days, have grown weary of the idiots who ride once a year but probably should just forget about it altogether.
Rides should be fun. And safe. Sounds simple, and it should be.