On the 2012 we found that there was a direct issue related to air pressures and ride stability. We had to find the sweet spot between front tire pressure, rear tire pressure and rear shock air pressure, all combined with the load we were carrying, and where the weight was on the bike (plus our combined weight). Once we had it dialed in by trial and error, the bike handled great and was very stable.
When you say "loaded up" make sure you are within the stated capacities of the bike...too much weight on the back (in the tourpak, on the luggage rack, in the saddle bags, plus rear seat passenger, etc.) can shift the balance of the bike to the rear and lighten the front end to the point where there is insufficient down-force to maintain good control. Shifting the heaviest loads to the lowest points on the bike, and setting rear tire and shocks for the upper max range of weight is about all you can do short of lightening the load.