I'd suspect the fitting on the bike side of the attachment, because it's either that or a bad set of pins in the bag. After securing the pins on that day, did you look down behind the bag where it joins the bike? There should be no gap at all there where the pin comes thru to join the bike. If there's a gap, there's a problem.
At the very least, try swapping one pin from the "good" side to the "bad" side and see if that makes a difference. Or swap both pins. Try to notice if they feel different as you turn and lock the other pins in place.
Visually compare your pins from left to the right side pins or even to other bikes pins or to new ones at a dealer. Get a flashlight and check your connection points. Do your pins on that "bad" side click into place with a secure, tight snap? If not, you probably have a problem. Do they secure with a different feel than the other side does? Again, if so, you probably have a problem.
Sounds like another trip to the dealer may be in order if they didn't replace something on your first trip.