Charke, Twisting and Pulling is exactly the situation I've got where the rivets are coming out. They are holding on leather attachment loops which dangle the bag from the mounting points on a bike saddle. I considered Chicago Screws as well but again, for the price point I'm working at they are too expensive. Although the biggest expense is having a bag fail because of a bad rivet, so the cost of using something like a Chicago Screw might be well worth the safety factor.
Busted, I must not have explained myself clearly. I'm using a steel Anvil on the bottom side of the rivet, the Latigo is what's being riveted.
I've got it like this: Rivet->Latigo Strap->Canvas(2 layers)->Latigo Strap->Rivet
I put one side of the rivet on my anvil, the other side I hit with the setter and my claw hammer.