Yes, that does happen. I've never tried buffing it off - it doesn't seem like it would work, but I dunno. I usually apply it after dying and before a finish. You will probably want to put some kind of finish on the raw veg-tan anyway. You could do a single layer of resolene, gives it a bit of a sheen without turning it too plastic-ey. Or you could use Aussie conditioner/wax. You can use a hair-dryer (or leave it in the sun) and the heat will help it melt in, after which it buffs up very nicely. Here's a picture of a bag I made for someone and left completely without a finish. I made it for a friend in town, so he brought it back for a spruce-up two years later. I put Aussie on it and it looks much, much better. I think people use Carnauba cream as well, but I've never tried it.