What are you using for a mallet? If it is a steel hammer, it's knocking little flakes of metal off your stamps. Make sure to use a mallet made of the right materials to avoid damaging your tools. Any filing, sharpening, sanding, wire brushing or similar activities involving iron based material could spread fine bits of iron over various surfaces. Need to keep the leather, tools, and work surfaces clean. Keep anything rusty away.
Another source is iron in your water. Try distilled or deionized or reverse osmosis treated water. Do you have rust stains in your sink where your water drips?
Another is contaminants in the leather that reacted when dampened, but not likely as it should have shown up in the tanning and finishing processes.
Phosphoric acid is fairly strong, so could do more than you wish. Should be rinsed off and neutralized.
Tom