Why not try using a piece of natural sponge? You are obviously having a chemical reaction with the synthetic sponge, which is petroleum based in its manufacture.
For one color dying I dip dye now. Airbrush works good too, but is a little too light an application for my tastes.
Fiebing's Pro Oil dyes applies pretty good with a dauber. Several successive applications (while still wet) help even out the color. Remember to let it dry completely before you fiddle with it. Wet leather always looks darker, I remind myself all the time to be more patient when dying, so that I see the real, dried color first, before I add another coat. Pro Oil Saddle Tan is terrible dark when first applied, but lightens up a lot after drying. I just used it for the first time the other day, and really like that color.
For water based dyes I have been using a Turtlewax yellow car sponge, or a no-name equivalent which is half the price. I keep a piece for each color in a plastic lidded container so they are always just a tad damp. So far no mold after 3 or 4 months. I rinse them good before I put them away.