I use pre-dyed chrome-tanned leather almost exclusively, so as you assumed, I have never had this issue when using tokonole. I use a swab, careful not over-saturate it, and apply 2-3 thin coats to my edges. Fortunately, I've never had any end up on the surface of the leather.
Below is what the swabs look like. They are used for cleaning printers and lenses, so there are no fibers to leave behind.