I sand 99% on the long axis of the edge. (with the grain) The only time I would go on the short axis (across the grain) is in rounding a corner, after which I hand sand again going with the grain.
Going back and forth with the grain has a tendency to raise as many fibers as it lays down. I sand (or stroke) in one direction only, which raises less fibers.
I also rough sand and finish with a finer grit, usually on a dremel tool at low speed with minimal pressure. You will likely find that stroking in one direction will work better than single stroking in the opposite direction also.
If you think about petting your dog, for example, stroking along his back from head to tail is a smoother experience than starting at the tail and stroking toward his head.
Both are long axis, but only one is really 'with' the grain.