Sounds like the needle bar has some wear on it. Leave the power off and turn the machine by hand. Place a piece of (clean meaning no holes) scrap under the needle leave the pressure foot down run the needle bar down with the hand wheel do this while keeping pressure on the needle bar with your left finger, thumb, hand what ever you want to use. Hold pressure to the left and change directions put pressure to the right. See how far you can move the needle left and right doing this. It won't take long to see what is loose or worn. It might not be this but is worth a try.