JP... there is no "easing" when it comes to leather. You need to make sure you have the same number of holes on both the crown and brim... etc. Count, count, count!!
Do you have a dremel? check that for small drill bits. Also a finish nail will work to pre punch holes.
Here's a tutorial for making a leather top hat.... perhaps some of these tips will help you.