I bought some small pieces of thin sheet stock at the hardware store and then used a shears to cut out some small washer-like shapes (I drilled the holes before cutting). There's a locating pin behind the tensioner and I hung different thicknesses of the sheet-stock washers on the pin until I found the right one that made the tensioner properly aligned (with the belt on). | 
