I got the replacement speaker today and I changed it out. Boom! the rattle is gone!!
It sounds so much better now. No air leak issues either.
BTW, I think the crossover needs to be set pretty low otherwise the bass sounds a bit muddy. I am using 24db per octave crossover and it is working pretty well.
Using my speaker level input. I am max on the plate amp volume and the cross over point is probably around 80 hz.