How badly does canceling a credit card impact your FICO score?

How badly does canceling a credit card impact your FICO score?
Added (1). I will be be paying off all of my credit cards this month, but am concerned about closing them as I have been warned that it will severely impact my credit score. My wife and I will be looking to possibly get a new mortgage in a year or two, and I don't want to get a higher interest rate because of it. It does seem silly to keep paying annual dues on these cards though. I have about 6 different cards and was planning on canceling 4 and dropping the limit on the remaining two to $2000 each.
Added (2). The reason for the decrease is not to be tempted to run up a card, which I've tended to do from time to time (it forces me to focus on using cash only).

Read more: How badly does canceling a credit card impact your FICO score?