Do you think that it's wise to pay off our mortgage?

Let me tell you some of the background info. My husband and I have been happily married for 14 years, and we have 3 children. We were in the military, and we both retired in 2010. In that time we sold our homes in Norfolk, and Chesapeake. Plus my home that I had in California, before we met.

We moved to Ohio (my husbands home state), and we bought a 4 bedroom, 2 1/2 bath, 2 car attached garage home for $64,000. We put at $30,000 down on our house, and we financed the rest through Navy Federal. I've recently inherited enough money to pay off the remaining $32,000 left on our mortgage, and still have some money left over. Navy Federal doesn't penalize you for paying off your mortgage early.

My husband thinks that we should pay off PART of our mortgage and invest the rest. I want to get out of debt for good. What do you think we should do?
Added (1). I meant Norfolk and Chesapeake, Virginia.
Added (2). Rob, we have ZERO credit card debt or lines of credit. We are retired/disabled vets. We both get pensions and VA disability checks every month. I'm a registered dietitian, and my husband is an accountant. We are employed full-time. I'm 42 (almost 43), and my husband is 44 years old.
Added (3). I also forgot to mention that we put 40% of our income from our full-time jobs into a Roth IRA. Plus we put $600.00 (split 3 ways) into our children's education accounts every month. We have a couple of investments, but they are slowly growing.
Added (4). Michelle, our current mortgage is $201.35 a month, but we pay $300.00. I just don't want to pay the bank more money than we have to. I'm tired of being in debt.

Read more: Do you think that it's wise to pay off our mortgage?