Will closing my credit card account effect my credit score negatively?

I've read several articles saying it will and it will not. Here's my situation. Several years back I opened a secure credit card account with capital one. I've been in good standing and never missed a payment. Now that my credit is in better shape I decided to apply for a rewards card. I figured why not benefit from using my credit card opposed to not. The limit on the current card I have is 750$. The new card I just was approved for has a limit of 5000$. I have a zero balance on the secure card so why not just keep it right? Well the secured card has an annual fee. Also upon cancelling I'll receive the initial money I invested into it back. I understand I'll reduce the amount of available credit I'll have, but seeing that I just increased it to 5000 I don't consider that a factor. Also I don't see myself using the secured card anymore due to the zero rewards so I fear that I may forget about it and possibly miss the annual fee payment which would hurt my credit. Any advice would be appreciated. Thanks in advance!

Read more: Will closing my credit card account effect my credit score negatively?