Co owned property and gifted money?

I invested in a second home with a family member. I know, dumb! He came to us with the proposal. He said he would cover the down payment, and leave his half in a trust for my children. His credit was not good enough to secure the loan or be on the loan at all, bank required him to gift the money to us (13k). We have the gift letter. My wife and I added him to the deed thru quit claim. Things have since degraded and he wants out, and is trying to get us to pay him the 13,000 plus back and force us to sell. We had to come up with 6500 for the down payment as his 10 percent was not enough, 2nd home loans require 15 percent. There is not enough equity in the house to recoup the investments even if we did sell (we don't want to). I did not get a contract. Can I be made to pay back the gift? Can I be made to sell?

Read more: Co owned property and gifted money?