A new study found that even creditworthy shoppers may be unable to find traditional mortgages. But the arrangements often lack typical consumer protections.