Researchers claim few beneficiaries would leave Britain, but advisers warn of damaging effect