More than 30% indicate bloc moving in wrong direction, with respondents citing ‘immigration control issues’