Former congressman said the controversy around his nomination had become a ‘distraction’