Canadians need to stop freaking out about the threat of United States annexation, Preston Manning says.