Security worries on the ground in Switzerland, not fear of a torture complaint over waterboarding, led organizers to cancel former President George W. Bush's speech at a gala this week in a luxurious Swiss hotel, the Geneva based director of an Israeli charity said Monday.