Create or cancel an alert