Reader Chris Connors is not entirely happy with interactions between the Calendar and Mail applications. He writes:

Although one might be tempted to simply trash Mail, try and do so and youll find that the OS prevents it. Like many of the applications bundled with Mountain Lion, this one is said to be required by the OS.

Another temptation would be to launch Mail, select Mail -> Preferences, and in the General tab choose a default email reader other than Mail. Regrettably, that doesnt stop the behavior your describe.

Surely, you say, there must be some simple knock it off command that puts a stop to this.

And Shirley replies, Yes, there is.

Launch the Calendar application, expose the Calendars pane by clicking the Calendars button at the top of the window, select the calendar that receives these invitationsa Google calendar, for exampleand press Command-I to produce the info sheet. Within this sheet enable the Ignore Alerts option and click OK. From this point forward Mail should no longer launch when an alert connected to this event goes off.