A technique for semantic search and retrieval that is event-based, wherein is event is composed of a sequence of observations that are user speech or physical actions. Using a first set of conversations, a machine learning model is trained against groupings of utterances therein to generate a speech act classifier. Observation sequences therein are organized into groupings of events and configured for subsequent event recognition. A set of second (unannotated) conversations are then received. The set of second conversations is evaluated using the speech act classifier and information retrieved from the event recognition to generate event-level metadata that comprises, for each utterance or physical action within an event, one or more associated tags. In response to a query, a search is performed against the metadata. Because the metadata is derived from event recognition, the search is performed against events learned from the set of first conversations. One or more conversation fragments that, from an event-based perspective, are semantically-relevant to the query, are returned.