Cut the clip where you want to stop and then where you want to restart seeing the speaker. Right click on the section you want to keep the audio for. Select delete video. You can then fill the gap with a clip of video which has had the audio removed. (As for delete video but delete audio instead). You can then highlight the whole thing, right click and select link items.
I hope this makes sense - I know what I mean!!