The reason Google was doing this was to learn my speech habits to convert it to text more accurately. It also said I was the only person who could access my archive. But it kinda creeped me out.
Fortunately it gave me a way to shut it all down. Here's how:
In a web browser sign in to your Google account. Find "Personal info and privacy". Select "Manage your Google Activity."
Click on "Go to activity controls."
Scroll down to "Voice and audio activity." Turn the slide switch off ("paused"). This turns off any future recording but does NOT delete any existing recordings.
Click on "Manage activity." You'll then see a list of all your speech to text activity ever. You can even play back every word you spoke.
In the left column click "Delete activity by". A pop-up will appear to delete your recordings by date. Under "Delete by date" click the drop-down menu and select "All time". Then click "Delete." All of your archived recordings will be deleted.