I may have found the solution - insofar as I haven’t had a crash in the last 4-5 hours of play or so. I’ll update this if the issue comes back up, but the solution for this is bizarre - you update your Xbox controller firmware.
I stumbled across this Reddit thread
The thread references using an Xbox Series X/S Controller, but I have an Xbox One controller and the steps to resolve things seemed to have worked:
Download the Xbox Accessories app from the Microsoft/Windows Store
Connect the controller
Update the firmware
Weirdly, this has stopped the crashes so far, but I’ll update if that changes.