Alright, I have 2 things for you to try:
Follow this link to find info on updating your controller firmaware: Regular crashing (xbox one controller) - #9 by Silas-Inservio-Pax
And a second possible fix is to share your Realm, invite someone. Let that invitee enter and walk around. Then try to join yourself.
Let me know the results!
Silas