This is certainly technically viable. However, the evils "bandwidth" and "distance" make it much more of a challenge. Full motion video streams take up (probably not the right words) considerable bandwidth continuously. The longer the distance the more power is needed to get the picture and commands to/from the aircraft from the operator. Given the ever increasing electronic warfare capabilities of all belligerents this could become problematic. The U.S. has begun moving away from full motion video data.
I have no doubt there any number of technical companies and scholastic institutions working to overcome this challenge, but to my knowledge they have not achieved totally viable results yet.
My first assumption would be to limit the total sent video to just that of the current view, plus however far to the sides of the current view signal lag takes.
Still a lot of video to deal with, but that cuts it down from 360x360 to probably 40x40 or maybe 60x60 if the lag is bad (assuming a ~20x20 degree field of view in the HMD).
I also remember hearing about some weird uses of COTS graphics cards, running them in reverse with a video signal input to create an easily compressible and encryptable signal that was used for secure video conferencing.
Some would argue we are getting to the point where unmanned platforms are capable of receiving data packet commands and executing the mission without further human intervention. This may, or may not, remain viable in the age of tactical quantum computing linked to AI.
That was demonstrated in a ground attack by LockMart
20 years ago when I was still in the Navy. And I believe in 2020ish in a dogfight.
My goal for AI wingman is to limit things to a small number of commands, maybe a dozen: "Weapons tight/free," "attack my target," "attack what I am not attacking," "attack this (ground) target," "defend me," "defend other," "destroy everything in this area," "reform on me," "RTB (without me)," maybe a couple more...
"Attack this ground target" comes with an assumed command of "deal with any defenses around it and targeting you as needed," and the single or group of CCAs then decide who is hitting what targets on their own.