Have to agree on the theory of it being primal. I felt this fascination before I reached sexual maturity. I had discovered masturbation before this even, well ahead of puberty even if I didn't know what it was called.
The first memories I have of getting excited via some kind of conflict were from my Nintendo and Gameboy. There was some old wrestling videogame on the Gameboy that there was a count to ten for the wrestlers outside of the ring. I purposely made them fight each other until one was counted out and the other "survived". This was not a visual fetish, this was not being attracted to any sexual identifiers. It was a fascination with two rivals desperately trying to control and subdue each other, the anxiety of a clock ticking away the seconds we had left driving my imagination crazy.
The other, was a battle chess game of some kind. Regular chess pieces were represented by magical and medieval creatures, my memory is more spotty on this one. But there was some kind of amoeba or slime creature with dreadlocks that ate enemy pieces in a short cutscene. I had no idea how to play chess, I couldn't even read at this point in early life. But again, it came down to a fascination with a desperate, survival based competition triggering something in my lizard brain.
Nothing about either experience was based on visuals or ideas of physical attraction. The allure was solely in my mind. I didn't know about sex, fetishes or anything like that. However the first time I saw a catfight via some soap opera, that was it. I knew exactly what I was looking for, it had a name. But before any of this, it was just a nameless obsession based on conflict.