Three mana is A LOT for this kind of effect. But it's kind of neat how the random element can hurt your opponent's ability to plan. There was an era where [[Pyrostatic Pillar]] was a popular card, and storm players would have to think things like: "I can cast 6 more cheap spells this game, I need to cast 9 spells in the same turn to win. I can afford to cast Ancestral any time, but I can only afford to cast this Mox if I KNOW I'm going to win this turn.". With Maddening Hex the storm math rapidly hits combinatorial explosion, e.g. "If I play a Mox and roll 2 damage, I can cast on average 5 more spells, which is enough to win, but if I roll 3 damage I can only cast on average 4 more spells, which is enough as long as I don't roll any 6s, but if I roll 4 on my initial damage roll, I'll have to stop going for it this turn, so I have a 33% of having an 80% chance of winning, an 18% chance of having a ....".
I'm not sure if a deck can afford to spend 1RR on this, but increasing the algorithmic complexity of your opponent's game state is an underrated tactic, and this card does a hell of a job of that.