Heavy armor, while not optimal, is totally viable. You’ll want to put some points into Stamina and Health that would normally be used for magicka in a mage, so your pool will be smaller. (My 1st character, pure mage, put nearly every point into magic, none at all in Stamina, and I was a glass cannon, wrecking enemies but dying fast if I screwed up.)
You’ll also have to spend perks on armor, which are perks a pure mage might use on another school of magic. And you’ll sacrifice double-strength stoneflesh-type spells, which make a cloth wearer extremely well armored for short timespans. (Of course, the best piece of gear for casting, the Dragon Priest Mask Morokei, is listed as armor, so I couldn’t make use of that perk on my pure mage anyway.)
From my experience, Destruction is quite strong enough to be your primary means of damage dealing, in spite of what others have said. My pure mage never touched a weapon (except when quest-required). It’s a bit tricky, definitely recommended for advanced players, but there was no time except the very early levels where I had much more trouble killing enemies than with my warrior. You have to use all the tools in the box, not just fling fireballs. Cloak spells, runes, and wall spells strategically placed are vital. For the later levels, it’s essential you do the quests to get the Master-level spells in the schools you use, but most essential for destruction. The knockback perk I highly recommend, too. Interrupting a great wopping dragon before he rains firey death can be the difference.
You won’t get the magic regen buffs of robes, so you’ll want to make that your primary enchantment or bring a lot of magic potions with you. Magic regeneration is, I think, the single most important factor for someone using Destruction as a focus.
tl;dr Yeah, go for it. It might be a bit more challenging, but that’s half the fun!