Elemental defense is definitely important in some fights. Having 30 points in the appropriate element can be the difference between a nasty AOE attack taking 80% of your HP in damage vs. 40%. The former means you have to stay at full health all the time, and even so might still die if stunned. The latter leaves way more room for error. Likewise the status effect immunities from resist skills can be a huge deal (just having 1-2 levels of a status effect resist I find mostly useless though; it’s either 0 or 3).
But… That’s assuming that you’re not giving up significant amounts of physical defense when optimizing for the elemental defense. Let’s say you’re fighting an Anjanath early: it’s much better to have 50 defense / 0 fire than 30 defense / 20 fire. You’re going to get hit by a normal attack a lot more than an elemental one.
The elemental damage might or might not be a red herring, it depends totally on your weapon. The thing to understand about the damage formula is that the physical damage and elemental damage numbers shown in the weapon information are at totally different scales and have different modifiers applied to them.
The basic idea is that for elemental/ailment attacks the only thing that matters is the number of hits you get; the weapon type, the attack type, sharpness, etc don’t matter. So a hammer user should totally ignore elemental damage (and maybe even avoid having an elemental weapon in the first place, so that they can slot in a non-elemental boost decoration). While for a dual blade user elemental damage is a big contributor to the total DPS.