Ah, good question. I guess it comes down to your average engagement time. If your average elite engagement time is 5-10 seconds, I would think you are at the right difficulty for progression. Progression being, "where is my drop rate optimised", vs pushing being "how high can I actually go".
Now your engagement time is also also a function of your skills and play-style, so yeah, that can have an effect. Generally, your skills should be optimised to your gear. If you have a few legs or items that boost an ability or element, look to stack that ability and element for best results. If you are confident your skill selection is leveraging the bonuses from your gear, then you are all good. Set the difficulty for a short engagement time accordingly while farming. The rest comes from getting your rotations right. And, to be fair, that can be annoying. I really dislike some rotation playstyles and they don't gel with me. I am playing wiz this season and the free set is Firebird, but I just dislike the playstyle of that one for some reason.
Also damage mitigation is important - eventually. I talked about this in the Path of Exile thread, but that and D3's structure ultimately made me realise that to be successful in an ARPG you need a couple of things. DPS, damage mitigation and to optimise for clear speed in order to progress. We have talked about the first (sets and crit, baby, crit) and the third above. The second, damage mitigation, D3 tries to make important during progression by breaking skills up into categories - primary, secondary, defence, etc, as they unlock. Now, the first thing you will do is turn on advanced mode which lets you put any skill anywhere (and by proxy stack all the same skill 'type' on all slots). That can make it easy to forget you need to slot a damage mitigation skill or two.
Sometimes the best defence is a good offence (stupid DPS) and that can get you reasonably far in D3, but sooner or later, even though you one-shot whites - all of them will also one-shot you. Many items in D3 will amplify defensive capability by stupid amounts. Aquila Cuirass is a good example - 50% damage reduction when above 90% primary resources! Brilliant if your build hardly uses any spenders. There are dozens of items like this in D3, custom built to compliment the sets on offer and every leaderboard build is leveraging one or more to a certain extent. The effect of this is that sometimes, regardless of DPS, you will be stuck at mid torments until you get that damn item that will make the survivabillity difference. The trick is recognising that and spending all of your crafting mats on trying to acquire that item, which, to be fair, requires fore-knowledge of what is available to begin with - annoying for a casual player.
So, D3 progression is get to 70, craft and re-roll items to crit, cube complimentary items, acquire legendary gems, farm or complete journey for set pieces, re-roll them to crit, acquire complimentary legendary damage mitigators/DPS multipliers, re-roll those to crit, win.
Win is typically faceroll torment X. After that comes grinding for ancients and pushing GR's, of which some moderate amount is required to complete seasonal journeys.