It took me several attempts but I finally did get him. You can see why I didn’t bother with the bonus objective.
The key is taking him out as fast as you can. I geared the Warden with as much DPS as I could and the ability that gives you another action point. I had the claymore with execute equipped since the boss starts next to several other low HP mobs I could use that on. I equipped the ninja in similar fashion including power attack and the free action point skill and ran her to the left to get the shard for the free action point the next turn. I gave her the axe and throwing knives.
I equipped a lot of dash and free movement skills on everybody that had them (get the one for the archer that gives a free move and attack). I gave the harmonica to the archer. I hit him with poison+piece+power counterattack and then ran her to the right to get the shard there and used the harmonica the call the mercenary for extra DPS.
It’s key to get spread out for when he teleports. Then I gave him the business with the warden and mage. The mage has the shield skill with a gem in it so he starts shielded. The warden also has that skill but starts unshielded. Shields allow you to survive boss retaliation if he goes into defensive mode. Once he gets below a certain HP he will teleport, even on the 1st turn.
You then have to get lucky. After a few tries, he teleported to where 3 of my group could reach him on the second turn and that allowed me to finish burning him down before he went into defensive mode. If that doesn’t happen you have to use your shields to survive defensive mode and hope you can reach him and finish him the following turn. The shards are helpful if they aren’t too much trouble to reach, otherwise I ignored them.
I was playing on normal so this might not work for you.