I wouldn’t. Many of the characters are those you should have met before, including some of the most important characters in the game. You’ll lose much of the story’s impact by starting in Summerset.
My advice: Quickly find the boat to your faction’s starting island (Khenarthi’s Roost, Strok M’Kai or Bleakrock for AD, DC and EP, respectively). The stories start there. You don’t have to do all the sidequests if you want to rush a bit, but every zone has a main quest. These all fit together with the Cold Harbour questline, which is the game’s overall main quest. As you finish each zone, you’ll get breadcrumbs to the next. Just make sure to do any quest involving your faction leader and your primary guide (whom you meet on that island).