You don't need worms or bait to fish and you can only add them to the better fishing rods. There's nowhere to put them on your starting bamboo rod. The bait makes the fish bite faster, but fishing without bait doesn't stop them biting.
There is not a shortcut key to get back to your farm. It's a time management game, so you have to run everywhere. It would be nice if they had an autorun key to make you run back to your farm though, just for the sake of my laziness.
You can unequip by clicking on an empty space, or pressing the shortcut key for an empty space. If your backpack is full then I tend to just click on the fishing rod. Occasionally I accidentally rod people in the face, but I do that in real life too.