It may be because I’m pretty hyper-sensitive to the bitterness in beer, but I find it makes an enormous difference, personally. To the point that some beers in some recipes can ruin things. I made beer battered fish with a strong IPA I had lying around after a party once and found the flavor to be miserable.
In something extremely intense already, like chili, it can be more subtle. In that case, you’re bigger benefit comes from “unlocking” extra flavor compounds in the tomatoes that are alcohol-soluble, but still, the added richness/depth from a heavy stout can be a nice background flavor in the same way that cracking a few ounces of intensely dark chocolate into the chili can help.
But in something like beer bread or especially beer batter? I find the flavors come through noticeably, though, again, I’m unusually sensitive to especially the bitterness from hops.