Personally, I feel that reviewing is useful for me, as it gives me an opportunity to read some new papers on topics that I don't necessarily follow very closely, which helps me keep up with the literature and broadens my understanding of the field as a whole. I would go as far as saying that I actually like reviewing (and not necessarily for the illusion of authority that often comes with it).
However, I think there is some great asymmetry in our academic community; authors spend months or even years on a paper, which is often judged by reviewers in a few days or -at times- in a few hours. And quite often the reviewers' confidence in their expertise leads them to seemingly irrefutable statements which are based on limited familiarity, as opposed to months of hard work by the authors.
What is even worse however is when reviewers don't even make an effort. I have received good and bad reviews (where "good" and "bad" are not with regard to whether the reviewers liked my paper or not) and I have encountered good and bad reviews as a PC member as well. Here are quick examples.
Once, I received a review that was a single dot. Obviously, the system did not allow for an empty review. Another time, I received a review that "was based on a previous version of the paper". The annoying thing was that we had spent significant effort to update our paper based on the reviews that we received in the previous version, resulting in a much different paper. I contacted the chairs (not to dispute the outcome but to point out the lack of professionalism); needless to say I never received a reply.
These things happened while I was a PhD student and these were things that my then-idealistic mind could not comprehend. I thought that I should bring them to the attention of more senior people. But I could read their faces while I was narrating my stories: "Another clueless PhD student that is complaining about his paper being rejected". I don't blame them, they've heard the stories before. But that wasn't the case. That was never the case.
And here I come back to the asymmetry. In pretty much any "real" job, you are evaluated by some mechanism that is there for that reason, whether it is your boss or your supervisor, or some authority appointed to evaluate your performance etc. In academia, your future lies in the hands of a few people that voluntarily spend a little of their time to evaluate your work and their assessment might be the deciding factor for whether you get your next job or not.
