I see your point about growing everything in response to font-size settings, but I'm not sure I agree. It's not the same as reinventing zoom because of how zoom behaves across domains. Meaning, if a user needs to zoom in on one website then comes to mine, they will need to zoom in again. But by using rem values for most things, you don't need to do that.

I do agree that there are cases where it doesn't make sense to grow everything proportionately, but for a simple rule, I think it works. There are exceptions.