I think users want a "back" button that goes back to where they were before they visited this web page. The browser knows how to do this. It doesn't need to know anything about the app running on that web page.
If the app on that web page wants a button that undoes some state change on that web page, they can create one of their own. It seems like a bad idea to confuse these two very different actions. I don't think it's beyond the capacity of users to understand the distinction if it were actually made consistently.
Most pages these days are like web apps in that they do partial refreshes rather than navigating to a completely new page (with most of the same markup) as you click around. From user's perspective, this looks like a new page, and logically it is. They don't care about how it is actually achieved.
For example, on GitHub, when you're browsing some source repo, and you open a folder there, do you expect the Back button to go to the previous folder, or to leave GitHub altogether?
Well, I"m arguing that combining two functions in one button is a bad idea. I expect there to be a button to leave github altogether, and a different button to move back within github.
Browsers have never worked that way, though. If GitHub was written back in late 90s, say, every time you clicked on a folder in the file tree, you'd get a page refresh with a new URL - and then the Back button would just go back to the previous URL.
Now, they do the refresh by fetching data from a web service and updating the DOM. But the URLs still change - as they should, since you are accessing different resources as you browse - and so the history tracks it accordingly.
The only time when we had something like you describe was in early Ajax days, when DOM updates were already done, but before the history API, and before the hacks that preceded it were devised. In those days, any website that did it would behave exactly as you described - the Back button in the browser would navigate off the website, even if you were clicking around it for the past hours. And then the website offered its own Back button implemented in HTML, that would navigate within it. Users hated it, because it broke all established conventions.
If the app on that web page wants a button that undoes some state change on that web page, they can create one of their own. It seems like a bad idea to confuse these two very different actions. I don't think it's beyond the capacity of users to understand the distinction if it were actually made consistently.