Customizer integration is very deep. I don’t think it’s possible to separate it from the core without ruining legacy and general architecture. We can’t remove it. (To my opinion). And since we have to support Customizer anyway, we have to consider related content structure and UX. If we try to ignore this functionality, we’ll have to run two parallel systems of theme customization. Duplication is very expensive (it multiplies all development costs: time, complexity, support, documentation, testing). And it is quite unsafe on a long distance, because it brings chaos which will progressively increase each time we add a new feature.
Yes, I agree that Customizer is hell-alike. It’s slow, it has poor usability and it’s absolutely senseless when you build projects with only 1 exclusive theme per site. But it’s a kinda build-in standard, so I think we have to use its api just in order to stay consistent.