The only thing I do is uncheck the year/month option under Settings > Media. So no plugins and nothing breaks.
However, for more advanced functionality like taxonomy-type folders (not actual directories on the server) I use HappyFiles. It also allows uploading and sanitizing SVGs 
I remember reading in an email developer is doing a complete rewrite in VueJS. Wouldn’t it be nice if the media library used VueJS instead of jQuery? I would call that a definite improvement.