Improved Media Library

[discobot]

I suggest an improved media library, because this is a feature of WP which has been strongly neglected in the past. Whenever someone came up with a new proposal, most of it went under, unheard or was post-poned till somewhere in the far-off future.

At least having some option to display files and directories in a classic tree view would be great. Oh, and a slightly more welcoming Media API.

Both features would be very welcome. The former one should sooth newcomers, who are used to having a “regular” overview of their media, while the latter should rush forward plugin development.

Update (2018-12-18)

• Discussion on the forums at Petititon: Improved Media Library
• Repository created for these improvements to start as a plugin: https://github.com/ClassicPress-research/improved-media-library

This petition was posted by Fabian Wolf.

It had 95 votes.

The status was open.

It had the following tags:

• difficulty-moderate
• request-modify-feature
• cp-research-plugin
• request-add-feature
• difficulty-hard

Read-only archive: https://petitions.classicpress.net/posts/51/improved-media-library

Comments:

In my mind, the media library has been WP’s biggest weakness, so I agree 100%.

~ posted by KTS915

Yup, this has my vote too.

~ posted by Pieter Bos

Yes please. Currently building a site where we are migrating over 1000 images and the WordPress media library is next to useless at things like: finding or preventing duplicate image uploads, easily organising images etc

~ posted by Ashley Goodman

Yes please! Media handling in WP was always problematic and here could ClassicPress really stand out. I mean getting rid of GB is just fine. Removing emojis, etc is welcome, but not a feature an average user would understand and would change their CMS for. Some idea for improving the media library:

• Allow to organize items in folders and in categories
• Possibility to save different sized images in different subfolders
• Allow to insert facebook-like photo albums as well (where you click on a thumbnail to see all the images in the album).
• Offer the possibility to have RANDOMLY generated pre- or suffixes at least for the original file (that would be some kind of copy protection). Right now it is really easy to figure out the valid url to the original image and just steal that.
• Add more metafields to media files (Author, copyright holder, etc., take a look what credit tracker plugin does)
• When inserting an image or gallery let people choose what they want to be shown in the caption (i mean letting them override the content of the description metafield with e.g. a combination of year taken and author).

~ posted by Peter Bankuti

Do any plugins exist that accomplish what people are suggesting and/or for inspiration if this moved forward?

~ posted by Dustin Snider

In WP one would need to install a bunch of plugins and even in that case one can not achieve all the suggestions above. Plugins that are worth to check their feature sets:
https://wordpress.org/plugins/credit-tracker/
adds some fields to each media (e.g. copyright or author and one can override image captions with these fields. For pictures bought from larger agencies like Getty images it actually can pull these data automatically. Great if you need mention the photographer)

https://wordpress.org/plugins/enhanced-media-library/
https://wordpress.org/plugins/media-library-assistant/
(These let you add taxonomies, tags and folders among other nice features)

https://wordpress.org/plugins/thumbnail-cleaner/
(great if you change your theme and want to get rid of such image sizes that remain existing on the server but are not part of the new theme, so they just need storage for nothing. Only the original sized files will remain on the server)

https://wordpress.org/plugins/regenerate-thumbnails/
(after using thumbnail-cleaner you just need to regenerate all the sizes that are part of your active theme)

https://wordpress.org/plugins/simple-image-sizes/
(Lets you register custom image sizes in WP without writing code)

https://wordpress.org/plugins/crop-thumbnails/
(In case of hard cropped image sizes you can recrop the given size based on the original image. This works wonders! Imagine a huge original image which looks just right but if your theme renders a special size that just does not look right with WPs automatic cropping you can change that)

https://wordpress.org/plugins/wp-content-copy-protection/
(disables right-click, F12 key, and code inspector in chrome, etc.)

Still no plugins cover:

• pre- or suffixing image URLs with random characters, so if one finds out any image url they can extremely easily go and grab your original image file… that will drive you crazy if you are a photographer for example)

• Updating EXIF/IPTC/XMP metadata in the image files on the server (imagine a photographer uploaded the images without metadata and later he wants to protect their image by adding copyright person to the files…just in case someone would use it without permission. In WP right now you can change metadata just in the database but you can not burn new metadata in the files themselves. And if the server uses GD library, the resized image files simply will not contain any metadata. Again great for those who plan to use images without asking the copyright holder).

So to sum it up, there are LOTS to be done here and i am sure i missed some nice features

~ posted by Peter Bankuti

Has my vote, but any changes here are likely to be pretty difficult. We’ll also need to break this up into small, actionable items in order to move forward.

~ posted by James Nylen

The current media library was one of the things that pushed me off WordPress ~2 years ago.

• there are countless hooks which are not supported on both the backend and the frontend;
• there is no built-in way to keep all metadata in thumbnails (some photographers would like this very much)
• no built-in watermarking
• media library originals should be kept away (directory level) from resized images and should be possible to block traffic access to originals
• srcsets and responsive images are a mess

I was never able to tackle all these. It would be great to see improvements on the media library. I’m willing to help, I have a couple of plugins that were helping me with these:

• https://github.com/petermolnar/wp-resized2cache
• https://github.com/petermolnar/wp-featured2rss
• https://github.com/petermolnar/petermolnar.eu/blob/822c4d540a1629e6b48cfa78794b371e705f2d27/classes/image.php

~ posted by Peter Molnar

When we get to this, is there any way to limit the number of files produced? Is it possible to do more of the resizing on the fly instead of making five copies of every image? The uploads folder is often the vast majority of a site’s overall size.

~ posted by Glenn Dixon

I’d like to add a couple of things I’ve dealt with the last ten years in more or less every client website we’ve built:

• The ability to change filename from the Media Library would be nice.
• Filename sanitizing (building websites mostly for Swedish/Scandinavian content this might be a much bigger problem for me than a UK or US developer). Change special chars åäöü etc. to aaou, replace spaces with dashes or underscores and use lowercases (seen problems related to this when switching server environment). I think some basic sanitizing should be built into core (maybe it is? In that case I just haven’t seen it work properly, like ever). Some might be optional/add-ons on Settings page.
• Media replacement. I don’t think you should need a plugin to replace uploaded media, it’s basic functionality in other CMS:es that I’ve worked with (see https://wordpress.org/plugins/enable-media-replace/)

~ posted by Erik

Current WP Media Library folder structure by year/month is totally obsolete and fits only for small blogs and doesn’t make any sense otherwise, especially if you have 100+ files and your site is 5+ years old.
Folder structure on server should be somehow customizable at least with categories or by user name. Plus all ideas that have been listed above should be considered in order to make this tool really useful for commercial sites.

~ posted by John

I would strongly recommend starting any enhancements here as a plugin, that way we can iterate there and recommend the plugin for testing.

If you’d like to put something up on GitHub under the ClassicPress name, let me know and I’ll create a repository for it. ClassicPress/improved-media-library is one idea for the name.

In order to be considered for merge into ClassicPress core, the plugin should have excellent automated test coverage and consistent code style.

~ posted by James Nylen

N.B. We’ve created https://github.com/ClassicPress-research for this purpose, so this repository would live there instead.

~ posted by James Nylen

Here is one thing I would REALLY like to see; the addition of register_image_type() where we could register a named type and them one of more sizes. Then when an image is added to the media library it could optionally be assigned an image type and if so resizing would be limited to just the sizes defined for that type.

This would be great for newspaper and magazine themes where they may need three different presentations (sizes) for feature images, but other images do not need to be those sizes, and vice versa.

Also, would love the option to have resize-on-first-use so that CP does not have to generate all applicable sizes when files are first uploaded but only has to resize when first requested. That way, if a size for an image is never requested, it is never created. Also, to “recreate all thumbnails” just delete all image files except the original ones, and they will eventually all be recreated.

(We implemented both of these concepts for an agency client in 2013 that used our code over and over for many different site implementations, but have not had the change since then to make the code completely generic.)

~ posted by Mike Schinkel

@mikeschinkel I’ve suggest the concept of what i called “image sets” similar to what you proposed in the first part of you post: https://github.com/ResponsiveImagesCG/wp-tevko-responsive-images/issues/45 maybe a few thoughts might be utilized from there as well.

~ posted by rkoller

@rkoller - Yep. “Image types” vs. “image sets”; different names, same concept.

Thanks for the link.

~ posted by Mike Schinkel

I like the image sets, but that also means initializing the media library in different contexts with only specific types visible, or else users will use images with the wrong type in the wrong place…

I like on the fly size generation better, but I’m not sure what a good implementation would look like that would play nice with nginx backproxied servers that bypass apache for all static resources… those server configs are pretty common these days.

~ posted by Greg Schoppe

maybe rather than generating the thumb on first view request, you could generate them when the user is first selected from the media library for insertion.

~ posted by Greg Schoppe

@greg - In our implementation if a we got a 404 for a file with a known image extension we would generate the file then redirect back to the same URL after which it would always just load the real image. So I think that would be compatible, though not sure.

~ posted by Mike Schinkel

@greg i like the idea of creating the according image sizes already within the media library. while setting alt text and alike you could also choose already from the list of available image set/image type. if the set/type changes at a later point and an image size is added or removed those could be generated on demand.

~ posted by rkoller

From Greg in Slack:

I have some code to remove attachment permalinks, that was written as a plugin, about a year ago, but could be much simplified with a few logical filters in core
they are meta on the attachment post type, iirc
https://gschoppe.com/wordpress/disable-attachment-pages/

~ posted by James Nylen

@mike That would work for some nginx backproxies, but not all. Some configs fall back to serving 404s through apache, but others provide their own minimal 404 page, regardless. WPEngine is one of the latter, where a 404 won’t be served via Apache, if the request is for a filename with an extension in their list of static file types.
(I spent about a year trying to work out a similar system to inject post keywords into image URLs on the fly, for Google Image Search… host compatibility is, unfortunately, a mess)

~ posted by Greg Schoppe

@greg Wow, that is a great point I was not aware of. When we did it it was before WPEngine launched and thus before the era of managed WordPress hosting.
(Now I have to figure out if there is another way to skin this cat…)

~ posted by Mike Schinkel

@Dustin Snider WP Real Media Library is a plugin that I have used that organizes the media library into folders. It really helps with sites that have thousands of images.

~ posted by Anji

Also would love to have an option to purge unused Media Files. This will make CP site smaller than WP powered site.

~ posted by W.V. Pelyn T. Palarao

See also:

• Discussion on the forums at Petititon: Improved Media Library
• Repository created for these improvements to start as a plugin: https://github.com/ClassicPress-research/improved-media-library

~ posted by James Nylen

As a photographer, I vote with pleasure.

1. Control the number of extra-generated images with different resolution without additional plugin would be great.
2. Organizing media files by post (instead of by month of upload) would give me a better orientation.
3. Maybe a setting for compression and image quality (if ever, and in the future, image compression is used) - a photographer would not compromise the quality of the photos on their site, even at the expense of a slower load . News blogs do not need super-quality images.

~ posted by KokoFresha

@KokoFresha: There already are settings for compression in place (just not very nice to work with from within the Admin Interface). I’d also be quite happy to have them sitting inside the WP Settings.

And even further, to be able to set DIFFERENT compression levels for each set image size. That would give us the chance to eg. compress thumbnails and small images more than huge header images, which might look bad when set to 60% quality, while the small images do not require to look too fancy.

On another note: Progressive JPEG (maybe as setting) would be great, too.

cu, w0lf.

~ posted by Fabian Wolf

@KokoFresha
Regarding point 3: the main problem is the majority of people use mobile devices to surf the internet and in many cases while they are not connected to a WiFi hotspot. This means they are often using the data volume bundled to their monthly payment-plan, that almost always has volume limitations. So if you as a photographer do not compress your images not only will be your website really slow, but it will also consume a lot of the above mentioned data volume making it also an expensive experience to visit your site. Not even a photographer would wish that i think

~ posted by Peter B

@Peter B
This is true. But it is good when there is a choice. My personal choice is to limit the resolution of uploaded photos more than compression. Of course, I would not want the system to change the original uploaded image because it was created in photographic software. My opinion related only to the low-resolution copies. If the code that generates them uses compression is good if we still have some control. One of the goals of a photographic site is to provide technically better pictures of facebook.

I’m not saying my point of view is the right one

~ posted by KokoFresha

@Fabian Wolf
I agree with you

~ posted by KokoFresha

@KokoFresha
For sites where above-average image quality is important i always let the hosting provider turn ImageMagick extension on. In that case WP will stop using GD Library and as a result the resized images will have superior quality while often also smaller data size as with GD.

Another trick that i use in such cases is the tinypng.com plugin. They have a very clever compression algorythm, you will not see any quality loss with your eyes while you get a really small file size. This process will be applied to all the image sizes generated by WP. The only downside of this plugin, that the free version has a monthly conversion limit, so if you are not ready to pay a one-time fee, you might need several months to convert all the images.

~ posted by Peter B

#jmtcw - Bottom line, I think users need a lot more control over image processing, with the option for the developer to lock anything down using settings for sites developed for task workers, such as some News, eCommerce sites, etc.

Every site/user’s need for images is potentially different; don’t go with the WP approach of deciding to only provide users with vanilla.

~ posted by Mike Schinkel

instead of:

decisions not choices

CP can aim for:

choices with sensible defaults

~ posted by Greg Schoppe

@Greg +1000

~ posted by Mike Schinkel

Enable Custom Media path Folder and URL

WordPress 3.5 removes the setting fields to change the media upload path and url.

It can be enabled manually from here /wp-admin/options.php

NAME-upload_path AND upload_path_url

Settings > Media, we can set the upload path(FOLDER) to folder like images,data etc.

It will be super useful for all.

https://petitions.classicpress.net/posts/150/enable-custom-media-path-folder-and-url

~ posted by Aryan Raj

Improvement suggestion: when I upload a high-resolution image to use as the featured image of a post, don’t serve the whole 2+ MB image along with the post! Instead, resize it to something reasonable like 1200 or 1400 px wide and then serve it.

~ posted by James Nylen

One way to solve file name conflicts and get rid of the year/month structure is to generate uniquely named folder for each image (and its thumbnails). Here’s a WP plugin I’ve written to accomplish this

https://github.com/CreunaFI/unique-media-file-folders

~ posted by Johannes

Yikes! Do I really want a ton of folders in my uploads folder?

~ posted by Ray Gulick

Nah, and neither is it a good solution - one could easily just use a non-insane renaming scheme, and be done with it (eg. {original-filename}-{number}.{ext}).

cu, w0lf.

~ posted by Fabian Wolf

The problem with a filename-based structure is that some operating systems use disk formats that only allow a certain number of files in each directory… without some form of subfolder system, many sites media libraries would be too big to store

~ posted by Greg Schoppe

I’ve occasionally used a system (not a plugin, but it could be) that places media into folders based on post type, and it’s worked out well so far. Media paths look like this:
/wp-content/uploads/post/filename.jpg
/wp-content/uploads/page/filename.jpg
/wp-content/uploads/listings/filename.jpg
/wp-content/uploads/news/filename.jpg

~ posted by Ray Gulick

While that looks really logical from a human standpoint, it assumes that any mass of images will be evenly spread among post types… ideally the system needs to not rely on any influenceable distribution and it needs to be able to grow over time to support more images as time goes on.

~ posted by Greg Schoppe

I don’t think it’s a matter of ‘evenly spread’, but a matter of whether any of the post types are likely to bump up against the limit of files contained in a folder. Not appropriate approach if there is a post type that adds 20 images a day, for instance.
BTW, what is the number of files allowed by typical disk formats; and is there a way to find or adjust that number at the webhost?

~ posted by Ray Gulick

unfortunately the answer to that question is highly dependent on both the file system and the commands running on top of it. For example, although EXT2 and 3 allow 1.3x10^20 files per folder, they also have a maximum number of blocks per folder, that can limit your number of files, if you have a small block-size set. Fat32 has a limit of 65k files per folder.

In both cases, however, many common commands like find and ls will operate extremely slowly if there are more than about 10k files in a folder, because they aren’t properly journaled, so many commands take linear time, based on number of files.

also keep in mind that this 10k files would include all thumbnail sizes, and some themes can have upwards of 10 registered image sizes.

As for adjusting the filesystem on the webhost, you will never be able to do so with a shared host. You can do it with most dedicated servers where you control the hardware (or virtualized hardware), but changing filesystems means wiping all the data on the server, so it tends to be unfeasible in practice.

A universal solution will have to work for systems where 10k files per folder is prohibitive, and will have to not be reliant on specific usage tropes, such as splitting by post type.

~ posted by Greg Schoppe

So, would 10k files per folder have to be considered the ‘practical’ limit in most cases?

~ posted by Ray Gulick

since it is a question of which filesystems we want to support and what level of performance we want to provide to those systems, It’s not a hard limit, but personally I would like to see folders stay under 10k files each. If we needed to count by attachments, rather than files (treating all thumbnails as one attachment), I’d see the practical cap as somewhere around 1k attachments per folder.

~ posted by Greg Schoppe

Since no one has mentioned it, one approach would be a form of sharding to assign a two (2) letter alphanumeric [a-z0-9] to each media item derived from a random two (2) character number in the range of 36^2 (less than 1300.) Then use that alphanumeric to create subdirectories in the current structure.

That would theoretically support ~1.5 million files per month before any one folder had more than 1300 files in a folder. And since this would not be a cryptographic issue the standard random functions should be sufficient.

This is not dissimilar to how Git handles objects in the .git folder.

#jmtcw

~ posted by Mike Schinkel

My experience matches the rough numbers above: 1k files per folder is still reasonable, 10k starts to get difficult to manage, and beyond that, various things will start to break or run very, very slowly. For the reasons previously discussed, it’s mostly impossible to come up with a hard, specific number.

I think the right approach to take here is “sane defaults, but allow for customization in extreme cases”. If you know that you’re going to be managing many thousands of media files on a ClassicPress site , then you should probably look into a customized storage scheme that fits your use case.

If there’s anything we can do to make this customization easier then that’s definitely something we should look into.

~ posted by James Nylen

one approach would be a form of sharding to assign a two (2) letter alphanumeric [a-z0-9] to each media item derived from a random two (2) character number in the range of 36^2 (less than 1300.)

I would use a value based on the hash of the filename rather than a truly random value, but this would also be a viable and useful approach here.

~ posted by James Nylen

Thats also why my idea of a more up-to-par Media DB API - so one could more easily than ever “plug in” a separate media management / storage solution, or write a custom one, which REALLY can compete with stand-alone solutions. Currently, stuff like Nextgen is nice, but not really up to the game.

cu, w0lf.

~ posted by Fabian Wolf

a pseudo-random directory structure certainly has some merit… although, if you used the first few characters of a hash of the file, instead of a random number, you could make the positions predictable (for scripts that want to see if a file is in the directory) and then you could store the hash, so we could finally implement deduplication.

~ posted by Greg Schoppe

Hash of the filename makes sense, unless the hashes in practices the hashes follow a long tail distribution, then we might want to consider something else.

~ posted by Mike Schinkel

w0lf’s comment about a pluggable media API has a lot of merit too. I can definitely see benefits there.

~ posted by Mike Schinkel

most modern hashes follow a pretty normal distribution. Even md5 should bbe sufficient for this (although I personally think something less broken would be a better idea)

~ posted by Greg Schoppe

also agreed about pluggable media library. I would suggest a complete ground-up rewrite of the media library. The monolithic backbone-based thing we have now is pretty awful.

~ posted by Greg Schoppe

This petition alone is enough material for a major release of ClassicPress.

It is obviously something that people want, now we “just” need to get some things implemented at https://github.com/ClassicPress-research/improved-media-library and try them out.

~ posted by James Nylen

There was an excellent plugin related to similar approach but has not been updated since many years:
https://wordpress.org/plugins/custom-upload-folders-plus/

~ posted by John

The plugin approach is a good one to explore the solution. I would find it very compelling if the final plugin-theme-core facing API makes it possible for a plugin to move media storage to an off-disk service, store rich metadata, and offload filtering (e.g. encoding/resize/crop).

~ posted by Will West

@will west: yeah, it would be totally awesome. also there is still the option, at least at the current state of CP, to use it in regular WP as well (as long as they dont destroy anything important aside of what has already been done …)

cu, w0lf.

~ posted by Fabian Wolf

I Love this Idea, however i don’t like the idea of categories - this will add extra queries to the mySQL / innoDB - folders could be created from a write command from the media library and media put in the created folder Much Like Media Library Folders Plugin within wordpress.

~ posted by Follow US

For adding chunked / resumable uploads: https://tus.io/

~ posted by James Nylen

[1stepforward]

I don’t think there’s any doubt that the media library can be improved but I think we should break it down into manageable chunks. We should agree what the main issues are and create a wishlist.

In the past, I’ve used this plugin - with good results - to replace the standard media library: https://codecanyon.net/item/justified-image-grid-premium-wordpress-gallery/2594251 but I have two go-to plugins that I install on nearly every site: https://wordpress.org/plugins/regenerate-thumbnails/ and https://wordpress.org/plugins/format-media-titles/.

Perhaps we could look at puling out the media library from the core and then tweak it as required. As has already been mentioned, there’s a research repository for this already at https://github.com/ClassicPress-research/improved-media-library if anyone wants to take that on?