Opened 6 weeks ago

Last modified 2 weeks ago

#16067 assigned Bug/Something is broken

preview images in nextcloud failing

Reported by: Jamie McClelland Owned by: Jamie McClelland
Priority: Medium Component: Tech
Keywords: nextcloud Cc:
Sensitive: no

Description

When trying to view an image in a shared Nextcloud folder, the browser returns an error: Error loading <name of file>.

In the Nextcloud logs we get:

{"reqId":"xxxxxxxxxxxxx","level":3,"time":"2020-10-22T13:05:08+00:00","remoteAddr":"xxxxxxxxxxxxx","user":"xxxxx","app":"index","method":"GET","url":"/core/preview?fileId=XXXXXXX&x=32&y=32","message":{"Exception":"OCP\\Files\\NotPermittedException","Message":"Could not create folder"

I modified the error logging to print out the folder it was trying to get. Then, I became the www-data user and tried to mkdir the folder it was requesting myself and got:

www-data@lucius:/var/lib/nextcloud/data$ mkdir appdata_XXXXXXX/preview/YYYYYYY
mkdir: cannot create directory ‘appdata_XXXXXXX/preview/YYYYYYY’: Too many links
www-data@lucius:/var/lib/nextcloud/data$

I've never seen this error before - apparently it means there are too many directories already.

I confirmed with:

www-data@lucius:/var/lib/nextcloud/data/appdata_XXXXXXX/preview$ ls |wc -l
64998
www-data@lucius:/var/lib/nextcloud/data/appdata_XXXXXXX/preview$

Wow, that is a lot. I'm not sure why there are so many - I'm working on searching the Nextcloud ticket queue and help system to see if others are having this problem but I'm not seeing much.

Change History (10)

comment:1 Changed 6 weeks ago by Jamie McClelland

I've found a good explanation of the preview directory including pro's and con's of deleting those files.

And, a blog and a another blog on the topic.

comment:2 Changed 6 weeks ago by Jamie McClelland

comment:3 Changed 6 weeks ago by Jamie McClelland

As an experiment, I found the oldest files in the preview directory with: ls -lt | tail and deleted the three oldest.

Then, I tried to load an image that was previously throwing an error and it worked.

We may want to consider a cron job that deletes entries older then a year.

comment:4 Changed 6 weeks ago by JaimeV

I am still seeing the error.

Shouldn't we be using ext4 for these partitions?

comment:5 Changed 6 weeks ago by Jamie McClelland

Oh - I didn't notice this was still ext3. Yes, we could convert to ext4 (probably not a good idea on this server - but when we migrate to the new hard ware we should migrate to a server with an ext4 partition).

However, it seems like ext4 has a similar limit. That page does mention that we can disable the limit with the dir_nlink option. I'm not sure what impications there are for using that option.

I'm inclined to first try cleaning up some of the preview images for older files since it seems odd that we have that many. But I'm on the fence about what the long term best option here is. I think the best option would be for Nextcloud to break this directory into sub directories, but that is probably beyond what we could implement (although we might consider opening an issue on their issue tracker). Seems like we can't be the only install with this many previews?

comment:6 Changed 5 weeks ago by Jamie McClelland

I'm starting with a conservative run:

find /var/lib/nextcloud/data/appdata_xxxxx/preview/ -maxdepth 1 -type d -mtime +900 -exec rm -rf '{}' \;

That deletes all previews older then 900 days (like 3 years or a bit less). I'm not sure if the nextcloud cron job re-creates all missing previews or not. I'll check in a few days.

Right now, we have:

0 lucius:~# ls /var/lib/nextcloud/data/appdata_xxxxxxx/preview/ |wc -l
59080
0 lucius:~#

That means we have space for another 5,000 previews or so.

If the cron job doesn't just re-create all the missing previews, then I'll add this as a daily cron job and set it to delete any preview folder older then one year. New previews will auto generate on demand and hopefully we can keep this under 64K.

comment:7 Changed 4 weeks ago by Jamie McClelland

So for so good - four days later and we still have a lot of slots available:

0 lucius:~# ls /var/lib/nextcloud/data/appdata_520aff6e7ca9f/preview/|wc -l
59779
0 lucius:~#

comment:8 Changed 4 weeks ago by Jamie McClelland

Owner: set to Jamie McClelland
Status: newassigned

comment:9 Changed 3 weeks ago by Jamie McClelland

It seems that the proper way to clean up this caceh may be to run occ files:cleanup. I'll test that during off hours.

The problem with directly deleting those preview images may be that the entries remain in the oc_filecache table, which on our install is enormous - 3,628,623 records and 7.5 GB.

comment:10 Changed 2 weeks ago by Jamie McClelland

Unfortunately, occ files:cleanup doesn't touch the preview directory or the oc_filecache table.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.