Opened 11 months ago

Closed 7 months ago

#16067 closed Bug/Something is broken (fixed)

preview images in nextcloud failing

Reported by: Jamie McClelland Owned by: Jamie McClelland
Priority: Medium Component: Tech
Keywords: nextcloud Cc:
Sensitive: no

Description

When trying to view an image in a shared Nextcloud folder, the browser returns an error: Error loading <name of file>.

In the Nextcloud logs we get:

{"reqId":"xxxxxxxxxxxxx","level":3,"time":"2020-10-22T13:05:08+00:00","remoteAddr":"xxxxxxxxxxxxx","user":"xxxxx","app":"index","method":"GET","url":"/core/preview?fileId=XXXXXXX&x=32&y=32","message":{"Exception":"OCP\\Files\\NotPermittedException","Message":"Could not create folder"

I modified the error logging to print out the folder it was trying to get. Then, I became the www-data user and tried to mkdir the folder it was requesting myself and got:

www-data@lucius:/var/lib/nextcloud/data$ mkdir appdata_XXXXXXX/preview/YYYYYYY
mkdir: cannot create directory ‘appdata_XXXXXXX/preview/YYYYYYY’: Too many links
www-data@lucius:/var/lib/nextcloud/data$

I've never seen this error before - apparently it means there are too many directories already.

I confirmed with:

www-data@lucius:/var/lib/nextcloud/data/appdata_XXXXXXX/preview$ ls |wc -l
64998
www-data@lucius:/var/lib/nextcloud/data/appdata_XXXXXXX/preview$

Wow, that is a lot. I'm not sure why there are so many - I'm working on searching the Nextcloud ticket queue and help system to see if others are having this problem but I'm not seeing much.

Change History (11)

comment:1 Changed 11 months ago by Jamie McClelland

I've found a good explanation of the preview directory including pro's and con's of deleting those files.

And, a blog and a another blog on the topic.

comment:2 Changed 11 months ago by Jamie McClelland

comment:3 Changed 11 months ago by Jamie McClelland

As an experiment, I found the oldest files in the preview directory with: ls -lt | tail and deleted the three oldest.

Then, I tried to load an image that was previously throwing an error and it worked.

We may want to consider a cron job that deletes entries older then a year.

comment:4 Changed 11 months ago by JaimeV

I am still seeing the error.

Shouldn't we be using ext4 for these partitions?

comment:5 Changed 11 months ago by Jamie McClelland

Oh - I didn't notice this was still ext3. Yes, we could convert to ext4 (probably not a good idea on this server - but when we migrate to the new hard ware we should migrate to a server with an ext4 partition).

However, it seems like ext4 has a similar limit. That page does mention that we can disable the limit with the dir_nlink option. I'm not sure what impications there are for using that option.

I'm inclined to first try cleaning up some of the preview images for older files since it seems odd that we have that many. But I'm on the fence about what the long term best option here is. I think the best option would be for Nextcloud to break this directory into sub directories, but that is probably beyond what we could implement (although we might consider opening an issue on their issue tracker). Seems like we can't be the only install with this many previews?

comment:6 Changed 11 months ago by Jamie McClelland

I'm starting with a conservative run:

find /var/lib/nextcloud/data/appdata_xxxxx/preview/ -maxdepth 1 -type d -mtime +900 -exec rm -rf '{}' \;

That deletes all previews older then 900 days (like 3 years or a bit less). I'm not sure if the nextcloud cron job re-creates all missing previews or not. I'll check in a few days.

Right now, we have:

0 lucius:~# ls /var/lib/nextcloud/data/appdata_xxxxxxx/preview/ |wc -l
59080
0 lucius:~#

That means we have space for another 5,000 previews or so.

If the cron job doesn't just re-create all the missing previews, then I'll add this as a daily cron job and set it to delete any preview folder older then one year. New previews will auto generate on demand and hopefully we can keep this under 64K.

comment:7 Changed 11 months ago by Jamie McClelland

So for so good - four days later and we still have a lot of slots available:

0 lucius:~# ls /var/lib/nextcloud/data/appdata_520aff6e7ca9f/preview/|wc -l
59779
0 lucius:~#

comment:8 Changed 11 months ago by Jamie McClelland

Owner: set to Jamie McClelland
Status: newassigned

comment:9 Changed 10 months ago by Jamie McClelland

It seems that the proper way to clean up this caceh may be to run occ files:cleanup. I'll test that during off hours.

The problem with directly deleting those preview images may be that the entries remain in the oc_filecache table, which on our install is enormous - 3,628,623 records and 7.5 GB.

comment:10 Changed 10 months ago by Jamie McClelland

Unfortunately, occ files:cleanup doesn't touch the preview directory or the oc_filecache table.

comment:11 Changed 7 months ago by Jamie McClelland

Resolution: fixed
Status: assignedclosed

I re-generated the oc_filecache to deal with the enormous size of the table on disk. I probably would not recommend doing it again. It took over 3 hours to complete with Nextcloud in maintenance mode. Bah! And, somehow, during the process, the config.php file was overwritten and had to be restored from backup (not sure why that happened).

The number of records in oc_filecache is unchanged, but since I truncated and vacuumed it, it takes up considerably less disk space (1.6GB instead of 7.5GB).

In any event, I think the file preview problem is solved.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.