0
Under review

Thumb generation is half / broken...

Mark 4 weeks ago updated 1 week ago 12

First of all the make_thumbs.php cron failed while trying to generate thumbs for a folder with 200k+ files.

-----------

Generating thumbnails for admin's files:
PHP Warning: Error while sending QUERY packet. PID=22438 in /cloud/web/system/classes/vendor/FileRun/Utils/DB.php on line 46

-----------

secondly:

thumbs generation fails for large 20 page+ word docx documents and randomly some other word documents, while spending huge amount of time for it and sill failing. 

In file thumbs folder I see 20+ png files and one *[1].png.lock file which stays there after the cron has finished. (Why would it need to generate thumbs for all pages rather then just the first one?)

Also sometimes some 2-8mb image files are not being opened in the browser however the thumb exists. After the loading circle just an image icon appears..


This spoils this beautiful design so much... :(

Any ideas??

----------------------------------

Debian 10

php 7.3

mariadb 10.4

nginx >> php-fpm CGI

I have noticed that while the make_thumbs.php cron produces error described above, it also struggles to create thumbs for some pdf files, movies, and images too. Even if I open the folder in browser thumbs for these files are not being generated.

However if I reboot the server, and remove generated thumbs folders, and access the folder with same files, all seems to be fine and thumbs are generated for all docs, pdf and images... (but not some movies)

----------------------------


php.ini is the same for php-cli.

Under review
PHP Warning: Error while sending QUERY packet.

What is the full command you are running when getting this error, and how often are you getting this?

thumbs generation fails for large 20 page+ word docx documents and randomly some other word documents, while spending huge amount of time for it and sill failing.

What LibreOffice version are you using?

What hardware are you using? (I ask because it sounds like you might be using a very low performance one. Perhaps low amount of available RAM.)

The command was simply: 

php /cloud/web/cron/make_thumbs.php

My VPS setup:

3xvCPU (AMD EPYC)

4GB RAM

40GB Nvme SSD cache >> Scaleway S3 rclone local mount, cached (10Gbit link to vps)

Nginx and php-fpm configs highly optimized for performance.



If you can't replicate the situation I am happy to record a video

Getting "Error while sending QUERY packet." every time you try to run the command? Is the MySQL server installed on the same machine?

atm I can't replicate the same error. But thumbs generation still doesn't work from cron:


root@album:~# php /cloud/web/cron/make_thumbs.php test

-------------------- START ----------------------

Generating thumbnails for test's files:

Getting directory listing...: 31 files found

Processing files:
30 /cloud/s3/private/test/docs/Test 1.docx ....................................... ... DONE
29 /cloud/s3/private/test/docs/Test 1A.docx ...................................... ... DONE
28 /cloud/s3/private/test/docs/Test 2.docx ....................................... ... Failed!
27 /cloud/s3/private/test/docs/aplhafreecoins.com_root.pdf ....................... ... Failed!
26 /cloud/s3/private/test/docs/cp_export[mailserver].csv ......................... ... Failed!
25 /cloud/s3/private/test/docs/file-sample_100kB.doc ............................. ... Failed!
24 /cloud/s3/private/test/docs/file-sample_100kB.docx ............................ ... Failed!
23 /cloud/s3/private/test/docs/file-sample_1MB.doc ............................... ... Failed!
22 /cloud/s3/private/test/docs/file-sample_1MB.docx .............................. ... Failed!
21 /cloud/s3/private/test/docs/file-sample_500kB.doc ............................. ... Failed!
20 /cloud/s3/private/test/docs/file-sample_500kB.docx ............................ ... Failed!
19 /cloud/s3/private/test/docs/file_example_XLSX_100.xlsx ........................ ... Failed!
18 /cloud/s3/private/test/docs/file_example_XLSX_1000.xlsx ....................... ... Failed!
17 /cloud/s3/private/test/docs/file_example_XLSX_50.xlsx ......................... ... Failed!
16 /cloud/s3/private/test/docs/file_example_XLS_100.xls .......................... ... Failed!
15 /cloud/s3/private/test/docs/file_example_XLS_1000.xls ......................... ... Failed!
14 /cloud/s3/private/test/docs/file_example_XLS_50.xls ........................... ... Failed!
13 /cloud/s3/private/test/images/2.png ........................................... ... DONE (Doesn't require processing)
12 /cloud/s3/private/test/images/RAW_KODAK_DCSPRO.DCR ............................ ... DONE
11 /cloud/s3/private/test/images/RAW_SONY_NEX7.ARW ............................... ... DONE
10 /cloud/s3/private/test/images/RAW_SONY_NEX7.jpg ............................... ... DONE (Doesn't require processing)
9 /cloud/s3/private/test/images/_DSC0002.NEF .................................... ... DONE
8 /cloud/s3/private/test/images/_DSC0003.NEF .................................... ... DONE
7 /cloud/s3/private/test/images/_DSC0005.NEF .................................... ... DONE
6 /cloud/s3/private/test/images/_DSC0006.NEF .................................... ... DONE
5 /cloud/s3/private/test/images/_DSC0010.NEF .................................... ... DONE
4 /cloud/s3/private/test/images/_DSC0012.NEF .................................... ... DONE
3 /cloud/s3/private/test/images/_DSC6977-27.jpg ................................. ... DONE
2 /cloud/s3/private/test/images/baxters.jpg ..................................... ... DONE (Doesn't require processing)
1 /cloud/s3/private/test/images/fray bentos slice2.psd .......................... ... Failed!
0 /cloud/s3/private/test/images/tomato soup 2.psd ............................... ... Failed!


It still seems to randomly fail on some image formats rather than jpg and fails all of the documents (but not the first 2 in the list?) when using cron as root or www-data (same user as php uses).

Screen from FTP thumbs folder for failed psd image "fray bentos slice2.psd" as example -  (special location specified in filerun config): (in browser no thumb is seen)

However when I purge the thumbs folder and visit the same cloud folder with clear browser cache. Image thumbs are being generated ok. But still some docs thumbs and psd etc.. thumbs are not being made and either .lock files only can be found or a list of png and lock file in thumbs folder. 

This how folder with documents look when accessed from browser while no thumbs were ready:

This errors might be related to my setup as all files are located on S3 storage which is mounted, however with the current mounting options I did not experience same issues with Nextcloud during the tests.

---------------

I use php7.3-fpm (nginx/1.17.5)

MariaDB 10.4

Installed on clean vps Debian 10

Storage: S3 mounted with rclone (local cached)

It would be useful to see the results of testing for ImageMagick and LibreOffice from the FileRun control panel. This shows their versions and supported formats. Perhaps it's a version compatibility issue.

It looks like a code that triggers thumbs generation differs from the cron. Maybe the command line string? 

I also have checked with the files stored locally and not on S3, the errors are is the same.

2 /cloud/test/docs/Test 1A.docx ................................................. ... DONE
1 /cloud/test/docs/file-sample_1MB.docx ......................................... ... Failed!
0 /cloud/test/docs/file_example_XLS_50.xls ...................................... ... Failed!

...........

14 /cloud/test/docs/Test 1.docx .................................................. ... DONE

the rest docs failed

------------------

strangely enough these Test 1A.docx thumbnails are generated ok even when in another order. 

but the rest documents thumbs fails. 

When accessed from browser most thumbs are being generated (random choice) but still some fails.

To see what goes on behind the scene, you can enable the Admin: Thumbnail troubleshooter plugin from the control panel, clear your browser's cache and reload FileRun, then right-click a file that is missing a thumbnail and choose Open withAdmin: Thumbnail troubleshooter.

Thumbnail debugging is on (1574765677)

Processing: /cloud/s3/cloud/_0_test/file_example_XLS_100.xls

As: xls

File size: 20480

Removing expired lock: /cloud/cache/thumbs/cloud/s3/cloud/_0_test/file_example_XLS_100.xls/20480/400_400_contain[1].png.lock

Creating lock: /cloud/cache/thumbs/cloud/s3/cloud/_0_test/file_example_XLS_100.xls/20480/400_400_contain[1].png.lock

Extractor: \FileRun\Thumbs\Extractors\office

Converting office document to image.

Running: soffice --headless --nologo --nofirststartwizard --norestore --convert-to jpg:calc_jpg_Export --outdir "/cloud/cache/thumbs/cloud/s3/cloud/_0_test/file_example_XLS_100.xls" "/cloud/s3/cloud/_0_test/file_example_XLS_100.xls" 2>&1

No file found at /cloud/cache/thumbs/cloud/s3/cloud/_0_test/file_example_XLS_100.xls/file_example_XLS_100.jpg

Displaying xls.png icon instead.


Thumbs folder:

Errors of third-party programs are not always displayed in the output, so you can directly run this command yourself and see what the result is:

soffice --headless --nologo --nofirststartwizard --norestore --convert-to jpg:calc_jpg_Export --outdir "/cloud/cache/thumbs/cloud/s3/cloud/_0_test/file_example_XLS_100.xls" "/cloud/s3/cloud/_0_test/file_example_XLS_100.xls"

update:

New error output in thumbnail troubleshooter: 

I did not make any changes, just reloaded the page and opened troubleshooter again.

Thumbnail debugging is on (1574771458)
Processing: /cloud/s3/cloud/IG/1 Brands/2019 12/Raw/_0_test/file_example_XLS_1000.xls
As: xls
File size: 140288
Removing expired lock: /cloud/cache/thumbs/cloud/s3/cloud/IG/1 Brands/2019 12/Raw/_0_test/file_example_XLS_1000.xls/140288/400_400_contain[1].png.lock
Creating lock: /cloud/cache/thumbs/cloud/s3/cloud/IG/1 Brands/2019 12/Raw/_0_test/file_example_XLS_1000.xls/140288/400_400_contain[1].png.lock
Extractor: \FileRun\Thumbs\Extractors\office
Converting office document to image.
Running: soffice --headless --nologo --nofirststartwizard --norestore --convert-to jpg:calc_jpg_Export --outdir "/cloud/cache/thumbs/cloud/s3/cloud/IG/1 Brands/2019 12/Raw/_0_test/file_example_XLS_1000.xls" "/cloud/s3/cloud/IG/1 Brands/2019 12/Raw/_0_test/file_example_XLS_1000.xls" 2>&1
Results:
* returned code: 134
* returned text: Array ( [0] => convert /cloud/s3/cloud/IG/1 Brands/2019 12/Raw/_0_test/file_example_XLS_1000.xls -> /cloud/cache/thumbs/cloud/s3/cloud/IG/1 Brands/2019 12/Raw/_0_test/file_example_XLS_1000.xls/file_example_XLS_1000.jpg using filter : calc_jpg_Export [1] => Error: Please verify input parameters... (SfxBaseModel::impl_store failed: 0xc10(Error Area:Io Class:Write Code:16)) [2] => Application Error [3] => [4] => [5] => Fatal exception: Signal 6 [6] => Stack: [7] => /usr/lib/libreoffice/program/libuno_sal.so.3(+0x3d523)[0x7efef3425523] [8] => /usr/lib/libreoffice/program/libuno_sal.so.3(+0x3d733)[0x7efef3425733] [9] => /lib/x86_64-linux-gnu/libc.so.6(+0x37840)[0x7efef324d840] [10] => /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7efef324d7bb] [11] => /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7efef3238535] [12] => /usr/lib/libreoffice/program/libmergedlo.so(+0x11ed84c)[0x7efef463384c] [13] => /usr/lib/libreoffice/program/libmergedlo.so(_ZN11Application5AbortERKN3rtl8OUStringE+0x90)[0x7efef6297c90] [14] => /usr/lib/libreoffice/program/libmergedlo.so(+0x1f078a7)[0x7efef534d8a7] [15] => /usr/lib/libreoffice/program/libmergedlo.so(+0x2e5725b)[0x7efef629d25b] [16] => /usr/lib/libreoffice/program/libuno_sal.so.3(+0x17762)[0x7efef33ff762] [17] => /usr/lib/libreoffice/program/libuno_sal.so.3(+0x3d5ff)[0x7efef34255ff] [18] => /lib/x86_64-linux-gnu/libc.so.6(+0x37840)[0x7efef324d840] [19] => /usr/lib/libreoffice/program/libuno_sal.so.3(osl_closeFile+0xf)[0x7efef341d45f] [20] => /usr/lib/libreoffice/program/libmergedlo.so(+0x13493e6)[0x7efef478f3e6] [21] => /usr/lib/libreoffice/program/libmergedlo.so(_ZN10comphelper16BackupFileHelper12tryPush_fileERKN3rtl8OUStringES4_S4_S4_+0x196)[0x7efef4793fc6] [22] => /usr/lib/libreoffice/program/libmergedlo.so(_ZN10comphelper16BackupFileHelper13tryPush_FilesERKSt3setIN3rtl8OUStringESt4lessIS3_ESaIS3_EERKS1_ISt4pairIS3_S3_ES4_ISB_ESaISB_EERKS3_SI_+0x86)[0x7efef4798466] [23] => /usr/lib/libreoffice/program/libmergedlo.so(_ZN10comphelper16BackupFileHelper13tryPush_FilesERKSt3setIN3rtl8OUStringESt4lessIS3_ESaIS3_EERKS1_ISt4pairIS3_S3_ES4_ISB_ESaISB_EERKS3_SI_+0x1c0)[0x7efef47985a0] [24] => /usr/lib/libreoffice/program/libmergedlo.so(_ZN10comphelper16BackupFileHelper13tryPush_FilesERKSt3setIN3rtl8OUStringESt4lessIS3_ESaIS3_EERKS1_ISt4pairIS3_S3_ES4_ISB_ESaISB_EERKS3_SI_+0x1c0)[0x7efef47985a0] [25] => /usr/lib/libreoffice/program/libmergedlo.so(_ZN10comphelper16BackupFileHelper7tryPushEv+0x7d)[0x7efef479948d] [26] => /usr/lib/libreoffice/program/libmergedlo.so(+0x1f08087)[0x7efef534e087] [27] => /usr/lib/libreoffice/program/libmergedlo.so(+0x1f0cbaa)[0x7efef5352baa] [28] => /usr/lib/libreoffice/program/libmergedlo.so(+0x2e589e6)[0x7efef629e9e6] [29] => /usr/lib/libreoffice/program/libmergedlo.so(_Z6SVMainv+0x30)[0x7efef629eae0] [30] => /usr/lib/libreoffice/program/libmergedlo.so(soffice_main+0x91)[0x7efef5370de1] [31] => /usr/lib/libreoffice/program/soffice.bin(+0x107b)[0x55c28316c07b] [32] => /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7efef323a09b] [33] => /usr/lib/libreoffice/program/soffice.bin(+0x10ba)[0x55c28316c0ba] ) Displaying xls.png icon instead.

While running the same command in terminal I've got no errors and thumbnail was generated .

soffice --headless --nologo --nofirststartwizard --norestore --convert-to jpg:calc_jpg_Export --outdir "/cloud/cache/thumbs/cloud/s3/cloud/IG/1 Brands/2019 12/Raw/_0_test" "/cloud/s3/cloud/IG/1 Brands/2019 12/Raw/_0_test/file_example_XLS_1000.xls"

Also the thumbs for office/pdf/other? docs not being displayed on every 4/5th file, even when thumbnail was generated OK according to Troubleshooter and file check. 

The only way to resolve it - is to completely erase browser's cache and revisit the page... 

All images btw, like jpg have no problem generating thumbs by visiting browser page or using make_thumbs.php

-----------------------

Win 10 / Opera

Any ideas on these issues?

it's around 2 weeks passed since the first report but without any actual response.