Solution to Traveller PDFs not showing images in PDFKit based renderers (macOS and iOS)

[MongooseMatt]

Image problems with Apple devices should now be fixed - simply re-download the file and try again.

Very sorry it took so long to fix, and please let us know if you still see any issues. However, assuming all goes well, we will start updating previous books to this format next week.

 

[apastuszak]

Which files did you update?

 

[MongooseMatt]

Which files did you update?

So far, This is Free Trader Beowulf and Fifth Frontier War, but we will be going over the rest from next week onwards. Expect updates!

 

[prunk]

So far, This is Free Trader Beowulf and Fifth Frontier War, but we will be going over the rest from next week onwards. Expect updates!

Thanks a lot for this! Is there a way to know which files are updated when looking at the download pages later? Are you appending the date on the file names? It helps a lot when I see the files with Apr24 or something on them.

 

[Evil Aardvark]

So far, This is Free Trader Beowulf and Fifth Frontier War, but we will be going over the rest from next week onwards. Expect updates!

Thank you.

 

[The Canterbury Tail]

Deepest apologies for having taken this long - but yes, I think we have cracked it. If all goes well with Beowulf tomorrow, expect a mass update over the coming days/weeks of all our other books affected by this.

Matt this would be absolutely wonderful. It is my single biggest issue with any of your products (which I love dearly I have to say) that I can't read them properly on my iPad. If you have nailed this and can update everything, that would be just an amazing early May Day present.

 

[apastuszak]

So, can anyone confirm the fix worked?

 

[Yatima]

So, can anyone confirm the fix worked?

Looking at two files (FFW and Free Trader Beowulf) the fix did work.

J

 

[BangsNaughtyBits]

Several PDFs were updated today, April 23, 2024. A quick check of a few pain points, the image immediately after the table of contents on page 2 of High Guard 2022 Update for example, are now working for me. I will have to do a side by side comparison at some point.

Edit: Ohh, thank you Mongoose.

The Great Apple PDF Update

Hey everyone, We are currently in the process of going through all our past PDFs and updating them so they display properly on Apple devices. There is rather a lot of them (!), but we will be steadily working through every PDF with this problem. We will update this post as new ones are added on...

forum.mongoosepublishing.com

!

 

[apastuszak]

So, I know we're getting fixed files, but if the one you're waiting for still isn't updated, I found yet another workaround, if you have a Mac. The App PDF Squeezer will allow you to create a custom compression profile and set an option to convert all images to RGB. The file created works fine on Apple devices and seems to keep all links.

Of course, you need a Mac or iOS device and $20.00 to buy the app. But it can also convert PDFs to greyscale for use on eReader with large e-ink screens like the Kindle Scribe.

 

[Timolution]

So, I know we're getting fixed files, but if the one you're waiting for still isn't updated, I found yet another workaround, if you have a Mac. The App PDF Squeezer will allow you to create a custom compression profile and set an option to convert all images to RGB. The file created works fine on Apple devices and seems to keep all links.

Of course, you need a Mac or iOS device and $20.00 to buy the app. But it can also convert PDFs to greyscale for use on eReader with large e-ink screens like the Kindle Scribe.

Well whadaya know! Been using this application for years and never thought about using this functionality for this reason… It does indeed fix the missing images issue while keeping all internal links and TOC metadata intact Live and learn I guess. Only thing I am a bit less thrilled about is that the RGB recompression deteriorates the images a bit (worse aliasing) and that is causes slight colour shifts (noticeable in dark areas), but that is mostly because of my OCD when it comes to these things Most probably won’t notice.

 

[apastuszak]

I asked the author if he could allow to not compress and just do color conversion as an option in one profile.

So you think it's the RBG conversion that's causing the issue or the compression of the images?

 

[Timolution]

Well, I given the aliasing and subtle colour shifts I presumed it was the conversion rather than the compression (in which case I would rather expect typical block artefacting). So I did a test where I twice 'compressed' a pdf with image quality at 100% and max image dpi at 150 (none of the images in the pdf had a dpi higher than 105, so this way no downsampling would take place). No other options selected save that one was done in the original colourspace and one converted to RGB colourspace. The former had no discernible image degradation, whereas the latter showed the aliasing and slight colour shift. So for me that seems to imply that it is the conversion, rather than the compression, being the cause.

Again, I think most people won't mind as you really need to zoom in to notice it, but still... So I guess I'll keep using the Cairo + pdftk datadump + PDF squeezer method, unless there are internal crosslinks in the main PDF text in which case the method you found would likely be preferable.

 

[apastuszak]

I'm curious what Cairo is doing. I wonder if it's converting all images to another format and that's how it "fixes" the PDF. I wonder if cairo converts all the JPEG2000 images to PNG, and that's why we see the increase in file size. If it did that, I was thinking it might preserve the CMYK color space. But I don't believe PNG supports CMYK.

There's still something odd about these PDFs. If I use ghostscript to convert the April Update of the 2022 Core Rulebook to greyscale, the file is over 700 MB in size. If I use ghostscript to convert other publishers' PDFs to greyscale they usually come in at half the size of the black and white. PDF Squeezer can shrink it back down to a decent size, and I can even use PDF Squeezer to do the conversion.

 

[Timolution]

Just did a quick check and indeed Cairo converts the whole lot to png (without any associated colourspace), and it seems to be generating a lot of extraneous images... Mongoose pdfs have a lot of white Jpeg and JPG2000 background images, which I presume live as as a sort of background under the actual images, and for each of these Cairo is producing a black 'shadow' image it seems. So png format combined with a lot of extra PNGs probably explains the file size increase (I also have to admit that looking at these results it seems some colour shift also happens with Cairo, though it is slightly more subtle than PDF squeezer):

Original PDF images


Cairo'd PDF images


Also I figured out the aliasing problem with PDF squeezer. Turns out that if you limit the max dpi to 150 for some reason the application shrinks all image dimensions by about 25%... No idea why it is doing that (bug maybe?), but putting the DPI limit to 300 fixes that issue.

PDF squeezer to RGB @ max 150DPI


PDF squeezer to RGB @ max 300DPI



And in this latter case the image quality is pretty much indiscernible after all, save for the aforementioned colour shift which is more pronounced in the PDF Squeezer output (compare the space background for example, may need to zoom in to see it):

 

[apastuszak]

So, if Cairo converts to PNG, it automatically converting to RGB colorspace, since PNG can't do CMYK. That would explain why it "fixes" the issue.

But it sounds like PDF Squeezer at 300 DPI with RGB conversion is acceptable. I wish PDF Sqeezer let you save a file, even if it's going to be larger than the original file. Unfortunately, if there is no space savings, it won't let you save.

 

[Timolution]

Well, it seems if you reduce the image quality to 99% you will almost always get an output that is smaller than the original PDF, with imperceptible image quality difference. Even at the 300dpi setting, as this basically only says that any image with a resolution >300DPI should be downsampled to 300 (if I understand the setting correctly). But given that none of the Mongoose PDFs contain images at that resolution it is not a problem at all.

But yes, I would also like that the app would allow exporting a PDF even if it has become bigger. Maybe something I'll contact them about (in addition to the strange image dimension reduction issue mentioned above).

 

[apastuszak]

Well, it seems if you reduce the image quality to 99% you will almost always get an output that is smaller than the original PDF, with imperceptible image quality difference. Even at the 300dpi setting, as this basically only says that any image with a resolution >300DPI should be downsampled to 300 (if I understand the setting correctly). But given that none of the Mongoose PDFs contain images at that resolution it is not a problem at all.

But yes, I would also like that the app would allow exporting a PDF even if it has become bigger. Maybe something I'll contact them about (in addition to the strange image dimension reduction issue mentioned above).

You'd think that was true. But for some of the PDFs I tried to shrink, I had to lower the image quality to 50%-60% before PDF Squeezer would let me save the file when doing an RGB conversion.

 

[Timolution]

Well, in the meantime you can always find the processed file (whether inflated or shrunk) at the following location as long as you don't quit PDF Squeezer:

~/Library/Containers/com.witt-software.PDF-Squeezer/Data/tmp/

Just copy it out and rename.

 

[apastuszak]

Well, in the meantime you can always find the processed file (whether inflated or shrunk) at the following location as long as you don't quit PDF Squeezer:

~/Library/Containers/com.witt-software.PDF-Squeezer/Data/tmp/

Just copy it out and rename.

Oh nice. Thank you! Explains why the save is so fast. It just needs to do a quick file move.