Solution to Traveller PDFs not showing images in PDFKit based renderers (macOS and iOS)

[Timolution]

=====================================================================================================
|THE METHOD IN THIS POST IS OUTDATED! PLEASE LOOK AT POSTS #31 AND #41 FOR THE BEST WAY |
=====================================================================================================


First and foremost, I am not sure if this the right place to put this, so moderators please feel free to ove it to a better subforum if necessary.

So over the last couple of Traveller PDF releases people using macOS or iOS have noted that often covers and other images in the PDFs do not render when viewing the files in the stock Preview app or any other app that uses the native PDFKit renderer. One solution up until now was using a third party application that uses a different rendering engine, be it Adobe Reader or a solution based on the third party PSPDFKit renderer.

Whatever the underlying reason, it is likely a combination of PDFKit being too overtly limited in its support of PDFs that are not conform the published specification and the PDF producing software used by Mongoose producing files not fully adhering to said specification.*

Anyway, it is rather annoying for us Mac/iOS users that we have to use 3rd party viewers to properly read these files, so I went and found a workaround solution which I'll leave here for anyone interested. Note that this will likely only work with files bought from Mongoose directly, as they are not DRMed/locked unlike files bought through Drivethru, and that you will need to use the CLI on your Mac or Linux machine.

One caveat to keep in mind however is that, likely due to the non-compliance of the PDFs to the published standard and the fact that you transform the PDF to a PDF/A standard, this method will sometimes transform text in rasterized characters. I notice this mostly happens on pages where text is overlaying a coloured background and thus probably a consequence of the image layering of those pages. So keep that in mind when you compress the PDF at the final step.

For those that are used to working in the CLI a lot of this will al be very basic level, apologies, but there may be readers out there who have never touched the command line or are not very familiar with a UNIX command prompt.

Mac users should first install a package manager for ease of CLI app installation. You can use Brew or MacPorts, whatever floats your boat, as we only need this to get GhostScript up and running. Linux users can probably get GhostScript from their preferred repo if it is not already installed as part of the default installation.

I personally use Brew, so here's how to install this on your Mac:

• Copy and paste the below command in your Terminal app, press enter and follow the instructions that will appear**:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

• Once Brew is installed, go ahead and install GhostScript by pasting the following command in the Terminal and pressing enter:

brew install ghostscript

​

Now that GhostScript is installed we can 'fix' the wayward Traveller PDF so it behaves normally when being rendered through PDFKit:

!!! I suggest you always work on a copy of the original file just in case !!!

• Put the copy of the probematic PDF in a folder of your choosing and name it original_copy.pdf (for ease of following along)

• In your terminal type

  cd

• Drag the folder with your PDF copy onto the Terminal window and press enter

• The prompt in your terminal should now read

  your_username@your_computername name_of_folder_you_just_dragged %

• If that is the case you have succesfully changed your working directory to the folder containing our PDF copy. If not, try to redoing the above steps.

• Now we will convert the copy of the problem PDF to a PDF/A file using Ghostscript (note that this will hugely inflate the size of the file, up to 500-600%) by copying and pasting the following command in the Terminal follwed by pressing enter:

  gs -dPDFA -dBATCH -dNOPAUSE -sColorConversionStrategy=UseDeviceIndependentColor -sDEVICE=pdfwrite -dPDFACompatibilityPolicy=2 -sOutputFile=converted_copy.pdf original_copy.pdf

• A lot of text will now start scrolling in your terminal window, with occasional errors and warnings. Just let it work until you see the command prompt again

  your_username@your_computername name_of_folder_you_just_dragged %

• In the folder where you put your copy oif the original pdf there should now be a very large file named converted_copy.pdf. You can confirm now that Preview will open this pdf and in fact show all images as intended.

You now have a pdf that is useable by the native macOS/iOS PDF rendering engine. However, you may notice that the file size of this PDF is quite huge, so I would suggest that we compress it down again. Here the amount of compression possible can go from very little to very much and will depend on how much image quality you want to preserve. This can be done through the CLI again, or by using a GUI compression app. I prefer and suggest PDF Squeezer (not affiliated, but this software has never let me down), though there are numerous other GUI options out there that probably can get the job done.

If you want to use a free solution though GhostScript can come to the rescue once again:

• Make sure you are still in the correct directory (the one where the newly made PDF is located)

• Copy paste the following command and press enter ***:

  gs -sDEVICE=pdfwrite -dCompatibilityLevel=2 -dPDFSETTINGS=/ebook -dNOPAUSE -dBATCH -sOutputFile=compressed_copy.pdf converted_copy.pdf

• Again text will start scolling and this is a slow process (one of the reasons I swear by PDF Squeezer as it is a lot faster), so go grab yourself a cuppa coffee or something in the meantime.

Once you see the command prompt again the process is done and you shoud now have a file called compressed_copy.pdf, which should be smaller than the converted pdf we made earlier but still readable in Preview.

You can of course rename your compressed PDF afterwards to whatever name you want.

==========

* Of the latter point I am pretty sure as this is the message you get when running the original PDF through GhostScript:

The following errors were encountered at least once while processing this file:​

error executing PDF token​

**** This file had errors that were repaired or ignored.​

**** The file was produced by:​

**** >>>> PDFlib+PDI 9.1.1p3 (PHP7/Linux-x86_64) <<<<​

**** Please notify the author of the software that produced this​

**** file that it does not conform to Adobe's published PDF​

**** specification.​

** Every now and then you might want to update your Brew installation and installed packages by using the following (combined) command:

brew update; brew upgrade; brew cleanup; brew doctor

​

*** You can specify the amount of compression by changing the -dPDFSETTINGS parameter in the command. You will have to search a bit yourself which option has the best file size versus image quality ratio for your preference, but the following options are possible:

-dPDFSETTINGS=/screen (screen-view-only quality, 72 dpi images)​

-dPDFSETTINGS=/ebook (low quality, 150 dpi images)​

-dPDFSETTINGS=/printer (high quality, 300 dpi images)​

-dPDFSETTINGS=/prepress (high quality, color preserving, 300 dpi imgs)​

-dPDFSETTINGS=/default (almost identical to /screen)​

 

[Timolution]

So, after some further noodling around it seems the root of the issue is that the images that don't show up are in fact JPG2000 (with extensions) files that have an embedded CMYK colour profile. The native ImageIO pipeline in macOS and iOS cannot deal with this kind of JPG2000 images (confirmed by extracting them from the PDF and changing the colour space to sRGB, after which they render without any issue) and this is an issue going back years already. So not a PDFKit rendering problem after all and no idea why Apple doesn't seem to want to fix this... Guess they want to focus on HEIC instead.

Otoh, not really sure why JPG2000 with CMYK profiles (instead of say sRGB) are needed in a PDF meant for screen reading purposes, so maybe Mongoose could look into changing that? Not sure how feasible that is in their PDF generation pipeline, but at least as an added benefit it would cut back on the filesizes of the images in the pdf I reckon (allowing maybe for higher resolution export, hinthint ).

 

[CordwainerFish]

at least as an added benefit it would cut back on the filesizes of the images in the pdf

It would also make the file icons no longer be absolutely blank.

 

[Timolution]

However, now that we know what the actual underlying issue is we can in fact employ a far easier method of 'fixing' the PDF for PDFKit renderers, using ghostscript, which that does not involve conversion to a PDF/A compliant file (and hence skips the issue of some text being rasterised)!

We will instead simply remux the PDF in an sRGB colourspace. You will lose a bit of image quality as ghostscript needs to rerender the images, but I feel it remains acceptable at the highest quality flag while still gaining a 37-40% file size reduction largely by booting the CMYK data.

Obviously you need to first have ghostscript installed as per the instructions above, and then you simply use the following command in terminal:

gs -sColorConversionStrategy=sRGB  -dBATCH -dNOPAUSE -dNOCACHE -dPDFSETTINGS=/printer -sDEVICE=pdfwrite  -sOutputFile=converted_copy.pdf original_copy.pdf

You should now have a file 'converted.pdf ' that still has all the bookmarks intact, with all text as actual searchable text and with all images showing as they should. Note that if you are willing to sacrifice more image quality you can get even higher space savings by using the various -dPDFSETTINGS flagslisted in my first post.

And indeed as @CordwainerFish noted above, this will now also allow macOS and iOS to generate a proper file icon.

Edit: It does however seem that some images will not have their transparancies properly preserved unfortunately...

 

[CordwainerFish]

We will instead simply remux the PDF in an sRGB colourspace.

How does this affect the file size? (I mean, yeah, it's 2023 and storage is cheap, but I had a 1MB disk quota when I was a Unix newbie and old fears die hard.)

Also, if you have the Rim Expeditions .pdf, does this fix page 118?

 

[Timolution]

How does this affect the file size? (I mean, yeah, it's 2023 and storage is cheap, but I had a 1MB disk quota when I was a Unix newbie and old fears die hard.)

Also, if you have the Rim Expeditions .pdf, does this fix page 118?

Well, in CMYK the colour information is defined by 4 channels, whereas in RGB only three channels are used (so 25% less data) though ironically the RGB colour space is actually far larger than the CMYK one.

And can confirm that page 118 is fixed in this way. So if you can live with grey semi-transparent boxes around some images and you'd rather not use 3rd party pdf readers, then this is the way to go if you use macOS/iOS and if Mongoose intends to keeps using CMYK extended JPEG2000 in their PDFs.

To be honest, after extracting all images from the PDF using pdfimage I have to say i am a bit baffled at the eclectic mix of formats jammed into one file JPG, PNG, TIFF, JP2, JPX... Why such an unoptimised mess of formats? Why not just use one format for all images? A really bizarre way of doing things imo...

 

[apastuszak]

However, now that we know what the actual underlying issue is we can in fact employ a far easier method of 'fixing' the PDF for PDFKit renderers, using ghostscript, which that does not involve conversion to a PDF/A compliant file (and hence skips the issue of some text being rasterised)!

We will instead simply remux the PDF in an sRGB colourspace. You will lose a bit of image quality as ghostscript needs to rerender the images, but I feel it remains acceptable at the highest quality flag while still gaining a 37-40% file size reduction largely by booting the CMYK data.

Obviously you need to first have ghostscript installed as per the instructions above, and then you simply use the following command in terminal:

gs -sColorConversionStrategy=sRGB  -dBATCH -dNOPAUSE -dNOCACHE -dPDFSETTINGS=/printer -sDEVICE=pdfwrite  -sOutputFile=converted_copy.pdf original_copy.pdf

You should now have a file 'converted.pdf ' that still has all the bookmarks intact, with all text as actual searchable text and with all images showing as they should. Note that if you are willing to sacrifice more image quality you can get even higher space savings by using the various -dPDFSETTINGS flagslisted in my first post.

And indeed as @CordwainerFish noted above, this will now also allow macOS and iOS to generate a proper file icon.

Edit: It does however seem that some images will not have their transparancies properly preserved unfortunately...

This worked. The file size was really reduced. But it did some font substitutions in the conversion. Hopefully Mongoose can find proper PDF export settings to allow us Apple users to be happy with the PDFs they generate and we won't need to use this workaround.

 

[Sigtrygg]

How about you Apple users get in touch with Apple and say you will stop using their products unless they fix the issue with their software.
I fail to see how the onus is on Mongoose.

 

[apastuszak]

How about you Apple users get in touch with Apple and say you will stop using their products unless they fix the issue with their software.
I fail to see how the onus is on Mongoose.

Well, Mongoose is the ONLY publisher I know that produces PDFs I can't view on my Mac or my iPad. The onus for this is ultimatelty on Apple. But it seems every other publisher has figured out a way to export their PDFs so they work on Apple hardware. As I said in the other thread, this is simply a matter of using the right export settings when making your PDF.

It will require that Mongoose put in an extra 15 minutes work to get the right kind of PDF generated. Would you spend 15 minutes of time on something you plan to sell, if you knew that you'd fix a problem for a couple hundred of your customers?

This isn't about your hatred of Apple products and how you wish Apple would go out of business tomorrow. It's just about making a good PDF files that works properly for all customers. I don't think that's too much to ask. Especially since I would a decent chunk of PDF purchasers are using them on an iPad.

 

[Timolution]

How about you Apple users get in touch with Apple and say you will stop using their products unless they fix the issue with their software.
I fail to see how the onus is on Mongoose.

I don't understand why people feel the need to come in threads on topics that do not affect them just to talk down to people trying to solve a problem, especially when they clearly have no idea what they're talking about.

Nobody is putting onuses on anyone in here, we are solely identifying a problem and as customers we try to bring this to Mongoose's attention as the easiest solution to this problem would come from their end. A few nerds not being able to correctly display their favourite nerd-game pdfs is not going to sway a trillion dollar corpoaration to turn around fundamental parts of or longstanding decisions made on the development and priorities for their operating systems, but maybe the nerd-game developer can do something from their side.

The problem is that the pdf specification has become a mess after spec 1.7, when Adobe handed it over to the ISO organisation but retained the right to make propietary extensions which only officially work in Adobe software. Hence third-party or open-source pdf implementations often have to kludge together spit-and-tape solutions to maintain parity with Adobe software, some doing a better job than others. The Apple PDF pipeline implementation is, as far as I know, spec compliant but generally does not stray outside said spec. Remember also that the Mongoose PDFs are in fact not spec-compliant, and implement some very curious PDF construction methods.

Secondly, the Jpeg2000 format is a mess with capital M. Developed to be a successor to JPEG, it was inept in its initial form and didn't even properly support ICC definition or transparency. So in order to be able to compete with PNG and the like they hacked together the JPX extension format. Take on top of that that the various encoding methods (kadaku, GDAL, etc..) are not 100% compatible and you get a guaranteed headache when dealing with this format.

So Mongoose produces out-of-spec PDFs (again, not unheard of in graphically rich PDFs but should ideally be avoided, especially for future-proofing), uses an image format (for some pictures) that is hardly ideal and also has a very peculiar PDF image pipeline where images are a messy mix of JPEG2000, PNG, JPG and TIFF files (pictures with transparency seem to build out of three layers: a mask, a shadow mask and the actual image and sometimes this consists of a PNG + JPG +JPG, other times of greyscale colourspace referencing JPEG2000s + JPG, sometimes of another combination). So to me it seems that their PDF output pipeline is weirdly setup when it comes to image handling and produces messy unoptimised files.

Now, can they easily change this? I have no idea, and I would also not be too quick to say that 'it can be done in 15 minutes'. Depending on how their pdf workflow is implemented this can take a lot of refactoring. On the other hand, the pdf for 'Prometheus' has no such issues whatsoever, and looking at a raw image extract there is not a single JPEG2000 in sight, and likely because of this the pdf is spec compliant according to GhostScript. So clearly their pipeline is capable of producing good PDFs that do not rely on that messy image format and can work cleanly on all platforms, so why this disparity between documents?

And as an aside, up to version 10 at least Windows also could not handle Mongoose PDFs properly as the internal rendering pipeline could not deal with the CMYK-referencing JPEG2000 images either apparently, so not an Apple-only problem. See the screenshot below that shows the thumbnails for the untouched Rim Expeditions PDF and the one run through the 'solution' I offered above (taken on a Windows 10 Enterprise system):



I don't know if Windows 11 handles this better (no access to a W11 system), but the big difference is that Windows does not have a native PDF reader and you are forced to use third-party solutions which usually use their own internal rendering pipelines. And these often try to accommodate non-spec issues, hence you rarely encounter this problem in Windows-land (though in fact I have seen some comments on DT that indicate that at least some 3rd party readers cannot handle the Traveller PDFs properly either).

 

[MongooseMatt]

Hey everyone,

We are going to try something this coming week - please stand by, and thank you for your patience.

 

[Timolution]

Hey everyone,

We are going to try something this coming week - please stand by, and thank you for your patience.

Thank you very much for entering this minefield of a thread! And I hope my posts don't come off as beligerent or demanding... Truth is that I was just curious why sometimes MGT2 PDFs work without issue on the native Mac/iOS reader and other times have these issues of 'invisible' images. And once I went through the rabbit hole I couldn't stop anymore

If my findings have some value for your team moving forward, then great! But if it woud be too disruptive to your workflow to change how you do things, don't worry too much about it I would say. We do have 3rd party options that display the PDFs as they should look, so this is probably not the most urgent of matters.

 

[apastuszak]

Hey everyone,

We are going to try something this coming week - please stand by, and thank you for your patience.

Thank you Matt. Hopefully it's a simple tweak to the export settings.

I do all my typesetting and layout in an open source app called Scribus. I have used it to generate PDFs that would not display properly on Apple software. I've made other PDFs that look fine on the screen, but don't print properly on my printers. I have managed to fix almost all those isues by changing the export settings and making a new PDF.

Now, my documents are not 200+ page PDFs, so I can't say the same thing applies here.

 

[apastuszak]

Any updates on this? Do we have a fix?

 

[Vormaerin]

There was discussion in another thread (the rim expeditions, maybe) where folks where saying that it was fixed for that book.

Edit: hmm. Not there. But I remember folks talking about it working now and confirming that some blank spots were blank in both apple and adobe style readers..

 

[Timolution]

As @Vormaerin said the last update of the World Builders book did indeed fix most (but not all) of the images for PDFKit-based readers. Some images were still CMYK-colourspace refering JPEG2000s, but the majority have been converted to an sRGB space and hence are showing.

That said, I keep repeating that if you are on a Mac or iOS device you can use other readers that use a different rendering engine to completely side-step this issue (no affiliation, but I really like PDF Viewer by PSPDFKit GHMB on the former, and PDF Pro by Dominic Rodemer on the latter; both using the PSPDFKit renderer which has no issues with the Mongoose PDFs). Adobe Reader also works (mostly), but is such a bloated and slow piece of dung that I generally recommend people to avoid that one.

 

[apastuszak]

PDF viewer and PDF Pro are subscription-based apps, and not really an option.

i understand the issue ultimately lies with Apple to fix this. But of all of purchased PDFs, only the Mongoose PDFs are giving me problems. As others have pointed out, there plenty of complaints from Apple users about this.

I would like every Apple user to leave Apple feedback about this and point the, to this thread. Squeaky wheel gets the grease.

Feedback - Books for iOS and iPadOS - Apple

 

[MongooseMatt]

Has anyone found any issues with Aliens of Charted Space 4?

 

[Timolution]

Has anyone found any issues with Aliens of Charted Space 4?

I do read in a different application, but quickly scrolling through the PDF in the native PDF reader on macOS after buying there did initially not seem to be any missing images this time.

Putting the PDF side by side in the native reader and one using a different renderer I do however notice that the small planet pictures in the sector maps are missing (confirmed by raw image extraction that these are CMYK referencing JPEG2000s). But to be honest that is such a minor issue imo, and I am more than happy to report that the rest of the beautiful art in the PDF is now actually showing.

Native macOS/iOS renderer​	3rd Party PSPDFkit Renderer​

 

[CordwainerFish]

Well, the planet images do let you know at a glance if you can wilderness refuel there. I still have to run through a mnemonic to pull that out of a UWP.