Solution to Traveller PDFs not showing images in PDFKit based renderers (macOS and iOS)

[NOLATrav]

There are plenty of tools that will go through a PDF and convert all the images to RGB JPEG. That should fix the problem.

While that may be true it should not be on the consumer to fix the problem, which is sort of the genesis of this thread. If the solution is to convert all images to RGB for digital publication, vs CMYK for print, then, well... some poor hero at Mongoose should be tasked with that. And should they do so, they would be loudly and fully feted and their name never forgotten!

 

[ThornPlutonius]

Yes, can confirm that this works perfectly!

It is true that you lose the TOC/Bookmarks this, but it is in fact very easy to restore these from the original PDF! You will need the following:

1. The original Mongoose PDF (mong.pdf in explanation below)
2. Your 'fixed' pdf using apastuszak's pdftocairo method (fixed.pdf in explanation below)
3. pdftk

If you haven't already, first install the Java 'port' of pdftk using homebrew:

brew install pdftk-java

This will also install any necessary dependencies (mainly openjdk).

When this is installed it becomes a matter of extracting the TOC/bookmark data (well, there's a bit more in there than just that, but we're mainly interested in the bookmarks) for the original Mongoose PDF and merging this data with the fixed pdf file.

First you extract the data*:

pdftk /path/to/mong.pdf dump_data_utf8 output data.txt

Then you merge the extracted data with the pdftocairo output file:

pdftk /path/to/fixed.pdf update_info_utf8 /path/to/data.txt output Fixed_Bookmarks.pdf

Et voilà, the Fixed_Bookmarks.pdf will now be your desired fixed pdf with all the original bookmarks intact!

*Note that you can now easily manipulate this data file in a text editor in order to add your own bookmarks or to change the nesting level of existing bookmarks.

Yeah...no. I just tried this, but am not sufficiently familiar with terminal to get it to actually work. So, Thank you for the advice, but...

Curses upon Mongoose for producing such inferior pdfs for their customers, who already pay too bloody much for Mongoose products. I take this continuing pdf problem as an indication of Mongoose's contempt for their customers. This customer will just live happily with the non-Mongoose "traveller" rules sets and leave Mongoose behind, shaking the dust off of my shoes as I walk away.

 

[Vormaerin]

If that's what you want to do. I just find it weird that *APPLE* has had reports of this problem for close to a decade that I am aware of and are just like "well, our customers don't need to read those kinds of pdf images." But it's definitely the tiny niche company making inferior pdfs and disrespecting their customers, not the megacorp making an inferior pdf reader.

Yes, Mongoose should jump through whatever hoops are necessary to stop this from happening if at all possible (which, as far as I can tell, is possible). But the idea that they are the ones deserving the rage is ridiculous.

 

[apastuszak]

If that's what you want to do. I just find it weird that *APPLE* has had reports of this problem for close to a decade that I am aware of and are just like "well, our customers don't need to read those kinds of pdf images." But it's definitely the tiny niche company making inferior pdfs and disrespecting their customers, not the megacorp making an inferior pdf reader.

Yes, Mongoose should jump through whatever hoops are necessary to stop this from happening if at all possible (which, as far as I can tell, is possible). But the idea that they are the ones deserving the rage is ridiculous.

Well, since I own hundreds of RPG PDFs and only the Mongoose ones have ever given me trouble, I think the problem lies on both sides. Sure, Apple's PDFKit renderer is failing to render Mongoose PDFs correctly. But Mongoose PDFs are not compliant.

I have been converting PDFs to greyscale for use on my Kindle Scribe. My PDFs usually decrease in size by 50% when I do this. The Mongoose Core Rulebook 2022 Update went from 48 MB to 751 MB in size. Instead of getting 50% smaller, it got 1500% bigger.

The gold standards for me for RPG PDFs right now are the Cyberpunk rulebooks from R. Talsorian Games and Savage Worlds by Pinnacle Entertainment.

The Cyberpunk books are full of hyperlinks all over the place. The TOC page is all hyperlinks. And if any section of the book references another section of the book, then there's a hyperlink there.,

And the Savage Worlds PDFs have hyperlinks AND layers. I can turn off the layer for the background and just have black text on a white background.

 

[CordwainerFish]

My PDFs usually decrease in size by 50% when I do this. The Mongoose Core Rulebook 2022 Update went from 48 MB to 751 MB in size. Instead of getting 50% smaller, it got 1500% bigger.

Now that's a head scratcher. What the hell are ghostscript and poppler doing?

 

[apastuszak]

Now that's a head scratcher. What the hell are ghostscript and poppler doing?

I didn't use Poppler. I went straight from original PDF to greyscale using ghostscript.

On my second attempt I went through poppler first and got a 300+ MB PDF. I then used Ghostscript to make that greyscale, and got the file down to about 28 MB.

And I did this on the latest Jun_23 PDF that I downloaded from Mongoose's website.

So, now I'm using a workaround of a workaround.

It is what it is, I guess. PDFs are a finicky beast.

 

[Vormaerin]

Well, since I own hundreds of RPG PDFs and only the Mongoose ones have ever given me trouble, I think the problem lies on both sides. Sure, Apple's PDFKit renderer is failing to render Mongoose PDFs correctly. But Mongoose PDFs are not compliant.

I have been converting PDFs to greyscale for use on my Kindle Scribe. My PDFs usually decrease in size by 50% when I do this. The Mongoose Core Rulebook 2022 Update went from 48 MB to 751 MB in size. Instead of getting 50% smaller, it got 1500% bigger.

The gold standards for me for RPG PDFs right now are the Cyberpunk rulebooks from R. Talsorian Games and Savage Worlds by Pinnacle Entertainment.

The Cyberpunk books are full of hyperlinks all over the place. The TOC page is all hyperlinks. And if any section of the book references another section of the book, then there's a hyperlink there.,

And the Savage Worlds PDFs have hyperlinks AND layers. I can turn off the layer for the background and just have black text on a white background.

Sure, Mongoose is definitely doing something that Apple PDF readers don't like. I'm not sure why they are doing it that way or how difficult it would be to fix it.

I'm just saying that I had this problem years ago with making PDFs from the books published by the psychologist I worked for. She used Apple and I didn't, so the pdf conversions I did worked fine for me. And not when she opened them. And when I google the issue *then* it was already a known issue.

Obviously, this is a situation where Apple isn't going to GAF, because they never have. So Mongoose needs to change their procedures if they want to avoid this constantly happening. I was just objecting to that other poster's suggestion that Mongoose hated their customers because Apple's PDF readers can't do things that everyone else's can.

 

[apastuszak]

Sure, Mongoose is definitely doing something that Apple PDF readers don't like. I'm not sure why they are doing it that way or how difficult it would be to fix it.

I'm just saying that I had this problem years ago with making PDFs from the books published by the psychologist I worked for. She used Apple and I didn't, so the pdf conversions I did worked fine for me. And not when she opened them. And when I google the issue *then* it was already a known issue.

Obviously, this is a situation where Apple isn't going to GAF, because they never have. So Mongoose needs to change their procedures if they want to avoid this constantly happening. I was just objecting to that other poster's suggestion that Mongoose hated their customers because Apple's PDF readers can't do things that everyone else's can.

Well, this has been a known issue with Mongoose PDFs for years now. The fact that a brand new book just came out this week and has PDF rendering issues on Apple devices would definitely alienate Apple users.

This is probably an Apple issue. Though some suggest on other forums that Apple's PDF renderer, PDFKit, is PDF spec compliant, and the things that cause these errors are not compliant, and the other renderers have worked around the issue. I'm not familiar enough with the issue to speak to that.

One thing I know is that PDFs "fixed" to work on Apple products render just fine on non-Apple products. So, if you shoot for iPad compatibility, then you'll get a file that works everywhere.

With the Poppler conversion, this is less of an issue. But Poppler is only a partial solution, because you lose all hyperlinks. You can copy the TOC over, but that's about it.

Would it have killed someone to open the Free Trader Beowulf PDF on an iPad and just flip through the pages to make sure everything looks OK? They could have still released it with a note that says there's a problem with Apple devices and they're working on a fix.

Or even a note to say that MacOS/iPadOS/iOS users should use PDF Viewer or PDF Pro to read these PDFs on the download page?

 

[CordwainerFish]

Or even a note to say that MacOS/iPadOS/iOS users should use PDF Viewer or PDF Pro to read these PDFs on the download page?

I don't consider that acceptable. My iDevices come with a vendor-contributed program which works on all standards conformant PDFen.

 

[Vormaerin]

Well, this has been a known issue with Mongoose PDFs for years now. The fact that a brand new book just came out this week and has PDF rendering issues on Apple devices would definitely alienate Apple users.

This is probably an Apple issue. Though some suggest on other forums that Apple's PDF renderer, PDFKit, is PDF spec compliant, and the things that cause these errors are not compliant, and the other renderers have worked around the issue. I'm not familiar enough with the issue to speak to that.

One thing I know is that PDFs "fixed" to work on Apple products render just fine on non-Apple products. So, if you shoot for iPad compatibility, then you'll get a file that works everywhere.

With the Poppler conversion, this is less of an issue. But Poppler is only a partial solution, because you lose all hyperlinks. You can copy the TOC over, but that's about it.

Would it have killed someone to open the Free Trader Beowulf PDF on an iPad and just flip through the pages to make sure everything looks OK? They could have still released it with a note that says there's a problem with Apple devices and they're working on a fix.

Or even a note to say that MacOS/iPadOS/iOS users should use PDF Viewer or PDF Pro to read these PDFs on the download page?

I agree that Mongoose should have bit the bullet and fixed this a long time ago. I can only assume they are trying to keep a single master file for each book so there aren't versioning issues between the pdf and print book. But it could be something else. I obviously don't work for Mongoose, so I don't know what actually goes on in their tiny office. Regardless, they aren't going to beat Apple on this, so they need to address it. Just like I had to do a bunch of extra work those years ago when I first encountered this issue.

I'm also sure that Apple's PDF reader meets the minimum standards for pdf reading, or someone would have made a big deal about that. It's still the only pdf reader I'm aware of that doesn't have the necessary features to read the print optimized image types.

 

[apastuszak]

I don't consider that acceptable. My iDevices come with a vendor-contributed program which works on all standards conformant PDFen.

Well, sometimes you need to use a non-standard app. Using the apps your device comes with, and not installing additional software is referred to as "the tyranny of the default." And it's a bad thing. It prevents people from finding new apps that do things better than the app they come with. It's the reason everyone in the US uses texting rather than far superior apps on their phones top communicate with one another..

The problem we have here goes deeper than that, unfortunately. PDFKit is part of the OS. And almost every PDF Reader/PDF Editor on Apple uses PDFKit, because it's there and built in. So, as an Apple user, you really need to do your homework to find a PDF reader/editor that does not use PDFKit if you want to use one of these PDFs.

 

[apastuszak]

I agree that Mongoose should have bit the bullet and fixed this a long time ago. I can only assume they are trying to keep a single master file for each book so there aren't versioning issues between the pdf and print book. But it could be something else. I obviously don't work for Mongoose, so I don't know what actually goes on in their tiny office. Regardless, they aren't going to beat Apple on this, so they need to address it. Just like I had to do a bunch of extra work those years ago when I first encountered this issue.

I'm also sure that Apple's PDF reader meets the minimum standards for pdf reading, or someone would have made a big deal about that. It's still the only pdf reader I'm aware of that doesn't have the necessary features to read the print optimized image types.

I am curious why no other RPG publisher has this issue. Are they testing on Apple hardware before release, or do their PDF renders work without special tweaking for Apple devices?

I open an issue for this with Apple every time they release a new version of an operating system. As much as I think Mongoose needs to do something about this, I am also pestering Apple about this.

 

[ThornPlutonius]

I am curious why no other RPG publisher has this issue. Are they testing on Apple hardware before release, or do their PDF renders work without special tweaking for Apple devices?

I open an issue for this with Apple every time they release a new version of an operating system. As much as I think Mongoose needs to do something about this, I am also pestering Apple about this.

No other RPG publisher has this issue because they understand how to create a pdf that is universally readable. Mongoose either does not, or is unwilling to do what is necessary. This is not an Apple problem. This is a Mongoose problem, a problem that only Mongoose seems to have.

 

[MongooseMatt]

Okay, we have been working on this, and think we have it nailed. We have just updated Wrath of the Ancients (and have already done so with Fifth Frontier War), so if anyone with iOS readers who already has these PDFs could give them a whirl and let us know how they look, we would be grateful.

We are making a few tweaks on This is Free Trader Beowulf and will be updating that to the same specs tomorrow morning.

 

[Yatima]

Okay, we have been working on this, and think we have it nailed. We have just updated Wrath of the Ancients (and have already done so with Fifth Frontier War), so if anyone with iOS readers who already has these PDFs could give them a whirl and let us know how they look, we would be grateful.

We are making a few tweaks on This is Free Trader Beowulf and will be updating that to the same specs tomorrow morning.

In my copy of FFW (viewed in Preview on my Mac) there were images missing on pages 124 and 184 – I just redownloaded the book and both images are now showing correctly. I'll have a look at This is Free Trader Beowulf when it's released tomorrow.

Thanks for resolving this.

 

[MongooseMatt]

Thanks for resolving this.

Deepest apologies for having taken this long - but yes, I think we have cracked it. If all goes well with Beowulf tomorrow, expect a mass update over the coming days/weeks of all our other books affected by this.

 

[NOLATrav]

Cheers, Matt - good on ya!!

 

[CordwainerFish]

Deepest apologies for having taken this long - but yes, I think we have cracked it. If all goes well with Beowulf tomorrow, expect a mass update over the coming days/weeks of all our other books affected by this.

Sweet - even if it means clicking on all my download links (and there's a bunch of them) to see if there's dates at the end.

 

[ThornPlutonius]

Deepest apologies for having taken this long - but yes, I think we have cracked it. If all goes well with Beowulf tomorrow, expect a mass update over the coming days/weeks of all our other books affected by this.

Thank you!

 

[apastuszak]

1. Thank you!!!!
2. What did you do? I work in IT. I'm curious about the technical details.