Solution to Traveller PDFs not showing images in PDFKit based renderers (macOS and iOS)

[Timolution]

I found another solution.

This requires that you install homebrew (https://brew.sh). Once you install Homebrew, you install the package Poppler.

 brew install poppler

The from the command line, you run:

pdftocairo -pdf mongoose_pdf.pdf mongoose_pdf_repaired.pdf

Yes, can confirm that this works perfectly!

Pdftocairo worked well for me, only it killed the bookmarks and hyperlinks. And then I ran it through a PDF compressor. PDF Expert to be precise. Will have to look at some of the PDFs I had issues with and re-mux and compress them and see how important the bookmarks are..

It is true that you lose the TOC/Bookmarks this, but it is in fact very easy to restore these from the original PDF! You will need the following:

1. The original Mongoose PDF (mong.pdf in explanation below)
2. Your 'fixed' pdf using apastuszak's pdftocairo method (fixed.pdf in explanation below)
3. pdftk

If you haven't already, first install the Java 'port' of pdftk using homebrew:

brew install pdftk-java

This will also install any necessary dependencies (mainly openjdk).

When this is installed it becomes a matter of extracting the TOC/bookmark data (well, there's a bit more in there than just that, but we're mainly interested in the bookmarks) for the original Mongoose PDF and merging this data with the fixed pdf file.

First you extract the data*:

pdftk /path/to/mong.pdf dump_data_utf8 output data.txt

Then you merge the extracted data with the pdftocairo output file:

pdftk /path/to/fixed.pdf update_info_utf8 /path/to/data.txt output Fixed_Bookmarks.pdf

Et voilà, the Fixed_Bookmarks.pdf will now be your desired fixed pdf with all the original bookmarks intact!

*Note that you can now easily manipulate this data file in a text editor in order to add your own bookmarks or to change the nesting level of existing bookmarks.

 

[BangsNaughtyBits]

Almost perfect. The hyperlinks on page two in the book's table of contents are still stripped. The PDF Table of Contents is all there for use in a side bar.

 

[Timolution]

Almost perfect. The hyperlinks on page two in the book's table of contents are still stripped. The PDF Table of Contents is all there for use in a side bar.

Ah, ok, you meant the internal hyperlinks. Yeah, I'm not sure how to easily transfer those programmatically unfortunately (if at all possible). But as you mentioned that you use PDF Expert it is easy, though a bit time-consuming, to add these yourself.

In the application, make sure to show the outline option in the left sidebar as this will show you the relative page numbers for each TOC item. And then it is just a matter of going into the link tool, selecting an entry on the textual tabel of contents and putting in the correct relative page number. Tedious, but I do not immediately see another way to achieve this.

 

[saltedcaramelplease]

Hey everyone,

We are going to try something this coming week - please stand by, and thank you for your patience.

Did Matt ever follow up on this?

 

[apastuszak]

There has to be some way to toss this PDF in Acrobat Pro and have it convert all the images to a format that will work on an iPad. Unfortunately, I don't have access to Acrobat Pro. I guess I could download a copy and do the free 7 day trial.

But the problem with that is that Acrobat is not a free tool that everyone has access to. pdftocairo is, but as pointed out, you lose hyperlinks.

I would love a PDF with hyperlinks all over the place. The current Cyberpunk Red rulebook does that, and it's insanely convenient. I wish there was a way to crowdsource this kind of thing, where people could create hyperlinks and then export them so others could import them.

 

[CordwainerFish]

The current version of Ghostscript appears to have serious issues - it produces ridiculously large .pdfs, like 40 times the size of the original. It also takes longer to process each page than the last. Further. it bobbles the picture on page 009 of The Fifth Frontier War. Poppler has the courtesy of only doubling the file size and runs much faster.

 

[apastuszak]

Why is this still an issue? Haven't we confirmed the issue is with JPEG2000 images with CMYK profiles? You should be able to run this through Acrobat Pro's optimizer and tell it to convert all images to JPEG and be done with it.

 

[CordwainerFish]

Why is this still an issue? Haven't we confirmed the issue is with JPEG2000 images with CMYK profiles? You should be able to run this through Acrobat Pro's optimizer and tell it to convert all images to JPEG and be done with it.

I do not have and will not get Acrobat Pro.

 

[BangsNaughtyBits]

I took my recent purchases like FFW and processed two versions: one with GhostScript (whatever I had installed) and one with Poppler. Then I set them up side by side in PDF Expert, the original PDF and the Ghostscript version and manually pulled in pages from Ghostscript processed into the original unless I noticed it had issues, then pulled the page from the Poppler processed version. Took forever, but I manually excluded the malformed transparencies and kept the hyperlinks intact. Processed three PDFs that way, including HighGuard 2022 which took longer because it wasn't always obvious when a bit of a layer was missing. Especially fuel tanks.

Apple or someone needs to look at this because it's maddening and only seems to be an issue for me with Mongoose.

And I think I'll link to this thread from some other Traveller-centric site and also copy the short versions of do this then that and you are good there as well as I just reformated my Mac and had to go digging all over again.

!

 

[Timolution]

I took my recent purchases like FFW and processed two versions: one with GhostScript (whatever I had installed) and one with Poppler. Then I set them up side by side in PDF Expert, the original PDF and the Ghostscript version and manually pulled in pages from Ghostscript processed into the original unless I noticed it had issues, then pulled the page from the Poppler processed version. Took forever, but I manually excluded the malformed transparencies and kept the hyperlinks intact. Processed three PDFs that way, including HighGuard 2022 which took longer because it wasn't always obvious when a bit of a layer was missing. Especially fuel tanks.

May I aks why you are specifically going through this complex exercise? Running the offending PDF through poppler and then reimporting the ToC through pdftk (and maybe do some light compression on the resulting file) will likely give you better* results faster.

* Swapping in and out pages in the original will destroy the ToC and likely also break internal links, so I’m not sure what you are gaining by going this route?

The current version of Ghostscript appears to have serious issues - it produces ridiculously large .pdfs, like 40 times the size of the original. It also takes longer to process each page than the last. Further. it bobbles the picture on page 009 of The Fifth Frontier War. Poppler has the courtesy of only doubling the file size and runs much faster.

No idea what changed, but this is of course the nature of OSS Always somebody somewhere who wants to squash a bug/add functionality they consider important and then breaking stuff up- or downstream… Unless there is something odd about the FFW pdf itself. I had no real issues using the Poppler > pdftk > compress route (Save for noting that for some reason a lot of the ToC is duplicated in the original’s metadata, but that was an easy fix.)

You can install older versions of brew formulae (if they are still available in the repository) by unlinking and then installing a user-specified version. See here on how to do that.

 

[apastuszak]

@Timolution I took the latest June 2023 Core Update Book and ran it through Poppler. It went from 48 MB to 308 MB.

I wanted to use pdftk to move the TOC over, then. But when I run the export command, it doesn't produce a text file.

 

[BangsNaughtyBits]

Much thanks to Mongoose for releasing an update of The Fifth Frontier War today which looks to be fully functional under PDFKit. Page 19 in the text is a good example page.

 

[Timolution]

@Timolution I took the latest June 2023 Core Update Book and ran it through Poppler. It went from 48 MB to 308 MB.

I wanted to use pdftk to move the TOC over, then. But when I run the export command, it doesn't produce a text file.

Hmmmm, not really sure what goes wrong here. I tried the same just now and had no issues with extracting the TOC data from the original file nor with inserting them in the 'fixed' file...

It is true that the pdftocairo method will massively blow up the file-size (in my case also about 308 MB), but using PDF Squeezer with the following settings brought it back to a manageable 53 MB (the default light compression method of 300 dpi max and 80% image quality will bring it down further to about 38 MB, but the images will have ever so slightly worse quality compared to the original if you zoom in):

 

[ThornPlutonius]

I am very disappointed that Mongoose pdfs are still having image problems in MacOS and iOS/iPadOS platforms. I have hundreds of pdfs that were produced by many sources and they all correctly display images in my readers. What does Mongoose Publishing not understand about producing pdfs that will universally display correctly? The new "This Is Free Trader Beowolf" continues their legacy of failed image content in pdfs. Please get your act together, Mongoose. For the prices that you charge, there should be NO such errors/problems with your products. Period. We, the purchasers, should not have to go to such efforts to get your products to be what they should be BEFORE you make them available for sale.

 

[apastuszak]

Especially since people identified the exact problem in this thread. Some at Mongoose must own and iPad or Mac that they can test this stuff on. I get that is an "Apple Problem." But I don't have problems with PDFs from any other RPG publisher. Part of QA should be to test on an Apple device.

The iPad far outsells any other tablet and probably has more usage at gaming tables than any other tablet.

 

[apastuszak]

Hmmmm, not really sure what goes wrong here. I tried the same just now and had no issues with extracting the TOC data from the original file nor with inserting them in the 'fixed' file...

It is true that the pdftocairo method will massively blow up the file-size (in my case also about 308 MB), but using PDF Squeezer with the following settings brought it back to a manageable 53 MB (the default light compression method of 300 dpi max and 80% image quality will bring it down further to about 38 MB, but the images will have ever so slightly worse quality compared to the original if you zoom in):

View attachment 1763

I got the TOC to copy over, but it threw an error of some kind.

I'm converting the Core Rulebook to greyscale, to use on my Kindle Scribe using Ghostscript. Most of the other books I convert drop to half their normal size and take about 2-3 min to convert. The Traveller Core Rulebook 2022 update took about 15 minutes and grew to 758 MB.

Kince Send To Kind limits file sizes to 200 MB, this was not going to work. So, I ran it through poppler, which took a long time, copied over the TOC and then converted that file to greyscale and that file was only 27 MB.

There's definitely something odd about these PDFs.

 

[ThornPlutonius]

Especially since people identified the exact problem in this thread. Some at Mongoose must own and iPad or Mac that they can test this stuff on. I get that is an "Apple Problem." But I don't have problems with PDFs from any other RPG publisher. Part of QA should be to test on an Apple device.

The iPad far outsells any other tablet and probably has more usage at gaming tables than any other tablet.

And if they still do not own a Mac/iOS/iPadOS platform, they have a responsibility to bloody well obtain them in order to do the testing necessary to ensure that ALL of their customers have a problem-free experience with their products. This is fundamental.

 

[Evil Aardvark]

I love Mongoose products, I've bought a lot of them. I picked up the PDF of This is Free Trader Beowulf and it is completely unreadable on my Mac. It is, from my perspective, the worst this problem has ever been.

People have explained again and again the problem, and Mongoose say they're going to fix it but they don't. I've waited, trusted, and hoped and the problem is getting worse.

At this point, I don't know if it is apathy or malice, but I don't care any more. I'm tapping-out on Mongoose products because they don't want me as a customer.

 

[apastuszak]

And if they still do not own a Mac/iOS/iPadOS platform, they have a responsibility to bloody well obtain them in order to do the testing necessary to ensure that ALL of their customers have a problem-free experience with their products. This is fundamental,.

All Apple products use the same PDF rendering engine. You could test this on any iPhone an employee has. But @Timolution identified the problem. There are plenty of tools that will go through a PDF and convert all the images to RGB JPEG. That should fix the problem.

 

[ThornPlutonius]

I love Mongoose products, I've bought a lot of them. I picked up the PDF of This is Free Trader Beowulf and it is completely unreadable on my Mac. It is, from my perspective, the worst this problem has ever been.

People have explained again and again the problem, and Mongoose say they're going to fix it but they don't. I've waited, trusted, and hoped and the problem is getting worse.

At this point, I don't know if it is apathy or malice, but I don't care any more. I'm tapping-out on Mongoose products because they don't want me as a customer.

I am tending to agree with you.