[cairo] [patch] type1-subset: always subset subroutines 0-3 (Flex/hint replacement)

[David Maxwell]

PDF files created with cairo that subsetted Type 1 font CMMI12.pfb caused the OS X printing filter /usr/libexec/cups/filter/cgpdftops to crash in a method TType1Font::CheckSubrs.  

The font (Computer Modern Math Italic) is a mathematics font pervasively used by LaTeX.  

The issue relates to subroutines 0-3, which have a special meaning with respect to the Type 1 Flex/hint replacement mechanism.  These subroutines are called indirectly via 'callothersubr' commands, which execute routines on the PostScript stack.  As a consequence, it cannot be determined until runtime on any particular PostScript interpreter if subroutines 0-3 will be called; this impacts font subsetting.  

The patch addresses the issue by including subroutines 0-3 in all cases.  In theory, I suppose that one should include all subroutines if there are any 'callothersubr' calls.  This seems too conservative, and I suspect that including subroutines 0-3 is probably enough nearly all of the time.  

Interestingly, the crashes in cgpdftops occurred even when the font subset only contained two glyphs, neither of which actually called callothersubr.  Presumably TType1Font::CheckSubrs is doing something with subroutines 0-3 simply based on the presence on the presence of an /OtherSubrs array corresponding to Adobe's standard code for Flex/hint replacement.  This shows that subroutines 0-3 need to be included even if no glyph actually uses them!

David Maxwell
Associate Professor
Department of Mathematics and Statistics
University of Alaska Fairbanks

From 895d2cb7f024b24138760b515033cbdb29b52487 Mon Sep 17 00:00:00 2001
From: David Maxwell <damaxwell at alaska.edu>
Date: Sun, 7 Oct 2012 00:01:53 -0800
Subject: [PATCH] type1-subset: always subset subroutines 0-3 (Flex/hint
 replacement)

---
 src/cairo-type1-subset.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/cairo-type1-subset.c b/src/cairo-type1-subset.c
index ba0b229..2ee40e6 100644
--- a/src/cairo-type1-subset.c
+++ b/src/cairo-type1-subset.c
@@ -1309,6 +1309,12 @@ skip_subrs:
 	    return status;
     }
 
+    /* Always include the first four subroutines in case the Flex/hint mechanism is
+       being used. */
+    for( i=0; i<MIN(font->num_subrs,4); i++){
+        font->subrs[i].used = TRUE;
+    }
+
     closefile_token = find_token (dict_end, font->cleartext_end, "closefile");
     if (closefile_token == NULL)
 	return CAIRO_INT_STATUS_UNSUPPORTED;
-- 
1.7.11.1