[HarfBuzz] pango opentype (harfbuzz) - huge memory waste (DejaVu font)

[Behdad Esfahbod]

Hi nsf!

Originally I thought "well, memory allocated but never accessed does not use
resources".  But testing proved otherwise.

After some analysis, I actually removed the code causing the bloat.  Thanks
for pointing out!  Oh, for me, with DejaVu Sans, this causes an 8MB
difference.  Am I crazy or what?!

I've committed my change to Pango for now. Will push into HarfBuzz soon.

Cheers,

behdad

nsf wrote:
> Hi. I was experimenting with gtk hello world app and notices that it
> eats a lot of memory.
> 
> --------------------------------------------------------------------
> [nsf @ ~]$ pmap -d `pidof hello`
> 4397:   ./hello
> Address   Kbytes Mode  Offset           Device    Mapping
> 08048000       4 r-x-- 0000000000000000 003:00004 hello
> 08049000       4 rwx-- 0000000000000000 003:00004 hello
> 08192000    1456 rwx-- 0000000008192000 000:00000   [ anon ]
> b6be7000    6240 rwx-- 00000000b6be7000 000:00000   [ anon ]
> b71ff000     608 r-x-- 0000000000000000 003:00003 DejaVuSans.ttf
> b7297000       8 r-x-- 0000000000000000 003:00003 pango-basic-fc.so
> < ... skipped ... >
> mapped: 22608K    writeable/private: 8088K    shared: 764K
> --------------------------------------------------------------------
> 
> Notice this anonymous memory blob which eats 6 megs. I started to
> look for a problem. And memory allocs profiling got me to "_hb_alloc"
> function with that kind of stack:
> 
> --------------------------------------------------------------------
> Leaked 0xb703a008 (262140 bytes)
> 	_hb_alloc()
> 	_HB_OPEN_Load_ClassDefinition()
> 	_HB_OPEN_Load_EmptyOrClassDefinition()
> 	Load_ChainContextSubst2()
> 	.L1434()
> 	_HB_OPEN_Load_LookupList()
> 	HB_Load_GSUB_Table()
> 	pango_ot_info_get_gsub()
> 	get_tables()
> 	pango_ot_info_find_script()
> 	pango_ot_ruleset_new_for()
> 	pango_ot_ruleset_new_from_description()
> 	pango_ot_ruleset_get_for_description()
> 	basic_engine_shape()
> 	_pango_engine_shape_shape()
> 	pango_shape()
> 	shape_run()
> 	process_item()
> 	pango_layout_check_lines()
> < ... skipped ... >
> 	main()
> 	__libc_start_main()
> 	_start()
> --------------------------------------------------------------------
> 
> There were few memory allocs with size of 262140 bytes, which is 
> (4 * 65535) exactly. I kept digging and found out where it was:
> pango/opentype/harfbuzz-open.c
> pango/opentype/harfbuzz-gsub.c
> pango/opentype/harfbuzz-gpos.c
> 
> The problem is that there is an allocation of memory "enough" for
> loading something (see function Load_ChainContextSubst2()), but there is
> no realloc later. I checked the real usage of memory and it was
> something like: 400 bytes out of total 262140! I mean literally it's a
> waste of 256 kilobytes per such table alloc (or whatever it is). 
> 
> So, I did a quick fix (patch attached) in function
> _HB_OPEN_Load_ClassDefinition(). I added a realloc with a real used
> size of array. And puff.. the problem is gone. I did LD_PRELOAD and
> here is the results:
> 
> --------------------------------------------------------------------
> [nsf @ ~]$ pmap -d `pidof hello`
> 21033:   ./hello
> Address   Kbytes Mode  Offset           Device    Mapping
> 08048000       4 r-x-- 0000000000000000 003:00004 hello
> 08049000       4 rwx-- 0000000000000000 003:00004 hello
> 0888c000    1432 rwx-- 000000000888c000 000:00000   [ anon ]
> b71ab000     608 r-x-- 0000000000000000 003:00003 DejaVuSans.ttf
> b7243000       8 r-x-- 0000000000000000 003:00003 pango-basic-fc.so
> < ... skipped ... >
> mapped: 16348K    writeable/private: 1824K    shared: 764K
> --------------------------------------------------------------------
> 
> No more strange anonymous bloated chunks. After that I did few tests of
> course. I tried different apps like: linuxdcpp, dia, sonata,
> claws-mail, midori (webkit-gtk) and everything works. I tried to reboot
> and load few programs with patch and then reboot again and do the same 
> without patch, memory usage was something like 130 megs versus 200 megs
> (~30%). And I couldn't believe, but!
> 
> Later I found out that this issue is valid only for DejaVuSans.ttf font
> for some reason (I didn't check other fonts). For example Bitstream
> Vera Sans doesn't cause that bloat.
> 
> Also I'm not sure is my hack correct and bug free. But it clearly works
> on my machine. Patch was made against SVN version of pango and affects
> only opentype component (one file, 3 functions).
> 
> Well, I don't know, maybe someone can comment that issue? :)
> Thanks for attention.
> 
> P.S. It's a cross post from gtk-i18n-list. And here are few things: I
> noticed on harfbuzz's web that Qt uses it too, and this waste probably
> true for Qt too (Qt's hello world apps eats a lot of memory on my
> machine too, same font).
> 
> nsf
> 
> 
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> HarfBuzz mailing list
> HarfBuzz at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/harfbuzz