[gst-devel] reducing memory usage in avidemux 8and other demuxers)

[Stefan Kost]

hi,

I have filed
http://bugzilla.gnome.org/show_bug.cgi?id=407433

would be nice if people with lots of avis could run this. This will  
check if a certain logline gets triggered. Thus would be the  
indication that there are avi's where this code is used. I havent  
found one so far.

Stefan

Quoting Stefan Kost <ensonic at hora-obscura.de>:

> hi,
>
> avidemux currently sucks quite a lot. One example, I have an avi that
> is 300 mb. Unfortunately it has 446774 index entries. One index entry
> uses 52 bytes. Therefore the index needs 23232248 bytes, that is 22.15
> Mb. I repacked that structure a bit and saved 4 bytes. This reduces
> memory consumtion to 21445152 20.45 Mb. Now the bigger problem is that
> the index is first generated as a list and then copied into an array.
> Thus the peak memory consumtion even doubles. I currently check if we
> can avoid that at all.
>
> Beside that I have some more ideas. This is the current structure:
> typedef struct {
>    guint   index_nr;      //
> =(entry-index_entries)/sizeof(gst_avi_index_entry);
>    guchar  stream_nr;
>    guchar  flags;
>    guint64 ts;
>    guint64 dur;           // =entry[1].ts-entry->ts
>    guint64 offset;
>    guint64 bytes_before;  // calculated
>    guint32 frames_before; // calculated
>    guint32 size;          // could be read from the chunk (if we don't split)
> } gst_avi_index_entry;
>
> as we always know the baseaddress of the indexentries, we could drop
> index_nr and calculate it. same for duration (would require to append
> a dummy entry). If we decide to add a configure switch to gstreamer
> core like e.g. --enable-memory-optimization, we could define
> conditional macros:
> #ifdef GST_ENABLE_MEMORY_OPTIMIZATION
> #define GST_AVI_INDEX_ENTRY_NR(entry) \
>    (entry-index_entries)/sizeof(gst_avi_index_entry)
> #else
> #define GST_AVI_INDEX_ENTRY_NR(entry) \
>    entry->index_nr
> #endif
> In the memory optimization case that would save 10 bytes per entry.
>
> If the target platform does not support files>4GB (#ifndef
> __USE_LARGEFILE64) offset, bytes_before can be defined as guint32.
> That would save another 8 bytes.
>
> In the avove example this would reduce the memory consumtion to 12.78 Mb.
>
> Finally I notice that we use g_malloc and g_new. Imho it would be
> better, that whenever we read stuff from media and according to that
> do mallocs, to use g_try_new and g_try_mallo and handle oom situation.
> This does not guarantee that the app gets not terminated because of
> oom, but makes it a lot less like likely.
>
> Any comments? Is there simillar stuff like the gst_avi_index_entry for
> other demuxers like ogg, mpeg?
>
> Stefan
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your   
> job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>