Re: package / tmp file usage ...
Attila Szűcs
attila.szucs at collabora.com
Wed May 17 14:42:55 UTC 2023
i just re-created the situation, and here is a callstack (made in windows), when i save_as an ods file.
it writes (still uncomressed) data to a stream, but that stream is a file on my harddrive: C:\\Users\\Szucsi\\AppData\\Local\\Temp\\lu99163n3gde.tmp\\lu9916{8AA24C62-674C-4CFF-BB49-581F27614405}.tmp
> mergedlo.dll!utl::OSeekableInputStreamWrapper::seek(__int64 _nLocation) Line 179 C++
mergedlo.dll!OWriteStream::writeBytes(const com::sun::star::uno::Sequence<signed char> & aData) Line 2057 C++
expwraplo.dll!`anonymous namespace'::SaxWriterHelper::writeSequence() Line 231 C++
expwraplo.dll!`anonymous namespace'::SaxWriterHelper::convertToXML(const char16_t * pStr, long nStrLen, bool bDoNormalization, bool bNormalizeWhitespace, char * pTarget, unsigned long & rPos) Line 527 C++
expwraplo.dll!`anonymous namespace'::SaxWriterHelper::writeString(const rtl::OUString & rWriteOutString, bool bDoNormalization, bool bNormalizeWhitespace) Line 601 C++
expwraplo.dll!`anonymous namespace'::SaxWriterHelper::startElement(const rtl::OUString & rName, const com::sun::star::uno::Reference<com::sun::star::xml::sax::XAttributeList> & xAttribs) Line 711 C++
expwraplo.dll!`anonymous namespace'::SAXWriter::startElement(const rtl::OUString & aName, const com::sun::star::uno::Reference<com::sun::star::xml::sax::XAttributeList> & xAttribs) Line 1207 C++
mergedlo.dll!SvXMLExport::StartElement(const rtl::OUString & rName, bool bIgnWSOutside) Line 2111 C++
mergedlo.dll!SvXMLElementExport::SvXMLElementExport(SvXMLExport & rExp, const rtl::OUString & rQName, bool bIWSOutside, bool bIWSInside) Line 2460 C++
sclo.dll!ScXMLExport::WriteCell(ScMyCell & aCell, long nEqualCellCount) Line 3268 C++
sclo.dll!ScXMLExport::WriteTable(long nTable, const com::sun::star::uno::Reference<com::sun::star::sheet::XSpreadsheet> & xTable) Line 2962 C++
sclo.dll!ScXMLExport::ExportContent_() Line 1943 C++
mergedlo.dll!SvXMLExport::ImplExportContent() Line 1159 C++
mergedlo.dll!SvXMLExport::exportDoc(xmloff::token::XMLTokenEnum eClass) Line 1384 C++
sclo.dll!ScXMLExport::exportDoc(xmloff::token::XMLTokenEnum eClass) Line 5393 C++
mergedlo.dll!SvXMLExport::filter(const com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> & aDescriptor) Line 812 C++
sclo.dll!ScXMLExport::filter(const com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> & aDescriptor) Line 5431 C++
sclo.dll!ScXMLImportWrapper::ExportToComponent(const com::sun::star::uno::Reference<com::sun::star::uno::XComponentContext> & xContext, const com::sun::star::uno::Reference<com::sun::star::frame::XModel> & xModel, const com::sun::star::uno::Reference<com::sun::star::xml::sax::XWriter> & xWriter, const com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> & aDescriptor, const rtl::OUString & sName, const rtl::OUString & sMediaType, const rtl::OUString & sComponentName, const com::sun::star::uno::Sequence<com::sun::star::uno::Any> & aArgs, std::unique_ptr<ScMySharedData,std::default_delete<ScMySharedData>> & pSharedData) Line 711 C++
sclo.dll!ScXMLImportWrapper::Export(bool bStylesOnly) Line 947 C++
sclo.dll!ScDocShell::SaveXML(SfxMedium * pSaveMedium, const com::sun::star::uno::Reference<com::sun::star::embed::XStorage> & xStor) Line 569 C++
sclo.dll!ScDocShell::SaveAs(SfxMedium & rMedium) Line 1861 C++
...
and here is the value of 'this' in (OSeekableInputStreamWrapper::seek):
-this 0x00000272b1388610 {...} utl::OSeekableInputStreamWrapper * {utl::OStreamWrapper}
- [utl::OStreamWrapper] {...} utl::OStreamWrapper
- cppu::ImplInheritanceHelper<utl::OSeekableInputStreamWrapper,com::sun::star::io::XStream,com::sun::star::io::XOutputStream,com::sun::star::io::XTruncate> {...} cppu::ImplInheritanceHelper<utl::OSeekableInputStreamWrapper,com::sun::star::io::XStream,com::sun::star::io::XOutputStream,com::sun::star::io::XTruncate>
- utl::OSeekableInputStreamWrapper {...} utl::OSeekableInputStreamWrapper
- cppu::ImplInheritanceHelper<utl::OInputStreamWrapper,com::sun::star::io::XSeekable> {...} cppu::ImplInheritanceHelper<utl::OInputStreamWrapper,com::sun::star::io::XSeekable>
- utl::OInputStreamWrapper {m_aMutex={...} m_pSvStream=0x00000272a992ff10 {mxFileHandle=0x00000000000049b0 nLockCounter=0 aFilename=...} ...} utl::OInputStreamWrapper
+ utl::OInputStreamWrapper_Base {...} utl::OInputStreamWrapper_Base
+ comphelper::ByteReader {...} comphelper::ByteReader
+ m_aMutex {...} std::mutex
- m_pSvStream 0x00000272a992ff10 {mxFileHandle=0x00000000000049b0 nLockCounter=0 aFilename=u"C:\\Users\\Szucsi\\AppData\\Local\\Temp\\lu99163n3gde.tmp\\lu9916{8AA24C62-674C-4CFF-BB49-581F27614405}.tmp" ...} SvStream * {SvFileStream}
- [SvFileStream] {mxFileHandle=0x00000000000049b0 nLockCounter=0 aFilename=u"C:\\Users\\Szucsi\\AppData\\Local\\Temp\\lu99163n3gde.tmp\\lu9916{8AA24C62-674C-4CFF-BB49-581F27614405}.tmp" ...} SvFileStream
+ SvStream {m_xLockBytes=nullptr m_nActPos=0 m_pRWBuf=unique_ptr 205 'Í' ...} SvStream
mxFileHandle 0x00000000000049b0 void *
nLockCounter 0 unsigned short
+ aFilename u"C:\\Users\\Szucsi\\AppData\\Local\\Temp\\lu99163n3gde.tmp\\lu9916{8AA24C62-674C-4CFF-BB49-581F27614405}.tmp" rtl::OUString
bIsOpen true bool
...
I can see this file on my hard drive, so it is really here.
i can view its content and it is a half made xml file
content.xml (the whole file) ~110mb big, but the compressed ods is only ~670kb.
so if it would be stored only in memory, we could avoid a lot of disk writing.
but, maybe this filestream cration (before zipping) designed to make sure we dont run out of memory during the save.. ? ..
i can understand that in most cases this extra file is not a big problem... and when the file is big enought to be a problem, than maybe the memory could be a bigger problem.. :)
On Sunday, May 14, 2023 12:46 BST, Noel Grandin <noelgrandin at gmail.com> wrote:
On Tue, 14 Feb 2023 at 17:40, Noel Grandin <noel.grandin at collabora.co.uk> wrote:On 2/13/2023 4:34 PM, Michael Meeks wrote:
>
> Noel might be interested in this. Noel, Attila suggests that every zip file we write we have to stream the full
> un-compressed data to /tmp files before compressing it. And these can be huge =)
>
I couldn't find any of this specific thing happening. Perhaps Attila has a codepointer or a stack trace? What I can see is that we save to a temporary file and then copy the temporary file to its final location.That part however would require major surgery to change since it starts in sfx and goes through the dreaded UCB layer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice/attachments/20230517/843f2dba/attachment.htm>
More information about the LibreOffice
mailing list