<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hey,</div><div><br></div><div>let me at least add some comments.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 7, 2019 at 5:42 PM Jean-Philippe MENGUAL <<a href="mailto:jean-philippe.mengual@libreoffice.org">jean-philippe.mengual@libreoffice.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
<br>
Le 06/03/2019 à 20:52, Samuel Thibault a écrit :<br>
> Hello,<br>
> <br>
> Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:<br>
>> On a slightly related note I think that we have already quite a few tests for<br>
>> the accessibility UNO layer but as that layer is full of bugs many of the tests<br>
>> are disabled. It might be a good idea to work on these tests before actually<br>
>> trying to implement more complex tests that depend on lower layers working<br>
>> correctly.<br>
> <br>
> I'm not sure which piece you are referring to. Is that the AWB? I<br>
> indeed see some source code in toolkit/test/accessibility but no<br>
> reference to it.<br></blockquote><div><br></div><div>There should be accessibility UNO API tests in most top level modules. E.g. from a quick git grep Accessible -- qadevOOo/Jar_OOoRunner.mk:</div><div><br></div><div>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessible \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleAction \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleComponent \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleComponent \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleContext \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleEditableText \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleEventBroadcaster \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleExtendedComponent \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleImage \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleSelection \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleTable \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleText \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/ifc/accessibility/_XAccessibleValue \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_basctl/AccessibleShape \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/AccessibleEditableTextPara_HeaderFooter \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/AccessibleEditableTextPara_PreviewCell \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sch/AccessibleDocumentView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessibleCell \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessibleCsvCell \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessibleCsvGrid \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessibleCsvRuler \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessibleDocument \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessibleDocumentPagePreview \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessiblePageHeader \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessiblePageHeaderArea \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewCell \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewHeaderCell \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewTable \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sc/ScAccessibleSpreadsheet \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sd/AccessibleDrawDocumentView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sd/AccessibleOutlineView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sd/AccessibleSlideView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sm/SmEditAccessible \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sm/SmGraphicAccessible \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBox \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxHeaderBar \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxHeaderCell \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxTable \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxTableCell \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleIconChoiceCtrl \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleIconChoiceCtrlEntry \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleTabBar \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleTabBarPage \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleTabBarPageList \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleTreeListBox \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svtools/AccessibleTreeListBoxEntry \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svx/AccessibleControlShape \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svx/AccessibleEditableTextPara \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svx/AccessibleGraphicShape \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svx/AccessibleOLEShape \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svx/AccessiblePageShape \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svx/AccessiblePresentationGraphicShape \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svx/AccessiblePresentationOLEShape \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svx/AccessiblePresentationShape \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svx/AccessibleShape \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_svx/SvxGraphCtrlAccessibleContext \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleDocumentPageView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleDocumentView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleEndnoteView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleFooterView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleFootnoteView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleHeaderView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessiblePageView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleParagraphView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleTableCellView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleTableView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleTextEmbeddedObject \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleTextFrameView \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_sw/SwAccessibleTextGraphicObject \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleButton \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleCheckBox \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleComboBox \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleDropDownComboBox \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleDropDownListBox \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleEdit \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleFixedText \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleList \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleListBox \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleListItem \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleMenu \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleMenuBar \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleMenuItem \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleMenuSeparator \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessiblePopupMenu \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleRadioButton \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleScrollBar \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleStatusBar \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleStatusBarItem \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleTabControl \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleTabPage \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleToolBox \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleToolBoxItem \<br>qadevOOo/Jar_OOoRunner.mk: qadevOOo/tests/java/mod/_toolkit/AccessibleWindow \</div><div><br></div><div><br></div><div>However you will quickly note that quite a lot of them are actually disabled because they are broken, the underlying interfaces found bugs that nobody cared or are just unreliable due to various issues. As a simple example for sc:</div><div><br></div><div>sc/qa/unoapi/knownissues.xcl:sc.AccessibleEditableTextPara_PreviewCell::com::sun::star::accessibility::XAccessibleEditableText<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleDocument::com::sun::star::accessibility::XAccessibleSelection<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvCell::com::sun::star::accessibility::XAccessibleText<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvRuler::com::sun::star::accessibility::XAccessibleText<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvGrid::com::sun::star::accessibility::XAccessibleTable<br>sc/qa/unoapi/knownissues.xcl:sc.AccessibleEditableTextPara_PreviewNote <br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvRuler <br>sc/qa/unoapi/knownissues.xcl:sc.AccessibleEditableTextPara_HeaderFooter<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleDocument<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvCell<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCsvGrid<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleDocumentPagePreview<br>sc/qa/unoapi/knownissues.xcl:sc.AccessibleEditableTextPara_PreviewCell::com::sun::star::accessibility::XAccessibleComponent<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessiblePageHeaderArea::com::sun::star::accessibility::XAccessibleEventBroadcaster<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessibleCell::com::sun::star::accessibility::XAccessibleText<br>sc/qa/unoapi/knownissues.xcl:sc.ScAccessiblePreviewCell::com::sun::star::accessibility::XAccessibleComponent<br>sc/qa/unoapi/sc_1.sce:# i84554 -o sc.AccessibleEditableTextPara_PreviewNote<br>sc/qa/unoapi/sc_1.sce:# i88241 -o sc.AccessibleEditableTextPara_HeaderFooter<br>sc/qa/unoapi/sc_1.sce:-o sc.AccessibleEditableTextPara_PreviewCell<br>sc/qa/unoapi/sc_1.sce:-o sc.ScAccessibleCell<br>sc/qa/unoapi/sc_1.sce:# i91044 -o sc.ScAccessibleCsvCell<br>sc/qa/unoapi/sc_1.sce:# i91044 -o sc.ScAccessibleCsvGrid<br>sc/qa/unoapi/sc_1.sce:# i84641 -o sc.ScAccessibleCsvRuler<br>sc/qa/unoapi/sc_1.sce:# i88330 -o sc.ScAccessibleDocument<br>sc/qa/unoapi/sc_1.sce:# i91045 -o sc.ScAccessibleDocumentPagePreview<br>sc/qa/unoapi/sc_1.sce:-o sc.ScAccessiblePageHeader<br>sc/qa/unoapi/sc_2.sce:# fdo#FOO -o sc.ScAccessiblePageHeaderArea<br>sc/qa/unoapi/sc_2.sce:-o sc.ScAccessiblePreviewCell<br>sc/qa/unoapi/sc_2.sce:-o sc.ScAccessiblePreviewHeaderCell<br>sc/qa/unoapi/sc_2.sce:-o sc.ScAccessiblePreviewTable<br>sc/qa/unoapi/sc_2.sce:# fdo#45337 -o sc.ScAccessibleSpreadsheet<br></div><div><br></div><div>All entries in "sc/qa/unoapi/knownissues.xcl" are disabled and all lines in the *.sce file starting with # are also disabled which means that for example for calc most accessibility tests are actually disabled.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> <br>
>> The focus handling can be easily integrated into the existing UI<br>
>> testing infrastructure and might benefit there from some of the concepts that<br>
>> should make them more stable<br>
> <br>
> Good :) So could you plan to work on it?<br></blockquote><div><br></div><div>No. But I can point you to the corresponding code. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
How could such plan be scheduled in LO qa? Should we report a bug? Or <br>
open a wiki roadmap?<br></blockquote><div><br></div><div>I think the only way is to actively do the work yourself. That means more than just writing tests. At least in the beginning you will need to make sure the framework is reliable and try to make using the test framework for developers convenient enough that it is easy enough to reproduce failures and debug them. Most importantly the framework needs to be reliable enough that you can convince developers to actually care about test failures. If your accessibility tests produce a noticable rate of false positives you will quickly see all your tests being disabled and no developer paying attention to any of your tests.</div><div><br></div><div>If you have a reliable test framework it can be integrated into our CI infrastructure similar to how all the other test frameworks are executed by our CI builds.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
The problem now is that it is not possible to fix accessibility bugs, <br>
fix regressions from 4.2, integrating in the code non-regression tests, <br>
and do the same for the three major programs in free software. It is <br>
less a will problem than a resource problem, because even with funds, we <br>
do not have enough persons to work on this with skills related to <br>
Libreoffice and accessibility in general. As you know more and more <br>
persons go the web or backend techno, less in programming for desktop <br>
software.<br>
<br>
So while we are ready to fix accessibility bugs, we need non-regression <br>
tests. And it is difficult to do both. The tool on which we are working, <br>
dogtail, is interesting because enables to test via the same framework <br>
different programs, without needing to know the code of each of one. I <br>
think gateway is possible between Libreoffice framework test and such <br>
tool. We also could imagine a dedicated machine with dogtail to test, <br>
but should be acceptd. Also, the thing is to know if LO is ready to <br>
prevent a release or a commit because introduces a regression in <br>
accessibility.<br></blockquote><div><br></div><div>Most likely no. If we have tests for something it is more likely that it will be fixed after it is discovered by in the end an accessibility regression is similar to any other regression. That means that a test started failing you are expected to inspect the test failure before commiting but we will most likely not stop a release if a new test discovers that something is broken. In general while we do much to avoid regressions and having tests helps we don't treat regressions as complete release blockers. There is always a case-by-case analysis neccessary.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Well to sum up, beyond our effort, that we try general and <br>
cross-software, we would need to know how we can together set a kind of <br>
roadmap to implement accessibility in the existing frameworks, and add a <br>
layer to make common scenarios between our general tool and LO's one. <br>
Having that may be in the easy hacks to have contributions? Or b a <br>
specific TDF tender just like we did for labels?<br></blockquote><div><br></div><div>Well, in the end the decision is mostly on you. I can only tell you based on experience that the most important part is to implement a reliable testing framework. You can start from scratch with an external tool or you can base it on one of the existing frameworks that we already have. The advantage of existing frameworks is that they have seen quite some use already and are usually accepted as stable by most developers. For any new framework you will need to convince existing developers about the value and the reliability of the framework before they will accept the framework. You can see with some parts of our UNO API tests, especially the accessibility related UNO API tests, that unreliable tests just lead developers to disable any failing test instead of spending time checking whether it was a genuine failure or a false positive.</div><div><br></div><div>I wish you the best with your work on the accessibility testing.<br></div><div><br></div><div>Regards,</div><div>Markus<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Regards<br>
<br>
> Samuel<br>
> _______________________________________________<br>
> LibreOffice mailing list<br>
> <a href="mailto:LibreOffice@lists.freedesktop.org" target="_blank">LibreOffice@lists.freedesktop.org</a><br>
> <a href="https://lists.freedesktop.org/mailman/listinfo/libreoffice" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/libreoffice</a><br>
> <br>
<br>
-- <br>
Logo Hypra JEAN-PHILIPPE MENGUAL<br>
DIRECTEUR TECHNIQUE ET QUALITÉ<br>
102, rue des poissonniers, 75018, Paris<br>
Tel : +331 84 73 06 61 <tel:+33184730661> Mob : +336 76 34 93 37<br>
<tel:+33676349337><br>
<a href="mailto:jpmengual@hypra.fr" target="_blank">jpmengual@hypra.fr</a> <mailto:<a href="mailto:jpmengual@hypra.fr" target="_blank">jpmengual@hypra.fr</a>><br>
<a href="http://www.hypra.fr" rel="noreferrer" target="_blank">www.hypra.fr</a> <<a href="http://www.hypra.fr/" rel="noreferrer" target="_blank">http://www.hypra.fr/</a>><br>
Facebook Hypra <<a href="https://www.facebook.com/hyprasoftware/" rel="noreferrer" target="_blank">https://www.facebook.com/hyprasoftware/</a>> Twitter Hypra<br>
<<a href="https://twitter.com/Hypra_" rel="noreferrer" target="_blank">https://twitter.com/Hypra_</a>> Linkedin Jean-Philippe<br>
<<a href="https://fr.linkedin.com/in/jean-philippe-mengual-800133135" rel="noreferrer" target="_blank">https://fr.linkedin.com/in/jean-philippe-mengual-800133135</a>><br>
<br>
<br>
</blockquote></div></div></div></div></div>