Mesa (master): glsl/glcpp: Treat CR+LF pair as a single newline

[Ian Romanick]

Module: Mesa
Branch: master
Commit: 04e40fd337a244ee77ef9553985e9398ff0344af
URL:    http://cgit.freedesktop.org/mesa/mesa/commit/?id=04e40fd337a244ee77ef9553985e9398ff0344af

Author: Carl Worth <cworth at cworth.org>
Date:   Tue Jul  1 16:31:07 2014 -0700

glsl/glcpp: Treat CR+LF pair as a single newline

The GLSL specification says that either carriage-return, line-feed, or both
together can be used to terminate lines. Further, it says that when used
together, the pair of terminators shall be interpreted as a single line.

This final requirement has not been respected by glcpp up until now, (it has
been emitting two newlines for every CR+LF pair).

Here, we fix the lexer by using a regular expression for NEWLINE that eats
up both "\r\n" (or even "\n\r") if possible before also considering a single
'\n' or a single '\r' as a line terminator.

Before this commit, the test results are as follows:

	\r:	135/143 tests pass
	\r\n:	  4/143 tests pass
	\n\r:	  4/143 tests pass

After this commit, the test results are as follows:

	\r:	135/143 tests pass
	\r\n:	140/143 tests pass
	\n\r:	139/143 tests pass

So, obviously, a dramatic improvement.

Reviewed-by: Ian Romanick <ian.d.romanick at intel.com>

---

 src/glsl/glcpp/glcpp-lex.l      |   11 ++++++-----
 src/glsl/glcpp/tests/.gitignore |    3 +++
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/src/glsl/glcpp/glcpp-lex.l b/src/glsl/glcpp/glcpp-lex.l
index aeaf8ab..48cc62c 100644
--- a/src/glsl/glcpp/glcpp-lex.l
+++ b/src/glsl/glcpp/glcpp-lex.l
@@ -172,6 +172,7 @@ SPACE		[[:space:]]
 NONSPACE	[^[:space:]]
 HSPACE		[ \t]
 HASH		#
+NEWLINE		(\r\n|\n\r|\r|\n)
 IDENTIFIER	[_a-zA-Z][_a-zA-Z0-9]*
 PP_NUMBER	[.]?[0-9]([._a-zA-Z0-9]|[eEpP][-+])*
 PUNCTUATION	[][(){}.&*~!/%<>^|;,=+-]
@@ -256,9 +257,9 @@ HEXADECIMAL_INTEGER	0[xX][0-9a-fA-F]+[uU]?
 	/* Multi-line comments */
 <INITIAL,DEFINE,HASH>"/*"   { yy_push_state(COMMENT, yyscanner); }
 <COMMENT>[^*\r\n]*
-<COMMENT>[^*\r\n]*[\r\n]    { yylineno++; yycolumn = 0; parser->commented_newlines++; }
+<COMMENT>[^*\r\n]*{NEWLINE} { yylineno++; yycolumn = 0; parser->commented_newlines++; }
 <COMMENT>"*"+[^*/\r\n]*
-<COMMENT>"*"+[^*/\r\n]*[\r\n] { yylineno++; yycolumn = 0; parser->commented_newlines++; }
+<COMMENT>"*"+[^*/\r\n]*{NEWLINE} { yylineno++; yycolumn = 0; parser->commented_newlines++; }
 <COMMENT>"*"+"/"        {
 	yy_pop_state(yyscanner);
 	/* In the <HASH> start condition, we don't want any SPACE token. */
@@ -289,7 +290,7 @@ HEXADECIMAL_INTEGER	0[xX][0-9a-fA-F]+[uU]?
 
 	/* Swallow empty #pragma directives, (to avoid confusing the
 	 * downstream compiler). */
-<HASH>pragma{HSPACE}*/[\r\n] {
+<HASH>pragma{HSPACE}*/{NEWLINE} {
 	BEGIN INITIAL;
 }
 
@@ -305,7 +306,7 @@ HEXADECIMAL_INTEGER	0[xX][0-9a-fA-F]+[uU]?
 	RETURN_TOKEN (LINE);
 }
 
-<HASH>\n {
+<HASH>{NEWLINE} {
 	BEGIN INITIAL;
 	RETURN_TOKEN_NEVER_SKIP (NEWLINE);
 }
@@ -521,7 +522,7 @@ HEXADECIMAL_INTEGER	0[xX][0-9a-fA-F]+[uU]?
 
 	/* We preserve all newlines, even between #if 0..#endif, so no
 	skipping.. */
-<*>[\r\n] {
+<*>{NEWLINE} {
 	if (parser->commented_newlines) {
 		BEGIN NEWLINE_CATCHUP;
 	} else {
diff --git a/src/glsl/glcpp/tests/.gitignore b/src/glsl/glcpp/tests/.gitignore
new file mode 100644
index 0000000..f18a9bb
--- /dev/null
+++ b/src/glsl/glcpp/tests/.gitignore
@@ -0,0 +1,3 @@
+subtest-cr/
+subtest-cr-lf/
+subtest-lf-cr/