Unrelated closing tags in script blocks interpreted as closing tags

[ehoogeveen-medweb]

Hi! I was trying out the lexer and I noticed that this block:

<script>"<style></style>"</script>

outputs the following sequence:

[ 'startTagStart', '<' ]
[ 'tagName', 'script' ]
[ 'tagEnd', '>' ]
[ 'rawtext', '"' ]
[ 'rawtext', '<style>' ]
[ 'endTagStart', '</' ]
[ 'tagName', 'style' ]
[ 'rawtext', '>"' ]
[ 'endTagStart', '</' ]
[ 'tagName', 'script' ]
[ 'tagEnd', '>' ]

As far as I know, the </style> should not be tokenized as an end tag because in a script block, the only valid end tag is </script> (not case sensitive and with optional whitespace between the /script and the >).

The readme mentions that script tags aren't fully handled, but I wasn't expecting this particular exception :) Is it expected behavior, given that the endTagStart and tagName are followed by a rawtext rather than a tagEnd and thus don't truly close the block?

[alwinb]

Hi! Thank you for spotting this. It is a regression; I’ll try and have a look at it sometime soon!

[ehoogeveen-medweb]

Thanks! I can work around it for now.

By the way, thank you for your work on this - I'm working on something that requires two-way conversion so preserving the exact whitespace, tag closing style etc. in the source is important, but none of the other parsers or tokenizers I looked at seem to pay attention to this or give enough control. Although a tokenizer is a bit low level for what I'm trying to do, it was a real relief to find something that handles my use case!

[ehoogeveen-medweb]

Not to pollute the bug report, but the following also seems pretty odd:

<div>
</span
</div>

gives

[ 'startTagStart', '<' ]
[ 'tagName', 'div' ]
[ 'tagEnd', '>' ]
[ 'newline', '\n' ]
[ 'endTagStart', '</' ]
[ 'tagName', 'span' ]
[ 'tagSpace', '\n' ]
[ 'attributeName', '<' ]
[ 'tagSpace', '/' ]
[ 'attributeName', 'div' ]
[ 'tagEnd', '>' ]

The HTML is obviously invalid, but turning < into an attribute name and / into attribute whitespace seems odd to say the least (especially given that end tags can't contain attributes).

[alwinb]

I see. The second example is actually correct though!

• In HTML5 slashes / within tags are treated as attribute separators. (I think this is a relic from SGML). As an example <span foo/=bar> will create an element with two attributes, one having a name foo and one having a name =bar and both having the empty string as their value!

• Furthermore, attributes on end tags are lexically parsed just as on start tags, but they are ignored in a later stage. This makes sense from an error recovery perspective and is described in the standard.

I agree that the name tagSpace here is a bit confusing and could be called attributeSeparator.

[ehoogeveen-medweb]

Oh, weird! Thanks for the explanation.