{"id":1607,"date":"2015-07-19T22:45:30","date_gmt":"2015-07-20T05:45:30","guid":{"rendered":"http:\/\/zackmdavis.net\/blog\/?p=1607"},"modified":"2015-07-19T22:45:30","modified_gmt":"2015-07-20T05:45:30","slug":"dollar","status":"publish","type":"post","link":"http:\/\/zackmdavis.net\/blog\/2015\/07\/dollar\/","title":{"rendered":"$"},"content":{"rendered":"

I used to think of $<\/code> in regular expressions as matching the end of the string. I was wrong! It actually might do something more subtle than that, depending on what regex engine you're using. In my native<\/a> Python's re<\/code> module, $<\/code><\/a><\/p>\n

\n[m]atches the end of the string or just before the newline at the end of the string, and in MULTILINE mode also matches before a newline.\n<\/p><\/blockquote>\n

Note! The end of the string, or just before the newline at the end of the string<\/em>.<\/p>\n

In [2]: my_regex = re.compile("foo$")\r\n\r\nIn [3]: my_regex.match("foo")\r\nOut[3]: <_sre.SRE_Match object; span=(0, 3), match='foo'>\r\n\r\nIn [4]: my_regex.match("foo\\n")\r\nOut[4]: <_sre.SRE_Match object; span=(0, 3), match='foo'><\/code><\/pre>\n
I guess I can see the motivation\u2014we often want to use<\/a> the newline character as a terminator of lines (by definition) or files (by sacred tradition<\/a>), without wanting to think of \\n<\/code> as really part of the content of interest\u2014but the disjunctive behavior of $<\/code> can be a source of treacherous bugs in the fingers of misinformed programmers!<\/p>\n
<\/p>\n
It happened to me while I was doing speculative pre-development of the speculative pre-prototype for my speculative Glitteral programming language<\/a>, specifically in the lexical analyzer\u2014the part of a compiler that recognizes strings of source code as representing tokens<\/em> that mean something in the language's grammar: this is a language keyword, that's an integer literal, this is an identifier, and so forth. My makeshift lexical analyzer (inspired by, but diverging significantly from, the more sophisticated thing that textbook said to do\u2014I was in a hurry) involved deciding if a segment of source code could represent a particular token (or prefix thereof) by checking if it matched the regular expression defining each token class (or a regex describing prefixes of that token class). I had prudently (but not prudently enough, as you see) anchored each of my token class regexes with $<\/code>, so that, for example, the source fragment fore<\/code> could not be erroneously recognized as the language keyword for<\/code>. But that just left me with a bug in which a newline immediately following a token would be recognized as part of the token: for example, you could end up lexing the string 3\\n<\/code> as an integer literal, even though the integer literal was supposed to be just 3<\/code>. After my first crude fix<\/a> later proved to be inadequate, I ended up fixing it by augmenting the $<\/code>s with the negative lookahead assertion (?!\\n)<\/code> immediately thereafter<\/a>, in effect saying, "match the end of the string or just before the newline at the end of the string, but<\/em> not just before a newline," the negative lookahead assertion canceling out the interpretation of $<\/code> that I didn't want. And then later I replaced all those $(?!\\n)<\/code>s with \\Z<\/code>s<\/a> (which actually<\/em> match the end of the string, like I wanted in the first place), after it was brought to my attention that \\Z<\/code> was a thing.<\/p>\n
But I'm not the only one who was confused! (Note: the previous sentence should be read in a tone of terror and despair at the tightness of the cruel grip of ignorance on our fragile world, not<\/em> relief that other people are as dumb as me.) The famous Django web application framework recently released patch versions 1.8.3<\/a>, 1.7.9<\/a>, and 1.4.21<\/a> due to security concerns, one of which being validators failing to guard against possible header injection vulnerabilities<\/a> owing to the use of $<\/code> instead of \\Z<\/code> in regular expressions<\/a>.<\/p>\n
All this that I have said about $<\/code> in regexes concerns the Python world. Apparently Perl is the same way<\/a> (maybe we got it from them?). But other regex engines don't have the "or just before the newline" semantic flourish in their interpretation of $<\/code>. In Java, for example<\/a> (and therefore, more importantly from my point of view, Clojure),<\/p>\n
\n[b]y default, the regular expressions ^<\/code> and $<\/code> ignore line terminators and only match at the beginning and the end, respectively, of the entire input sequence.\n<\/p><\/blockquote>\n
user=> (re-matches #"foo$" "foo")\r\n"foo"\r\nuser=> (re-matches #"foo$" "foo\\n")\r\nnil<\/code><\/pre>\n
Whereas in Ruby<\/a>, $<\/code> is explicitly the end of line<\/em> anchor (like in Python's MULTILINE<\/code> mode), \\Z<\/code> matches the end of the string or just before the newline if the string ends with a newline (like Python's default), and \\z<\/code> is for the end of the string!<\/p>\n
I guess the moral is that if you want to write a kind of regular expression that you're not already intimately familiar with, you should think carefully and read the owner's manual of your chosen regex engine. What you find there may surprise you!<\/p>\n","protected":false},"excerpt":{"rendered":"
I used to think of $ in regular expressions as matching the end of the string. I was wrong! It actually might do something more subtle than that, depending on what regex engine you're using. In my native Python's re … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[20],"tags":[70,21,64],"_links":{"self":[{"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/posts\/1607"}],"collection":[{"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/comments?post=1607"}],"version-history":[{"count":4,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/posts\/1607\/revisions"}],"predecessor-version":[{"id":1611,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/posts\/1607\/revisions\/1611"}],"wp:attachment":[{"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/media?parent=1607"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/categories?post=1607"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/zackmdavis.net\/blog\/wp-json\/wp\/v2\/tags?post=1607"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}