Saturday, February 2, 2013

Evimosaurus Rex

E-Rex

because everyone deserves extended regular expressions

A couple posts back I compared some idiomatic Ruby with equivalent Vim forms. One of those was the multiline, whitespace-insensitive extended regular expression mode that the /x flag gives the PCRE wielder. I showed how this might be achieved in Vim as:

let PHONE_NUMBER_PATTERN = substitute(substitute('
      \ ^
      \ \%(
      \   \(\d\)           # prefix_digit
      \   [\ \-\.]\?       # optional_separator
      \ \)\?
      \ \%(
      \   (\?\(\d\{3}\))\? # area_code
      \ [\ \-\.]           # separator
      \ \)\?
      \ \(\d\{3}\)         # trunk
      \ [\ \-\.]           # separator
      \ \(\d\{4}\)         # line
      \ \%(:\ \?x\?        # optional_space_or_x
      \   \(\d\+\)         # extension
      \ \)\?
      \ $', '# \S\+', '', 'g'), '\\\@<! ', '', 'g')

echo string(PHONE_NUMBER_PATTERN)
let a_phone_number = '1-234-567-0987:1234'

echo matchlist(a_phone_number, PHONE_NUMBER_PATTERN)

Which is all good and well, but wouldn’t it be nicer if we could just do this instead:

let PHONE_NUMBER_PATTERN = ERex.parse('
      \ ^
      \ \%(
      \   \(\d\)           # prefix_digit
      \   [\ \-\.]\?       # optional_separator
      \ \)\?
      \ \%(
      \   (\?\(\d\{3}\))\? # area_code
      \ [\ \-\.]           # separator
      \ \)\?
      \ \(\d\{3}\)         # trunk
      \ [\ \-\.]           # separator
      \ \(\d\{4}\)         # line
      \ \%(:\ \?x\?        # optional_space_or_x
      \   \(\d\+\)         # extension
      \ \)\?
      \ $')

echo string(PHONE_NUMBER_PATTERN)
let a_phone_number = '1-234-567-0987:1234'

echo matchlist(a_phone_number, PHONE_NUMBER_PATTERN)

Where the ERex object was just magically waiting in the background for all of our extended regex needs…?

Install https://github.com/Raimondi/VimRegStyle and you can!


“There is no charge for awesomeness.”
— Po

This just in! the awesomeness of Extended Regular Expressions has just been added to vimple.

RFC on VimRegStyle


VimRegStyle

more than just a pattern library for sexy regex

This is a request for feedback on a new plugin: https://github.com/Raimondi/VimRegStyle

VimRegStyle is designed to be a repository of PCRE ←→ Vim regex equivalences and general pattern library for Vimmers.

Our goal is to have somewhere for the collection of common regex patterns expressed in the Vim regex flavour. We frequently get regex help requests on #vim that are answered with “look that common pattern up on a regex lib site” only to have them come back and say “it didn’t work” because they plugged the PCRE into Vim and got spanked for it.

No more! Now we have a library of our own!

So… what do I want from you? Feedback on:
  1. Design issues (concerns & improvements)(*)
  2. Ideas for tools, commands, maps and scripts surrounding such a pattern library.
(*) This might be the place to mention that we’ve taken several rather bold steps in the implementation of the pattern repository files:
  1. the patterns/ directory contains *.vrs files with the following format:
    name flavour pattern
    Where:
    • name contains no whitespace
    • flavour can be vim or pcre
    • pattern is not delimited — use a bare regex
  2. the patterns are further enhanced in the following two ways; they:
    • accept PCRE style multiline, whitespace insensitive syntax. All multiline patterns must commence on the line below the named entry and must be indented with whitespace.
    • accept a new regex atom: \%{name,count,separator} providing pattern composition by inline-expanding the named pattern at the current point in the regex (optionally count times, each one separated by separator.)

      As an example:, assuming the VRS library had a pattern called _ip4_segment that represented a single 0-255 chunk, an ip4 regex could then be written using this composition atom as: \<\%{_ip4_segment,4,.}>
      As a side note, the Extended Regex features mentioned here will be further elaborated on in a separate article. These features can be freely used in other projects, pulling from arbitrary lookup sources, including the runtime vars in Vim itself. But more on that later.
When we’re finished with this RFC process we will be looking for pattern contributions. Thanks to all who contribute their time and energy into this project! :-)

Monday, January 28, 2013

Say it with VimL

I like Ruby. She’s a beautiful language. So expressive and elegant and yummy. VimL is Vim’s scripting language. While she may not win the sort of aesthetic awards Ruby deserves, she certainly is expressive and elegant in her own way.
Gregory Brown recently showed a handful of idioms for elegantly working with text and files in Ruby. I thought I’d show their VimL analogues here.

idioms for text processing

1. Multiline Matches

Vim has its own regular expression flavour. It’s a bit shocking to PCRE lovers at first — it uses an older, more arcane syntax that can reduce the most ardent Perler to pitiful puling instead. Sulk as they may though, Vim’s regex flavour got here before PCRE and isn’t going anywhere fast. The good news is that Vim’s regex flavour is quite strong — equally up to the machinations of PCRE in almost all aspects (and certainly so in all that count within the context of editing text). I digress — this is not the place to wage that war.
Gregory showed the PCRE idiom of using the /s flag<*> to enable DOTALL mode which allows the . atom to match newlines (as well as its default match-any-character behaviour.) Vim uses \_. to achieve this result:

echo matchlist("foo\nbar\nbaz\nquux", 'foo\n\(\_.*\)quux')[1]

<*> The astute reader will have noticed my sleight play there. Ruby’s flavour of PCRE uses the /m flag to mean what the rest of the PCRE speaking world knows /s to do. I must admit, I was scratching my head when I first read Gregory’s article thinking he’d given the wrong example to suit the /m flag. Thanks goes to kotigid on #regex for pointing me at the Ruby regex page.

2. matchlist()

While Vim doesn’t have the global match variables ($1 et al) that Gregory is recommending avoidance of, it does have his preferred method baked right in. The matchlist() function returns a list containing the whole match as the zeroth element and any submatches from index 1 onwards.

echo matchlist("---\na\nb\nc\n---\n", '^\(---\s*\n\_.\{-}\n\?\)\(---\s*\n\?\)')

3. Extended Regular Expression Syntax

The /x flag in PCRE allows complex regular expressions to be spread out over multiple lines with embedded comments for easier readability and clarity. We can approximate that in VimL:

let PHONE_NUMBER_PATTERN = substitute(substitute('
      \ ^
      \ \%(
      \   \(\d\)           # prefix_digit
      \   [\ \-\.]\?       # optional_separator
      \ \)\?
      \ \%(
      \   (\?\(\d\{3}\))\? # area_code
      \ [\ \-\.]           # separator
      \ \)\?
      \ \(\d\{3}\)         # trunk
      \ [\ \-\.]           # separator
      \ \(\d\{4}\)         # line
      \ \%(:\ \?x\?        # optional_space_or_x
      \   \(\d\+\)         # extension
      \ \)\?
      \ $', '# \S\+', '', 'g'), '\\\@<! ', '', 'g')

echo string(PHONE_NUMBER_PATTERN)
let a_phone_number = '1-234-567-0987:1234'

echo matchlist(a_phone_number, PHONE_NUMBER_PATTERN)

4. Using join()

VimL doesn’t have string interpolation like Ruby.
Given a dictionary (a.k.a associative array, or hash) such as:

let filedata = {'year' : 2013, 'month' : 1, 'day' : 28}

To include variable values in strings we have to use catenation:

echo filedata["year"] . '/' . filedata["month"]. '/' .  filedata["day"]

Of course, the join() trick Gregory showed also works in VimL:

echo join([filedata["year"], filedata["month"], filedata["day"] ], "/")

Another approach in both languages would be to use a printf string:

echo printf("%d/%d/%d",filedata["year"], filedata["month"], filedata["day"])

On the down side, this only works when you know how many fields you need to print, and you’re forced to insert the / characters manually. On the up side, you can easily format the values to show, for example, leading zeros in the day and month fields:

echo printf("%d/%02d/%02d",filedata["year"], filedata["month"], filedata["day"])

idioms for working with files and folders

1. Filenames

Ruby has File.dirname, File.basename, and File.extname for munging filenames. Vim uses the expand() function to do this. Ruby’s FILE (available in Vim as % and VimL as expand('%')) is expressed in Vim as:

echo expand('%:p')

The :p there is called a Modifier in the :help expand() docs. The :p modifier means full path.

To get the dirname only (called the head in vimspeak):

echo expand('%:p:h')

To get the basename (called tail):

echo expand('%:p:t')

Which will return the basename.extension form of the filename. To get just the basename with no extension, use the :r (root) modifier to strip off one level of extension:

echo expand('%:p:t:r')

To get the extension (mnemonically equivalent in vimspeak):

echo expand('%:p:e')

2. Pathname Objects

The closest analogue in Vim to Pathname objects is the fnamemodify() function which provides the same filename manipulations as the expand() function above. You can find more functions like this in :help file-functions.

3. Reading and Writing Files

Vim has two builtin functions for reading and writing files: readfile(fname) (which returns a list of lines) and writefile(list, fname). Semantically simple interfaces.

4. Dir.mktmpdir

Vim doesn’t have a mktmpdir() function but more importantly, VimL doesn’t have the beautiful code blocks of Ruby. As such, we have to use a more procedural idiom of manually creating a temporary directory (with :help tempname()), doing what we want in it and finally remembering to clean up after ourselves. Ruby wins here.

Reflections

Gregory’s intent behind his article was to lead the misguided Rubiest away from using needlessly laborious low-level functions for achieving what can be more beautifully expressed using idiomatic Ruby and elegant thinking. My intent with this article is twofold: firstly to show that not only can VimL easily do the sort of text and file manipulations Gregory showed, but also in most cases just as elegantly (read: semantically simple). Sure, VimL’s actual syntax in places might make your skin crawl, but once you overcome that and appreciate the deeper aesthetics, VimL doesn’t deserve the derision it receives as being a gnarled and impotent language.

Thursday, December 20, 2012

Learn Vimscript the Hard Way, Commentary II

As I wrote in my first commentary on Steve Losh’s Learn Vimscript the Hard Way, I think it is a worthwhile book for people to read to get more familiar with the mechanics of customising their Vim environment.

What about those who want to use it to actually learn Vimscript?

It does an acceptable job of that too.

The chapters on Vimscript itself (19-27 and 35-40) cover the syntax and semantics of the language with examples and exercises spread throughout to give the learner necessary hands on experience.

I want to stress here that I feel Steve didn’t intend the what of those examples to be used literally in anyone’s vimrc files or personal plugins (in fact, I believe that to be true of the whole book at large) but rather the how of techniques shown. Don’t create your own little maps in your ~/.vimrc file for commenting lines in various filetypes and don’t write your own toy snippets system — very good plugins exist for these purposes already. Do learn that you can do these sorts of things so that when the time comes for you to really write something new, you will know how to.

Steve also provides two larger exercises starting respectively at chapters 32 and 41. The first is a new operator to grep for the motioned text, and the second is a full blown Plugin for a new programming language. Both serve as good models for the sort of larger works of the practising VimLer.

I do recommend this book because it’s freely available to read online. Another resource I would recommend for learning Vimscript is Damian Conway’s five part developerWorks article series, Scripting the Vim Editor — that’s how I first got into VimL (VimL is short for Vim Scripting Language and is another name for Vimscript). Vim’s built-in :help usr_41 is the user guide to writing Vim scripts and :help eval.txt is the reference manual on VimL’s expression evaluation.

Do you have a favourite resource for learning VimL?

[update]
Oops... I forgot to add my remarks on some of the technical aspects of Steve's work:

  • As Steve says, always use :help nore maps until you know you need otherwise.
  • In the same vein, always use :normal! (instead of the oft shown :normal) to avoid user-defined keymaps on the right hand side.
  • The :echom command (and friends) expects the evaluations of its expressions to be of type string. Use :help string( to coerce lists and dictionaries to strings for use in these commands. E.g.   :echom string(getline(1, '$'))
  • Vim's help system is context aware based on the format of the help tag. See :help help-context for the list of formats.

Learn Vimscript the Hard Way, Commentary I

Steve Losh has written a book called Learn Vimscript the Hard Way.

It’s badly titled, imho. His definition of the book explains why I say that: a book for users of the Vim editor who want to learn how to customize Vim. With that description in mind, I think the book achieves its goal — a goal that all vimmers would aspire to master. However with the title of the book, I fear even many proficient vimmers would assume that the material is out of their reach or too dense to absorb right now with their busy schedules, relegating it to a later reading pile, at best.

For all you up and coming Vimmers looking to read something to take you beyond all of the beginner tutorials out there, read chapters: 0-18, 28-32, 43-48, 50 & 56.

For those of you who picked the book up specifically because of its title, that review is coming soon. :-)

Sunday, December 16, 2012

call() for a Good Time

Simple functions in Vim are declared like this:

function! A(a, b, c)
  echo a:a a:b a:c
endfunction

call A(1, 2, 3)

There’s probably nothing surprising there except for the a:a syntax, which is how Vim insists on accessing the function’s arguments (mnemonic: a: for argument).
Just as simple is calling function A() from another function, B(), passing its arguments directly along to A():

function! B(a, b, c)
  return A(a:a, a:b, a:c)
endfunction

call B(1, 2, 3)

Nothing surprising there at all. But we’ve just laid the groundwork for the main attraction tonight. In VimL, you can call a function using the library function call(func, arglist) where arglist is a list. If you’re calling a function that takes multiple arguments, collect them in an actual list like this:

function! C(a, b, c)
  return call("A", [a:a, a:b, a:c])
endfunction

call C(1, 2, 3)

If you already have the elements in a list, no need to wrap it in an explicit list:

function! D(a)
  return call("A", a:a)
endfunction

call D([1, 2, 3])

Let’s step it up a notch. What if you want to be able to accept the args as either separate arguments or as a list? Vim has your back with variadic functions cloaked in a syntax similar to C’s: …

Variadics in the key of V:
  • a:0 is a count of the variadic arguments
  • a:000 is all of the variadic arguments in a single list
  • a:1 to a:20 are positional accessors to the variadic arguments


So now it doesn’t matter how we receive the arguments — standalone or in a list — we can keep Vim happy and call A() appropriately.

function! E(...)
  if a:0 == 1
    return call("A", a:1)
  else
    return call("A", a:000)
  endif
endfunction

call E(1, 2, 3)
call E([1, 2, 3])

Ok. That’s not too bad; it’s perhaps a little awkward. We’re calling A() directly here, but it shouldn’t be a surprise to see that we can call C() in the same way too:

function! F(...)
  if a:0 == 1
    return call("C", a:1)
  else
    return call("C", a:000)
  endif
endfunction

call F(1, 2, 3)
call F([1, 2, 3])

Pretty straightforward. What about calling D() instead which expects a single list argument? Hmm… if Vim wants a list, give him a list:

function! G(...)
  if a:0 == 1
    return call("D", [a:1])
  else
    return call("D", [a:000])
  endif
endfunction

call G(1, 2, 3)
call G([1, 2, 3])

It’s worth stopping briefly here to consider what call() is doing to that arglist: It’s splatting it (extracting the arguments and passing them as separate members to the called function). Nice. Wouldn’t it be nice if we could splat lists ourselves? Well, be envious of Ruby coders no more because we can splat lists in VimL!

Splat!
To splat a list into separate variables (a, b and c here):
let [a, b, c] = somelist
Read :help :let-unpack for the juicy extras.

I like the splatting approach because it gives us variable names to play with inside our function:

function! H(...)
  if a:0 == 1
    let [a, b, c] = a:1
  else
    let [a, b, c] = a:000
  endif
  return D([a, b, c])
endfunction

call H(1, 2, 3)
call H([1, 2, 3])

Of course, it works just as well for calling functions with explicit multiple arguments, like C():

function! I(...)
  if a:0 == 1
    let [a, b, c] = a:1
  else
    let [a, b, c] = a:000
  endif
  return C(a, b, c)
endfunction

call I(1, 2, 3)
call I([1, 2, 3])

You’ll notice that the splat semantics are identical between H() and I() and only the call of D() and C() change, respectively. This is very neat, I think.
So far we’ve been calling through to functions that call A() directly. Happily, we can call through to one of these dynamic functions (like E(), but any would work as well) and have it Just Work too:

function! J(...)
  if a:0 == 1
    let [a, b, c] = a:1
  else
    let [a, b, c] = a:000
  endif
  return E(a, b, c)
endfunction

call J(1, 2, 3)
call J([1, 2, 3])

So, that’s it. Vim has variadic functions and splats. And splats are my recommended pattern for handling deep call chains between variadic functions.
There’s one last, cute, little thing about splats: you can collect a certain number of explicit arguments as you require, and then have any remaining arguments dumped into a list for you. The rest variable here will be a list containing [4, 5, 6] from the subsequent calls:

function! K(...)
  if a:0 == 1
    let [a, b, c; rest] = a:1
  else
    let [a, b, c; rest] = a:000
  endif
  echo "rest: " . string(rest)
  return E(a, b, c)
endfunction

call K([1, 2, 3, 4, 5, 6])
call K(1, 2, 3, 4, 5, 6)

And I thought this was going to be a short post when I started. I almost didn’t bother posting it because of that reason.

Saturday, December 1, 2012

Rapid Programming Language Prototypes with Ruby & Racc, Commentary

I just watched a tolerable ruby conference video by Tom Lee on Rapid Programming Language Prototypes with Ruby & Racc.

What he showed he showed fairly well. His decision to "introduce compiler theory" was, he admitted, last-minute and the hesitation in its delivery bore testimony to that. The demonstration of the compiler pipeline using his intended tools (ruby and racc) was done quite well with a natural progression through the dependent concepts along the way. By the end of the talk he has a functional compiler construction tool chain going from EBNF-ish grammar through to generated (and using gcc, compiled) C code.

I was surprised that nobody in the audience asked the question I was burning to ask from half way through the live-coding session: Why not use Treetop? (or the more generic: why not use a peg parser generator or a parser generator that does more of the heavy lifting for you?)

The whole point of Tom's presentation is: use ruby+racc because it saves you from all the headaches of setting up the equivalent tool chain in C/C++. And it does, he's right. But it feels to me that Treetop does even more of that hard work for you, allowing you to more quickly get to the fun part of actually building your new language. I'm angling for simplicity here.

I could be wrong, though, so let me ask it here (as Confreaks seems to not allow comments): Why not treetop (or an equally 'simple' parser generator) for something like this? (and answers along the lines of EBNF > PEG are not really what I'm after, but if you have a concrete example of that I'd like to hear it too.)

On a completely separate note: Tom, you need to add some flying love to your Vim habits. :-)