Say it with VimL

I like Ruby. She’s a beautiful language. So expressive and elegant and yummy. VimL is Vim’s scripting language. While she may not win the sort of aesthetic awards Ruby deserves, she certainly is expressive and elegant in her own way.
Gregory Brown recently showed a handful of idioms for elegantly working with text and files in Ruby. I thought I’d show their VimL analogues here.

---

idioms for text processing

1. Multiline Matches

Vim has its own regular expression flavour. It’s a bit shocking to PCRE lovers at first — it uses an older, more arcane syntax that can reduce the most ardent Perler to pitiful puling instead. Sulk as they may though, Vim’s regex flavour got here before PCRE and isn’t going anywhere fast. The good news is that Vim’s regex flavour is quite strong — equally up to the machinations of PCRE in almost all aspects (and certainly so in all that count within the context of editing text). I digress — this is not the place to wage that war.
Gregory showed the PCRE idiom of using the /s flag<*> to enable DOTALL mode which allows the . atom to match newlines (as well as its default match-any-character behaviour.) Vim uses \_. to achieve this result:

echo matchlist("foo\nbar\nbaz\nquux", 'foo\n\(\_.*\)quux')[1]

<*> The astute reader will have noticed my sleight play there. Ruby’s flavour of PCRE uses the /m flag to mean what the rest of the PCRE speaking world knows /s to do. I must admit, I was scratching my head when I first read Gregory’s article thinking he’d given the wrong example to suit the /m flag. Thanks goes to kotigid on #regex for pointing me at the Ruby regex page.

2. matchlist()

While Vim doesn’t have the global match variables ($1 et al) that Gregory is recommending avoidance of, it does have his preferred method baked right in. The matchlist() function returns a list containing the whole match as the zeroth element and any submatches from index 1 onwards.

echo matchlist("---\na\nb\nc\n---\n", '^\(---\s*\n\_.\{-}\n\?\)\(---\s*\n\?\)')

3. Extended Regular Expression Syntax

The /x flag in PCRE allows complex regular expressions to be spread out over multiple lines with embedded comments for easier readability and clarity. We can approximate that in VimL:

let PHONE_NUMBER_PATTERN = substitute(substitute(' \ ^ \ \%( \ \(\d\) # prefix_digit \ [\ \-\.]\? # optional_separator \ \)\? \ \%( \ (\?\(\d\{3}\))\? # area_code \ [\ \-\.] # separator \ \)\? \ \(\d\{3}\) # trunk \ [\ \-\.] # separator \ \(\d\{4}\) # line \ \%(:\ \?x\? # optional_space_or_x \ \(\d\+\) # extension \ \)\? \ $', '# \S\+', '', 'g'), '\\\@<! ', '', 'g') echo string(PHONE_NUMBER_PATTERN) let a_phone_number = '1-234-567-0987:1234' echo matchlist(a_phone_number, PHONE_NUMBER_PATTERN)

4. Using join()

VimL doesn’t have string interpolation like Ruby.
Given a dictionary (a.k.a associative array, or hash) such as:

let filedata = {'year' : 2013, 'month' : 1, 'day' : 28}

To include variable values in strings we have to use catenation:

echo filedata["year"] . '/' . filedata["month"]. '/' . filedata["day"]

Of course, the join() trick Gregory showed also works in VimL:

echo join([filedata["year"], filedata["month"], filedata["day"] ], "/")

Another approach in both languages would be to use a printf string:

echo printf("%d/%d/%d",filedata["year"], filedata["month"], filedata["day"])

On the down side, this only works when you know how many fields you need to print, and you’re forced to insert the / characters manually. On the up side, you can easily format the values to show, for example, leading zeros in the day and month fields:

echo printf("%d/%02d/%02d",filedata["year"], filedata["month"], filedata["day"])

---

idioms for working with files and folders

1. Filenames

Ruby has File.dirname, File.basename, and File.extname for munging filenames. Vim uses the expand() function to do this. Ruby’s FILE (available in Vim as % and VimL as expand('%')) is expressed in Vim as:

echo expand('%:p')

The :p there is called a Modifier in the :help expand() docs. The :p modifier means full path.

To get the dirname only (called the head in vimspeak):

echo expand('%:p:h')

To get the basename (called tail):

echo expand('%:p:t')

Which will return the basename.extension form of the filename. To get just the basename with no extension, use the :r (root) modifier to strip off one level of extension:

echo expand('%:p:t:r')

To get the extension (mnemonically equivalent in vimspeak):

echo expand('%:p:e')

2. Pathname Objects

The closest analogue in Vim to Pathname objects is the fnamemodify() function which provides the same filename manipulations as the expand() function above. You can find more functions like this in :help file-functions.

3. Reading and Writing Files

Vim has two builtin functions for reading and writing files: readfile(fname) (which returns a list of lines) and writefile(list, fname). Semantically simple interfaces.

4. Dir.mktmpdir

Vim doesn’t have a mktmpdir() function but more importantly, VimL doesn’t have the beautiful code blocks of Ruby. As such, we have to use a more procedural idiom of manually creating a temporary directory (with :help tempname()), doing what we want in it and finally remembering to clean up after ourselves. Ruby wins here.

---

Reflections

Gregory’s intent behind his article was to lead the misguided Rubiest away from using needlessly laborious low-level functions for achieving what can be more beautifully expressed using idiomatic Ruby and elegant thinking. My intent with this article is twofold: firstly to show that not only can VimL easily do the sort of text and file manipulations Gregory showed, but also in most cases just as elegantly (read: semantically simple). Sure, VimL’s actual syntax in places might make your skin crawl, but once you overcome that and appreciate the deeper aesthetics, VimL doesn’t deserve the derision it receives as being a gnarled and impotent language.

Learn Vimscript the Hard Way, Commentary II

As I wrote in my first commentary on Steve Losh’s Learn Vimscript the Hard Way, I think it is a worthwhile book for people to read to get more familiar with the mechanics of customising their Vim environment.

What about those who want to use it to actually learn Vimscript?

It does an acceptable job of that too.

The chapters on Vimscript itself (19-27 and 35-40) cover the syntax and semantics of the language with examples and exercises spread throughout to give the learner necessary hands on experience.

I want to stress here that I feel Steve didn’t intend the what of those examples to be used literally in anyone’s vimrc files or personal plugins (in fact, I believe that to be true of the whole book at large) but rather the how of techniques shown. Don’t create your own little maps in your ~/.vimrc file for commenting lines in various filetypes and don’t write your own toy snippets system — very good plugins exist for these purposes already. Do learn that you can do these sorts of things so that when the time comes for you to really write something new, you will know how to.

Steve also provides two larger exercises starting respectively at chapters 32 and 41. The first is a new operator to grep for the motioned text, and the second is a full blown Plugin for a new programming language. Both serve as good models for the sort of larger works of the practising VimLer.

I do recommend this book because it’s freely available to read online. Another resource I would recommend for learning Vimscript is Damian Conway’s five part developerWorks article series, Scripting the Vim Editor — that’s how I first got into VimL (VimL is short for Vim Scripting Language and is another name for Vimscript). Vim’s built-in :help usr_41 is the user guide to writing Vim scripts and :help eval.txt is the reference manual on VimL’s expression evaluation.

Do you have a favourite resource for learning VimL?

[update]
Oops... I forgot to add my remarks on some of the technical aspects of Steve's work:

• As Steve says, always use :help nore maps until you know you need otherwise.
• In the same vein, always use :normal! (instead of the oft shown :normal) to avoid user-defined keymaps on the right hand side.
• The :echom command (and friends) expects the evaluations of its expressions to be of type string. Use :help string( to coerce lists and dictionaries to strings for use in these commands. E.g.   :echom string(getline(1, '$'))
• Vim's help system is context aware based on the format of the help tag. See :help help-context for the list of formats.

Learn Vimscript the Hard Way, Commentary I

Steve Losh has written a book called Learn Vimscript the Hard Way.

It’s badly titled, imho. His definition of the book explains why I say that: a book for users of the Vim editor who want to learn how to customize Vim. With that description in mind, I think the book achieves its goal — a goal that all vimmers would aspire to master. However with the title of the book, I fear even many proficient vimmers would assume that the material is out of their reach or too dense to absorb right now with their busy schedules, relegating it to a later reading pile, at best.

For all you up and coming Vimmers looking to read something to take you beyond all of the beginner tutorials out there, read chapters: 0-18, 28-32, 43-48, 50 & 56.

For those of you who picked the book up specifically because of its title, that review is coming soon. :-)

call() for a Good Time

Simple functions in Vim are declared like this:

function! A(a, b, c) echo a:a a:b a:c endfunction call A(1, 2, 3)

There’s probably nothing surprising there except for the a:a syntax, which is how Vim insists on accessing the function’s arguments (mnemonic: a: for argument).
Just as simple is calling function A() from another function, B(), passing its arguments directly along to A():

function! B(a, b, c) return A(a:a, a:b, a:c) endfunction call B(1, 2, 3)

Nothing surprising there at all. But we’ve just laid the groundwork for the main attraction tonight. In VimL, you can call a function using the library function call(func, arglist) where arglist is a list. If you’re calling a function that takes multiple arguments, collect them in an actual list like this:

function! C(a, b, c) return call("A", [a:a, a:b, a:c]) endfunction call C(1, 2, 3)

If you already have the elements in a list, no need to wrap it in an explicit list:

function! D(a) return call("A", a:a) endfunction call D([1, 2, 3])

Let’s step it up a notch. What if you want to be able to accept the args as either separate arguments or as a list? Vim has your back with variadic functions cloaked in a syntax similar to C’s: …

Variadics in the key of V:

a:0 is a count of the variadic arguments a:000 is all of the variadic arguments in a single list a:1 to a:20 are positional accessors to the variadic arguments

So now it doesn’t matter how we receive the arguments — standalone or in a list — we can keep Vim happy and call A() appropriately.

function! E(...) if a:0 == 1 return call("A", a:1) else return call("A", a:000) endif endfunction call E(1, 2, 3) call E([1, 2, 3])

Ok. That’s not too bad; it’s perhaps a little awkward. We’re calling A() directly here, but it shouldn’t be a surprise to see that we can call C() in the same way too:

function! F(...) if a:0 == 1 return call("C", a:1) else return call("C", a:000) endif endfunction call F(1, 2, 3) call F([1, 2, 3])

Pretty straightforward. What about calling D() instead which expects a single list argument? Hmm… if Vim wants a list, give him a list:

function! G(...) if a:0 == 1 return call("D", [a:1]) else return call("D", [a:000]) endif endfunction call G(1, 2, 3) call G([1, 2, 3])

It’s worth stopping briefly here to consider what call() is doing to that arglist: It’s splatting it (extracting the arguments and passing them as separate members to the called function). Nice. Wouldn’t it be nice if we could splat lists ourselves? Well, be envious of Ruby coders no more because we can splat lists in VimL!

Splat!

To splat a list into separate variables (a, b and c here): let [a, b, c] = somelist Read :help :let-unpack for the juicy extras.

I like the splatting approach because it gives us variable names to play with inside our function:

function! H(...) if a:0 == 1 let [a, b, c] = a:1 else let [a, b, c] = a:000 endif return D([a, b, c]) endfunction call H(1, 2, 3) call H([1, 2, 3])

Of course, it works just as well for calling functions with explicit multiple arguments, like C():

function! I(...) if a:0 == 1 let [a, b, c] = a:1 else let [a, b, c] = a:000 endif return C(a, b, c) endfunction call I(1, 2, 3) call I([1, 2, 3])

You’ll notice that the splat semantics are identical between H() and I() and only the call of D() and C() change, respectively. This is very neat, I think.
So far we’ve been calling through to functions that call A() directly. Happily, we can call through to one of these dynamic functions (like E(), but any would work as well) and have it Just Work too:

function! J(...) if a:0 == 1 let [a, b, c] = a:1 else let [a, b, c] = a:000 endif return E(a, b, c) endfunction call J(1, 2, 3) call J([1, 2, 3])

So, that’s it. Vim has variadic functions and splats. And splats are my recommended pattern for handling deep call chains between variadic functions.
There’s one last, cute, little thing about splats: you can collect a certain number of explicit arguments as you require, and then have any remaining arguments dumped into a list for you. The rest variable here will be a list containing [4, 5, 6] from the subsequent calls:

function! K(...) if a:0 == 1 let [a, b, c; rest] = a:1 else let [a, b, c; rest] = a:000 endif echo "rest: " . string(rest) return E(a, b, c) endfunction call K([1, 2, 3, 4, 5, 6]) call K(1, 2, 3, 4, 5, 6)

And I thought this was going to be a short post when I started. I almost didn’t bother posting it because of that reason.

Rapid Programming Language Prototypes with Ruby & Racc, Commentary

I just watched a tolerable ruby conference video by Tom Lee on Rapid Programming Language Prototypes with Ruby & Racc.

What he showed he showed fairly well. His decision to "introduce compiler theory" was, he admitted, last-minute and the hesitation in its delivery bore testimony to that. The demonstration of the compiler pipeline using his intended tools (ruby and racc) was done quite well with a natural progression through the dependent concepts along the way. By the end of the talk he has a functional compiler construction tool chain going from EBNF-ish grammar through to generated (and using gcc, compiled) C code.

I was surprised that nobody in the audience asked the question I was burning to ask from half way through the live-coding session: Why not use Treetop? (or the more generic: why not use a peg parser generator or a parser generator that does more of the heavy lifting for you?)

The whole point of Tom's presentation is: use ruby+racc because it saves you from all the headaches of setting up the equivalent tool chain in C/C++. And it does, he's right. But it feels to me that Treetop does even more of that hard work for you, allowing you to more quickly get to the fun part of actually building your new language. I'm angling for simplicity here.

I could be wrong, though, so let me ask it here (as Confreaks seems to not allow comments): Why not treetop (or an equally 'simple' parser generator) for something like this? (and answers along the lines of EBNF > PEG are not really what I'm after, but if you have a concrete example of that I'd like to hear it too.)

On a completely separate note: Tom, you need to add some flying love to your Vim habits. :-)

Vim Motions

One of the more frequent admonishments delivered on #vim to the whining novice or the curious journeyman is to master the many motions within the editor. Previously, a bewildering list of punctuation and jumbled letters was unceremoniously dumped on the complainant with the misguided expectation that they'd then take themselves off and get right to the task of memorising the eighty odd glyphs. We mistook their silence for compliance but I rather suspect it was more bewilderment or repulsion or sheer paralysis. In an attempt to friendly that mess up, I have started an infographic series intended to cover the twelve major categories, probably spread over six separate infographics.

The Vim Motions Infographic Series (in 9 parts):

1. Line & Buffer
2. Column
3. Word
4. Find
5. Search
6. Large Objects
7. Marks, Matches & Folds
8. Text Objects (not motions, but mesh nicely at this point)
9. Creating your own Text Objects

I plan to have a different expression on the chibi's face in each of the pages. I'll move the crying one from the Large Object page (as shown below) to page 1 and then progressively improve her mood through the remaining pages: something like -- crying, disappointment, resignation, hope, amazement, happiness, confidence, smugness and something devilish. As an update on that, I have inked five of the chibis now. I look forward to having them all up in their own infographics.

I decided to have the background colour change to suit the mood of the chibi, starting from black in image number one to represent depression and despair. I will roughly follow the same colour spread I used on the How Do I Feel graphic.

I have no experience in putting together a multi-page piece like this. Feedback certainly welcome. I was vaguely thinking of having it a bit like a magazine or comic book spread, but I don't know how to do that or whether it's the right or even a good approach.


Legend:
Green indicates cursor origin before issuing the motion.
Red indicates cursor destination at the end of the motion.
Orange shows the area covered by the motion. This would be the same area highlighted in Vim if a visual operator was used with these motions.

1. Line & Buffer Motions

2. Column Motions

 
6. Large Object Motions

The Many Faces of % in Vim

Pity the poor Vimmer for he has so many a face to put to percent:

Help Topic	Description
N%	go to {count} percentage in the file
%	match corresponding [({})] (enhanced with matchit.vim plugin)
g%	enhanced match with matchit.vim plugin — cycle backwards through matches
:%	as a range, equal to :1,$ (whole file)
:_%	used as an argument to an :ex command as the name of the current file
"%	as a register, the name of the current file
expr-%	in VimL as modulo operator
expand(), printf() and bufname()	in VimL use % in printf-like format specifiers
'grepformat', 'errorformat', 'shellredir', 'printheader' and 'statusline'	various options use % as a printf-like format specifier
Regular Expression Atoms:
Match locations:
\%#	cursor position
\%'	position of a mark
\%l	specific line
\%c	specific column
\%v	specific virtual column
\%(	non-backref capturing group
\%[	sequence of optionally matched atoms
Numeric character specifier in matches:
\%d	decimal
\%o	octal
\%x	hex (2 digits)
\%u	hex (4 digits)
\%U	hex (8 digits)
Absolute file or string boundaries:
\%^	start of file (or start of string)
\%$	end of file (or end of string)
\%V	match inside visual area