Now You Know When To Macro

Use macros to reformat multi-line text
that would be too finicky
with :ex's linewise commands.

What Are Macros?

Macros are just recorded keystrokes of what the user would normally do manually when faced with the given editing task. As such, they do not represent much of a conceptual challenge and therefore the barrier to adoption is quite low. Once users gingerly try their first few macros, they're hooked and look for new and interesting screws to bang with their dull new hammer.

Regular expressions, on the other hand, are a mess of complicated hieroglyphics to the layman. Like nothing they've used or done before, they appear opaque and unfathomable. The meaning of various atoms within a regular expression are overloaded (they take on different meanings within different contextual sections of the expression) which only adds to the surprise and frustration of those who make the effort to learn them. Let's not even mention Vim's silly variable magic-ness.

For a great many editing situations, though, regular expressions are the sharpest tool in a Vimmer's toolbox. It slices and dices text cleanly and, when you master them, efficiently. I urge every Vimmer to vault the regex wall; glory awaits you on the other side.

For the right task, however, macros are the right tool - a better tool than regular expressions (alone).. But... what is the right task? In my opinion, macros should be used for multi-line edits where Vim's :ex commands (the :substitute (regular expression driven search & replace command) among them) prove fragile and stubborn.

A quick duck around the net revealed several good explanations and examples of macros being used correctly. It also produced a few cases of macros being used where the better choice would have been one or several regexes.

I won't bore you with yet another contrived example here, I'll simply point to some good ones:

• http://riceball.com/d/node/565
• http://lucapette.com/vim/rails/vim-for-rails-developers-recursive-macros-and-ack/
• http://joaopizani.hopto.org/en/2011/12/vim-macros/

Actually... As I was readying to publish this, I just noticed a macro that I always use. Moving to the first paragraph, I type:

  qavipJ}jq

I then run it on the second paragraph with @a and the subsequent ones with @@. I don't make it crawl over the whole file automatically because the asciidoc multi-line headings, lists and admonition blocks get messed up with this macro. for the length of my articles and the builtin } command to move to the next paragraph, it's not a big deal.

Recursive Macros

One neat trick about macros is that they can be recursive. This is often used to make the macro auto repeat without having to specify a range or count of lines when executing it. While it is fairly trivial to specify a range of lines for a macro to execute over, I'd suspect that if it _were_ that easy then using a macro in the first place was probably the wrong choice. With a macro that utilises a find up front to set the position of the subsequent edits and then calls itself as the last step, you have a simple and neat solution that finds and changes all occurrences for you.

Here is a simple example of a recursive macro:
http://dailyvim.blogspot.com/2009/06/recursive-macros.html

And I've discussed recursive macros before too.


Cases Where Macros Were The Wrong Tool

Of the many poor examples of macros, I'll highlight why I consider macros to be a poor choice with the following two cases (chosen randomly):

• http://www.joeldare.com/wiki/linux:vim_macro

The source data is all on separate, unrelated lines. This is a clear signal that plain :ex (in this case, a regex :substitute) should be employed instead of a macro.

Original source lines:

  First
  Last
  Phone

Desired destination lines:

  <label>First:</label><input type="text" name="First">
  <label>Last:</label><input type="text" name="Last">
  <label>Phone:</label><input type="text" name="Phone">

Simple regex solution:
  :%s/.*/<label>&:<\/label><input type="text" name="&">

• http://www.oreillynet.com/mac/blog/2006/07/more_vim_save_time_with_macros_1.html

Again, the source data is strictly single-line, requiring again, a simple :substitute.

  %s/^\d\+ -- \(.*\)\( (\d\{4})\)$/<li><em>\1<\/em>\2<\/li>/

.Follow up with some normal commands to wrap the list:
  o</ol><esc>
  <c-o>O<ol><esc>


I've tried to show in this article the following things:

• Vim's macros are cool when used in the right place - where the source involves complex, interdependent multi-line text.

• Don't use a macro when the source does not consist of multi-line, interdependent text. Use an :ex command, like :substitute then.

• LEARN regular expressions. They'll save you a LOT of time in the long haul.

The Art of Edits I - Weaponry

When faced with vicious hordes of marauding tweaks, ghastly repeats and soul shaking changes to the landscape that would make a lesser mortal wine and pule under their desks, Vimmers do not wince or cringe or shy away from their keyboards. This is their domain. This is what they've been practising for. The accomplished warrior does not foolhardily rush in to battle without first examining his lot carefully. What is the nature of the beast? What weapon shall I vanquish it with? How many scalps will I collect before my anime starts?

The Armoury

Various weapons adorn the armoury of the Valiant Vimmer:

Edits

A small hand-held weapon, most effective in cramped and dynamic situations. CAUTION: Sole reliance on this weapon will cause fatigue. Remember to sheathe your Insert Edits when you're not using them. These are the normal, insert and visual mode commands that constitute a lot of ordinary editing work. Read the left column of topics in the table of contents of :help quickref for a summary of awesome tips, tricks and techniques this weapon offers.

Abbreviations

Grips and tactics to wield Edits more gracefully.  These are rehearsed and prepared shortenings that automatically expand out to longer forms while we're typing. They're available in both insert and command-line modes. Read :help :abbreviate for the craft and maintenance of this weapon.

Completions

A mystical amulet that bestows upon the wearer the ability to complete an attack automatically. Adept users of this power can call on recent battlefield attacks, those from historic battles, from battlefield manuals or even from an omniscient guiding hand. Read :help ins-completion for access to this gem of valour.

Regexes

The most devastating hand-held weapon in the Vimmer's arsenal. Due to its ungainly size and the need to retreat from battle while preparing an offensive with this weapon, it is not recommended for close quarter or highly dynamic combat. Often best suited for ranged attacks. The adept Vimmer is an effective user of Vim's regular expressions. To sharpen your regex skills, read :help pattern.txt and you can practise with VimRegexTutor.

Ex

One of a warrior's greatest strengths is the ability to command his troops amidst the frenzy of battle. Ex is the powerful tool Vimmers use to coordinate and execute larger strategies on the battlefield. Raining fire down upon your enemies from afar, changing the very landscape beneath their feet, and even travelling in time are but a few of the awesome powers at the command of a Vimmer versed in Ex. These are all of the : (colon) commands in Vim. These commands are inherently linewise and can therefore be ungainly on multiline monsters (macros are the recommended weapon when facing multi-lined monstrosities). Read :help ex-cmd-index for an overview of battlefield orders you can issue.

Maps

The map is a ranged weapon providing a sortie of commands with the flick of a wrist. Skilled Vimmers spend hours in training creating map weapons for special purpose attacks. The builtin Edits occupy a vast majority of the available body (keyboard) space.  Careless novices unwittingly replace valuable original Edits with inferior maps, only to be defeated or fatigued in extreme battle.  The master Vimmer chooses his maps carefully, regularly preferring to place them on the purpose-built shield (:help mapleader) so as not to interfere with his perfected mastery of the native Edits. Read :help map.txt to start crafting your own maps of devastation.

Macros

Macros are pre-rehearsed battle maneuvers that can be performed instantly with precise execution. A powerful weapon but dangerous when employed on the wrong terrain. The macro will blindly repeat the instructions it was established with. Regardless of whether this weapon is fired when facing the wrong enemy or even friendly troops, it will charge headlong in to battle until its target is demolished or it has been overcome (an error occurs).  Here are several good explanations of macros and their uses.

VimL

The arcane language of the truly wizened Vimmer, capable of wreaking havoc on even the most worthy opponent. Once a spell is cast its effects are almost instantaneous, but the incantation process is long and complicated. This discipline is better practised at quieter times of relative peace whereupon its power may be distilled into volatile potions, sacred scrolls, charged wands and ready tomes that can be more freely and immediately used on the battlefield. Such artefacts in the land of Vim are called plugins.  To begin on your path as a Vim Mage, read Damian Conway's excellent introduction to Vimscript. The accomplished Vimlglot might prefer a deeper analysis. The holy arcanum is kept in :help usr_41.txt.

Veteran Vimmers and Alacritous Acolytes alike are well advised to review their personal arsenal frequently, polishing tarnished weapons and sharpening dull and forgotten ones. Develop a regimen of daily practise in each of these tools and skills so that you may face your next editing evil with brave heart, right mind and quick hand.

Being virtuous in preparation and valiant in battle,
the nature of a Vimmer is irrepressible!

Advanced Macros in Vim, Commentary

This is a bloated comment on: http://blog.sanctum.geek.nz/advanced-vim-macros/

Sharks in Shells

Firstly, there are at least two plugins that can tabulate text in vim: Align and Tabular. Using plugins written in pure vimscript is usually better than shelling out to system tools, because it's more OS independent. On that same note, Vim has a builtin :sort command, so no need to shell out for that either.

You warn against reaching for the hammer when holding screws; that it's sometimes better to use a splash of :ex and a sprinkle of VimL. Indeed, that is true in this example (starting with the cursor on the first line of the original data table):

  :Tabularize / \+/
  :sort
  :%s/\d\{4}/\=strftime('%Y')-submatch(0)/

Mentioning Macro Mechanics

Yanking a macro line from your buffer into a register with "ayy captures the trailing ^J which, if you have something bound to <Enter>, will produce unwanted side-effects. Use ^"ay$ instead (or the VimLocalMacros plugin).


When assigning macros in a let expression, the double-quote form does allow for Vim map-style markup:

  let @a = 'must have ^M literal chars'
  let @a = "can have \<cr> escaped markup"

There's no inherent benefit from moving this sort of macro into a function - it's so uniquely dependent on your current text file (and year!) that it simply doesn't make sense in this case. However, this trick might be useful in more generic situations. Using a function to preserve macro contents allows us to reuse the register that macro was occupying. Personally, I use macros on an ad hoc basis, crafting them quickly for the need at hand and discarding them when done. I might overwrite register @a several times over within a single edit session with various macros on an as needed basis. However, I was asked one day how to persist particular macros for a given file. The embedded-in-a-function approach shown in the article is one solution. I also wrote VimLocalMacros for this purpose.

Macros stop executing if an error occurs. You can use this to your benefit. Design your macros so they deliberately fail - easy way to not have to count and just provide a large-enough number of runs: 999@a

Recursive Macros have a use and are not something to be dismissed as the wrong tool instead of reaching for a scripting language.

My Vimrc

bairui, y u no?

I've been asked before why my ~/.vimrc is not available online.

I guess, deep down, there are probably feelings of apprehension about releasing something so personal and close to one's heart; that insensitive malcontents might throw stones through the windows and graffiti the walls.

A bigger and more real concern is that of the endemic affliction known as Cargo Culting. Vim is like a martial art. Mastery comes from years of rigorous, dedicated practice, lots of trials and accidents and occasional brilliant successes, study at the feet of other masters and plain old Time at the coal-face. There is no short-cutting this process. You can speed it along with the right approach, but you can not just dump a master's vimrc file at $HOME and think you're playing with the big boys now. That will only lead to tear-stained keyboards and hosed code.

Heh... I'm distorting the truth only slightly here, but I was just told a funny story that allegorises this tragedy: A good friend and fellow vimmer had his Vim set up with :cursorcolumn=80 to remind him where to constrain his code lines. A colleague passing by gasped, "Oh no! Your monitor is broken!". Upon explaining that everything was ok, that the monitor was indeed working fine and that Ti... er, I mean, my friend wanted the line there, the enquirer left mollified and quietly envious of this awesome feature that his Ec, er, I mean, editor didn't have. A week later, my friend discovered his colleague's computer had a dark red line too; only, his was drawn on a transparent plastic strip sticky-taped to the monitor.

:-)

In fairness... the colleague was mocking my friend, so this is not really a story of Cargo Culting, but it shows the phenomenon well, I think. Someone sees something awesome in another person's setup and blindly copies it in the vain hopes it will bring them equal fortune too.

While not a Vim Master, I do have the occasional sharp implement that is best kept away from curious beginners. To that end, I am not just pushing my vimrc up for you to copy blindly. I give you here a snapshot in time. I have annotated it in the hope that it shows you why I chose a certain option or implemented a particular solution. It is my intention that you learn how to wield this cutting edge tool with ample guidance and clear models.

So much for the What and Why. Now a little bit about the How. An expert is not born expert. Years of training go into the growth of his skills, techniques, tools and knowledge. My vimrc has grown over the years in the same way, from humble beginnings through proud days of development and embarrassing times of foolishness all the while with unremitting hubris.

You can see examples of that growth in the snapshot below. In various places I note pieces that might eventually grow to the point of becoming their own plugin. When they do, they will be cut out of my vimrc and put into a standalone plugin of their own. Doing this helps keep the core vimrc down to a manageable size. It also allows me to share it more easily with others.

Finally, the process of cleaning up my vimrc for public release was quite a long and intellectually grueling one. It generated the philosophical challenge: what is my Vim configuration? By this I mean, it occurred to me that my vimrc was not the entirety of my Vim configuration. I have extracted and encapsulated within plugins many times over productive growths from my vimrc throughout the years. They serve me well still, sitting mostly quietly at the sides, doing their jobs unnoticeably well. To forget these pieces and not include them in my configuration would be disingenuous. They provide many of the commands that my muscle memory relies on within Vim today. They are my Vim configuration; certainly as much as my vimrc can claim to be, anyway.

My Vimrc

" ~/.vim/vimrc_common
" Barry Arthur

" My 'office' has several people using Vim who would like to have a pleasant
" configuration without needing a neckbeard to get it. Sharing all of my vimrc
" with them would not only give them unicorns and rainbows, but would also
" leave them with dangerous sharps and nasty unexpecteds. To solve this
" problem, the vimrc is split into three parts:
"
" * A common shared by all (this file)
" * A $USER pre for things that the common depends on (like mapleader)
" * A $USER post for things that the user would like to override

" Use Vim not vi
"----------------
set nocompatible


" Initialize Plugin Manager
"---------------------------
" https://github.com/Raimondi/vim-pathogen
" Raimondi's pathogen allows multiple bundle dirs and
" provides a command interface to interrogate, enable and
" disable plugins.

call pathogen#infect('bundle/shared', 'bundle/local')

" bundle/shared contains plugins used by all
" bundle/local is for personal plugins


" Filetype & Syntax Highlighting
"--------------------------------
filetype plugin indent on
syntax on


" User Pre File
"---------------
" For things like (:help mapleader)
" Note: use <leader>gf to jump to the file
source $HOME/.vim/vimrc_${USER}_pre


" Options
"---------
" Use :help 'option (including the ' character) to learn more about each one.

" Buffer (File) Options
set hidden                     " Edit multiple unsaved files at the same time
set confirm                    " Prompt to save unsaved changes when exiting
set viminfo='1000,f1,<500,:100,/100,h  " Keep various histories between edits

" Search Options
set hlsearch                   " Highlight searches. See below for more
set ignorecase                 " Do case insensitive matching
set smartcase                  "   except when using capital letters
set incsearch                  " Incremental search

" Insert (Edit) Options
set backspace=indent,eol,start " Better handling of backspace key
set autoindent                 " Sane indenting when filetype not recognised
set nostartofline              " Emulate typical editor navigation behaviour
set nopaste                    " Start in normal (non-paste) mode
set showmode                   " Necessary to show paste state in insert mode
set pastetoggle=<f11>          " Use <F11> to toggle between 'paste' and 'nopaste'

" Indentation Options
set shiftwidth=2               " Number of spaces for each indent level
set softtabstop=2              " Even when using <Tab>'s
set expandtab                  " When pressing <Tab>, replace with spaces

" Status / Command Line Options
set wildmenu                   " Better commandline completion
set wildmode=list:full         " Expand match on first Tab complete
set showcmd                    " Show (partial) command in status line
set laststatus=2               " Always show a status line
set cmdheight=2                " Prevent "Press Enter" message after most commands
set statusline=%f%m%r%h%w\ [%n:%{&ff}/%Y]%=[0x\%04.4B][%03v][%p%%\ line\ %l\ of\ %L]

" Interface Options
set number                     " Display line numbers at left of screen
set visualbell                 " Flash the screen instead of beeping on errors
set t_vb=                      " And then disable even the flashing
set mouse=a                    " Enable mouse usage (all modes) in terminals
set ttimeout ttimeoutlen=200   " Quickly time out on keycodes
set notimeout                  "   but never time out on mappings
set list                       " Show tabs and trailing whitespace
set listchars=tab:⇥\ ,trail:·  " with nicer looking chars
set shortmess=atI              " Limit Vim's "hit-enter" messages


" Key Maps, Functions, Commands and Abbreviations
"-------------------------------------------------

" General Editing
""""""""""""""""""

" Help terms can include characters typically not considered within keywords.
function! ExpandHelpWord()
  let iskeyword = &iskeyword
  set iskeyword=!-~,^),^*,^\|,^",192-255
  let word = expand('<cword>')
  let &iskeyword = iskeyword
  return word
endfunction

" F1 to show context sensitive help for keyword under cursor
" Deliberately left off the terminating <cr> to allow modification.
" Moves the cursor to the beginning of the term for contextual markup.
" (:help help-context)
nnoremap <F1> :help <c-r>=ExpandHelpWord()<cr><c-left>

" F9 to show VimL function list
nnoremap <silent> <F9> :help function-list<cr>

" Grep help files for a pattern
command! -nargs=* -complete=help HG helpgrep <args>


" Resize windows
nnoremap <leader>h :exe "resize " winheight(0) / 2<cr>
nnoremap <leader>H :exe "resize " winheight(0) * 2<cr>
nnoremap <leader>w :exe "vertical resize " winwidth(0) / 2<cr>
nnoremap <leader>W :exe "vertical resize " winwidth(0) * 2<cr>

" Move between windows
nnoremap <a-left>    <c-w>h
nnoremap <a-down>    <c-w>j
nnoremap <a-up>      <c-w>k
nnoremap <a-right>   <c-w>l


" Move up and down visually across wrapped lines
nnoremap <Down>  gj
nnoremap <Up>    gk

" Move left and right across whole (:help WORD)s
nnoremap <Left>  B
nnoremap <Right> W


" Execute current line
nnoremap <silent> <leader>E :exe getline('.')<cr>

" Toggle list mode
nnoremap <silent> <leader>L :set invlist list?<cr>

" Reformat paragraph
nnoremap <silent> Q gqap

" F2 to toggle spelling
nnoremap <F2> :set invspell spell?<cr>

" F3 to trigger snipmate
" Note: This map might change in the future; it's
" currently part of an extended <Tab> experiment.
let snips_trigger_key = '<F3>'

" F6 to Toggle the TagBar
nnoremap <silent> <F6> :TagbarToggle<cr>


" Diffing
"~~~~~~~~~
" Note: This is begging to be moved into a Diffing plugin

" View changes made to a buffer since it was last saved
" storing original line in global g:diffline because it's used by several
" different diff commands - do and dc
function! DiffOriginal()
  let g:diffline = line('.')
  vertical new
  set buftype=nofile
  " 0read prevents a blank line at the top of the buffer
  " ++edit retains current option values for read operation
  " # is the alternate buffer (:help :_#)
  0read ++edit #
  diffthis
  exe "normal! " . g:diffline . "G"
  wincmd p
  diffthis
  wincmd p
endfunction

" Demonstration of creating a command that calls a function.
command DiffOrig :call DiffOriginal()<cr>

" Demonstration of a map calling a command.
nnoremap <silent> <leader>do :DiffOrig<cr>

" Close a DiffOrig session
nnoremap <silent> <leader>dc :q<cr>:diffoff<cr>:exe "norm! ".g:diffline."G"<cr>


" Highlighting
"~~~~~~~~~~~~~~
" Note: These might move into the SearchParty plugin

" Temporarily clear highlighting
nnoremap <silent> <c-l> :nohlsearch<cr><c-l>

" Toggle search highlighting (:help set-inv  :help E518)
nnoremap <c-Bslash> :set invhlsearch hlsearch?<cr>

" F8 to highlight all occurrences of word under cursor
" Note: Needs a better key. <F8> seems arbitrary
nnoremap <silent> <F8> :let @/='\<'.expand('<cword>').'\>'<bar>set hlsearch<cr>


" Make
"~~~~~~
" Note: Again, a candidate for extraction into an external plugin

function! QuietMake()
  write
  silent! make
  copen
  if empty(len(filter(getqflist(), 'v:val["valid"]')))
    cclose
  else
    wincmd w
  endif
  redraw!
endfunction

" Demonstration of calling a function from a map
nnoremap <silent> <leader>ma :call QuietMake()<cr>


" Find-in-Files
"~~~~~~~~~~~~~~~
" (:help :abbrev :lvimgrep :lwindow)
cabbrev lvim
      \ lvimgrep /\<lt><c-r><c-w>\>/gj
      \ *<c-r>=(expand("%:e")=="" ? "" : ".".expand("%:e"))<cr>
      \ <bar> lwindow <c-left><c-left><c-left><right>


" Plugins
"---------

" Matchit - comes with Vim but needs to be enabled (:help matchit-activate)
runtime macros/matchit.vim

" Third-party plugins:

"" https://github.com/tpope/vim-surround.git
"" https://github.com/scrooloose/nerdcommenter.git
"" https://github.com/scrooloose/nerdtree.git
"" https://github.com/garbas/vim-snipmate.git
"" https://github.com/honza/snipmate-snippets
"" https://github.com/vim-scripts/Tagbar.git
"" https://github.com/vim-scripts/NumberToEnglish.git

" Home-grown plugins:

"" https://github.com/dahu/Insertlessly.git
"" https://github.com/dahu/Nexus.git
"" https://github.com/dahu/Firstly.git
"" https://github.com/dahu/Zzzomg.git
"" https://github.com/dahu/vim-mash.git
"" https://github.com/dahu/SearchParty.git
"" https://github.com/dahu/Vimple.git
"" https://github.com/Raimondi/vim-buffalo.git
"" https://github.com/Raimondi/vim-pathogen


" Auto Commands
"---------------
" (:help :autocmd)
if has("autocmd")
  augroup vimrc
    au!

    " Jump to last-known-position when editing files
    autocmd BufReadPost *
          \ if line("'\"") > 1 && line("'\"") <= line("$") |
          \   exe "normal! g'\"" |
          \ endif

    " Default omni completion based on syntax
    if exists("+omnifunc")
      autocmd Filetype *
            \ if &omnifunc == "" |
            \   setlocal omnifunc=syntaxcomplete#Complete |
            \ endif
    endif
  augroup END
endif


" GVim
"------
if has("gui_running")
  set t_vb=                    " t_vb gets reset when entering GUI mode
  set guifont=Terminus\ 14
endif


" User Post File
"----------------
" For overriding the common settings defined here
" Note: use <leader>gf to jump to the file
source $HOME/.vim/vimrc_${USER}_post

Resist Read Only Realities, Part I

Part I - The Beast Bespoke

One thing I frequently find limiting in Vim is the disturbingly all
too frequent read-only commands that display data in a pager-like dump
with the only options being: space, d, j, b, u, k and q for moving
down and up and quitting the pager. This is lame. Sometimes we want
to be able to interact with that information, or see a filtered subset
of it, or have at it with even more freedom and flexibility to do with
as we choose.

Take for example, the :scriptnames command that shows all of the
scripts Vim has loaded since the session started, in load order. We
most commonly use this list to check for the presence of a new plugin
we've installed, to confirm that it's loading correctly. Indeed, this
is one of the diagnostics #vim denizens will demand of you when
presenting with plugin failures.

If all you have is a small handful of scripts, then perusing the
output of :scriptnames manually in a dumb pager is tolerable, if not
enjoyable. If you have a lot of plugins then this quickly becomes
painful. Wouldn't it be nice to be able to filter your :scriptnames
list so that it only showed scripts matching the pattern you're
looking for? What we crave is something like:

:scriptnames /\cnerd/

to select only the loaded scripts matching the case-insensitive
pattern 'nerd' (presumably to catch one of either NERDCommenter or
NERDTree.)

That would be nice. That is what we desire. But we are left wanting.
In fact, we're left with the unsavoury message:

E488: Trailing characters

Are we doomed to a baleful life of blind bumbling through screenfuls
of awkwardly navigated, 1970's style non-interactive data dumps?

No. We don't have to suffer this injustice. We are vimmers. We have
recourse. We can solve this problem with Vim's :redir command, in
various ways, such as:

• :redir > somefile
• :redir => somevar
• :redir @{a-zA-Z*+"}>   (to a register)

The Standard Redir Recipe

Solving the :scriptnames problem above with the file redirection solution:

:redir >myscripts
:silent scriptnames
:redir END
:edit myscripts
:v/\cnerd/d

This works, and isn't too painful to do manually when you need it,
so long as you don't need it too often. The :scriptnames command is a
prime example of one you wouldn't need to call very often. Having to
choreograph the five line dance above once in a blue moon is no great
ordeal, provided you can easily remember the right steps in the right
order.

Is this as good as it gets?

Well, actually, no. This is just the first step, showing you the crux
of the problem and the simplest of Vim's built-in solutions. Coming up
next, I will show you how to take it to the next level with a splash
of split, filter, join love.

Vim Can Haz Math

I got asked today how I might tackle the problem of calculating the arithmetic difference between values in a series. Take for example this mythical data file:

date: 2012 01 01
weight: 90kg
date: 2012 02 01
weight: 87.5kg
date: 2012 03 01
weight: 85.4kg
date: 2012 04 01
weight: 83.2kg
date: 2012 05 01
weight: 80kg

With the desired outcome of:

date: 2012 01 01
weight: 90kg
date: 2012 02 01
weight: 87.5kg (-2.5)
date: 2012 03 01
weight: 85.4kg (-2.1)
date: 2012 04 01
weight: 83.2kg (-2.2)
date: 2012 05 01
weight: 80kg (-3.2)

This is what I used:

:g/weight:/ copy . | silent! ?weight:??? copy . | - join | s/// | s//-/ | s/kg//g | s/.*/\='('.string(eval(submatch(0))).')'/
:g/weight:/ join

Slow Motion Replay
If that makes perfect sense to you and you're even wondering why I was so verbose in some places, then I have nothing left to teach you here. Otherwise, let's break this down to show what's going on:

Note:

• Vim uses the | character as a command separator, like the ; in C.
• Vim's :ex mode has a notion of the current line which many commands use as their default source or target address.

1. :g// is Vim's global command. It finds all lines in the buffer matching the specified pattern, in this case: /weight:/. On each match, it sets the current line to the line of the match and runs the series of | separated commands given to it.

Note: Some of the internal commands within this sequence can alter the current line, affecting the subsequent command's notion of where the current line is.

2. copy . duplicates the given address range to below the current line. With no explicit address given then the implied source address is the current line. This effectively duplicates the weight: found by the :g// command below itself. It also sets the current line to the destination address, so the next command in our chain will implicitly operate on the duplicated line as its current line.

3. silent! is used in this pattern in case the user has :set nowrapscan (which would cause the ?weight:??? to fail on the first matching line in the file, prematurely terminating our :g// command. If the user has :set wrapscan enabled, then the last weight: in the file will be found here. Either way, cleaning up the first line is a manual exercise for our hapless user in this tutorial.

4. ?weight:? searches back from the current line to the prior match of weight:. Because our current line was reset by the copy command to be the duplicated line, then a single search backwards will merely find the line we copied from. That's not good enough. We want the next one back again from that one. However, the ?? command, when chained, sets the current line again, so a subsequent ?? immediately after will find the next prior weight: line.  When a search (either forward with // or backward with ??) is used without an explicit search term, Vim uses the prior search term implicitly, so ?? means search backwards for... weight: (because that's what we last searched for, in the ?weight:? command to start with).

Question for the attentive: Why did I give the explicit pattern ?weight:? in that command and not rely on the prior implicit search pattern?

5. Remember that copy . duplicates the given address range to below the current line. The given address in this case was explicitly provided by the search in #4, which points at the actual prior weight: . The current line is the duplicated weight line from step #2, so this command will copy the second prior weight line to below the duplicated weight line. Also remember that the copy command resets the current line to that of the destination address. It might be instructive to see what a sample of the file would look like if we halted our command here. That is, if we were to run the command:

:g/weight:/ copy . | silent! ?weight:??? copy .

We would get:


... 
date: 2012 02 01
weight: 87.5kg
weight: 87.5kg
weight: 90kg               <-- current line
date: 2012 03 01
...

That is assuming you have :set nowrapscan . However, if you have :set wrapscan , you will actually see:

... 
date: 2012 02 01
weight: 87.5kg
weight: 87.5kg
weight: 80kg               <-- current line
date: 2012 03 01

...

Note: That rogue 80kg comes from the fact that the first match of the :g// command is at the top of the file so the subsequent ?weight??? wrapped backwards around the file, finding the bottom-most entry instead — which is 80kg in our sample data file.

But we're not done, so let's continue on. Here's the whole :g// command again:

:g/weight:/ copy . | silent! ?weight:??? copy . | - join | s/// | s//-/ | s/kg//g | s/.*/\='('.string(eval(submatch(0))).')'/

Remember that our current line is as indicated in the samples above.

6. - join moves back a line (from the current line!) and joins the following line to the current line. The result of step #6 from the three weight: lines shown in step #5 is:

weight: 87.5kg
weight: 87.5kg weight: 90kg     <-- current line

It's really instructive for you to run this partial command up to this point yourself to see the goblins I'm ignoring by deliberately choosing the second weight: match within the file. It's only a white lie that will all be cleared up later anyway, so I don't feel too bad about it.

The remaining 4 commands in the chain are substitutions which strip unwanted non-numeric pieces from the line, shape it into an arithmetic subtraction expression and evaluate it to produce the arithmetic difference between the two numbers. The following lines show the result of each command in turn applied to the result of step #6.

the s/// results in:

 87.5kg weight: 90kg

the s//-/ results in:

 87.5kg - 90kg

the s/kg//g results in:

 87.5 - 90

and, finally, the s/.*/\='('.string(eval(submatch(0))).')'/ results in:

(-2.5)

At this stage, the file looks like this:

date: 2012 01 01
weight: 90kg
weight: 90kg
date: 2012 02 01
weight: 87.5kg
(-2.5)
date: 2012 03 01
weight: 85.4kg
(-2.1)
date: 2012 04 01
weight: 83.2kg
(-2.2)
date: 2012 05 01
weight: 80kg
(-3.2)

We want the differences at the end of the preceding weight: lines, so we'll use another :g// command for that:

:g/weight:/ join

Which leaves us almost done. The only bugbear remaining is the duplicated first weight in the file:

weight: 90kg weight: 90kg

Clean that up manually.

Reflection

I could have approached this in a number of different ways, but to my mind, this seemed to be the quickest and easiest approach. Other approaches might include using a macro instead of a global command, or writing a full-blown VimL script.

I tend to build these things up piecemeal, testing as I go. I follow the same methodology when constructing SQL commands. Run a partial to prove to yourself that it's good so far. Press the u key to undo and add your next chunk. Repeat until you're done.



Writing up this article to explain the solution took twenty times longer than scratching the solution out for the requester in the first place.

Oh, the answer to my earlier question? Did you figure it out? The explicit pattern given in s/kg//g forces me to be explicit again at the start of the chained commands within the :g// command.

Undo & Continue Confirmed Substitution

Janni on #vim asks:

When using substitution (:s) with the [c] flag one can use several keys to
confirm/skip concrete substitutions. I'd really like to get the additional
option to undo the last substitution.  (without exiting the substitution
process as a whole)

Here's how I would handle that situation:

For demonstration purposes, let's replace "e" with "z".

We have to start on the first line, so type   gg

The following command operates on all lines from the cursor to the end of the
file which contain an "e":

:.,$g/e/s//z/gc

The /c on the :substitute command there prompts us at every replace position. See :help :s_flags for an explanation of the interactive options.

If you want to undo the previous one, press   ctrl-c   to cancel the :global
command and then press   u   to undo and   j   to move down a line (skipping
over the line you just undid so as not to be asked about it again). Then re-run
the   :.,$g/e/s//z/gc   command. The easiest way to do that is to press
:<up><enter>

It'd be nice to have an extra   u   option in the interactive confirmed
substitution command to undo the last replacement. Until then, enjoy.

Update:

The ever  vigilant Raimondi caught me with my pants down again. It turns out, we don't need the extra :g// at all here. A simple:

:.,$s/e/z/gc

is sufficient. It has exactly the same runtime mechanics as the method described above — that is, use ctrl-c to cancel the operation midway, u to undo changes and then :<up><enter> to recommence the substitute from the current line onwards. Thanks for keeping me honest, Raimondi. <3

Your gem, my rock:

Sir Raimondi followed up with another eloquent query:

y u no @: bairui?!

Ah... good question. Ostensibly, I didn't show @: because I rarely use it myself. For my needs, :<up><cr> is in the muscle memory... Besides, @: is not necessarily easier for me to reach. Also, it gives me a chance to eye-ball it before accidentally re-running the wrong command. My supply of brown underwear is limited so it behooves me to be a little cautious around my :ex.

But yes, @: and then subsequent @@ is faster if you know you have to re-run an :ex command many times, as is the case here... but if you're having to do that a lot manually... I smell fishies... :-) Of course, I just polluted your wading pond with a load of carp by suggesting you do exactly that above — what can I say? When you know the rules, you can break the rules?   :-p   Love as always, R.