Thursday, March 29, 2012

Saturday, March 17, 2012

Paste Me A Column

Have you ever needed to paste one chunk of text linewise besides another column? The *nix tool, paste, does this nicely and is the recommended way to solve this problem (and is demonstrated in solution #2). In this article, I attempt to compare various approaches to solving this problem in Vim.

Two Data Sets

Data Set 1: (possibly in a file called data_set_1.txt)


Data Set 2: (possibly in a file called data_set_2.txt)


(goal): (possibly clobbering data_set_1.txt)

  if    you
  don't know
  say   so

(actual tab separators are not necessary — a single space is sufficient)

NOTE: I am assuming that all data-sets are equal in length (number of lines)


1. (insert mode) — hand editing the column of data

Certainly the lowest-tech solution considered here, and for small data sets, probably the fastest solution. For larger or complicated data sets, though, this solution is just not workable. It's mentioned here only for completeness.

The following solutions all assume that the current buffer holds data_set_1.txt and the alternate buffer holds data_set_2.txt

2. (external command) — :!paste

  :%!paste - #

NOTE: The - tells the paste command to read data set 1 from the standard input stream — which is being sent to paste by Vim with the   :%   part of that command line (in a range, % means every line in the buffer).

If you don't want to load data_set_2.txt into the alternate buffer, you could instead use:

  :%!paste - data_set_2.txt

You'd want a really good reason not to be using this solution. It's the shortest, fastest, most reliable solution presented here. If your operating system doesn't offer you a paste command, then you can download yourself a real one for free.

This solution requires at least the second data set to be saved in a file and not just represented as a clump of lines further down in the same file. In that situation, you can either cut them out to a separate file and use the :!paste command, or look at some of the following approaches.

3. (normal mode commands only) — using blockwise cut & paste

  ctrl-6                  " switch to the alternate buffer (data set 2)
  1G 0                    " move to line 1, column 1
  ctrl-v G $ y            " blockwise yank data set 2
  ctrl-6                  " switch back to buffer holding data set 1
  1G                      " move to first line
  A<space><esc>           " append as many spaces as necessary past
                          " longest entry in left column
  p                       " (blockwise) paste

Next to hand-typing the data in, this is the most labour-intensive approach, especially if you have many paste operations to do. Some of the later examples have a costlier setup (having to type more initially), but will execute faster over anything more than a very few paste jobs. If you only have to do one, this solution might not be too bad. Another problem with this solution is having to ensure that you paste past the longest entry in the left column by padding with spaces beforehand.  This is clumsy, frustrating and prone to cause blunders if not boredom.

Data Set 2 in a Register or Variable

The following examples assume that data set 2 has been read into a register or variable, as in:

  let data = readfile('data_set_2.txt')

OR, if the data was in a chunk in the original file, like:


you could:

  "{visually select the lines using shift-v}
  :let data = split(@", "\n")

4. (ex) - substitute


This solution requires that the data sets are the only things in the buffers and that they start at the first line.

A slight modification to the range allows an alternate start of the data set. In all the cases below, the range starts on line 6:

Using either a visual selection over the range:


Or an explicit range:


5. (ex) — global substitute

  :let x = 0 | .,$g/./s/$/\="\t".data[x]/ | let x += 1

The beauty of the :global command over the plain :substitute command is that it handles the | (bar — command separator in Vim) differently. The :global command sees the | separated pieces as part of its argument — executing the series of commands on each line matched in the global pattern. (The :substitute command does not treat | as part of its argument and is therefore terminated/separated by the |, allowing another command to execute after the :substitute has finished (across all of the lines it was issued against in its range).

In this example, a variable (x) is used to index the data array instead of the line('.') kludge used in the :%s/// solution #4.

Data Set 2 below Data Set 1 in same file

The following examples assume that the two data-sets are in the same file, separated by a single blank line.

6. (ex) — global move

  "{position the cursor on the first line of data set 2}
  :let x = 1 | .,$g/^/exe "move " . x | let x += 2
  :g/./normal J

This example assumes that data set 1 starts on line 1. Change the let x = 1 to the line number starting data set 1 if that's not the case. Also, I am assuming that the two data sets are the only thing in the file — you can adjust the ranges or global patterns to suit otherwise.

7. (macro) — normal commands on meth

  let @m="}jdd\<c-o>pkJj"

and then:

  "{navigate to the first line of the top-most data-set}
  3@m                     " execute the macro 3 times

Of course, you'd replace 3 there with the actual number of times you needed. If you don't know the exact number, or you're lazy to type it, you can fudge it with an obviously oversized alternative, as in:

  999@m                   " automatically stops on error

NOTE: My choice of   m   as the macro register there was totally arbitrary. You have 26 named registers (a-z) to do with as you choose.

Also, although I show the macro assignment as a let statement here, it is typically not crafted as such. The more usual approach when creating macros is to bang away at the set of normal commands needed like a monkey chained to a typewriter. Eventually you will mash out your Shakespearean macro masterpiece. The   q   command in Vim is used to record macros for later playback with the   @   command. A typical way to record a macro into register m might look like:


Where the first line clears register m and the second begins, sets and ends the recording — with q stuff q respectively. That hideous   ^O in there represents the   ctrl-o   needed to jump backwards in the position list, and is an actual control character. For this reason, explicitly let-ting macro expressions as shown above is clearer and more portable.

Thanks to osse for his cleaner macro example than I'd originally chosen.

So much for insert, external, normal and ex vim approaches. How about VimL?

8. (vanilla viml)- procedural approach

Vim doesn't have an explicit zip array function that merges successive source elements into a result array, as in [a,b,c].zip(1,2,3) ->
[[a,1],[b,2],[c,3]]. If it did, we could use it to solve our problem here. So, let's write one:

  function! Zip(a, b, sep)
    let i = 0
    let r = []
    let n = min([len(a:a), len(a:b)])
    while i < n
      call add(r, join([a:a[i], a:b[i]], a:sep))
      let i+= 1
    return r

Assuming that the current buffer holds only data set 1 and the alternate buffer holds only data set 2, then:

  :call setline(1, Zip(getline(1,'$'), readfile(expand('#')), "\t"))

9. (newVimL) — because you just knew this had to be here :-)

It turns out, when you're dealing with functionalish languages, they do provide more wow-factor than the first glance over the docs would suggest. Awesomeness is lurking beneath innocuous looking, humble little function names that you'd just glaze right over if they weren't pointed out to you. We're going to look at the functional power-house, map().

  "{navigate to the blank line separating the two data sets}
  :call setline(1, map(getline(1,line('.')-1), 'v:val."\t".getline(line(".")+1,"$")[v:key]'))

Again, I assume here that our file contains only the two data sets and that the first one starts on line 1 and the second set starts after a blank line after the first set.

This code will not delete the second data set — effectively a copy of it was made beside the first data set. In some cases it might be desirable to have this unaltered copy of the second set to remain in the file, but if not, just delete it after the column paste is done.

This map has an ugly wart at the moment — it's inefficient. The replacement expression re-calculates   getline(line('.')+1,"$"   for every line in data set 1. Yuck. We can optimise that away using the same data array trick we did earlier, yielding the cleaner and more efficient:

  :call setline(1, map(getline(1,'$'), 'v:val."\t".data[v:key]'))

NOTE: This version is not affected by the location of the cursor as the previous version was.

If you delete into that data array then you clean up your duplicate copy of data set 2 problem from the outset.

Price of Tea

If we knew from the outset that shelling out to the external paste command was the best way to do this, why did we bother with all of the other approaches? For one thing, we have the paste command for this example — your next hairball might not have a *nix command lurking beneath the shell to help you. Secondly, it's the journey, not the destination. Having walked this way and picked at the various shrubs along the path we have collected a nice potpourri of Vim buds to burn at our leisure on a quiet Sunday afternoon, or torch in anger when faced with the next gnarly editing requirement our tempestuous day-jobs throw at us.

Prior Preparation Prevents Piss-Poor Performance.

Vim on.

Tuesday, March 13, 2012

Friday, March 9, 2012

Protocol Based Lifestyle

I've had some cool mentors throughout my working life. Sadly, I never worked under a coding master, but I did have a few mentors in other aspects of my professional (heh, and personal) development.

One such mentor was Master Juffy (Jeffrey Parker). He taught me both indirectly (through his many jokes, stories and real-life examples (occasionally (and most hilariously) illustrating what not to do)) and directly through his protocols.


Juffy had protocols for all aspects of his working and home life. We had Lift Protocol clearly stipulating that those on the lift alight before those standing outside attempt to get on. There was Coffee Protocol wherein the first staff member to arrive at work in the morning was responsible for putting on the first carafe of coffee; and the closely related Refill Protocol requiring the staff member who drained the last drop of coffee from the carafe to put on a new one.

We had many such colourful and humorous protocols, almost all created by Juffy to maintain a happy workplace atmosphere. Exemplary among them was Fuck Off Protocol. This afforded any staff member who was feeling the pinch of a deadline to coldly say, "Fuck off!" to anyone who interrupted them without causing hard feelings, insult or injury to the audience. And it worked. We understood that there was no malice in those words; it was the protocol - the right thing to do at the time. And we were glad for the comfort of it.

Dumb Protocol

One of the more amusing (and often used) protocols was Dumb Protocol. This protocol was prompted by Juffy's then 3 year old boy. Whenever the child had detected that a human inefficiency had just occurred, he would loudly proclaim to the room, Dumb! Dumb! Dumb! Dumb! Dumb!!! He usually operated in broadcast fashion like this, but he had a point-to-point protocol as well, like when Juffy made a goof at home: Dumb, daddy! Dumb!!!

Sadly, my work took me away from the cultivated cloister of Juffyland, but I always remembered my Protocol Based Living lessons learned under the tutelary Master Juffy.

My Modern Protocols

My girlfriend, being a nerd like me, was an agile adopter of the Protocol Based Lifestyle. Between us, we have our house and daily routines whipped in to shape with a poignant pack of protocols. Washing Protocol: clothes (yes, separated, darks from lights) are placed in to the washing machine throughout the week and only when it is filled is it started (always checked at around 6am when I get up). Loo Protocol: succinctly (and colourfully) expressed with the pithy poem - if it's yellow, let it mellow; if it's brown, flush it down. Shower Protocol: household members stagger their showers (typically around dinner and the evening vids) sufficiently far apart from one another so as to allow the hot water heater time to do so, affording hot showers for all in the house. Kettle Protocol: Our gas kettle has a little flip lid over the spout that serves no real purpose other than disrupting the flow of water when poured so as to spill messily over the tabletop and avoid wetting the insides of the cup beneath it (yes, we learned quickly to flip it out of the way before pouring from it, but only after painting several puddly portraits). As you'll read below, we have a real need in this part of the world to be certain that the water in the vessel has indeed been boiled. Kettle Protocol requires us to flip that spout lid as soon as it's boiled, advertising its availability for all approaching suitors. And I could go on with many more protocols that keep our lives running smoothly and often humorously... but let's talk about the birth of a new protocol:

Urn Protocol

We have a new electric push-button urn with a little indicator light that shows when it is re-boiling. This can be a bit hard to see, especially when your mind is still trying to parse that last piece of hairy newLISP or decide what it's going to enjoy for dinner. One weekend a few years ago in Chiang Mai (Thailand) my girlfriend and I helped ourselves to the hostel's urn for a nice cup of hot coffee each. It wasn't, though. Hot, I mean. But we foolishly finished our coffees anyway and spent the next three days violently spraying various smelly liquids out of both ends all over the walls and floors of our poor, rented, little bathroom. Lesson learned - in Asia, let the urn come to the full boil before drinking from it. Which brings me back to our swanky new push-button urn... That little indicator light just wasn't enough to warn us that she wasn't yet done boiling our water for us... We tipped out several cups of coffee & tea before realising we were in desperate need of a New Protocol. It has a swinging handle which rests at the back of the unit when not in use. If rotated all the way forward it actually sits across the control panel, preventing access to the pour button. Awesome! That's our re-boiling indicator! There's the birth of Urn Protocol: When the handle is forward and the little red light is on, the unit is not safe to drink from; wait. When refilling the urn, rotate the handle forward to show that it's not yet potable. When finding the urn with its handle rotated forward AND the little green ok-to-drink-now light is on, rotate the handle back out of the way and drink happily knowing the bathroom walls will be dry tonight.

Protocol Based Lifestyle

We can go months without creating any new protocols, or several new ones might appear in a week, as needed. We sometimes review old protocols and discard ineffective ones for more efficient ones. If something goes wrong we analyse it with an eye to crafting a new protocol around it to prevent subsequent recurrence. This is how we live our Protocol Based Lifestyle. I heartily recommend it to any erudite nerd attending this now... Adoption Protocol.

Sunday, March 4, 2012

It's the Spirituality, Stupid

I'm not a god person

I established that aphorism when I was in grade two, primary school. I got my mother to write a note excusing me from Religious Instruction classes because even at that age, I knew that shit was not for me.

In the many years since then, I have variously been exposed to, seen or investigated a multitude of both religious and spiritual things, the outcome of which allowed me to change my sutra to:

I am not a religious person

God (for certain values of god) gets a second look, but all that religion malarkey is just not for me.

What's the difference?

I see it as spirituality vs religion; the belief in a set of universal laws vs man-made garbage; the territory vs the map.

We made religion. It's a set of protocols and procedures, many of which may have once had a significant and useful purpose, but have long since lost relevance; or we've lost the real understanding behind their purpose.

The problem I have with religion

Actually, I have many, but here's the essence of them all:

It tends to be used as a sheep herding tool to control the masses. Baaah.

It's the Spirituality, Stupid

In my early days I hated all things goddy so much that I threw the spirituality baby out with the religious bath water. It took years of sometimes guided, often blind exploration in the spiritual swamplands to finally stumble onto my own beliefs in this area. Spirituality is the essence of godness that survived scrutiny.

Again, what's the difference? Or, what's the essence? How about an example?

Praying 2.0

Take prayer for example. I don't know what you were told about prayer, whether you were told how to pray or what to pray for, but due to my complete lack of RI (thank god) in the formative years, I have absolutely nfi about the ins and outs of prayer. For many years, Prayer(tm) was just not a part of my daily routine. It never has been, isn't now and never will be! Well, at least, that's how I saw it until a few months ago (or years, I don't really know when I had the first inkling of this idea form in my little skull, but conscious acknowledgement of it can't be more than a few years at most).

I can't believe it's not prayer

Without realising it or referring to it as prayer, I have been praying; kicking it new-school.

What's wrong with the old prayer model?

I see two major faults:

1. maybe you're not told how to pray, OR WORSE
2. maybe you're told the wrong way to pray

If you're suffering from problem #1, there's a good chance you're wasting your time, or at worst, you're accidentally doing it the way sufferers of problem #2 were instructed to do so, causing more harm than good.

What's the wrong way to pray?

There are a bunch of different wrong ways, like focusing on the sad or bad parts of life, or worrying about the hows (details). Let's look at what I believe to be the right way to pray. It boils down to two fundamental aspects: gratitude and intentions.


Spend a little bit of time every day (or more often as appropriate) being grateful for the good things you have & have done, and the good people you know. Find something to be grateful about in all aspects of your life - work, colleagues, home, neighbourhood, circle of friends (both in RL and online), and even daily chores, like shopping, cooking, etc.


Spend even more time every day intending better. This can be done in a big session just before falling asleep at night (*cough* *prayer* *cough*), and in small JIT sessions throughout the day. JITI, or Just In Time Intending is such a fun little game to play: As you're walking to work (or moving toward some next event), envision a great outcome (or what Ferris calls Wild Success) for that next event.

When envisioning greatness, focus on the what (what you want and what you'll do with it once you've got it) and not on the how (the details of how to achieve what you want - let the universe fill in that part). Dooley suggests adding trivial details to add realism; spice it up.

I use a third intending session just before getting out of bed in the morning. I think to myself: What cool shit am I gonna do today? By thinking about the various awesome things I could be doing instead of just lying there in bed, I get excited about dragging my arse from under the warm quilt out into the cold morning. It works for me! I'm usually up around 05:30 eagerly diving into a half-finished programming book, trying out new languages, finishing off various coding projects, chatting with friends and enjoying that delicious first cup of coffee... FLML. :-D

Further Reading

Some of you will have smelt the Law of Attraction underpinnings in this post, or possibly older throwbacks to one of Covey or Robbins or Carnegie or Hill's books. It's all good. ;-)

Thursday, March 1, 2012

First Storm of Spring

Ha! It's March 01, the first day of Spring and we're gifted with our first storm since... last Summer or early Autumn. Love storms... Haven't heard a single peal of thunder all winter; grumbling weather appeals to my inner grouch. Boom!