How to process log files with vi

Add newlines where needed

:s/></>^M</g

Remove the lines that are irrelevant

:g/thevalue/d

Remove patterns in a line that are irrelevant

:s/long.\{-}value//g

Modifying logs files so that only the relevant information is shown does wonders when finding a problem. The steps above are 'vi' commands which will help with log file processing.


Add newlines where needed

Quite often we work with XML of JSON types of documents. All the data is squished together and a bit of space will help.

^M - the way to achieve this is by entering <control> v <return>

Command breakdown:

    :   - go into ex mode in vi
    s/   - substitute the pattern that follows
    ><   - the pattern of less than and greater than
    /    - the pattern next will replace the previous 
    >^M< - new pattern ^M 

Before:

<content>joe</content><content>sally</content>  

After:

<content>joe</content>  
<content>sally</content>  

Remove the lines that are irrelevant

Removing lines that you know are not useful will clean-up your log. Do make sure that this is a copy of the log and not the original. Also always keep in mind that the lines you think are irrelevant can actually be relevant.

Command Breakdown:

    :   - go into ex mode in vi
    g/   - global action
    thevalue   - the pattern to look for
    /    - end of pattern 
    d    - delete the whole line 

Remove patterns in a line that are irrelevant

There will be times when the line is quite long and there are parts of the line which have repetitive information. Removing the information can make the relevant data clearer to see.

This command is identical to the first command.
Two tricks here:
1. You're replacing the pattern with nothing
2. You want a non-greedy search

The regular expression : long.*value
represents all characters between long and value. The reason why non-greedy is needed is because of this:

long abcdefg value useful content value again  

The greedy result will be:

again  

The non-greedy result will be:

useful content value again  

Command Breakdown:

    :   - go into ex mode in vi
    s/   - substitute action
    long.{-}value   - the pattern to look for
    //    - end of pattern 
    g    - global