Using Regular Expression and log4net input formats with Log Parser Lizard

NOTE: This article is obsolete. Described input formats are replaced by RegEx Enhanced and GROK Input Parser

Both these formats are based on regular expressions to parse the log lines but the difference is that RegEx input format is parsing the log files line by line (if there is some line that doesn’t match the regular expression, it will be marked as error). Log4Net input format also uses regular expressions to parse the log file but the logged messages is spread in more than one line. Field “Full Message” is what goes after the first line that matches the regular expression and “Exception” field isn’t null only if the “Full Message” begins with “Exception:” word (this was made for my own needs but maybe someone else will find it useful). Regex and Log4Net input formats are not the ultimate solution for every text-log-file-format but they are flexible enough to meet most of your needs.

 

Here is a step by step guide for using RegExp and log4net input formats:

  1. In Log Parser Lizard install path create XML file which defines regular expression and its fields and their data types. Something like this:

    <?xml version="1.0" encoding="UTF-8"?>
    <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="C:\src\LogParserCSWebServiceInputFormat\LogParserRegexInputFormat.xsd">
     <regex>^(?&lt;DateTime&gt;(?:\d{4})-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})\s+(?&lt;ThreadID&gt;\d*)\s*\[(?&lt;ProcessID&gt;\d+)\]\s+(?&lt;LogType&gt;\w+)\s+(?&lt;Loger&gt;\w+)\s+-\s+(?&lt;Message&gt;.*)$</regex>
     <fields>
                <field name="DateTime" type="Timestamp" format="yyyy-MM-dd HH:mm:ss,fff"/>
                <field name="ThreadID" type="Integer"/>
                <field name="ProcessID" type="Integer"/>
                <field name="LogType" type="String"/>
                <field name="Loger" type="String"/>
                <field name="Message" type="String"/>
     </fields>
    </config>


    For writing regular expression you can use Expresso, excellent and free tool for building regular expressions and for editing XML files you can use XML Notepad from Microsoft.
  2. Create a new query
  3. From a drop down list select “Regular expression input format” or “log4net input format”
  4. Click on properties button (next to the drop down list) and set “config file” property to configuration file name that you have created in step 1.
  5. Test created the query against your text file (ex. select * from c:\mylog.txt) and if you have some troubles try to fix the config file.

You can look at log4net examples provided in installation directory of LogParser Lizard. And remember, the difference between RegEx input format and log4net input format is that RegEx log files are read as one record per text line. In log4net log files, one record can be in more text lines (for instance when exception is logged).

Feedback and Knowledge Base