Using Regular Expressions (RegEx) and GROK

A regular expression is a pattern that the regular expression engine attempts to match in input text. A pattern consists of one or more character literals, operators, or constructs. For a brief introduction, see .NET Regular Expressions.

LPL RegEx built in support

Log Parser Lizard has built in Input Formats to parser text based log files using regular expressions and GROK also (also supported by popular Logstash platform so there are plenty of samples online). The built in formats are:

- "GROK/RegEx Text Parser" - for easy to write simple text files. Most users will find this enough.
"Regular Expression (RegEx) Enhanced" - advanced features to parser records.



File Stream support

These plugins can also read files from several file-stream sources: by default from Windows file system, but also from http(s), ftp(s) and standard output (stdout) of console applications if you prefix the file with one of these prefixe: ftp://, ftps://, http://, https://, stdout:// 

FTP and HTTP streams supprt basic authentication ex. SELECT * FROM  http://user:pwd@sample.com/logs/file.log

Read compressed logs

All built in LPL plugins can read .gz (gzip) compressed logs. To compress gzip files and save storage space, you can use popular tools like gzip or 7zip.  After compression files can also be encrypted (check LPL Tools folder and find a helper .BAT script for compression and/or encryption of log files that you can modify).

Samples

To see how GROK format can be used in LPL, open a new "Read Lpl Log" query (from favorite input formats - those with the star), and see the Input Properties. There are also some samples in sample Queries in installation package.

External resources 

Read more about Logstash GROK syntax on these links:





Regular Expression Builder

Log Parser Lizard has built in feature to help you building your regular expressions. Enter a sample record from your log and write and test the pattern. Use RegEx groups to set log fields. Also when you use GROK, you can set group name after and field type (see comments and example below). There are also many "favorite patterns", GROK aliases and other options when you click on Show Builder button (read the options and explore). You can change these or add your own in a text editor. Here is a screenshot of RegEx Builder:




Feedback and Knowledge Base