Using Regular Expressions (RegEx) and GROK
LPL RegEx built in support
Log Parser Lizard has built in Input Formats to parser text based log files using regular expressions and GROK also (also supported by popular Logstash platform so there are plenty of samples online). The built in formats are:
- "GROK/RegEx Text Parser" - for easy to write simple text files. Most users will find this enough.
- "Regular Expression (RegEx) Enhanced" - advanced features to parser records.
File Stream support
These plugins can also read files from several file-stream sources: by default from Windows file system, but also from http(s), ftp(s) and standard output (stdout) of console applications if you prefix the file with one of these prefixe: ftp://, ftps://, http://, https://, stdout://
FTP and HTTP streams supprt basic authentication ex. SELECT * FROM http://user:pwd@sample.com/logs/file.log
Read compressed logs
All built in LPL plugins can read .gz (gzip) compressed logs. To compress gzip files and save storage space, you can use popular tools like gzip or 7zip. After compression files can also be encrypted (check LPL Tools folder and find a helper .BAT script for compression and/or encryption of log files that you can modify).
Samples
To see how GROK format can be used in LPL, open a new "Read Lpl Log" query (from favorite input formats - those with the star), and see the Input Properties. There are also some samples in sample Queries in installation package.
External resources
Read more about Logstash GROK syntax on these links:
Regular Expression Builder
Log Parser Lizard has built in feature to help you building your regular expressions. Enter a sample record from your log and write and test the pattern. Use RegEx groups to set log fields. Also when you use GROK, you can set group name after and field type (see comments and example below). There are also many "favorite patterns", GROK aliases and other options when you click on Show Builder button (read the options and explore). You can change these or add your own in a text editor. Here is a screenshot of RegEx Builder:
Parsing tricky log? Send us a few lines of raw data on support@lizard-labs.com and we will reply with a Log Parser Lizard regular expression in a few working days.