Using Regular Expressions (RegEx) and GROK
LPL RegEx built in support
Log Parser Lizard has built in Input Formats to parser text based log files using regular expressions and GROK also (also supported by popular Logstash platform so there are plenty of samples online). The built in formats are:
- "GROK/RegEx Text Parser" - for easy to write simple text files. Most users will find this enough.
- "Regular Expression (RegEx) Enhanced" - advanced features to parser records.
File Stream support
These plugins can also read files from several file-stream sources: by default from the Windows file system, but also from HTTP(s), FTP(s), and standard output (stdout) of console applications if you prefix the file with one of these prefixes: FTP://, FTPS://, HTTP://, HTTPS://, STDOUT://
FTP and HTTP streams support basic authentication ex. SELECT * FROM http://user:pwd@sample.com/logs/file.log
Read compressed logs
All built-in LPL plugins can read .gz (gzip) compressed logs. To compress gzip files and save storage space, you can use popular tools like gzip or 7zip. After compression files can also be encrypted (check the LPL Tools folder and find a helper .BAT script for compression and/or encryption of log files that you can modify).
Samples
To see how GROK format can be used in LPL, open a new "Read Lpl Log" query (from favorite input formats - those with the star), and see the Input Properties. There are also some samples in sample Queries in the installation package.
External resources
Read more about Logstash GROK syntax on these links:
Regular Expression Builder
Log Parser Lizard has a built-in feature to help you build your regular expressions. Enter a sample record from your log and write and test the pattern. Use RegEx groups to set log fields. Also when you use GROK, you can set the group name after and field type (see comments and example below). There are also many "favorite patterns", GROK aliases, and other options when you click on the Show Builder button (read the options and explore). You can change these or add your own in a text editor. Here is a screenshot of RegEx Builder:
Parsing tricky log? Send us a few lines of raw data at support@lizard-labs.com and we will reply with a Log Parser Lizard regular expression in a few working days.