Why am I getting an out of memory exception when I have more memory than is being used?

Q: I just downloaded your program, and after running a query in excess of 700,000 records, I am now getting an out of memory exception, even after the parsing was complete. I cannot save this as an excel spreadsheet file, even though I have a total of 12 GB's of RAM, and roughly 6 GB's of that are not in use by your program or my system.

Why am I getting an out of memory exception when I have more memory than is being used? Can you please help? If your product cannot produce 700k record files, I have no reason to purchase the full version for use at work.

A: There is no limit on the size of the file(s) that can be processed with log parser. You may process any number of very large files. But there is a limit in the amount of records that can be displayed in LPL grid for browsing since LPL stores the output records in system memory. Unfortunately LPL runs in 32 bit mode even on 64 bit OS since it uses 32 bit log parsing engine from Microsoft. When you run the query lpl stores the results in memory in order to display them in a data grid. Exporting to Excel won't help you here because it also stores the results in memory before writing to file.

You must find the way to write your queries to produce smaller output (by using WHERE for instance) since the size of your log files is very big.

For instance use the TOP keyword to limit the output (this is my common practice to take a quick look at what is going on in the log files) :

Select top 10000 * from... Where rownumber > 0

Then change the where parameter when needed.

Or, for instance, if you want to get all usernames from the log file you can use something like this:

Select distinct username from...

This should narrow down the output.

If you want to see how fast the engine will process the files, write the query like this:

select count(*) from c:\logs\*.log

This query will process all files, but it has only one output record so LPL wont crush due to insufficient memory.

If you use LP to output in a file instead in the grid, you also wont have any troubles (since the output is not stored in memory). For instance:

select * INTO 'c:\out.csv' from c:\www\logs\*.log

I hope this was helpful. I also hope that you will like LPL, find it useful, share your experience with others and eventually buy the full version with all features :)


User Response:

I have not purchased your software just yet! I wasn’t sure if it’d meet my needs. However, your prompt and in depth explanation shows that you are dedicated to supporting your software and helping those who use it, even with the free users.

Therefore, I plan on purchasing this software quite soon, simply to support such a great company and on-the-fly data parsing.

I have used your recommendations and the software parses things MUCH MUCH faster. Part of my difficulty was not understanding how the tool was meant to be used (operator error).



Log Parser Lizard: FAQ

  1. How to use inline Visual Basic .Net code?
  2. Can I try Log Parser Lizard before I buy it?
  3. I own a previous version of Log Parser Lizard. Do you offer upgrade pricing from a previous version?
  4. How can I modify formula field?
  5. How do I download updates and upgrade?
  6. How do I enter my license code in Log Parser Lizard?
  7. How do I specify in the regex that I want to match something on the second line of the output?
  8. How to get user input for queries?
  9. How to view Windows Event Log using Log Parser Lizard
  10. I just wanted to find out what extra features are enabled after registering this product
  11. I purchased a license for Log Parser Lizard, but I can not find where to download it from. Please advise where it can be downloaded from.
  12. I want Log Parser Lizard GUI to be much better and I want to help. How to help?
  13. I was unable to download the Log Parse Lizard. What to do?
  14. I'm getting Retrieving the COM class factory for component with CLSID {............ID.........} failed due to the following error: 80040154 anytime I click on any of the options
  15. Is there a listing of example log queries beyond those listed in the UI?
  16. Who is using Log Parser Lizard?
  17. How to look at the logs of multiple servers?
  18. Can I use Log Parser Lizard to Query MS SQL, Oracle or My SQL database?
  19. My anti virus software is telling me it found security risk in LogParserLizard.msi
  20. Why am I getting an out of memory exception when I have more memory than is being used?
  21. I've ordered today but my license code not working
  22. Please can you send us a reseller quote for this license, and more important, is there a reseller discount available?
  23. Can I use Log Parser Lizard for Google BigQuery BigData services?
  24. I did not receive the confirmation email after requesting free trial. What should I do?

Feedback and Knowledge Base