Why am I getting an out of memory exception when I have more memory than is being used?
Why am I getting an out of memory exception when I have more memory than is being used? Can you please help? If your product cannot produce 700k record files, I have no reason to purchase the full version for use at work.
A: There is no limit on the size of the file(s) that can be processed with log parser. You may process any number of very large files. But there is a limit in the amount of records that can be displayed in LPL grid for browsing since LPL stores the output records in system memory. Unfortunately LPL runs in 32 bit mode even on 64 bit OS since it uses 32 bit log parsing engine from Microsoft. When you run the query lpl stores the results in memory in order to display them in a data grid. Exporting to Excel won't help you here because it also stores the results in memory before writing to file.
You must find the way to write your queries to produce smaller output (by using WHERE for instance) since the size of your log files is very big.
For instance use the TOP keyword to limit the output (this is my common practice to take a quick look at what is going on in the log files) :
Select top 10000 * from... Where rownumber > 0
Then change the where parameter when needed.
Or, for instance, if you want to get all usernames from the log file you can use something like this:
Select distinct username from...
This should narrow down the output.
If you want to see how fast the engine will process the files, write the query like this:
select count(*) from c:\logs\*.log
This query will process all files, but it has only one output record so LPL wont crush due to insufficient memory.
If you use LP to output in a file instead in the grid, you also wont have any troubles (since the output is not stored in memory). For instance:
select * INTO 'c:\out.csv' from c:\www\logs\*.log
I
hope this was helpful. I also hope that you will like LPL, find it
useful, share your experience with others and eventually buy the full version with all features :)
User Response:
I have not purchased your software just yet! I wasn’t sure if it’d meet my needs. However, your prompt and in depth explanation shows that you are dedicated to supporting your software and helping those who use it, even with the free users.
Therefore, I plan on purchasing this software quite soon, simply to support such a great company and on-the-fly data parsing.
I have used your recommendations and the software parses things MUCH MUCH faster. Part of my difficulty was not understanding how the tool was meant to be used (operator error).