I have a process that will load a couple of million records in BO to do a duplicate record search. My process will import thousands (360K+) of records and then I need to make sure those records does not exist in the database, so instead of querying the data 360k+, I load a BO with all possible duplicate records to check against and this is the million records BO.



Now the problem is that even when I do a BO.Clear to remove those records the application keeps using a lot of memory (over 1gb) and that memory is not released until the whole application is closed, so when the import process starts again, it will blow up with a not enough memory to complete the process.



So what should I do in order to make sure that the memory used once the million record BO is filled is released when using BO.Clear?



Also I am open to suggestions to change my approach if I need to.



Thanks!

Edhy Rijo

BOs implement IDisposable, so you need to call Dispose method on it in order to release the memory. Dispose on the BO will in turn call Dispose on the DataTable, which is likely what is using the memory. At minimum you need to use a try/catch/finally block and call Dispose in the finally block. However, the best practice is to use the Using statement:







This will ensure that the BO is disposed of, and available for GC, even if an exception occurs or the app is just shut down.

Hi Greg,



Thanks for jump in here.



This is a bit more complicate because the use of the BO is not in a single method, the BO instance is being created as follow:





Then, there is a process running in a BackGroundWorker that will call a method, that will call another method and in those methods the _bizDuplicateCardRecords BO is being used.



In the BackgroundWorker.DoWork() I have code like this:





Then in the other methods I use the _bizDuplicateCardRecords, but once a transaction has completed, a new instance is created and filled in the DoWork() event.



But still with all this, if I process 2 transactions the memory just keeps going up. Any other suggestion?

Edhy Rijo

I kind of wondered when you asked the question...seemed too easy!



The next thing I'd check into would be events. I.e. I'm wondering if there is a reference to the BO that isn't getting cleaned up related to events. I thought WithEvents was supposed to handle this kind of automagically for you, but...?? The other thing that throws a wrinkle in is threads. If an object is created in one thread, then disposed of in another, is there anything weird about that?



As you can tell my usefulness related to this is rapidly coming to a close...time for the real programmers* to step in...











* As an aside, check out what a real programmer is...http://www.pbm.com/~lindahl/real.programmers.html

Hi Greg,



Yeap, I believe the BackgroundWorker may have something to do in here, because the BO is being created and filled in the BackgroundWorker.DoWork() event, which I believe is already in a separate thread as the one when the BackgroundWorker.Completed() is fired.



I am kind of stock now, today's testing has been pretty bad due to this situation which is blowing out with out of memory error.



Hope Trent or anybody else can step in soon here, I am kind of desperate

Edhy Rijo

Hi Edhy,



I think my approach would be to do this in the database, e.g. import the 360k records into a temp table (I presume they are in an external flat file) and do all the heavy lifting in SQL Server.



Cheers, Peter

Hi Peter,



Thanks for jump in.



The problem is not importing the records, that is working fine.



The problem is with the lookup BO data which can have 2-4 millions records and when I am done with this data I need to release the memory used by that BO so the process can be done again.



At this point I see the memory used in the task manager and it just keeps going up until it blow out with a not memory error or something like that.

Edhy Rijo

Should be pretty easy to find with a memory profiler, I use ANTS and they have a free trial.

Keith Chisarik

Hi Keith,



Thanks for the reference, I downloaded the trial and so far discover that my forms are not being removed from the MDI control in the main/parent form and even when I close any child form, there will be an instance there.



I am reviewing the code to see how to get rid of the form instance that should not be there.

Edhy Rijo

Hi Trent,



In this application I am following the same code as in StrataFlix to launch forms and create the _MDIClient objct.



After using ANTS Memory Profiler I noticed that if I open any form, and then close it, a reference of this form will still be hanging. See MovieMaintenance.png picture.



The same is happening with my forms, except that my form is using a large memory chunk and being left hanging it is causing the "out of memory" exception. See MyFormProfiler.png. In some cases the memory used by this form is over 1GB and when I close it and open again, the memory used will just keep increasing.



I have tried several things to get rid of the hanging form without any luck. Could you please take a look at this in the StrataFlix sample and see if you can come up with a way to really close and dispose the MDI child forms?



Thanks!

Edhy Rijo