I have a process that will load a couple of million records in BO to do a duplicate record search. My process will import thousands (360K+) of records and then I need to make sure those records does not exist in the database, so instead of querying the data 360k+, I load a BO with all possible duplicate records to check against and this is the million records BO.



Now the problem is that even when I do a BO.Clear to remove those records the application keeps using a lot of memory (over 1gb) and that memory is not released until the whole application is closed, so when the import process starts again, it will blow up with a not enough memory to complete the process.



So what should I do in order to make sure that the memory used once the million record BO is filled is released when using BO.Clear?



Also I am open to suggestions to change my approach if I need to.



Thanks!

Edhy Rijo

Hi Greg,



Thanks for jump in here.



This is a bit more complicate because the use of the BO is not in a single method, the BO instance is being created as follow:





Then, there is a process running in a BackGroundWorker that will call a method, that will call another method and in those methods the _bizDuplicateCardRecords BO is being used.



In the BackgroundWorker.DoWork() I have code like this:





Then in the other methods I use the _bizDuplicateCardRecords, but once a transaction has completed, a new instance is created and filled in the DoWork() event.



But still with all this, if I process 2 transactions the memory just keeps going up. Any other suggestion?

Edhy Rijo

I kind of wondered when you asked the question...seemed too easy!



The next thing I'd check into would be events. I.e. I'm wondering if there is a reference to the BO that isn't getting cleaned up related to events. I thought WithEvents was supposed to handle this kind of automagically for you, but...?? The other thing that throws a wrinkle in is threads. If an object is created in one thread, then disposed of in another, is there anything weird about that?



As you can tell my usefulness related to this is rapidly coming to a close...time for the real programmers* to step in...











* As an aside, check out what a real programmer is...http://www.pbm.com/~lindahl/real.programmers.html

Hi Greg,



Yeap, I believe the BackgroundWorker may have something to do in here, because the BO is being created and filled in the BackgroundWorker.DoWork() event, which I believe is already in a separate thread as the one when the BackgroundWorker.Completed() is fired.



I am kind of stock now, today's testing has been pretty bad due to this situation which is blowing out with out of memory error.



Hope Trent or anybody else can step in soon here, I am kind of desperate

Edhy Rijo