StrataFrame Forum

How to properly dispose a BO?

http://forum.strataframe.net/Topic24782.aspx

By Edhy Rijo - 9/30/2009

I have a process that will load a couple of million records in BO to do a duplicate record search. My process will import thousands (360K+) of records and then I need to make sure those records does not exist in the database, so instead of querying the data 360k+, I load a BO with all possible duplicate records to check against and this is the million records BO.



Now the problem is that even when I do a BO.Clear to remove those records the application keeps using a lot of memory (over 1gb) and that memory is not released until the whole application is closed, so when the import process starts again, it will blow up with a not enough memory to complete the process.



So what should I do in order to make sure that the memory used once the million record BO is filled is released when using BO.Clear?



Also I am open to suggestions to change my approach if I need to.



Thanks!
By Greg McGuffey - 9/30/2009



BOs implement IDisposable, so you need to call Dispose method on it in order to release the memory. Dispose on the BO will in turn call Dispose on the DataTable, which is likely what is using the memory. At minimum you need to use a try/catch/finally block and call Dispose in the finally block. However, the best practice is to use the Using statement:



Using bo As New MyBigBO()

  '-- Load with millions of records.

  '....

  '-- Load another BO

  Using notherBo As New MyOtherBO()

    '-- Both BOs available and in scope...

  End Using

End Using




This will ensure that the BO is disposed of, and available for GC, even if an exception occurs or the app is just shut down.
By Edhy Rijo - 9/30/2009

Hi Greg,



Thanks for jump in here.



This is a bit more complicate because the use of the BO is not in a single method, the BO instance is being created as follow:

Private WithEvents _bizDuplicateCardRecords As New bizTransactionItemsImport




Then, there is a process running in a BackGroundWorker that will call a method, that will call another method and in those methods the _bizDuplicateCardRecords BO is being used.



In the BackgroundWorker.DoWork() I have code like this:



If _bizDuplicateCardRecords IsNot Nothing Then

'-- Make sure the BO is disposed to avoid memory leak from previous use.

_bizDuplicateCardRecords.Dispose()

End If

_bizDuplicateCardRecords = New bizTransactionItemsImport

_bizDuplicateCardRecords.FillPreviousRecordLookup(Me.BizTransaction1.PK_Transaction, Me.BizTransaction1.FK_Vendor_Carrier)







Then in the other methods I use the _bizDuplicateCardRecords, but once a transaction has completed, a new instance is created and filled in the DoWork() event.



But still with all this, if I process 2 transactions the memory just keeps going up. Any other suggestion?
By Greg McGuffey - 9/30/2009

I kind of wondered when you asked the question...seemed too easy! Wink



The next thing I'd check into would be events. I.e. I'm wondering if there is a reference to the BO that isn't getting cleaned up related to events. I thought WithEvents was supposed to handle this kind of automagically for you, but...?? The other thing that throws a wrinkle in is threads. If an object is created in one thread, then disposed of in another, is there anything weird about that?



As you can tell my usefulness related to this is rapidly coming to a close...time for the real programmers* to step in... Ermm











* As an aside, check out what a real programmer is...http://www.pbm.com/~lindahl/real.programmers.html
By Edhy Rijo - 9/30/2009

Hi Greg,



Yeap, I believe the BackgroundWorker may have something to do in here, because the BO is being created and filled in the BackgroundWorker.DoWork() event, which I believe is already in a separate thread as the one when the BackgroundWorker.Completed() is fired.



I am kind of stock now, today's testing has been pretty bad due to this situation which is blowing out with out of memory error.



Hope Trent or anybody else can step in soon here, I am kind of desperate w00t
By Peter Jones - 9/30/2009

Hi Edhy,



I think my approach would be to do this in the database, e.g. import the 360k records into a temp table (I presume they are in an external flat file) and do all the heavy lifting in SQL Server.



Cheers, Peter
By Edhy Rijo - 9/30/2009

Hi Peter,



Thanks for jump in.



The problem is not importing the records, that is working fine.



The problem is with the lookup BO data which can have 2-4 millions records and when I am done with this data I need to release the memory used by that BO so the process can be done again.



At this point I see the memory used in the task manager and it just keeps going up until it blow out with a not memory error or something like that.
By Keith Chisarik - 9/30/2009

Should be pretty easy to find with a memory profiler, I use ANTS and they have a free trial.
By Edhy Rijo - 9/30/2009

Hi Keith,



Thanks for the reference, I downloaded the trial and so far discover that my forms are not being removed from the MDI control in the main/parent form and even when I close any child form, there will be an instance there.



I am reviewing the code to see how to get rid of the form instance that should not be there.
By Edhy Rijo - 9/30/2009

Hi Trent,



In this application I am following the same code as in StrataFlix to launch forms and create the _MDIClient objct.



After using ANTS Memory Profiler I noticed that if I open any form, and then close it, a reference of this form will still be hanging. See MovieMaintenance.png picture.



The same is happening with my forms, except that my form is using a large memory chunk and being left hanging it is causing the "out of memory" exception. See MyFormProfiler.png. In some cases the memory used by this form is over 1GB and when I close it and open again, the memory used will just keep increasing.



I have tried several things to get rid of the hanging form without any luck. Could you please take a look at this in the StrataFlix sample and see if you can come up with a way to really close and dispose the MDI child forms?



Thanks!
By Trent L. Taylor - 10/2/2009

To All:



We have made some relatively large changes within StrataFrame recently that have not yet been released which will resolve this issue along with other potential memory leaks. Before releasing this build, we are running and testing it to make sure that there are no residual side-effects. It has been running smoothly and effectively in the field for nearly 2 weeks and will be released to the SF community at large next week.



Note: Though many issues will be resolved without any code changes at all, clean development and the release of any handlers or references is required to clean up memory and prevent memory leaks. StrataFrame has been changed to try and clean up things that may not be handled from the developers side to reduce the risk of this happening. However, it is still the developers responsibility to clean up within their code (i.e. if a handler is added make sure it is removed, etc.).
By Edhy Rijo - 10/2/2009

Hi Trent,



Those are great news. Thanks!



Beside those changes, I don't think they cover the situation explained here about using MDI forms with the same logic used in StrataFlix. Right now my project is being beta tested and this particular form dealing with a couple of million of records got stock with out of memory error, normally closing the form should fix the problem, but I noticed that a form reference is still hanging when I close the form and the same thing happens on the StrataFlix project.



I have tried many things to fix the problem, but I have not been able to fix it. Could you please take a look at the StrataFlix, using ANT or any other memory profiler and notice that the form reference is left hanging?



At this point I am desperate and open to any suggestion.



Thanks!
By Trent L. Taylor - 10/2/2009

Edhy,

The offending line of code within the StrataFlix sample is this:

    ''' <summary>
    ''' Launches a dialog and adds it to the MDI
    ''' </summary>
    ''' <param name="dialogType"></param>
    ''' <remarks></remarks>
    Private Sub LaunchDialog(ByVal dialogType As System.Type)
        '-- Establish Locals
        Dim f As Form = CType(Activator.CreateInstance(dialogType), Form)

        '-- Add the handlers to the form
        AddFormHandlers(f)

        '-- Set the parent MDI
        f.MdiParent = Me

        '-- Show the dialog
        f.Show()
    End Sub

If you comment out the code that associates the form with the MDI, the form will release.  I tested this a number of different ways including removing any handler, threading close, etc.  If it was attached to an MDI, it stuck around.  Even on a sample dialog outside of StrataFrame, if you do this (and load the dialog as we are here) it is sticking around.  I will play with this some more to see if there is a way to get rid of it.  In ANTS, it indicates that the propertStore is holding onto the disposed reference....which indicates that this is lower level.  I am going to look at .NET through Reflector and see if I can tell what it is doing.

By Edhy Rijo - 10/2/2009

Thanks a lot Trent, I really appreciate the time you are taking on this.
By Trent L. Taylor - 10/2/2009

For a further test, I 100% removed StrataFrame from the formula.  Attached here is a plain 100% very simple .NET app that uses a parent/child.  If you create the child and close it, it doesn't get released.  There are no handlers, code, objects, etc.  This is as basic as you can get and it won't release.  I even used the plain IsMdiContainer property versus adding an MDIClient.  This looks like a .NET issue.

Now I am curious and will continue to mess with this...but it does prove StrataFrame has nothing to do with it. Crazy

By Trent L. Taylor - 10/2/2009

One more thing worth noting.  A reference does stick around, but only one.  If you open the dialog 10 times only a single live instance is sticking around [scratching head].  Since there are not any shared or static references, this is a bit strange.
By Trent L. Taylor - 10/2/2009

It does look to be a .NET issue.  Once I started looking this up on Google it appears that this is not uncommon.  Looking for a resolution, but I would be curious to see if you get the same results on ANTS as I did on the project I posted.
By Edhy Rijo - 10/6/2009

Hi Trent,



Yes I got the same results in ANTS. During the past weekend I have been looking like crazy in all google findings about this issue and have not been able to find a solution.



I know this is not related to SF, but have you have a time to look at this deeper?



Thanks again!
By Edhy Rijo - 10/10/2009

Here is some information I found on the web about how to overcome the "System.OutOfMemoryException".



Apparently, .Net runtime does not let you go up to 2GB by default, and not matter how much memory you have it will give you the Out Of Memory exception if your application passes 1.1GB of memory.



That is exactly my case, I am loading some lookup data 3-5 millions records for a process that needs to check on those records. I am importing around 350 thousand records from a CSV file and each record needs to be checked against those 3-5 millions records.



First I had a store procedure to do the checking, but that was painfully slow since it had to go to the database thousands of time to do the check. I could not do the check in groups because a decision needs to be done at the time of checking. So loading the lookup data triggered the Out Of memory exception and it has been a nightmare since the last 2 weeks.



There is a program named Editbin.exe which can be use to flag the VB.NET assembly to use memory after the 2GB with the /LARGEADDRESSAWARE parameter.



I added this code to my project's Post-Build:



call "$(DevEnvDir)..\..\VC\vcvarsall.bat" x86

Editbin.exe /LARGEADDRESSAWARE “$(TargetPath)”




So far, with this change I am able to run the import process with less out of memory exceptions. I am still getting the error, but not so frequenly now. Monitoring the memory with task manager, I seen it past the 2GB, then drops to 800k and up again, and without leaving the form I have done at least 3 import test and have not seen the out of memory error.



My computer have 8GB Ram, so now will test on one of the beta tester computer with less ram and see what happen.



I don't like this temporary solution since it does requires the main .exe file to be hacked with EditBin.exe but until I can test the upcoming SF changes that address possible memory leaks in the BOs I have no other choice so far.



I am compiling in .Net 2.0 and have not tested yet the 3.5 runtimes to see if there is a difference.



Here is copy of the out of memory exception:





OutOfMemoryException

Exception of type 'System.OutOfMemoryException' was thrown.



Source : mscorlib



Stack Trace:

at System.Collections.Generic.Dictionary`2.Initialize(Int32 capacity)

at System.Collections.Generic.Dictionary`2..ctor(Int32 capacity, IEqualityComparer`1 comparer)

at System.Data.DataView.ResetRowViewCache()

at System.Data.DataView.UpdateIndex(Boolean force, Boolean fireEvent)

at System.Data.DataView.UpdateIndex(Boolean force)

at System.Data.DataView.SetIndex2(String newSort, DataViewRowState newRowStates, IFilter newRowFilter, Boolean fireEvent)

at System.Data.DataTable.get_DefaultView()

at MicroFour.StrataFrame.Business.BusinessLayer.ChangeCurrentDataTable(DataTable NewTable, Boolean AcceptChanges, Boolean IsSharedTable)

at MicroFour.StrataFrame.Business.BusinessLayer.FillDataTable(DbCommand CommandToExecute)

at CardTrackingSystem.Business.bizTransactionItemsImport.FillPreviousRecordLookup(Int32 pFK_Transaction, Int32 pFK_Vendor)

at CardTrackingSystem.frmFirstUsageImport.PreAnalyzeRecord_Step1()

at CardTrackingSystem.frmFirstUsageImport.cmdAnalyzeImportedRecords_Click(Object sender, EventArgs e)

at System.Windows.Forms.ToolStripItem.RaiseEvent(Object key, EventArgs e)

at System.Windows.Forms.ToolStripMenuItem.OnClick(EventArgs e)

at System.Windows.Forms.ToolStripItem.HandleClick(EventArgs e)

at System.Windows.Forms.ToolStripItem.HandleMouseUp(MouseEventArgs e)

at System.Windows.Forms.ToolStripItem.FireEventInteractive(EventArgs e, ToolStripItemEventType met)

at System.Windows.Forms.ToolStripItem.FireEvent(EventArgs e, ToolStripItemEventType met)

at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEventArgs mea)

at System.Windows.Forms.ToolStripDropDown.OnMouseUp(MouseEventArgs mea)

at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)

at System.Windows.Forms.Control.WndProc(Message& m)

at System.Windows.Forms.ScrollableControl.WndProc(Message& m)

at System.Windows.Forms.ToolStrip.WndProc(Message& m)

at System.Windows.Forms.ToolStripDropDown.WndProc(Message& m)

at System.Windows.Forms.Control.ControlNativewindow.OnMessage(Message& m)

at System.Windows.Forms.Control.ControlNativewindow.WndProc(Message& m)

at System.Windows.Forms.Nativewindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
By Edhy Rijo - 10/13/2009

Hi Trent,



Have you have a chance to take a look at this issue with the MDIClient not disposing the MDI child forms?
By Trent L. Taylor - 10/13/2009

A little. As we had discussed, this is something that we reproduced outside of SF. I have played with this for a while and .NET is holding on to the MDI reference. As of yet, I have not found a solution. But I will look at this some more at the first opportunity. I would be curious to see what Microsoft has to say on the issue. I think that we need to send the sample that we have to them and ask them why this continues to hold on to the reference.



But I will still look at this again.
By Edhy Rijo - 10/13/2009

Thanks!
By Edhy Rijo - 1/31/2010

Hi Trent,



Will the new SF release 1.7.0.6 have any effect in this issue with the MDI forms?



My trial version of ANTS expired and I cannot test your sample application with new SF assemblies.
By Trent L. Taylor - 1/31/2010

Most likely the two are unrelated since the code you are referring to is not SF specific but rather .NET specific.  But I will run a test to see.