I have a routine that is importing Foxpro Data. This week  made some changes and added a few larger tables(more records and more fields) to the import routine that has been running for months. These "new" tables are by no means crazy huge as the largest one is right around 75,000 records and 30 or so fields. This routine has been running at night so I never really noticed if this has always been an issue. however while debugging changes noticed that the performance was lacking..  

Once I started looking onto it I was very suprised at the perfomance vs a plain datatable. I did expect a business object to have some additional overhead over a vanilla datatable , however I was expecting a very minimal difference compared to the results I am seeing. Perhaps I am missing something easy like not setting a business object property or I am looking at something completly wrong. Whatever it is there seems to be a huge performance difference that I would like to get to the bottom of it.

To take anything I was doing out of the mix I created a test project that uses the stataframe sample orderitems table as a source and destination bo. I then created a standard plain old vanilla BO mapped to the items table. The only business object property that is changed from default is the "allowsnullvaluesonnewrow" is set to true.

Basically I fill a 'source' business object Then loop through the 'source' business object and create a new row on a destination object and sets the value of a field on the destination object from the source .

Here are my results based on 50,000 records from the strataframe order items table. These times do not include any filling of data or saving of data.

Use a bo to create a new row and set no values                    .54 seconds

use a bo to create a new row and set the value of one field     5.04 seconds

use a bo to create a new row and set the value of two fields   9.51 seconds

use a bo to create a new row and set the value of 3 fields     14.17 seconds

----------------Data Table--------

Use a datatable to create a new row and set no values            .21 seconds

use a dt to create a new row and set the value of one field      .31 seconds

use a dt to create a new row and set the value of two field      .45 seconds

use a dt to create a new row and set the value of 3  fields      .51 seconds

 

I guess I forgot to attach the sample app

Hey guys did this get skipped\missed?

Do you need anything additional from me? Just would like to know you guys are looking at this problem and it is on the radar.

Thank you,

Paul 

I am taking a look at it right now.

Paul,



I will have to look at this some more tomorrow. I am going to run it through a profiler to see where the hard spots are. Thanks.

Thanks Trent.

Paul,



The code issue is actually happening within the DataTable. I think that there is a handler somewhere within the DataTable managing the ListChanged event which is tied into the FieldPropertyDescriptors that is causing the delay. I am going to have to break this down to the angle iron as the root of the issue is actually within the ADO.NET data table, but is manifesting itself through a handler. I have killed all handlers, etc. and not been able to isolate it just yet.



I am going to add this to my list. However, in the interim, if you are dealing with massive records and just want to circumvent all BO handlers, you can extract the datatable, add the rows, and the BO will still be looking at the same BO reference:







Basically this will circumvent all DataTable list changed events. Like I said, I am going to revisit this when I can dedicate 4+ hours as I have already spent 2 with little progress as there is a handler or delegate conflict going on. But the above would at least be a temp solution.

One more thought...the main thing here is that before we make any changes, we are going to have to totally understand the bottleneck and make sure that any change made doesn't break something else as this would be a low level change. Thus the time necessary to diagnose needs to be allocated prior to making any change.

Thanks Trent.

For the most part the import routine that this affect's runs after hours unless there is some type of exception that would require a manual rerun. So rather than refactor the mapping methods to use a datatable I will leave it as is for now as I do have some other stuff going on that I might break. So it is not a life or death issue for me but I did want to find out what was going on, and obviously resolve it.

I understand 100% that this will not be a "quick" fix as the underlying datatable events have to be one of the worms at the bottom of the business object can.

Appreciate ya,

Paul  

Hi Paul,

   What's the old joke? A guy says "Doctor, it hurts when I do this" and smacks his head. Doctor says "Then don't do it..."

   Back in the heyday of FoxPro, I wrote an article very similar to your comment about the listboxes in the "two-column mover" control created by Alan Griver, who is now in charge of languages at MS. I was able to load a 50,000-line listbox must faster than his method did, using arrays. His answer was "why do you need 50,000 items in a listbox?". I had to admit that he was right.

   There's probably a way to speed up listview loading internally from a business object, but maybe a BrowseDialog will give your users a more manageable way of filtering and selecting records than would a ListView. That's what I ended up doing. By the time my users got to the two-column-mover, they were looking at only a few hundred records, having already filtered out what they weren't interested in.

Les