By Kirk M Sherhart - 7/1/2009
I notice that the private BO.MergeDataTables procedure uses a simple linear list for gathering PK values in order to check for duplicates. In my app, I can easily have the case of merging 10,000 rows into an existing datatable with 50,000 rows (large GIS datasets.) The O(m*n) behavior kills the performance!
If you simply switched over to using a SortedList, at least I'd get O(m*log(n)) behavior, which is OK.
Another (more dangerous) option would allow a NoCheckForDuplicates option on the BO.AppendDataTable and BO.MergeDataFrom routines, leaving duplicate PK checking in my hands.
TIA
|
By Trent L. Taylor - 7/1/2009
To be honest, if you are dealing with this many records in a single record set I wouldn't be using this approach anyway. There are very few times that it is ever necessary to deal with this many records at any one time. In our medical application we have some databases and customers with millions of records in a single table, and to date, I have never had to sacrafice performance for size. I do not know what you are trying to accomplish, but if you can deal with a reader if this is a linear approach or a do the work within a SPROC to filter these records down before they come to the client side, you would be better off.
If this is something that you intend on continuing, then I would recommend moving up a level and creating your own static class within one of your application basics classes to accept two data tables (or views) and merge them versus relying on any checking so that you can optimize what you need. Again, I still recommend against ever having this many records client side regardless of task...it takes time and performance degradation to get this much data client side. Like I said, there are a very few rare times that this may not be avoidable, but if you catch yourself doing this a lot, I would imagine that most of this work should be getting done within a SPROC or CLR SPROC.
|
|