Edhy Rijo
|
|
Group: StrataFrame Users
Posts: 2.4K,
Visits: 23K
|
Hi guys, I have an import process of CSV files with more than 300 thousand records, I have to read each one and compare against my records and then insert them in another table for future reference with other imports. My approach is to loop through the imported records collection and then add them to the BO using BO.NewRow(), so far so good, the problem comes when I need to save them using BO.Save(), it takes a long time to actually insert those 300+K records. I am using the DDT for all my data modeling, and this table is setup to use Store Procedures for all Insert/Update/Delete and also is using an AutoIncrement PK. In the BO I have turn off the Update/Delete ConcurrencyType and there is no value in the RowVersionOrTimestampColumn. Is there anything else I may need to do to save those records faster?
Edhy Rijo
|
|
|
Keith Chisarik
|
|
Group: StrataFrame Users
Posts: 939,
Visits: 40K
|
SQLBULKCOPY Somthing like this: (pass in the currentdatatable fo your BO) Private Sub bulkLoadFromMultipleTables(ByVal dtToInsert As DataTable)Dim x As DateTime = DateTime.NowUsing destinationConnection As SqlConnection = _New SqlConnection(MicroFour.StrataFrame.Data.DataLayer.DataSources("sql").ConnectionString)destinationConnection.Open() Using bulkCopy As SqlBulkCopy = New SqlBulkCopy(destinationConnection)bulkCopy.DestinationTableName = _sqlTable.Trim Try'bulkCopy.NotifyAfter = 10000'bulkCopy.SqlRowsCopied += New SqlRowsCopiedEventHandler(s_SqlRowsCopied)bulkCopy.BatchSize = 1000 'should already match due to the join process and field aliasing doneFor Each row_conversionrule As BUSINESS.bo_mapfields In Me.Bo_mapfields.GetEnumerable()bulkCopy.ColumnMappings.Add(row_conversionrule.mapf_sqlfield, row_conversionrule.mapf_sqlfield) NextbulkCopy.WriteToServer(dtToInsert)bulkCopy.Close() Dim y As DateTime = DateTime.NowDim diff As TimeSpan = y - xMsgBox( "seconds: " & diff.Seconds & ":" & diff.Milliseconds)Catch ex As ExceptionMsgBox(ex.ToString) Finally' Close the SqlDataReader. The SqlBulkCopy' object is automatically closed at the end' of the Using block.dtToInsert.Dispose() End TryEnd UsingEnd UsingEnd Sub' of the Using block.
Keith Chisarik
|
|
|
Keith Chisarik
|
|
Group: StrataFrame Users
Posts: 939,
Visits: 40K
|
if your SQL table matches the structure of your BO, you can skip the mapping line, ohterwise us it to map your datatable fields to your SQL table fields before the copy. Optional: bulkCopy.ColumnMappings()
Keith Chisarik
|
|
|
Edhy Rijo
|
|
Group: StrataFrame Users
Posts: 2.4K,
Visits: 23K
|
Hi Keith, Thanks a lot for the code sharing. I was just reading an article about SqlBulkCopy here and then saw your post. Yes, my BO match the SQL data structure. I will try to implement this one.
Edhy Rijo
|
|
|
Keith Chisarik
|
|
Group: StrataFrame Users
Posts: 939,
Visits: 40K
|
This saved my butt way back when I was using DB2 (uhggg) , I use it a decent amount with SQL and it has treated me good Enjoy!
Keith Chisarik
|
|
|
Edhy Rijo
|
|
Group: StrataFrame Users
Posts: 2.4K,
Visits: 23K
|
Hi Keith, I am very close to make this work. I only having some problem with the bulkCopy.ColumnMappings.Add() For Each row_conversionrule As BUSINESS.bo_mapfields In Me.Bo_mapfields.GetEnumerable()
bulkCopy.ColumnMappings.Add(row_conversionrule.mapf_sqlfield, row_conversionrule.mapf_sqlfield)
Next This code is not very clear to me, I guess I need to get all the field names from the BO to be mapped in the bulkCopy. Would you please post the For...Each code for a real BO sample?
Edhy Rijo
|
|
|
Edhy Rijo
|
|
Group: StrataFrame Users
Posts: 2.4K,
Visits: 23K
|
Never mind, I found it: For Each fieldName As String In Me.BizTransactionItemsImport1.AllFieldsList()
bulkCopy.ColumnMappings.Add(fieldName, fieldName)
Next
Thanks Keith, it works the first run, I will now do a test with couple of thousand of records.
Edhy Rijo
|
|
|
Keith Chisarik
|
|
Group: StrataFrame Users
Posts: 939,
Visits: 40K
|
ahh yes I was pulling from a fields mapping table because my BO structure does not match my SQL tables structure. Looks like you got it. FYI, if they ,match you dont need to do the column mapping step and it will 'just work'
Keith Chisarik
|
|
|
Edhy Rijo
|
|
Group: StrataFrame Users
Posts: 2.4K,
Visits: 23K
|
Keith Chisarik (09/01/2009)
ahh yes I was pulling from a fields mapping table because my BO structure does not match my SQL tables structure. Looks like you got it. FYI, if they ,match you dont need to do the column mapping step and it will 'just work' Yes they match, but it did not work without the mapping, so I figured to be in the safe side to just do the Field Name loop and it will not cost anything, I am still gaining a lot of time now, I will do the test in a couple of minutes with 365K records which took almost 10 minutes to save before, will post the results later.
Edhy Rijo
|
|
|
Keith Chisarik
|
|
Group: StrataFrame Users
Posts: 939,
Visits: 40K
|
My bet is ... 6 seconds.
Keith Chisarik
|
|
|