Bulk Saving?


Author
Message
Edhy Rijo
E
StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)
Group: StrataFrame Users
Posts: 2.4K, Visits: 23K
Hi guys,



I have an import process of CSV files with more than 300 thousand records, I have to read each one and compare against my records and then insert them in another table for future reference with other imports.



My approach is to loop through the imported records collection and then add them to the BO using BO.NewRow(), so far so good, the problem comes when I need to save them using BO.Save(), it takes a long time to actually insert those 300+K records.



I am using the DDT for all my data modeling, and this table is setup to use Store Procedures for all Insert/Update/Delete and also is using an AutoIncrement PK.



In the BO I have turn off the Update/Delete ConcurrencyType and there is no value in the RowVersionOrTimestampColumn.



Is there anything else I may need to do to save those records faster?




Edhy Rijo

Keith Chisarik
Keith Chisarik
StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)
Group: StrataFrame Users
Posts: 939, Visits: 40K
SQLBULKCOPY

Somthing like this: (pass in the currentdatatable fo your BO)

Private Sub bulkLoadFromMultipleTables(ByVal dtToInsert As DataTable)

Dim x As DateTime = DateTime.Now

Using destinationConnection As SqlConnection = _

New SqlConnection(MicroFour.StrataFrame.Data.DataLayer.DataSources("sql").ConnectionString)

destinationConnection.Open()

Using bulkCopy As SqlBulkCopy = New SqlBulkCopy(destinationConnection)

bulkCopy.DestinationTableName = _sqlTable.Trim

Try

'bulkCopy.NotifyAfter = 10000

'bulkCopy.SqlRowsCopied += New SqlRowsCopiedEventHandler(s_SqlRowsCopied)

bulkCopy.BatchSize = 1000

'should already match due to the join process and field aliasing done

For Each row_conversionrule As BUSINESS.bo_mapfields In Me.Bo_mapfields.GetEnumerable()

bulkCopy.ColumnMappings.Add(row_conversionrule.mapf_sqlfield, row_conversionrule.mapf_sqlfield)

Next

bulkCopy.WriteToServer(dtToInsert)

bulkCopy.Close()

Dim y As DateTime = DateTime.Now

Dim diff As TimeSpan = y - x

MsgBox("seconds: " & diff.Seconds & ":" & diff.Milliseconds)

Catch ex As Exception

MsgBox(ex.ToString)

Finally

' Close the SqlDataReader. The SqlBulkCopy

' object is automatically closed at the end

' of the Using block.

dtToInsert.Dispose()

End Try

End Using

End Using

End Sub

' of the Using block.

 



Keith Chisarik
Keith Chisarik
Keith Chisarik
StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)
Group: StrataFrame Users
Posts: 939, Visits: 40K
if your SQL table matches the structure of your BO, you can skip the mapping line, ohterwise us it to map your datatable fields to your SQL table fields before the copy.

Optional: bulkCopy.ColumnMappings()

Keith Chisarik

Edhy Rijo
E
StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)
Group: StrataFrame Users
Posts: 2.4K, Visits: 23K
Hi Keith,



Thanks a lot for the code sharing. I was just reading an article about SqlBulkCopy here and then saw your post.



Yes, my BO match the SQL data structure. I will try to implement this one.

Edhy Rijo

Keith Chisarik
Keith Chisarik
StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)
Group: StrataFrame Users
Posts: 939, Visits: 40K
This saved my butt way back when I was using DB2 (uhggg) , I use it a decent amount with SQL and it has treated me good Smile

Enjoy!

Keith Chisarik

Edhy Rijo
E
StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)
Group: StrataFrame Users
Posts: 2.4K, Visits: 23K
Hi Keith,



I am very close to make this work. I only having some problem with the bulkCopy.ColumnMappings.Add()



For Each row_conversionrule As BUSINESS.bo_mapfields In Me.Bo_mapfields.GetEnumerable()



bulkCopy.ColumnMappings.Add(row_conversionrule.mapf_sqlfield, row_conversionrule.mapf_sqlfield)



Next




This code is not very clear to me, I guess I need to get all the field names from the BO to be mapped in the bulkCopy. Would you please post the For...Each code for a real BO sample?

Edhy Rijo

Edhy Rijo
E
StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)
Group: StrataFrame Users
Posts: 2.4K, Visits: 23K
Never mind, I found it:

For Each fieldName As String In Me.BizTransactionItemsImport1.AllFieldsList()

bulkCopy.ColumnMappings.Add(fieldName, fieldName)

Next





Thanks Keith, it works the first run, I will now do a test with couple of thousand of records.

Edhy Rijo

Keith Chisarik
Keith Chisarik
StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)
Group: StrataFrame Users
Posts: 939, Visits: 40K
ahh yes I was pulling from a fields mapping table because my BO structure does not match my SQL tables structure.

Looks like you got it. FYI, if they ,match you dont need to do the column mapping step and it will 'just work' Smile

Keith Chisarik

Edhy Rijo
E
StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)StrataFrame VIP (4.7K reputation)
Group: StrataFrame Users
Posts: 2.4K, Visits: 23K
Keith Chisarik (09/01/2009)
ahh yes I was pulling from a fields mapping table because my BO structure does not match my SQL tables structure.



Looks like you got it. FYI, if they ,match you dont need to do the column mapping step and it will 'just work' Smile




Yes they match, but it did not work without the mapping, so I figured to be in the safe side to just do the Field Name loop and it will not cost anything, I am still gaining a lot of time now, I will do the test in a couple of minutes with 365K records which took almost 10 minutes to save before, will post the results later.

Edhy Rijo

Keith Chisarik
Keith Chisarik
StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)StrataFrame VIP (1.7K reputation)
Group: StrataFrame Users
Posts: 939, Visits: 40K
My bet is ... 6 seconds.

Keith Chisarik
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Reading This Topic

Login

Explore
Messages
Mentions
Search