Best way?


Author
Message
Kevin Lingofelter
Kevin Lingofelter
StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)
Group: Forum Members
Posts: 77, Visits: 804
I have a BO that is being used to transform records using a template the user creates. The parsing routine works fine, but filling the BO takes several minutes!

The number of records is quite large (about 100k records). Indexes are in place and the query executes within about 5 seconds inside Access, so the time is getting used filling the BO.

This is a read-only view. The fill routine is just "select * from blah" (blah is a view). Can anyone suggest a way to speed the fill routine up?

Thanks!

Gary C Wynne
Gary C Wynne
StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)
Group: Forum Members
Posts: 26, Visits: 236

Hi Kevin,

>The number of records is quite large (about 100k records). Indexes are in place and the query executes within about 5 seconds inside Access, so the time is getting used filling the BO.

Anything I say about this is just a SWAG because I have never used a BO to handle 100,000 records before! However, what stands out to me is that this is an inordinate amount of records to hold in any BO. Underpinning the BO is a simple ADO.Net DataTable so, I guess the question is, how long does it take for an ADO.Net DataTable to create 100,000 rows?

Inside Visual FoxPro, you could query and build a table with millions of records in a couple of seconds so, the fact that your Access query is quicker to select than filling the SF BO is not really a factor here, IMHO. What may be a good idea is to use the SF "GetDataTable" method and see whether you get any difference in time just loading into another temporary DataTable in the same BO (unlikely).

Also, you might like to create another instance of your BO on the form and from that instance, use it's "CopyDataFrom" method to copy the data from the original BO after you have filled it. This should rule out the querying aspect of the problem and give you an idea of how quickly the SF BO fills its inner DataTable from an existing BO.

Finally, you could write a small method in the BO that creates an ADO.Net DataTable and simply fills it from the CurrentDataTable of your BO by doing a foreach through it, constructing a row object which you can then add to the new DataTable. There are more optimized methods to do this like :-

TempTable.Merge(YourBO.CurrentDataTable);

But, this will not likely be a fair comparison of what happens when SF fills the inner table of the BO from the result of a query.

Sorry there is nothing conclusive here but, you could try some of these suggestions and see whether they shed any light on the problem. In all honesty though, it would seem that asking ADO.Net to create a 100,000 record DataTable is very likely to be the botteneck.

Best

-=Gary


Kevin Lingofelter
Kevin Lingofelter
StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)
Group: Forum Members
Posts: 77, Visits: 804
Thanks for the reply!

I thought I was going to have to use ADO.NET directly. I'll try it out. I know working with this many records is not typical, so maybe I need to consider other ways? I will eventually need to "touch" all of these records for this functionality.

I'm pretty new to .NET programming (I'm an old-school VB dev). Is there a way to asychonously grab the data and begin working with it while the table is getting filled?

Kevin

Gary C Wynne
Gary C Wynne
StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)
Group: Forum Members
Posts: 26, Visits: 236
Hi Kevin

There are a number of ways you could optimize this; some that spring to mind would be pretty funky and esoteric threading alternatives so, maybe we shouldn't go down that route Smile

It seems that the problem is the query/load time coupled with the additional process time. By far the fastest way to read the data is by using an ADO.Net DataReader.

I have attached a sample.txt file with my attempt at VB (I am a VFP/C# guy) so, it may be buggy but you should get the gist of it. Also, I have referred to SQL as the dataprovider so, you should replace these references with the appropriate ones that you currently use for Access.

The solution is that you essentially "For Each" the DataReader instance and get an IDataRecord. You then cast this to a DataRow. The DataRow can be processed in whatever way you want and then after processing it, you simply add the row to the CurrentDataTable of your BO.

So, in this way, you process the record as you read it (and the DataReader is extremely fast at reading your data in the first place). I'm sorry, I haven't actually tried/tested this code but, I am pretty certain that, bad syntax apart, it should work for you.

FYI, the CurrentDataTable property of the BO is simply a reference to the inner DataTable. Also in the sample code, I have included a way to access the "default" connection string used by StrataFrame so, you don't have to worry about re-specifying it.

Let me know if you are confused with any of this and I will try and explain further.

HTH

Best

-=Gary

 


Attachments
sample.txt (149 views, 1.00 KB)
Kevin Lingofelter
Kevin Lingofelter
StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)
Group: Forum Members
Posts: 77, Visits: 804
Hey Gary,

Just wanted you to know that I have ready your reply. I've actually been working on 3 different projects and haven't had a chance to give your suggestions a try, but I will...BigGrin

I'll keep you posted. Thanks again for the help!

Kevin

Trent Taylor
Trent Taylor
StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)
Group: StrataFrame Developers
Posts: 6.6K, Visits: 7K
Kevin,

Gary has a wealth of experience in StrataFrame.  He has been with us from the beginning and has a lot of good ideas.  I just thought I would tell you a little about his background on StrataFrame.

Kevin Lingofelter
Kevin Lingofelter
StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)StrataFrame Novice (123 reputation)
Group: Forum Members
Posts: 77, Visits: 804
Just wanted to follow-up. This method worked great! The records are returned in a fraction of the time it took to fill the business object.

it takes a little more coding because I can't take advantage of the strong typing the BOs provide, but it is working just peachy.,..BigGrin

Thanks again, Gary.

Gary C Wynne
Gary C Wynne
StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)StrataFrame Beginner (46 reputation)
Group: Forum Members
Posts: 26, Visits: 236
Hi Kevin

I'm glad the datareader trick worked out for you!

Best

-=Gary

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Reading This Topic

Login

Explore
Messages
Mentions
Search