Saving big text files in a Varchar Max field?


Author
Message
Edhy Rijo
E
StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)
Group: StrataFrame Users
Posts: 2.4K, Visits: 23K
Hi guys,



I am currently saving big text files (450,000+ lines) in a Varchar(Max) field, for a while the performance of getting a record with such data decreased and it is noticeable in the browser dialogs and reports.



I am wonder is there is a better way to deal with so much data in a single field, so would it be possible to kind of compress the data in the field somehow?



I know that I may have this field in another table to separate it from the main table and do some magic to make it look as part of the record in the form, but if that is no the best way I am open to any suggestions.



Please keep in mind that currently I am showing the data in the form in a readonly textbox, but I can change the design to not show it up when the form loads and put a button so the user can get the data and then show it, but at least that will not slow down the browser dialog.



Thanks!

Edhy Rijo

Replies
Edhy Rijo
E
StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)
Group: StrataFrame Users
Posts: 2.4K, Visits: 23K
Trent L. Taylor (02/18/2010)
Edhy,



You are aware that you can update a single field as a blob or chunk with events that notify you of the progress, right?




Trent, Dustin,



I was not aware of the Blob field, while working with VFP in the past we always tried to stay away of Blob fields for inestability reasons and it did not cross my mind to even think about them in this case. Since these CSV files are really big, I like the idea of compressing them and be able to show a progress bar during this process, so I will take you on your offer for the sample in VB if possible (I know you are switching to C# Tongue)

Edhy Rijo

Trent Taylor
Trent Taylor
StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)
Group: StrataFrame Developers
Posts: 6.6K, Visits: 6.9K
I will try to get you a sample...I kind of think I have posted one out here before for Kenneth Langley. Might do a quick search to see if anything comes up. Also, we just called this method blob as some databases (i.e. Oracle) call it blob. But in essence you could do this for any field type. But in SQL Server, I would use a VarBinary(MAX) field to store the data. So you would just convert your file into a byte array and then save that into a VarBinary field.



I will try and allocate some time to write this sample...sorry for the delay. I am getting somewhat hammered this morning (and I am not referring to beer drinking contests! w00t )
Edhy Rijo
E
StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)
Group: StrataFrame Users
Posts: 2.4K, Visits: 23K
Hi Trent,



Sorry to bother you on this one, but I searched the forums and could not find any sample code on how to do this. Please when you have some time see if you can post something quick to get the idea.



Thanks!

Edhy Rijo

Trent Taylor
Trent Taylor
StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)
Group: StrataFrame Developers
Posts: 6.6K, Visits: 6.9K
I'm sorry Edhy...this slipped off of my radar. I will try and get you something right now. Blush



I've been dealing with bureaucrats the last few days, so I was a bit preoccupied (not to mention frustrated!). Smile
Trent Taylor
Trent Taylor
StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)StrataFrame Developer (14K reputation)
Group: StrataFrame Developers
Posts: 6.6K, Visits: 6.9K
OK, here is an example I posted a while back. Let me know if this is not enough to work off of.



http://forum.strataframe.net/FindPost21474.aspx
Edhy Rijo
E
StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)StrataFrame VIP (6.4K reputation)
Group: StrataFrame Users
Posts: 2.4K, Visits: 23K
Thanks a lot, I believe that will be good enough, will try to implement this today and will post results.



About the bureaucrats, just don't pay too much attention to them Hehe


Edhy Rijo

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Threaded View
Threaded View
Edhy Rijo - 15 Years Ago
Teddy Jensen - 15 Years Ago
Edhy Rijo - 15 Years Ago
Russell Scott Brown - 15 Years Ago
Trent L. Taylor - 15 Years Ago
Trent L. Taylor - 15 Years Ago
Edhy Rijo - 15 Years Ago
Trent L. Taylor - 15 Years Ago
                         Hi Trent,

Sorry to bother you on this one, but I searched...
Edhy Rijo - 15 Years Ago
                             I'm sorry Edhy...this slipped off of my radar. I will try and get you...
Trent L. Taylor - 15 Years Ago
                                 OK, here is an example I posted a while back. Let me know if this is...
Trent L. Taylor - 15 Years Ago
                                     Thanks a lot, I believe that will be good enough, will try to...
Edhy Rijo - 15 Years Ago
Edhy Rijo - 15 Years Ago
Dustin Taylor - 15 Years Ago
Charles R Hankey - 15 Years Ago
Greg McGuffey - 15 Years Ago
Edhy Rijo - 15 Years Ago
                     That's what I'd try.

BTW, I took a look and my memory...
Greg McGuffey - 15 Years Ago
Charles R Hankey - 15 Years Ago
Edhy Rijo - 15 Years Ago
Teddy Jensen - 15 Years Ago
Edhy Rijo - 15 Years Ago
                         OK - thanks, was just curious on what route You choose. I will...
Teddy Jensen - 15 Years Ago
                             [quote][b]Teddy Jensen (02/26/2010)[/b][hr]I will probably later try...
Edhy Rijo - 15 Years Ago
                                 This is all good stuff. One other thing I thought I would mention is...
Trent L. Taylor - 15 Years Ago
                                     OK - i tried to get this to work, but am apparently missing something,...
Teddy Jensen - 15 Years Ago
                                         Teddy, are you referring to the PullBlob or PushBlob here?
Trent L. Taylor - 15 Years Ago
                                             It was the PushBlob, but I have it working now. Ihad a null value...
Teddy Jensen - 15 Years Ago
                                                 Great....glad you got it going! :) Yeah, it is really nice when you...
Trent L. Taylor - 15 Years Ago

Similar Topics

Reading This Topic

Login

Explore
Messages
Mentions
Search