Is there a maximum filesize with fileload?

Author: i2stiller@gmx.de (istiller)

Hi We just doing some profiling on our application and increase the "rounds" of calls. As a result, we got a profiling log-file of about 900 MB and around 4 500 000 lines. This log file  I do load by lfileload and go through the lines and analyize them. But after 122Mb the file is cut off ? Try to validate this, I did start the debuger after loading the file. $status give the ~960000000 lines but a few seconds later UnifAce crashes (with out any other action) So is there any upper limit? And if yes, how to analyze  profiling log as UnifACe gives us no application that will do this job :-) Ingo

10 Comments

  1. diseli said Why not use the TXT-driver? I did a quick test and loading a 1GB profiling log-file does not seem to be problem. I can also loop through the 3.8+ million lines. It probably is also a good idea to use discard (after analyzing a profiling line) in order to safe/release memory (when not needed anymore). Hope this helps. Regards, Daniel  

    Hi Daniel How could I open an arbitrary text file? Ingo


    Author: istiller (i2stiller@gmx.de)
  2. istiller said
    diseli said Why not use the TXT-driver? I did a quick test and loading a 1GB profiling log-file does not seem to be problem. I can also loop through the 3.8+ million lines. It probably is also a good idea to use discard (after analyzing a profiling line) in order to safe/release memory (when not needed anymore). Hope this helps. Regards, Daniel  
    Hi Daniel How could I open an arbitrary text file? Ingo  

    Hi Ingo, I was asked already once to solve this kind of issue and just rediscovered it this morning in my archives! We solved this way: 1) Front-end let user choose its file 2) File chosen is verified solving variances like CR vs LF vs CRLF as line terminator 3) Verified file is copied where ASN file address it 4) Application is loading the copied file 5) Copied file is renamed/deleted after loading This initial implementation was single user only and file size was limited. In a later stage we added more functionalities: 2bis) Splitting verified file in chunks when too large 3) Verified file is copied where ASN file address a set of them to enable multiuser usage 4bis) Checking if a further chunk should be loaded Hope it helps. Gianni


    Author: gianni (gianni.sandigliano@unifacesolutions.com)
  3. Hi Gianni, nice solution to still have a choice of the file but with the comfort of TXT. IIRC, once there was a problem that one couldn't close "$TXT" the workaround was to implement the universal close without parameter close this way, the TXT input file was no longer locked and you could process the next file


    Author: ulrich-merkel (ulrichmerkel@web.de)
  4. Hi Uli, Oh YES...I remember that issue! I remember also another tricky situation, again about file locking: when a TXT file was generated embedding record types in it (1=document header, 2=document details), an application should open same TXT file more than once, while using functional subtypes to read embedded tables. ASN file was pointing two model tables to the same physical TXT file: the second open returned an error! I remember a /nolock to be added to assignment file to avoid getting error but as of today I do not see it in current documentation. We are differently young in these scenarios... :-) :-) :-) Gianni


    Author: gianni (gianni.sandigliano@unifacesolutions.com)
  5. gianni said Hi Ingo, I was asked already once to solve this kind of issue and just rediscovered it this morning in my archives! We solved this way: 1) Front-end let user choose its file 2) File chosen is verified solving variances like CR vs LF vs CRLF as line terminator 3) Verified file is copied where ASN file address it 4) Application is loading the copied file 5) Copied file is renamed/deleted after loading This initial implementation was single user only and file size was limited. In a later stage we added more functionalities: 2bis) Splitting verified file in chunks when too large 3) Verified file is copied where ASN file address a set of them to enable multiuser usage 4bis) Checking if a further chunk should be loaded Hope it helps. Gianni  

    Hi Gianni That's the workaround I also thought about :-) @UnifAce: Let there be a function $tableproperties() Then a topic "TABLE=<tablename>" resp. "FILE=<filename>" to open other files then defined in model Ingo


    Author: istiller (i2stiller@gmx.de)
  6. Why not use the TXT-driver? I did a quick test and loading a 1GB profiling log-file does not seem to be problem. I can also loop through the 3.8+ million lines. It probably is also a good idea to use discard (after analyzing a profiling line) in order to safe/release memory (when not needed anymore). Hope this helps. Regards, Daniel


    Author: diseli (daniel.iseli@uniface.com)
  7. Perhaps a matter of your memory ?? In most cases where you process large files line by line, using the good old (since 1996) USEQREAD.DLL would be a real booster. because it empowers linear processing speed and minimal memory consumption (only for 1 record) See the old uniface listserver entry from https://groups.google.com/forum/#!topic/comp.soft-sys.app-builder.uniface/-WBbREJqco4


    Author: ulrich-merkel (ulrichmerkel@web.de)
  8. Hi Ingo, I made some stress test with a simple form loading from a profiling log file, duplicating in memory and dumping to another file with 6.644.800 lines sizing 1.280MB!; Uniface (32bit) was using about 3GB as maximum. The issue we are facing with could be related more probably to other objects in memory or DBMS driver. I am currently doing this small test using SQLLite on U9.7.04.G422 plain; which DBMS are you using? Gianni


    Author: gianni (gianni.sandigliano@unifacesolutions.com)
  9. diseli said Why not use the TXT-driver? I did a quick test and loading a 1GB profiling log-file does not seem to be problem. I can also loop through the 3.8+ million lines. ... Regards, Daniel  

    Hi Daniel, very interesting! Could you provide your quick test configuration? UPDATE: No need for this...I just made it! Using U9704.G422 (32bit) I've retrieved a profiling log file sized 1.724GB containing 8863744 rows and looped into it forward and backward. Maximum RAM usage for Uniface process while looping was more than 2.8GB. I am guessing using Uniface 32bit platform (RAM costrained!) maximum profiling log file manageble could be around 10 millions rows...not bad! The only thing I would like to be explained from ULab is the huge overhead Uniface is applying: 1.724GB requires more than 2.8GB to be managed: it is more than 60% overhead. Could I ask for a rough explanation? Thanks in advance. Regards, Gianni


    Author: gianni (gianni.sandigliano@unifacesolutions.com)
  10. Hi Ingo, I am currently doing some stress test on this same subject. I can confirm the maximum profiling log size I was able to load is a little bit more than 120MB, about 630k profiling lines in my case. Regards, Gianni


    Author: gianni (gianni.sandigliano@unifacesolutions.com)