Developing services|forms to manage large number of occurrences

Author: gianni.sandigliano@unifacesolutions.com (gianni)

Hi all, Developing a Uniface functionality able to manage a large number of occurences (hundred thousands or millions), either as a real Uniface Service in a 2/3tier env or as a Uniface Form in a 2tier env, trying have the minimum overhead, alias the best performances, it is better to avoid to paint any field on the tableau, just the entities markers. In the same kind of functionality usually a log file must be generated to enable the end user to verify the correct execution; obviously in this log file every type of warning/error should be recorded. If the developer needs to catch errors also at field level, those managed in the "on error" trigger like a missing part of the PK, it is forced to paint those fields on the tableau. To me this is a little nonsense... Confused Confused Confused Am I missing something? Is performances in a Uniface service not influenced from fields picted on the tableau? Is anyone knowing a different or better way to implement it in Uniface? Thanks for your opinions! Gianni

10 Comments

  1. Hi Uli and Knut, nice to hear from you again! Thanks Theo for your input... Reading your answers had let me recognize I should describe the problem a little bit more in detail. I need to cleanup (part of) the historical db in a custom health system. This cleanup operation is needed because informations are not fully consistent during the past times neither the db was designed with default values in mind. We need to catch all "wrong things" either at technical level or at semantic level. There are two sets of tables, online and historical, composed each one of 5 (large) tables; the main table is "ADMISSIONS" while the others are connected with a many relationship: ADMISSIONS |-- DAYHOSPITALS |-- TRANSFERS |-- OPERATIONS \-- COMPLICATIONS Simplifying my functions should: 1) browse through all historical ADMISSIONS 2) if current admission it is found in online ADMISSIONS found differerences between the two records (whole records...so field list = *) then repeat the same check for all related records of the many entities 3) if current admission it is NOT found in online ADMISSIONS: a) clean the online related entities of eventual orphaned records b) copy admission and all related entities (whole records...so field list = *) in another set of tables The scope of this function is to identify all possible "problems" in the db (missing mandatory fields, missing part of PK, whatever...) The function is already developed as a Uniface form (the customer is still working C/S on a Gb network...) and is acceptably fast and write a log file quite complete! In the path 2) is managing about 20.000 admissions and related records per minute In the path 3) is managing about 500 admissions and related records per minute Considering there are around half millions admissions and more than 1 million of related record I was forced to enable the user to work per periods (months / years) to complete the task in batches. Technically as I told already to catch all possible errors and their details into the LOG files I managed: - accurately check all possible errors in my code. - adapt all "ON ERROR" triggers at entity level. What is still missing is to catch those errors trapped in the "ON ERROR" triggers at field level... but...to catch those only in this function I need to paint them on the tableau... ...and painting them on the tableau just to change the on error trigger means having the whole form slower... Is there any way to change the On Error trigger at field level in a form without painting the field on the tableau? Thanks, Gianni


    Author: gianni (gianni.sandigliano@unifacesolutions.com)
  2. still don't see what the on-error trigger is about and why it's necessary to change it locally, but: Create a subtype, modify the on-error-code in the model could be an option but the changed names may affect your code. Another one is use a precompiler constant to select versions of the common on-error-code at compiletime: #ifdefined check_wrong_things ..... #else ........ #endif P.S. think there are much better ways to specify the batches than date ranges, just a little trick can help a lot.


    Author: ulrich-merkel (ulrichmerkel@web.de)
  3. Hi Gianni, Thank you for explaining a bit more..... My 1st thought here would be to; 1) send the PK of whichever entity that needs to be validated to a separate service component (suggest using a separate component just for the sake of keeping things simple) 2) retrieve the entire record with all fields included / formatted - but on a separate PATH (you don't want to complete the hitlist - right?) 3) execute a validate statement 4) trap all the onerror field level events and process from there (maybe with a call back to the originating component to do the updates / changes) 5) complete and return to calling component 6) repeat Good luck. Knut


    Author: Knut (knut.dybendahl@gmail.com)
  4. Hi Gianni, on your list, there are a lot of tasks which can be done much more efficient directly on the database. The cleaning of potential orphans is just one SQL command in the database. Incomplete PKs must not occur because the PK fields have to be NOT NULL by convention. Empty mandatory fields are only a problem if they are in the "variable length" part of the record all the others are just a mandatoryfield IS NULL


    Author: ulrich-merkel (ulrichmerkel@web.de)
  5. It may sound strange coming from me, but I would not use Uniface for this. For a one-time cleanup action like this you are probably better of in SQL. Or first use SQL for whatever you can simply do in SQL and after that run a Uniface program for the more refined/complicated actions that you need to take.


    Author: Theo Neeskens (tneeskens@itblockz.nl)
  6. Theo Neeskens said ... but I would not use Uniface for this. For a one-time cleanup action like this you are probably better of in SQL. ...

    Theo, unfortunately IS NOT a one time cleanup because all existing functionalities dealing with these tables are not (yet) fully fixed!

    ulrich-merkel said ... Incomplete PKs must not occur because the PK fields have to be NOT NULL by convention. ...

    Uli, this was one of the "wrong things" should have been fixed in the past but unfortunately it was NOT... That's one of the reasons I need ON ERROR trigger at entity level...

    KnutD said ... 1) send the PK of whichever entity that needs to be validated to a separate service component (suggest using a separate component just for the sake of keeping things simple) 2) retrieve the entire record with all fields included / formatted - but on a separate PATH (you don't want to complete the hitlist - right?) 3) execute a validate statement 4) trap all the onerror field level events and process from there (maybe with a call back to the originating component to do the updates / changes) 5) complete and return to calling component 6) repeat ...

    This is good idea...

    ulrich-merkel said ... Create a subtype, modify the on-error-code in the model could be an option but the changed names may affect your code. ...

    This is another good idea... ********** ********** ********** ********** ********** ********** ********** ********** My choice: I'll desing functional subtypes in the main form to check validation at all levels, with code either in local or model triggers... taking care of a proper hitlist management...(already in place!) and avoiding memory issues [often but not always raising in U7 (yes guys, still U7 here...hoping to move to U9 soon...) when separate components and a lot of occurrences are put together...] Thanks a lot guys, for yours "guru" level inputs! Ciao, Gianni


    Author: gianni (gianni.sandigliano@unifacesolutions.com)
  7. Hi Gianny, .... but incomplete PKs are nothing more as NULLs on mandatroy fields. You can extract the problem and don't need to catch this in the entities on error trigger. As found on many large cleanup projects, the SQLs performs much much better than doing the same checks via uniface.


    Author: ulrich-merkel (ulrichmerkel@web.de)
  8. Hi Gianni, Leo is right in terms of only specifying the fields you need in the field list - but there's another challenge remaining; Uniface will always fetch the entire record - no matter what you specify in the field list. So, if your record is 5000 bytes long, that's what Uniface will fetch. Also, if you happen to have a VC* or similar in the entity, it gets even more fun... An alternative to a large entity (here I mean a large number of fields defined) - two options I've used in the past; Option 1; Copy your big entity A_BIG to entity A_SMALL. Delete all the fields you don't need from entity A_SMALL from the model - and, yes, I said delete. In the ASN file, map A_SMALL.model to A_BIG - and Uniface is happy to execute the limited set of fields against A_BIG table. Option 2; If your component needs multiple entites (up entities esp) - consider adding a view to your database, and (again) only specify the fields you need in the view.... One caveat here - as long as the records cannot be (or not likely to be) modified by others (outsiders or Uniface), the two options works fine as the checksums for the records will remain the same from your process perspective. If the data is modified (esp the fields you've removed / not selected) - well, you have an issue with data integrity... Final caveat - this doesn't work on a record management system like RMS...


    Author: Knut (knut.dybendahl@gmail.com)
  9. Hi Gianni, if a lot of database records have to be processed like in accounting where you process all the input they entered during the day in a nightly batch job, it takes a lot of design issues to implement a functionality like this in uniface with a good performance which not runs in exponential time. I spent a lot of time investigating and benchmarking to find solutions for a couple of bottlenecks. Prehaps you can contact me outside of this forum, Uli


    Author: ulrich-merkel (ulrichmerkel@web.de)
  10. Hi Gianni, Just to zoom in on a part of your question: Uniface works with a "field list". The component entity field list specifies the fields that are loaded into the component at runtime, and managed and formatted by Uniface. By limiting the number of fields that are loaded (known as field subsetting), you can enhance performance. There is a full explanation in the uLibrary topic "Field List". It is not so relevant if the fields are painted or not, it is relevant if they are in the field list or not. And if a field is not in the field list there is no data and of course no OnError trigger.


    Author: Theo Neeskens (tneeskens@itblockz.nl)