separate or combined value-list source tables?

Hi all,

This is a design question that, in my db engineering past with FileMaker Pro, I historically have handled using a compressed structure wherein any number of tables used as data sources for value lists are compressed into two tables - List and List Items.

I am wondering, if one can give an answer that holds true generally across all the commonly used SQL-based database platforms, whether it still makes sense to use the same approach with SQL tables, or better to move to using a separate table for each value-list data source.

Here is an example of two separate tables one could create as the data sources for two value lists:

[attachment=2]separate_vl_tables.jpg[/attachment]

The data in these two tables might look like this:

[attachment=1]separate_data.jpg[/attachment]

Whereas … what I mean by a flattened VL data-source structure would look be done like this:

[attachment=0]combined_data.jpg[/attachment]

The obvious advantage to the combined-list structure is that one can create new value lists by simply adding data rows rather than adding tables. Further, in the parent “vl_type” table one can add a column to store a semaphore indicating which user priv level can edit the list, and this value, again, can be modified without touching the code under the hood. (There were further advantages to the compressed-table approach when using FMP, that don’t apply to SQL 'bases.)

But I have much less experience in optimizing design for SQL engines, and with the snags I might run into when coding in Servoy. So I wonder whether would there be any downside to the compressed approach? I’m not anticipating any but thought I’d post this anyhow “just in case”.

TIA for any thoughts/shared experience,

kazar

separate_vl_tables.jpg

Hi kazar!

this is a very good topic indeed, thanks for bringing it! I’m sure that there are lots of different opinions on this and that it is debatable and always good to know the best practices of others!

You touch here a design problem that IMHO is not as clear cut as you show it here and for which there is always decisions to make depending on what you want to achieve. I can actually see a few drawbacks to your approach, even though it works very well in some case.

The biggest problem here, I think, is database constraints. In your design, how do you ensure referential integrity in your database?
If every value list is in one table and you actually use fk constraints in the database, you will have them coming from a lot of different places in your graph, and each of them will have to have the relevant type in a compound foreign key.
Consider this design for example (inspired by yours):
[attachment=1]example_design.gif[/attachment]
(BTW this was done in Servoy with Aquarius ORM Studio :wink:)

No you can see that I have constraints from student to student_status, and that I have constraints from semesters_courses (a many-to-many table) to semester. Now If I put everything into one single table, these 2 tables will point on it, with the additional overhead of having to deal with a compound key (2 fields to define the one relation).

I’m not saying that your idea is not valid, it can well be depending on the way you code it - from what I undestand of your value lists, they are used only for “selecting” values, and the “selected” value is then copied into the “calling” table, is that it?

It has some advantages but it also have disadvantages when users change the values of the value list table, because then the copied value is standalone and will not be updated as well.

In some cases, this is exactly what you want, of course: imagine tax values, you wouldn’t want all the tax values of the already billed invoices to change when a tax changes, do you? But then in some other cases you will want the values to be updated, meaning that the real values will only live in the vl table and you only store an id in the calling table (like in the design above).

As for the advantage of adding record in the vl_type and corresponding values in the vl_items table to have a new type with no coding at all, I doubt that you will not have to adapt your code to be able to use your new type anyway. In any case you will have to change the other tables to use this new “kind” of vl, and to change the code accordingly. So the “no code” advantage is not that obvious to me.

And then there is the performance factor. I’m not sure that having everything inside one table is such a good idea, it might become a little bit crowded with time, you can always use indexes but they too have their own pitfalls. And every time you will hit this table in SQL you will have to add the filter to retrieve the relevant type…

I have had to build (in java for the web) a search engine capable of dealing in SQL with any kind of runtime searches (meaning the user can add any kind of criteria, with any kind of operators and combine them at runtime) and I can tell you that dealing with different “lookup” tables is actually easier than having to deal with one filtered by a type.
Consider the interface below, and understand that behind each line of query that deal with a value list (these are the combo-box here), you would have to add two criteria instead of one if the value lists were all into one table:
[attachment=0]searchEngine.gif[/attachment]

So I would say that your approach works great depending on your code and what you want to achieve, but it is not always the best answer.
I’m sure that others will have again diverging opinions on that matter, and I would love to see the definitive answer to that kind of design problem, but I fear that it will always be a design time decision that you will have to do every time you design a system.

But I’d love to hear the opinions of those with a long experience of Servoy in production and see what has worked best for them from the Servoy perspective…

Much to chew on in that answer, Patrick, thanks! I will be posting again after I process & re-re-think the question…

kazar

(BTW this was done in Servoy with Aquarius ORM Studio)

I would LOVE such a tool inside Servoy, and automaticly show my DB-tables and the Servoy relations, between the tables.

possible??

Harjo:

(BTW this was done in Servoy with Aquarius ORM Studio)

I would LOVE such a tool inside Servoy, and automaticly show my DB-tables and the Servoy relations, between the tables.

possible??

Hi Harjo,

I used to use Azzurri Clay for DB design, but since the pro version is not available out of Japan and the free version lacked the HTML documentation export that I wanted, I looked for another one…
Aquarius ORM Studio is an Open Source tool that works quite well on Servoy (with still a few quirky bugs comparing to the same on MyEclipse, that hopefully will be corrected with Tano being on Galileo)…

It doesn’t show the Servoy relations (it knows nothing of Servoy), the relations you see above are the database contraints, they live in the database. Still, you can reverse-engineer your database, and it will create the DDL for you, and it’s a lot nicer to build your schema visually (it helps documenting your solution too :)), and if you use the “description” fields in the DB, Table and Fields, it will produce automatically a very decent HTML documentation of your schema.

Check it out here: http://orm.aquatool.org/nl/en/US/
You will have to download the plugin from the “install plugin version” link, choose the “Download Link (For Eclipe3.2,3.3,3.4)” and unzip directly into your Servoy/developer folder.

Hi Harjo

I think it’s even today a good idea to clearly separate between the relational model (http://en.wikipedia.org/wiki/Relational_model), introduced by E.F. Codd, see http://en.wikipedia.org/wiki/Edgar_F._Codd
and the entity relationship mode ( http://en.wikipedia.org/wiki/Entity_relationship_model) introduce by P.P.S Cheng, see

The Relational Model (RM) knows (only) the terms: domain, relation (table), tuple, and attribute. It especially doesn’t know the term relationship, as this term was introduced only 4 years later by Cheng after the RM was introduced by Codd. Servoy uses unfortunatly in a for me still confusing way the name relation, i. e. table, but meaning actually a relationship.

The relational model in short is a mathematical model to avoid redundency (1st, 2nd and 3rd normal form). The Entity Relationship Model (ERM) is used to design a data base, a complete other task than what was aimed with the relational model.

Luckely, in general, if you use the entity relationship model (carefully), you get at least the 2nd normal form for free, often the 3rd.

If you are talking about tables,and therefor about the relational model, there is no such thing as relationship (as said Servoy calls it relation, but a relation is a table, see Wikipedia). The problem is, many, not the least tool developers, messed and mixed everything with these 2 theories, using terms vice a versa :-(

So, contrary to your wish, I think it’s best Servoy does never introduce such a tool! It is very demanding to build such a tool and its code generator and others have already done it. If you design an Entity Relationship Model, the big advantage is that it really can automatically implement it (create the real needed SQL code), after you made some design decisions, for example how you like to have implemented Super Type/Sub Type entities (horizontally, vertically, or all-in-one).

And yes, I nearly forgot it, I think I more or less completely agree with Patrick’s explanations. We use combinde VL for very generic VLs, otherwise you often clearly “see” where they belong to and should be designed at the appropriate place as Patrick said.
I think you also mix Value Lists and Entity Life History (like student status, which actually describe the stati of a students entity life cycle). These are 2 different things. Ilyse, if you like we can discuss this at ServoyCamp.
By the way, the table semester looks a bit funny with 3 entries, this looks more like a trimester :D
If you arbe building a school admin software (as we do), you may need a period as Taktgeber (sorry, don’t know the english term for it) like a quartz for a CPU , built up by year and a fraction (name) to be flexible whether it’s a semester or trimester etc.

Best regards, Robert

PS: A very usable tool is Entity Modeler, as a whole environment WOLips, which was formerly WebObjects. As WOLips, it’s since some years Open Source and available as a standalone or Eclipse plug-in. So you can have it in Servoy.

Harjo:

(BTW this was done in Servoy with Aquarius ORM Studio)

I would LOVE such a tool inside Servoy, and automaticly show my DB-tables and the Servoy relations, between the tables.

possible??

Harjo:

(BTW this was done in Servoy with Aquarius ORM Studio)

I would LOVE such a tool inside Servoy, and automaticly show my DB-tables and the Servoy relations, between the tables.

possible??

I can agree that it would be great to have a tool like that in Servoy, but on the other hand I see also problems.
First I don’t want all defined relations in my view, so in that case you need to have a setting within Servoy ‘Use for diagram yes/no’
Second I have already more than 100 tables within my application. I don’t want a diagram on one single A4 page. It will be unreadable.

And maybe a better solution will be to have API’s/Plugin/Bean available in Servoy to create a diagram using one of the programs mentioned in this topic.
Within Servoy you can loop through all your relations.
If you put some mark in the relationname that signals that this relation needs to be shown in your diagram (for example: relationname orderline_to_article$IN_DIAGRAM as relation from orderline to article)
Then if you have a plugin where you can create tables, fields and relations like you do in that tool manually, then this would be a solution.

So,

Is it possible to create a plugin around such a tool?
Who is capable to write such a plugin?
Who has time to write such a plugin?

Martin, I totally agree with you.

it was not my intention, te replace it with the functionality we now have, but just an extra.

kazar:
This is a design question that, in my db engineering past with FileMaker Pro, I historically have handled using a compressed structure wherein any number of tables used as data sources for value lists are compressed into two tables - List and List Items.

The most common issue with value lists from a customer standpoint is the power to modify themselves. The user interface implications of this starting point drives how value lists are setup technically for us. Most Servoy solutions are not going to run into the table performance drawbacks Patrick brings up and changing value list values is the same issue no matter how you set your value lists up.

Since creating a user interface for multiple (and unrestricted growth) tables is not fun, we use one table for all value lists. Have an additional column for the value list name, create a global with the value list name, relate global to the value list name column, and create a value list based on this relation. Servoy does all the work from there.

You can also setup chained “related” value lists with a single table. A few additional steps but not difficult.

Put the interface for your value lists in one module and include in every workflow module as a “resource” module. Even if a particular value list is defined from another table, create that value list in this value list module.

Lastly, you need a good mechanism for the customer/user to be able to do a find and replace across a foundset. A technical user can then easily change a value list value and update a foundset to reflect the new value.

Thanks for your input on this David. Largely for the UI reasons you cite I also decided to stick with my old approach of using a single Lists table.

In the UI for modifying value lists, for some lists I present only one column, for others two columns for Value and Description (to aid during data entry).

I also find it handy to include an Editable switch on each row that is accessible only by a dev or a solution super-user. This way I can provide VLs that have one or more hard-wired choices that may be necessary to satisfy business rules and that cannot be deleted or edited by anyone but a super-user, but admin users are allowed to add (and edit) additional choices to the list.

kazar

kazar:
I also find it handy to include an Editable switch on each row that is accessible only by a dev or a solution super-user. This way I can provide VLs that have one or more hard-wired choices that may be necessary to satisfy business rules and that cannot be deleted or edited by anyone but a super-user, but admin users are allowed to add (and edit) additional choices to the list.

You’ll see this in future versions of our frameworks :) Good tip.

david:
You’ll see this in future versions of our frameworks :) Good tip.

Cool. I imagine I’ve racked up a $0.10 discount?

oh, and here’s another idea, david.

Instead of offering the user the ability to head out across various tables/form of a solution to perform a Replace on a field whose VL item has been edited (thus introducing a cognitive gap in the space between editing the VL in one UI, and then locating where across the solution that VL is used and performing a Replace via a different UI) … what I’d love to do is make an “update existing records” button become visible when a VL item has been edited.

How could we easily track on which columns across a whole solution a particular VL has been applied, so that the user could choose a simple ‘update existing’ one-click operation? I would think we could write an array of table|column pairs into a column in the Lists table, and then completely automate the process of updating solution-wide? In most circumstances it’s likely that any value list that would be used in more than one table’s data would be contain choices one would never be editing (example: Yes/No) and that most updates, therefore, would only need to be done in one table. But “just in case” a more esoteric value list is indeed used for data entry in more than one table, i’m workin’ on this idea of using (and keeping maintained) a list of where the VL is used…

your thoughts? (if you like it that’s another $0.10 credit due me!)

[The other option, of course, would be to always work of the the list_uid and pull the actual value into forms via unstored calc. Certainly not optimal for searching! Not a real option, IOW.]

I like it.

  • Technically, figuring out where a value list is being used can be done by checking the Servoy solution meta data (which we do for all kinds of things – file system if in developer and repository if deployed).

  • Possibly give the user a choice by returning a list of where a value list is located by navigation set and navigation item (frameworks engine terms).

  • On the selected locations, run find and replace code.

Removing some drudgery and reducing business data junk seems worth the effort. Troy and I will give it a review.

On a side note, there will be a place to discuss all this coming soon. Participation will have a reputation and rewards system as a little extra added incentive. My impression is that you are undervaluing your potential contributions by a few pennies :)

On a side note, there will be a place to discuss all this coming soon. Participation will have a reputation and rewards system as a little extra added incentive. My impression is that you are undervaluing your potential contributions by a few pennies :)

Soon…?

Are you able to be more specific or is it related to Servoy World tour… :lol:

Working on a public release build for Servoy 4.1.x. 2 to 4 weeks out.

Troy and I will be at the NYC tour date October 5. Thinking about hitting the LA tour date October 9 as well. Get some surfing lessons/abuse from Portnoy…