Because I use a lot of importing-scripts in my solutions, I did a test.
I have a tab-seperated file of 7248 records with three text-fields.
Filemaker imports this file in 3 seconds.
Servoy imports this file in 8 minutes!!
For us, such a big difference, is not an option. Is there something to do about it??
Import is indeed quite slow at this moment. There are a few optimizations that are still going to happen and we are investigating how we could ease the bulk loading of data. Currently Servoy does a row by row insert to ensure checks, auto-enters and logging which certainly slows down the import process.
We would like to know from you and other users:
How are you using import?
In the client?
Do you need it on a regular basis?
Would you consider running the import separately (batched/triggered) on the server?
We make applications (for so far in Filemaker) that imports for example data from a datalogger (like a blackbox in cars) Mostly the data is pulled out from the datalogger by another program (sometimes thru GSM) This program makes from every datalogger a csv-file (or tab-seperated). Everytime a blackox is readed, the data has to be imported to make calculations and reports.
For so far we use the import thrue the client, because filemaker doesn’t allow import on the server-side. But if it is possible (and quicker!) why not?
Depently of how many datloggers there are, this function is used everyday!. Very important for us is that we must have complete control over the import-function (I mean that everything is done and set in the scripts(methods), so the customers does not have to point to tab-files or has to assign the field to the right collums. Now we use DialogMagic plugin to do that in Filemaker!)
Hopefully this gives some background information for how we used it.
Sorry, the upper-message is from me. I forgot to login!
For the application you describe I would definetely use server side loading of the data. Unfortunately I havent been able to find free tools that make this easy for Firebird, I’m still searching for them and will update this thread as soon as I find them.
You could consider using postgresql (free), mysql (free) oracle or mssql with which it is very easy to bulk load data. If you choose for postgresql I can send you some sample scripts on how to get the data in. If you choose for mysql you can find information here:
http://www.mysql.com/documentation/mysq … ing_tables
If you choose for Oracle or mssql: both provide excellent documentation on buld loading data.
On the necessity of scripting imports and exports:
I’m considering deploying servoy instead of Filemaker for a large national nonprofit with offices in 5 cities that regularly recruits people to its programs. For complicated business/political reasons they can’t all share the same backend.
Every few days one office will want to send an export of its recruits to another office to see if they are interested. So the export must be scripted (and emailed) and then retrieved and imported, all by script, and all FAST (8 minutes, as in the above example, would be too slow).
FM can do all of this with plugins. How soon will this be possible and will it be as fast as FM?
Thanks,
Chris
How many rows are you transfering? Servoy has become significantly faster in importing, to load large amounts of data it is still better to use the database backend tools though. You can integrate those into servoy to fairly easy.
Ok, using external tools is better (this goes for deleting large sums of rows too) but:
I would not mind using a server-side import when it is a -not-so-many-times-thing-. Otherwise I wouldn’t be happy as a customer walking over to the server every time I need to do an import. And this is the same for an external tool too…
We have to bare in mind that we work with SMA customers sometimes with all the right infrastructure and often without enough knowledge. Surely one of the plilars of succes for FM.
Ciao
I agree with ITBE
I think a server-side plugin could maybe help.
the client points to a file (by hand or thru a method), the file is transfered to the server and server takes care of the rest. (not the client)
I am just thinking!
jaleman:
Servoy has become significantly faster in importing, to load large amounts of data it is still better to use the database backend tools though. You can integrate those into servoy to fairly easy.
Hello - At issue for me is that users must be able to click “import recruits from another office” and that the system will take care of the rest. It doesn’t matter if it’s going through the server or the client.
If it’s a simple matter to integrate a tool to import through the database backend, could you please explain how to setup a method to automate such an import into Firebird, with, for example, the file needing to be imported starting out on the client’s machine on the c: drive?
Thank you so much!
Hate to sound like a broken record but replication/synchronization is a built in feature of SQLAnywhere. This is not a slapped together solution but has been a key feature since 1997 (I believe).
And better yet - if I’m reading other posts correctly - Servoy will soon bundle SQLAnywhere as the default database.
New to Servoy so I did a remote import test. I created a Filemaker table with two fields and did the same with Servoy. I imported from my laptop to a server hosting both the Filemaker file and Servoy. I used a full version of FMP and the Servoy’s smart client. 2,620 records took 4:04 in Filemaker and 12:40 in Servoy.
Bill Belanger
And next, do the search test. Have a FileMaker table with 1.000.000 records and do a search in 3 fields. Now do the same with Servoy and a proper database… Everything has its pros and cons.
I agree with Patrick and, apart from that, I think 4 minutes for 2600 records is long. 12 minutes too long so there must be something special going on…
I guess the question to ask is what kind of data are we importing.
2600 records doesn’t say much either…are there 3 columns per record or 300 ?
Also how are you importing things in Servoy, using a custom method or with the excel/csv import plugin?
If with a method what does the code look like?
Also when you import you might want to start with a transaction and end with a commit.
Depending on the back-end database this can speed up things significantly.
I created a Filemaker table with two fields
I assume the ‘two fields’ means ‘two columns’…
Remember that unlike Filemaker Servoy is not a database. If you need to bulkload data all sql databases have tools for this. eg in sybase loading of 2600 rows using it’s loader tool would take about a second.
Patrick, don’t leave me hanging…What is the result for a 3 field search with a million records? If you’ve done it, tell me so I don’t have to create a million records. That would have been my next test today. I’m importing the records using the “Import” menu item in the smart client, then choosing the xcel file, same as I did in Filemaker. I’m not even close to being able to construct methods and use plugins, I’m a beginner here.
Robert, there were two “columns”. Why do you guys call them columns sometimes and sometimes fields? When I’m making a form in Servoy, the elements menu has add a “field”. Every other tools I’ve used called them records and fields. If you want to call them rows and columns, that’s fine. How about choosing one?
Don’t jump on me all at once guys. I’m only trying to justify the added cost of hosting a database with Servoy over Filemaker. I’m hosting a county wide sheriff’s department now on my server with Filemaker. We have items that Servoy certainly can do better, but I can host 100 concurrent web clients for 2500 bucks with Filemaker, 8 with Servoy for the same money. I need to sell this to my client, that’s all. There’s no question that Servoy is a better tool, I’m sold on that, but at some point it’s a matter of economics and what we can live with for the price.
As far as imports, my reason for testing that is that we would like to download mug shots to their laptops in the cruisers. Just viewing them isn’t sufficient because they don’t always have access to the internet.
Bill Belanger
Bill, I don’t have the feeling anybody jumps on you.
It is more that you ask questions with little or no information to give a proper answer so there you have it. All of us want to give you an answer that makes sense but need to find out ‘what information is behind the question’.
Robert, there were two “columns”. Why do you guys call them columns sometimes and sometimes fields? When I’m making a form in Servoy, the elements menu has add a “field”. Every other tools I’ve used called them records and fields. If you want to call them rows and columns, that’s fine. How about choosing one?
We call ‘them’ fields when they are on a form. In this case they are not columns since they refer to table columns.
Rows, records refer to the same thing and usage of the word depends on somebodies background I guess.
You might try taking a look at this form post which has a few alternative methods for importing data, which can be faster.