Import Speed?

Classic Servoy
1

Hi Again Guys:

I’ve got my “test” server running. Clean install of W2K Advanced Server, 2..4ghz P4, 1GB or RAM, Oracle Instant Client, Fresh Install of Servoy 2.2.

Oracle is on a seperate box.

I setup a new file, just a dozen basic field, miz of text, dates and number. I setup a DB SEQUENCE with an index.

Importing from Developer on the Servoy server was very slow.

I moved to my 1.8ghz Pentium M notebook, and it improved dramatically, but still not blindingly fast.

Are there ways to optimize the speed of Imports from within Servoy. Does servoy commit each record, or do it in batches? Is there a way to adjust the commit count? Any idea why my notebook, which is also running Servoy Developer, would run so much faster then a W2K Server with a faster processor?

Thanks,
Lee Snover

2

Lee

There have been several previous threads on Importing large datasets & general advice is to use the backend DB tools since they are optimised for the process.

Graham

3

grahamg:
Lee

There have been several previous threads on Importing large datasets & general advice is to use the backend DB tools since they are optimised for the process.

Graham

Graham:

I did a search on Import and Importing, but there where too many topics that come up and none of the Subject headers seem to be relavant. I can’t seem to find a way to search just on the Message Title/Headers and that might help.

Again, I know there are better tools, but my users will be in Servoy, so I am attempting to work within the Servoy environment as much as possible.

Thanks,
Lee Snover

4

Hi Lee,

It is probably difficult for you and other forum members here as there seems to be a lot of assumptions being made regarding what level of flexibility you need for your users.

So, a la FM, do you just need to allow access to the drop down menus or do you need to script the opening of an import dialogue so that the users can do their own totally adhoc thing ?

Or is this import a one-off process ?

Perhaps if you could outline more the level of interaction then we may suggest things which may work well for you !?

Harry

5

Harry Catharell:
Hi Lee,

It is probably difficult for you and other forum members here as there seems to be a lot of assumptions being made regarding what level of flexibility you need for your users.

So, a la FM, do you just need to allow access to the drop down menus or do you need to script the opening of an import dialogue so that the users can do their own totally adhoc thing ?

Or is this import a one-off process ?

Perhaps if you could outline more the level of interaction then we may suggest things which may work well for you !?

Harry

Harry:

It’s a real mixed bag. We have stuff that we do on a regular basis that can be scripted, have a standard interface and more structure. Then we do quite a bit of Adhoc stuff.

Basically, our business involveds analyzing other companies data. We have a standard program that does very sophisticated tasks and calculations on the data, once we have it massaged into our standard format.

To get there, we get very raw and very varied data dumps from the companies. Everything from fixed length cobal generated files to excel spreadsheets and SQL Database dumps. Sometimes these DBs are quite large, and we don’t get a great deal of instructions. We are very good at intrinsically figuring out what the data is all about and then getting it into our formats. We’ve been doing this for years. and FM has been a valuable tool. But it is SLOW on big sets of data.

Once the data is moved into our program by our “mid-level” users, we then allow the clients to access the system. They are also often relatively sophisticated. They will often want to review our database and even pull our refined data back out for thier own analysis in spreadsheets, reports, etc. But they can’t get to everything, just their data.

So, you see that this environment provides some big challenges. FM7 has been good, but slow, and the deployment to outside users is very difficult.. As a programmer, I find scripting very limiting. So, I am hoping that Servoy can provide the right mix of sophistication and ease of use.

Regards,
Lee Snover