Huge database

Classic Servoy
1

Hi,

I would to import a huge amount of csv files (240,000) in Servoy.
Of course there is no chance that I can do this on my internal HD. Is there a way to either crate a database server or install Servoy on an external HD?

Thanks in advance
Irene

2

Hi Irene,

irene.meyer1422261887:
I would to import a huge amount of csv files (240,000) in Servoy.

That is indeed a lot of files. But how much data is this? How much are the files combined in Megabytes?

irene.meyer1422261887:
Of course there is no chance that I can do this on my internal HD. Is there a way to either crate a database server or install Servoy on an external HD?

Why would you say that? For instance here I created 30 millions records that was 3.9GB of data. How much space do you have on your internal HD?

3

Hi Robert,

Thanks for your prompt reply.

The bunch of files are aprox 5 Tb.
Our internal hd is 1 tb but the free space is 450 Gb.

We have a RAID system of 12 Tb attaced to it. so I think it will be enough to transport the data.

My main question is can I install Servoy on the RAID, or how can I instal there the database server and let it communicate with Servoy.

Cheers
Irene

4

irene.meyer1422261887:
The bunch of files are aprox 5 Tb.
Our internal hd is 1 tb but the free space is 450 Gb.

We have a RAID system of 12 Tb attaced to it. so I think it will be enough to transport the data.

Pretty easy to do with a small amount of NodeJS code using Baby Parse (handles the remote file streaming and csv processing) and a database connector to whatever db flavor you’re targeting.

5

Hi Irene,

irene.meyer1422261887:
The bunch of files are aprox 5 Tb.

Okay, now we’re talking. :)

irene.meyer1422261887:
We have a RAID system of 12 Tb attaced to it. so I think it will be enough to transport the data.

My main question is can I install Servoy on the RAID, or how can I instal there the database server and let it communicate with Servoy.

That depends. What database server do you (want to) use? PostgreSQL? If so do you use the bundled one or the EnterpriseDB installer?
If it’s the bundled one you simply install Servoy on that RAID drive and you are set. No extra work needed. If you want to use the EDB installer then it requires a lot more work.
So it depends on what you want to do and/or use.

6

As you have a large amount of data, you need to import the files directly into the backend database with some kind of script.
Don’t try to do it through Servoy unless you really have to – it will be very, very, very slow…

7

Correct.
PostgreSQL’s COPY command comes to mind.

8

Hi Robert and Christian,

Thank you very much for the help.
I will install Servoy on the RAID and of course I use sql query’s so much I can they are always much faster.