Huge database

Questions and answers regarding general SQL and backend databases

Huge database

Postby irene.meyer1422261887 » Tue Jan 19, 2016 6:26 pm

Hi,

I would to import a huge amount of csv files (240,000) in Servoy.
Of course there is no chance that I can do this on my internal HD. Is there a way to either crate a database server or install Servoy on an external HD?

Thanks in advance
Irene
User avatar
irene.meyer1422261887
 
Posts: 34
Joined: Mon Jan 26, 2015 10:44 am

Re: Huge database

Postby ROCLASI » Tue Jan 19, 2016 10:38 pm

Hi Irene,

irene.meyer1422261887 wrote:I would to import a huge amount of csv files (240,000) in Servoy.

That is indeed a lot of files. But how much data is this? How much are the files combined in Megabytes?

irene.meyer1422261887 wrote:Of course there is no chance that I can do this on my internal HD. Is there a way to either crate a database server or install Servoy on an external HD?

Why would you say that? For instance here I created 30 millions records that was 3.9GB of data. How much space do you have on your internal HD?
Robert Ivens
SAN Developer / Servoy Valued Professional / Servoy Certified Developer

ROCLASI Software Solutions / JBS Group, Partner
Mastodon: @roclasi
--
ServoyForge - Building Open Source Software.
PostgreSQL - The world's most advanced open source database.
User avatar
ROCLASI
Servoy Expert
 
Posts: 5438
Joined: Thu Oct 02, 2003 9:49 am
Location: Netherlands/Belgium

Re: Huge database

Postby irene.meyer1422261887 » Tue Jan 19, 2016 11:11 pm

Hi Robert,

Thanks for your prompt reply.

The bunch of files are aprox 5 Tb.
Our internal hd is 1 tb but the free space is 450 Gb.

We have a RAID system of 12 Tb attaced to it. so I think it will be enough to transport the data.

My main question is can I install Servoy on the RAID, or how can I instal there the database server and let it communicate with Servoy.

Cheers
Irene
User avatar
irene.meyer1422261887
 
Posts: 34
Joined: Mon Jan 26, 2015 10:44 am

Re: Huge database

Postby david » Tue Jan 19, 2016 11:38 pm

irene.meyer1422261887 wrote:The bunch of files are aprox 5 Tb.
Our internal hd is 1 tb but the free space is 450 Gb.

We have a RAID system of 12 Tb attaced to it. so I think it will be enough to transport the data.


Pretty easy to do with a small amount of NodeJS code using Baby Parse (handles the remote file streaming and csv processing) and a database connector to whatever db flavor you're targeting.
David Workman, Kabootit

Image
Everything you need to build great apps with Servoy
User avatar
david
 
Posts: 1727
Joined: Thu Apr 24, 2003 4:18 pm
Location: Washington, D.C.

Re: Huge database

Postby ROCLASI » Tue Jan 19, 2016 11:55 pm

Hi Irene,

irene.meyer1422261887 wrote:The bunch of files are aprox 5 Tb.

Okay, now we're talking. :)

irene.meyer1422261887 wrote:We have a RAID system of 12 Tb attaced to it. so I think it will be enough to transport the data.

My main question is can I install Servoy on the RAID, or how can I instal there the database server and let it communicate with Servoy.

That depends. What database server do you (want to) use? PostgreSQL? If so do you use the bundled one or the EnterpriseDB installer?
If it's the bundled one you simply install Servoy on that RAID drive and you are set. No extra work needed. If you want to use the EDB installer then it requires a lot more work.
So it depends on what you want to do and/or use.
Robert Ivens
SAN Developer / Servoy Valued Professional / Servoy Certified Developer

ROCLASI Software Solutions / JBS Group, Partner
Mastodon: @roclasi
--
ServoyForge - Building Open Source Software.
PostgreSQL - The world's most advanced open source database.
User avatar
ROCLASI
Servoy Expert
 
Posts: 5438
Joined: Thu Oct 02, 2003 9:49 am
Location: Netherlands/Belgium

Re: Huge database

Postby swingman » Wed Jan 20, 2016 12:57 pm

As you have a large amount of data, you need to import the files directly into the backend database with some kind of script.
Don't try to do it through Servoy unless you really have to -- it will be very, very, very slow...
Christian Batchelor
Certified Servoy Developer
Batchelor Associates Ltd, London, UK
http://www.batchelorassociates.co.uk

http://www.postgresql.org - The world's most advanced open source database.
User avatar
swingman
 
Posts: 1472
Joined: Wed Oct 01, 2003 10:20 am
Location: London

Re: Huge database

Postby ROCLASI » Wed Jan 20, 2016 2:17 pm

Correct.
PostgreSQL's COPY command comes to mind.
Robert Ivens
SAN Developer / Servoy Valued Professional / Servoy Certified Developer

ROCLASI Software Solutions / JBS Group, Partner
Mastodon: @roclasi
--
ServoyForge - Building Open Source Software.
PostgreSQL - The world's most advanced open source database.
User avatar
ROCLASI
Servoy Expert
 
Posts: 5438
Joined: Thu Oct 02, 2003 9:49 am
Location: Netherlands/Belgium

Re: Huge database

Postby irene.meyer1422261887 » Wed Jan 20, 2016 4:14 pm

Hi Robert and Christian,

Thank you very much for the help.
I will install Servoy on the RAID and of course I use sql query's so much I can they are always much faster.
User avatar
irene.meyer1422261887
 
Posts: 34
Joined: Mon Jan 26, 2015 10:44 am


Return to SQL Databases

Who is online

Users browsing this forum: No registered users and 6 guests