Importing webpage requiring a js enabled browser

Questions and answers on designing your Servoy solutions, database modelling and other 'how do I do this' that don't fit in any of the other categories

Importing webpage requiring a js enabled browser

Postby rioba1401913570 » Wed Jul 03, 2019 7:12 pm

I am trying to import the content of pages ( example: https://www.iso.org/obp/ui/#iso:code:3166:IT ) mfrom a site which requires that javascript is enabled in the brower. Also saving directly the webpage from within the browser is only possible using Firefox on a Mac. In Servoy I am using this funcion
Code: Select all
function mt_iso3166_import() {
   var http_iso = plugins.http.createNewHttpClient();
   var isopage = http_iso.createGetRequest(iso3166_baseurl);
   var result = isopage.executeRequest();
   var txt12 = result.getResponseBody();
   plugins.file.writeTXTFile("/Users/rioba/DBFA/2019/ISO-3166/test.txt",txt12);
}

This is the result I get: You have to enable javascript in your browser to use an application built with Vaadin

Code: Select all
<!doctype html>
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=11;chrome=1">
  <style type="text/css">html, body {height:100%;margin:0;}</style>
  <link rel="shortcut icon" type="image/vnd.microsoft.icon" href="./../VAADIN/themes/iso-red/favicon.ico">
  <link rel="icon" type="image/vnd.microsoft.icon" href="./../VAADIN/themes/iso-red/favicon.ico">
</head>
<body scroll="auto" class="v-generated-body">
  <div id="obpui-105541713" class="v-app iso-red obpui">
   <div class="v-app-loading"></div>
   <noscript>
    You have to enable javascript in your browser to use an application built with Vaadin.
   </noscript>
  </div>
  <script type="text/javascript" src="./../VAADIN/vaadinBootstrap.js?v=7.7.9"></script>
  <script type="text/javascript">//<![CDATA[
if (!window.vaadin) alert("Failed to load the bootstrap javascript: ./../VAADIN/vaadinBootstrap.js?v=7.7.9");
vaadin.initApplication("obpui-105541713",{"theme":"iso-red","versionInfo":{"vaadinVersion":"7.7.9"},"widgetset":"org.iso.obp.ui.widgetset.applicationWidgetset","widgetsetReady":true,"comErrMsg":{"caption":"Communication problem","message":"Take note of any unsaved data, and <u>click here</u> or press ESC to continue.","url":null},"authErrMsg":{"caption":"Authentication problem","message":"Take note of any unsaved data, and <u>click here</u> or press ESC to continue.","url":null},"sessExpMsg":{"caption":"Session Expired","message":"Take note of any unsaved data, and <u>click here</u> or press ESC key to continue.","url":null},"vaadinDir":"./../VAADIN/","standalone":true,"heartbeatInterval":300,"serviceUrl":"."});
//]]></script>
</body>
</html>

Any idea on how to copy the contents of the page using Servoy? How can I emulate the behavior of the Firefox save as text menu command? Thank you for any suggestion.
rioba1401913570
 
Posts: 28
Joined: Wed Jun 04, 2014 10:26 pm

Re: Importing webpage requiring a js enabled browser

Postby rgansevles » Fri Jul 05, 2019 9:43 am

This is a dynamic web page, the content is not loaded directly, but created using the javascript in the loaded js file.
You cannot scrape websites like that, you need a browser to run the javascript.

Doesn't have this site a proper REST/SOAP interface to get the info you need?

Rob
Rob Gansevles
Servoy
User avatar
rgansevles
 
Posts: 1927
Joined: Wed Nov 15, 2006 6:17 pm
Location: Amersfoort, NL


Return to Programming with Servoy

Who is online

Users browsing this forum: No registered users and 9 guests