{"id":1807,"date":"2024-04-16T14:36:47","date_gmt":"2024-04-16T12:36:47","guid":{"rendered":"https:\/\/dah001.staging.360vier.net\/sucess-stories\/webcrawling-dashboard\/"},"modified":"2024-05-07T10:08:36","modified_gmt":"2024-05-07T08:08:36","slug":"webcrawling-dashboard","status":"publish","type":"sucess-stories","link":"https:\/\/www.datahouse.ch\/en\/sucess-stories\/webcrawling-dashboard\/","title":{"rendered":"Webcrawling Dashboard"},"content":{"rendered":"\n
\n
\n
<\/div>\n\n\n\n
\n

SENS<\/a>\u00a0wants to track the reuse of electrical appliances in Switz\u00aderland in order to implement potential measures and decisions based on data. Just a few weeks after the kick-off, SENS was able to examine the first data sets on a customized analysis dashboard.<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
<\/div>\n\n\n\n
\n
\"\"<\/figure><\/div>\n<\/div>\n\n\n\n
<\/div>\n<\/div>\n\n\n\n
\n
<\/div>\n\n\n\n
\n

Customer need<\/strong><\/p>\n\n\n\n

SENS<\/a>\u00a0promotes the sustainable recycling of electrical and electronic equipment. In addition to recycling, the reuse of equipment is also of interest to SENS. In order to promote circular economy measures in a targeted manner and thus conserve valuable resources, our customer requires infor\u00admation on products as well as on regional and seasonal volume trends on the secondary market. SENS would therefore like to identify and analyze infor\u00admation on the reuse of electrical and electronic equipment.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Project procedure<\/strong><\/p>\n\n\n\n

For this project, we chose an iterative approach in order to be able to react as flexibly as possible to new findings. We developed the corre\u00adsponding functio\u00adnality in relatively short iterations, which was then presented and discussed at a review meeting. Based on the insights gained, the planning for the next iteration was created in each case.<\/p>\n\n\n\n

At the kickoff meeting, we reviewed possible data sources with SENS. Based on their needs and technical feasi\u00adbility, we selected a suitable trading platform as the primary source. In a first iteration, we developed the webcrawler within two weeks to retrieve the relevant data regularly and store it processed in a suitable database. In the subsequent review meeting, we discussed the data quality and data prepa\u00adration with SENS. We also outlined various options for the dashboard based on the analysis needs.<\/p>\n\n\n\n

SENS plans to conduct explo\u00adratory data analyses. For this, we chose a flexible and inter\u00adactive approach for the dashboard so that emerging issues could be easily explored. During the next iteration, we developed the dashboard within three weeks in an ongoing exchange with SENS, who already received test access to follow the development. In the subsequent review meeting, we discussed and prioritized final optimization approaches. In a final development iteration, the web crawler and the dashboard were finalized within a few weeks.<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
<\/div>\n\n\n\n
\n

\u201eDatahouse perfectly met my needs, kept me informed about the development status and always took my wishes seriously.\u201c<\/p>\n\n\n\n

Pasqual Zopp, Stv. Gesch\u00e4ftsf\u00fchrer, Manager Back Office<\/p>\n\n\n\n

<\/div>\n\n\n\n

End produkt<\/strong><\/p>\n\n\n\n

SENS has exclusive access to a customized analysis dashboard. On this, SENS can analyze all collected data and export it as desired. Datahouse ensures the operation of the dashboard and the web crawler in the background and conti\u00adnuously monitors the data quality from the automated regular queries.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Techno\u00adlogies used<\/strong><\/p>\n\n\n\n

The dashboard was developed using the R Shiny framework. For webcrawling we used Python Scrapy and our proven special infra\u00adstructure for coordi\u00adnation and monitoring. All data is stored in a MongoDB database.<\/p>\n\n\n\n

<\/div>\n\n\n\n
<\/div>\n\n\n\n

More<\/strong><\/p>\n\n\n\n

Are you facing similar challenges or do you have a similar project that you need help with? Then don\u2019t hesitate to contact our Senior Expert Data Scientist\u00a0Thomas Maier<\/a>.<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Inter\u00adactive dashboard for analysis of crawling results<\/p>\n","protected":false},"author":13,"featured_media":1806,"template":"","sucess-stories-category":[60,61,62],"class_list":["post-1807","sucess-stories","type-sucess-stories","status-publish","has-post-thumbnail","hentry","sucess-stories-category-data-science-en","sucess-stories-category-diverse-en","sucess-stories-category-success-stories"],"acf":[],"image":{"url":"https:\/\/www.datahouse.ch\/wp-content\/uploads\/refcase-sens.webp"},"_links":{"self":[{"href":"https:\/\/www.datahouse.ch\/en\/wp-json\/wp\/v2\/sucess-stories\/1807","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.datahouse.ch\/en\/wp-json\/wp\/v2\/sucess-stories"}],"about":[{"href":"https:\/\/www.datahouse.ch\/en\/wp-json\/wp\/v2\/types\/sucess-stories"}],"author":[{"embeddable":true,"href":"https:\/\/www.datahouse.ch\/en\/wp-json\/wp\/v2\/users\/13"}],"version-history":[{"count":2,"href":"https:\/\/www.datahouse.ch\/en\/wp-json\/wp\/v2\/sucess-stories\/1807\/revisions"}],"predecessor-version":[{"id":1870,"href":"https:\/\/www.datahouse.ch\/en\/wp-json\/wp\/v2\/sucess-stories\/1807\/revisions\/1870"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.datahouse.ch\/en\/wp-json\/wp\/v2\/media\/1806"}],"wp:attachment":[{"href":"https:\/\/www.datahouse.ch\/en\/wp-json\/wp\/v2\/media?parent=1807"}],"wp:term":[{"taxonomy":"sucess-stories-category","embeddable":true,"href":"https:\/\/www.datahouse.ch\/en\/wp-json\/wp\/v2\/sucess-stories-category?post=1807"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}