If you have not heard of web scrapping, it is the term used to extract data from websites. You may have visited a website and they kindly provide data tables for you to download. On other occasions, you may have thought. As you manually copied and pasted the data into your database or spreadsheet there must be a better way.

Warning: Before extracting any data, check you won't be breaking any copyright infringement rules. 

Fortunately, there is an array of software tools available. 

Sequentum (desktop or server) aimed at the enterprise market. Pricing you need to contact Sequentum. I have no experience with this software. The fact they will not provide pricing on their website suggests it is not for everyone. 

Dexio is a suite of software to meet all your extraction transform, load (ETL) needs. Again, frightened to provide pricing and I have no experience of using Dexio. 

UniPath provides robotic (algorithm) automation  process software. An example job candidate submits their resume, a robot automates resume data extraction. Most end-users should be able to use this software. 

Web Scraper is a free browser extension with paid options. To collect web data, it has a point and clicks interface. They call this a scraper. Once you have configured a scraper and ran your scraper you can download the data in CSV format. 

To use either R or Python you would have to invest time learning how. Without question R and Python are the most robust solutions but not for everyone. 

Web scraping with Power BI, Power Query

As an example, let’s suppose I wanted to extract data from Amazon. I seek books to do with ‘collect combine and transform data.’ I require the book title. Author, the number of reviews, average rating and published date.

Open Microsoft Power BI. Click the Get Data icon to reveal the drop-down and select web. 

You should see the next window. Click ‘Add Table Using Examples.’

You should see the example window below. 

Tip: Maximise the window.

In the top half of the window you should be able to scroll through the list to view the available books.

In the bottom half of the window. On the left you should see a table with one column. Named ‘column 1.’ Double click the ‘column 1.’ Rename the column to ‘Title.’ Below the title row click the empty row. Type the first few letters of the book title. In this case ‘Learn Microsoft PowerApps.’ You should not need to type as many letters. Before the title becomes obvious to you. Click book title to add to the column. Add the next book title. On rare occasions you may need to add another book title. To have the column fully populate with webpage book titles.

On the right next to ‘Title’ column is a thin column with an asterisk. Click to add another column. Name the column ‘Author.’ Repeat the previous steps you did to add book title but this time author name. Repeat the same steps to add ‘Average Rating’ and ‘Release Date.’ 

You should find the table rows self-populate. 

Once you completed click ‘OK.’ 

Power BI should open with the table populated. If you do not see the table. On the left hand of the screen you should see three icons. Click the table icon see below. 

You can now manipulate the data to suit your needs see below. 

Conclusion

Power Query offers you a lot of flexibility and for many Power Query is all they will ever need. 

Predictive modelling seeks to answer the question. 

“Based on known past behaviour, what is the most likely thing to happen in the future?

How is predictive modelling helpful?

Predictive modelling applies statistics to predict outcomes. It is a process that uses data mining and probability to forecast outcomes.

Models are made up of several predictors.

These are variables that are likely to influence future events. For example when and where a crime is likely to occur.

The BBC Horizon documentary ‘The Age of Big Data’ is good example. Unfortunately, no longer available on BBC i-player but you can view the program on Vimeo. 

Modelling can be applied to any type of unknown event, regardless of when it occurred. For example, predictive models are often used to detect crimes and identify suspects. Before the crime has taken place. 

Predictive modelling example applied to marketing.

Predictive modelling also called predictive analytics. Analytics is used by marketers to take the trial and error out of their marketing campaigns. Models seek to predict which actions and strategies are more likely to succeed.

Predictive analytics forecasts outcomes by pairing historical data with analytical algorithms. Businesses use software to track marketing efforts. Such as calls, clicks, web leads and more. The data is collected and added to the model.

The model seeks to account for various elements of a marketing campaign. Such as demographics time and location. Marketers can adjust different variables. For example when and where they are running their ads. Based on these adjustments the software generates predicted outcomes. Findings allowing marketers to see how campaign changes can impact their marketing outcomes. 

Predictive modelling gives marketers an advantage.

By forecasting data driven outcomes exposing blind spots and highlighting potential revenue streams. Marketers can feel just a little more confident about their latest campaign strategy.