Recently I have heard many people talking about the wonders of scraping websites and how it will solve all of our data gathering problems. And unfortunately, I must disagree. But before delving into why I believe web scraping is a flawed practice, it is important to truly understand what it is.

While web scraping can be done manually by a software user, the term typically refers to automated processes implemented by websites, data aggregators, and others using a bot or web crawler. Essentially, it is a form of copying, in which, specific data is gathered and copied from the web - typically into a central local database or spreadsheet - for later retrieval or analysis.

I could go into further detail about all of the technical complications of web scraping and how it is a behemoth task for software developers. Still, those details are not imperative to the implications it has. Let's talk about why web scraping is not the panacea it is positioned to be.

Limited unit availability
When listing apartment homes on an ILS, operators may deliberately withhold the number of units that appear online. This may be due to a limited number of units available at the property or, in the case of new development properties, the operator may want to create the appearance of low supply. Either way, if you are relying on web scraping, you are only getting the data that properties are making available.

Leasing Statistics
Leasing statistics, such as occupancy, leased, and availability percentage is not available on any website and therefore cannot be scraped or copied. Moreover, you can forget about finding out how much traffic or leases a property had the prior week. These are property-level details that play a significant role in property performance. Without this knowledge, decisions are being made based on inaccuracies and limited visibility.

Pricing Clarity
It is nearly impossible to tell if rents on an ILS or property website are market rents or lease rents. Some operators even have a call to action that suggests the visitor "Call for Pricing," meaning their pricing on the website will not give the whole picture. The only way to get actual pricing from web scraping is to know that the property has a daily Revenue Management pricing feed. Otherwise, you may miss out on concessions offered.

This is simply a small window into the world of web scraping. And while it may seem like an easy, automated way to procure your data, the gaps in information cannot be ignored. The longer they are ignored, the more negative impact it will have on your performance.