The importance of data technology in delivering accurate hotel rate shopping

With the evolving distribution landscape changing the relationship between OTAs and hotels, the data used by revenue managers to make informed decisions is now more important than ever. Rate shopping tools provide hoteliers with knowledge on their overall market and competitor pricing; insights that are relied on daily. With pricing strategy changing sometimes hour to hour to reflect local events and demand, this requires the data that powers rate intelligence tools to be accurate and up to date.

Big Data, in particular, continues to expand within hospitality as hotels and OTAs alike seek to use guest data to fuel traveller segmentation and personalisation, especially as the distribution landscape changes and direct bookings become more crucial to hoteliers. Big Data is also important in providing key revenue management insights by allowing for predictive analysis using past occupancy rates, PMS data and various other KPIs.

OTA Insight is well versed in Big Data, with over one billion data points processed every day. With so much data needed for revenue management and OTA Insight’s role in helping hoteliers to visualise and leverage the insights, it’s crucial that the data we provide is the industry’s most accurate and reliable.

We’re able to do this by using a multi-source strategy that combines APIs and state of the art crawler technologies. Our crawler technology is used to deliver accurate rates as they are displayed live online, while our API integrations, across a broad range of brand.com and OTAs, provide efficient rate shop data delivered through a unified interface.

Data with a difference

Our effective strategy is implemented by our award-winning engineering team, who last month took their turn to host a Tech Meet-Up focusing on data processing on the Google Cloud Platform.

The event attracted over 60 attendees, one of our largest gatherings to date, who descended on our engineering hub in Ghent, Belgium, to catch up in a relaxed and informal setting before getting down to discussions. As well as a presentation from OTA Insight, we were joined by Sven Degroote, an engineer from machine learning provider ML6, who spoke about video analytics with Apache Beam.

OTA Insight CTO Joeri De Turck’s session showcased our rate shopping tool, describing its functionality and detailing how the data is collected, transformed, and stored, before being aggregated and presented visually in the finished product; shown in the image below.

Our method of processing data has a significant impact on the tools OTA Insight provides. Joeri explains: “The way that we process our data is what makes our data quality so good. The live shops wouldn’t be possible if you didn’t have an efficient and quick streaming pipeline. It gives us flexibility in the way we do shops and helps us with our data quality checks so we can reprocess certain parts of the data quickly and easily to proactively handle changes on websites.”

The streaming pipeline is indeed critical to our process, and to ensure data completeness we employ an industry-leading follow-up system for our whole data retrievement pipeline. Joeri expanded on our pipeline developments during his session, detailing the move in 2017 from our old data processing pipeline where we had been using the Apache Kafka queue that was executed using a processing framework built on Apache Spark. Our new pipeline involved a switch to Google’s managed Cloud Pub/Sub using Google Kubernetes. OTA Insight was an early adopter of Kubernetes 4 years ago for our APIs. The move allowed us to have 1 infrastructure for both APIs and data processing using the same tools and deployments; making it cost-efficient and easier for developers. As well as handling application deployment, Kubernetes is a powerful open-source “container orchestration system” for automating scaling and management. Scalability is an important consideration when processing the volume of data OTA Insight accesses every day.

OTA Insight streams data through very small pieces in order to scale out and do thousands of data points at the same time instead of one by one. By transforming data piece by piece, the time between shopping the data and it then appearing on the dashboard is just a few seconds. Joeri explains, “It’s also essential to store it in a database that’s scalable and made for our kind of workload, which is why we use Bigtable, the same database that powers Google Search. Bigtable is made to store substantial amounts of data and offers an efficient way to query large ranges of data.”

He continues: “Data streaming is what enables our tools to be so interactive. We don’t pre-aggregate things, and that allows users to have a flexible view of their data, such as the dynamic room-type mapping and meal selections. All those things are made possible because of how we process data and how we store it.”

Ahead of the curve

OTA Insight has always benefitted from an expert tech stack and we can count ourselves as one of the earliest adopters of the Google Cloud Platform. Director of Engineering Mathias Verhoeven, who earlier this year blogged about our rate shopping reinventions, comments that “we’re super proud of our technology choices because a few years afterwards you see how they have grown. For example, Google Cloud has advanced to be such a well-known tool, and we made that choice five years ago.”

Go is another tool that’s key to our technology approach. Mathias explains: “One of the things that makes OTA Insight unique on a technical level is that we collect over 1 billion data points a day. We store them in a database that already contains trillions of data points, but it’s all unaggregated. A lot of competitors just collect data, process it and then store it, whereas we store it and then process it at the point at which the client needs it.”

“This makes our dashboard super-flexible because we can combine any data points. We started using Go three years ago and it allows us to do data combinations, filters and aggregations on the fly, which made it one of our core technologies.”

Principles of perspective 

Since evolving from a start-up created by 3 friends in 2012, to growing quickly to become an essential SaaS company, some may wonder how we retain a fresh view and perspective on data. Joeri feels this is achieved by “keeping up with what’s new from big players like Google, Amazon and Microsoft. But it’s not about using something new because it’s new, it’s about being aware of all the new things out there and verifying if they fit certain use cases that we have and then testing them.”

Mathias adds that this perspective comes from the engineering team itself: “Part of it is the culture. We hire people who have an open mind and that keeps the ball rolling. We’re still looking for new people with a curious mindset who are able to make us question things. That’s a unique aspect here: you can actually challenge the CEO and he will gladly accept!”

One thing we can’t be challenged on is providing reliable and accurate data that gives hoteliers the full picture. For more on how you can use the high-quality data we provide for pricing strategy, source markets and distribution, or to tackle rate parity, visit our resources centre.