“I go on holiday to change the way I feel…”

Can technology detect how we want to feel, in order to help us make better holiday choices?

We recently embarked on a journey to push the limits of how we as consumers go about searching and choosing holidays. Following our extensive research and customer interviews we identified a key insight that powered the rest of our thinking: we actually choose to go on holiday based on the way we want to feel.

The challenge with delivering on this insight is from one person to the next, we all experience things differently. We have different interpretations of what a busy city vibe feels like. A beach experience may be perceived as adventurous, to others it could be peaceful.

So why do all travel businesses structure the online holiday discovery journey in the same way? They use the same categorisation of holidays which assumes we all interpret them in the same way. This, coupled with decision paralysis of having to pick 1000’s of options across 100’s of brands, means we end up painfully trawling through countless websites, day after day.

To find a better way to solve this pain point, we jumped into creating and experimenting with a new concept, to see what the future search experience could look and feel like, if the experience of discovering holidays was driven by feelings.

→ Enter Gini.

Can Gini work in real life?

While we explored at how brands could move towards a Gini-like approach over time right here, for this article we looked at the concept’s validity with our tech heads on.

Overall tech approach

The key to an online experience like this is the fluidity of keeping the user engaged, through to taking the desired action you want as a business. To allow for smooth interactions, animations, complex media interaction along with external data integration, a single page application using frameworks like React or Vue.js would deliver this experience. A single page application (SPA) using these frameworks allows for flexibility in complex interactions with the page, whilst making integration with different media formats (such as Gini’s soundscapes) and data sources (holiday data) efficient. 

Why an SPA?

We wanted to make sure Gini feels like one experience, rather than several pages or steps.  It was one of the key requirements we had for this concept - to retain the fluidity of the experience as you navigate through. SPAs are great at enabling this. Thinking of Gini as an SPA meant the user experience was carefully considered, as Gini hones the buildup of anticipation, as you start reacting to sound, video and images that trigger the feeling you’re after.

Sensory experiences

To draw out from people how they want to feel when on holiday, we started looking into sensory elements that will help the user immerse deeper into the atmosphere of their perfect holiday. Giving people micro control to adjust sound, video and image levels to their personal taste would then give a more realistic feeling of the holiday they truly wanted.

Gini uses soundscapes to create your perfect holiday atmosphere.

3D immersive sound

The first of the sensory elements was sound. The user is presented with the ability to toggle various sounds they would expect to hear on different holidays around the World. Turning up or down the distant chatter of people, the sound of gentle waves, or a boat on the water going past... 

To ensure the experience would trigger a strong emotional reaction, we wanted to go to another level of realism. So we scoped out 3D sound using binaural audio to really transport you into that environment. Binaural audio has been around since the 1980’s, however it's only in recent years that it's taken off as a cost effective way to draw a user into a virtual experience.

A binaural microphone, source Headrec Audio

It works by capturing sound using a binaural microphone, which is shaped like a human ear. The beauty of this is in the way the technology replicates how sound waves bounce in and around the human ear. Once recorded, any budget headphones can be used to listen and experience a high level of spatial realism.

Here's a clip from a BBC Click episode where they refer to binaural audio as "VR for your ears". We recommend wearing headphones:

Device orientation detection (but for sound)

Mobile web app technology has dramatically improved over recent years to give functionality that used to only be available through native apps, now into web apps. Many native features of your phone such as accessing the camera, the orientation of the device, gesture control, can now all be used within a web experience. This means you can build rich web experiences without relying on native apps.

To draw the user further into Gini, we started looking into how this functionality could be used to offer a deeper experience for the user. One area was the mobile device orientation API. This works by detecting the angle and direction of your phone in real-time. Using this data we allowed the user to move around whilst playing the sound effects to alter where the sounds were virtually coming from. Gentle waves in front of you one minute, turn around and you hear music playing in front of the beach. 

So we have an immersive sound experience to draw you into your ideal holiday experience, triggering that feeling of being there, but what else could we do to detect how you want to feel and the things that bring you positive emotions?

Emotion detection

How we feel is triggered by emotions, so how can we measure them? Machine learning (ML) has been able to do this for some time now, by measuring your subtle facial expressions and more recently the tone in your voice. Taking this capability of ML, we looked at our concept and began exploring how we could gather further insight into the users’ feelings by tracking their emotional reaction to holiday scenes through rich media. 

Here's me training an algorithm to detect my emotional state based on facial reactions.

Firstly, we needed to build a dataset to train an ML model to detect facial emotion reactions... Grabbing who we could in the studio, we started exploring how accurate this technology is, and to our surprise, once we had enough data tracking how different people express emotions, it felt like a solid contender to drive our emotion-led holiday choice experience.

Applying emotion detection to Gini’s algorithm

We took this technology and designed an experience that showed the user a series of video scenes, tracking the person’s reaction to how they are feeling whilst looking at them. Using your webcam, our ML model could then track their reaction when seeing different scenes, and then filter out the best experiences that the model thinks will be most appealing to them.

In short, we incorporated a “smile detector” within Gini. Powerful emotions create physical changes, and an involuntary smile will help identify the most delightful triggers and account for them accordingly later.

Just for fun: How else could Gini use emotion to find your perfect holiday?

Here are a few examples of technology already on the market that could elevate Gini’s emotion-led decisions even further. We didn’t take Gini this far, but we could just about imagine it: 

Voice analysis like this (currently used to detect health conditions) could be used to determine mood and likes/dislikes.

If Gini was to be delivered via VR or AR, its accuracy in recognising and learning how a person feels could also be done through scientifically objective measures - brainwaves and heart rate. Brainwaves could be captured through an EEG headband. Heart rate via tech like this.

Of course, taking Gini to this level for the sake of a holiday feels rather insane, but what's science without experiments or crazy ideas?...

Gini in the context of the travel industry

The next step was going to be taking this data and seeing how it could be matched with the extensive holiday provider listings.

The biggest hurdle was that Gini’s model of holiday categorisation is based on a person’s feelings. All external holiday listing data from providers is of course categorised by type (beach, city break, adventure). So how could we create a matching capability that's scalable?

Gini’s biggest hurdle? Holiday data matching

The answer to Gini’s scalability is in data. So is its biggest hurdle. There’s loads of data to process and match. Despite that, there is also some data missing. Think of the soundscapes example, and one of Gini’s most immersive features. A person is likely to want to book a getaway to a place that has some fidelity to the audio experience created during that phase. Now imagine getting there and realising it’s nothing like it. While assumptions can be made on whether a particular destination might feature some sounds (a hotel right by the sea will surely have the sound of waves as part of the sensory experience), how can we make sure there is sufficient data to feed into the holiday choice algorithm? 

Sound aside, there is plenty of data going around. Many holiday providers offer API’s that allow other businesses to use their data to promote their offers through different channels. Getting access to this data is relatively simple via API’s. You can use any map search service to make sure your favourite cuisine is nearby. You can use image recognition to find suitable matches.The bigger challenge is creating a new capability to enhance the incoming data with our additional “feelings” tags, and use this as a true differentiator in how the future of holiday search might happen. Like we identified in the initial pain point, what is relaxing to you, is stressful to others. We scoped out how a machine learning model could be trained over time to enhance the incoming holiday data by understanding images, videos and sentiment in text with how our user wants to feel on holiday, offering people their ideal holiday options. 

Over time, people will truly feel like Gini truly gets them, their emotions and how they want to feel on holiday.

It’s a reimagined travel search experience.

Liked this thinking? See Gini's full story.

🧠

Feed your brain with more stories like this one

Subscribe to receive interesting and insightful links shared on our studio Slack channel every couple of weeks 🤓

🚀 Sign up to Brainfood