Relevo, a Spanish sports news website, known for its innovative mobile home page that emulates the look and feel of TikTok or Instagram’s reels, was launched in October 2022. Though it aims to build a cross-over audience of sports fans, it is focused on Gen Z and Millennials, who feel disengaged or disillusioned with traditional sports coverage.
To make the most of the innovative interface and measure success, the newsroom introduced a new ad-hoc metric and AI-supported software coding. This improved the homepage engagement by a third in just two months.
The project also proved that non-coders in the team were able to effectively use generative AI to create basic software tools to meet the challenges of a data-driven digital newsroom.
Since Relevo’s launch, team members have actively discussed the ideal length of headlines for the homepage. The newsroom has gravitated towards longer titles, of 100 characters or more. The product team felt this put too much cognitive load on users, contradictory to the fast-swiping appeal of the interface. A counter-argument quickly emerged: Fast swiping powered by shorter headlines could discourage users to click on headlines, which in turn could damage the most sacred metric of all: pageviews.
And so, the four-person team turned its attention to the target: What was the ideal use case they envisioned for their new product? Only then could they meaningfully discuss metrics, said Germán Frassa, Digital Strategy Director.
Creating the Pringles metric
The use case the newsroom agreed on was one of a user navigating everyday to the Relevo homepage on their mobile phones, swiping until they find a headline that grabs their attention, clicking to read the article, going back, swiping again, then clicking, and so on. In doing this exercise, the team understood that “swipes vs. clicks” was a false dilemma. They go hand-in-hand. One leads to the other.
There is no out-of-the-box unit to measure both swiping and clicking, so Relevo crafted its own hybrid metric by combining swipes and clicks. They called it “slick” (swipe + click).
The formula is: (swipes/visit) 10 + (clicks/visit) 100
The team modelled it with the editors in mind, so it had more to do with cognitive psychology than hard math. To make the figure bigger, rounded and easier to recall, they multiply both swipes and clicks. In the case of clicks, the factor is bigger because they are less frequent than swipes, and they wanted the two components to be roughly similar in magnitude.
This case study is part of a deeper research report titled “AI in Action: 10 Global Use Cases.” Download it here.
Using AI to measure headlines
The goal of the project was to break the loop of the argument between the newsroom and the product teams by:
- Defining a new metric (the aforementioned “slick”) that could properly frame the desired user behaviour Relevo aims for with this new home page interface.
- Using this metric to formulate a hypothesis about headline length: the shorter the headlines, the more the slicks.
- Testing this hypothesis to agree on the best way to increase user engagement, for which the use of AI to hack the software needed was crucial.
Calculating the new metric was straightforward. They used Adobe Analytics to define their metrics. However, measuring the headlines was a different thing. “Our web developers were fully booked with other initiatives, and we felt that we needed to be very agile in developing a solution to engage all parties and capture their attention while the issue was still being discussed,” said Frassa.
By mid-2023, the team had been using AI tools for a while on an individual level to provide support in various editorial and product tasks, such as summarising, translating, defining use cases. “But we knew that generative AI is also quite good at writing software code. And, as in many newsrooms, a few people in our team has a basic programming knowledge,” he noted.
And so: What if they used AI to code a scraping tool to extract and measure headlines from the homepage?
In July 2023 the team started measuring the average character length of headlines. We used ChatGPT Plus to develop a Python script that scrapes the home page every 15 minutes, measures headlines, and saves the data to a csv file.
The typical value for the first five months of measuring was 90 characters per headline. The trend was increasing by about one or two characters per month, with a minimum of 87 in July and a peak of 93 in November. (Words tend to be longer in Spanish than in English.)
Meanwhile, the slick index for the period was 119. The best mark (132) was recorded in September, when Relevo jumped to international relevance thanks to scoops in the coverage of the “Rubiales case”.
“We retroactively calculated the slick index down to launch date, and discovered it was even lower: only 93, on average, during the first half of 2023. And although we didn’t have headline length data prior to July, we all had the impression that it was even longer,” said Frassa.
Making data visible
Here, again, the sensible hypothesis emerged: The shorter the headline, the more the slicks.
But data needs to be visible to be acted upon, and Relevo wanted its editors to be aware of these figures. How could they this without having to set up any infrastructure? A Chrome browser extension was the answer. They turned to ChatGPT, and quickly learned that it was proficient at crafting Chrome extensions.
The extension (code-named “Pringles” because Relevo wanted its headlines to be small and of equal shape, as the famous potato chips) injects a purple box into the page with current data about headlines, including character length. The team also created a mobile dashboard in Adobe Analytics to put the data on slicks at hand for everyone.
With both AI-powered tools up and running, an editorial policy to post shorter headlines was enacted in December 2023.
Outcomes and lessons learned
Three months after the launch, the figures seemed to confirm the hypothesis: As headlines became shorter, the slick index improved to 142 from 107 in 2023 – a 32.7% increase. On a scale from -1 to 1, there is a Pearson Correlation Coefficient of -0.7 between headline length and slicks. “Although statisticians always remind us that correlation does not imply causation, with no other evident variables at play, the trend strongly suggests the shorter the headline, the better the engagement,” Frassa said.
Standard metrics have also significantly improved, with a 19% increment in visits to the homepage per visitor, and 26% more home page views per visitor.
Frassa summarised their lessons learned:
- Metrics are not an end, but just a proxy to verify human interaction with digital products.
- It’s okay to craft your own measurement to make sure you are hitting your target.
- AI is a powerful tool to support hybrid roles across the newsroom and product teams.
- People with basic programming skills can become fully-fledged software developers with AI help.
- This new “superpower” can help the non-technical side of the team to quickly set up small and mid-sized software-based initiatives without distracting engineering resources.
