AI eliminates camera distractions to enhance live soccer broadcasts

While the sports industry is continuously improving the at-home viewing experience for spectators, some problems remain. One issue for soccer fans is camera operators accidentally appearing in each other's shots during live broadcasts. Not only can this ruin key moments of the match, but it can also lead to revenue losses for broadcasters due to viewer dissatisfaction.

To address this, researchers at Kaunas University of Technology (KTU) developed an end-to-end system that improves the viewing experience by eliminating visual disruptions caused by overlapping camera angles.

“Our new invention is an algorithm adapted to detect video operators,” says KTU professor Rytis Maskeliūnas, one of the creators of the innovation.

Serhiy Postpayev, another member of the research team, noted that due to the complexity of live sports coverage and the large number of cameras installed around the stadium, cameramen being caught in the frame is a frequent issue in soccer broadcasts.

“At prestigious tournaments, the number of camera views can start at nine, and the large number of overlapping views contributes to the problem of visual clutter. These issues severely constrain camera teams, as they have to film the match while constantly trying to avoid each other, which can lead to the context of some game moments being lost and the dynamic and immersive nature of the broadcast being compromised,” Postpayev explains.

Eliminate visual distractions

To solve this problem and remove unwanted objects during live broadcasts, KTU scientists designed and implemented an end-to-end system.

In terms of functionality, the company employed the YOLOv8 model, a state-of-the-art object detection system known for its speed and accuracy. YOLOv8 stands for “You Only Look Once,” and is able to detect and classify objects in an image in a single pass, making it ideal for real-time events such as live football broadcasts.

“The technique works by dividing the image into a grid and predicting the bounding box, class probability and segmentation polygon for each grid cell, which allows us to identify and segment the cameraman,” says Serhii Postupaiev, a recent KTU graduate with a master's degree in artificial intelligence in computer science.

To train the YOLOv8 model to accurately detect and segment cameramen during soccer matches, we needed to create a dataset.

“I created this dataset to include cameramen of different sizes, shapes, and equipment types, captured in different situations and at different stages of the game. Now, YOLOv8 uses this dataset to identify the cameramen’s positions within the video frames,” Postupaiev adds.

As the inventor explains, this process was necessary in order to create the basis for actually removing the operator: for this purpose, video restoration techniques were used.

“The term inpainting in deep learning refers to the process of reconstructing lost or degraded parts of an image or video. Specifically, in this case, it is used to remove a cameraman from a football video broadcast,” Postpayev said.

Artificial intelligence (AI) and computer vision-based technology analyzes video frames, detects unwanted objects, such as cameramen, and fills the removed areas with relevant background details. The corrected frames are streamed to viewers, making for a more immersive and professional broadcast.

Maskelyunas added that the algorithm allows television servers to process footage recorded a few seconds after the actual moment it was shot before it is broadcast and still be considered live. He believes that as equipment improves, AI will be able to perfectly bridge the gap.

Shifting the focus from simply capturing the action

This new technology will bring many improvements to watching soccer matches at home, one of which will be a smoother viewing experience.

“The broadcast will be more polished and professional, without the confusion caused by cameramen appearing in unnecessary places. This improvement will reduce the number of times important moments of a match are missed because of distracting shots,” emphasizes Postpayev, who earned his master's degree in the project.

According to Postpayev, further research in this area could herald a new era of sports broadcasting, shifting the focus from simply capturing the action to creating a fully immersive, uninterrupted viewing experience.

“Implementing cameraman inpainting will enable broadcasters to explore innovative camera angles, perspectives and effects to bring the game to life in new and exciting ways,” he says.

Additionally, cameraman inpainting can also extend beyond live broadcasts to enhance pre- and post-match analysis, offline highlight reel processing, and archival footage restoration.

“This could also breathe new life into old recordings of classic matches,” the KTU graduate added.

This invention is not limited to soccer but can be applied to other sports with similar broadcasting challenges: dynamic sports such as futsal or basketball that require immersive broadcasting can also benefit from this technology.

“This is another example of what modern AI applications are capable of. We often hear about medical applications, but here we see a consumer-oriented approach to editing out images that we don't like. In the future, such technologies will make it possible to constantly update content with such precision that it is unnoticeable to the human eye, for example by removing advertisements or replacing them with different ones,” said Professor Maskeliunas from KTU's Faculty of Informatics.

The paper, “Real-time Camera Operator Segmentation with YOLOv8 in Soccer Video Broadcasting,” has been published in the journal artificial intelligence.