About the Project
FascinatE is a €9.5m EU funded project involving a group of partners from across Europe. FascinatE stands for: Format-Agnostic SCript-based INterAcTive Experience and is looking at broadcasting live events to give the viewer a more interactive experience no matter what device they are using the view the broadcast.
The FascinatE project will develop a system to allow end-users to interactively view and navigate around an ultra-high resolution video panorama showing a live event, with the accompanying audio automatically changing to match the selected view. The output will be adapted to their particular kind of device, covering anything from a mobile handset to an immersive panoramic display. At the production side, this requires the development of new audio and video capture systems, and scripting systems to control the shot framing options presented to the viewer. Intelligent networks with processing components will be needed to repurpose the content to suit different device types and framing selections, and user terminals supporting innovative interaction methods will be needed to allow viewers to control and display the content.
Enhancing a live event
In 2013 Jim is at the final concert of the Rolling Stones, and is listening entranced by his favourite music. In the stadium the concert is being recorded with a cluster of fixed cameras to record a high-resolution panoramic view, with additional detail being added to key areas of interest from the adjacent manned HD broadcast cameras. Jim uses his mobile to connect to the FascinateURmobile service. After a connection has been established, his mobile initially shows the picture from its own camera on its screen. He points the phone camera to the stage and selects the drummer to be in the centre of his picture. He presses the OK button and the picture is replaced by a high quality close-up live stream of the drummer, as recorded by the camera system and repurposed for mobile usage. Jim can see the wrinkles of Charlie Watts and is very happy. He pans a bit by using his touch screen and watches Charlie do his thing. After a while he gets bored and selects Mick Jagger in the same manner. He presses the button ‘Follow me’ on his screen to make sure Mick will not walk out of the viewing frame on the mobile, as he jumps up and down on the stage. When the concert has finished, the FascinateURmobile service informs Jim that an edited version of the concert is available. Jim watches it on his mobile during his trip home, just to enjoy the concert again. But he is disappointed with what the directors have selected for scene cuts and framing. So he activates the free navigation mode to get access to the whole ‘database’ and navigates freely (in both time and viewing window) to watch his favourite parts of the concert again.
Enhancing home viewing
Jim has returned home from his business trip to Barcelona, and decides to watch the UEFA match Ajax – Barcelona on his 60-inch LCD set. He selects the FascinateURtv service and the current match. The match has already started but it is still 0-0. Jim points his remote at the screen, ‘grabs’ a player with his remote and drags him to the top of the screen. After ‘releasing’ the player, the top left corner of his screen shows Thierry Henry in close up, always from the best possible camera angle. In the same manner he selects the two goals, but now he chooses to watch always from the back of the goal, to have a good view on the 16-metre area. The match is halfway through when Jim’s daughter comes in and starts making gestures to the TV. She is using the gesture recognition system to try to change the channel to MTV, to watch her favourite show. The TV set refuses since Jim has control over the set. Jim sighs and allows the TV to change channels. He moves to another room with a small TV set. The screen is much smaller, but here he can at least watch the game without being disturbed. But he notices that, after watching the game on such large screen for a long time, his preferences are different from those of the director’s default scripting for a small screen. So he changes the mode and navigates freely in the scene to follow his favourite Thierry Henry again or to select more wide shots than the default setting gives.
To implement a system to achieve these aims requires a paradigm shift in video production, towards capturing a format-agnostic representation of the whole scene from a given viewpoint, rather than the view selected by a cameraman based on assumptions about the viewer’s screen size and interests. It should be noted that we are not proposing to capture a 3D representation of the scene, nor to support true ‘free viewpoint’ rendering. Such systems, in the experience of the project partners, will be incapable of presenting really high-quality images for the foreseeable future, so we are concentrating on an approach that will deliver true high-quality images, whilst allowing a significant degree of interactivity, and which is practical within the time frame of the project.
By ‘format-agnostic’ we mean that the resolution, field-of-view, aspect ratio, frame rate and colour depth of the captured image are chosen based on the requirements of the particular production, rather than being tailored to a particular delivery format. Indeed, there will be no single camera with a given set of parameters covering the scene; rather, different parts of the scene will be captured with different types of camera, clustered around one or more viewpoints.