VP BASELINE
The NEXTGEM project rests upon insights from prior projects, a long list of previous research results, field testing and ongoing dialogue with industry leaders in the Virtual Production (VP) sectors. From these sources the project partners have established a shared baseline.
An ideal VP workflow to frame the EMDM process, as well as inform and inspire future programme applicants.
ideation
Once use of VP is decided, the steps towards a script defines how actions play out. Concurrently, the set design specifies how the physical and virtual props interact. At this stage, art and tech ideas must align.
conception
This initial phase involves defining the project’s vision, story, and desired aesthetic. It includes establishing the overall look and feel, identifying key elements, and creating initial storyboards or animatics to visualise the virtual environments and camera movements.
development
This stage focuses on pre-visualization (previs), creating detailed 3D models and environments. This involves designing virtual sets, creating digital assets, and planning camera shots and lighting setups. Technical planning, including equipment selection and workflow design, occurs in this phase.
finalisation
This phase refines the virtual environments and assets based on the previs and feedback. Any necessary adjustments to the virtual sets, lighting, or camera plans are made. Detailed planning for the shoot, including actor blocking and technical rehearsals, is completed.
execution
This phase involves the actual filming on set. Actors perform within the virtual environments displayed on LED walls or green screens, displaying the real-time rendered imagery. The VP supervisor and tech. team ensure technology is working correctly, and the director guides the performance and camera work.
follow up
After the recording is done, any further VFX intentions are evaluated, and footage is edited. Once shots are assembled, various VFX software is used for further integration and tweaking of elements’ visual matching.
click to go
click to go
click to go
click to go
click to go
click to go
ideation
Any Film/TV project looking to use Virtual Production (VP) may consider how it could be used – while developing the basic storytelling elements. Whether VP is used for artistic or practical reasons, ideas for its use can be an inspiring part of the creative process as the storyline evolves via Synopsis, Treatment and Script.
However, an important evaluation is how VP ideation may affect the writing process. In some cases, initial artistic flow can be disrupted by early introduction to design- centric ideas. In other cases, awareness of creative opportunities benefits the writing.
Each step in the process of developing the story can prompt VP related ideas. To capture and articulate those ideas, roles across storytelling and VP should exchange thoughts and insights
Mobilising team members, with complementary skills, for joint ideation sessions, is effective for rapid development, but simply sharing story points and visuals for feedback may suffice. A collective ideation process can be done in several ways – whichever works best depends on the time, and resources available. There is no set configuration but looping in Script, Direction, Cinematography, Production Design, and VP supervision, warrants a solid ideation.
During early ideation, a core team—Director, Producer, Cinematographer, VP Supervisor, and Production Designer—collaborates on ideation. They explore ways to enhance storytelling with virtual production, using simple references or mockups to visualise ideas without focusing on detailed execution yet. This stage prioritises creative exploration within practical limits, establishing a foundation for the project’s next phases. The small team enables focused decision-making and minimises communication delays, ensuring streamlined development for the project’s future stages.
In Ideation, planning is crucial for aligning artistic vision with technical capabilities. The planning process begins by clearly defining art objectives and criteria, ensuring they align with the overall project vision. Next, available tech solutions are mapped in order to see if technical objectives meet operational needs, while also identifying potential limitations. Finally, test objectives are outlined to determine which elements require testing, preparing the team to explore artistic concepts alongside technical possibilities. This comprehensive approach ensures that both artistic ambitions and technical feasibility are addressed.
concept
Once the Script is done, the main storyline is unlikely to radically change. Yet, some tweaking is to be expected during the Conceptualisation phase, as the detailing of VP designs may lead to Script updates. However, it is important for the Script to be fundamentally reliable as it is the core source of information for breaking each scene into locations, sets and props. A process that translates narrative intentions into practical designs and constructions, which makes the “Script breakdown” essential to the VP conceptualisation.
Previsualisation or “previz”, should be an integral part of the Conceptualisation process. In conjunction with the script breakdown, referencing, e.g. “moodboards” and visualisations like “storyboards” begin. Complex sequences, difficult camera movements and visual effects (VFX) are analysed to define the needs of each VP set up and how to design it. A process ranging from design of physical/digital set elements, untangling VP from VFX tasks, to defining the needed VP production capacity.
First step is to fuse specifications from script breakdown with staging ideas from the Ideation process and define practical solutions and testing needs. Then, solutions and testing needs are validated in dialogue with experts. Next step is the design of key elements. A process that must stay in tune with location research, storyboards, floor plans and/or construction mock-ups of complex set arrangements.
The next step is detailing and validation of the practicalities of the VP designs. Blueprints for set elements are made and key needs are defined. To ensure overview and solid communication, it is recommended to use digital visualisation tools. A best practice is to build a “digital mock-up” of the full VP studio setup.
Conceptualisation entails more planning than ideation. This is the phase when technical, practical and economic details of the VP designs are ironed out as blueprints, plans and budgets. Because VP is a highly interdisciplinary practice, workflows from across film, TV and games are combined, which involves an array of shared checkpoints. Some of which are unfamiliar in either sector. To minimise risk of misunderstandings and errors during expensive production, planning should involve early validation of designs, identification of challenges, and measures for improved communication. Therefore, the conceptualisation process requires agile coordination and frequent communication between production departments.
Budgeting for VP is likely to happen in two ( or more ) steps, because even though some aspects of VP budgeting align with traditional routines, use of advanced technology, front-loaded workflows, and integration of new specialised skills insert unfamiliar territory in the process. As an example, the extended pre-production phase, involving both new types of asset creation and tech. setups for the virtual set building, can be difficult to estimate from a traditional film budgeting perspective. The use of previs, aids in creating a realistic budget, ensuring financial targets align with art and tech. needs. Consulting VP experts during conceptualisation also helps identify costs and points of awareness.
First budgeting step is an early cost-estimation, which relies on the script breakdown in combination with creative discussions and analyses that happen during previs. Consulting with department heads to roughly estimate virtual asset creation needs, obtaining initial quotes for general tech infrastructure, preliminary crew estimations, and a basic project timeline is key to the estimation. Also, referencing data from prior virtual productions as benchmarks and including a larger contingency to account for technological uncertainties is recommendable.
development
Test definition, testing planning and testing is useful for both successful Development and effective Requirement Capture. Based upon a digital mock-up and prototypes, tests for various components of the VP setup can be defined. Test definition allows for early evaluation of feasibility and identification of technical- and artistic challenges and should be done as a collaboration between creative- and technical- teams. This step involves identifying any technical or creative constraints that need addressing. Key aspects include integrating artistic and technical requirements and alignment of expectations.
Clear test definitions in combination with a thorough test plan sheds light on risks, helps reveal unknown issues and promotes validation of production capacity. All of which are natural focus points in the Requirement Capture process. Evaluation of floor plans, blueprints, and technical setups through the lens of test planning encourages stakeholders to look for possible problems, provide input and suggest improvements. An iterative process beneficial for exposing requirements but also for exploring solutions to make artistic vision and technical functionality work in tandem.
As the Requirement Capture progresses beyond test definition and planning it is beneficial for actual testing to begin. Having testing and Requirement Capture inform each other is recommendable because it promotes a higher degree of precision in both processes. Furthermore, both testing and Requirement Capture can produce findings that call for changes to design and the sooner the need for design adaptations is realised, the better.
In the Development phase design has a “Top Down” approach, driven by artistic intentions and creative objectives. During the Requirement Capture phase the design process shifts to a “Bottom Up” approach, guided by responses to limitations or solutions to practical problems. Having a Digital Mock Up is valuable for the general development but for Requirement Capture’s more granular definitions, made in dialogue between creatives and technologists, it may not suffice. Therefore, creating dummy assets for focused tests can be a useful development routine.
The digital artists building the virtual sets and props will often benefit from creating dummy assets. Initially, these are simplistic placeholders but as more features are added and details are approved by artistic and technical leads, the dummy assets evolve to final versions. This step-by-step, or incremental, development process fits well with the testing process, because the elements’ evolution can be tailored to meet testing needs as they emerge. In turn, making the Requirement Capture process more precise.
Incremental alignments between physical and digital elements also supports the Cinematographers, Production Designers and VP supervisors who guide the Development and Requirement Capture process. On the one hand, they will be able to evaluate the blend of physical and digital staging from early on, which grants them more creative leeway. On the other, they gain more opportunities to ensure artistic requirements are captured. Similar benefits apply for the technologists configuring the combined physical/digital set’s functionalities. Incremental development establishes an ongoing dialogue between the leading creatives, stagecraft, digital art and engineering that warrants a final product that meets both artistic and technical goals.
When moving from Conceptualisation to Development, the team usually scales up, as more work can be delegated.
Particularly, it is important to identify and recruit qualified experts for all aspects of the VP setup. On the physical side of the set, this means engaging with specialists in areas like AV-tech., lighting, rig, and set construction. On the digital side, it involves consulting with relevant VFX and post processing specialists. A guiding principle in this process is that the team’s focus shifts from exploration to detailed definition and production.
The need for coordination and clear communication grows in sync with the expansion of the team and activities must become more structured. This adds a need for increased management to the organisational requirements. During this expansive phase of Development, Requirement Capture should maintain a focus on testing, design quality assurance, and budgeting to ensure a clear path from concept to completion. Aligning project execution with artistic and organisational objectives alike, is a key managerial focus point at this stage.
As previously mentioned, VP budgeting is likely to happen in two ( or more ) steps. Extending from the estimate, made during Conceptualisation, a more precise budget can be rendered in conjunction with the Requirement Capture process. An opportune moment for making the granular budget is after all VP assets are identified and core tech. infrastructure is defined – but before detailed tech and infrastructure planning, final definition of digital assets’ complexity levels, hiring of the full VP crew and department (re)structuring, as well as before making the detailed VP production schedule. This way, producing a realistic budget to guide financial, artistic and contractual commitments.
In the second budgeting stage, one discreet factor that should be considered, is that crew training in unfamiliar VP workflows and tech. adds complexity. This complexity is partially addressed by the creative / tech. dialogue installed by the iterative workflows involving testing, design and Requirement Capture, but even though iterative development is beneficial for “learning while doing” it must be carefully planned and managed to avoid time and budget overruns.
Another point of budget awareness is the artistic flexibility VP’s real-time on-set adjustment capacity allows. This is highly appealing and useful but also entails risk of “scope & feature creep”. Standardised interactive features, such as changing the sun’s position or the weather are often readily available but making bespoke solutions for specific artistic variations can lead to a wide array of prepared interaction scenarios. This can entail data volumes that necessitate robust data management and/or lead to tech. setups that involve risk of on-set problem solving. Data handling and risk management for tech. failure are both crucial but costs can be difficult to quantify in advance.
finalisation
During the Finalisation stage Quality Assurance is a critical process. It covers several key aspects. One is to ensure elements for the planned VP recordings have the required accuracy, functionality, and coherence. Another is the integration of the project’s general production plan and the VP specific plans. Migrating the refined and tested VP elements into the filming environment requires solid planning and management. Integrating the digital/physical sets, actors and combined live-action/VP crews, while taking subsequent VFX into account, is a meticulous process, which involves a wide range of experts. Planning and coordinating the process is a demanding managerial task, which requires ongoing Quality Assurance of both artistic, technical and logistical aspects.
In VP productions, the line between artistic and tech. considerations is often blurred. In a management perspective this means an integrated Finalisation and Quality Assurance process that encourages collaboration and shared communication is ideal. However, the process must maintain distinct focuses in terms of review processes and dependencies between teams should be minimised. The goal is a coordinated workflow where artistic and technical considerations continuously inform each other, not a complex interdependent process with slow decision-making.
As the project approaches the final stages, managers must ensure the artistic team focuses on final visual quality checks, fine-tuning the aesthetic aspects of the virtual environments. In parallel, the tech teams must perform final performance optimisations and conduct failsafe testing to ensure system reliability. Finalisation concludes with system checks using full-quality assets and all functionality, allowing creatives, software operators and AV-techs to assess the complete VP setup under real-world conditions. This also works as the finalising Quality Assurance.
More often implementation work happens incrementally across several production stages but in this phase the work concludes as final implementation and Quality Assurance prepare the VP recordings. During Finalisation, the artistic, software tech, and AV tech teams need to work together to plan a precise and functional VP setup. Therefore, the digital artists must produce and combine the final digital assets, ensuring the final results aligns with the artistic vision and the practical circumstances on set. Meanwhile, the tech team must concentrate on Quality Assuring the technical infrastructure, performing tests to guarantee system reliability, final performance optimisations and bug fixes.
In the late-stage implementation, digital artists perform final asset refinement and adjustments based on performance tests. They collaborate on plans for light integration, and polish environment details. Color grading integration, animation refinement, and LOD optimisation are also crucial focus points. Optimisation for virtual cameras and frustum selection are done, and are followed by final artistic review. Finally, the digital artists prepare on-set support.
Throughout implementation, various film crew members actively participate and give feedback. Directors collaborate on virtual scouting, shot planning, and plans for blocking/rehearsals. The DP and camera team define virtual camera settings and calibrate real and virtual cameras to match. Production designers prepare alignment of virtual and physical sets and costume, makeup, and hair departments prepare their elements for the VP set. For this, colour is a point of awareness because some colours may be affected by LED light and appear differently in front of the screen.
Grips and electricians prepare to integrate physical and virtual lighting and effects. The sound department defines recording techniques suitable for the virtual space and VFX supervisors prepare to integrate traditional and real-time visual effects. Finally, assistant camera and Digital Imaging Tech teams define data workflows/pipelines and on-set monitoring needs. When possible, key routines and pipelines should be tested.
recording
Rigging for VP shoots requires close collaboration between set design, physical construction, lighting and camera tracking. Prepared set designs, Previz and intended camera movements, guide the configuration of physical vs. virtual elements. Requirement Capture done for the tracking system’s technical requirements and the plan for lighting must inform rigging rigging adjustments to avoid conflicts. Physical set pieces with integrated mechanisms and control systems must also be considered and should be tested in advance. The rigging team should work closely with the camera and art departments to identify potential limitations imposed by the tracking system, and adapt the physical set pieces and their corresponding rigging accordingly.
During set dressing artistic vision must be balanced with technical realities. The production designer-, director- and cinematographers’ shared vision must adapt to the limitations of physical and virtual environments’ scale and depth to facilitate integration of physical and digital elements. Often, the set dressing team cannot fine-tune alignment with those constraints until they are on the actual set, so time and resources must be allocated for this process. Open communication between tech. practitioners and creative leads requires the relevant crew members to be available on the set. Real-time adjustments, balancing practicalities with artistic vision, is a core part of VP set dressing.
Establishing a cohesive VP recording environment that blurs the lines between the physical and digital set elements is a key objective for the set dressing process. As part of that process the set dressing team must consider factors such as camera angles, lighting, and the interactivity of the set pieces to ensure that the final result is artistically and technically feasible. Simultaneously, the AV tech team handles final AV equipment integration and calibration. This involves synchronisations, calibrations, and tests of the system to ensure a cohesive and well-functioning AV experience. The two teams work concurrently, maintaining communication and collaboration to address any issues and deliver an integrated final VP staging.
The lighting design, which is critical to the overall aesthetic, must consider the unique properties of the LED volume and the way it interacts with practical lighting sources. The set dressing team must work closely with the lighting department to ensure that the placement and characteristics of physical set pieces and props enhance, rather than detract from, the desired lighting effects. Particularly, in terms of matching light effects such as blinking, fading or movement/animation of lights. In this process, the set dressing team must maintain dialogue with the production designer, the director, and the cinematographer to ensure that practical considerations promote the artistic vision
Interactions between elements in the digital environment and the physical set is a potentially enriching aspect of VP staging. It can significantly enhance the illusion that reality and the virtual world are one and the same. A dynamic relationship between the two can potentially be sophisticated because most game engines allow for a wide range of interaction features, providing novel creative possibilities. A challenge in achieving this is ensuring physical elements and actors can interact convincingly with digital elements. This requires precise alignment and calibration of physical set pieces with their virtual counterparts. Whether it is achieved as a LED- or green / blue screen based setup.
The choice between LED and green/ blue screens, or a combination depends on several factors. Some can be determined during previs and testing across various development stages, but other factors rely on the interplay between elements on set. Ultimately, the director’s and cinematographer’s vision plays a crucial role, influencing the choice between the tangible interactivity of LEDs and the expansive possibilities of green / blue screen compositing.
On the one hand, realistic lighting and interactive environments favor using LED screens. This solution enhances real-time performances and capitalises on physical light phenomena such as reflections / refractions. On the other hand, hi-res digital environments can be demanding to manage in real-time and better suited for post production.
Dynamic scenes will benefit from LEDs’ ability to simulate in real-time and make interactive changes on set, while green / blue screens’ capacity for complexity in the digital environments can grant a higher image quality and its flexibility for frequent environment changes can save time on set. Therefore, a hybrid approach, combining LED and green / blue screens, is sometimes used. It is not uncommon to film in a mixed environment where the parts of the screen that is behind the performers is green / blue, while the rest of the screens display the background used for lighting and reflections. Making it easier to replace background, create multilayer effects, etc. during post production.
Prior to recording, the VP stage is set up with either LED screens or a green / blue screen. Virtual sets are loaded onto the screens or monitors, allowing the crew and actors to see the scene. At this point cameras and tracking systems are calibrated for synchronisation with the virtual environment. Then the director, the actors and the cinematographer discuss key actions and emotions for each planned scene to determine blocking, dialogue, and timing. Actors rehearse movements and positions relative to both physical and virtual elements, while the cinematographer rehearses the shots. During the rehearsals, lighting, visual effects, and set design may be adjusted in real time. Several full run-throughs (dry runs) are usually done before shooting to perfect timing and interactions.
During principal photography, the cinematographer collaborates closely with the director, VP supervisor and gaffer to capture the actors’ performances and compose the intended camera angles, framings, and lighting. At the same time, the production designer ensures visual consistency between the virtual and physical sets, carefully considering how physical props interact with the digital environment. In tandem, the virtual production supervisor oversees all technical aspects, monitoring the integration of virtual and physical elements and providing guidance as needed.
As each shot progresses, the director and cinematographer stay aware of the actors’ placement relative to the virtual environment and guide actor performances to react authentically to the digital environment.
After each take the production designer checks visual consistency and lighting; the director assesses the storytelling impact of the virtual elements, collaborating with the VP supervisor and production designer on adjustments. The cinematographer evaluates lighting, composition, and overall visual impact, suggesting improvements to camera angles, framing, or lighting. The VP supervisor addresses technical issues like tracking errors, coordinating the crew to ensure a smooth workflow. Actors actively participate, adapting their performances based on feedback to refine their portrayals and align with the overall vision. This joint and highly collaborative process is key to the effectiveness of the shooting, the efficiency of post production and achieving high-quality results.
follow_up
The post-production workflow for VP is fundamentally similar to that of traditional film making. It begins with the editor organising the raw footage, collaborating with the director to create a rough cut. VPs extensive pre-visualisation, often means “pick-up” shoots for storytelling purposes are rare. If continuous review of shots has been part of the on-set routines, reshoots for technical reasons are also uncommon, as technical issues are addressed on the set. However, post production for VP shots made in front of a LED screen can involve significant colour correction due to the nature of LED lighting. In a similar vein, post production of shots produced in a LED “volume” can involve post processing of recorded audio as LED volumes tend to create echo and taint sound in unwanted ways.
VP usually entails additional VFX work. Integration of set extensions, adding foregrounds and volumetric effects such as smoke and rain is often part of the artistic vision and is done during post production. When such VFX intentions are known at an early stage, the VP design can plan for a mixed environment where parts of the screens, behind the performers, is green / blue screened while the rest of the volume carries backgrounds, making it easier to replace backgrounds, etc. Yet, even if a mixed environment has not been used, set extension and other VFX augmentations can still be added during post production. VFX artists can extend the digital environments to create a more expansive view, beyond what was initially shown on the LED screen, digital elements can be added and re-lighting techniques can be used for light source manipulation and focusing effects.
Grading is a standard process in post production but can be challenging in a VP context. When actors are filmed in front of LED screens, the colours and light emitted from these screens can reflect on their faces and costumes. While this enhances integration, it can complicate grading, as reflections may need to be adjusted to ensure consistency with the intended color schemes Due to the nature of LED lighting’s limited colour granularity, it is recommended that the cinematographer and colour grading experts test color management workflows through the entire pipeline. If the grading process is not managed carefully, it can lead to discrepancies in color between shots, especially if scenes were shot under varying lighting setups.
When working with VP, planning for post production during pre production is a managerial best practice. Producers should prioritise comprehensive assessment of VFX needs, technology analyses and budgeting during pre production. To this end, it is recommended to foster early cross-functional collaboration, engaging not only with the lead creatives and the VP supervisor but also aligning the creative vision with post production teams. A holistic approach, which helps establish a post production strategy early on. A validated post production strategy can guide decision making and promote VFX integration, frictionless colour grading and help deliver a successful VP result.
Another recommendation for effective VP post production is to maintain the editing team as a central hub throughout this phase. Having the editing team in a coordinating role helps maintain quality control because editors can evaluate how well audio and VFX work together, ensuring all elements align harmoniously and provide lead creatives and producers with versions of the VP footage in its storytelling context. Placing the editing team in a pivotal position not only enhances communication, streamlines workflows, and fosters creative collaboration but also ensures that the artistic vision is consistently executed throughout the post production process. Often, editors have a comprehensive understanding of both narrative and audio visual aspects of the project. From a central position, they can ensure that storytelling remains the priority and support the artistic result effectively.
Finally, installing “post mortem” reviews at the beginning and end of the post production is a good practice.
Post mortems are collective analyses of events and results done after a project or process concludes. They typically identify what went well and what went wrong, and define how to improve outcomes moving forward. Conducting a post mortem early in post production allows the team to pinpoint immediate challenges that may have been noted during recording or been detected in the initial stages of editing or VFX integration. Addressing these concerns early can prevent them from escalating and affecting the overall timeline. Also, early and collaborative problem-solving, helps identify gaps in communication, enabling managers to establish clearer information channels and protocols. This is particularly useful in VP contexts, as VP methods have yet to become native to post production routines.
Having a post mortem at the end of post production capitalises on the knowledge and experience of the project, allowing for the next project to be more artistically effective and practically efficient.
