Welcome to External Verification for Netflix. Through this program you will be responsible for reviewing Master QC reports, making corrections or additions to the issues noted, and being the final set of eyes on Netflix Original content.
During the QC process, Netflix preferred vendors are instructed to review our content in full and note every instance of any issue that may be a technical problem or content inconsistency. Not all of the issues reported will be be accurate or actionable. It is your job to make sure the final report that goes to our production partners for arbitration is as clear, accurate and complete as possible, so they can take further steps in addressing any issues.
Verification takes place in the Netflix Backlot for Secondary Audio Dubs/ Audio Description and Primary A/V files. Verification for M&E and Atmos files takes place in the Netflix Asset QC platform. This is where you will find what assets you are assigned to, check the reports, download the files and submit your results.
- Review every issue in context, with an understanding of how we review issues internally at Netflix
- Completion of all assigned Report verifications within standard 24hr SLA from assignment
- Includes file transfer time
- Verifying Master QC reports (includes, but NOT limited to):
- Full context and instructions will be provided for any
- Deleting inaccurate error code flags
- Adjusting error severity in accordance with Netflix quality standards
- Amending error descriptions to be more specific
- Confirming issues that have been flagged are actually present in content
- Accepting or Rejecting the QC report based on issues present and decisions in playback
- Tagging the correct report viewers for each asset
- Investigating issues flagged on previously completed reports
- Overturning reports should full context on issues be provided by the Master QC Specialists
- Escalating blocker issues to the internal Netflix Master QC specialist
- Performing subsequent 5-point spot check of content for additional technical issues
- Additional responsibilities may be assigned by Netflix Master QC specialists as needed.
What kinds of Files do Verifiers work with?
M&E, Secondary Audio dubs/ Audio Description, Atmos:
- Quicktime MOV wrapped mixes or individual discrete WAV files
- Uncompressed linear PCM format
- 16 or 24 bit depth
- 48,000 Hz
- 8 Channels in Netflix standard audio config (L, R, C, LFE, LS, RS, LT, RT)
- Atmos BWAV ADM
- H.264 proxy with burn-in timecode and Netflix watermark (for alignment)
M&E (music and effects) files will be the source for all secondary audio dubs. If the M&E files are QC’d, the secondary audio dubs will not require source-level QC and thus, will not require verification.
Secondary audio dubs will be in foreign languages. You do not need to be a native speaker of the dubbed language in order to verify the file. You are not responsible for reviewing the linguistic content of dubbed audio. If you do happen to be fluent in (or even just semi-familiar with) the language of the dub and catch a translation error, that’s great. But, this is certainly not a requirement.
Audio description is a secondary audio file containing narration for the visually impaired. A narrator will describe on screen elements and action. Audio description files tend to be in English but may also be provided for foreign languages.
Atmos files will be the English Printmaster with a 7.1.4 channel bed config. These can include up to 128 separate audio objects with all necessary object placement information.
- 4K HDR IMF or 4K/1080p ProRes
- IMF packaged to Netflix Original Delivery Specification
- ProRes with theatrical or nearfield audio
- Primary audio (will usually be muxed with the primary video)
We typically refer to video sources as “Primaries.” They are the asset from which many encodes are created. The report structure and verification procedure will be the same as it is for audio files, except you will be responsible for all picture notes as well.
Assignment and Receipt/ Download of files:
You will login to one of the following to check the verification queue/ see what you are assigned:
https://backlot.netflix.com/cqc/requests/state/5/languageCode/any/master/true - Secondary audio dubs, Audio description, Primary A/V
- You will receive files via Source Share and will download them to your local station for review.
https://assetqc.netflix.com/qc?counter=ALL# - M&E and Atmos
- Files will be attached to the QC report and downloaded directly from within the Asset QC platform
If you are unable to download assigned files due to errors in Backlot/ Asset QC, please email email@example.com.
Playback Hardware and Setup:
Your dedicated workstation must have playback hardware and software that can support full speed 4K playback as well as frame-by-frame step through analysis.
Examples of playback software/hardware:
- Mac Pro with Adobe Premiere
- Mac Pro with Final Cut Pro 7
- Windows 10 PC with Adobe Premiere
- Rohde & Schwarz Clipster
- Colorfront Transkoder
- Professional or Pro-sumer grade monitor (high quality computer monitor is acceptable)
- 5.1 surround speaker system or true 5.1 headphones (6 drivers, NO virtualization)
- Stereo monitor headphones
- Centralized file storage (local RAID or SAN)
- High Speed Internet connection to Backlot/ Asset QC
- Dolby Media Meter, Nugen LM Correct, or Nugen VisLM (for LKFS/ True Peak verification)
All playback hardware should be calibrated to be at a reference level of -79dB SPL for audio and monitors should be calibrated to be color accurate. Please reach out to StudioTechnologies@netflix.com if you have any further questions about calibration.
Align audio and video files in your playback software/ hardware of choice, and verify they line up. Any major sync or length discrepancies should be flagged. If you believe an incorrect file has been delivered to verification request you are assigned, stop down and reach out to firstname.lastname@example.org.
Once you have confirmed you have the correct assets and your playback is properly set up, you are now ready to begin verifying your assigned report.
- When you login to Backlot, all assets assigned to you will be listed:
- You will click on a title link in the left most column to open the QC report for that asset:
- You will see the following three icons above each issue:
- Edit Icon - Select this to edit Severity, Timecode and Description of a given issue.
- Clone Icon - Select this to duplicate an issue, if the error code chosen was incorrect. You will choose the correct error code under the newly cloned issue, and then delete the original notation.
- Viewed Icon - Select this to mark an issue viewed (once an issue has been updated or if no changes are required). The issue will disappear from the list once “viewed”.
- To show the full list of reviewed issues, uncheck the “Hide Viewed Issues” box at the top of the issue list.
Checking Priority and Version:
The three possible priorities that can be listed are Low, Normal and High.
For Normal or High priority requests, Netflix is striving for pixel and sample level perfection on these assets. Consider not only the customer experience but also the long term quality of the asset for archival - i.e. Don’t just assume “the average viewer will never see that.”
For Low priority requests, consider the asset under a lesser scrutiny. This should have been the equivalent of a broadcast level of QC.
Please only keep the following error code flags as Needs Fix or Needs Review, please mark all others as FYI:
- Audio Static
- Audio Channel Issue - Audio Mix Missing
- Audio Channel Mismap
- Audio Channel Issue - Mixdown Issue
- Audio Channel Issue - Panning
- Audio Drop Out
- Audio Drop Out - Dialogue
- Audio Drop Out - Mute
- Loudness LKFS Measurement
- Volume/Level - Peaks
- Ident - Mismapped
- Ident - Mix Down
- Ident - Peak Levels
- Ident - Sync
- Ident Altered
- Sync - Dialogue if over 5 frames off and on screen
- Sync - Effects if over 5 frames off and on screen
- Sync - Global Drift Early
- Sync - Global Drift Late
- Sync - Global Offset
- Extraneous Content - Description
- Extraneous Content - Dialogue
- Extraneous Content - Music
- Missing Content - Dialogue
- Missing Content - Effects - If on screen
- Incorrect Asset - Title
- Incorrect Asset - Language
- Incorrect Asset - Asset Type
- Frame Error - Blanking Shift
- Frame Error - Freeze Frame
- Extraneous Content
- Extraneous Content - Commercial Blacks
- Extraneous Content - Censorship
- Extraneous Content - Burned in Subtitles
- Missing Content - No Black at head or tail
- Missing Content - Truncation
- Conversion Error - Aliasing
- Conversion Error - Ghosting
- Conversion Error - Combing
- Luminance - Harding PSE Failed
- Color - SDR Issue
- Color - HDR Issue
- Incorrect Asset - Title
- Incorrect Asset - Metadata
- Video Hit
- VFX Error - Bad Edit - Includes spelling and grammatical errors in graphics and lower thirds.
Assets may be re-delivered (in some cases) multiple times to incorporate fixes and to address new issues noted during the fix check/ re-QC process. The “version” of the asset must be considered when assessing if issues brought up after the initial QC are actionable.
There should be two main considerations that go into this decision:
- Is the issue blatant and/ or jarring?
- Can I immediately see it in playback?
- If I were an average customer, would this issue impact my viewing experience?
- Has the issue been flagged before?
- Does this issue (or instances of the same issue) appear in the history?
- Has the content partner reviewed the issue in previous versions and signed off as “best possible” or “creative intent”?
Check the history of an asset before diving into the individual issues to better inform your decisions:
- Select the “History” tab at the top of the issue list:
- The “Result” defaults to the current request, so you must select a previous date from the dropdown:
- Typically, you will want to review the most recent QC results prior to the current request (the second date from the top).
- If there is only one date under the Result dropdown, it means the current report is the first version of an asset. You will be reviewing each issue for the first time, so extra scrutiny should be given to ensure accuracy for possible subsequent versions.
- Select “Latest Verification” from the “Issue Type:”.
- Your History tab will now look like this:
- You will see the list of issues from the previous version, along with the notes and changes the Verifier made. Use this history as context when verifying newly flagged issues in the current version.
Move down through the list, type in the timecode, bring up the issue in playback and find it in the audio or video content. Then, based on the error code and description, you must reproduce each issue in playback and designate a severity. Make sure that the issue description is also as succinct and accurate as possible. Our goal is to make it so that a producer or post supervisor can look at the QC report once and know what action needs to be taken, without opening a session or calling in personnel.
If the description is not correct or could be more accurate, click the pencil icon in the top right corner to edit. The issue will change to look like this once editable:
Add any more information you can to the report, like this:
We want to include as much information as possible to decrease the amount of time needed to fix these issues. If you can add more information, always include it.
If the error code is not correct:
Perhaps the QC operator incorrectly flagged a Camera Operator in frame as an Audio tick.
This is an accurate description, and the timecode is correct, but the error code used is not.
In this case, we would clone the issue by pressing the Clone Icon (middle icon in the top left of the issue) and then deleting the old error. Cloning the issue brings up a new flag at the bottom of the report with all the old information included. You can then update the error code, severity, timecode or description and save the issue:
Click the drop down with the error code to change it with the correct error code:
Once the issue is correctly updated, click save in the bottom right corner and the issue will look like this:
The old flag, with the incorrect error code will still be in the report, this must be deleted by clicking the trash can icon in the upper right corner of the flag:
By deleting this issue, you remove it from the report and mark the QC operator incorrect in their entry. It is important to provide this feedback to our QC vendors so they may learn and get better.
If the Timecode flagged is incorrect:
When an operator flags an issue and does not input the correct timecode, please adjust the timecode to be accurate. If the timecode is wildly inaccurate, for example, more than 20 frames off, then delete the issue.
To change a timecode, click the edit issue icon and change the in and out timecodes in their respective fields.
If the Error Severity is not correct or you are overturning the issue:
Our QC operators flag a lot of issues. We instruct them to so that we may get the most knowledge about every asset. Sometimes, they will flag issues that are very minor or not actionable. In those cases we do not delete the issues, as they are present and don’t want to affect their metrics. So we change the severity to be from Needs Review or Needs Fix to FYI.
Below are some example images:
Here is an issue that is OK as is:
We have decided that it’s not a Needs Fix issue and want to change it to FYI. Click on the edit issue icon.
Then click on the severity drop down menu and select FYI.
Then click the save button and the issue will look like this:
The issue is now complete and saved.
Please repeat this as many times as necessary on a per issue basis until every issue has been reviewed.
It is extremely important to review every issue. When our QC reports are completed and sent out, we are saying that every issue contained therein is accurate and to the point.
Remember that some issues should not be included due to the priority of the report. Refer to the request priority in the top metadata field to know which priority of Verification you should apply to the report. Page 3 goes into detail about what issues should make it on the the final QC report.
Some issues may appear on a report that are blockers.
We define a “blocker” issue as an issue so severe that the end user cannot consume the material. An example of this would be a Japanese dub delivered to a German dub source request, a global sync issue or large scale corruption of the file. Think to yourself, if this issue was in a show you were watching, would you be able to enjoy it or would you turn it off? If the answer is that you would turn it off, flag this as a BLOCKER severity and email email@example.com immediately. This will help us get a fix quickly.
Completing a Report - Spot Check and Accept/Reject
After all issues have been verified, adjusted or deleted and you feel they are all in the best shape possible, you will perform a 5 point spot check of the file. This will include watching down the first 2 minutes of content (including double checking the Ident), a 1 minute section near 25% runtime, a 1 minute section near 75% runtime and the last 2 minutes of content (including double checking the Ident.) Keep an eye out for any issues missed by the QC operators at these points.
If you find an issue that was not found by the QC operator, click the Add Issue button in the top right corner of the report to add it to the list. Be as specific as possible and use the correct error code.
If you do not find any issues and the content is good, you are now ready to complete the report. We complete reports by clicking the Accept or Reject buttons.
Accepting a Report
If you have gone through all the issues flagged in a report and deemed them to be be FYI severity or were incorrect and have removed them, then you will click the Accept button at the top of the report. By clicking the Accept button, you are saying the asset has passed QC and requires no more review by the Content Partner or by Netflix. Be very sure that all issues flagged have been checked and deemed FYI before clicking the Accept button.
After you click the Accept button, the page will update the QC status for a few seconds. Then the status will change in the top metadata to Accepted:
Once the status has successfully updated, there is no more action required on the part of the Verifier. You may close this report and move on to your next asset.
Rejecting a Report
When an asset has issues that should be addressed or fixed by the Content Partner, or they have neglected to include of address an issue on a later version of a file, we Reject the QC report.
Once all the issues have been checked, the severities updated and the descriptions edited to be very succinct and accurate, click the red “Reject” button at the top of the report. This will cause the partner to be notified that their QC report has been rejected and they have issues they need to address.
After clicking the Reject button the status will update for a few seconds, and change in the top metadata to “Rejected.”
Then a box will appear below the Accept and Reject buttons with the header “Report Viewers.” This is a very important step in the Verification Process.
Report Viewers List and Tagging
In order for external and internal teams to see the QC report you have just completed, they must be tagged individually in this box. Report viewers are only tagged on Rejected reports. They are not added for Accepted reports.
You do not have to know by heart who should be tagged in each report. You will have access to this list, called “Report Viewers List by Show,” that is a title by title list of who should be tagged for a Primary A/V file, a Secondary dub or an Audio Description asset.
To tag someone in the report, reference the Report Viewers List by Show and type their name or email address into the box that appears when you rejected the report. It will look like this:
Once the correct viewers name is highlighted, hit the enter key and they will be tagged.
The email address will receive an email instantly that they have been given access to this report.
If you make a mistake and tag a person who is not supposed to have access, that is OK. Simply delete them from the Report Viewer box with the backspace key and retype the name of who is supposed to receive the report. If you have any questions about report tagging or who is supposed to see a report, email MasterQC.Support@netflix.com. Also email Master QC if the show is not listed or there are any errors when trying to tag a name.
Questions or Help
If you ever have any questions on how to operate the tools in Backlot, how to flag or grade a report, or about verification in general please do not hesitate to reach out to firstname.lastname@example.org. We are happy to help with any questions you may have.
- Tips and Tricks
- Flags for Polish voice over Forced Narratives (FNs) should be deleted, our QC NPVs have been instructed to not flag them.
- Voice over FNs in other languages (excluding Polish) should be flagged, we are supposed to cover FN elements with on screen text.
- Any Ident sync flags that are less than 1 frame (2000 samples) should be waived and turned to FYI severity
- The head and tail idents should have 1 second of black and silence before and after their appearance.
- Color space issues on the Ident should be flagged, these appear as a washed out appearance in the whites and a lack of gradient around the logo. The easiest way to tell if an ident is in the wrong color space is to look at the N. If the border on the left side of the N blends into the background, the color space on the Ident has been incorrectly converted.
- When all else fails on the Ident, compare the flagged Ident to a source Ident of the same kind. That will show you what and how to describe the issue accurately.
- If the audio file does not match the length of the proxy, that should be flagged. If the proxy does not match the length of the audio file, that is an internal bug and should not be flagged.
- ALL Blocker issues should be emailed to email@example.com as soon as they are found to rush a fix.
- Ident peak levels will usually be flagged for Audio description. These are OK as is, as the Ident must be dipped to accommodate the narrator over the content.
- The Audio Description for the Ident should always be “Letters pop out from a white background and then turn red, Netflix.”
- When saving an issue, wait for it to complete the saving process. If you update another issue before a previous issue has saved, that next issue will not update.
- Keep your assignments list in one window and your active report in another, open your report by right clicking and opening in new tab or window.
- For Video assets: watch issues in context with handles, rewind 10 seconds before the issue and play through to see how it interacts.
- If there are frame edges visible due to image stabilization on one shot, there are probably more throughout program. Check other areas around shots with frame edges creeping in.
- Don’t flag Lens Flares as Needs Fix unless they seriously impede the on screen action, making the image obscured, otherwise they should be FYI or Review.
- QC operators are allowed to indicate an issue over many shots, do not mark incorrect if a span is noted, but the total in and out timecodes must be accurate. Add how many shots it occurs on if this has not been included.
- SDR issues need to be checked through a hardware or software CMU to confirm their presence.
- Be sure to reproduce every issue and check every description.
- If you get any errors with a red dialogue box at the top of the report, try reloading the page. If that doesn’t work, please email MasterQC.Support@netflix.com, this may be a tool issue that needs to be addressed by Engineering.
- The SLA (Service Level Agreement) for turn around time is 24 hours from assignment to completion of the verified asset report. This means that from when you are assigned an asset to verify, you have 24 hours to download, verify and complete the report.
- Consistency is also important. If you see the same or similar issues in multiple versions of an asset, please make sure the issue is noted the same across all versions.
- If you are unsure about whether an issues present is Creative intent or the best possible state of the asset, then mark it as a Needs Review issue and the Content Partner can provide context on the issue.
- 2/7/2018 - Added workflow and error codes for Low Priority requests - Page 3