Nov 30, 2022
FSQC Protocol
- Saashi Bedford1
- 1University of Cambridge
Protocol Citation: Saashi Bedford 2022. FSQC Protocol. protocols.io https://dx.doi.org/10.17504/protocols.io.kxygx9m6wg8j/v1
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We use this protocol and it's working
Created: September 19, 2022
Last Modified: November 30, 2022
Protocol Integer ID: 70208
Abstract
This protocol is a guide to conducting a visual quality control (QC) of Freesurfer outputs. Quality control is important to make sure that the cortical surface reconstruction and grey and white matter classification are accurate.
To examine outputs, ten images per subject are generated (see https://github.com/sbedford0/Generate_FSQC_images), showing the cortical segmentation and surface boundaries at different slices and views of the brain (3 axial; 3 coronal; 4 sagittal). The pial surface is delineated in red, and the white matter surface in blue. These images should be examined to check that the reconstructed surfaces follow the cortical boundaries accurately.
Guidelines
Rating criteria:
BAD
- Missing large area, or multiple small/medium areas, of cortex
- Large errors in grey/white matter boundary
MINOR ERROR
- Missing small area of cortex
- Minor over or under segmentation
- Misclassification of GM/WM in one or two regions
GOOD
- Accurate delineation of cortical boundary across the brain
- No cortex missing
MOTION / POOR SCAN QUALITY
- Visible motion artefact or poor overall scan quality (i.e. ringing, blurring)
- Note: If Freesurfer surface looks ok or has minor errors, but scan quality is poor, rate as MOTION. If there is visible motion in the scan and Freesurfer has also performed badly (large or significant errors), rate as BAD
Materials
Example participants : good output
Example participant 1:
Example participant 2:
Example participant 3:
Examples of “issues” that are actually ok:
- No Freesurfer output or cortex visible (should still be rated as “good”, as long as it doesn’t look like there is cortex that should have been included)
- Large sulci or spaces that can look like holes in the reconstruction, but are actually accurately following the cortical surface (should be rated as “good” as long as you can’t see cortex and/or white matter in those spaces that was clearly missed)
- Neck or shoulders included in field of view (should be rated as “good” as long as the surface reconstruction looks good - ie. ignore the extra tissue and just rate the brain as usual)
Examples of issues/bad outputs
Minor error: common issues
- Missing grey matter in frontal cortex
- Overestimation of cortex into skull
- Slight temporal pole underestimation
Bad:
- Large chunks of cortex missing
- Larger sections bleeding into skull in multiple places
- Temporal pole underestimation (more extensive)
- More examples of temporal pole :
- Note: these subjects also have visible motion in their scans. However, the Freesurfer reconstruction has large errors, so we would rate this as “bad”, to signify that it is unusable.
Motion:
Look out for: ringing or rippling effect in scans; blurry or fuzzy images
- (If you would like a better idea of what to look out for when identifying motion, and to see some examples in raw scans, see the link below - but ignore the ratings described in this link as it is a different scale): https://github.com/CoBrALab/documentation/wiki/Motion-Quality-Control-(QC)-Manual
Some examples of outputs with motion:
- Sagittal view:
Ringing effect:
Blurring (with a bit of ringing)
- Coronal view
- Axial view
- In the example below, see the blurriness of the cortex and white matter, and ringing artefact in the cerebellum:
Motion in scan (and temporal pole underestimation)
- This subject has only slight underestimation of the temporal pole (only visible in sagittal view) which would usually be rated as “minor error”. However, here, the motion is more of an issue; therefore this scan would be rated as “motion”.
- Another example:
Instructions
Instructions
Download, save and unzip the Image Rating QC app from https://github.com/sbedford0/imageratingQCApp
Follow the instructions in the “readme” file to run the program:
- The first time you run the program, in a terminal, run ‘npm install’ in the app directory
- Run ‘npm run build’ to build the app for production in the build folder and create a local version (you only need to do this once)
- Once you have done this, any time you want to start the app, run ‘npm run start’ in the app directory to launch the app in a web browser
- Click ‘browse’ to select and load in your images and start rating!
Note: if you have a large dataset, try to limit yourself to maximum ~2000 subjects in one session to avoid fatigue and distraction
Images will be displayed one by one
Rate by either clicking the buttons or using arrow keys
- Thumbs down (← key): BAD
- Shrug (↓ key): MINOR ERROR
- Thumbs up (→ key): GOOD
- Brain with ripples (↑ key): VISIBLE MOTION / POOR SCAN QUALITY
See Guidelines for rating criteria, and Materials for examples of outputs and ratings
When you have gone through all of the images, a download button will appear. Click this to download your ratings as a csv file.
After downloading your ratings, convert scores to numeric
- GOOD = 1
- MINOR ERROR = 2
- MOTION = 3
- BAD = 4
Calculate an average to generate a score between 1-4 for each subject
Ideally have (at least) one other rater score your images
- Compare scores (average score per participant), and take an average and/or resolve any major discrepancies between scores
- Check correlation and/or ICC between raters to make sure inter-rater reliability is reasonable
Apply your exclusion criteria to exclude scans with a score below a certain threshold or cut off, or use these scores as a covariate in your analysis