How to build a benthic coral reefs analyser

In this post we explore the development of a benthic coral reef analyzer, built in partnership with ReefSupport to improve the tools for monitoring coral reefs and marine environments.

For the full picture, here is the coral analysis pipeline — tap through for the project:

Capture → segment → classify → measure coral cover over time

Leveraging computer vision for the segmentation of coral reefs in benthic imagery holds the potential to quantify the long-term growth or decline of coral cover within marine protected areas

Project Scope

Our collaboration develops an underwater benthic imagery model that identifies and locates the functional groups in a reef — flexible enough to use across marine regions worldwide.

The benthic imagery analysis system by Reef Support

We start by distinguishing hard from soft coral, then add finer taxonomic detail as the system matures — a broad foundation to build comprehensive reef analysis on.

Provided Datasets

The data provided by ReefSupport is made available on a publicly hosted Google Cloud bucket. It comes in two forms:

1 / 3

Each image is associated with a dense stitched mask made of all the individual coral instances.

Associated labels from the ReefSupport dataset - hard and soft coral instances masks

Exploratory Data Analysis

Before modelling, we explored the dataset closely — and it surfaced several data-quality issues worth fixing first.

Data quality issues

Empty masks

Some stitched masks were entirely black — 532 in SEAVIEW_PAC_USA and 328 in SEAVIEW_ATL. Removing these empty masks gave a cleaner dataset and improved performance during training and evaluation.

Empty Masks samples

Low quality labels

The dense labels in SEAVIEW/PAC_USA covered almost all the coral in an image as one big mask rather than outlining each individual, so we excluded them from the training set.

Low quality Masks samples

Mismatched sparse and dense labels

We compared ReefSupport’s dense (mask) labels against the sparse (point) labels for the same images.

Mismatch summary and distribution

The presented samples illustrate instances where point labels contradict dense labels. Each white cross signifies a label mismatch, with the first sample showing a 17% error mismatch and the last sample demonstrating a complete 100% error mismatch.

Samples to illustrate sparse and dense label mismatches

Data leakage

In one region, many images overlap with their neighbours within the same quadrats. If overlapping images land in different splits, the model effectively sees test content during training — inflating its scores. Ordering by image ID makes it clear: most images share content with their neighbours.

A sequence of 4 photos that share overlaps with their neighbours

Another sequence of 4 photos that share overlaps with their neighbours

Since it is confined to one region, we gauge its impact by evaluating the model region by region.

Class imbalance

The dataset skews heavily towards hard coral — roughly five times more instances than soft coral. Such imbalance biases a model towards the majority class and hurts its performance on the minority one.

Class imbalance distributions

Data Preparation

YOLOv8 TXT format

To use the YOLOv8 ecosystem, we first convert the raw datasets into its expected format.

YOLOv8 TXT format conversion

Each line represents an instance of a class with a defined contour. It has the following format:

class_number x1 y1 x2 y2 x3 y3 ... xk yk
class_number x1 y1 x2 y2 x3 y3 ... xj yj

Where the coordinates x and y are normalized to the image width and height accordingly. Therefore, they always lie in the range [0,1].

Example:

1 0.617 0.359 0.114 0.173 0.322 0.654
0 0.094 0.386 0.156 0.236 0.875 0.134

Therefore, each line corresponds to an individual mask instance.

The OpenCV library is employed to convert the dense individual masks into contour coordinates.

Data Modeling

Data Split

In this section, we elucidate the methodology employed for the train/val/test splits across different datasets.

For each region, a dedicated dataset is created with an 80/10/10 split ratio for train/val/test. Simultaneously, a comprehensive global dataset is established using the same split ratios. Importantly, any image allocated to the test set for a region-specific dataset is also included in the test set for the global dataset (similarly for train and val splits). This design facilitates the evaluation of models trained on region-specific datasets against the global dataset.

Instance Segmentation vs Semantic Segmentation

Semantic segmentation assigns a class label to each pixel in an image, such as ‘person,’ ‘dog,’ or ‘flower,’ grouping together pixels of the same class. Conversely, instance segmentation distinguishes between individual instances of objects within the same class, treating each one as a separate entity.

Semantic segmentation vs Instance segmentation

For analyzing benthic coral reefs, an instance segmentation approach proves superior as it enables precise localization and counting of reef organisms.

Evaluation Metrics

We evaluate segmentation with mean IoU (mIoU) and the Dice coefficient, avoiding mean pixel accuracy since it’s misleading on skewed datasets.

mIoU (Jaccard index) measures the overlap between the predicted and ground-truth masks — higher is better:

Dice coefficient (F1) also rewards overlap but weights true positives more heavily, which makes it well-suited to imbalanced data:

YOLOv8

Overview

We took a pretrained YOLOv8 model and fine-tuned it for our instance segmentation task. YOLOv8 is fast, accurate, and easy to work with, and it handles a range of tasks — object detection, tracking, instance segmentation, image classification, and pose estimation.

Segmentation on a benthic image: a photo in, each colony mapped out

Training

Baseline

We first established a baseline to gauge the approach: a medium-size pretrained model fine-tuned for 5 epochs on the train split.

Results / Quantitative - Training metrics (left) and pixel level confusion matrix (right)

Results / Qualitative

The initial results are highly promising, prompting us to further optimize the performance of the modeling approach through meticulous selection of hyperparameters.

Best Model

After hundreds of GPU-hours of hyperparameter search, we arrived at the best-performing models.

Given the uncertainty about ReefSupport’s hardware configurations and the intended use of the models (including the possibility of running on live video streams from underwater cameras), we aimed to offer a diverse range of models. These span from models suitable for embedding on edge devices, enabling real-time video stream segmentation, to high-end GPUs delivering peak performance. This approach ensures flexibility to accommodate various deployment scenarios.

The pre-trained YOLOv8 models undergo fine-tuning for 140 epochs with images resized to 1024x1024 pixels. Additionally, random flipping and rotation of images up to 45 degrees are applied during training.

Data Augmentation / Batch Samples

Results / Quantitative - Training metrics (left) and pixel level confusion matrix (right)

Results / Qualitative

Evaluation

The subsequent table provides a summary of the performance of the best model on the test sets for each region:

As the various evaluation metrics are weighted in proportion to the number of pixels per region, we provide a summary below, illustrating the different weights assigned to regions based on their respective pixel counts:

Model size vs Model accuracy

The table below summarizes the performance of the different YOLOv8 models that are trained on the same training set, using the same test set for evaluation.

The top-performing model is the l size model, as indicated in the table above. As the model size decreases, there is a slight degradation in performance—from a mIoU of 0.85 to 0.83. However, the advantage of smaller models lies in their faster execution and compatibility with smaller hardware devices.

Conclusion

YOLOv8 proved a strong fit for this instance-segmentation task — accurate even on modest hardware, and fast enough to run on live underwater video streams, which makes it practical for real deployments.

Benthic Segmentation / Hard Coral

In conclusion, while YOLOv8 presents a robust solution for the instance segmentation task, it is crucial to carefully address issues related to regional model performance, data leakage, and dataset quality. The insights gained from our findings are invaluable for refining and optimizing computer vision applications in marine biology and underwater image segmentation.

You can try the segmenter yourself on real benthic imagery — the interactive demo runs right in your browser.