pytorch-unet-segmentation Members : PyeongEun Kim, JuHyung Lee, MiJeong Lee Supervisors : Utku Ozbulak, Wesley De Neve Description This pro…
pytorch-unet-segmentation Members : PyeongEun Kim, JuHyung Lee, MiJeong Lee Supervisors : Utku Ozbulak, Wesley De Neve Description This pro…
Members : PyeongEun Kim, JuHyung Lee, MiJeong Lee
Supervisors : Utku Ozbulak, Wesley De Neve
This project aims to implement biomedical image segmentation with the use of U-Net model. The below image briefly explains the output we want:
The dataset we used is Transmission Electron Microscopy (ssTEM) data set of the Drosophila first instar larva ventral nerve cord (VNC), which is dowloaded from ISBI Challenge: Segmentation of of neural structures in EM stacks
The dataset contains 30 images (.png) of size 512x512 for each train, train-labels and test.
…We preprocessed the images for data augmentation. Following preprocessing are :
Original Image
Image
Flip Vertical
Horizontal
Both
Gaussian noise Standard Deviation: 10
Standard Deviation: 50
Standard Deviation: 100
Uniform noise Intensity: 10
Intensity: 50
Intensity: 100
Brightness Intensity: 10
Intensity: 20
Intensity: 30
Elastic deformation Random Deformation: 1
Random Deformation: 2
Random Deformation: 3
CropLeft Bottom
Left Top
Right Bottom
Right Top
Padding process is compulsory after the cropping process as the image has to fit the input size of the U-Net model.
In terms of the padding method, symmetric padding was done in which the pad is the reflection of the vector mirrored along the edge of the array. We selected the symmetric padding over several other padding options because it reduces the loss the most.
To help with observation, a 'yellow border' is added around the original image: outside the border indicates symmetric padding whereas inside indicates the original image.
PadLeft Bottom
Left Top
Right bottom
Right Top
We have same structure as U-Net Model architecture but we made a small modification to make the model smaller.
We used a loss function where pixel-wise softmax is combined with cross entropy.
.png)
.png)
In attempt of reducing the loss, we did a post-processing on the prediction results. We applied the concept of watershed segmentation in order to point out the certain foreground regions and remove regions in the prediction image which seem to be noises.
The numbered images in the figure above indicates the stpes we took in the post-processing. To name those steps in slightly more detail:
* 1. Convertion into grayscale
* 2. Conversion into binary image
* 3. Morphological transformation: Closing
* 4. Determination of the certain background
* 5. Calculation of the distance
* 6. Determination of the certain foreground
* 7. Determination of the unknown region
* 8. Application of watershed
* 9. Determination of the final resultThe first step is there just in case the input image has more than 1 color channel (e.g. RGB image has 3 channels)
Convert the gray-scale image into binary image by processing the image with a threshold value: pixels equal to or lower than 127 will be pushed down to 0 and greater will be pushed up to 255. Such process is compulsory as later transformation processes takes in binary images.
We used morphologyEX() function in cv2 module which removes black noises (background) within white regions (foreground).
We used dilate() function in cv2 module which emphasizes/increases the white region (foreground). By doing so, we connect detached white regions together - for example, connecting detached cell membranes together - to make ensure the background region.
This step labels the foreground with a color code: red color indicates farthest from the background while blue color indicates closest to the background.
Now that we have an idea of how far the foreground is from the background, we apply a threshold value to decide which part could surely be the foreground.
The threshold value is the maximum distance (calculated from the previous step) multiplied by a hyper-parameter that we have to manually tune. The greater the hyper-parameter value, the greater the threshold value, and therefore we will get less area of certain foreground.
From previous steps, we determined sure foreground and background regions. The rest will be classified as 'unknown' regions.
We applied connectedComponents() function from the cv2 module on the foreground to label the foreground regions with color to distinguish different foreground objects. We named it as a 'marker'.
After applying watershed() function from cv2 module on the marker, we obtained an array of -1, 1, and many others.
* -1 = Border region that distinguishes foreground and background
* 1 = Background regionTo see the result, we created a clean white page of the same size with the input image. then we copied all the values from the watershed result to the white page except 1, which means that we excluded the background.
Optimizer
Learning Rate
Lowest Loss
Epoch
Highest Accuracy
Epoch
SGD
0.001
0.196972
1445
0.921032
1855
0.005
0.205802
1815
0.918425
1795
0.01
0.193328
450
0.922908
450
RMS_prop
0.0001
0.203431
185
0.924543
230
0.0002
0.193456
270
0.926245
500
0.001
0.268246
1655
0.882229
1915
Adam
0.0001
0.194180
140
0.924470
300
0.0005
0.185212
135
0.925519
135
0.001
0.222277
165
0.912364
180We chose the best learning rate that fits the optimizer based on how fast the model converges to the lowest error. In other word, the learning rate should make model to reach optimal solution in shortest epoch repeated. However, the intersting fact was that the epochs of lowest loss and highest accuracy were not corresponding. This might be due to the nature of loss function (Loss function is log scale, thus an extreme deviation might occur). For example, if the softmax probability of one pixel is 0.001, then the -log(0.001) would be 1000 which is a huge value that contributes to loss. For consistency, we chose to focus on accuracy as our criterion of correctness of model.
Accuracy and Loss Graph
SGD
(lr=0.01,momentum=0.99) RMS prop (lr=0.0002) Adam (lr=0.0005)
We used two different optimizers (SGD, RMS PROP, and Adam). In case of SGD the momentum is manually set (0.99) whereas in case of other optimizers (RMS Prop and Adam) it is calculated automatically.
Model trained with SGD can be downloaded via dropbox: https://www.dropbox.com/s/ge9654nhgv1namr/model_epoch_2290.pwf?dl=0
Model trained with RMS prop can be downloaded via dropbox: https://www.dropbox.com/s/cdwltzhbs3tiiwb/model_epoch_440.pwf?dl=0
Model trained with Adam can be downloaded via dropbox: https://www.dropbox.com/s/tpch6u41jrdgswk/model_epoch_100.pwf?dl=0
Input Image
Results comparsion
original image mask
RMS prop optimizer
(Accuracy 92.48 %) SGD optimizer (Accuracy 91.52 %) Adam optimizer (Accuracy 92.55 %)
Following modules are used in the project:
* python >= 3.6
* numpy >= 1.14.5
* torch >= 0.4.0
* PIL >= 5.2.0
* scipy >= 1.1.0
* matplotlib >= 2.2.2[1] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation, http://arxiv.org/pdf/1505.04597.pdf
[2] P.Y. Simard, D. Steinkraus, J.C. Platt. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis, http://cognitivemedium.com/assets/rmnist/Simard.pdf
No open issues yet, or sync has not completed.