Kaggle DSTL Dataset and Dataloader


#1

Hi @jakub_czakon,

I’m trying to develop a Dataset and Dataloader for the Kaggle DSTL Satellite Imagery Feature Detection challenge dataset.

The original dataset contains 16-band images, 3-band images,1 P-Band image, train csv file, grid sizes csv file and the shapes file containing polygons that can be used to generate training masks.

The NN model need a concatenated input dataset of 20-channels containing image data (X_train), and masks (y_train).

What would be the appropriate way to develop the Dataset class and Dataloader class for the DSTL dataset?

My workflow is:

  1. Load the high-res image, get the shape of the image
  2. Up-scale the low-res image
  3. Align the M and A-Band images to the RGB-Band image.
  4. Concatenate the RGB, P, M and A-Band images into a single X_train sample
  5. Generate the ground truth image mask from the shapes polygon file, and create a high-res 10-ch image mask corresponding to the 10 different class to be segmented
  6. Crop 512x512 regions, and generate around 2500 images from a single high-resolution image and the image mask.
  7. Save the cropped images to disk, i.e. cache them.

I have normal python scripts that do all this at the moment.

The Dataloader would have to use the Dataset class and just get a batch of 512x512x20 shaped X_train and 512x512x10 shaped y_train samples.

I was wondering if I were to implement a Dataset and Dataloader, at what point should I use the Steps library, since most of the steps from 1) to 7) happen in the dataset class.


#2

I would suggest that you check out https://github.com/minerva-ml

We are currently participating in the mapping challenge on crowdai https://www.crowdai.org/challenges/mapping-challenge where the objective is to build the best model for building instance segmentation.
We are open-sourcing our solution and you are welcome to use it. The loaders are defined here:
https://github.com/minerva-ml/open-solution-mapping-challenge/blob/master/loaders.py


#3

Do you have the dataset and data-loader for the Kaggle DSTL competition, that some of the guys from DeepSense.ai participate on?

I looked at the CrowdAI one and it is slightly different. If you have it ready, it would help save some time, redeveloping the classes for the DSTL dataset, since I am working on that dataset.


#4

Unfortunately I don’t have that one.

I can ask around but that competition took place when pytorch was really young so I don’t think we have that.


#5

Ah, I understand.

I’ll try to adapt the CrowdAI dataset and dataloaders to match the Kaggle DSTL one.