Neural Networks in CUDA from Scratch - 3. Multiple Layers

We are extending our CUDA Kernel so it can work through multiple layers in a single call.

The code can be found here: https://github.com/ThoenigAdrian/Neur...

What we did so far:

Part 1 - Implement Cuda Kernel which can execute the sigmoid activation function for many values at once
Part 2 - Add the Linear Layer (Dot Product) to our Kernel. So far we only had inputs and outputs but no hidden layers.

This will be a multipart series:
I am planning to cover the following topics
1. Implementing activation layer
2. Linear layer
3. Feedforward with multiple layers
4. Forward pass with multiple inputs
5. Backpropagation
6. Applying gradient descent
7. Training loop with multiple iterations
8. Cost function
9. Scaling

If you are interested in general CUDA Programming, I can recommend the following the following resources:

Cuda Crash Course: @CoffeeBeforeArch

   • CUDA Crash Course: Vector Addition  

Cuda Crash Cource v2: @CoffeeBeforeArch

   • C++ Crash Course (v2): Text Editors and Co...  

NVIDIA CUDA Tutorial @Creel

   • NVIDIA CUDA Tutorial 1: Introduction  

Nvidia Documentation: https://docs.nvidia.com/cuda/cuda-c-p...
(Yes the official documentation, it's actually really good)