gradient-descent

A simple implementation of Gradient Descent written in C++. This project demonstrates how a machine learning model can learn the parameters of a straight line by repeatedly reducing its prediction error.

The example learns the linear equation:

y = kx + b

from a small set of training data.

---

What is Gradient Descent?

Gradient Descent is one of the most important optimization algorithms in machine learning.

Its goal is to find the values of a model's parameters that minimize a loss (error) function.

Instead of solving for the parameters directly, Gradient Descent starts with random guesses and gradually improves them by taking small steps in the direction that reduces the error.

For a linear model:

y = kx + b

the parameters are:

• k = slope
• b = intercept

At each iteration the algorithm:

1. Predicts outputs using the current parameters.
2. Measures the prediction error.
3. Computes how the error changes with respect to each parameter (the gradient).
4. Updates the parameters.
5. Repeats until the error is minimized.

---

Example

Training data:

x	y
1	3
2	5
3	7
4	9
5	11

The hidden equation is:

y = 2x + 1

Initially, the model starts with

k = 0
b = 0

Its predictions are poor, producing a large error.

After many Gradient Descent updates, it learns

k ≈ 2
b ≈ 1

which perfectly fits the data.

---

Features

• Beginner-friendly implementation
• Demonstrates Gradient Descent step by step
• Learns the slope and intercept of a line
• Prints training progress over multiple epochs
• No external libraries required
• Well-commented and easy to understand

---

Project Structure

.
├── gradient_descent.cpp
└── README.md

---

Compilation

Compile using g++:

g++ gradient_descent.cpp -std=c++17 -O2 -o gradient_descent

Run the program.

Windows

gradient_descent.exe

Linux / macOS

./gradient_descent

---

Sample Output

Learning line y = kx + b

Epoch   1   Loss = 33.000000   k = 1.700000   b = 0.700000
Epoch  25   Loss = 0.034818    k = 2.074345   b = 0.731927
Epoch  50   Loss = 0.004275    k = 2.026049   b = 0.906392
Epoch  75   Loss = 0.000525    k = 2.009111   b = 0.967258
Epoch 100   Loss = 0.000064    k = 2.003187   b = 0.988553

Learned model:

y = 2.000000x + 1.000000

---

How the Algorithm Works

1. Initialize the parameters k and b.
2. Predict each output using

ŷ = kx + b

3. Measure the prediction error.
4. Compute the gradients of the loss with respect to k and b.
5. Update both parameters using Gradient Descent.
6. Repeat until the loss becomes very small.

---

Formula Used

Linear Model:

ŷ = kx + b

Mean Squared Error (per sample):

E = ½(y - ŷ)²

Gradient Descent Update:

k = k - η ∂E/∂k

b = b - η ∂E/∂b

where

• η = learning rate
• ∂E/∂k = gradient with respect to the slope
• ∂E/∂b = gradient with respect to the intercept

---

Advantages

• Simple and intuitive optimization algorithm
• Works with many machine learning models
• Scales well to large datasets
• Foundation of modern machine learning and deep learning
• Easy to visualize and understand

---

Limitations

• Sensitive to the learning rate
• Can require many iterations
• May converge slowly on complex problems
• Can become trapped in local minima for non-convex optimization

---

Common Applications

• Linear Regression
• Logistic Regression
• Neural Networks
• Deep Learning
• Recommendation Systems
• Computer Vision
• Natural Language Processing

---

Learning Objectives

This project demonstrates:

• Linear Regression
• Gradient Descent
• Loss Functions
• Mean Squared Error (MSE)
• Partial Derivatives
• Parameter Optimization
• Model Training
• Supervised Machine Learning

---

Requirements

• C++17 or later
• GCC / g++
• Any standard C++ compiler

---

License

This project is licensed under the MIT License.