PCA (ACP) From Scratch Using Python

A complete from-scratch implementation of Principal Component Analysis (PCA)
(Analyse en Composantes Principales – ACP) using NumPy and Matplotlib.

This project follows the classical PCA pipeline and focuses on understanding both:

• The mathematical foundations of PCA
• The practical implementation step by step

The algorithm is applied to a small Iris flowers dataset containing 15 individuals and 3 variables.

---

1. Introduction

Principal Component Analysis (PCA) is a statistical technique used to analyze datasets with multiple variables and reduce their dimensionality while preserving as much information as possible.

When the number of variables becomes large, visualizing relationships between variables becomes difficult. PCA solves this problem by projecting the data into a lower-dimensional space formed by new axes called principal components.

These axes are chosen so that:

• They maximize variance
• They minimize information loss
• They allow visualization in 2D or 3D

---

2. Dataset Description

The dataset contains:

• 15 Iris flowers
• 3 variables:

x1 : Sepal length
x2 : Sepal width
x3 : Petal length

Matrix size:

15 × 3

---

3. Objectives of the Project

• Implement PCA manually using Python
• Display intermediate matrices
• Compute eigenvalues and eigenvectors
• Calculate explained variance
• Project data onto principal components
• Visualize individuals and variables

---

4. PCA Methodology (Mathematical Steps)

4.1 Mean Calculation

For each variable j, compute its mean:

μ_j = (1 / n) * Σ x_ij

Meaning:
Add all values of variable j, then divide by the number of samples n.

---

4.2 Data Centering

Subtract the mean from each value:

Xc(i, j) = X(i, j) − μ_j

Meaning:
Each column will have an average equal to zero.

---

4.3 Standardization

Divide each centered value by its standard deviation:

Y(i, j) = Xc(i, j) / σ_j

Meaning:
All variables will have the same scale.

---

4.4 Normalization

Z = Y / √n

Meaning:
Divide the standardized matrix by the square root of the number of samples.

---

4.5 Correlation Matrix

R = corr(Z)

Meaning:
R measures how strongly variables are related to each other.

Properties:

• Diagonal values = 1
• Off-diagonal values = correlation between variables

---

4.6 Eigenvalues and Eigenvectors

Solve:

R · v = λ · v

Where:

• v : eigenvector (direction of a principal component)
• λ : eigenvalue (amount of variance)

---

4.7 Explained Variance

ExplainedVariance_k = ( λ_k / Σ λ ) × 100

Meaning:
Percentage of total information carried by component k.

---

4.8 Projection

Scores = Z × V

Meaning:
Transform data into the new PCA coordinate system.

---

5. Project Structure

PCA-From-Scratch/ ├── ACPtesting.py
├── images/
│ ├── Figure_1.png
│ ├── Figure_2.png
│ ├── Figure_3.png
│ ├── Figure_4.png
│ ├── Figure_5.png
│ ├── Figure_6.png
│ └── Figure_7.png
└── README.md

---

6. Installation

pip install numpy matplotlib

---

7. Execution

python ACPtesting.py

---

8. Results and Visualizations

Figure 1 – Projection on PC1 and PC2

Figure 2 – Initial Data Matrix

Figure 3 – Centered Matrix

Figure 4 – Standardized Matrix

Figure 5 – Normalized Matrix

Figure 6 – Correlation Matrix

Figure 7 – Correlation Circle

---

9. Interpretation

• PC1 captures the maximum variance
• PC2 captures the second highest variance
• Using PC1 and PC2 preserves most information
• Variables pointing in the same direction are positively correlated
• Variables pointing in opposite directions are negatively correlated

---

10. Author

Hachim Fernane
Master Student – Computer Science
University of Guelma

---

11. License

FEEL FREE TO USE JUST PRAY FOR ME