DATA CLEANING

Data Cleaning (Data Preprocessing Guide)

Data Cleaning is the process of detecting and fixing errors, inconsistencies, and missing values in datasets before analysis or modeling.

It’s one of the most critical steps in Data Science and Machine Learning.

🧠 Why Data Cleaning Matters

✔ Improves model accuracy

✔ Ensures reliable insights

✔ Removes noise and errors

✔ Saves time during analysis

⚠️ Common Data Issues

1️⃣ Missing Values

Null or empty fields

2️⃣ Duplicates

Repeated records

3️⃣ Inconsistent Data

Different formats (e.g., “USA” vs “us”)

4️⃣ Outliers

Extreme values that distort analysis

5️⃣ Incorrect Data Types

Numbers stored as strings, etc.

⚙️ Data Cleaning Steps

🔹 1. Handle Missing Values

import pandas as pd

df = pd.read_csv("data.csv")

df = df.fillna(df.mean())

🔹 2. Remove Duplicates

df = df.drop_duplicates()

🔹 3. Fix Data Types

df['age'] = df['age'].astype(int)

🔹 4. Normalize Data

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

df[['salary']] = scaler.fit_transform(df[['salary']])

🔹 5. Handle Outliers

df = df[df['salary'] < df['salary'].quantile(0.95)]

📊 Tools for Data Cleaning

Python

Pandas

NumPy

OpenRefine

🚀 Best Practices

✔ Always explore data first (EDA)

✔ Keep original data unchanged

✔ Document cleaning steps

✔ Automate pipelines

#DataCleaning

#DataScience

#MachineLearning

#Python

#Pandas

#DataPreprocessing

#DataAnalytics

#BigData