Pandas Basics: Series, DataFrames & Data Cleaning

Pandas is the Python library for working with data sets — analyzing, cleaning, exploring and manipulating them. The name references both “Panel Data” and “Python Data Analysis”; it was created by Wes McKinney in 2008 and it’s built on top of NumPy.

Pandas answers questions about data:

• Is there a correlation between two columns?
• What is the average / max / min value?
• Which rows are wrong, empty or duplicated — and how do I fix them?

That last one is huge: real data is messy, and Pandas is how you clean it.

# pip install pandas

import pandas as pd      # pd is the universal convention

print(pd.__version__)

The two core objects

Object	What it is	Analogy
Series	one-dimensional labelled array	a single column
DataFrame	two-dimensional labelled table	the whole spreadsheet

The idea that makes Pandas different: the index

NumPy arrays are addressed by position — element 0, 1, 2. Pandas adds one thing on top: every row carries a label, and the collection of labels is called the index.

That sounds minor and changes everything. Labels mean your data can be addressed by what it is (df.loc["day2"]) rather than where it happens to sit. It also means Pandas can align data automatically: when you combine two Series, Pandas matches rows by label, not by position, so adding sales data indexed by date to costs indexed by the same dates just works — even if the rows are in a different order or one has gaps.

Keep this in mind as you read: almost every confusing Pandas behaviour makes sense once you ask “what is it doing with the index?”

Series — a single column

import pandas as pd

a = [1, 7, 2]
myvar = pd.Series(a)
print(myvar)
# 0    1
# 1    7
# 2    2
# dtype: int64

Labels

If you don’t specify labels, values are numbered from 0. Use that label to access a value:

print(myvar[0])      # => 1

Custom index labels

myvar = pd.Series([1, 7, 2], index=["x", "y", "z"])
print(myvar)
# x    1
# y    7
# z    2

print(myvar["y"])    # => 7    access by your own label

From a dictionary

Keys become the index automatically:

calories = {"day1": 420, "day2": 380, "day3": 390}
myvar = pd.Series(calories)
print(myvar)
# day1    420
# day2    380
# day3    390

# Select only some keys with index=
myvar = pd.Series(calories, index=["day1", "day2"])
# day1    420
# day2    380

Filtering a Series

print(myvar[myvar > 400])
# day1    420        <- only rows where the condition is True

DataFrames — the whole table

import pandas as pd

data = {
    "calories": [420, 380, 390],
    "duration": [50, 40, 45],
}

df = pd.DataFrame(data)
print(df)
#    calories  duration
# 0       420        50
# 1       380        40
# 2       390        45

Locating rows with loc

print(df.loc[0])          # one row (returned as a Series)
# calories    420
# duration     50

print(df.loc[[0, 1]])     # multiple rows (returned as a DataFrame)
#    calories  duration
# 0       420        50
# 1       380        40

Named indexes

df = pd.DataFrame(data, index=["day1", "day2", "day3"])
print(df)
#       calories  duration
# day1       420        50
# day2       380        40
# day3       390        45

print(df.loc["day2"])     # locate by your own label

loc vs iloc — the key distinction

	Selects by	Example
loc	label (your index name)	df.loc["day2"]
iloc	integer position (0, 1, 2…)	df.iloc[0]

df.iloc[0]           # first row by POSITION, whatever it's labelled
df.loc["day1"]       # the row LABELLED "day1"

df.loc["day1", "calories"]   # => 420   (row label, column name)
df.iloc[0, 0]                # => 420   (row 0, column 0)
df.iloc[0:2]                 # first two rows

Why both exist. With a default index the two look identical — df.loc[0] and df.iloc[0] both give the first row, because the label is 0. They diverge the moment the index isn’t a plain 0,1,2 sequence:

df = pd.DataFrame(data, index=["day1", "day2", "day3"])
df.iloc[0]        # works - "the first row"
df.loc[0]         # KeyError! there is no row LABELLED 0

# And after filtering, labels survive but positions shift:
big = df[df["calories"] > 385]     # keeps day1 and day3
big.iloc[1]       # => day3's row  (second row of the result)
big.loc["day3"]   # => the same row, addressed by name

That last case is where people get bitten: after a filter or a sort, the labels stay attached to their original rows while the positions renumber. If you want “the row for day3”, use loc; if you want “whatever is now second”, use iloc.

One more asymmetry worth memorising: loc slices are inclusive of the end label, while iloc slices exclude the end position, matching normal Python:

df.loc["day1":"day2"]   # includes day2  (2 rows)
df.iloc[0:2]            # excludes row 2 (2 rows: 0 and 1)

Selecting columns

df["calories"]              # one column -> a Series
df[["calories", "duration"]]  # several columns -> a DataFrame

Reading data from files

CSV

CSV (comma-separated values) is the simplest way to store a big data set.

import pandas as pd

df = pd.read_csv('data.csv')
print(df)                # prints the first & last 5 rows if it's large
print(df.to_string())    # prints the ENTIRE DataFrame

Pandas only displays the first and last 5 rows of a big frame. You can change that limit:

print(pd.options.display.max_rows)     # => 60   (the current cap)
pd.options.display.max_rows = 200      # raise it

JSON

df = pd.read_json('data.json')

# ...or from a Python dict already in memory:
data = {
    "Duration": {"0": 60, "1": 60, "2": 60, "3": 45},
    "Pulse":    {"0": 110, "1": 117, "2": 103, "3": 109},
}
df = pd.DataFrame(data)

Inspecting your data

The first four commands you run on any new data set:

df.head()       # first 5 rows (the default)
df.head(10)     # first 10 rows
df.tail()       # last 5 rows - great for spotting trailing junk
df.info()       # column names, non-null counts, dtypes, memory use
df.describe()   # count/mean/std/min/quartiles/max for numeric columns
df.shape        # => (rows, columns)

info() is the most valuable of these — the non-null count per column immediately tells you where the empty cells are.

df.info()
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 169 entries, 0 to 168
# Data columns (total 4 columns):
#  #   Column    Non-Null Count  Dtype
# ---  ------    --------------  -----
#  0   Duration  169 non-null    int64
#  1   Pulse     169 non-null    int64
#  2   Maxpulse  169 non-null    int64
#  3   Calories  164 non-null    float64   <- 5 missing values!

Data cleaning

Data cleaning means fixing bad data in your data set. Bad data comes in four flavours:

1. Empty cells
2. Data in the wrong format
3. Wrong data
4. Duplicates

1. Empty cells — remove the rows

df = pd.read_csv('data.csv')

new_df = df.dropna()          # returns a NEW DataFrame; original untouched
print(new_df.to_string())

df.dropna(inplace=True)       # modifies the ORIGINAL, returns nothing

Understanding inplace=True

This parameter appears on most cleaning methods and trips up nearly everyone, so it’s worth being precise about.

• Without inplace the method returns a new DataFrame and leaves the original untouched. You must capture the result: new_df = df.dropna().
• With inplace=True the method modifies the original and returns None. You must not capture it.

The classic bug is mixing the two:

df = df.dropna(inplace=True)     # WRONG - df is now None!
print(df)                        # => None

df.dropna(inplace=True)          # right: modify in place, don't assign
new_df = df.dropna()             # also right: keep the original, take a copy

It’s the same trap as xs.sort() returning None for lists — Python’s convention is that methods which mutate in place return nothing.

Which should you prefer? Assignment (new_df = df.dropna()) is generally safer — it keeps your raw data intact so you can re-run a cleaning step after getting it wrong, which you will. Modern Pandas is also gradually moving away from inplace, so the assignment style is more future-proof.

1b. Empty cells — fill them instead

Deleting whole rows over one missing cell is wasteful. fillna() replaces them:

df = pd.read_csv('data.csv')
df.fillna(130, inplace=True)         # fill EVERY empty cell with 130

# Only fill one specific column:
df.fillna({"Calories": 130}, inplace=True)

1c. Fill with mean, median or mode

The most common approach — fill gaps with a typical value for that column:

df = pd.read_csv('data.csv')

x = df["Calories"].mean()        # average: sum / count
x1 = df["Calories"].median()     # middle value once sorted
x2 = df["Calories"].mode()[0]    # most frequent value ([0] since mode can tie)

print(x, x1, x2)

df.fillna({"Calories": x}, inplace=True)

Statistic	Meaning	Best when
mean	the average	data is roughly symmetric
median	the middle value when sorted	there are outliers
mode	the most frequent value	the data is categorical

Why the choice matters. Suppose five houses cost 30, 32, 35, 38 and 900 lakhs. The mean is 207 — a value no house is anywhere near, dragged up entirely by one mansion. The median is 35, which actually describes a typical house. That’s the rule: outliers pull the mean but barely move the median, so use the median whenever a column has extreme values (prices, incomes, response times). Use the mode when the column isn’t numeric at all — you can’t average “red”, “blue”, “red”, but you can say the most common value is “red”.

And why filling can be dangerous. Every value you invent makes the data look more certain than it is. Filling 40% of a column with its mean creates an artificial spike at the average, shrinks the apparent variance, and weakens any correlation that column has with others. Before filling, ask why the values are missing:

Missing because	Better action
A sensor randomly dropped readings	Filling with mean/median is reasonable
The field is optional and users skipped it	Consider a real “unknown” category
Values are missing systematically (e.g. only for low earners)	Filling introduces bias — investigate first
The column is mostly empty	Drop the column, not the rows

dropna() is the honest alternative, but it has its own cost: one empty cell discards the entire row, including its good data. With a small data set that can delete more information than it saves.

2. Wrong format

df['Date'] = pd.to_datetime(df['Date'])   # convert a column to real dates
df.dropna(subset=['Date'], inplace=True)  # drop rows whose date couldn't parse

3. Wrong data

“Wrong data” isn’t empty or badly formatted — it’s just wrong, like someone typing 199 instead of 1.99. Fix a single cell directly:

df.loc[7, 'Duration'] = 45     # row 7's Duration was 450 - clearly a typo

For larger data sets, set rules and enforce boundaries by looping over the index:

# Cap any duration above 120 at 120
for x in df.index:
    if df.loc[x, "Duration"] > 120:
        df.loc[x, "Duration"] = 120

Or delete the offending rows entirely:

for x in df.index:
    if df.loc[x, "Duration"] > 120:
        df.drop(x, inplace=True)

4. Duplicates

print(df.duplicated())            # True for every row that is a repeat
df.drop_duplicates(inplace=True)  # remove them from the original

Correlations — finding relationships

corr() computes how strongly each pair of numeric columns moves together.

df.corr()

The result is a table of numbers from -1 to 1:

Value	Meaning
1	perfect positive correlation — both rise together
0.6 to 1	good positive relationship
0	no relationship
-0.6 to -1	good negative relationship — one rises as the other falls
-1	perfect negative correlation

A rough rule: anything above 0.6 (or below -0.6) is a meaningful relationship worth investigating.

df.corr()
#           Duration     Pulse  Maxpulse  Calories
# Duration  1.000000 -0.155408  0.009403  0.922717   <- strong! long workouts burn more
# Pulse    -0.155408  1.000000  0.786535  0.025120
# Maxpulse  0.009403  0.786535  1.000000  0.203813
# Calories  0.922717  0.025120  0.203813  1.000000

Remember: correlation is not causation — see cognitive biases for why that distinction matters.

Quick reference

Task	Code
Read a CSV	pd.read_csv('data.csv')
First / last rows	df.head() / df.tail()
Structure & nulls	df.info()
Summary statistics	df.describe()
Row by label / position	df.loc[label] / df.iloc[0]
One column	df["col"]
Drop empty rows	df.dropna(inplace=True)
Fill empty cells	df.fillna(value, inplace=True)
Column average	df["col"].mean()
Fix one cell	df.loc[7, 'col'] = 45
Remove duplicates	df.drop_duplicates(inplace=True)
Correlations	df.corr()

Takeaways

• Series = one column, DataFrame = the whole table — both carry labels.
• loc selects by label, iloc by integer position. That’s the single most common Pandas confusion.
• Start every data set with head(), info() and describe().
• Cleaning is the real work: dropna / fillna for gaps (mean, median or mode), .loc assignment or rules for wrong values, drop_duplicates for repeats.
• inplace=True modifies the original; without it you get a new DataFrame back.

Next: Exploratory Data Analysis — visualizing and understanding a data set once it’s clean.