Data Handling Using Pandas

About this unit

This unit introduces data analysis in Python with the Pandas library, supported by NumPy. You will create Series (one-dimensional) and DataFrames (two-dimensional) from lists, dictionaries and CSV files, select and change rows and columns, filter data with conditions, and join tables. You will then summarise data with statistical functions, group and pivot it, handle missing values, and move data between Pandas and a MySQL database.

Key ideas

• NumPy arrays hold elements of one type and support fast element-wise maths.
• A Series is a labelled one-dimensional array; a DataFrame is a labelled two-dimensional table.
• loc selects by labels (end label included); iloc selects by positions (end excluded).
• Boolean indexing such as df[df['Marks'] > 60] filters rows.
• merge(), concat(), read_csv() and to_csv() combine, import and export data.
• Functions such as sum(), mean(), median(), mode(), quantile(), var() and std() summarise data; groupby() and pivot() reshape it.
• Missing values (NaN) are found with isnull() and handled with dropna() or fillna().

How to prepare

Type the examples into Python and check the output yourself; most questions show short code and ask for the result, shape or number of rows. Learn the default axis of each function (columns, axis = 0) and remember that Pandas statistics skip NaN values.