Numpy Tutorial Part 2 – Vital Functions for Data Analysis

Numpy is the core package for data analysis and scientific computing in Python. This is part 2 of a mega numpy tutorial. In this part, I cover the advanced features of numpy that are essential for data analysis and manipulation.

This post has interactive code, click ‘Run’ or press Ctrl+Enter on any code block to execute it directly in your browser. The first run may take a few seconds to initialize.

Introduction

In part 1 of the numpy tutorial, we learned why numpy matters for working with datasets in Python. We covered how to create arrays, explore them, do indexing, reshaping, flattening, and generate random numbers.

In part 2, I pick up where we left off. We will tackle more advanced but essential topics for data analysis.

I will assume you know basic Python, some math, and have read part 1. The best approach is to read the full article once, then come back and run the examples yourself.

Let’s begin.

1. How to get index locations that satisfy a given condition using np.where?

In part 1, you saw how to extract items from an array using boolean indexing. But sometimes you need the index positions of items that match a condition, not the items themselves.

np.where finds exactly those positions.

Once you have the positions, extract the values using the array’s take method.

np.where also accepts two optional arguments: x and y. When the condition is true, it yields x. Otherwise, it yields y.

Below, I create an array that labels each element as 'gt5' or 'le5' based on whether it exceeds 5.

You can also find the positions of maximum and minimum values.

2. How to import and export data as a csv file?

A standard way to import datasets is np.genfromtxt. It handles web URLs, missing values, multiple delimiters, and irregular columns.

A simpler alternative is np.loadtxt, which assumes no missing values.

Let’s read a .csv file from a URL. Since every element in a numpy array must share the same data type, text columns get imported as nan by default. The filling_values argument lets you replace those missing entries.

# Turn off scientific notation
np.set_printoptions(suppress=True)

# Import data from csv file url
path = 'https://raw.githubusercontent.com/selva86/datasets/master/Auto.csv'
data = np.genfromtxt(path, delimiter=',', skip_header=1, filling_values=-999, dtype='float')
data[:3]  # see first 3 rows
#> array([[   18. ,     8. ,   307. ,   130. ,  3504. ,    12. ,    70. ,
#>             1. ,  -999. ],
#>        [   15. ,     8. ,   350. ,   165. ,  3693. ,    11.5,    70. ,
#>             1. ,  -999. ],
#>        [   18. ,     8. ,   318. ,   150. ,  3436. ,    11. ,    70. ,
#>             1. ,  -999. ]])

Notice that the last column shows -999 everywhere. That happened because we set dtype='float'. The last column in the file contains text values, and numpy could not convert them to floats.

2.1 How to handle datasets that has both numbers and text columns?

If you need the text column intact, set the dtype to 'object' or None.

# data2 = np.genfromtxt(path, delimiter=',', skip_header=1, dtype='object')
data2 = np.genfromtxt(path, delimiter=',', skip_header=1, dtype=None)
data2[:3]  # see first 3 rows
#> array([( 18., 8,  307., 130, 3504,  12. , 70, 1, b'"chevrolet chevelle malibu"'),
#>        ( 15., 8,  350., 165, 3693,  11.5, 70, 1, b'"buick skylark 320"'),
#>        ( 18., 8,  318., 150, 3436,  11. , 70, 1, b'"plymouth satellite"')],
#>       dtype=[('f0', '<f8'), ('f1', '<i8'), ('f2', '<f8'), ('f3', '<i8'), ('f4', '<i8'), ('f5', '<f8'), ('f6', '<i8'), ('f7', '<i8'), ('f8', 'S38')])

Now the text column is preserved. To export an array as a csv file, use np.savetxt.

# Save the array as a csv file
np.savetxt("out.csv", data, delimiter=",")

3. How to save and load numpy objects?

Sometimes you want to save large numpy arrays to disk. This avoids re-running data transformation code every time.

Numpy provides .npy and .npz file formats for this. Use np.save to store a single ndarray as a .npy file. Use np.savez to store multiple arrays in one .npz file. Both can be loaded back with np.load.

# Save single numpy array object as .npy file
np.save('myarray.npy', arr2d)

# Save multiple numpy arrays as a .npz file
np.savez('array.npz', arr2d_f, arr2d_b)

Load back the .npy file:

# Load a .npy file
a = np.load('myarray.npy')
print(a)
#> [[0 1 2]
#>  [3 4 5]
#>  [6 7 8]]

Load back the .npz file:

# Load a .npz file
b = np.load('array.npz')
print(b.files)
b['arr_0']
#> ['arr_0', 'arr_1']
#>
#> array([[ 0.,  1.,  2.],
#>        [ 3.,  4.,  5.],
#>        [ 6.,  7.,  8.]])

4. How to concatenate two numpy arrays column-wise and row-wise?

There are three ways to concatenate numpy arrays:

• Method 1:np.concatenate with axis=0 (rows) or axis=1 (columns)
• Method 2:np.vstack and np.hstack
• Method 3:np.r_ and np.c_

All three produce the same result. One key difference: np.r_ and np.c_ use square brackets instead of parentheses.

Let me create the arrays first.

Now let’s stack them vertically (row-wise).

Now let’s stack them horizontally (column-wise).

You can also use np.r_ to build complex 1D sequences.

5. How to sort a numpy array based on one or more columns?

Let’s sort a 2D array based on the first column.

If you use np.sort with axis=0, each column gets sorted independently. This breaks the integrity of rows. Values from different rows get mixed together.

To keep rows intact, use the indirect approach with np.argsort.

5.1 How to sort a numpy array based on 1 column using argsort?

np.argsort returns the index positions that would sort a 1D array. It does not sort the array itself.

How do you read this? The 0th item is smallest, the 3rd item is second smallest, and so on. Use these indices to reorder the array.

Now apply this idea to sort a 2D array by its first column. Run argsort on that column, then use the result to reorder the full array.

For descending order, reverse the argsorted index.

5.2 How to sort a numpy array based on 2 or more columns?

Use np.lexsort for multi-column sorting. Pass a tuple of columns. Place the primary sort column at the rightmost position in the tuple.

6. Working with dates

Numpy handles dates through np.datetime64. It supports precision down to nanoseconds. Create one using a standard YYYY-MM-DD string.

numpy.datetime64('2018-02-04T23:10:10')

You can include hours, minutes, and seconds down to nanoseconds. Let’s strip the time component from date64.

Adding a plain number increases the date by that many days. For other time units like months, hours, or seconds, use np.timedelta64.

To convert dt64 back to a string, use np.datetime_as_string.

When working with dates, you often need to filter business days. Use np.is_busday() to check, and np.busday_offset() to shift dates by business days.

6.1 How to create a sequence of dates?

Use np.arange with datetime64 objects. It works just like creating numeric ranges.

6.2 How to convert numpy.datetime64 to datetime.datetime object?

Use the .tolist() method to convert a datetime64 to a Python datetime.date object.

datetime.date(2018, 2, 4)

Once you have a datetime.date object, you can extract the day, month, year, and more.

7. Advanced numpy functions

7.1 vectorize – Make a scalar function work on vectors

np.vectorize() lets a function designed for single numbers work on entire arrays.

Here is a simple example. The function foo squares odd numbers and halves even numbers. It works fine on individual values but fails on arrays.

x = 10 returns  5.0
x = 11 returns  121

Now let’s vectorize foo() so it works on arrays.

x = [10, 11, 12] returns  [   5.  121.    6.]
x = [[10, 11, 12], [1, 2, 3]] returns  [[   5.  121.    6.]
 [   1.    1.    9.]]

The optional otypes parameter specifies the output data type. Setting it makes the vectorized function run faster.

7.2 apply_along_axis – Apply a function column-wise or row-wise

np.apply_along_axis applies a 1D function across rows or columns of a 2D array. It takes three arguments:

1. A function that operates on a 1D vector
2. The axis (1 = row-wise, 0 = column-wise)
3. The array

Let’s find the difference between the max and min value in each row.

Without apply_along_axis, you would need a for-loop. This approach is cleaner and works for any custom function.

7.3 searchsorted – Find the location to insert so the array will remain sorted

np.searchsorted tells you where to insert a value to keep the array sorted.

With a smart hack by Radim, you can use searchsorted for weighted random sampling. It is much faster than np.random.choice.

# Randomly choose an item from a list based on a predefined probability
lst = range(10000)  # the list
probs = np.random.random(10000); probs /= probs.sum()  # probabilities

%timeit lst[np.searchsorted(probs.cumsum(), np.random.random())]
%timeit np.random.choice(lst, p=probs)
#> 36.6 µs ± 3.93 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
#> 1.02 ms ± 7.16 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

7.4 How to add a new axis to a numpy array?

Sometimes you need to convert a 1D array into a 2D array without adding data. This is useful for saving a 1D array as a single column in a CSV, or for concatenating with another 2D array.

Use np.newaxis to insert a new axis. This raises the array’s dimension by one.

7.5 More Useful Functions

Digitize

np.digitize returns the bin index for each element in an array.

Clip

np.clip caps values within a range. Values below the lower limit get replaced by that limit. Same for the upper limit.

Histogram and Bincount

Both np.histogram and np.bincount count frequencies, but in different ways.

bincount counts how often each integer appears, from 0 to the max value. It includes zeros for missing values. histogram groups values into bins you define and counts how many fall in each bin.

8. What is missing in numpy?

We have covered many techniques for data manipulation with numpy. But there are things numpy cannot do directly:

1. Merge two 2D arrays based on a common column
2. Create pivot tables
3. Do 2D cross tabulations
4. Compute grouped statistics (like mean by category)
5. And more…

These gaps are filled by the pandas library, which I will cover in the upcoming pandas tutorial. Meanwhile, test your skills with these numpy practice exercises.