0209 Structured Data NumPy

中文版：NumPy 结构化数据

Structured Data: NumPy’s Structured Arrays

While often ourdatacanbewell represented by a homogeneous array of values, sometimes thisisnotthecase. This section demonstrates theuseofNumPy’s structured arrays and record arrays, which provide efficient storage for compound, heterogeneous data. While the patterns shown here are useful for simple operations, scenarios likethisoften lend themselves totheuseof Pandas Dataframes, which we’ll explore in Chapter 3.

import numpy as np

Imagine thatwehave several categories ofdataona number of people (say, name, age, and weight), and we’dliketostore these values foruseina Python program. It would be possible to store these in three separate arrays:

name = ['Alice', 'Bob', 'Cathy', 'Doug']
age = [25, 45, 37, 19]
weight = [55.0, 85.5, 68.0, 61.5]

Butthisisabit clumsy. There’s nothing herethattells usthatthethree arrays are related; it would be more natural ifwecould use a single structure to store allofthisdata. NumPy can handle this through structured arrays, which are arrays with compound data types.

Recall that previously we created a simple array using an expression like this:

x = np.zeros(4, dtype=int)

We can similarly create a structured array using a compound data type specification:

# Use a compound datatypefor structured arrays
data = np.zeros(4, dtype={'names':('name', 'age', 'weight'),
 'formats':('U10', 'i4', 'f8')})
print(data.dtype)

Here 'U10' translates to “Unicode string of maximum length 10,” 'i4' translates to “4-byte (i.e., 32 bit) integer,” and 'f8' translates to “8-byte (i.e., 64 bit) float.” We’ll discuss other options for these type codes in the following section.

Nowthatwe’ve created an empty container array, wecanfillthearray withourlists of values:

data['name'] = name
data['age'] = age
data['weight'] = weight
print(data)

Aswehadhoped, thedataisnow arranged together in one convenient block of memory.

The handy thing with structured arrays isthatyoucannowrefer to values either by index orbyname:

# Getallnames
data['name']

# Get first rowofdata
data[0]

# Getthenamefromthelastrow
data[-1]['name']

Using Boolean masking, this even allows youtodosomemore sophisticated operations such as filtering on age:

# Get names where ageisunder 30
data[data['age'] < 30]['name']

Notethatifyou’dliketodoany operations thatareanymore complicated than these, you should probably consider the Pandas package, covered inthenext chapter. As we’ll see, Pandas provides a Dataframe object, which is a structure built on NumPy arrays that offers a variety of useful data manipulation functionality similar towhatwe’ve shown here, aswellasmuch, much more.

Creating Structured Arrays

Structured array data types can be specified in a number of ways. Earlier, wesawthe dictionary method:

np.dtype({'names':('name', 'age', 'weight'),
 'formats':('U10', 'i4', 'f8')})

For clarity, numerical types can be specified using Python types or NumPy dtypes instead:

np.dtype({'names':('name', 'age', 'weight'),
 'formats':((np.str_, 10), int, np.float32)})

A compound typecanalsobe specified asalistof tuples:

np.dtype([('name', 'S10'), ('age', 'i4'), ('weight', 'f8')])

Ifthenames ofthetypes do not matter to you, you can specify the types alone inacomma-separated string:

np.dtype('S10,i4,f8')

The shortened string format codes may seem confusing, buttheyarebuilt on simple principles. The first (optional) character is < or >, which means “little endian” or “big endian,” respectively, and specifies the ordering convention for significant bits. The next character specifies thetypeofdata: characters, bytes, ints, floating points, andsoon (seethetable below). The last character or characters represents thesizeofthe object in bytes.

| Character | Description | Example | | 'b' | Byte | np.dtype('b') | | 'i' | Signed integer | np.dtype('i4') == np.int32 | | 'u' | Unsigned integer | np.dtype('u1') == np.uint8 | | 'f' | Floating point | np.dtype('f8') == np.int64 | | 'c' | Complex floating point| np.dtype('c16') == np.complex128| | 'S', 'a' | String | np.dtype('S5') | | 'U' | Unicode string | np.dtype('U') == np.str_ | | 'V' | Raw data (void) | np.dtype('V') == np.void |

More Advanced Compound Types

It is possible to define even more advanced compound types. For example, you can create atypewhere each element contains an array or matrix of values. Here, we’ll create adatatypewitha mat component consisting of a $3\times 3$ floating-point matrix:

tp = np.dtype([('id', 'i8'), ('mat', 'f8', (3, 3))])
X = np.zeros(1, dtype=tp)
print(X[0])
print(X['mat'][0])

Now each element in the X array consists of an id and a $3\times 3$ matrix. Why would youusethis rather than a simple multidimensional array, or perhaps a Python dictionary? The reason isthatthisNumPy dtype directly mapsontoaC structure definition, so the buffer containing the array content can be accessed directly within an appropriately written C program. Ifyoufind yourself writing a Python interface to a legacy C or Fortran library that manipulates structured data, you’ll probably find structured arrays quite useful!

RecordArrays: Structured Arrays withaTwist

NumPy also provides the np.recarray class, which is almost identical to the structured arrays just described, butwithone additional feature: fields can be accessed as attributes rather than as dictionary keys. Recall that we previously accessed theagesby writing:

data['age']

Ifweviewourdataasa record array instead, we can access this with slightly fewer keystrokes:

data_rec = data.view(np.recarray)
data_rec.age

The downside isthatfor record arrays, there issomeextra overhead involved in accessing the fields, evenwhenusing the same syntax. Wecanseethishere:

%timeit data['age']
%timeit data_rec['age']
%timeit data_rec.age

Whether the more convenient notation is worth the additional overhead will depend onyourown application.