Hi everybody! My week was full of python and pandas coolness. I refactored a central bit of code to traverse all directories in a tree and parse any metadata source it finds there. When I did that I learned that there was a lot of data and some of the files I was parsing are huge, so I had to optimize the code for performance.

Traversing all directories in a tree

The wise folks at python know that lots of people have to traverse directories recursively, so they built a function right into the os module that does this for you. It’s called, appropriately, walk. Here’s how I use it to find JSON files and process them:

import os

for in_dir in options.input_dir:

    for root, dirs., files in os.walk(in_dir):

        for file in files:

            fname = os.path.join(root,file)

            if fname[-5:] == ‘.json’:

                print “processing JSON %s “ % fname

My very first recursive function: Telling JSON files apart

These JSON files have a variety of dictionary structures in the, each depending on the kind of data it contains. In one file, Mike writes out a dictionary with a function name as the key and an integer (a count of the unit test scripts generated by his TestGenerator) as the value. This could have any kind of name on it, so I don’t want to use the name to identify the file type. I want to use “duck typing” and ask it what kind of file it is.

So I made a file describer, like this. I’m assuming that the file has already been read into a dict:

def describe_json_structure(data, description):

    key = data.keys()[0]

    description.append(type(data[key]))

        if type(data[key]) == type({}):

            describe_json_structure(data[key], description)

        elif type(data[key]) == type([]):

            description.append(type(data[key][0]))

return description

All this does is check the first item in a dict and see what the nature of its value is. If it’s a dict, then it passes the nested dict to the same exact function and runs it again. The description array tags along, getting updated with the types of all the layers.

When I want to use the description to tell which file type it is, I compare it to a pre-defined file-type list I set up:

jtype_static = [type({}), type([]), type([])]

jtype_calls = [type({}), type({}), type({}), type(1)]

jtype_report = [type({}), type({}), type({}), type({}), type(1)]

jtype_coverage = [type({}), type({}), type({}), type(u’x’)]

jtype_counters = [type(1)]

The compare function:

if description == jtype_coverage:

    #take care of bidness

 

Performance optimization

I had to get away from using a pandas DataFrame for constructing the big tabular structure everything about this coverage report fits into – it’s great for operating on the data when it’s in, but constructing it row by row as I have to do is painful. So I used the fabulous and fast dict.iteritems() function to build the structure, and then transformed the dict into a DataFrame using the very fast pd.concat() method.

Using a dict (couldn’t python have come up with an easier name to say for that feature? It’s so common, and yet so fraught. It’s like owning a Shih-Tzu) was perfect because it automatically deduplicates the data.

def convert_coverage_to_df(coverage):

    print “converting coverage dict to DataFrame”

    func_frames = []

    func_a = []

    path_a = []

    line_a = []

    for func, a in coverage.iteritems():

        path_frames = []

 

        for path, b in a.iteritems():

            line_frames = []

            for line, c in b.iteritems():

                line_frame = DataFrame([c])

                line_frames.append(line_frame)

                func_a.append(str(func))

                path_a.append(str(path))

                line_a.append(line)

            path_frame = pd.concat(line_frames)

            path_frames.append(path_frame)

        if len(path_frames) > 1:

            func_frame = pd.concat(path_frames)

            func_frames.append(func_frame)

        elif len(path_frames) == 1:

            func_frames.append(path_frame)

    cov_df = pd.concat(func_frames)

    cov_df[‘func’] = func_a

    cov_df[‘path’] = path_a

    cov_df[‘line’] = line_a

    cov_df = cov_df.set_index([‘func’, ‘path’, ‘line’])

    cov_df[‘app’] = cov_df[‘app’].fillna(‘ ‘)

    cov_df = cov_df.fillna(0)

    return cov_df

 Until next time, enjoy your programming, and let me know what you think!


Phase 5: Outreach

The theme for this weeks’ post is outreach. We asked lots of people for feedback on our approach, we tied up a bunch of loose ends, we wrote and gave a tidy presentation on the project, and somebody other than Mike and me ran our code. We are ready to start looking at the big picture: making the fixes.

Housekeeping

For the past few weeks Mike and I have been churning out python scripts that do various things, as we have ideas and want to try them out. These scripts produce artifacts that sometimes we later realized we could re-use in the pursuit of another purpose, so we wrote another script to consume it, and put out another artifact, etc.

For instance, Mike wrote a script called construct_call_graph.py, which scans a directory structure for Python files, reads them and creates an ast_list (which is what gets fed to the Python parsing routines and turned into running code), and parses it for recognizable function calls. Then it puts out several artifacts:

·       Graphs of the function calls in DOT format

·       Images of the graphs in PNG format

·       A dict of dicts (in a JSON file) containing a list of all the function calls for the library we care about, each a key for a dict listing the files containing the call, each filename a key to an array of line numbers for each call.

We reprocess the DOT files into a larger graph, and pull the dict out of the JSON file into a pandas DataFrame to transform it into a CSV which we read into Excel to report, using pivot tables and conditional formatting, the coverage area of the project and our progress through it.

All totally useful stuff. But – those interfaces! We have to clean them up. We have to clear out the #print statements in the code, check it all into GIT, make a Confluence page describing it. Guests are coming! Clean the house!

Meanwhile we are also clearing out all the ad hoc scripts we didn’t end up using long-term, and all their squirrelly little output.

Coverage, round 2

Wait, what? How is that again? We’re creating a coverage map of the problem space by reading the call graph into a pandas DataFrame and exporting to CSV. Here’s how.

The call graph dict structure looks like this:

{

“Index”: {

    “core.query.py”: [1385, 1389, 2215],

    “core.collection.py”: [41, 259],

    “core.series.py”: [80, 164, … 212, 217]

    },

“np.load”: {

    “core.index.py”: [17],

    “core.series.py”: [114, 115]

    }

}

The function (e.g. Index) is a key to a set of modules that call it (e.g. core.series.py), each of which is a key to an array of line numbers (e.g. 80).

We want to seed a pandas DataFrame with these call references, then later we will insert more data into the DataFrame, and finally put out a CSV.

First, let’s define the DataFrame we want to end up with.

from pandas import DataFrame

df_coverage = DataFrame(columns=[ ‘idx’,

                                  ‘func’,

                                  ‘path’,

                                  ‘line’)]

df_coverage.set_index(‘idx’, inplace=True)

Now let’s instantiate the dict from JSON:

import json

import os

with open(“_calls.json”) as static_file:

    dict_static = json.load(static_file)

Next we will turn the data in the dict into a series of rows in a format that we can add to the DataFrame, then add the rows to the DataFrame one at a time:

for func in dict_static:

    for path in dict_static[func]:

        for line in dict_static[func][path]:

            idx = func + ‘::’ + path + ‘::’ + line

            if idx not in coverage.index:               

                    row = DataFrame({‘idx’: [idx], ‘func’: [func], ‘path’: [path], ‘line’: [line]})

                    row.set_index(‘idx’, inplace=True)

                    df_coverage = pd.concat([df_coverage, row])

                    df_coverage = df_coverage.drop_duplicates()

And let’s put out a CSV of the file:

df_coverage.to_csv(‘coverage.csv’)

What does it look like?

idx,func,path,line

Index::core.query.py::1385,Index,core.query.py,1385

…

np.load::core.series.py::115,np.load,core.series.py,115

If we run across more data we want to add to the DataFrame to flesh out the numbers, we can do that. Let’s say we run a profiler on the code while it’s running, and find that some calls are made very frequently and others are more rare. We can add that to the DataFrame in a ‘freq’ column, like this:

import csv

with open(‘heatmap.csv’) as heat_file:

    arr_heat = csv.reader(heat_file)

    df_coverage[‘freq’] = 0

    for hm in arr_heat:

        hm_func = hm[1]

        hm_freq = hm[2]

        df_coverage.ix[df_coverage.func == hm_func, ‘freq’] = hm_freq

And when we find out how many unit tests were created for each call at each line of each file, we can insert those numbers into the DataFrame the same way.

First contact

Our project supervisor, Gil (not his real name but his office has glass walls like a fish tank) wanted to get his hands dirty a little bit, and asked how to run the TestGenerator. We showed him, and fortunately he was able to run it. This is a pretty big milestone for us since we want to bring in as many people as we can to give us feedback.

He immediately made a few suggestions, which I’ll summarize.

The TestGenerator is at the heart of the library upgrade process: as you might imagine, its function is to generate a series of unit test files and accompanying pickle objects. To do this it has to wrap the functions, and to do that, it has to know what functions to wrap. Up until now we’ve been coding the list of functions by hand, but there are hundreds of them and it’s pretty messy that way.

Quick recap on AOP wrapping: we are using two styles of AOP wrappers. For functions that are no part of the built-in collection (mostly pandas functions) we are using the b3j0f library to replace the guts of the call. For all other functions, we are using a simple monkey patching technique to substitute an extended function for the base function.

Some functions can only be wrapped using one method, some can only be wrapped using the other. We have been specifying each type of functions differently when we call the TestGenerator(), but that leads to call code that looks like this:

import TestGenerator

import pandas as pd

import numpy as np

#pandas wrapper

tg=TestGenerator.TestGenerator([pd.isnull, pd.notnull])

#numpy wrappers

np.unique = TestGenerator.wrapper_decorator(np.unique)

np.concatenate = TestGenerator.wrapper_decorator(np.concatenate)

np.array = TestGenerator.wrapper_decorator(np.array)

That is kind of a mess. So we switched it up. We put in a simple try / except block to see which method works for each call, and made TestGenerator do the work of wrapping the code according to its logic. Now we just have to pass a single array with all the calls we want wrapped, and they get wrapped.

tg=TestGenerator.TestGenerator([pd.isnull, pd.notnull, np.array, np.concatenate, np.unique])

The great thing is that we can find this array anywhere – even in a text file. So we are updating construct_call_graph to put a list of functions out to a file, and the TestGenerator call can just pass that file into TestGenerator which will operate on it automatically.

That’s it for this week. See you around at the next exciting episode!

Your comment will be posted after it is approved.