OK, so – the last couple weeks I have focused on creating reporting tools that show our coverage. What I mean by coverage is “the problem space and the extent to which we have explored it”. In our world here, the problem space has the following dimensions:

• The pandas and NumPy calls we can identify as having been made in the business code
• The files and line numbers where the calls are made

Our tools for exploring the space are:

• The business applications that run this code
• The input data to the business applications which may force it down one or another code path

We set up a map of the territory with pandas calls down the left and files across the top:

Our coverage tools find each call in each file, then we color the cell corresponding to the call. If we haven’t created any tests yet for that call in that file, we color it blue. If we have some tests, we color the cell green – and the more tests, the greener the cell.

We can add some detail to this by making the map show which applications we ran to get the tests to appear. Once we move past our first app we will see if that is a desired or necessary detail, and if we need it we will add it.

The most interesting thing to do though is to create a time series of these maps – at ProChain we did this for project status reports by taking snapshots of the buffer consumption across a fever chart over time, and offering a slider on the web page to let users review the project’s progress quickly. I’m not sure how I’ll do it here, but I definitely want to.

Making our Performance Public

The point of course of creating coverage maps is to report on our progress. We are proud of our work and think it’s great that everyone know what we are doing and how far we have come. So the next step, now that the tools are in place and the deliverables are determined, is figuring out the actual delivery. For now I’m emailing a spreadsheet to Gil, but I think I want to be a little more exhibitionistic – I’m going to put the results up on Confluence.

Now – how do I automate that? If you have any cool ideas please share them! I know that pandas has a “to_html” method, but I may just end up going with Excel’s “Save as Web Page” option – scripted, of course. I am SDET, you know.

Of course, I could just stand up a Django framework and feed the coverage DataFrame straight to it… Maybe in a couple weeks I’ll try it.

Fun with AOP wrapping

On a more technical topic, Mike and I have had to do some pretty interesting maneuvers to intercept the various pandas and NumPy calls. The vast majority of the pandas calls in v0.7 are amenable to using the b3j0f library, but the ones that rely on NumPy calls beneath the surface can’t be intercepted that way. NumPy is coded in C for performance reasons, and gets included in the Python built-ins, so it does not expose the attributes required for b3j0f weaving to work. But, a simple wrapper decorator, aka monkeypatch, does the trick for these functions. So the question is, how do you tell a priori which function to wrap using which strategy?

Answer: You don’t. You let the function tell you.

Try to weave it, then test the weave. If it succeeded, then great, we’re done. If it failed, try to monkeypatch it. If they both fail, try again, but meantime you’ve got some of your questions answered.

Step 1: Send all calls to b3j0f for weaving

(If anyone wants me to blog in more detail about exactly how to go about weaving calls using b3j0f, please say so. I’ve been leaving that side of the business to Mike but I’d like to know more about how it’s done. I can’t simply cut and paste his code here, but if I need to make some samples showing the procedure I can certainly try to take time for that.)

Step 2: Check all calls for successful woven-ness using is_intercepted

This is actually quite a bit harder than it sounds. The is_intercepted(my.pandas.call) syntax works to check if the call is successfully woven, but it will return false negatives if it is run too soon after the weave has taken place. So we have to wait a bit. What we have chosen to do is write the code to do the check out to a file called aopverify.py. Since it comes with a .py extension, and contains valid Python code, the file is available for importation and we can run the aopverify() method defined in it. So the step looks more like: write verification file, then run it to check all calls.

    def verify_aop(self,_dict):       

        with open(‘aopverify.py’, ‘w+’) as aopfile:

            aopfile.write(“import TestGenerator\n”)

            aopfile.write(“import numpy as np\n”)

            aopfile.write(“import pandas as pd\n\n”)

            aopfile.write(“from b3j0f.aop.advice import is_intercepted\n\n”)

            aopfile.write(“def aopverify():\n”)

            aopfile.write(”    fp0 = open(‘aop_b3j0f’,’w+’)\n”)

            aopfile.write(”    fp  = open(‘aop_monkeypatch’,’w+’)\n”)

            aopfile.write(”    fp1 = open(‘aop_bad’,’w+’)\n”)

            aopfile.write(”    fp2 = open(‘aop_err’,’w+’)\n”)

            for k in sorted(_dict.keys()):

                func = _dict[k]

                index = k.rfind(‘.’)

                prefix = k[:index]

                name = k[index + 1 :]       

                aopfile.write(“\n    ni = True\n”)

                aopfile.write(”    try:\n”)

                aopfile.write(”        if is_intercepted(” + prefix + “.” + name + “):\n”)

                aopfile.write(”            fp0.write(‘” + prefix + “.” + name + “\\n’)\n”)

                aopfile.write(”            ni = False\n”)

                aopfile.write(”    except Exception as e:\n”)

                aopfile.write(”        print ‘” + prefix + “.” + name + “: ‘ , e\n”)

                aopfile.write(”        fp2.write(‘” + prefix + ‘.’+ name + “: ‘ + str(e) + ‘ on b3j0f weave check\\n’)\n”)

                aopfile.write(”    if ni:\n”)

                aopfile.write(”        try:\n”)

                aopfile.write(”            if ” + prefix + “.” + name + “.__name__ != ‘” + ‘wrapper’ + “‘:\n”)              

                aopfile.write(”                print ‘monkey patch did not work for ” + prefix + “.” + name + “‘\n”)

                aopfile.write(”                fp1.write(‘” + prefix + “.” + name + “\\n’)\n”)          

                aopfile.write(”            else:\n”)

                aopfile.write(”                fp.write(‘” + prefix + ‘.’+ name + “\\n’)\n”)

                aopfile.write(”        except Exception as e:\n”)

                aopfile.write(”            print ‘” + prefix + “.” + name + “: ‘ , e\n”)

                aopfile.write(”            fp2.write(‘” + prefix + ‘.’+ name + “: ‘ + str(e) + ‘ on mp wrapper check\\n’)\n”)

            aopfile.write(”    fp0.close()\n”)

            aopfile.write(”    fp.close()\n”)

            aopfile.write(”    fp1.close()\n”)

            aopfile.write(”    fp2.close()\n”)

        print ‘invoking aop verify ‘

        import aopverify

        aopverify.aopverify()

       

The coolest part here is at the very end: we spit out a python file, then immediately import it and run the code!

The generated code in aopverify.py looks like this:

import TestGenerator

import numpy as np

import pandas as pd
 
from b3j0f.aop.advice import is_intercepted

def aopverify():

    fp0 = open(‘aop_b3j0f’,’w+’)

    fp  = open(‘aop_monkeypatch’,’w+’)

    fp1 = open(‘aop_bad’,’w+’)

    fp2 = open(‘aop_err’,’w+’)

    ni = True

    try:

        if is_intercepted(np.DataSource):

            fp0.write(‘np.DataSource\n’)

            ni = False

    except Exception as e:

        print ‘np.DataSource: ‘ , e

        fp2.write(‘np.DataSource: ‘ + str(e) + ‘ on b3j0f weave check\n’)

    if ni:

        try:

            if np.DataSource.__name__ != ‘wrapper’:

                print ‘monkey patch did not work for np.DataSource’

                fp1.write(‘np.DataSource\n’)

            else:

                fp.write(‘np.DataSource\n’)

        except Exception as e:

            print ‘np.DataSource: ‘ , e

            fp2.write(‘np.DataSource: ‘ + str(e) + ‘ on mp wrapper check\n’)
…

    ni = True

    try:

        if is_intercepted(np.zeros):

            fp0.write(‘np.zeros\n’)

            ni = False

    except Exception as e:

        print ‘np.zeros: ‘ , e

        fp2.write(‘np.zeros: ‘ + str(e) + ‘ on b3j0f weave check\n’)

    if ni:

        try:

            if np.zeros.__name__ != ‘wrapper’:

                print ‘monkey patch did not work for np.zeros’

                fp1.write(‘np.zeros\n’)

            else:

                fp.write(‘np.zeros\n’)

        except Exception as e:

            print ‘np.zeros: ‘ , e

            fp2.write(‘np.zeros: ‘ + str(e) + ‘ on mp wrapper check\n’)

    fp0.close()

    fp.close()

    fp1.close()

    fp2.close()

Notice that aopverify() puts out a list of calls that were not woven. We can feed this list back to the TestGenerator in the next step.

Another benefit of this approach is that we have the aopverify() function available to use whenever we need it. We can use it to monitor the health of the wrapped functions during the course of the testing.

Step 3: Try monkeypatching all the calls that were not woven in step 1

TestGenerator accepts a separate list of calls for monkeypatching. You might ask, why did you get away from sending all calls to the monkeypatch that didn’t get woven? There are two reasons.

First, we couldn’t reliably tell when a call was successfully woven right away. We had to complete the weave attempts for everything and actually end the loop that the calls were being made in.

The second reason is that some calls, when fed into b3j0f for weaving, will cause the Python session to crash. We can’t afford to crash the session, so we have to specifically identify them as monkeypatch-only calls. Once we have that list, why not put everything on it that we know must be monkeypatched?

So – the verification step puts out a list of calls that are not currently being wrapped. We started with a list of calls to wrap, tried them all in b3j0f, and this list is the leftovers. We feed these into TestGenerator as a second list which it will try to wrap using monkeypatching.

TestGenerator again writes out a python file, imports it and runs it to start wrapping the calls using the monkeypatch technique. The code we spit out, import, and run looks like this:

import TestGenerator

import numpy as np

import pandas as pd

 

def aopscript():

    try:

        if np.DataSource.__name__ == ‘DataSource’:

            np.DataSource = TestGenerator.wrapper_decorator(np.DataSource,”)

            print ‘np.DataSource sent to monkeypatch’

        if np.DataSource.__name__ == ‘wrapper’:

            print ‘np.DataSource wrapped with wrapper decorator’

    except:

        print ‘np.DataSource threw exception’

        pass

…

    try:

        if np.zeros.__name__ == ‘zeros’:

            np.zeros = TestGenerator.wrapper_decorator(np.zeros,”)

            print ‘np.zeros sent to monkeypatch’

        if np.zeros.__name__ == ‘wrapper’:

            print ‘np.zeros wrapped with wrapper decorator’

    except:

        print ‘np.zeros threw exception’

        pass

We do it this way because we (sometimes) need to run the verification step first, and this gives us the option to re-wrap the calls if they fail for any reason. Once this runs, we can re-run the verification step, and we will see the full list of calls wrapped by each method, plus any calls that are still unwrapped, and any calls that throw errors on verification. These errors are usually the same ones thrown during an attempt to wrap them, so they give Mike a heads up on what the issue likely is.

Your comment will be posted after it is approved.