Debugging with Functions¶
IPython notebooks are great for interactive use. You can perform a bunch of calculations in the global scope, look at the results etc. If some code in a loop crashes, you can see the current values of all the variables etc.
However, working in the global scope is generally bad practice, especially in a notebook where you might be executing cells out of order. For example:
- A variable you think you have defined might have actually been created later on, so if you run the notebook from scratch, it will break.
- Variables might be overwritten later, and so rerunning code may have unexpected behaviour that depends sensitively on how you toyed with your notebook.
For these and many other reasons, it is often better to do certain calculations in a function. (This function can then be passed to other tools like profilers, timers, parallel computations, animation libraries etc.). However, now if something goes wrong, you need to fire up a debugger which does not work so well in a notebook.
My previous solution was to do something like the following once I started the debugger:
ipdb> !import sys;sys._l = locals()
ipdb> q
Then I can get out of the debuger and put all of the local variables in my environment:
import sys
locals().update(sys._l)
Decorating Functions for Debugging¶
Note: This is now implemented through the decorator mmfutils.debugging.debug with the only caveat that you must pass locals() explicitly.
I recently came across this question on SO: Python: Is there a way to get a local function variable from within a decorator that wraps it?. Here we explore this a bit.
import sys
class persistent_locals(object):
"""Decorator that stores the functions local variables
in an attribute `locals`.
"""
def __init__(self, func):
self._locals = {}
self.func = func
def __call__(self, *args, **kwargs):
def tracer(frame, event, arg):
if event=='return':
self._locals = frame.f_locals.copy()
# tracer is activated on next call, return or exception
sys.setprofile(tracer)
try:
# trace the function call
res = self.func(*args, **kwargs)
finally:
# disable tracer and replace with old one
sys.setprofile(None)
return res
def clear_locals(self):
self._locals = {}
@property
def locals(self):
return self._locals
This seems to work well:
@persistent_locals
def func():
local1 = 1
local2 = 2
func()
print(func.locals)
It also works if an exception is raised:
@persistent_locals
def func():
local1 = 1
raise Exception
local2 = 2
try:
func()
except:
pass
print(func.locals)
Now we can add this to a debug decorator that automatically adds the variables to the "global" scope.
def debug(locals):
"""Decorator to wrap a function and dump its local scope.
Arguments
---------
locals : dict
Function's local variables will be updated in this dict.
Use locals() if desired.
"""
def decorator(f):
func = persistent_locals(f)
def wrapper(*v, **kw):
try:
res = func(*v, **kw)
finally:
locals.update(func.locals)
return res
return wrapper
return decorator
def f():
l1 = 1
l2 = 2
x = 1/0
l3 = 3
f()
env = {}
debug(env)(f)()
env
This works, but I would rather not see all of the wrapper details in the exception.
def debug(locals=None):
"""Decorator to wrap a function and dump its local scope.
Arguments
---------
locals : dict
Function's local variables will be updated in this dict.
Use locals() if desired.
"""
if locals is None:
locals = globals()
def decorator(f):
func = persistent_locals(f)
def wrapper(*v, **kw):
try:
res = func(*v, **kw)
except Exception, e:
# Remove two levels of the traceback so we don't see the
# decorator junk
exception_type, exception, traceback = sys.exc_info()
raise exception_type, exception, traceback.tb_next.tb_next
finally:
locals.update(func.locals)
return res
return wrapper
return decorator
debug(env)(f)()
If we do not specify the environment, they appear in the global scope:
assert 'l1' not in locals()
try:
debug()(f)()
except:
pass
print(l1, l2)
del l1, l2
Now the final touch would be to not have to call the decorator if we are not passing in the environment.
def debug(*v, **kw):
"""Decorator to wrap a function and dump its local scope.
Arguments
---------
locals (or env): dict
Function's local variables will be updated in this dict.
Use locals() if desired.
"""
func = None
env = kw.get('locals', kw.get('env', None))
if len(v) == 0:
pass
elif len(v) == 1:
if isinstance(v[0], dict):
env = v[0]
else:
func = v[0]
elif len(v) == 2:
func, env = v
else:
raise ValueError("Must pass in either function or locals or both")
if env is None:
env = globals()
def decorator(f):
func = persistent_locals(f)
def wrapper(*v, **kw):
try:
res = func(*v, **kw)
except Exception, e:
# Remove two levels of the traceback so we don't see the
# decorator junk
exception_type, exception, traceback = sys.exc_info()
raise exception_type, exception, traceback.tb_next.tb_next
finally:
env.update(func.locals)
return res
return wrapper
if func is None:
return decorator
else:
return decorator(func)
Here are the four calling styles:
def f():
l1 = 1
l2 = 2
return l1 + l2
env = {}
res = debug(f, env)(); print env, res
res = debug(f, locals=env)(); print env, res
res = debug(env)(f)(); print env, res
res = debug(locals=env)(f)(); print env, res
If you want to just update the global dict, there is one calling style:
assert 'l1' not in locals()
res = debug(f)()
print l1, l2, res
del l1, l2
And now as a decorator:
@debug
def f():
l1 = 1
l2 = 2
return l1 + l2
env = {}
@debug(env=env)
def g():
l1 = 1
l2 = 2
return l1 + l2
g()
print env
assert 'l1' not in locals()
f()
print l1, l2; del l1, l2