Profiling isn’t often important for my python programs, but when a performance black hole appears out of nowhere it’s very useful for narrowing down the problem, even on smaller scripts.
(Recently I had cause to profile a script manipulating large-ish (<50Mb) files, whose performance had taken a turn for the worse. I had a suspicion that poor list manipulation was to blame, but the results spoke for themselves: 99% of CPU time was spent in the function list.pop(0)
! Based on this I discovered PEP-290 which describes the collections.deque
structure, significantly more efficient than a basic list — the replacement function, deque.popleft()
, barely registers a percentage of overall execution time(!)
I could have discovered this using basic profiling, but whilst playing with the profiler I discovered a very straightforward, easy-to-read python profile visualiser, and thought it worth sharing. It doesn’t do anything very beautiful or complicated, but it’s very quick, provides a neat “squaremap” view of your code performance, and importantly Just Works. RunSnakeRun is its name.
You can obtain RunSnakeRun and its dependencies using the awkwardly-named easy_install
:
easy_install SquareMap easy_install RunSnakeRun
Well. That was easy.
A note about running on the Mac — this requires TCL/Tk so on a standard Snow Leopard installation can’t be executed as the 64-bit Python binary is invoked. You can get around this by running in 32-bit – a useful general tip for any situation where you need to enforce an architecture:
$ arch -i386 python runsnake ./my.profile
So far so good, but you’ve no profiling data. A brief profiling-in-python primer follows, though really it’s not complicated and this section is little more than a condensed version of the python docs; you should look there in preference! In summary, however, you can profile any python app using the cProfile
module; any interactive script can be run against cProfile.py
(you’ll find it in your Python library directory). Running cProfile.py <your-script-name> <your-script-args>
without an output file will produce some mostly-helpful statistics, but really nothing beats a graphical view so we’ll press on.
You can create some profiling data using the cProfile.py
script as follows:
cProfile.py -o myscript1.profile <your-script-name> <your-script-args>
or programmatically:
import cProfile, mymodule cmd = "mymodule.main(args)" outfile = "myscript1.profile" cProfile.run(cmd, outfile)
having run either of these you simply need to launch your visualiser:
$ runsnake myscript1.profile
Beautiful.