How to debug django web application with autoreload.

Edit:

• After Pydev 2.2.1, this feature has been added, just act like this one, please see http://pydev.org/manual_adv_remote_debugger.html#django-remote-debugging-with-auto-reload

If you are a django newbie and want to develop django application in pydev, see this article first: Configuring pydev to work with django.

This post will focus on how to debug django without –noreload option.

Configuring pydev to work with django this article mentioned that:

“If you want to keep the auto-reloader and still debug, you’ll have to use the remote debugger that comes with the extensions to debug, as it does not rely on having the process started from Eclipse. Check http://www.fabioz.com/pydev/manual_adv_remote_debugger.html for more details.”

I tried it and feel it is a very bad way to debug django application. I use auto reload feature because its dynamic and convenient. But if follow the article to do that, I must insert a line: “import pydevd; pydevd.settrace()” in my source code to set a breakpoint. How silly of this way? If I want to trace the source code in django itself, I must modify the source code of django just for a breakpoint. What if I forget to remove this line…?

Ok, you can give it up that, debug django with –noreload option, not really bad, right? This is true if your web application do not need to login. Every time my partner debug its code, he must change the code, restart django server and login again, then to see the result and do real debug. Restart and login again is a redundant operation in debugging and very wasting time (I know I can use some automation tool such as Selenium to reduce my pain, but it is also inconvenient). The django’s charming productivity is gone.

After some digging into source code of django and pydev, I found a easy way to debug django in pydev with autoreload feature. This way is also using pydev extension’s remote debugger, but the annoy operation which inserts a “import pydevd; pydevd.settrace()” is no more needed. You can set a breakpoint like before and debug it with autoreload feature. The django’s productivity is back!

How to do that? This is simple, you must modify your manage.py and insert these code:
(This code requires pydevd module, please see Configuring pydev to work with django to know where to find it.)

    if settings.DEBUG and (command == "runserver" or command == "testserver"):

        # Make pydev debugger works for auto reload.
        try:
            import pydevd
        except ImportError:
            sys.stderr.write("Error: " +
                "You must add org.python.pydev.debug.pysrc to your PYTHONPATH.")
            sys.exit(1)

        from django.utils import autoreload
        m = autoreload.main
        def main(main_func, args=None, kwargs=None):
            import os
            if os.environ.get("RUN_MAIN") == "true":
                def pydevdDecorator(func):
                    def wrap(*args, **kws):
                        pydevd.settrace(suspend=False)
                        return func(*args, **kws)
                    return wrap
                main_func = pydevdDecorator(main_func)

            return m(main_func, args, kwargs)

        autoreload.main = main

This code overrides the django’s autoreload behavior to make pydev can communicate with it. The knowledge behind it is a little complex, so I don’t explain it in here.

If you want to debug without –noreload, you must start remote debugger first, then run django server in pydev (no matter debug mode or not). Eclipse will suspend while it run on your breakpoints. You can modify code and do not need to restart django server to debug it. Try it!

Using PythonTidy in pydev as code formatter.

My favorite python ide is Eclipse + Pydev. Pydev is a good eclipse plugin to make develop with python efficiently. But you will find its Autoformat (code format) feature is too simple to make things work.

I worked with some python newbie, they are just learning python and don’t know what is PEP08 (Style Guide for Python Code). Usually, they are experts in some other languages like C++, Java or C# with powerful IDE. They rely on code completion and code format feature too much. When they use Eclipse + Pydev, the source code they written is terrible…

I searched python code formatter (beautifier) for a long time. Unfortunately, there is no good product or tool to do that. Finally, the tool which is the closest to what I want is PythonTidy. This script format your code to fulfill PEP08 and it can be customized to satisfy the coding standard in your team.

The problem is PythonTidy is a standalone script to run it. This is not very convenient for us to use it. We must make pydev to integrated with PythonTidy to enhance its code format feature.

Basically, the pydev is a extensible tool to make us integrated PythonTidy easily. You can write some Jython script to extend pydev if you familiar with Java and Eclipse also. I wrote a simple script to make PythonTidy work while we press Ctrl+Shift+F.

If you want to use it, save the code below in somewhere you want as pyedit_pythontidy.py (pyedit_ is required).

"""
This code is public domain.
The original author is Bear Huang (http://bear330.wordpress.com/).
"""
if False:
    from org.python.pydev.editor import PyEdit #@UnresolvedImport
    cmd = 'command string'
    editor = PyEdit

assert cmd is not None
assert editor is not None

if cmd == 'onCreateActions':
    # from org.eclipse.jface.action import Action
    from org.python.pydev.editor.actions import PyAction
    from org.python.pydev.core.docutils import PySelection
    from java.lang import Runnable
    from org.eclipse.swt.widgets import Display
    from org.eclipse.jface.text import IDocument
    from org.eclipse.jface.text import TextSelection

    from java.io import FileWriter
    import java.lang.Exception

    FORMAT_ACTION_DEFINITION_ID = "org.python.pydev.editor.actions.pyFormatStd"
    FORMAT_ACTION_ID = "org.python.pydev.editor.actions.navigation.pyFormatStd"

    class PythonTidyAction(PyAction):

        def __init__(self, *args, **kws):
            PyAction.__init__(self, *args, **kws)

        def run(self):
            import tempfile
            import os

            try:
                ps = PySelection(editor)
                doc = ps.getDoc()
                startLine = ps.getStartLineIndex()

                p1 = tempfile.mktemp()
                p2 = tempfile.mktemp()
                f1 = FileWriter(p1)

                formatAll = False
                if ps.getTextSelection().getLength() == 0:
                    # format all.
                    c = doc.get()
                    f1.write(c)
                    formatAll = True
                else:
                    # format selection.
                    #c = ps.getSelectedText()
                    #f1.write(ps.getSelectedText())
                    print "Format selected text is not supported yet."
                    f1.write("")
                    # A kind of solution is to insert a special comment in
                    # front and end of selection text, pythontidy it, and
                    # extract text according that comment. 

                f1.close()
                os.system('PythonTidy.py "%s" "%s"' % (p1, p2))
                f2 = open(p2, "r")
                result = f2.read()
                f2.close()

                os.remove(p1)
                os.remove(p2)

                if startLine >= doc.getNumberOfLines():
                    startLine = doc.getNumberOfLines() - 1

                if formatAll:
                    doc.set(result)
                else:
                    #doc.replace(doc.getLineOffset(startLine), 0, result)
                    pass

                sel = TextSelection(doc, doc.getLineOffset(startLine), 0)
                self.getTextEditor().getSelectionProvider().setSelection(sel)
            except java.lang.Exception, e:
                self.beep(e)

    def bindInInterface():
        act = PythonTidyAction()

        act.setActionDefinitionId(FORMAT_ACTION_DEFINITION_ID)
        act.setId(FORMAT_ACTION_ID)
        try:
            editor.setAction(FORMAT_ACTION_ID, act)
        except:
            pass

    class RunInUi(Runnable):
        '''Helper class that implements a Runnable (just so that we
        can pass it to the Java side). It simply calls some callable.
        '''

        def __init__(self, c):
            self.callable = c
        def run(self):
            self.callable ()

    def runInUi(callable):
        '''
        @param callable: the callable that will be run in the UI
        '''
        Display.getDefault().asyncExec(RunInUi(callable))

    runInUi(bindInInterface)

Next, go to eclipse pydev preferences page to specify the script location:

Now, you can simply press Ctrl+Shift+F while edit python code.

(Format selected text is not supported yet, this is because I am too lazy to implement it in this time. If I have time and wish to do that, I will publish it as soon as posible.)

Any contributions are appreciated.