Python Buzz Forum - A Python syntax highlighter

In a recent post I described my first ever Ruby program – which was actually a syntax highlighter for Python and written in Ruby, ready to be used in a Typo web log. Since the post was rather a long one, I decided to post the code itself separately. Here it is, then.

The Test Code

As you can see, currently only comments, single- and triple- quoted strings, keywords and identifiers are recognised. That’s really all I wanted, for now. For completeness, I may well add support for numeric literals. Watch this space!

typo/vendor/syntax/test/syntax/tc_python.rb

require File.dirname(__FILE__) + "/tokenizer_testcase"

class TC_Syntax_Python < TokenizerTestCase

  syntax "python"

  def test_empty
    tokenize ""
    assert_no_next_token
  end
  def test_comment_eol
    tokenize "# a comment\nfoo"
    assert_next_token :comment, "# a comment"
    assert_next_token :normal, "\n"
    assert_next_token :ident, "foo"
  end
  def test_two_comments
    tokenize "# first comment\n# second comment"
    assert_next_token :comment, "# first comment"
    assert_next_token :normal, "\n"
    assert_next_token :comment, "# second comment"
  end
  def test_string
    tokenize "'' 'aa' r'raw' u'unicode' UR''"
    assert_next_token :string, "''"
    skip_token
    assert_next_token :string, "'aa'"
    skip_token
    assert_next_token :string, "r'raw'"
    skip_token
    assert_next_token :string, "u'unicode'"
    skip_token
    assert_next_token :string, "UR''"
    tokenize '"aa\"bb"'
    assert_next_token :string, '"aa\"bb"'
  end
  def test_triple_quoted_string
    tokenize "'''\nfoo\n'''"
    assert_next_token :triple_quoted_string, "'''\nfoo\n'''"
    tokenize '"""\nfoo\n"""'
    assert_next_token :triple_quoted_string, '"""\nfoo\n"""'
    tokenize "uR'''\nfoo\n'''"
    assert_next_token :triple_quoted_string, "uR'''\nfoo\n'''"
    tokenize '"""\'a\'"b"c"""'
    assert_next_token  :triple_quoted_string, '"""\'a\'"b"c"""'
  end
  def test_keyword
    Syntax::Python::KEYWORDS.each do |word|
      tokenize word
      assert_next_token :keyword, word
    end
    Syntax::Python::KEYWORDS.each do |word|
      tokenize "x#{word}"
      assert_next_token :ident, "x#{word}"
      tokenize "#{word}x"
      assert_next_token :ident, "#{word}x"
    end
  end
end

The Python Tokenizer

typo/vendor/syntax/python.rb

require 'syntax'

module Syntax

  # A basic tokenizer for the Python language. It recognises
  # comments, keywords and strings.
  class Python < Tokenizer
    # The list of all identifiers recognized as keywords.
    # http://docs.python.org/ref/keywords.html
    # Strictly speaking, "as" isn't yet a keyword -- but for syntax
    # highlighting, we'll treat it as such.
    KEYWORDS =
      %w{as and del for is raise assert elif from lambda return break
         else global not try class except if or while continue exec
         import pass yield def finally in print}
    # Step through a single iteration of the tokenization process.
    def step
      if scan(/#.*$/)
        start_group :comment, matched
      elsif scan(/u?r?'''.*?'''|""".*?"""/im)
        start_group :triple_quoted_string, matched
      elsif scan(/u?r?'([^\\']|\\.)*'/i)
        start_group :string, matched
      elsif scan(/u?r?"([^\\"]|\\.)*"/i)
        start_group :string, matched
      elsif check(/[_a-zA-Z]/)
        word = scan(/\w+/)
        if KEYWORDS.include?(word)
          start_group :keyword, word
        else
          start_group :ident, word
        end
      else
        start_group :normal, scan(/./m)
      end
    end
  end
  SYNTAX["python"] = Python

end


	Web Artima.com