An academic approach to ‘import’

As they say, Python is a language that does not force you to do things. Instead, it assumes the programmer to be a consenting adult who, knowingly or unknowingly has the power to bend and mold existing paradigms to fit whatever task is at hand. While this easy-peasy mindset sometimes can be an absolute asset, it will inevitably (hopefully not more than once) lead you down the winding path of destruction. Personally I feel that nothing in Python is quite as capable at dragging people down like the “import” statement.

Standard Imports:

import os
from os.path import isfile as isFile
from time import *

Hopefully this is not the first time you’ve heard this, but you should avoid the asterisk on imports. Unless you own the import (like a file with a bunch of common functions in it) then you can never be sure that a function defined within the import has not just blown away a function with the same name in your application.

User-specific Imports:
But what about the following… (Linux only I believe)

import os
import sys
sys.path.append(os.path.join(os.environ['HOME'], 'scripts'))
import userScript

This can be really helpful to provide some user-specific functionality. Basically the username determines the home directory, and the home directory then provides you with a specific subdirectory of Python scripts. If you log in as “root” you will get one set of functionality, whereas if you log in as “user” you will get a different set of functionality.

Watch out though, as environmental variables are very inaccurate. Anybody and anything can change these variables, and can leave you depending on incorrect or invalid values. One of the easiest ways to see this is by having a proper-daemon C/C++ application that calls a Python script. When you log on as any user and execute the binary, you will see the expected results. If you (via socket or RPC or whatever) call it remotely, the user defaults to the system, which unfortunately does not have certain environmental variables (like a “HOME” directory)… in other words, remote call = failure.

Application-relative (not path-relative) Imports:
Here is another one that I have seen, but have rarely used. Similarly to the previous condition, it requires appending to the system path, but this time we are working with a relative directory.

import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
import someClass

There are two special things happening here. First, we are extracting the direct relative path to the file containing this code. If we were to run this through the interpreter, we would get an error because there is no “__file__”. Second, because “__file__” is always the relative path to the Python file, it doesn’t matter where you execute from, absolute path or not, you will always be referring to “..” from the Python file. This is EXTREMELY useful if you need to create an application that can be run via both relative and absolute path.

Path-agnostic Import Best Practices:
Although I have given multiple examples of how to get the system path to not care where you are executing from, I have for the most part found that the best practice with import paths is to use a configuration file. Why did I just complicate the import process even more? Well, lets take a look at the code:

import os
import sys
lines = open(os.path.join(os.path.dirname(__file__), "config.cfg")).splitlines()
paths = [line.split("=")[1] for line in lines if "path" is line.split("=")[0]]
for path in paths: sys.path.append(path)
import customModule

The benefit here is that now when you want to swap out “customModule” with a different version for testing, you dont need to move any directories, or rename any imports. All you do is change the config file. And yes, I did go a little overboard here, and allow you to add as many paths to the system path list as you want. The other big benefit here is that since you are always specifying an absolute path in the configuration file, you are always guaranteed that the import will succeed.

Using a Property to get a Class name into an Attribute

For one reason or another, I found myself in a slight predicament. I needed to get the name of my class into an attribute.

Thanks to Google + KyLev I managed to procure a workable solution:

class Problem(object):
    moduleIndex = 0
    def __init__(self):
        self.__class__.moduleIndex = self.__class__.moduleIndex + 1

class Solution(object):
    moduleIndex = 0
    moduleName = property(fget=lambda self: self.__class__.__name__)
    def __init__(self):
        self.__class__.moduleIndex = self.__class__.moduleIndex + 1
        print "[%s] %s"%(self.moduleIndex, self.moduleName)

if __name__ == "__main__": t = Solution()

C/C++ memcpy() for structs

At some point you will find that you need a permanent reference to a struct, but all that you have available is a non-static pointer to that struct. Now you will note that this commonly exists already in the dup2() function, taking a file-pointer and duplicating it to another file-pointer. So how can we do this for some other struct?

struct MY_STRUCT*
copyStruct(const struct MY_STRUCT *s) {
  struct MY_STRUCT *d = (struct MY_STRUCT*)malloc(sizeof(struct MY_STRUCT));
  if (d == NULL) return NULL;
  memcpy(d, s, sizeof(s));
  return d;
}

Of course, name it whatever you want, and use whatever struct you want. Obviously you get a pointer-to-struct on success, otherwise you get NULL… so make sure the caller checks… and calls free() when appropriate.

Calculating Network Addresses

I have frequently run up against the problem of generating network addresses (strictly speaking IPv4) but I am positive that the principles demonstrated below can carry over into IPv6.

The biggest issue is that in calculating addresses, the best way to find the “Nth address from here” is to convert to decimal and add. Well it’s not so difficult to convert 4 octets into a 32-bit binary number, and crunch that into a single decimal number, add “N”, convert back to binary, divide into octets, and convert back to decimal… but it takes forever. So why take forever when you don’t need to. The trick? Divide & Conquer.

After walking through the theory, masks, addresses, ranges, compliments… I finally put together the following as a helper. Feel free to comment.

#!/usr/bin/env python

import re
import sys
from optparse import OptionParser

# ============================================================================
# Global Variables
# ============================================================================
use_string = "usage: %prog"
ver_string = "v.0.1.0"

# ============================================================================
# Global Functions
# ============================================================================
def is_ipv4(string):
    regex = r'^([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)$'
    i = re.finditer(regex,string)
    match = [m.groups() for m in i]
    if len(match) != 1: return False
    octets = match[0]
    for octet in octets:
        try: number = int(octet)
        except: return False
        if number < 0 or number > 255: return False
    return True

def dec2octets(number):
    original = number
    octets = 4
    answer = []
    remainder = 1
    while (remainder != 0):
        if octets == 0:
            raise Exception("'%s' is too large to be an IP."%original)
        octets = octets - 1
        value = number / (256**octets)
        remainder = number % (256**octets)
        answer.append(value)
        number = remainder
    while (len(answer) != 4): answer.append(0)
    return answer

def ip2list(ip):
    assert is_ipv4(ip)
    return [int(x) for x in ip.split('.')]

def ip2bin(ip):
    assert is_ipv4(ip)
    tmp = [[('0','1')[x>>j&1] for j in xrange(7,-1,-1)] \
           for x in [int(x) for x in ip.split('.')]]
    return ''.join([''.join(x) for x in tmp])

def bin2dec(x):
    assert type(x) == type("")
    assert x.strip('01') == ''
    return int(x,2)

def numify(n):
    assert type(n) == type(1)
    if str(n)[-1] == "1": return "%sst"%(n)
    elif str(n)[-1] == "2": return "%snd"%(n)
    elif str(n)[-1] == "3": return "%srd"%(n)
    else: return "%sth"%(n)

# ============================================================================
# Main Method
# ============================================================================
if __name__ == '__main__':
    parser = OptionParser(version=ver_string, usage=use_string)
    parser.add_option("-i","--ip",
                      action="store",
                      default=None,
                      dest="ipAddress",
                      help=("Use this to specify the IP address to use in "
                            "calculations. Alternatively, specify the IP address"
                            "in the 'x.x.x.x/N' form to specify both IP and "
                            "subnet mask."),
                      metavar="IPADDR",
                      )
    parser.add_option("-m","--mask",
                      action="store",
                      default=None,
                      dest="subnetMask",
                      help=("Use this to specify the subnet mask to use in "
                            "calculations."),
                      metavar="MASK",
                      )
    parser.add_option("-n","--find",
                      action="store",
                      default=0,
                      dest="lookFor",
                      help=("Use this to specify the IP address to "
                            "us in calculations"),
                      metavar="VALUE",
                      )

    (options, args) = parser.parse_args()

    # ========================================================================
    # Massage input...
    # ========================================================================
    if (options.ipAddress != None and options.subnetMask != None):
        assert is_ipv4(options.ipAddress)
        assert is_ipv4(options.subnetMask)
        addr = ip2list(options.ipAddress)
        mask = ip2list(options.subnetMask)
        bits = ip2bin(options.subnetMask).count('1')
        comp = [255-x for x in mask]
        netw = [addr[x] & mask[x] for x in range(len(addr))]
        cast = [comp[x] | netw[x] for x in range(len(comp))]
    elif (options.ipAddress != None and options.subnetMask == None):
        assert options.ipAddress.find('/') != -1
        (temp, bits) = options.ipAddress.split('/')
        assert is_ipv4(temp)
        assert int(bits) < 32
        assert int(bits) > 0
        addr = ip2list(temp)
        bits = int(bits)
        temp = "".join(["1"]*bits)+"".join(["0"]*(32-bits))
        mask = [bin2dec(temp[0:8]), bin2dec(temp[8:16]),
                bin2dec(temp[16:24]), bin2dec(temp[24:32])]
        comp = [255-x for x in mask]
        netw = [addr[x] & mask[x] for x in range(len(addr))]
        cast = [comp[x] | netw[x] for x in range(len(comp))]
    elif (options.ipAddress == None and options.subnetMask != None):
        assert is_ipv4(options.subnetMask)
        addr = [0,0,0,0]
        mask = ip2list(options.subnetMask)
        bits = ip2bin(options.subnetMask).count('1')
        comp = [255-x for x in mask]
        netw = [addr[x] & mask[x] for x in range(len(addr))]
        cast = [comp[x] | netw[x] for x in range(len(comp))]
    else: raise Exception("An IP or Subnet mask must be specified!")
    diff = dec2octets(int(options.lookFor))
    size = bin2dec("".join(["1"]*(32-bits)))-1

    # Is it valid? (aka. Is it in our specified range?)
    test = [diff[x] & mask[x] for x in range(len(diff))]

    # ========================================================================
    # Massage output...
    # ========================================================================
    address = ".".join([str(x) for x in addr])
    network = ".".join([str(x) for x in netw])
    netmask = ".".join([str(x) for x in mask])
    brdcast = ".".join([str(x) for x in cast])
    print ("There are %s addresses available in the %s/%s network "
           "range."%(size, network, bits))

    print ("Network: %s  Netmask: %s  "
           "Broadcast: %s"%(network, netmask, brdcast))

    if (int(options.lookFor) <= 0): sys.exit()
    if (test == [0,0,0,0]):
        plus = ".".join([str(netw[x] + diff[x]) for x in range(len(netw))])
        print ("The %s address in the %s/%s network "
               "is %s"%(numify(int(options.lookFor)), network, bits, plus))
    else:
        print ("The %s address is beyond the %s addresses in the %s/%s "
               "network."%(numify(int(options.lookFor)),size,network,bits))

RegEx – Lists & Hashes with Escape Characters

I recently ran across a need to match escape characters within a regular expression. The key here is that you need to make use of a fun little RegEx feature called “Zero-Width, Negative, Look-Back Assertion”. Its a loaded title, but crams a LOT of functionality into a little space… None. This all stems from a post I made on Python-Forum…

Just FYI, so that nobody hassles me later, you need to include the Python Regular Expression module in order to use any of these examples.

import re

Problem #1: How do you represent an “escaped character” in a Regular Expression?
Solution #1: Use positive look-back assertion on the existence of a single backslash prior to any character.

text=r'\f\o\o'
regex = r'(.(?:(?<=\\).))'
i = re.finditer(regex, text)
for m in i: print m.groups()

Problem #2: It is not possible to put the previous regex inside a character set. So how then do you create a set of “escaped characters + non-terminators”?
Solution #2: Use the “?” on each “character” in the set.

text='{foo:bar}{a:\/foo\.bar}'
regex = r'(\{(?:(?:(?<=\\).)?[^\{\}\[\]\/\.]?)+\})'
i = re.finditer(regex, text)
for m in i: print m.groups()

Problem #3: How do you keep multiple occurrences from matching?
Solution #3: Explicit match using anchors. In other words, use “^” and “$” to bound your matching appropriately.

text='{a:\/foo\.bar}'
regex = r'^(\{(?:(?:(?<=\\).)?[^\{\}\[\]\/\.]?)+\})$'
i = re.finditer(regex, text)
for m in i: print m.groups()

Problem #4: How do you match practically any character that could occur within 2 bounding characters (like /regex/ in a sed-esque fashion)?
Solution #4: Exactly like before, except now, the only character we cant have in the middle is one of the plain (non-escaped) bounding characters.

text='/g[@#$%\/{foo:bar}\/^&*()]/{a:\/foo\.bar}'
regex = r'(\/(?:(?:(?<=\\).)?[^\/]?)+\/)'
i = re.finditer(regex, text)
for m in i: print m.groups()

Problem #5: How do we avoid matching when two valid matches are split? Like “/foo/{a:b}/bar/”
Solution #5: Explicitly define the possible combinations of each regex subsection

text1='{a:\/foo\.bar}/foo\/bar/'
text2='/foo\/bar/{a:\/foo\.bar}'
regex=r'^(\{(?:(?:(?<=\\).)?[^\{\}\[\]\/\.]?)+\})(\/(?:(?:(?<=\\).)?[^\/]?)+\/)$'
i = re.finditer(regex, text1)
for m in i: print m.groups()
regex=r'^(\/(?:(?:(?<=\\).)?[^\/]?)+\/)(\{(?:(?:(?<=\\).)?[^\{\}\[\]\/\.]?)+\})$'
i = re.finditer(regex, text2)
for m in i: print m.groups()

Code Highlight

Just a small test of a new code-highlighting feature.

#!/usr/bin/env python
def test():
    try:
        print "Trying something..."
        int("foo") # Should cause an exception
        return "TRY"
    except:
        print "Excepting something..."
        return "EXCEPT"
    finally:
        print "Finalizing something..."
        return "FINALLY"
    return "FUNCTION"

if __name__ == "__main__":
    print test()

Honig Vineyard & Winery

Monterey Bay Aquarium

Adonia and I took a trip down to the Monterey Bay Aquarium.  On the way we ran into some interesting photography… although she might yell at me for paying more attention to that than the actual fun we were having.

On the way from Mountain View to Monterey, one of the first places we traveled through is Santa Cruz.  Of course there is the boardwalk, the beaches, and the shops on Main street… but one of my favorite places to stop is Marians Home-Made Ice Cream Shop.  There are dozens of different flavors like Lychee and Green Tea.  I think we went with the Oreo-Cookie…

Anyway, Monterey County has always been the home of many different businesses during the past century.  Sardines, Almonds, and… Artichokes… like the “Worlds Largest Artichoke” found a short minute off the road to Monterey.  Of course, it is pretty darn heavy.

Finally we got to the Aquarium, and just our luck, we got sidetracked on the way to a shark-feeding by some of the brightest Jellyfish I had ever seen.

I have to say I like the way the Jellyfish are purposely placed in bright blue tanks so that they show off their stark contrasting bodies.  Of course, I like the fact that this makes for better photography too though.

We finally wrapped the day up with a tour through the Pebble Beach area in Carmel.  Granted they charge you an arm and a leg just to drive the 17-mile road ($9.50 I believe), but they ARE definitely guarding some prime real-estate… which unfortunately I cannot share with you, as Adonia’s camera ran out of battery right about the start of the 17-mile drive.

Weekend in Las Vegas

Ok, so regardless of my feelings about gambling, and how much I despise losing money, I succomed to the nagging of my friends, and went to Las Vegas.

One of the biggest things that bothers me about casinos (aside from the “taking your money” business) is the look-and-feel.  Some casinos, regardless of what they may have been advised, do not go for the “timeless beauty” look… which 20 years later means they look dated… because they ARE dated.  Of course others (Bellagio, Venecian, etc) actually look good.

I know people always say that you should go see a show when you are in Las Vegas, but this brings up another point.  I am pretty sure that every single one-hit-wonder comedian I have ever heard of has a show in Las Vegas.  As an example, I was stunned to find out that Louis Anderson still existed, given that my childhood memorys were of the dozen-or-so “Life with Louie” episodes… but I digress.

The weather was great. The walk was long. There were definitely too many things to even SEE in one day.  There were private escort services & “girls for hire” being advertised by the thousands.  There were public displays of thousands of empty alcohol containers.  I was asked more than once if I was wanted to purchase crack or heroine.  AND, did I mention that the weather was great?

Altogether as I expected, and to quote Obi Wan: “You will never find a more wretched hive of scum and villainy”.

In the business of costumes…