This document is intended to lay out a path for how a typical Pynto compilation will take place. It begins with an empty Pynto environment and shows compilation will be done as different modules are added to the system and compiled. The example source code in question is as follows: module counter: class Counter: def __init__ (self, initial, inc): self.value = initial self.inc = inc return def add (self): self.value += self.inc return def __str__ (self): return "%s (%s)" % (self.value, self.inc) pass module firewall: import counter badrequests = counter.Counter (0,1) def filter_list (requests): res = 0 for req in requests: res += filter (req.data) pass return res def global_count (): return badrequests.value def filter (requestdata): def isbad (requestdata): if len (requestdata) < 3 or requestdata[0] != 'G' or \ requestdata[1] != 'E' or requestdata[2] != 'T': return 1 return 0 if isbad (requestdata): badrequests.add () return 1 return 0 ---------------------------------------------------------------------- Let's imagine that the module 'counter' is loaded first. When python loads a module it automatically runs the initialization code; it's very difficult to intercept the code before this point, so we don't bother. This means that Pynto begins operation with the Module object, which essentially consists of a dictionary. Pynto iterates through the keys of the dictionary to find the functions and classes contained. Each entry in the dictionary also receives a Symbol entry; in certain cases, the value is used to give the symbol an assumed value or type. In particular, globals mapped to function and class values are assumed to be constant; it'd be nice to find a way to determine which integer constants will be constant, but it's probably too risky to optimize them otherwise. All global symbols must be marked read/writable from outside jitted code. Once we have loaded up the initial symbols for the module, we process them recursively. It is possible not to, but it is desirable to jit as much as possible. We therefore begin processing the Class counter; like modules, classes are basically dictionaries by the time we see them*. Similarly to modules, we construct a Class code node and iterate through the keys and values in the class objects, creating a Symbol for each one. Like the module globals, these symbols are read-writable by non-jitted code. Unlike the module case, we should probably make the assumption that they won't change values no matter what they map to. After jitting the class itself, we jit each method. The init method's disassembly looks like this: Disassembly of __init__: 3 0 LOAD_FAST 1 (initial) 3 LOAD_FAST 0 (self) 6 STORE_ATTR 2 (value) 4 9 LOAD_FAST 2 (inc) 12 LOAD_FAST 0 (self) 15 STORE_ATTR 3 (inc) 5 18 LOAD_CONST 0 (None) 21 RETURN_VALUE 22 LOAD_CONST 0 (None) 25 RETURN_VALUE We want to convert this to a standard 3-operand form. This procedure is described in algorithms.txt. The initial 3-operand form generated is fairly uninteresting. It will look like: ATTR (self, "value") <- initial ATTR (self, "inc") <- inc RETURN None Now we want to do an initial alias analysis to create object tags. We create a generic "know nothing" object tag. However, since this is a method, we decide to also create an "instance" object tag that has type information indicating it is a class instance of type Counter or some single-inheritance subclass thereof. We now have two object tags: 'any' and 'instance'. Therefore, the symbol for 'self' is given the object tag of 'instance'. The parameters 'value' and 'inc' are given both 'any' and 'instance'; we don't know that they don't alias, especially since __init__ can be called at times other than construction. When we examine the 3-operand form now, we are able to determine then that an object of type Counter has two likely attributes: "value" and "inc". These are added to the type object for this class. This information can be used when generating code to compile attribute gets and sets into direct derefs rather than painful hashtable lookups. We can also use those entries in the type object to associate what typing information we know, which at this point is nothing. We are now finished with the __init__ method. We add the IR to a list, storing information about its type signature: -> None. i.e., it takes three arguments, one of which is a Counter object, the other can be of any type, and returns an object of type None. Let us examine the add() method now. The byte code for the ADD method looks like: 8 0 LOAD_FAST 0 (self) 3 DUP_TOP 4 LOAD_ATTR 1 (value) 7 LOAD_FAST 0 (self) 10 LOAD_ATTR 2 (inc) 13 INPLACE_ADD 14 ROT_TWO 15 STORE_ATTR 1 (value) 9 18 LOAD_CONST 0 (None) 21 RETURN_VALUE 22 LOAD_CONST 0 (None) 25 RETURN_VALUE This generates the following IR instructions: ATTR (self, value) -> t0 ATTR (self, inc) -> t1 t2 = t0 += t1 ATTR (self, value) <- t2 RETURN None We can apply the same typing guesses to the parameters of this function, meaning that we specialize self to be an instance of Counter. We don't need to add any attributes to the class type object because we already have value and inc, but we do look up the attributes and discover any typing information: at this point there is none, however. t0 and t1 are given new object tags based on the information in the class type object; in addition, they are also given the tags of all parameters and other external tags, because for all we know they could point at the same object. Now, the form of the in-place add may look strange to you. Well, the semantics aren't quite what the C-pseudo-code would suggest. The in-place add operaton (t1 += t0) potentially modifies the object that 't0' points at, but it also potentially creates a new object and leaves 't0' unmodified (this is what would happen if 't0' pointed at an integer constant, for example). In either case, the pointer to the result (which may be the same as 't0') is returned and assigned to t2. We then store t2 back into the attribute in case it has changed. We could only store t2 back if it is different from t0, but we don't do this optimization yet. Maybe later. Anyway, we have now covered the counter object in sufficient detail, I think. * - Usually ---------------------------------------------------------------------- Now the module 'firewall' is imported. It too arrives in the form of a dictionary with five entries. We iterate down the hashtable, creating symbols. In the case of 'counter', we detect that this is a reference to a module we already have in our listings, so we create a constant and point it at the appropriate code node for 'counter' we created in the last section. There are three functions, which we create code nodes for and point at. In addition, there is a global variable, badrequests: we create a Variable Symbol for this and an object tag that contains the typing information. Because the counter module was loaded (and jitted) first, this instance will be created by the version of Counter() we export from there, which means it will return to us a version with the self pointers optimized in the methods. This is good. If, however, we had not yet jitted the counter module, then this would be a "natural" instance of the Counter class. Natural means that it is was generated using the byte code and lacks the specializations we may apply to it --- in that case we would be able to do very little by way of optimization. As it is, when we see accesses to 'badrequests', we can choose to optimize based on the type information. The main thing this would effect is attribute access, which would be optimized to avoid hashtable lookups and use direct pointer manipulation. However, we will only make the assumptions about the type of badrequests if we are on an aggressive optimization mode. This is because the user could change the value of badrequests to be any sort of object; while this wouldn't cause us to do the wrong thing, as we would catch the change and revert the global functions to be more conservative, it would wind up forcing us to use a more unoptimized version. the code. The main optimizations that we do on firewall.py is to convert calls to the functions into true function calls rather than name lookups. In addition, we can inline the call to isbad() in filter().