Skip to content
October 8, 2012 / ionrock

Forcing a Refresh

Web applications are interesting because it is exceptionally challenging to maintain state. The client and the server act independently and there very little a web application developer can do to reliably keep the client and server in perfect sync. Yet, there are times where you need to sync your client with the server.

When a server API changes it means your client code needs change. When the update to the server involves multiple processes, there is even more chance for the client and server to get out of sync. One way to make sure our client and server processes are in sync is to force the client to reload its resources. This involves downloading the new versions of static resources such as JavaScript, CSS and images.

In order to force our client to refresh, we need a couple pieces in place. First we need to be sure our client communicates what version it is running. Second we need our client to understand when the response from the server is indicating we need to refresh. There are other pieces that can be developed such as a UI explaining to the user what is happening, but to start this is all we need.

To start we should make sure our client has access to some version number. The version number is defined by the server and can be evaluated however you want. An easy way to do add the version is via hidden element in the HTML. If you wanted to do something clever you could limit a refresh when there is a major version bump. For simplicity sake, I’d recommend using the application version and only do simple equality comparisons when deciding whether to return a refresh error.

Whatever version number is in the HTML needs to be sent to the server on each request. How you send that data is totally up to you. I’ve used a ‘__version__’ named value submitted as a form value. You could have a __version__ key in a JSON document you post, make it part of the URL, use a header or use a cookie value. Whatever it is needs to be understood by the server API.

Once you have a version number being sent by your client, the server then needs to test whether or not the versions match before returning a response. If they don’t match, then you should send an error message. For example, you could return a 400 status along with a JSON message that says to reload the page. It is important that you return an error to the client rather than simply return a redirect because the client needs to refresh the page and make sure to avoid cached resources on the refresh. When the client gets the error message, the JavaScript can call ‘window.location.reload(true)’ in order to reload the page, avoiding the cache.

It should be noted that this doesn’t avoid using things like timestamps in the paths to static resources. Making the URL paths reflect the age of the file is still very helpful as it makes sure that subsequent reloads will reference completely different resources that are not cached. The system I’ve described is focused on reloading the initial page vs. performing some operation to clear the cache or determine what resources need to be reloaded. By keeping these concerns separate, we can keep the implementation simple.

I don’t think any of this is extremely difficult, but I think it is a critical part of any distributed web app. When you release new software to a cluster of applications, you want to slowly roll it out. This methodology ensures that as a roll out occurs, your client code can be upgraded at the same time as well as client code that has become stale.

September 16, 2012 / ionrock

Better CherryPy Tools

CherryPy allows creating extensions to the request level functionality by means of Tools. There are a standard set of tools that provide extra functionality such as gzipping responses, HTTP auth, and sessions. You can also create your own tools to provide specialized functionality as needed for your application.

From a design perspective, Tools are exceptionally powerful. Like WSGI middleware, they provide a way to layer features on top of request application handlers without adding extra complexity. The only problem is that writing your own custom tools can be a little tricky, depending on what you want to accomplish.

What makes them a little tricky is that they can be used in many different scenarios. You can attach them to hook points in the request cycle and they can be used as decorators. A Tool also requires that you attach a callable to a specific hook point, which is convenient when only one hook point is necessary for some functionality. When you need a tool to attach to more than one hook point, it requires a somewhat awkward extra method that can be a little confusing. None of this is impossible to understand, but it can be less than obvious when you are first working on writing Tools.

In order to make this process easier, I wrote an alternative Tool base class to make writing tools a little easier. Lets start with a simple tool that logs when a request starts and ends.

import cherrypy


class BeforeAndAfterTool(cherrypy.Tool):

    def __init__(self):
        # Attach the initial handler. This is also the method that
        # would be used with a decorator.
        super(BeforeAndAfterTool, self).__init__('before_handler',
                                                 self.log_start)

    def _setup(self):
        # CherryPy will call this function when the tools is turned
        # "on" in the config. 
        super(BeforeAndAfterTool, self)._setup()

        # The cherrypy.Tool._setup method compiles the configuration
        # and adds it to the hooks.attach call so it gets passed to the
        # attached function.
        conf = self._merged_args()
        cherrypy.request.hooks.attach('before_finalize', 
                                      self.log_end, **conf)

    def log_start(self, **conf):
        cherrypy.log('before_handler called')

    def log_end(self, **conf):
        cherrypy.log('before_finalize called')


cherrypy.tools.before_and_after = BeforeAndAfterTool()

If you’ve never written a tool the above is a decent example to explain the process of attaching more than one hook. It should also make it clear why this API is not exactly obvious. Why do you attach the ‘before_handler’ in the __init__ call? There is really no reason. The Tool API has to accept an initial callable in order to allow using it as a decorator. Similarly, we could have used the log_start method to attach our ‘before_finalize’ and avoid using the _setup at all. What methodology is correct? There isn’t a right or wrong way.

It would be nice if we had a slightly more straightforward way of creating tools that was clearer in how they worked. This was my goal in writing the SimpleTool base class. SimpleTool provides a very simple wrapper around the above pattern to make creating a tool a bit more straightforward.

Here is the code for the actual base class.

from cherrypy import Tool
from cherrypy._cprequest import hookpoints


class SimpleTool(Tool):

    def __init__(self, point=None, callable=None):
        self._point = point
        self._name = None
        self._priority = 50
        self._setargs()

    def _setup(self):
        conf = self._merged_args()
        hooks = cherrypy.request.hooks
        for hookpoint in hookpoints:
            if hasattr(self, hookpoint):
                func = getattr(self, hookpoint)
                p = getattr(func, 'priority', self._priority)
                hooks.attach(hookpoint, func, priority=p, **conf)

Here is the example tool above, using this new base class.

class BeforeAndAfterTool(SimpleTool):

    def before_handler(self, **conf):
        cherrypy.log('before_handler called')
    callable = before_handler

    def before_finalize(self, **conf):
        cherrypy.log('before_finalize called')

How does that look? Is it a little more obvious to see what is going on? I think so.

There are still some features we haven’t covered yet. When hook points are attached, they are ordered according by a priority. There are two ways to set the priority. The first is by setting a priority value on the method itself.

class HighPriorityTool(SimpleTool):

    def before_handler(self, **conf):
        cherrypy.log('this is high priority!')
    before_handler.priority = 10

The second way to set the priority is via the _priority attribute. This will be the default priority for any hook functions. Here is an example using the _priority attribute.

class PriorityTwentyTool(SimpleTool):

  _priority = 20

  def on_start_resource(self, **conf):
      prep_db_connections()

The last aspect of tools we haven’t covered yet is how to use these tools as decorators. As I mentioned earlier, the initial callable passed to the Tool is used for the decorator functionality. Using the SimpleTool base class it is simply a matter of setting the callable attribute.

CherryPy is pretty much voodoo free, so implementing the default tool behavior where the initial callable is applied when the tool is used as a decorator is straight forward.

class DefaultTool(SimpleTool):
    def on_start_resource(self, **conf):
        cherrypy.log('starting a resource')
    callable = on_start_resource

Pretty simple right?

If you use CherryPy and give this base class a try, let me know how it works out. Likewise, if you think the API could be improved I love to hear any suggestions.

As an aside, if you are curious what Tools bring to the table over WSGI middleware, there is an important distinction. WSGI ends up nesting function calls where tools are called directly. The result is that if you utilize a lot of tools, the cumulative effect is much smaller compared to WSGI middleware. Most of the time this doesn’t make a huge difference, but it is good to know that if you use Tools in your application design, you be insured against the tools eventually becoming a bottleneck. The other benefit of tools is that they are much simpler to write and can be easily applied to content consistently via the CherryPy framework (ie via the config) rather than a simple decorator. These are not huge gains, but as complexity grows over time, Tools are a great way to keep the code simple.

August 19, 2012 / ionrock

Silly Lawsuits

I just skimmed an article arguing the Apple vs. Samsung case doesn’t really matter. It is a trend I’ve noticed on Slashdot as of late where articles increasingly touch on things like patents and lawsuits rather than actual technology.

These copy and patent cases have little actual meaning, yet they are really dangerous. These companies are using the courts and patents to hurt competitors. It is nothing more than an expensive tactic that we pay for as a society. We should be angry that companies are using our courts as tools in their quest to gain marketshare and fight competitors. If that was the intent by providing things like patents, then we need to completely rethink the way we protect the ideas of others.

The root of the issue comes down to the ability to compete. A patent can be a valuable asset when you are talking about an inventor trying to build a business. The inventor can use the patent as a means of protecting the idea against those that can immediately copy and bring a product to market. When we are talking about companies like Apple and Samsung, both have no problems whatsoever taking an idea from patent to product reasonbly quickly. Do they really need patent protection when they have the means to beat competitors fairly? Should we allow these companies to sue simply because the cases look the same?

The situation is similar to privacy for celebrities. The courts have ruled that when a person is considered a celebrity, you can use images taken in public without that person’s consent. I’m not entirely sure of the logic of the court, but it makes some sense in my mind. Famous people could create an entire industry based on suing news corporations for unsanctioned photos. Similarly, if a celebrity doesn’t want a photo to be taken, they have the means to prevent it from happening. Most of these patent lawsuits are not about protecting ideas. They are celebrity companies using the patent system as business strategy.

August 14, 2012 / ionrock

Decoupling Data Storage and The UI

One thing that has always frustrated me about MongoDB is that it is very difficult to assert your data has been written. Eventual consistency is a tricky thing. You can’t always be positive the next read you make includes all the data. How do you get around this limitation?

One technique is to use a smart client. Rather than store everything in the database and rely completely on it as your primary source of data, your client loads the state and maintains its own state internally. This does not work when the client needs to consider a rich hierarchy of records, but for things like a user session’s settings, it is pretty reasonable.

For example, say you have a list of records you are displaying in your browser. Your initial request loads the records and displays them all. When you change the name of some record, the user interface is updated directly from the submitted data, while it is persisted to the database in the background. If the person managed to open another browser and reach the same UI, the record might not yet be updated, but that is the tradeoff you make for eventual consistency.

This technique works relatively well as long as the client is able to take the state and directly apply it to the user interface. If the new data impacts a wide variety of data or requires information that must be coordinated / evaluated centrally, the result is the client becomes much more complicated. For example, imagine if you TurboTax had to do all the calculations for the tax code in the browser. It ceases to be a valid tactic.

Just because you need to visit the server for some evaluation of your data, it doesn’t mean you need to store the data in your database prior to evaluating the data. Most data driven applications use some concept of a model in order to translate the data from the data store format to your application format. There is no reason your application cannot submit data that gets translated to the same model format and is evaluated for a result.

I’ll use an example from my job to explain how to implement this kind of design.

As I mentioned numerous times, we do opinion polling. This involves writing questionnaires in a specialized DSL. The DSL supports things like logic blocks and flow control so authors can ask different questions and process data based on the answers. The flow control and logic are too complex to consider implementing them in the browser. It would mean introducing client issues into a core element of our processing, which is an unnecessary addition of complexity. The problem is that we need to consider this complex logic when finding the next question (or questions) we want to ask.

Currently we require that for each submitted answer, we have to write it to the database before returning our next question to the UI. This requirement is because the user might contact another node in the cluster. Therefore, we need to be sure when there is a node change, the new node pulls it state from our database.

What I propose then is that we decouple the data submission from the evaluation for the next page. When a question is answered we submit our data to a data queue that will process it and store it in the database. Rather than waiting on an answer, we send our newly updated state and current location in the interview to an evaluation service that will take the state and find the next question we need to ask.

It is possible that the user could try to change clients and switch nodes between the time the data is actually written. The worst case scenario in this situation is they might be asked the same question twice. It is reasonable to require that the data be written and made available within a few seconds, which would virtually eliminate this scenario from happening.

One thing I haven’t touched on is whether or not the data is validated before writing or when looking for the next question. I would presume the former is a better tact as the validation should be relatively quick and we reduce the chance of bad data being written. I should also mention that in our scenario, our data, once written is effectively static. This means that when requesting the data on a node change, there is a minimum requirement that it be available in a format suitable for interviewing. This might not be the same format we use for analysis. The analyzed format could require more time to process, but that time need not be considered for maintaining state during interviews.

Has anyone tried this methodology before? Any pitfalls or obvious problems I’m missing?

August 14, 2012 / ionrock

Decoupling Data Storage and The UI

One thing that has always frustrated me about MongoDB is that it is very difficult to assert your data has been written. Eventual consistency is a tricky thing. You can’t always be positive the next read you make includes all the data. How do you get around this limitation?

One technique is to use a smart client. Rather than store everything in the database and rely completely on it as your primary source of data, your client loads the state and maintains its own state internally. This does not work when the client needs to consider a rich hierarchy of records, but for things like a user session’s settings, it is pretty reasonable.

For example, say you have a list of records you are displaying in your browser. Your initial request loads the records and displays them all. When you change the name of some record, the user interface is updated directly from the submitted data, while it is persisted to the database in the background. If the person managed to open another browser and reach the same UI, the record might not yet be updated, but that is the tradeoff you make for eventual consistency.

This technique works relatively well as long as the client is able to take the state and directly apply it to the user interface. If the new data impacts a wide variety of data or requires information that must be coordinated / evaluated centrally, the result is the client becomes much more complicated. For example, imagine if you TurboTax had to do all the calculations for the tax code in the browser. It ceases to be a valid tactic.

Just because you need to visit the server for some evaluation of your data, it doesn’t mean you need to store the data in your database prior to evaluating the data. Most data driven applications use some concept of a model in order to translate the data from the data store format to your application format. There is no reason your application cannot submit data that gets translated to the same model format and is evaluated for a result.

I’ll use an example from my job to explain how to implement this kind of design.

As I mentioned numerous times, we do opinion polling. This involves writing questionnaires in a specialized DSL. The DSL supports things like logic blocks and flow control so authors can ask different questions and process data based on the answers. The flow control and logic are too complex to consider implementing them in the browser. It would mean introducing client issues into a core element of our processing, which is an unnecessary addition of complexity. The problem is that we need to consider this complex logic when finding the next question (or questions) we want to ask.

Currently we require that for each submitted answer, we have to write it to the database before returning our next question to the UI. This requirement is because the user might contact another node in the cluster. Therefore, we need to be sure when there is a node change, the new node pulls it state from our database.

What I propose then is that we decouple the data submission from the evaluation for the next page. When a question is answered we submit our data to a data queue that will process it and store it in the database. Rather than waiting on an answer, we send our newly updated state and current location in the interview to an evaluation service that will take the state and find the next question we need to ask.

It is possible that the user could try to change clients and switch nodes between the time the data is actually written. The worst case scenario in this situation is they might be asked the same question twice. It is reasonable to require that the data be written and made available within a few seconds, which would virtually eliminate this scenario from happening.

One thing I haven’t touched on is whether or not the data is validated before writing or when looking for the next question. I would presume the former is a better tact as the validation should be relatively quick and we reduce the chance of bad data being written. I should also mention that in our scenario, our data, once written is effectively static. This means that when requesting the data on a node change, there is a minimum requirement that it be available in a format suitable for interviewing. This might not be the same format we use for analysis. The analyzed format could require more time to process, but that time need not be considered for maintaining state during interviews.

Has anyone tried this methodology before? Any pitfalls or obvious problems I’m missing?

August 13, 2012 / ionrock

Announcing AutoRebuild

At work we’ve started using sass for our CSS. The developer actually working on the code asked if we had a good way to rebuild the CSS when the sass file changed. I said no, but it would be really easy to write a CherryPy plugin for it.

AutoRebuild is that plugin.

AutoRebuild will let you configure a set of glob patterns and watch the files for changes. If they change, it will trigger a function to be called. It is up to you to write the function, but the README provides an example of how you configure a function to call make.

This plugin is really simple, but that is really the point. CherryPy makes it really easy to add integration points so working with other services and tools is easy and automated.

August 12, 2012 / ionrock

Libertarian Programming

If you haven’t already read it, Steve Yegge wrote a post providing an axis of alignment for programmers. The basic idea is that programmers tend to fall on spectrum between liberal and conservative based on the aversion to risk.

For the most part I get it and can agree to an extent. My one beef is that the perspective on risk seems rather one sided. The question of whether something will break for users is only part of the story. The boring nature of “conservative” developers might appear to be solely in fear for the user’s well being, but in my experience, the aversion of risk actually involves everyone involved with the code.

People, while having some pretty strong differences, still have survival instincts. If you are getting burned, your natural instincts kick in and you move extremely quickly. In other scientific fields, we’ve been burned plenty of times and as such, our industries reflect good instincts. Companies have had to shut their doors due to mistakes and carelessness, which in turn “burned” the entire industry in such a way as to cause an instinctual reaction.

Why is it in programming we haven’t seen a similar trend? Sure, we’ve seen source control and the Joel Test become relatively standard, but past that, what have we learned as an industry that is clearly the best practice for everyone involved in coding?

I have a theory why we have been somewhat slow developing our survival instinct.

If you follow sites like Slashdot, Hacker News and Reddit, you’ll see articles come up every once in a while that deal with age and programming. A quick search for “programming age” quickly returns some examples questioning if age is a limiting factor for a programmer. Generally, programming is considered to be something for the young, even if we don’t say so out right.

The result of this subtle ageism is that we often think new concepts can only come from young minds who have yet to be jaded by the old ways of thinking. The liberal programmers who don’t acquiesce to the conservative curmudgeon down the hall are the ones who will truly revolutionize the industry. As a culture, the new technology is what is most important, with the youth having the loudest voice.

Steve suggests that the longer you are a programmer and experience the pains of failure, you become more risk averse, hence becoming more conservative. I’d argue that the real risk you are adverse to involves spreading incomprehensible code and wasting the time of those that will come after you.

The developers I respect most have a knack for taking old code that makes no sense and turning it into clear and concise code with tests, tooling and documentation. Even though they are not making huge strides in new user functionality or creating new models for dealing with big data, they are exponentially saving the precious time of every other developer that works on the code. The potentially boring decisions and technology they used manages to hide complexity so effectively, it never seems to rear its head again.

If I were to put a political category on this line of thinking, I’d say it was Libertarian. New technology is great, as long as it doesn’t effect the liberty of other developers. Every new feature and bug fix should be tested for validity as well as reviewed in light of whether it adversely effects the ability of other developers to work on the code.

If I were describe myself, I’m a libertarian programmer. My desire is to write code that helps the users while staying out of the way of others I work with. I don’t always do a great job of this, but I’ll claim the naivete of youth and mention that I’m listening to my elders in order to overcome my lack of understanding.

Passionate programmers are going to disagree. It can be helpful to put terms on the different perspectives because, as Steve says, it provides an easy way to agree to disagree. At the same token, our industry makes a lot of mistakes and assumes that our flexibility and disagreements are simply clashes of opinion. The fact is our metric is often too focused on the result without considering the means. The time we save our users can be exponentially increased by saving our developers time.

The next time you find it difficult to read some code or set of tests it might be worthwhile to consider what impact your code has on others. The freedom you feel when writing new code is because you have not been encumbered by the code of others. There is no reason that when you are working on code others have written, the same freedom is not possible. As a libertarian coder, my goal is to write code that protects liberty of my fellow programmer.

Follow

Get every new post delivered to your Inbox.