Ionrock Dot Org

My Weblog

Seemingly Disconnected

It seems that my DNS service expired over the last week or so, the result being I missed some email. Life has also gotten in the way of my writing, which has added to the appearance of my disconnectedness. Even though I’m sure no one really even noticed, it still annoys me.

One of the reasons I haven’t been writing this past week is that my computer seemed to receive an update that it wasn’t happy with. I have a i7 MacBook Pro and unfortunately it has a history of locking up. I’m 99% it is a driver issue and my money is on thunderbolt being the cause. To make things work, VMWare Fusion, the tool I use to run Linux and do real work, seems to make things worse. It has been really frustrating to recognize I didn’t write anything for the day to sit down at my computer, see some odd pixelization and know the computer froze up again.

The other pain in my neck (and wallet) has been our van. We played a show on Tuesday and after loading the gear into the van at the practice space it wouldn’t start back up. It had been running poorly that day, but that happens every once in a while and it usually seems to work itself out. This time it didn’t. Fortunately, after we practiced it did start up and I drove it home. I’m going to take it up to the shop this afternoon and probably end up dropping way too much money to get it fixed.

My only saving grace is that if it is something that commonly breaks on the van, then my mechanic usually has a means of fixing it where it won’t break again. After buying the van we ended up finding out that there are a lot of components that fail. My mechanic is something of a diesel genius and can fix most of them in a way where they don’t break again (or he can get rid of them completely!). This is definitely a good thing as I have heard horror stories of people with similar engines fixing the same thing over and over again.

For those that do read and pay attention, my apologies. I should be back on track now, at least until my computer freezes up again.

Using Env Vars

Someone at work mentioned The Twelve Factor App in our IRC topic. It is a great read as it simply and concisely states the current general best practices for writing a distributed app.

One thing that did confuse though was the mention of Environment Variables for configuration. Here is what it said:

The twelve-factor app stores config in environment variables
(often shortened to env vars or env). Env vars are easy to change
between deploys without changing any code; unlike config files,
there is little chance of them being checked into the code repo
accidentally; and unlike custom config files, or other config
mechanisms such as Java System Properties, they are a language- and
OS-agnostic standard.

The idea of using env vars as a means of passing configuration information makes sense in a general case. But, if you application has more complicated configuration, it seems like it could be somewhat daunting.

That is just my initial impression coming from a YAML based configuration system. After taking a look at our production configuration I could see how most details could be reflected as traditional env vars. Where things do become more complicated it seems reasonable to consider moving that information to a service.

As a though experiment, I’m going to consider moving some of our configuration to env vars to see if it feels reasonable. Starting off, the vast majority of values seem like they could easily be migrated to prefixed variable names. As we do use YAML, we have some dictionary and list structures. The lists seem easy enough to parse as strings. For example:

# set the env var as JSON
import os
os.environ['APP_HOSTS'] = '["123.45.22.3", "123.45.22.8", "123.45.22.10"]'

# load the env var
import json
hosts = json.loads(os.environ['APP_HOSTS'])

While JSON seems like it might be somewhat heavy handed, it is important to have some standard otherwise you’d end up seeing lots of code that looks like:

hosts = os.environ['APP_HOSTS'].split()

Maybe this paradigm is totally fine. I would imagine it could break down though depending on the values and if they needed spaces.

For dictionaries it seems looking at our config that most could, and probably should, be moved to some service. It is still possible to use something like JSON, but again, I think that might prove cumbersome in some situations. Writing raw JSON in a BASH script might be rough. I’m also noticing that many times a dictionary really is more of a visual organizing tool. It is possible to flatten the dictionary using an obvious delimiter that could be used to expand things later if need be.

Here is an example:

conf = {
    'csv': {'extra': {'repair': True, 'delimiter': '|'},
            'type': 'text/csv'},
    'xml': {'extra': {'xsl': 'http://host/path/to/foo.xsl'},
            'type': 'text/xml'}
}

This might be flattened in the env vars as the following:

APP_CSV_EXTRA_REPAIR = True
APP_CSV_EXTRA_DELIMITER = "|"
APP_CSV_TYPE = "text/csv"

APP_XML_EXTRA_XSL = "http://host/path/to/foo.xsl"
APP_XML_TYPE = "text/xml"

What we need to do is take this kind of configuration and use it via env vars to configure a download service. Here is one option:

class DownloadService(object):
    @classmethod
    def from_env(cls, type):
        kwargs = {}

        vkey = ('app_%s' % type).upper()
        for key, value in os.environ.iteritems():
            if key.startswith(vkey + '_EXTRA'):
                kwargs[key.split('_', 3).lower()] = value

        kwargs['headers'] = {'Content-Type': os.environ['APP_%s_TYPE' % type.upper()]}

        return cls(**kwargs)

Offhand it feels a little bit kludgy, but I think the reality is that it is not that complicated. The fact I wrote this method inline in a minute or two suggests that wasting any more time to something cleaner is probably just a waste of time.

In summary, it seems pretty reasonable to define configuration via environment variables. One concern is that keeping the configuration for development and testing might be cumbersome at times. But with that in mind, there is no reason one couldn’t keep them in a YAML or JSON or even Python and simply generate them on the fly when you run the application. I’ve also found at times using environment variables can be tricky with scripts where I preferred to use command line flags. My gut tells me that the real issue is that I’m not doing the right thing wrapping some script or configuring how it is run.

I hope this exercise has been somewhat helpful. Obviously a complex configuration is going to end up with pretty massive set of environment variables. The gut reaction might be that it seems sloppy or that it will become too complex. This might very well be true, but I have a feeling it can be pretty easily managed. Writing a simple YAML to env var script would be pretty trivial. Maintaining a global namespace of env var names is effectively what Emacs does and I can say from experience that while the namespace has become huge over the years, I’ve never hit a collision.

The Envconf Tool

Continuing to think about using env vars, it seems like a really easy way to start using them would be to have a way of configuring them using a file in development. For example, in our apps at work we traditionally have some sort of a base config YAML file that keeps all our defaults. The YAML format is pretty helpful in keeping things organized for development but there is really no reason you couldn’t use environment variables.

Without further adieu, I wrote envconf. It is really meant for development where you have some configuration files and you want to read and create env vars from them. It only supports YAML files and for lists it uses JSON to create a more universally parseable value. I’m not crazy about this list support, but one easy work around is to explicitly use a string and parse it yourself. Not ideal, but probably good enough.

In theory if someone actually uses this and wants to use another config format like .ini, JSON or some other format, it should be pretty easy to add a new type. Implementing a different configuration type is a matter of reading a file and returning a dict, which shouldn’t be terribly hard.

I should also mention that envconf doesn’t overwrite anything currently in the environment. The result is that you could layer things if you wanted. Here is an example of how you could do it:

envconf -c dev.yaml envconf -c base.yaml run-my-server

You could also just pass it in explicitly in your shell:

APP_EXAMPLE=foo envconf -c base.yaml run-my-server

Unfortunately, I haven’t had a chance to try it out at my job as we have a semi-involved configuration pattern that would need to be adjusted and I don’t have the time. But hopefully I’ll get a chance to give it a go.

Coding Pride

Are you proud of the code you write? What is it about some code are you specifically proud of? For me, I’ve become less and less proud of my code.

This doesn’t seem like a bad thing. It helps me to be humble and less attached. Lacking pride doesn’t imply that I don’t care about the code, but simply that when I’m done, it doesn’t feel like something I’m excited about showing people.

I can say it does concern me slightly. My perspectives seem to have changed slightly looking at code that I respect. It is typically pretty boring and nothing too interesting really happens. The meat of the code ends up being comments that describe rather clearly what should be going on. None of this is a bad thing, but I’m also finding myself less interested in new language features or using a feature that that is not well supported. In short, my code has become rather conservative.

Some would argue that this is a good thing and generally, and it should be obvious, I agree. With that said, it also seems to be indicative of a more fundamental issue of questionning my skills. Is the reason I don’t use some experimental feature because I don’t trust it will continue to be developed or is it really that the feature isn’t very easy for me to use? That slick new language feature that everyone is talking about looks helpful, but I’m not sure I really understand what the big deal about it is. Maybe I’m missing something.

It is important that I recognize when my code is wrong or none optimal, but it also is important that I feel I can code confidently. My lack of pride in my code actually feels more like a lack of confidence. As is with most of life, the solution is plain old hard work.

Dad and adding Build Steps

I’ve been working on the same project at work for a while so when I have a little free time I like to work on Dad. I’m not sure if I’ve mentioned it before, but its a process manager application that is meant to support a more comprehensive set of management tasks. It came from my idea of a devserver much like Foreman, with biggest difference being that instead of simply controlling whether a process is up or down, you can ask it to perform tasks within its own sandboxed environment.

It got me thinking about how most people in the Python community end up looking to virtualenv for their deployment, when there might be other tactics. Virtualenv is a great tool and use it all the time for development. There are tons of people that find great success with it as a deployment tool as well, so I don’t play to sway the opinion away from it. I do think there could be better ways though.

One of the reasons developers choose to write applications in languages like Ruby or Python is because it is easy to get a large portion of the functionality done quickly. Honestly, during this phase of development a tool like virtualenv makes a ton of sense because it provides a similar speed when deploying. Installing all your dependencies at deploy time is not a huge deal because there aren’t that many and most of the time they don’t change. In other words, it isn’t broken so why fix it?

The reason to consider a different methodology early on is because it is easier to do it early on. Eventually the simple Django app you wrote is going to push the limits of what the framework offers out of the box. You start adding functionality that doesn’t have an obvious fit within the repo. As these problems evolve the application gets complex. The complexity moves from being a simple repository level complexity to abstractions via services and processes. There is nothing wrong with this, but it presents a different sort of problem to manage. You no longer are simply refactoring code, you are orchestrating services both in development for testing and in production. There is a lot that can change going from dev to production and I don’t believe that tools like virtualenv really help that issue. They do too many things.

With that in mind if you start out your application with a healthy appreciation for processes and services, there is a good chance you can avoid much of the pain of learning to orchestrate these details consistently. This is why I started writing Dad. It is a process manager that you can use in development to orchestrate your services. You can then use it to run your tests and eventually the idea is that your development process from code to test to production is same on your local machine as it is when you deploy to production.

The key to making this transition is to recognize the need for different steps of deployment. In Python it is easy to just pull some code on a server, run setup.py install in a virtualenv and fire off a command to start it up. The problem is when you app actually needs 10 other services, one of which is a database written in C++ and another is a Java application. When you have more steps for deployment, you have points in the process where you can create the necessary pieces to make deploying as simple as untarring a tarball vs. running setup.py, finding dependencies, downloading them, installing console scripts, etc.

The first step in a deployment process is to make a package. This is something that makes it easy to put files in the execution environment easy. Creating a tar.gz via setuptools is probably a good first step. RPMs or dpkg might be another option. But in either case, you need something you create that can be used to build the actual “thing” that will be deployed.

Once you have a package, you then need to make a build. The difference between a package and a build is that a build has to be able to be run in a production-like environment without having to do anything but copying all the files from the build. Pip supports a “freeze” operation where it takes all the currently installed requirements and their versions to create a requirements file that can be used to recreate an environment. This sort of function should happen at the package phase, prior to the build. The build stage should find all the requirements and compile them together accordingly. The result of that operation then is what is used to create the actual build.

After the build, it is time to actually deploy. The deployment should be really simple. You should have an execution environment where you are going to put the files. That is where the build is installed. This is where Dad comes in. When you add an application to a Dad server it creates a sandbox for you to create an execution environment. You have your executable files and supporting files as needed and Dad is configured to call commands in order to manage the processes.

While all the above seems like a ton of work it really is a lot easier than you might think. It is pretty easy to write a short and simple script or Makefile that has to run “python setup.py sdist”. From there it is pretty simple to run a command that installs it into a brand new virtualenv and resolves dependencies. At the end you can pop the build in your Dad sandbox and run the tests. All in all, it really should be simple, especially if start when your application is still manageable.

Lastly, Dad is no where near finished. The design has been hashed out a few times with different models and only now would I argue that the model is correct. There is still a lot to do. If it interests you, feel free to fork it and try hacking on it.

Devops Mindset

I’ve never said it, but I’ve never really been a fan of the devops concept. I have no problem with developers doing deployments or having root access to servers or something silly like that. My main complaint is that the divide between a sysadmins and developers is helpful as it forces an interface and protocol. I’m now thinking that perspective was wrong.

The devops movement always seemed to be born of the startup trend where developers are hired and must be responsible for every part of the system. Services like Amazon EC2 only made it a logical position to create in a small company that aims to build on the shoulders of giants. Seeing as I have no historical context to say whether my assumption was right or not, I’ll assume it was correct for the purpose of making my point.

The real impact of devops is providing empathy for both concerns in applications and system design. When you throw away the preconceived misconception each type of stake holder has, there are some really powerful solutions that can be created that help everyone out. The developers can write code to help cross the chasm between development and production while sysadmins can have their needs met more intimately with applications packaged and written with the system in mind.

In a way it is simply an optimization. Prior to devops, the consensus was that there was an actual wall you had to work around. Devops doesn’t remove the wall, but instead suggest that both sides work to build doors and windows that each can depend on. This is a much better situation. I’m glad I finally started to see it.

Talking About Music

It is extremely difficult to talk about how something should sound. When we practice we hear different ideas for how the song should progress. You have to reference where in the song you want to try something and give some semblance as to what you are going for. Those that are going to play whatever it is you’re thinking have to have some idea they can internalize and try to actually play. Rarely is the process simple even though it can feel like it should have been extremely simple.

One thing that I’ve noticed is that I usually want to communicate these ideas quickly. Part of the reason is simply because I don’t want to lose the idea. A rhythm or melody can be easy to forget when first write it because your hands haven’t memorized it yet and your ears most definitely haven’t. Another reason I want to try and speak quickly about what I’m thinking is that experimenting takes a very finite amount of time. When you have a song that is 4 minutes and you are practicing for 2 hours, the most you could play the song is 30 times. The reality is you’re going to take a minute or two in between each play and if someone messes something up (which is not uncommon at all), you might have to start over. When you have such a limited time period to work, communicating efficiently is critical.

That last sentence though should speak volumes because trying to say something quickly is rarely going to be the most effective means of saying what you mean. This is especially when you’re talking about music where the concepts can be really abstract and qualities are subjective. I’m coming to the conclusion if you want to communicate some musical idea there is no better way than to just try and play it. Take the time to set a context and emphasize the part(s) you are working on so everyone see when and what you’d like to change. This kind of patience is not something I seem to have, but I’m most definitely going to try and learn how to take my time and become more effective.

Testing MongoDB

We’ve been noticing some issues with our data in MongoDB. Seeing as others have had less than stellar experiences, it seemed like we should make sure that MongoDB wasn’t the one causing problems.

My main goal is to see if incrementing updates by a large set of clients can cause data to be corrupted. I’m defining corruption as anything that diverges from the original document. Specifically, we’re looking for some keys that should have an answer but don’t. My theory is that under some conditions it’s possible develop a race condition when writing the documents.

The good news is that so far I haven’t found anything suggesting my theory is correct. The bad news is that the test takes an extremely long time to run. This could be a function of the python client not being very fast, but I suspect MongoDB is at least partially to blame based on the logs. There are some long operations like moving indexes and creating new datafiles that seem to be slower than expected, which I’m assuming is because of the heavy write load.

The larger picture this test paints is that incrementing documents in MongoDB is probably a bad idea for any mission critical or highly available app. The incremental updates play into MongoDBs weaknesses such as the global write lock and the way it uses mmap to manage how files are created. For our application I have long thought that we should have a two phase system. One data store for “live” data and keep static data in another store. In an initial iteration of this design I would picture a MongoDB instance for both and migrate data inbetween, doing some compaction along the way. Hopefully these tests, even if they prove MongoDB has been doing fine, will help support my design ideas.

The Skill of Listening

Usually a couple times a year most or all of the remote developers get together to take some time to meet and reset. The goal is to have some time together to discuss and design some of the bigger picture details of our systems as well as work together to sprint on some code.

I remember one specific instance where we had some new developers and we were discussing some potential changes to our deployment system. One of the new developers had a rather strong opinion about keeping our own sources and packaging everything together when we release. My opinion was the exact opposite of course. It seemed ridiculous to avoid the convenience things like setuptools and easy_install for what I thought was no real gain.

Every once in a while when there is a technical discussion it is helpful for me to think back to that discussion. My opinions have changed and now I agree with his points. At the time I had a bias that was unsubstantiated. The fuel driving me to disagree was really my own close mindedness to new ideas. It was certainly OK to disagree, but my resolution should have been to recognize the impasse and contemplate on the ideas presented. Instead, I was impatient and kept arguing.

None of this was damaging or that heated, but lately I’ve been thinking about it a lot along side other technical arguments I’ve gotten myself into. The theme I keep coming back to is the importance of listening. For myself, that means explicitly shutting my mouth and thinking about the problem. Sometimes it means talking about it to myself out loud or trying to write some code. The main thing is that I stop arguing and start listening in order to consider and compare the ideas.

The other thing about listening is that it does not imply that my idea or direction is wrong. I’m not giving up on the argument or becoming acquiescent. This is also important because assuming I’m wrong means I’m not listening to myself and giving my own ideas a chance to mature and develop.

This process is something that takes a lot of practice and between writing code and writing music, I get my fair share. The irony is that only now am I realizing how poor my listening skills actually are, even though I could have been practicing for most of my life. You never stop learning.

Broken Python Packaging

Over the weekend I took some time to take a project at work and try to build a set of binary packages. My goal was to be able to untar a package into a virtualenv and it work correctly. This isn’t that different from distributing egg packages or a RPM, so I figured it would be pretty easy.

My first step was to determine all its requirements. To do this, I created a blank virtualenv and installed my app. I then used pip freeze to record all the eventual requirements. Again, my goal was to have a set of package like files that can be untarred (or something similar) that will install the files in the virtualenv correctly such that I don’t need to perform some sort of build (compile C extensions for example) or get other requirements.

Having my requirements in place, I then starting downloading their packages from our local cheeseshop using pip’s “–no-install” flag. Again, this was very convenient and felt promising.

Once everything was downloaded, my script would then visit each package and try to build a binary package. I tried quite a few options here but none seemed to work correctly. The biggest problem was that each package had some inconsistencies that made using the same command for all of them fail. One package gave an odd character error when trying to create a tar.gz. Another didn’t recognize different bdist formats. Trying an RPM format on a whim was useless. Taking a step back I tried doing a “build” of each package and manually putting them together in a tar, but that was non-trivial and different per-package.

I’m going ahead and giving up for the time being as it is clear that Python packaging currently has a requirement to use source vs. pre-built packages. This is really too bad because often times distributing source and forcing your end-user system to have the necessary tools makes a package unusable. In my case it means that releasing a new deployment requires running the through the entire setuptools process for each package that gets installed. The benefit of avoiding this process is that you reduce the number of variables present when deploying. This makes a deployment much faster, which becomes more and more important in a distributed environment.

Hopefully with distutils2 and Python 3 the community can find some better solutions for packaging. I understand that the source based installation makes a lot of sense in development and even in many production environments, but that shouldn’t make a consistent binary package system impossible.

Paver and Throw Away Scripts

I’ve been making an effort to automate as much as I can recently. Part of this effort has been to utilize more Unix tools, but as a Python programmer, it isn’t always obvious how to combine the two. Paver is a build tool written in Python that aims to be similar to Rake for Ruby. If you’ve ever tried to maintain a Makefile and have felt out of your element, Paver is a great tool to help bridge the gap. It doesn’t have the same target based design that make does, but in terms of keeping a collection of operations dealing with files and/or running processes, Paver does a great job to bridge the gap between Python and the shell.

Beyond being helpful in builds, Paver provides a great tool for writing throw away scripts. If you’ve ever had a task that you needed some code for that didn’t need a full fledged module or package, then a pavement file can help to keep things organized. Its concept of tasks helps to keep small operations organized and allows you to keep your code semi-modular as you hack away. Paver also provides some helpful tools to make things like command line flags and input simple and direct. If the throwaway code does end up becoming a module that to needs stick around, you are one step closer to making it official as Paver provides some tooling for setuptools.

Using the simple example of examining log messages, I’ll show you how Paver makes the process really simple and intuitive.

First off, you have a log file somewhere you want to copy to your system. The easiest thing to do would be to copy it via scp. Here is an example in Paver.

# we'll assume the rest of these examples import this
from paver.easy import *

@task
def grab_logfile():
    sh('scp eric@myhost.org:/var/log/myapp.log .')

The ‘sh’ function calls a command much like the Popen class. You can capture the output as well in a variable. Another helpful aspect is that you can add command line arguments. Here is a good example of how you can find the package name and version in a Python package.

@task
@cmdopts([('pkg=', 'p', 'The path to the package')])
def pkg_name_version():
    if not options.pkg_name_version.get('pkg'):
        print 'I need a package name'

    scratch_dir = path('scratch_dir')
    scratch_dir.mkdir()

    pkg = path(options.pkg_name_version.pkg)
    pkg.copy(scratch_dir)
    sh('tar zxvf %s' % (pkg.basename()), cwd=scratch_dir)
    name = sh('python setup.py --name', cwd=(scratch_dir / pkg.basename())).strip()
    version = sh('python setup.py --version, cwd=(scratch_dir / pkg.basename())).strip()
    print 'Name: %s' % name
    print 'Version: %s' % version

I’m making an assumption that you have a tar.gz package that has the same name as the package. The code uses a ‘-p’ or ‘–pkg’ flag to get the package name. It makes a scratch directory where it will copy the tar.gz into. From there we unpack the tar.gz and ask the package’s setup.py to tell us the name and version.

You can see it is really simple to do things like run commands in specific directories and capture input as needed. Also, Paver includes a really handy path library that helps to make path operations more intuitive.

I haven’t really covered anything that isn’t in the docs, but hopefully you can see how some of its tools help make simple throw away scripts easier. It should also be clear that Paver doesn’t make these scripts perfect. You can see from my example above that in order to provide a better interface you’d probably want to use something than Paver’s built in command line options support. But for a throw away script, it is simple and gets the job done. The same goes for the path library. Sometimes it can be a little verbose at times because you need to be more specific by using things like the basename or abspath method. Again, it gets the job done adding just the right amount of framework to make things easier, not perfect.

The big win with Paver is that you have all the benefits of a Python environment while having easy access to the shell. Here is an example of reading and filtering a log file with Popen vs. Paver.

@task
def read_with_popen():
    log = path('/var/log/app.log')

    p1 = Popen(['tail', '-f', log], stdout=PIPE)
    p2 = Popen(['grep', 'foo'], stdin=p1.stdout, stdout=PIPE)
    p3 = Popen(['grep', '-v', 'bar'], stdin=p2.stdout)

@task
def read_with_sh():
    log = path('/var/log/app.log')
    sh('tail -f %s | grep foo | grep -v bar' % log)

You can always redirect the output to some file or you could use the ‘capture=True’ argument in the ‘sh’ call to do further processing. Either method certainly works, but for a quick script, Paver does a great job utilizing an obvious pattern that allows quick access from Python.

I’ve never been a huge advocate of Paver in the past because it didn’t seem like a critical tool, but I’m beginning to become a real fan. It is a tool that you can live without if you’ve never had it, but once you recognize where it excels and you begin to use it, it quickly becomes indispensable.

Instinctual Programming

Programmers often are searching for optimizations to their workflow. Editors, shell scripts and customizable tools are all examples making your development experience faster. As a programmer, you dedicate your life to knowing tools well in order to use them effectively. Effectiveness with many tools is a matter of making their use instinctual.

The irony of instinctual programming is that the tools you’ve become dedicated to finally get out of the way of your thinking and focusing on a problem. You find flow in thinking about and solving problems. The code ceases to be something you are typing into your editor and takes shape on your screen as though you are publishing your thoughts as you think them. Your body ceases to act based on your command and instead listens to an instinct you’ve developed through dedication to your tools and environment.

As nice as this may sound it is critical to recognize what I said in my previous paragraph. The tools you are dedicated to finally get out of your way. All the tooling and customizations you’ve made to optimize your development experience fall away to the background. Many times the tools you believe really help may actually be standing in the way of your focus and the natural instincts you have to solve the problem.

I’m not suggesting you shouldn’t use powerful tools. I can say from experience there have been many a time when my job was not to conceptualize and implement complex algorithms. It was actually mundane text editing that required next to nothing in terms of critical thinking. In these cases, mastering my editor became an optimization to help in hurrying through the mundane to get back to the important work of the day.

But just as my editor and tools have helped me to optimize the mundane, they have proven to be distracting. An infinitely customizable piece of software offers endless optimizations as well as an endless supply of pointless things to change. Change inhibits instinct. Instinct is when you body takes over and your mind is free to do other things. There is a reason cars all have roughly the same interface. Drivers immediately know how to drive any vehicle and to do so instinctively because they do not have to think about the basic tasks like steering, breaking and acceleration.

In the development world this relates to things like working with source control, build/test scripts and deployment processes. All these things should be instinctual processes that do not require excessive thinking. The same goes for documentation. Imagine your favorite programming language didn’t have a central place for documentation. You had to constantly search for random articles in order to find out how to use the language. You’d quickly change your opinion of the language and find something that was more usable.

Instinctual programming is not about optimization. It is about repetition to the point of mastery. It is important to recognize when the process you are repeating may not be optimal, but at the same time, it is important to beware that constant change will never allow you to act instinctively.