By David Stewart on August 26, 2015
What would happen if you were hauling a trailer down the road, and suddenly you realized that the trailer you thought you were pulling passes you on the highway. (Which I guess can actually happen, I just saw a video capture of this exact disaster, happening to a trucker). You might not feel kind of blind-sided!
This is kind of the way I feel when a technology trend sneaks up on me and suddenly I feel like I'm way behind the curve.
It's a little how I felt when I first learned about how powerful Python is as a programming language. I had always thought of Python as the next generation after Perl, a popular scripting language which encompasses features from sed, awk, the Unix shell and inherits syntax from most of these. Another scripting language, in other words. OK, thought I, I know what a scripting language can be used for. Creating scripts. Simple automation. Maybe some data analysis. File processing, that kind of thing. Not real programming, after all.
I was sitting in a cafe in San Francisco in 2010 with Richard Purdie, who is a fellow of the Linux Foundation. Richard and I were discussing some technical challenge he was working through on Bitbake, which is the core build engine of the Yocto Project and written in Python.
BitBake is frankly pretty amazing, no matter what language it was implemented in. This code allows you to build a complete and fully functional Linux operating system from sources on a desktop computer in about an hour. The Python code will extract sources from about 1000 open source projects from around the internet, and knows how to interpret all of the source code repositories out there (Git, Subversion, Mercurial, etc etc). Once it parses a separate configuration file for each of these 1000 projects, it extracts the sources, applies source code patches, configures and compiles the code, creates a package for each of them in a package repository and then constructs a runnable Linux image and an application development SDK. Like I said, impressive!
We started digging into his technical challenge, which involved some matter of serializing objects using pickle onto a disk and later deserializing them as a performance optimization between runs. And then I was like, "Hold on. Objects? In a scripting language?" Yes, and much more. BitBake tries to consume as many processors as your system has, and on my little MacBook Air would start the fan blowing air as the processor heats up from the use. A scripting language!
In fact, Python has become the first programming language taught to incoming freshmen at universities as prestigious as MIT, as I heard from an MIT parent. It's loose typing and quick development turnaround helps get the mechanics of coding out of the way and helps students focus on the real learning value of making algorithms work.
Since it's so quick to create working code, a lot of data scientists are using Python to access their big data systems. There are Python bindings to many large ISV systems. Surprisingly, scientists in the high performance computing field are implementing their algorithms in Python.
The server world has really embraced Python in a big way. For example, the OpenStack project is a very popular Infrastructure as a Service offering, and most of it is written in Python. This makes Python a leader for Software Defined Infrastructure (SDI), Software Defined Storage (SDS) and Software Defined Networking (SDN).
Does this make Python important? You bet it does!
What about other dynamic languages? Are they important in the server world as well?
Well yes, as a matter of fact, how about PHP? If Python is easy to code in, PHP makes developing web pages even easier. After all, to quickly create a web page using PHP is simply a matter of annotating a web page with code which gets run on the web server when the page is invoked. Important? One survey I read claimed that 80% of the web pages on the internet use PHP! The most recent language survey I read claims that there are 9 million PHP programmers out there in the world.
However, interpreted languages with loose typing are, by their nature, fairly slow at executing code. Now if your entire multi-million-user web site is implemented in PHP, this can create a major cost for a company. Extend this to a site like Facebook which has over a billion users, and you get really excited to address slow performance without reimplementing your entire site in a more efficient language.
Enter HHVM, which is the open source implementation of the PHP engine that Facebook created.
In 2015, I was asked to create an engineering team which would work on speeding up these dynamic languages (Python, PHP, HHVM, Node.js, etc) and implementing Intel processor features where they make a difference. We have some interesting results now, and I think it's a good time to share some of these results and how you can take advantage of them for your server work.