NYT4, the fourth and longest living version of nytimes.com died in March, 2018 after serving New York Times content for 17 years.
NYT4 is survived by Jeff Damens and Oliver Karlin, the engineers who invented Context, a programming language that powered the site, and dozens of other engineers who worked on the site over the years.
Context was born in 2001 in the office of Scott Meyer, then general manager of nytimes.com. Meyer wanted to make the New York Times website more personal by adding readers’ usernames to the top of the homepage. At the time, nytimes.com was a static site that could not be customized for each person, but Meyer wanted that to change.
Damens and Karlin could have hacked something together just to add the username. They could have used PHP and CGI, which were common technologies in 2001. But, as Karlin pointed out, “You didn’t want to CGI-up the homepage to put one dynamic text in there.”
The developers had concerns that CGI wouldn’t scale to handle traffic on the homepage, which even then was growing quickly. They were also skeptical that Meyer’s vision for personalizing nytimes.com would end with the addition of a username.
Instead, they created Context, a lightweight language that was compiled to run on The New York Times’s servers and optimized for speed. Damens said they were able to get speeds a hundred times faster than CGI in early tests.
But they didn’t stop at creating a language.
Anticipating that editors would one day want to add more dynamic sections to the site, Damens and Karlin designed Context to support newsroom workflows.
Editors were familiar with adding tags to their page layouts to change elements, such as the weather, for different regional editions. Rather than asking editors to learn a programming language, the developers tied the language into the macros the editors used to send formatting instructions to the printing press.
What started as a request for a username turned into a programming language, a compiler, a pre-processor that scanned every page on the site and converted it into a context program, and a virtual machine to run those context programs on The Times’s servers.
“We probably over-engineered it, I guess,” Damens said.
But Context wasn’t just an elaborate username-printing system for long. Soon, the system would be used to add a weather widget to the homepage. Then it was used to streamline the process for publishing movie showtimes online. Within a few months, a Context program was put into service to paginate articles, which was a major driver of page views. Context quickly became the technology powering the bulk of nytimes.com.
While it had a reputation for being fast and reliable, Context was also controversial. Engineers bristled at having to learn a programming language only used at The Times. Only a few learned enough to help add new features or extend its abilities. Recruiting new engineers was sometimes a challenge because candidates had to accept that they were joining an organization with a homegrown programming language and build system. Looking back, Damens and Karlin regret not picking an existing syntax of a language, like PHP, to make it easier to adopt.
Context probably would have been phased out quickly were it not for another personalization challenge that required its speed. This time the problem was ads. Later in 2001, Zip-code targeted ads were slowing down the site and by this time Context had credibility as a fast and reliable solution. Damens and Karlin were called in to help.
They used Context to create a fast ad server called AdX, which used regular expressions to match ad slots according to targeting parameters for each ad campaign. The regular expressions ran against metadata in the page (such as a reader’s zip code and what section of the paper they were visiting) to target ads to specific users. AdX also kept track of how frequently ads were shown so it could manage overall ad inventory.
All of this was able to run on the existing servers of The Times.
By the end of 2001, Context was powering nearly all pages and advertising for The Times. It went on to serve more than a decade in this capacity, seeing the transition from a supplement to the printed newspaper, to the way most readers consume The Times. It survived traffic surges during elections and major news events. AdX expanded to serve increasingly sophisticated targeting, including marketing messages on the site.
With all that Context did at The Times, it never ventured outside the walls of the building; it was never open sourced or sold. The years that Context was the most creative and powerful as a system were long before The Times had a culture that embraced open source.
But perhaps that isolation helped keep it alive until earlier last year. With only a handful of contributors, the language was spared being pulled in different design directions. Fewer than a dozen engineers knew how the build system worked. Maybe a hundred engineers learned to ever write Context code. Without a large community of engineers, Context was spared from the drag that new features can add to a system, which meant it stayed small, nimble and able to scale to meet the growing traffic of The Times for a decade after its creation.
As the last files of nytimes.com were rewritten using newer technologies, Context disappeared entirely from The Times. While the language could have grown into something larger had it been open sourced, it instead served to help The Times find its footing on the internet. Damens and Karlin were right: The Times news offerings have expanded from a static website to include apps and more personalized features.
But one thing has stayed the same: readers can still find a link to their username on the homepage.
