Worth a Read: The Unreliable Bus

Four posts, explaining in near-laymen’s terms, the factors that affect service reliability. The author goes into more detail than I did on the subject.

In this first post, I am going to try and convince you that while on the surface this complaint seems simple enough, addressing it is a devilishly tricky problem. After we understand what the problem actually is, we can start looking at ways to fix it.

The National ITS Architecture and its Environs

This is part 3 of a series of thoughts on the state of the transit tech industry, public and private.

We left off with Google’s introduction of the GTFS, but that was just the beginning of the importance of transit to this space. The iPhone ushered in a world of opportunities for end users. Smartphones have driven ubiquitous information. But that ubiquitous information took a little while to get into this form factor. In 2008, with the addition of apps to iOS, Google brought transit directions to the iPhone.

In 2011, Google introduced GTFS-RT, allowing agencies to provide real time updates for an entire fleet. A number of transit apps sprung up merging GTFS, GTFS-RT, and other information together to provide useful information in one place.

These have gotten serious attention and funding from the tech community. For example, in 2014 Citymapper received $10M in funding, and in January 2016 got another $40M.

Moovit has raised a total of $81.5M. What’s most interesting is who is funding them. In addition to regular tech venture capitalists, BMW and Keolis (a branch of the French National Railways) have invested. Clearly, the existing transportation industry is hedging their bets with technology.

Not to be outdone, Google snapped up the largest player in the driving space, Waze, for just under $1 Billion.

What these investors see in apps are not just national but global ITS architectures. And they do this not out of the goodness of their hearts– they see money extracted from surveillance of our everyday activities. Some companies are willing to do some crazy stuff to get data at any cost.

 “The assault we face is driven in large measure by the exceptional appetites of a wholly new genus of capitalism, a systemic coherent new logic of accumulation that I call surveillance capitalism. Capitalism has been hijacked by a lucrative surveillance project that subverts the “normal” evolutionary mechanisms associated with its historical success and corrupts the unity of supply and demand that has for centuries, however imperfectly, tethered capitalism to the genuine needs of its populations and societies, thus enabling the fruitful expansion of market democracy.”

Meanwhile, back at the government ranch…

As a result of scale and mandate, the model of surveillance capitalism is not something that most public agencies can capitalize on, so the private sector is left to reap the gains. Perhaps because of this lack of incentive, public sector driven regional ITS architectures have not reached the same level of sophistication or prominence. (With one notable exception,  and a possible newcomer).

In conclusion, the private sector uses sizable public investments for profit (or at least leverage for sizable private investments), and gives (or is perceived to give) little back to those that have provided it.  Understandably, this causes consternation.
The real question is, knowing this, what are the incentives for the public sector to push the next generation of public investment forward?

Background Nerd Reading: On Predictions

This paper goes into detail about the most common arrival prediction algorithm– using the published schedule. This algorithm is not perfect, but it is a substantial improvement on the schedule alone:

This scheme was found to systematically underestimate the remaining waiting time by 6.2% on average. The provision of real-time information yields a waiting time estimate that is more than twice closer to the actual waiting times than the timetable is. This difference in waiting time expectations is equivalent to 30% of the average waiting time.

Oded Cats* & Gerasimos Loutos. Real-Time Bus Arrival Information System: An Empirical Evaluation. Journal of Intelligent Transportation Systems, 2015

On the emergence of a Global ITS Architecture

This is the second in a series of posts on what I’m calling the Global ITS Architecture. The first gave background on the National ITS Architecture.

Below the fold, edited notes from a talk I gave last year on “Have Open Standards Delivered ITS Architectures that Matter?” Hint: I argue Yes.

Note that I engage in some of the lies of storytelling, duly noted.

I’m going to take a strong stand on this question and make three arguments on my way to an answer:

  1. The evolution of ITS has seen the scale of information distribution grow over time.
  2. The private sector has really driven the state of the art for passenger information.
  3. In this space, the work put into National ITS standards have not had good ROIs.

First, a bit of history. Electronic passenger information is not something of the 21st century. The first subway in America, Boston, had electronic passenger information signs in the 1890s.

imageBut both the provision and reach of information was severely limited. Driving this was someone behind a switchboard.

Fast forward to the end of the 20th century. We have systems that make passenger information available at a city-wide scale. Here is one such sign.


While the data were available systemwide, in the beginning it was only available to the agency and through this manufacturer’s product1) Lie of omission: HSL in Helsinki is an awesome data provider. The only reason we know what the interface for these signs looks like is because of a Finnish hacker. She was playing around with a spectrum analyzer, found something interesting and decided to take a look.

Growing Pains

At the dawn of the 21st century, after deploying a host of single vendor solutions, the industry began to understand the downsides of end-to-end functionality from one vendor. from a panel of experts on passenger information and, after a standard standards-development process, TCIP was born, good, bad, and ugly.

  • Good: Standards Process
  • Bad:
    • No major implementations
    • Backward looking: Year 2000 technology (Not thought through for web / mobile clients)
  • Ugly: Documentation “cumbersome at best, impossible at worst”

Because of these issues, and no one has used the Passenger Information messages at large scale. No one took this standard as an example and said, “hey I want to build a system with that.”

Contrast this with the NTCIP standards, where the documentation is succinct and accessible.


The tech sector has taken a separate track.

In 2006 Google finalized the first version of GTFS. This was part of moving from a search company, to a data company.
Rather than trying to understand national ITS diagram,  they addressed this as a tech company would, in an agile fashion with a minimum viable product first. They worked with one agency, TriMet in Portland, to iterate a working product before putting it out.
Put simply, GTFS proved that without a flagship project, a standard is worthless.

Google saw world of opportunities for data, and they have driven the market ever since. But will they continue to in the future?

I’ll address that in in part 2.

References   [ + ]

1. Lie of omission: HSL in Helsinki is an awesome data provider

Primer: The National ITS Architecture

This is background material for a subsequent post.

The [US] National ITS Architecture provides a common framework for planning, defining, and integrating intelligent transportation systems. It is a mature product that reflects the contributions of a broad cross-section of the ITS community (transportation practitioners, systems engineers, system developers, technology specialists, consultants, etc.).

The architecture defines:

  • The functions (e.g., gather traffic information or request a route) that are required for ITS
  • The physical entities or subsystems where these functions reside (e.g., the field or the vehicle).
  • The information flows and data flows that connect these functions and physical subsystems together into an integrated system.


Basically, the National ITS architecture brings taxonomy to Intelligent Transportation Systems, all through the confusing lens of the typical American engineer (c.f. the usage of the ackronym ISP). It looks like this:

The National ITS Architecture
The National ITS Architecture

The ne-plus ultra of the National ITS Architecture is the connected vehicle. Below the fold, a shiny video.

Personally, I remain cynical of the next big thing.

For more on the National ITS Architecture (and a reminder that slides make bad documents), see here. If you need bedtime material, there is a ~300 page theory of operation from USDOT.

Querying GTFS with CSVkit

A while ago I wrote a post about using built-in tools from the Bash command line shell to perform quick and dirty analysis of GTFS files. Those examples depended on the order of the columns in the GTFS, something which is not standard across all feeds. There is another set of utilities, csvkit, which can work around this problem and enable the same analysis to be performed regardless of column order.

Continue reading “Querying GTFS with CSVkit”

The Quartz Bad Data Guide

Following my previous post, online journalism outfit Quartz recently posted a Guide on Bad Data, organized into the following parts:

  • Issues that your source should solve
  • Issues that you should solve
  • Issues a third-party expert should help you solve
  • Issues a programmer should help you solve

It is well worth a read. Also, if you have any comments or additions, it is open for your updates!