Talk
Observability in software applications
A talk about a 2007 plane crash, the orange box that survived it and the standard our industry has not come close to.
- Event
- GigCity Elixir 2023
- Date
- 19 to 20 May 2023
- Location
- The Edney, Chattanooga, Tennessee
- Length
- 20 minutes
- Observability
- Instrumentation
- Elixir
- Incident review
Takeaways
I opened by asking who wanted to learn about observability in Elixir applications, watched the hands go up and told them they were about to be disappointed. Then I talked about Kenya Airways for fifteen minutes. The tools slide comes at the end. I skip through it on purpose.
Aviation sets the benchmark. We are nowhere near it
In 2022 there were around 22 accidents across roughly 22 million flights. Not only that: for essentially all of them, we know what happened. Ask whether your systems could claim either half of that.
The aircraft was destroyed and the data survived anyway
Nothing usable was recovered from the airframe. We still have the altitude, the pitch, the speed, the moment a knob was touched and the words spoken in the cockpit, second by second. That is not luck. Somebody designed for it.
Observability is two jobs, not one
It told the pilots what weather they were flying into, in the moment. It also told investigators what had happened, years later. Instrumentation that only does the first is half a system.
Record the silence, not just the events
The most damning line in the report is a first officer who does not respond. In aviation the captain calls, the first officer confirms. The absence of the confirmation is the signal. Most logging cannot express that.
Design the recorder to survive the failure
The box sits in the tail because that is where impact is lowest. It is wrapped against fire, it takes about 3.5 times the impact weight, its beacon transmits from 14,000 feet under water. If the case is gone you pull the chip and read it. Your telemetry should be as hard to lose as your service is to keep up.
The name is wrong and it matters
The black box is orange. It is painted to be found.
Watch the talk
Getting there
I started with the journey, because it turned out to be the talk. Kisumu to Nairobi, thirty minutes. Nairobi to Dubai, five hours, then a five-hour layover. Dubai to Chicago, about fourteen hours, then another five-hour layover, then an hour to Chattanooga. I landed around midnight and Bruce picked me up from the airport.
I also showed people where Kisumu is, since most of the room had been politely pretending to know. Red dot on one side of the map, another dot over here, plus a note that maps will lie to you about the distance between them.
Kenya Airways 507
Kenya Airways is state-owned and loss-making and gets bailed out the way governments bail out businesses they have an interest in. That is background. The story is the night of 4 to 5 May 2007.
A Boeing 737-800 flying Abidjan to Douala, then Douala to Nairobi, with a stop to refuel. The captain was 52. The first officer was 23. The aircraft was about six months old to the airline. There had been minor autopilot trouble, since fixed. The leg into Douala was fine.
They checked the weather radar, saw the storm, planned a route around it, got clearance and took off. At around a thousand feet the captain engaged the autopilot. Then the aircraft began banking right.
Reading the report on stage
Rather than describe it, I opened the accident report in a browser and read it with the room, because the granularity is the argument.
00:06, landing gear retracted
Twenty-four seconds later they are passing a thousand feet and climbing. The captain says "select checked". The report gives the altitude, the pitch and the speed at that instant.
One second later, a knob is touched
The captain says he will keep it somewhere around here, which most likely means they are threading the thunderstorm.
"Okay command" and nothing back
That call means the autopilot is engaged and the first officer is supposed to confirm it. He never does. The report records the silence.
The bank angle warning sounds
Roughly a minute and a half after the wheels left the ground. All 114 people on board died.
It was a very sober moment in Kenya. May they rest in peace.
How we know any of this
The airframe is unrecoverable. Nothing useful comes out of it. And yet years later there is a report describing the flight second by second. So where did the data come from?
Not the cloud, which was my joke. I did make the room sit through a bad pun about calling head and tail on an aeroplane first. Calling tail gets you the tail, which is where the recorders live.
There are two of them, together making up what the press calls the black box, which is orange.
The cockpit voice recorder
Everything said in the cockpit, including over the microphone. This is how we know the crew spoke to the tower and what they said to each other in flight. It typically holds about two hours. On some aircraft the crew may erase it once on the ground.
The flight data recorder
Position and instruments. This is what gets used to replay the accident and work out the cause.
Both live in the tail, because that is the part of the aircraft that takes the least of the impact. Almost all of it lands on the fuselage. The units are surrounded by material rated for the temperatures of a fuel fire, so fire rarely destroys them and when it does there are ways to recover the data anyway. You open the case, take the chip out and read it with a chip reader.
If the aircraft goes into water, a beacon transmits for about thirty days, from as deep as 14,000 feet. That is why hope of finding a lost aircraft starts to fade at the thirty-day mark, as it did with the Malaysian flight. Very few recorders are never found, however severe the accident.
The question I left them with
Twenty-two million applications running, twenty-two failing and in each of those cases knowing exactly what happened. Can we get to that?
Because after this crash, pilots were retrained on specific procedures. Airlines get measurably safer every year and they get safer because the data from the last failure changes the next flight. That loop is what observability buys you. Not dashboards.
Then, yes, the tools slide. A grid of the usual names. I skipped through it deliberately and said so. Some of them do a lot, some you leave to ops, some you will need. That was never the point of the talk.
The point is that observability is what let the pilots see the weather they were flying into and what let us find out what happened after it went wrong. Build your applications so you can know what is going on while they run and know what happened once they are dead and maybe we get to 22 in 22 million.
Ending on visas
I closed with a photo from a session we had with José Valim, mostly Kenyans in the frame and mentioned that ElixirConf Africa 2023 was the following week, virtual again.
Then a screenshot from a webinar the previous day, where a lot of cameras were off. That is not a design choice, it is a situation. Some of those people did not have the data to spare.
I was in Chattanooga because I wanted to be and because I was able to be. The room was there for the same two reasons. Some of our members chose not to come and some could not, because they are refused passports and visas over and over again. They cannot get out. You can get in. That was the pitch for making ElixirConf Africa 2024 a physical event in Kenya, with a safari attached and it is why I ended a talk about telemetry with a photo of a giraffe centre.
Think your organisation has outgrown its systems?
Let's figure out what is actually broken.