Cole BrennanStaff Site Reliability EngineerDenver, CO
Demo
View PDF

Professional Summary

Treats observability as a product with owners, SLOs, and a budget. Kubernetes is the substrate; the work is making a page name the failing dependency before the war room does.

At Highwater, cut MTTR from 47 minutes to 14 and dropped paging noise 58% across 9 clusters. Previously ran Summit's tracing mesh and Alpine's first on-call that did not page on every 5xx.

Skills

Group by:
13 years:
Python
12 years:
AWS
11 years:
Observability
10 years:
Distributed Systems
Go
9 years:
Kubernetes
8 years:
System Design
6 years:
Technical Leadership

Work Experience

Highwater Observability
Denver, CO, November 2020 to Present
Denver shop that sells managed telemetry to teams drowning in their own cardinality. Cole is the staff SRE who still carries a pager.

Staff Site Reliability Engineer Highwater Observability

Replaced threshold alerts with error-budget burns on 31 services so a page named a user-facing burn, not a noisy saturation. Pages per week fell 58%; MTTR on customer-facing incidents went from 47 minutes to 14. Saturation without a burn still goes to a ticket, which is how the pager got quiet.
Unified traces across 9 EKS clusters with consistent baggage so a request did not become three tools and a guess. Time-to-name-the-dependency in incidents dropped from 18 minutes to 3. The war room still happens; it starts with a trace ID instead of a theory.
Go admission webhook that rejects unbounded label sets before they become a finance conversation. Time-series growth flattened 71% and the monthly telemetry bill stopped surprising finance. A new high-cardinality label is now a design review, which is the point of a budget.

Summit Mesh Ops
Boulder, CO, April 2016 to October 2020
Boulder consultancy that ran other people's clusters. Cole owned the tracing story and the on-call that had become a group chat.

Senior Site Reliability Engineer — February 2018 to October 2020Summit Mesh Ops

Defined the span contract six clients adopted so an outage stopped opening four Slack channels with four clocks. Duplicate incident channels fell from 4 per outage to 1; postmortems started with a trace ID. The standard is a page, not a working group that never shipped.
Split noisy batch onto dedicated pools so a report job stopped being a latency incident for product. Noisy-neighbor latency events fell 64% and the batch team stopped paging product on-call. Isolation is cheaper than another apology in the incident channel.

Site Reliability Engineer — April 2016 to February 2018Summit Mesh Ops

Python reports that mailed weekly budget burn to service owners, with a name on the page instead of a shared inbox. Unowned 5xx alerts dropped 40% once names were on the dashboard. Leadership still reads the mail; that is why the names stayed accurate.
Moved Prometheus scrape onto private links so a metrics endpoint stopped being a public API with worse auth. Accidental public metric endpoints went from 11 to 0 in a quarter. The scrape path is boring on purpose; the findings it closed were not.

Featured Projects

High-cardinality time series budget admission webhook

Reference admission controller that keeps high-cardinality labels out of the time-series store. Written after a single deploy doubled a client's Prometheus bill. A new unbounded label is a design review, not a surprise invoice.

Go
Kubernetes
System Design

Reusable SLO and paging contract template

A reusable SLO and paging contract: burn-rate alerts, ownership fields, and the rule that saturation without a user-facing burn does not wake anyone. Includes the weekly budget-burn mail that kept names on the dashboard accurate.

Observability
Technical Leadership
Python

Education

University of Colorado Boulder

B.S. Computer Science — May 2013

Certifications

Certified Kubernetes Administrator

Cloud Native Computing Foundation — August 2019

Credential ID: CKA-COLE-0816

AWS Certified SysOps Administrator – Associate

Amazon Web Services — January 2021

Credential ID: SOA-COLE-0129