Skip to content

tags: - infrastructure-engineering - envoy-proxy - cloud-platform - open-service-broker - load-balancing Original URL: https://www.youtube.com/watch?v=55pTFVoclvE

My 8 Years at Atlassian: Building Infrastructure from the Ground Up

A reflective look at an engineer's journey through Atlassian's internal platform team, covering the creation of a self-service load balancing platform, an Envoy control plane, and the lessons learned along the way.

Introduction

After being laid off from Atlassian following eight years of employment, the speaker took time to reflect on the technical systems they built. This article summarizes the key projects, from the open service broker that enabled self-service provisioning, to the Envoy-based edge infrastructure that replaced enterprise load balancers, to the operational challenges and non-technical growth that defined the experience.

The Interview and First Tasks

  • The interview process included a HackerRank coding quiz (aced with full marks), a white paper reading exercise on Cloudflare's custom domains, a troubleshooting exercise simulating a real Atlassian incident, and a values interview.
  • When asked to envision what success would look like in 12 months, the interviewer described a need for a self-service load balancing platform for internal developers.
  • The speaker promised to build it using Python, leveraging confidence in web development skills.
  • The first task was to build an Open Service Broker using the Open Service Broker API spec, initially with the connexion library (auto-generating routes from an OpenAPI document), later migrating to Flask and then FastAPI.

The Open Service Broker Architecture

  • A FastAPI web app received provisioning requests from clients and pushed tasks asynchronously to an SQS queue.
  • A worker process handled provisioning tasks—creating DNS records, CloudFront distributions, and other API calls—then wrote results to DynamoDB.
  • The client polled for completion status; the web server checked the database and responded with success or failure.
  • This broker enabled self-service provisioning of load balancing for internal Atlassian services.

The Envoy Control Plane (Sovereign)

  • An architect proposed replacing expensive enterprise load balancers with the open-source, cloud-native Envoy proxy.
  • The speaker built Sovereign, an open-source Envoy management server (also available on Bitbucket).
  • Sovereign used FastAPI, accepting templates and dynamic context to render Envoy configuration (clusters, routes, listeners) on demand.
  • Context was pulled from the broker's database and an S3 bucket, fed into templates, and pushed to proxies.
  • Proxies were provisioned via AWS CloudFormation templates across ~13 regions, creating ~2,000 proxies with autoscaling groups referencing a standard AMI.

AMI Provisioning with Packer and SaltStack

  • The AMI was built using Packer with an EC2 provisioner, uploading SaltStack configuration for declarative configuration management.
  • The image included states for installing and configuring Envoy, logging agents, security hardening, network tuning, containers, and observability agents (logging, tracing, metrics).
  • CloudFormation templates referenced the AMI, passed runtime parameters (secrets, keys), and provisioned EC2 instances running the full proxy stack.

Migrating Products and Adding Platform Features

  • After the foundation was laid, the team migrated major products (Jira, Confluence, Bitbucket, Statuspage) onto the centralized edge infrastructure.
  • The platform enforced migration by disabling basic load balancing exposure—services had to explicitly opt into the centralized infrastructure.
  • Once the migration was complete, the team centralized cross-cutting concerns at the proxy layer:
  • DoS protection via CloudFront
  • Access logging through Envoy network filters (HTTP Connection Manager)
  • Authentication, authorization, and rate limiting via sidecar containers running alongside Envoy
  • Sidecars were configured dynamically, with authentication written in Rust by the speaker, and authorization and rate limiting contributed by other teams.
  • All configuration was dynamic, templated, and abstracted away from developers who provided simple parameters.

Non-Technical Lessons and Growth

  • Diplomacy and conflict resolution: Working with diverse personalities and management styles required self-awareness and psychological understanding to manage interpersonal conflict effectively.
  • Mentoring vs. training: The speaker excelled at breaking down complex topics for colleagues ("bread and butter"), but found one-on-one mentoring an intern challenging—balancing guidance without giving answers while preventing frustration.
  • Maintenance awareness: Recognized the importance of onboarding, documentation, and monitoring for long-term sustainability. Noted that code churn in specific areas signals growing complexity that demands architectural attention.
  • Code maintainability: Building something is easy; maintaining and evolving it over time is difficult. Coupling increases with changes, and detangling becomes a major effort—something the speaker anticipates AI-assisted tools may help with in the future.

Conclusion

Over eight years, the speaker helped transform Atlassian's edge infrastructure from basic load balancing into a fully programmable, multi-tenant platform built on Envoy proxies managed by a custom control plane. Key takeaways include the value of templated, dynamic configuration for abstraction, the importance of centralized cross-cutting concerns, and the critical but often overlooked soft skills—diplomacy, mentoring, and maintenance awareness—that sustain long-term engineering success.