Skip to content

Reflections on Eight Years at Atlassian: Building a Self‑Service Load‑Balancing Platform

Original URL: https://www.youtube.com/watch?v=55pTFVoclvE

Introduction

The speaker shares a reflective memoir of an eight‑year career at Atlassian, spanning from the hiring interview through the creation of a centralized self‑service load‑balancing system for internal developers, to the maintenance and enhancement of that platform. The narrative offers a candid look at both technical achievements and the soft‑skill challenges that arise in a large engineering organization.

Early Days: Interview & First Projects

Interview Process

  • Coding quiz: Hacker Rank, perfect score
  • Technical interview: White‑paper on Cloudflare custom domains; discussion of microservices and container architectures
  • Troubleshooting exercise: Simulated denial‑of‑service incident, probing latency‑based DNS
  • Values interview: Presented an opportunity to build a self‑service load‑balancing application; confident Python web‑app skillset secured the job

First Task: Open Service Broker

  • Goal: Build a web app + API that provisions resources in a Kubernetes‑centric environment for internal developers
  • Technologies:
  • Initial implementation with connexion (Python) auto‑generating routes from an OpenAPI spec
  • Migration to FastAPI for better performance and type support
  • Asynchronous worker decoupled from the API via AWS SQS
  • Persistence in DynamoDB; status polling for provisioning completion

Building the Core Platform

Proxy Architecture

  • Entry point: AWS NLB (Layer‑4) directing traffic to an Envoy proxy fleet
  • Proxy provisioning:
  • CloudFormation templates create VPC, subnets, security groups, and an Auto‑Scaling Group
  • AMI creation with Packer + SaltStack: installing Envoy, logging, tracing, and security hardening
  • Sidecar model: additional containerized services (auth, rate limiting, logging) run alongside Envoy

Control Plane: “Sovereign”

  • FastAPI‑based control plane that reads configuration templates (cluster, route, listener) and context (service metadata, external data sources)
  • Generates dynamic Envoy config on demand
  • Bridges the broker, provisioning tasks, and the proxy

Scaling to the Enterprise

Enabling Cloud Products

  • Migrated major Atlassian services (Jira, Confluence, Bitbucket, Status Page) onto the new edge infrastructure
  • Introduced a public access flag in the service metadata to render the service reachable via the central load balancer, preventing accidental exposure

Centralized Concerns

  • Security: Authentication via Envoy sidecar; authorization delegated to a separate team
  • Rate limiting: Implemented as a sidecar, configurable per‑service
  • Access logs & DoS protection: Leveraged CloudFront and Envoy’s network filters
  • Observability: Structured logging, tracing, and metrics baked into the AMI

Maintenance & Organizational Lessons

Technical Stability

  • Established onboarding documentation for new engineers; focused on log analysis and metric troubleshooting
  • Introduced health checks for the worker queue (SQS) and the Broker’s database (DynamoDB)
  • Implemented automated retries and fallback paths for critical provisioning steps

Professional Growth

  • Improved diplomacy, conflict avoidance, and mentorship skills through interaction with diverse teams and managers
  • Developed a deeper understanding of codebase churn: identified high‑change areas and refactored for easier future modifications
  • Learned the balance between providing guidance and encouraging autonomous problem solving in mentorship roles

Conclusion

This reflection encapsulates how an engineer transitioned from a single‑project coder to a platform builder, and later to a maintainer and mentor within a large tech organization. The key takeaways include the importance of:

  • Decoupling (API, worker, control plane) for scalable, fault‑tolerant systems
  • Dynamic configuration (templates + context) to avoid duplicated logic across services
  • Centralized security and observability to reduce workload on individual teams
  • Continuous learning of both technical and soft‑skill dimensions to thrive in a complex, evolving environment