Internal traffic: from the "central airport" to sidecar, mTLS, and Istio

01 · CONTEXT

A chaotic airport inside Fury

Fury's internal traffic, in the Mercado Libre engineering team's account, resembled a chaotic airport, with millions of passengers traveling to thousands of destinations. In that airport, everyone traveled without passports, without immigration controls, and without customs inspection.

At access peaks, internal traffic alone exceeds 400M per minute, with more than 30k applications split into more than 75k instance groups. So any solution or change must take this scale into account.

1. The diagram shows access to the platform entering through a central traffic layer, and an outbound arrow from application A returning to that same layer.

2. Access enters through the central traffic layer, which routes the request to the destination application.

3. When an application needs to talk to another, it sends a new outbound request through the same layer, which forwards it to the destination.

4. The source describes this central airport as the convergence point of all flights.

5. From the destination's point of view, all flights appear to come from the same origin.

02 · IDENTITY

No passport and no immigration

Many applications operated without proper authentication, so it was not possible to verify the authenticity of the origin. This allowed anyone inside the platform to use any API with no form of control.

Without identity there was also no way to create access rules for a specific client, because it was not possible to identify them uniquely.

03 · ROUTES

A central notebook and all roads leading to Rome

Every route and every existing destination was written in a single place, the central notebook. Adding, removing, or changing a destination required changing that same notebook, and the source mentions more than 100K defined routes.

Communication was also centralized: even between nearby airports, passengers had a layover at the central airport before reaching the destination. For the receiver, all flights seemed to come from the same origin.

04 · SIDECAR

Local airports: sidecar, certificates, and mTLS

The first step was to relieve the central airport by distributing the new security controls to local airports. Each local airport is a traffic sidecar that runs next to the application, acting as a reverse proxy on ingress and as a transparent proxy on egress.

The sidecar gave a presence in every application in the Fury ecosystem and was the kickoff for better control, observability, and governance of traffic. In the QCon SF talk, the team also mentions Traffic Middleware among the platform's sidecars.

1. The diagram shows a sidecar next to each application, with the certificate authority and Traffic Authorization connected to the sidecars.

2. The sidecar runs next to the application, as a reverse proxy for inbound traffic and a transparent proxy for outbound traffic.

3. The certificate authority generates the certificates when services or scopes are created and renews them on every deploy.

4. All traffic is stamped at the origin and validated at the destination.

5. In the third version of identity, mTLS is handled by the sidecar and protects communication in both directions.

6. Traffic Authorization stores what each application may access, and the destination's sidecar enforces these rules with Envoy RBAC filters.

To identify the passengers, the team deployed a certificate authority that generates the certificates, the passports, when services or scopes are created and renews them on every deploy. Traffic is stamped at the origin and validated at the destination. Identity went through three versions:

05 · AUTHORIZATION

Immigration and routes per team

Once identity is confirmed, authorization comes in. Traffic Authorization lets teams specify the endpoints, methods, and domains accessible in an application, as well as those needed to interact with others. With this, internal traffic was cataloged and it is possible to trace the origin and destination of each flight.

The rules started as custom validations at the destination, with Nginx, Lua, and Golang customizations. In the sidecar migration from Nginx to Envoy, the team took the opportunity to standardize the process and began using Envoy RBAC filters.

On the routes side, Traffic Routes gives each development team its own domain, where it maps paths to destinations. Teams can define conditions by request headers, query parameters, or HTTP methods, and test the routes before going to production.

06 · SERVICE MESH

The control tower: Istio

The source states that the service mesh reduces congestion, for example by removing the balanceador and the centralized traffic layer from the equation, and that without it they would not have been able to decentralize all the traffic. In the role of control tower, the mesh directs flights along safe or less congested routes.

After analyzing several options, including in-house development, the team chose the Istio control plane, among other factors because it is a mature, open-source solution with the features they needed. Integration into the Fury ecosystem included the creation and deploy processes, security controls, and the exchange of routing information with Traffic Routes.

1. The diagram shows a sidecar next to each application, with the mesh control plane connected to the sidecars and Traffic Routes connected to the control plane.

2. Access to the platform still starts at the centralized traffic layer.

3. The request is intercepted by the sidecar, which passes it to the application.

4. In communication between applications, the outbound request is intercepted by the sidecar and routed directly to the next destination.

5. The mesh plays the role of control tower, directing flights along safe or less congested routes.

6. Istio's integration into Fury includes the exchange of routing information with Traffic Routes.