Service Mesh Security
Adopting the cloud can put strains on DevOps teams. Developers leverage microservices to architect for portability, meanwhile operators are managing extremely large hybrid and multi-cloud deployments.
A Service Mesh helps reduce the complexity of these deployments, and eases the strain on development teams. It is usually an open source component that layers transparently onto existing distributed applications. It is also a platform, including APIs that let it integrate into any logging platform, telemetry, or policy system.
Well-known Service Mesh solutions of the Cloud Native Computing Foundation (CNCF) are:
- Istio (developed by Google)
We'll focus on the security of Istio, because this service mesh is closely integrated into the dominating Kubernetes orchestration component.
Istio Security OverviewAn Istio service mesh is logically split into a data plane and a control plane:
- the data plane is composed of a set of intelligent proxies (Envoy) deployed as sidecars. These proxies mediate and control all network communication between microservices along with Mixer, a general-purpose policy and telemetry hub
- the control plane manages and configures the proxies to route traffic. Additionally, the control plane configures Mixers to enforce policies and collect telemetry.
EnvoyIstio uses an extended version of the Envoy proxy. Envoy is a high-performance proxy developed in C++ to mediate all inbound and outbound traffic for all services in the service mesh. Istio leverages Envoy's many built-in features:
- Dynamic service discovery
- Load balancing
- TLS termination
- HTTP/2 and gRPC proxies
- Circuit breakers
- Health checks
- Staged rollouts with %-based traffic split
- Fault injection
- Rich metrics.
MixerMixer is a platform-independent component. Mixer enforces access control and usage policies across the service mesh, and collects telemetry data from the Envoy proxy and other services. The proxy extracts request level attributes, and sends them to Mixer for evaluation. Mixer includes a flexible plugin model. This model enables Istio to interface with a variety of host environments and infrastructure backends. Thus, Istio abstracts the Envoy proxy and Istio-managed services from these details.
PilotPilot provides service discovery for the Envoy sidecars, traffic management capabilities for intelligent routing (e.g., A/B tests, canary rollouts, etc.), and resiliency (timeouts, retries, circuit breakers, etc.). Pilot converts high level routing rules that control traffic behavior into Envoy-specific configurations, and propagates them to the sidecars at runtime. Pilot abstracts platform-specific service discovery mechanisms and synthesizes them into a standard format that any sidecar conforming with the Envoy data plane APIs can consume. This loose coupling allows Istio to run on multiple environments such as Kubernetes, Consul, or Nomad, while maintaining the same operator interface for traffic management.
CitadelCitadel enables strong service-to-service and end-user authentication with built-in identity and credential management. Citadel can be used to upgrade unencrypted traffic in the service mesh. Using Citadel, operators can enforce policies based on service identity rather than on relatively unstable layer 3 or layer 4 network identifiers. Istio's authorization feature can be used to control who can access services.
GalleyGalley is Istio's configuration validation, ingestion, processing and distribution component. It is responsible for insulating the rest of the Istio components from the details of obtaining user configuration from the underlying platform (usually Kubernetes).
Security ArchitectureBreaking down a monolithic application into atomic services offers various benefits, including better agility, better scalability and better ability to reuse services. However, microservices also have particular security needs:
- to defend against man-in-the-middle attacks, they need traffic encryption
- to provide flexible service access control, they need mutual TLS and fine-grained access policies
- to audit who did what at what time, they need audit tooling.
- Security by default: no changes needed for application code and infrastructure
- Defense in depth: integrate with existing security systems to provide multiple layers of defense
- Zero-trust network: build security solutions on untrusted networks.
- Citadel for key and certificate management
- Sidecar and perimeter proxies to implement secure communication between clients and servers
- Pilot to distribute authentication policies and secure naming information to the proxies
- Mixer to manage authorization and auditing.
- Kubernetes: Kubernetes service account
- GKE/GCE: may use GCP service account
- GCP: GCP service account
- AWS: AWS IAM user/role account.
- On-premises (non-Kubernetes): user account, custom service account, service name, Istio service account, or GCP service account. The custom service account refers to the existing service account just like the identities that the customer's Identity Directory manages.
AuthenticationIstio provides two types of authentication:
- Transport authentication (aka service-2-service authentication): verifies the direct client making the connection. Istio offers mutual TLS as a full stack solution for transport authentication. This solution
- provides each service with a strong identity representing its role to enable interoperability across clusters and clouds
- secures service-2-service communication and enduser-2-service communication
- provides a key management system to automate key and certificate generation, distribution, and rotation.
- Origin authentication (aka end-user authentication): verifies the original client making the request as an end-user or device. Istio enables request-level authentication with JSON Web Token (JWT) validation and a streamlined developer experience for open source OpenID Connect provider ORY Hydra, Keycloak, Auth0, Firebase Auth, Google Auth, and custom auth.
How to authenticate on Kubernetes
- a. Citadel watches the Kubernetes apiserver, creates a SPIFFE certificate and key pair for each of the existing and new service accounts. Citadel stores the certificate and key pairs as Kubernetes secrets.
- b. When a pod is created, Kubernetes mounts the certificate and key pair to the pod according to its service account via Kubernetes secret volume.
- c. Citadel watches the lifetime of each certificate, and automatically rotates the certificates by rewriting the Kubernetes secrets.
- d Pilot generates the secure naming information, which defines what service account or accounts can run a certain service. Pilot then passes the secure naming information to the sidecar Envoy.
How to authenticate on-premises
- a Citadel creates a gRPC service to take Certificate Signing Requests (CSRs).
- b. Node agent generates a private key and CSR, and sends the CSR with its credentials to Citadel for signing.
- c. Citadel validates the credentials carried with the CSR, and signs the CSR to generate the certificate.
- d. The node agent sends both the certificate received from Citadel and the private key to Envoy.
- e This CSR process repeats periodically for certificate and key rotation.
AuthorizationAlthough it's strongly recommended to using the Istio authorization mechanisms, Istio is capable of working with 3rd party authentication & authorization plugins via the Mixer component. Istio's RBAC-based authorization feature provides namespace-level, service-level, and method-level access control for services in an Istio service mesh. Its featured are:
- Role-based semantics, which are simple.
- Service-2-Service and Enduser-2-Service authorization.
- Flexibility through custom properties support, for example, conditions in roles and role-bindings.
- High performance, as Istio authorization is enforced natively on Envoy.
- High compatibility, supports HTTP/S and HTTP2, as well as any plain TCP protocol.
Policies and TelemetryInfrastructure backends are usually designed to provide support functionality used to build services. They include items such as access control systems, telemetry capturing systems, quota enforcement systems, billing systems, etc. Services traditionally directly integrate with these backend systems, creating a hard coupling and baking-in specific semantics and usage options. Istio provides a uniform abstraction that makes it possible for Istio to interface with an open-ended set of infrastructure backends. This is done in such a way to provide rich and deep controls to the operator, while imposing no burden on service developers. Istio is designed to change the boundaries between layers in order to reduce systemic complexity, eliminate policy logic from service code and give control to operators. Mixer is the Istio component responsible for providing policy controls and telemetry collection: The Envoy sidecar logically calls Mixer before each request to perform precondition checks, and after each request to report telemetry. The sidecar has local caching such that a large percentage of precondition checks can be performed from cache. Additionally, the sidecar buffers outgoing telemetry such that it only calls Mixer infrequently. At a high level, Mixer provides:
- Backend Abstraction: Mixer insulates the rest of Istio from the implementation details of individual infrastructure backends.
- Intermediation: Mixer allows operators to have fine-grained control over all interactions between the mesh and infrastructure backends.