The nOps data platform uses three strategies to ingest data from our customer environments: pulling data from AWS and Azure data sources, forwarding events in NRT from AWS, and forwarding high resolution metrics from both managed and standalone Kubernetes clusters. Here’s how we deploy into customer’s in an easy automated fashion:
IAM roles which can be applied through our Cloudformation automation or using our account-registration IaaC project allow our backend processes to pull multiple data sources, including reading progressive log files from S3 buckets (Cost and Usage, VPC Flow logs), resource metadata, and cloud trail events.
The nops-aws-forwarder is a Lambda function that forwards events for real-time automation. This allows us to drive mission critical optimization workflows like Reserved Instance automation and keeping our Graph Data API refreshed. The Lambda forwarder can also be launched in minutes using our Cloudformation template.
The nOps Kubernetes agent is a lightweight client that allows us to ingest high-resolution metrics from Kubernetes clusters for optimization and detailed shared resource cost allocation. It can be deployed into any type of Kubernetes environment, including EKS. The agent is deployed via IaaC using helm charts and creates a pod in the customers cluster to forward metrics to our data platform.