AWS re:Invent 2018 Recap: Kubernetes, Serverless, ML, Security
AWS re:Invent continues to explode—there were 50,000+ attendees this year and many really good sessions. I was able to get to quite a few and, between those and walking the floor and visiting booths, I saw several topics rise to the top across the conference.
Kubernetes has become a de-facto industry standard as the open-source system for automating deployment, scaling, and management of containerized applications. There were multiple talks on how to run high-performance clusters and how to operate at scale using Amazon EKS, as well as several exhibitors talking Kubernetes. At the same time, there are no major Kubernetes announcements, which I found interesting.
There were some significant improvements announced around AWS Lambda and API Gateway. In the next year or two, I expect to see more fully managed (and secure) layers for Lambda, with pre-built basic operations like database access, as well as a fully-operational Lambda on-prem edition. Given that AWS is pushing towards smaller function footprint and faster startup times, they might be planning to develop a new standard for building microservices without the need to containerize applications and orchestrate such containers.
Clouds, Multi-cloud and What Not
Competition between cloud providers is apparently heating up and now AWS cannot just claim that they are simply the best, they have to explain how they are different even in the basic cloud computing components (compute, storage and network). A significant amount of time was spent on explaining the architecture of S3, virtual machines and improvements that were made over time. There were lots of new announcements around very specialized use cases, new types of databases, the blockchain, new instance types, new hardware, etc. “Cloud nativeness” becomes the ability of underlying architecture and hardware to match the needs of the specific applications and services on top of it and bring significant improvements in performance. And as we see clouds diverge in the implementation of even basic things the only options left for real multi-cloud approaches would be either to use each cloud in its own way or restrict usage to very basic components. Another interesting move is AWS Outpost—if this works as advertised it could be a real game-changer for on-prem and hybrid solutions.
Storage and Data
One of the oldest services that Amazon offers, S3 still gets lots of attention. It keeps getting new features and is transforming to be the main driver behind all data-related use cases: data lakes, big data, analytics, insights, forecasting, machine learning, ETL processes to prepare data for machine learning, long-lived archives and so on. Virtually every single presentation relating to data included S3 in one way or another, as well as the evolution of ways to ingest the data and govern it over time at reduced cost. Data management is another impediment into going multi-cloud as compute needs to be located close to storage. This is definitely the area where AWS holds its ground very strong.
No surprise, there was a huge push in this area at AWS re:Invent, from specialized hardware to specialized services like Amazon SageMaker RL for managed reinforcement learning, a diverse variety of data science frameworks and a marketplace for ML algorithms for SageMaker. With all those advancements it’s still not easy to get into advanced production-ready machine learning models, but building proof of concept on top of existing S3 data continues to get easier .
Lastly, “security” was everywhere. Orchestration, automation, auto-remediation, DevSecOps, shifting left were all topics of conversation. Amazon’s AWS Security Hub announcement focuses on infrastructure security, compliance, monitoring, alerts, and access control with Security Hub as the…”hub”. Interestingly though there was virtually nothing around code and application security and applying the same approach there. I think the logical next step for AWS would be to add application security to serverless. Even though Security Hub makes sense mainly within AWS environment it’s a great attempt to tackle very important issues related to the security practices:
- The high volume of vulnerabilities data and alerts that are very complicated to deal with
- The absence of unified data formats among different security tools
- Lack of visibility and top-level view across multiple sources of information
Stay tuned for our next blog on how CYBRIC handles the same challenges for code, container, application and infrastructure security both on-prem and in the cloud.