Tuesday, October 23, 2018

its always good to have one Ambassador in your Kubernetes cluster @getambassadorio

I have been working with Kubernetes in production for 2 years now. Over this time we have migrated 3 or 4 times clusters, changed techinques and tools of deployments, changed habits or pushed back to some of the hype and trends coming up every day on the kubernetes scene. I guess nothing new here, this is what almost everybody does.

My personal take on this ever changing scene is to keep things simple - as much as possible, especially while your underlying infrastructure is changing from version to version. Kubernetes 1.4 (the first version I used in production) has the same basic constructs as version 1.7 and 1.10 currently in production, but many other other things come and go or change. When you are adapting or trying to adopt kubernetes on your org you need to rely on these simple and basic constructs that will make your transition from one version to the other painless, through with time you master more advanced and `cryptic` features. Anyway the point is, keep it simple.

On the early adoption days, we followed what felt natural based on our previous experiences. We were (still are) creating services spreading them around accomodating for the different business needs and features. Kubernetes is our golden cage and creating new services / apps never felt so easy. Going back to our previous experiences, while creating a new app/ service you feel the urge to `expose` it somehow, to your private network, to some namespace, to some context. Using the service type `LoadBalancer` while in AWS spinning ELB's felt like total magic! At first you have a couple of services, talking to each other, exposing some public API and it was good, you were provisioning your ELB's, you could related to the old concept a DNS entry for a specific service. Old school I know but it felt good.

Time goes by the 2 services are 15 now, they talk to each other, they integrate. Creating some many ELB's or Services becomes a messy and error prone business! Even within the kubernetes namespace context - dealing with some many different service entries is challenging - especially when you follow point to point integration. It was obvious that you need some kind of proxy. A step further

  1. You might need a public Inbound Proxy (Ingress)
  2. You might need internal proxies for your `service` mesh!  (or mess  haha)
Personally I always thought that the proposed Ingress API / Solution from kubernetes despite the `good` intentions - was too much, overcomplicated for my over simplifying mind in its ever ending battle to keep things simple. Also, I never felt good with the idea to have 1 everything under the same ingress hose. So despite having examples of things like Traefik or Nginx as the reference Ingress controllers, personally I thought that I was increasing my platform's configuration entropy and complexity. Maybe a bit spoiled by all the nice things abstracted by Kubernetes after all this time, adding some serious config for doing something like simple never made it to my book. So yes to Ingresses and Proxies but most probably multiple and independent Ingresses within the same cluster and not one.
On the other end internal proxies within the kubernetes network - was always something OK to do, but kubernetes with its DNS service was always easing the pain- at the end of the day everything is just a label in a `service.yml` definition.  Internal service names change or move easily.
One night I end up reading this article by R.Li (CEO of Datawire) and I was like `yes!! I totally agree with you`. This is how I decided to give Ambassador a go. What I really liked was the fact that Ambassador which is a kubernetes wrapper around Envoy proxy (a bit oversimplified definition apologies), takes away all the config that you would have to do using the Kubernetes Ingress resources, glues almost instantly on your existing deployment - being aware of your existing services and then leaving you only the task to configure its proxying rules and proxying behavior.

So in 1 place, you get to spin a horizontally scaled L7 proxy, no news here, the same way you scale your pods you would scale Ambassador, and either you can feed it  your proxying config or  you can enhance your existing service definitions with metadata that Ambassaor will detect (since it talks and understands the Kubernetes API).  Currently I prefer (the first) - adding directly to the Ambassador deployment all the configs instead of adding labels and metadata to our existing services. In away our  Services and the pods behind our services do not have a hint that are being proxied and this gives us great freedom to play around and change configs in 1 place without risking errors or mistakes to existing pieces of the puzzle!

Configuring Ambassador was really easy

Service definition


  • I like that I don't have to deal with the Ingress resources of Kubernetes especially when it comes to internal use scenarios and shadowing internal services.
  • Installing Ambassador is dead simple - especially the Helm sub charts can be easily adapted or templated for your own case.
  • All you have to do is decide if
    • You will make the proxying mapping in 1 single file (like I did on the example)
    • Or you will enhance your existing service definitions (kubernete services) with the same config that we added above, so that Ambassador can `discover` them through the Kubernetes API
  • Once you are up and running and you start doing some basic proxying things get more interesting
    • Canary deployments 
    • Weighted deployments controlling - the traffic weight for each service
    • Re-play production or pre-production traffic to new services (shadowing etc)
  If you are looking for a really easy L7 proxy, kubernetes ready then invite Ambassador to your cluster you will be amazed with him!

Sunday, May 06, 2018

Bonus επαναπατρισμού..

Δεν είναι λίγες οι φορές, που πιάνουμε την συζήτηση με φίλους και γνωστούς εδώ στα ξένα, για υποθετικά σενάρια επιστροφής στην Ελλάδα. Ιδιαίτερα αν η παρέα σας ή ο κυκλος σας έχει να κάνει με την πληροφορική, τέτοιες συζητήσεις μπορεί να είναι αρκετά συχνές μιας και ανήκουμε σε μια αγορά που δίνει ευκαιρίες παντού, γι'αυτό και αρκετοί από εμας δεν ζουν πια στην Ελλάδα . 

Έχω ξαναγράψει και παλιότερα, ότι ανήκω ή έτσι νιώθω τέλος πάντων, στο group των επαγγελματιών που μάλλον άργησε να κάνει το βήμα ή μάλλον πολέμησε αρκετά μέσα του μέχρι να κάνει το λεγόμενο leap of faith. Άρα τα σχεδόν 3+ χρόνια εκτός Ελλάδας δεν είναι καμία τεράστια εμπειρία.

Τον τελευταίο καιρό διαβάσαμε λοιπόν ανακοινώσεις για εταιρίες στην Ελλάδα οι οποίες προς τιμήν τους, προσφέρουν ένα κάποιο οικονομικο βοήθημα σε όσους θα ήθελαν να γυρισουν πίσω στην Ελλάδα και να ενταχθούν ξανά στην Ελληνική αγορά και πραγματικότητα. Δεν θα μείνω πολυ στα ποσά αν τα χρήματα είναι αρκετά ή όχι.Για όποιον έχει κάνει μετακόμιση στο εξωτερικό, ξέρει ότι το κόστος είναι σχεδόν όσο λεφτά μπορεί να είχες καταφέρει να αποταμιεύσεις σε 3-4 χρόνια στην Ελλάδα. Η κίνηση σαν κίνηση ομως μετράει και νομίζω είναι θετική αλλά μέχρι εκεί.

Αποφάσισα να φύγω από την Ελλάδα όταν ένιωσα ότι η επαγγελματική μου πορεία είχε μία στατική πορεία και μάλιστα ερχόταν σε αντίθεση με κάποιες προσωπικές μου βλέψεις, φιλοδοξίες αλλά και θεματα επαγγελματικής ηθικής. Δυστηχώς για μένα η brutally honest σκέψη και τρόπος δουλειάς δεν με έκανε ποτέ το αγαπημένο παιδί για πολλους. Όταν κατάλαβα ότι οι απολαβές μου είχαν φτάσει σε  ενα συγκεκριμένο σημείο και μάλλον ήταν καιρός που θα έπρεπει να ξεκινησω να σκέφτομαι να δέχομαι λιγότερα χρήματα για χάρη μονιμότητας, ή βαρεμάρας ή να πρέπει να γίνομαι αρεστός μόνο και όχι απλά χρήσιμος.  Καθώς μπαίνω σε μια πορεία να θέλω να κάνω οικογενεια, παιδιά έπρεπε να σκεφτώ καλά τι έπρεπε να κάνω και πως θα μπορέσω να παρέχω αυτά που θέλω εργαζόμενος σε μια μικρή αγορά. Έβλεπα πως αυτό που μου αρέσει να κανω γίνεται devalue χρόνο με το χρόνο και η δυνατότητα μου να βαζω χρήματα στην άκρη θα μειώνεται όσο μεγαλώνει η ηλικία μου και οι αναγκες μου. 

Θα πει κάποιος, σιγά ρε μεγάλε αυτό μπορεί να είναι η δικη σου εκδοχή. Να συμφωνήσω εν μέρη, αλλά όταν μιλάω με άλλους ανθρώπους που έχουν τα ίδια προβλήματα αλλά και φιλοδοξίες με μένα κάπου τις ίδιες απαντήσεις παίρνω.

Τα πράγματα που θα πρέπει να αλλάξουν είναι πολλα και για να καταφέρεις να φερεις ανθρώπους πίσω θα πρέπει η γενικότερη κατάσταση να αλλάξει.

Μπορεί κάποιος να πει, ότι κοίτα ναι σίγουρα βασικά θέλουμε να κρατήσουμε τον νέο κόσμο στην χώρα που έτσι και αλλιώς δεν έχει τα ίδια expectation μ'εσενα που είσαι λιγο πριν τα 40 ή να φέρουμε νέο κόσμο, εσύ είσαι μια κατηγορία που δεν μπορεί η αγορά, οι εταιρίες και η χώρα να ευχαριστήσει πια. Το καταλαβαίνω απόλυτα το σέβομαι και λέω ότι και αυτό είναι ένα θετικό σημάδι. Εξάλλου σε μια Ελλάδα που το εργασιακό κόστος (αν εισαι νόμιμος και σωστός) είναι μεγάλο, είναι πιο ευκολο να έχεις 10 junior με 500-1000 euro παρά ακριβούς με πολλά χρόνια εμπειρίας.

Άρα το να σκέφτομαι να γυρίσω στην Ελλάδα σε μια αγορά που θα με πληρώσει απο 50-70% και κάτω απο τα λεφτά που μπορώ να βρώ έξω δεν είναι και φοβερό κίνητρο. Μιλάω για πραγματικους μισθους που παίζουν στην Ευρώπη και λογικά λεφτά (οχι υπερβολές που ακουω απο πολλους ξυπασμένους). Επίσης να γυρίσω πίσω σε μια αγορά με 10-15 εταιρίες, μεσα σε 10+ χρόνια καριέρας έχεις ήδη γυρίσει τα μισα μαγαζιά και ξέρεις τι παίζει.

Μερικές φορές κάποιοι ρωτάνε, 'ΑΝ σου έδινε η τάδε εταιρία ένα φοβερό πακετό, λεφτά που δεν ειχες ξαναπάρει ποτέ στην καριέρα σου θα γύριζεις;' . Η απάντηση είναι όχι. Γιατί οι εταιρίες δεν είναι η οικογένεια σου, αν είσαι ακόμα ένας υπαλληλος. Και τι θα γίνει ρωτάω αν μετά απο 1-2 χρόνια αυτή η εταιρία για τους δικούς τις λόγους δεν θέλει να με κρατήσει πια, ή εγώ δεν αντέξω είμαι δυσαρεστημένος; Θα βρεθώ ξανά στην ίδια μικρή και δύσκολη αγορα. 

Οι λόγοι που θα κάνουν ανθρώπους σαν και μένα να γυρίσουν πίσω αυτή την στιμή είναι μη επαγγελματικοί πχ οικογένεια, ανάγκη και μάλλον τότε θα έχουμε αποφασίσει να αρκεστούμε με αυτό που αφήσαμε πίσω πριν χρόνια για χάρη άλλων αναγκων.  Δηλαδή να ξεχάσουμε καριέρες, ευκαιρίες επιλογές, αποταμίευση κτλ.

Η κατάσταση της χώρας, η φορολογία της, η μη σταθερότητα, η μικρή αγορά πληροφορικης, η ανομία και η φορολογική αταξία (ΟΧΙ δεν θέλω να φοροδιαφευγω για να βάζω κανένα χιλιάρικο στην άκρη με μαιμουδιές και να λέω ΦΟΒΕΡΑ περνάω στην Ελλάδα) είναι οι λόγοι που αρκετός κόσμος δεν θα γύρίσει πίσω. 

Όταν οι 15 εταιρίες γίνουν 150, όταν οι μισθοί γίνουν ανταγωνιστικοί, όταν μεγάλο ποσοστό των εταιριών δεν θα ειναι δημοσιο-επιδοτούμενες, όταν η φορολογία στους ταπεινούς μισθωτους θα γίνει ξανά λογική και δεν θα με αντιμετωπίζει το κράτος σαν cash cow, όταν το κόστος ζωής θα είναι λογικό και θα μπορώ να κάνω αποταμίευση και όχι να δουλευω 5 χρόνια για να βάλω στην άκρη λεφτά που δεν μπορούν να υποστηρίξουν μια γέννα ή μια έκτακτη ανάγκη, τότε να επιστρέψουμε στην Ελλάδα.

Μέχρι τότε, δίνουμε τον αγώνα μας στο εξωτερικό που φυσικά τα πράγματα δεν είναι ρόδινα ουτε τέλεια, αλλά τουλάχιστον εχουμε επιλογές!

Sunday, March 11, 2018

My take on terraform #terraform

Disclaimer - don't shoot the pianist

It's been some time since I wanted to write this post. For those that are going to read it, I need to say that a) I hate the term DevOPs as a way to 'label' engineers b) I'm a software developer  meaning that I am the type of guy who really enjoys high-level languages like Java/ Kotlin etc and has been following DevOps principles per occasion or workplace, but being honest I did not have a solid Ops/Admin background. In other words, not all shops out there do or encourage DevOps as a engineering culture, so it really depends if you re going to experience it or not.

It is evident though, that through time in many companies, the typical developer stereotype is changing, a developer is asked to engage more often with then overall release life cycle of software rather than limiting himself/herself to only writing the code and then delegating to others to configure, ship, deploy and maintain software in production.

IMHO there is no better person to bring the code alive and take care of it, from the person who wrote it and knows its strengths and weaknesses. So please don't shoot the pianist in case you don't agree with my views or my perception of things.


My experience with Terraform

I started seriously working with terraform 2 years ago. At first, it was like oh ok I don't know what I am doing, I don't understand what is going on.  Then I progressed on copy-paste-ing this template run it and see what is happening, then I was like 'OK I am going to copy paste someone else's thing modify it a bit and support it'. Now I am able to follow through all the templates, modules and in general kind of find my way around most of the production deployments.

So definitely there is a learning curve, if you happen to work in a place that Terraform is being adopted the higher the chances are that you are going to step up your game once you seriously invest some time. It's not rocket science, after all from a user's perspective,  its some simple notation, where you express how you want your cloud infrastructure to look like. Example, terraform please create for me a VM, some networking, a load balancer, some access right on the VM and please initialize them. Boom!

Having already lots of day to day experience with kubernetes and being very confidend with Kubernetes deployment descriptors and tools like Helm, Terraform feels familiar in its core. I would argue that terraform is very easy to use, the only thing you need is a couple of good resources, some reading and your personal cloud account (e.g AWS). It is not a corporate-only tool, neither requires some magic heavy infrastructure.

Just a command line utility and a couple of API keys to access your cloud.


Books and resources

Through this ongoing journey, I used the following resources :
  1. Terraform Up And Running
  2. A comprehensive guide to Terraform by the GruntWork guys , here.  
  3. The Terraform book
  4. The official Terraform documentation - but is better if you start with one of the books and then take over the official documentation (IMHO). At least this is what I did.
  5. Terraform videos / short courses on Plurarsight.
I would highly suggest you start with Terraform Up and Running and them move to 'The Terraform Book'. I kind of did it the opposite way, which I think it was wrong, so I high recommend you go through the above resources with that order.


So what is Terraform? (for newbies)

Its a utility that you can download on your machine (I used brew) or use it as a docker container, which will enable you to change and shape your cloud infrastructure. This utility is fed with a set of  files (deployment descriptors) where you will describe what things you want to run and instantiate on your cloud  (e.g AWS or google cloud).  E.g spin vms, databases, s3 buckets, etc.

Terraform will read these files and translate them to actual API calls to the cloud provider, will also do the dirty job instantiating the resources for you with the correct order and 'save' the state, meaning save somewhere what is already deployed so that next time that you run it can check what is already deployed and peform an more fine grained update (like a diff).

Some might argue, why you want to do that, since you can login to the console of your cloud provider and eventually perform all these actions. Well for sure you can do that but
  • Manual changes to your cloud infrastructure especially as it gets bigger and complex are error prone
  • Manual means - manual labor - which means that you need to do the same things all over again - where you could just automate and script them.
  • Every time you do a change you need to make sure that you dont break or alter something else, and we all know how complex the settings are in many cloud providers. 
  • You dont have continuous deployment semantics. Repeatable builds and configuration.
  • Support more than one users, that try to 'mutate' the state, e.g development teams or organizations.


Should I invest time learning terraform as a skill?

Definitely yes if you ask me. As I said my background is not Ops so mastering the details of different cloud providers proves to be a full time job for many people. In my mind whatever tool or technology helps to reduce this learning curve is a good thing. Terraform abstracts in a way your interaction with the cloud provider. It does not abstract the cloud provider, meaning that if I write some terraform for AWS and I want to do more or less the same things for Google cloud, you will end up writting something a bit different, since cloud providers are different. But at least you will have a common way of interacting with the different clouds, and you will develop skills on how to quickly leverage and re-use all your terraform resources. Instead of becoming a master on the AWS cloud console and the google Cloud console, you will be a master on writting template files that instruct terraform to perform stuff for you.


Terraform is not only about AWS

Since AWS holds a large portion of the market ( 50/60%) meaning is the no1 cloud of choise for individuals and companies, many people assume that Terraform is somewhat AWS specific, which is of course not true. Terraform has this thing called providers, which is actually an abstraction for many different clouds or services. So yes the most commonly used is AWS but you will find an extensive list of different services that can be 'terraformed' . Lately I am spending lots of time with the Fastly Provider, fastly being a CDN! 


If you know terraform you dont become a cloud expert! A small trap.

Well it is not  a trap of Terraform, but no matter if you master the way terraform works and you quickly find yourself 'terraforming all the things', using its powerful template engine and all the nice utilities and resources, you must really know what you are doing.

In plain words, if you are good at terrraform does not mean that suddenly you became an AWS or Google Cloud expert! Becoming an expert on the different clouds, is a full time job and requires time and effort. Thats why we have AWS experts, Google cloud experts etc. Maybe more modern clouds, e.g the Google cloud abstract more things compared to traditional (older clouds) like AWS, but still at some point you are going to end up, searching or having to master the details. 

I have to admint my journey on mastering thenAWS cloud is still on going and I feel I have a looong way, but this is part of the process I guess!