https://echorand.me/ Recent content on Explorations - computing, research, etc. Hugo -- gohugo.io en-us Mon, 04 May 2026 00:00:00 +0000 https://echorand.me/posts/looking-for-answers/ Mon, 04 May 2026 00:00:00 +0000 https://echorand.me/posts/looking-for-answers/ Some examples of where i get limited by a specific programming language/framework’s syntax/particular way of doing things, and how I navigate it by trying to see if i can solve it myself based on what i know or good old internet search, (i.e. manually looking at links/posts) and learning a bit more in the process. I know there is research now showing the impact of using LLMs on our brain, but to me, this is just how I want to do my work. https://echorand.me/posts/devops-notes/ Sun, 19 Apr 2026 00:00:00 +0000 https://echorand.me/posts/devops-notes/ Some suggestions that i offered to someone else, looking for some general guidance around setting up their DevOps processes. Overall, the idea is that you run as many automated checks/processes as possible before the code is merged into your main/master repository and deployed to production so that you can: Prevent bad code from being merged - bringing down your systems post deployment Ensure code quality Ensure uniform code style Unit tests and integration tests are run before code is merged Code reviewers understand what they are reviewing Code mergers know how the change being merged can be reverted Secrets are not being committed accidentally to version control The techniques have come to be referred to “shifting left” (imagine that left is the developer’s laptop and the right is the production system): https://about. https://echorand.me/posts/dmwp-and-ai-notes/ Mon, 16 Mar 2026 00:00:00 +0000 https://echorand.me/posts/dmwp-and-ai-notes/ In what can only be described as sign of the times, one of the books I wrote, arguably, the most impactful one, Doing Math With Python is one of the works in the Anthropic Copyright Settlement. This is just a post to commemorate this time in history and in a bizarre twist, me being part of it. Off late, I have realized, I don’t like an “agent” doing my software work. https://echorand.me/posts/sepsis-onset-calcualation-mimic-iii/ Mon, 10 Nov 2025 00:00:00 +0000 https://echorand.me/posts/sepsis-onset-calcualation-mimic-iii/ In this blog post, I am going to desribe how we can calculate sepsis onset time using sepsis-3 criteria for the MIMIC-III patient data. Reading the relevant tables Sepsis-3 Criteria Operationalization in code Identifying the onset of suspected infection Time of drawing of the first culture Time of administering of antibiotics Identifying infection onset Identifying organ failure/dsyfunction Calculation of total SOFA score Finding sepsis onset time Code Data access Assumptions References Reading the relevant tables We first read the data from the CSV files (corresponding to the relevant tables): https://echorand.me/posts/slowing-down-read-api-documentation/ Tue, 26 Aug 2025 00:00:00 +0000 https://echorand.me/posts/slowing-down-read-api-documentation/ I remember the days of reading API documentation, and figuring things out and experiencing a joy having done so. The experience was enriching and that’s really what enabled me keep doing it, because of a sense of joy I derive from it. I remember the days, when that was just how things happened. Now, my first call is to open my favorite LLM powered tool and ask it to do what I need to be done. https://echorand.me/posts/downloading-mimic-cxr-jpg/ Sun, 14 Jul 2024 00:00:00 +0000 https://echorand.me/posts/downloading-mimic-cxr-jpg/ I recently downloaded MIMIC-CXR-JPG, from Google cloud storage. The reason I had to eventually use Google cloud storage download was because of the suggestion that was offered for working around the bandwidth constrained physionet.org servers. These are some notes that might help someone else. Before you get started: Right off the bat, the first thing you want to keep in mind is the bucket’s configured so that the downloader will bear the cost of network transfer. https://echorand.me/posts/github-oauth/ Wed, 21 Sep 2022 00:00:00 +0000 https://echorand.me/posts/github-oauth/ In this post, we are going to discuss an implementation detail about how we can use a custom HTTP client to use the Go GitHub SDK and the https://pkg.go.dev/golang.org/x/oauth2 package. Introduction Using a custom HTTP client with golang.org/x/oauth2 Using the custom HTTP client with Go Github SDK Summary References Introduction Let’s create an example *oauth2.Config for communicating with GitHub.com oauth provider: oauthConf = &oauth2.Config{ ClientID: getEnvironValue("CLIENT_ID"), ClientSecret: getEnvironValue("CLIENT_SECRET"), Scopes: []string{"repo", "user"}, // see the project desrciption for understandng why we need full scopes here Endpoint: github. https://echorand.me/posts/go-http-client-middleware/ Mon, 19 Sep 2022 00:00:00 +0000 https://echorand.me/posts/go-http-client-middleware/ In this post, I am going to share what I have learned about writing HTTP client middleware in Go. Let’s get started! HTTP client and Transport Writing your own RoundTripper implementation Returning static responses Summary HTTP client and Transport Go’s http.Client defines a default value for the Transport field when one is not specified: type Client struct { // Transport specifies the mechanism by which individual // HTTP requests are made. https://echorand.me/posts/shared-functionality-using-middleware-ii/ Tue, 06 Sep 2022 00:00:00 +0000 https://echorand.me/posts/shared-functionality-using-middleware-ii/ I delivered a talk at the recently concluded Kiwi PyCon XI. Checkout some photos of the event from Kristina DC Hoeppner here. I attended it virtually as I was unable to travel due to last minute interruptions from a certain virus. The organizers were of course very well prepared for that. Massive kudos to the Next day video team and everyone else involved in the process, including the paper chair and the director who did go the extra mile in convincing me that I should make use of the virtual presentation opportunity rather than withdrawing my talk altogether. https://echorand.me/posts/shared-functionality-using-middleware/ Mon, 30 May 2022 00:00:00 +0000 https://echorand.me/posts/shared-functionality-using-middleware/ I delivered a talk at the recently concluded PyCon US 2022 conference. Amazing conference, great work by the organizers and everybody involved. This was my third time at PyCon US and my first time participating in the post conference sprints. I contributed to the Pyodide project during the sprints, and was a great experience to work with the core developers during the sprints and for a couple of days after the sprints. https://echorand.me/posts/py-script-monkey-patch-requests/ Sun, 08 May 2022 00:00:00 +0000 https://echorand.me/posts/py-script-monkey-patch-requests/ At PyCon US 2022, Anaconda announced PyScript: Python in the Browser. So far my understanding is that it builds on Pyodide and makes it magically easy to bridge the world of the Browser - the Document Object Model (DOM) and Python. It’s so magical that you can simply copy scripts that you were running using Python installed on a computer and they just run in the browser. Check out the blog post for some demos. https://echorand.me/posts/hard-to-not-be-a-computer-user/ Mon, 07 Feb 2022 00:00:00 +0000 https://echorand.me/posts/hard-to-not-be-a-computer-user/ I saw a computer for the first time at school when I was 10 years old (~1995/96). I failed horribly in the first few years to actually get programming. Then i had this teacher with whom i took group classes, who taught me computer programming, properly. GW Basic, then some Java, some C and C++. I picked up Linux, installed Red Hat Linux 6/7 i think on a PC I managed to acquire citing that it will be useful for my father’s shop - yes i knew exactly what I was doing. https://echorand.me/posts/on-my-learning-style/ Sun, 12 Dec 2021 00:00:00 +0000 https://echorand.me/posts/on-my-learning-style/ I have finally gotten around to reading (listening) “The Code Breaker: Jennifer Doudna, Gene Editing, and the Future of the Human Race” by Walter Isaacson. And in one of the first chapters, the writer talks about Jennifer Doundna’s thoughts (I think) on how often it’s the practical application of science or in solving a problem, we improve/revise our understanding of the fundemental building blocks (basic sciences) involved in solving the problem. https://echorand.me/posts/go-values-references-etc/ Sun, 21 Nov 2021 00:00:00 +0000 https://echorand.me/posts/go-values-references-etc/ Update: Please note that my usage of “Call by reference” is not correct here as pointed out by a Reddit community member here. Read it as “Call by passing pointer values”. While trying to understand this a bit more, I came across a few more links to the topic. You can find them here in my reply. I will do a follow up post on this topic perhaps. I am grateful that the reddit commenter helped me revise my mental model on the topic. https://echorand.me/posts/go-flag-option-append/ Sat, 13 Nov 2021 00:00:00 +0000 https://echorand.me/posts/go-flag-option-append/ While working on the solutions to exercises for my soon to be published book Practical Go, I needed to implement a way to implement an option in my command line application which could be specified multiple times. The result would be that all the values specified would form a list of the values. To make that concrete, consider that you are writing a command line HTTP client application. You want to add one or more headers to an outgoing request, so this is the user interface that you want for your application: https://echorand.me/posts/go-test-stdin/ Sat, 06 Nov 2021 00:00:00 +0000 https://echorand.me/posts/go-test-stdin/ While working on the solutions to exercises for my soon to be published book Practical Go, I wanted to write a test function which would simulate a user not providing an interactive input when one was asked for. However, I noticed that the test function would not wait for me to provide the input and just continue the execution. Consider the following test function: import ( "bufio" "fmt" "os" "testing" ) func TestInput(t *testing. https://echorand.me/posts/serialize-deserialize/ Sat, 18 Sep 2021 00:00:00 +0000 https://echorand.me/posts/serialize-deserialize/ I always get confused between “serialization” and “deserialization”. Perhaps, that is because I am trying to memorize what they are, and then trying to recall from memory. Of course, it’s sufficient to remember only one of them correctly. So, here’s my trick that I am going to use from now on. It is derived from this Wikipedia article on the subject, in the Drawbacks section: Serialization breaks the opacity of an abstract data type by potentially exposing private implementation details. https://echorand.me/posts/computing-transparency/ Wed, 15 Sep 2021 00:00:00 +0000 https://echorand.me/posts/computing-transparency/ One word that has always baffled me in the context of software/computing is “transparency”. “This change is completely transparent to the application” - or some version of it is often used to tell consumers of an application/library that a certain change won’t be noticeable by the consumer at all. If I were to put in my English language hat on (a non-native at that), i would think that means “so, the consumer will see the change” and not the intended “the consumer will not see the change”. https://echorand.me/posts/chaos-engineering-talk/ Mon, 13 Sep 2021 00:00:00 +0000 https://echorand.me/posts/chaos-engineering-talk/ I delivered a talk at the recently concluded PyCon Australia conference. Amazing conference, great work by the organizers and everybody involved. Fantastic experience! My talk My talk was in the DevOops track titled “Planning for Failure using Chaos Engineering”. Since I joined Atlassian, I have come to realize the value chaos engineering/wargaming brings to the table when it comes to assess the production readiness of your services. However, as I discuss in the talk, it’s not really just about your software’s readiness, it is also about your people systems. https://echorand.me/posts/go-named-return-values/ Mon, 05 Jul 2021 00:00:00 +0000 https://echorand.me/posts/go-named-return-values/ In Go, there are a couple of ways to return values from a function. Non-named return values Until today, I had been exclusively using the following style of what i am going to refer to as “Non-named return values”: func myFunc() (int, error) { return 1, errors.New("An error") } You declare in the function signature that you will be returning an int and and an error. Then in your code, you return those values. https://echorand.me/posts/go-module-check-path/ Thu, 15 Apr 2021 00:00:00 +0000 https://echorand.me/posts/go-module-check-path/ The Go module path enforces certain restrictions as expected on what constitutes a valid path. Try running go mod init https://foo.bar/baz for example. Now, what if you as a Go programmer needed to run this check yourself? That’s where the golang.org/x/mod/ package comes in. It has a number of functions, one of them being the CheckPath function, which you can use as follows: // Using Go 1.16 package main import ( "fmt" "golang. https://echorand.me/posts/go-file-mutex/ Mon, 18 Jan 2021 00:00:00 +0000 https://echorand.me/posts/go-file-mutex/ Using a file for persistent storage (and not a database - datastore/object store) sounds like an academic exercise. For me, it brings back memories of writing a structure in C (programming language) to a file to simulate a student record database. However, there may be situations where you may just get by using it especially when you just want to run a single copy of your application. Let’s see a pattern for implementing concurrency safe file access in Go using sync. https://echorand.me/posts/go-embed/ Sun, 20 Dec 2020 00:00:00 +0000 https://echorand.me/posts/go-embed/ Demo - Embedding a template Demo - Serving files from a directory Learn more The most exciting feature for me in the Go 1.16 release is the new “embed” package which allows you to embed a file contents as part of the Go application binary. This ability so far was most easily available via using various third party packages and they worked great. You could also use go generate to roll out your own solution, if needed. https://echorand.me/posts/micros-auth-kubernetes/ Sun, 13 Dec 2020 00:00:00 +0000 https://echorand.me/posts/micros-auth-kubernetes/ In my latest article for the folks at learnk8s, I write about establishing authentication between services deployed in Kubernetes. Specifically, we discuss how you can use the Kubernetes primitives - Service accounts with a new feature - Service Account Token Volume Projection to setup authentication between two HTTP services. You can find the article here with the accompanying code repository here. https://echorand.me/posts/policy-enforcement-kubernetes/ Mon, 27 Jul 2020 00:00:00 +0000 https://echorand.me/posts/policy-enforcement-kubernetes/ My two recent articles, Validating Kubernetes YAML for best practice and policies and Enforcing policies and governance for Kubernetes workloads looks at the topic of enforcing policies for your Kubernetes workloads. Check them out and let me know if you have any comments. https://echorand.me/posts/gatekeeper-kubernetes/ Thu, 28 May 2020 00:00:00 +0000 https://echorand.me/posts/gatekeeper-kubernetes/ Introduction Gatekeeper allows a Kubernetes administrator to implement policies for ensuring compliance and best practices in their cluster. It makes use of Open Policy Agent (OPA) and is a validating admission controller. The policies are written in the Rego language. Gatekeeper embraces Kubernetes native concepts such as Custom Resource Definitions (CRDs) and hence the policies are managed as kubernetes resources. The GKE docs on this topic are a good place to learn more. https://echorand.me/posts/getting-started-with-golang-plugins/ Tue, 26 May 2020 00:00:00 +0000 https://echorand.me/posts/getting-started-with-golang-plugins/ Introduction In this post, I will share some of my learnings and explorations on plugins in Golang. We will write a “driver” program which will load two plugins and execute a certain function which are present in both of them. The driver program will feed an integer into the first plugin, which will run some processing on it. The result of the first plugin is fed into the second plugin and finally the driver program will print the result. https://echorand.me/posts/kubernetes-pod-security-policies/ Wed, 20 May 2020 00:00:00 +0000 https://echorand.me/posts/kubernetes-pod-security-policies/ Welcome to this new blog post! Introduction Enforcing policies Using kustomize to manage policies Rolling the policy changes out Multiple matching policies Conclusion Introduction Pod security policies are cluster level resources. The Google cloud docs has some basic human friendly docs. A psp is a way to enforce certain policies that pod needs to comply with before it’s allowed to be scheduled to be run on the cluster - create or an update operation (perhaps a restart of the pod? https://echorand.me/posts/fluentbit-kubernetes/ Thu, 07 May 2020 00:00:00 +0000 https://echorand.me/posts/fluentbit-kubernetes/ Metadata I have posted this article on dev.to where I welcome comments and discussions. Introduction Log forwarding is an essential ingredient of a production logging pipeline in any organization. As an application author, you don’t want to be bothered with the responsibility of ensuring the application logs are being processed a certain way and then stored in a central log storage. As an operations personnel, you don’t want to have to hack your way around different applications to process and ship logs. https://echorand.me/posts/hugo-reverse-sort-modified/ Tue, 05 May 2020 00:00:00 +0000 https://echorand.me/posts/hugo-reverse-sort-modified/ This blog is managed as a git repository and I use Hugo as the framework for managing it. I am using the Hugo classic theme which I have tweaked slightly and store it along with the blog source. I wanted to modify the sorting of the blog post titles on the index page so that the most recently modified page was displayed first. The support was added to Hugo 4 years back, so the following template worked for me: https://echorand.me/posts/cloud-custodian-aws/ Tue, 28 Apr 2020 00:00:00 +0000 https://echorand.me/posts/cloud-custodian-aws/ Introduction In this post, I will describe my experiments with using Cloud Custodian to perform various tasks usually falling into the bucket of compliance and sometimes convention. I encourage you to take a look at some of the example policies. Some of the areas I will cover are resource tagging and unused resources across multiple AWS accounts. Installation and setup Cloud Custodian is a Python 3 application, so you will need that installed. https://echorand.me/posts/python-restrict-attributes/ Thu, 02 Jan 2020 00:00:00 +0000 https://echorand.me/posts/python-restrict-attributes/ A reader of my book “Doing Math with Python” wrote to me a few weeks back about a strange problem they were having. They were trying to create an animated projectile motion from the code listing in the book. However, they were not seeing the expected results. Worse, there were no errors. They figured the issue on their own eventually since I didn’t get the time to reply back and the issue was there was an attribute center on a class that was being set in the correct version. https://echorand.me/posts/kubernetes-manifest-management/ Fri, 20 Dec 2019 00:00:00 +0000 https://echorand.me/posts/kubernetes-manifest-management/ Introduction There are two broad discussion points in this post: Managing the lifecycle of Kubernetes YAML manifests Static guarantees/best practices enforcements around Kubernetes YAML files before they are applied to a cluster Prior art and background Please read this article to get a more holistic view of this space. What follows is my summary of what I think is the state at this stage and how it fits in with my two gols above. https://echorand.me/posts/correlated-error-logging/ Thu, 12 Dec 2019 00:00:00 +0000 https://echorand.me/posts/correlated-error-logging/ Please note this document is currently in progress. You may read this other post instead which illustrates what I wanted to discuss in this post with an implementation in Go. Introduction In a service oriented architecture, more popular these days as a microservice oriented architecture it’s notoriously difficult and time consuming to find out whether a certain error observed in a particular service may have been caused to an error in another service. https://echorand.me/posts/golang-ops-automation/ Sat, 07 Dec 2019 00:00:00 +0000 https://echorand.me/posts/golang-ops-automation/ Since around 1.5 years ago, I have been working in roles which has required to automate various things usually classified as “infrastructure and operations” work. Sometimes they have involved working with cloud infrastructure and at other times they have involved interacting with databases and other services via HTTP APIs. During this period, to solve such tasks I have started using the Go programming language to implement these things. I shared my experience and some examples in a talk at the combined Go/Women who Go and Infrastructure as code meetup in Sydney. https://echorand.me/posts/jenkins-docker-workflow-inside/ Tue, 26 Nov 2019 00:00:00 +0000 https://echorand.me/posts/jenkins-docker-workflow-inside/ Introduction The docker workflow plugin enables leveraging Docker containers for CI/CD workflows in Jenkins. There are two broad patterns one would generally use containers in their CI/CD environment. The first would be as “side car” containers - these are containers which run alongside your tests/other workflow and provide services such as a database server, memory store and such. The second would be as base execution environments for the tests/builds. The documentation for the plugin explains these two patterns and how to achieve either using Jenkins workflow plugin. https://echorand.me/posts/bash-functions-exit/ Fri, 18 Oct 2019 00:00:00 +0000 https://echorand.me/posts/bash-functions-exit/ Monday was just beginning to roll on as Monday does, I had managed to work out the VPN issues and had just started to do some planned work. Then, slack tells me that new deployment had just been pushed out successfully, but the service was actually down. Now, we had HTTP healthchecks which was hitting a specific endpoint but apparently that was successful, but functionally the service was down. So I check the service logs, which shows something like this: https://echorand.me/posts/user-management-aws-eks-cluster/ Fri, 23 Aug 2019 00:00:00 +0000 https://echorand.me/posts/user-management-aws-eks-cluster/ Introduction When implementing a solution for allowing users other than the cluster creator to access the cluster resources we are faced with two fairly old generic problems - authentication and authorization. There are various ways one can solve these problems. I will discuss one such solution in this post. It makes use of AWS Identity and access management (IAM) features. This in my humble opinion is the simplest and hopefully secure enough solution when it comes to EKS. https://echorand.me/posts/go-notes/ Wed, 07 Aug 2019 00:00:00 +0000 https://echorand.me/posts/go-notes/ Repeating the same argument to Printf If we wanted to repeat the same argument to a call to fmt.Printf(), we can make use of “indexed” arguments. That is, instead of writing fmt.Printf("%s %s", "Hello", "Hello"), we can write fmt.Printf("%[1]s %[1]s", "Hello"). Learn about it in the docs. Multi-line strings Things are hassle free on the multi-line strings front: package main import ( "fmt" ) func main() { s := `Multi line strings are easy. https://echorand.me/posts/kubernetes-notes/ Fri, 02 Aug 2019 00:00:00 +0000 https://echorand.me/posts/kubernetes-notes/ Introduction This in-progress page lists some of my findings while working with Kubernetes. EKS cluster setup You may also find this guide from spacelift.io useful. This section will have findings that are relevant when working with an AWS EKS cluster. Terraform configuration for master This is based on the tutorial from the Terraform folks here. Unlike the tutorial though, I assume that you already have the VPC and subnets you want to setup your EKS master in. https://echorand.me/posts/nginx-non-www-www-redirect/ Fri, 19 Jul 2019 00:00:00 +0000 https://echorand.me/posts/nginx-non-www-www-redirect/ Introduction I wanted a Nginx configuration which would satisfy the following requirements: Any example.com requests should be redirected to www.example.com The above should happen for http and https http://example.com should redirect directly to https://www.example.com Solution We will need four server blocks: http - example.com (listen on 80) http - www.example.com (listen on 80) https - example.com (listen on 443) https - www.example.com (listen on 443) I obviously went through a bit of hit and trial, but my main issue was around how I would setup (3) correctly. https://echorand.me/posts/linux_shell_autocompletion/ Wed, 03 Jul 2019 00:00:00 +0000 https://echorand.me/posts/linux_shell_autocompletion/ Welcome to this new blog post! Introduction Setting up DIY bash completion Data provided to completion handlers Single <TAB> and double <TAB><TAB> Getting good old BASH completion back Magic of bash-completion package compgen built-in command Learning more Bash completion for applications cobra (Golang), click (Python) and clap (Rust) complete (Golang) and shell_completion (Rust) Conclusion Introduction What happens when you press <command> <TAB> or <command> <TAB><TAB>? You get a bunch of suggestions with one of them just the one you had in mind you were going to type. https://echorand.me/posts/pg-sql-notes/ Thu, 27 Jun 2019 00:00:00 +0000 https://echorand.me/posts/pg-sql-notes/ Some notes on PostgreSQL which you may find useful. Thanks to all those numerous StackOverflow answers that helped me do my job at hand. Schemas and Database The database is the highest level of organization. A database can have one or more schemas. The public schema is present by default and all tables created are created in this schema, if not otherwise specified. Learn more about it in the documentation https://echorand.me/posts/strace-nginx/ Wed, 19 Jun 2019 00:00:00 +0000 https://echorand.me/posts/strace-nginx/ I was debugging a issue where we were getting truncated logs in ElasticSearch in the context of a setup as follows: Application Logs -> Fluentd (logging) -> Nginx -> ElasticSearch The original problem turned out to be on the application side, but my first point of investigation was what are we getting on the nginx side? Do we get the entire message that we are expecting and something is going on the ElasticSearch side? https://echorand.me/posts/nginx-geoip2-mmdblookup/ Fri, 24 May 2019 00:00:00 +0000 https://echorand.me/posts/nginx-geoip2-mmdblookup/ I wanted to setup Nginx logging so that it would perform GeoIP lookup on the IPv4 address in the X-Forwarded-For header. Here’s how I went about doing it on CentOS 7. This nginx module integrates Maxmind GeoIP2 database with the RPMs being available by getpagespeed.com. Once I had installed the module, the hard part for me was how to get the data I wanted - city, timezone information and others from nginx and the geoip2 module integration. https://echorand.me/posts/terraform_from_toml/ Thu, 04 Apr 2019 00:00:00 +0000 https://echorand.me/posts/terraform_from_toml/ In this post, we will see how we can use Golang to generate Terraform configuration from a TOML specification. That is, given a TOML file, like: subnet_name = "SubnetA" rules = [ {rule_no=101, egress = false, protocol = "tcp", rule_action = "allow", cidr_block = "127.0.0.1/32", from_port = 22, to_port = 30}, ] We will generate: # This is a generated file, do not hand edit. See README at the # root of the repository resource "aws_network_acl_rule" "rule_SubnetA_ingress_101" { network_acl_id = "${lookup(local. https://echorand.me/posts/docker_logs_variable/ Mon, 11 Mar 2019 00:00:00 +0000 https://echorand.me/posts/docker_logs_variable/ docker logs by default shows the container’s stdout and stderr logs. However, what I discovered was that the stderr logs from the container are output to the host system’s stderr as well. I was expecting everything from the container to be on the host’s stdout. Let’s see a demo. Consider the Dockerfile: FROM alpine:3.7 CMD echo "I echoed to stdout" && >&2 echo "I echoed to stderr" Let’s build it and run it: https://echorand.me/posts/aspnet_compiler_node_modules/ Tue, 05 Mar 2019 00:00:00 +0000 https://echorand.me/posts/aspnet_compiler_node_modules/ I love working in software. Mostly things work as expected, but at times no. I changed A, how can Z be affected - after all they are all miles apart. Right? Wrong. Z can be affected. Today’s story is the latest - totally unexpected, but not surprising. Background We run ASP.NET compilation on our code base as part of every build in our Continuous Integration (CI) pipeline. I changed some part of the pipeline so that we started caching node_modules directory between different parts of a pipeline. https://echorand.me/posts/traefik-aspnet/ Mon, 21 Jan 2019 00:00:00 +0000 https://echorand.me/posts/traefik-aspnet/ Recently, I wrote two articles about using traefik as a reverse proxy. The first article discussed deploying a ASP.NET framework application and the second discussed deploying ASP.NET core applications. In both cases, I demonstrated the following: Docker native integration In-built support for LetsEncrypt SSL certificates One of the things I didn’t discuss was how we could setup an architecture which allowed us to do zero-downtime deployments without any external help. By external help I mean taking the application instance out of the DNS pool, having another healthchecking process automatically taking it out of a load balancing pool or something like that. https://echorand.me/posts/scheduled-task-docker-prune/ Mon, 21 Jan 2019 00:00:00 +0000 https://echorand.me/posts/scheduled-task-docker-prune/ Windows docker images can be bulky and on a server that you are deploying your application as docker images, the free disk space becomes a metric to watch out for. The following script will setup a Scheduled tasks to be run at a 7.0 PM UTC which will prune all unused images: # Scheduled tasks if (-Not (Test-Path "C:\ScheduledScripts")) { mkdir C:\ScheduledScripts } $command="docker image prune --all -f" $command | Out-File -encoding ASCII C:\ScheduledScripts\DockerImagePrune. https://echorand.me/posts/docker-build-and-ephemeral-port-issues/ Mon, 14 Jan 2019 00:00:00 +0000 https://echorand.me/posts/docker-build-and-ephemeral-port-issues/ TLDR; If you are having trouble with docker build and ephemeral port ranges, we can use iptables to solve the issue: $ sudo iptables -t nat -I POSTROUTING -p tcp -m tcp --sport 32768:61000 -j MASQUERADE --to-ports 49152-61000 I have written previously about how things get interesting with ephemeral port ranges in a Windows and Linux environment and AWS network acls. Today’s post is related to the same topic but specifically relevant if you are building docker images in such an environment. https://echorand.me/posts/aws-codedeploy-autoscaling-group/ Wed, 19 Dec 2018 00:00:00 +0000 https://echorand.me/posts/aws-codedeploy-autoscaling-group/ When we create an AWS Code Deploy deployment group via Terraform or CloudFormation and integrate with an Auto Scaling Group, it also by default creates an initial lifecycle hook which ensuresthat a new code deployment gets triggered when a scale-out event occurs. It is all very “magical” and it is one of those cases where you have troublesome behavior especially when you are managing your infrastructure as code. The troublesome behavior happens as a result of the lifecycle hook creation being a side-effect of creating a deployment group rather than an explicit operation that the user performs. https://echorand.me/posts/hard-links-soft-links-linux/ Fri, 09 Nov 2018 00:00:00 +0000 https://echorand.me/posts/hard-links-soft-links-linux/ Much has been written (and asked) on the topic of hard links and soft links (a.k.a symbolic links) on Linux. I have read a few of those more than once. However, I end up getting confused between the two, specifically the differences between the two. So, here’s my post on the topic with the hope that I will stop getting confused ever again. Our setup Let’s create a file and write a line into it: https://echorand.me/posts/manual-letsencrypt-dns-challenge/ Thu, 08 Nov 2018 00:00:00 +0000 https://echorand.me/posts/manual-letsencrypt-dns-challenge/ I wanted to create a new SSL certificate for IIS hosted ASP.NET framework application. The key data that may make this post relevant to you are: Let’s Encrypt Challenge mode: DNS TXT record DNS provider: GoDaddy Target web server: IIS Target operating system: Windows Local operating environment/system: Linux (including Windows Subsystem for Linux). Why I chose certbot? I decided to use certbot since it allowed me do create the DNS TXT entries manually. https://echorand.me/posts/fedora-29-scientific-vagrant-boxes/ Thu, 01 Nov 2018 00:00:00 +0000 https://echorand.me/posts/fedora-29-scientific-vagrant-boxes/ Fedora Scientific brings together the most useful open source scientific and numerical tools atop the goodness of the KDE desktop environment. From the beginning, our focus has been to provide scientists, engineers and programmers in numerical/scientific computing a Linux distribution which has most of the tools that they would need to use as part of their work. With the Fedora 29 release, we have also started to provide Vagrant boxes for Fedora Scientific to make it even easier to use Fedora Scientific while not doing away with your present operating system. https://echorand.me/posts/on-sane-defaults/ Fri, 26 Oct 2018 00:00:00 +0000 https://echorand.me/posts/on-sane-defaults/ My task at hand was simple. Build a Docker image of a ASP.NET application (full framework) hosted in IIS on a build host (host1) and move it to a deployment host (host2) and run it. This is a story of how I spent close to two full working days trying to debug a simple issue which sane default behavior of a tool would have cut it to seconds. Key details The key details that are important to my story are: https://echorand.me/posts/aws-vpc-internet-connectivity-subnets/ Mon, 15 Oct 2018 00:00:00 +0000 https://echorand.me/posts/aws-vpc-internet-connectivity-subnets/ Welcome to this new blog post! Public subnet - Public IP Ingress Egress Public subnet - No Public IP Private subnet - Public IP Ingress Egress Private subnet - No Public IP Ingress Egress Learn more We can have two kinds of subnets inside a AWS VPC - private and public. A public subnet is one which is attached to an Internet Gateway. This essentially adds a routing table entry to the subnet’s routing table sending all Internet traffic to an Internet Gateway. https://echorand.me/posts/flyway-baseline-migration/ Fri, 12 Oct 2018 00:00:00 +0000 https://echorand.me/posts/flyway-baseline-migration/ If you are trying to introduce flyway to an existing database with the schemas and tables already created, you may find flyway baseline command useful. Let’s say you already have the migration scripts written, or perhaps dumped out of your existing DB setup, and they are: V1__foo.sql V2__foo_new.sql. V3__bar.sql You basically want to say - flyway, ignore all scripts upto V3, but if there are newer migrations, run them. To do so, you run the baseline command, like so: https://echorand.me/posts/git-only-stage-partial-changes/ Thu, 11 Oct 2018 00:00:00 +0000 https://echorand.me/posts/git-only-stage-partial-changes/ Let’s say you have made a few changes to a file and only want to stage only some of those changes for commit. You may want to do so either to create a nice commit history or may be you just want to discard some of the changes. Either way, git add -p is our friend. Let’s see an example: $ git add -p file1 ... index b82819b..a281865 100644 --- file1 +++ file2 @@ -139,8 +139,14 @@ else docker run \ -e ConnectionStrings__Db="Data Source=\"tcp:$SQL_SERVER_IP\";Persist Security Info=True;Initial Catalog=$DB_NAME;User ID=$APPLICATION_USER_NAME;Password=$DB_APPLICATION_PASSWORD" \ - -e App__RedisCS="$REDIS_IP" \ + -e "${APPLICATION_NAME}_Web__CI__ConnectionStrings__Db"="Data Source=\"tcp:$SQL_SERVER_IP\";Persist Security Info=True;Initial Catalog=$DB_NAME;User ID=$APPLICATION_USER_NAME;Password=$DB_APPLICATION_PASSWORD" \ --sysctl net. https://echorand.me/posts/building-fedora-vagrant-boxes-for-virtualbox-with-packer/ Wed, 10 Oct 2018 00:00:00 +0000 https://echorand.me/posts/building-fedora-vagrant-boxes-for-virtualbox-with-packer/ In a previous post, I shared that we are going to have Fedora Scientific Vagrant boxes with the upcoming Fedora 29 release. Few weeks back, I wanted to try out a more recent build to script some of the testing I do on Fedora Scientific boxes to make sure that the expected libraries/programs are installed. Unexpectedly, vagrant ssh would not succeed. I filed a issue with rel-eng where I was suggested to see if a package in Fedora Scientific was mucking around with the SSH config. https://echorand.me/posts/sql-server-flyway-non-empty-schema/ Fri, 28 Sep 2018 00:00:00 +0000 https://echorand.me/posts/sql-server-flyway-non-empty-schema/ While attempting to use flyway for SQL server schema migrations, I was consistently getting an error of the form Found non-empty schema xxx ithout metadata table! Use init() or set initOnMigrate to true to initialize the metadata table.. Okay, so easy - let me just delete the tables, drop the schema and we will be good. No luck! Okay, let’s try cleaning it - a destructive process potentially, but the risks were low in my case. https://echorand.me/posts/golang-templates/ Tue, 18 Sep 2018 00:00:00 +0000 https://echorand.me/posts/golang-templates/ While working on creating a template file for a Go project, I wanted to better understand how to work with data in Go templates as available via the html/template package. In this post, I discuss a few use cases that may arise. Accessing a variable Let’s consider our first program: package main import ( "html/template" "log" "os" ) func main() { var names = []string{"Tabby", "Jill"} tmpl := template.New("test") tmpl, err := tmpl. https://echorand.me/posts/curl-download-redirect/ Fri, 14 Sep 2018 00:00:00 +0000 https://echorand.me/posts/curl-download-redirect/ I wanted to download the prometheus binary using curl. My first attempt: $ curl https://github.com/prometheus/prometheus/releases/download/v2.4.0/prometheus-2.4.0.linux-amd64.tar.gz <html><body> You are being <a href="https://github-production-release-asset-2e65be.s3.amazonaws.com/6838921/5c87dc00-b5d1-11e8-8a3a-fd29b54e0c60?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180914%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180914T004135Z&X-Amz-Expires=300&X-Amz-Signature=5a4887918cf75526c045d236dd5a8b22dace657900a1d131a7ffa947be66fc81&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprometheus-2.4.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream">redirected</a>. Sure, that’s a redirect, let’s try: $ curl --location https://github.com/prometheus/prometheus/releases/download/v2.4.0/prometheus-2.4.0.linux-amd64.tar.gz curl: (23) Failed writing body (0 != 16360) What does that mean? I frantically then google, “download prometheus using curl” and hit upon this link. So, the -O option will help: $ curl --remote-name --location https://github.com/prometheus/prometheus/releases/download/v2.4.0/prometheus-2.4.0.linux-amd64.tar.gz <file downloaded> The -O option is equivalent to --remote-name which basically says two things: https://echorand.me/posts/docker-user-namespacing-remap-system-user/ Fri, 24 Aug 2018 00:00:00 +0000 https://echorand.me/posts/docker-user-namespacing-remap-system-user/ In this post, we learn how we can make use of docker’s user namespacing feature on Linux in a CI/build environment to avoid running into permission issues. Using user namespacing also keeping things a bit sane without adopting sub-optimal alternatives. Introduction Let’s consider that we are leveraging docker in a continuous integration (CI)/build environment and the usage scenario looks as follows: CI agent/slave runs as an unpriviliged user agent on the host agent clones the repository during a build on the host The build happens in a docker container spawned by scripts running as agent with the repository volume mounted On a new build, the agent doesn’t do a fresh clone if a clone already exists, but instead does a git clean followed by git fetch of the commit. https://echorand.me/posts/aws-vpc-private-dns/ Wed, 15 Aug 2018 00:00:00 +0000 https://echorand.me/posts/aws-vpc-private-dns/ AWS Route 53 private hosted zones enable you to have private DNS names which only resolve from your VPC. This is great when working from EC2 instances since everything is setup and ready to go. This however becomes a problem when using docker containers on a systemd system. On such a system, systemd-resolved sits in between your host applications and name resolution. The entry in /etc/resolv.conf is basically, 127.0.0.53 which doesn’t mean much when you want name resolution from a docker container which defaults to 8. https://echorand.me/posts/aws-network-acl-docker/ Tue, 14 Aug 2018 00:00:00 +0000 https://echorand.me/posts/aws-network-acl-docker/ In this post, I discuss a problem (and its solution) I encountered while working with AWS (Amazon Web Services) Network ACLs, docker containers and ephemeral port ranges. Infrastructure setup A Linux EC2 instance with docker engine running in a VPC with inbound and outbound traffic controlled by Network ACLs. I was connecting to another hosted service running on a separate VM, service1 running on port 10001 inside the same subnet with security groups allowing traffic from the host IP (via CIDR). https://echorand.me/posts/managing-aws-lambda-functions-terraform/ Thu, 02 Aug 2018 00:00:00 +0000 https://echorand.me/posts/managing-aws-lambda-functions-terraform/ AWS lambda functions look deceptively simple. The devil is in the details though. Once you have written the code and have created a .zip file, there’s a few more steps to go. For starters, we need an IAM profile to be defined with appropriate policies allowing the function to access the AWS resources. To setup the lambda function to be invoked automatically in reaction to another event, we need some more permissions and references to these resources. https://echorand.me/posts/windows-docker-containers/ Thu, 26 Jul 2018 00:00:00 +0000 https://echorand.me/posts/windows-docker-containers/ I went into working with Windows docker containers after having been worked with docker on Linux exclusively. My goal was to have isolated environments for each build in a continuous integration pipeline. That is, each build happens on an exclusive build host (AWS EC2 VM instance) and every database and service the application needs access to for the integration tests (including selenium tets) are run on docker containers on the same host. https://echorand.me/posts/measuring-seconds-elapsed-powershell/ Wed, 25 Jul 2018 00:00:00 +0000 https://echorand.me/posts/measuring-seconds-elapsed-powershell/ I have been working with PowerShell for three months now, and my approach to using it has been pretty much google and trial - copying things, modifying things till they work and learning new things on the way. This post talks about a discovery which I made today. I had a script like this: while ($health -ne 'healthy') { $elapsedTime = $(get-date) - $StartTime Write-Output "--- Waiting for service to become ready - $($elapsedTime. https://echorand.me/posts/terraform-remote-state-consul/ Wed, 27 Jun 2018 00:00:00 +0000 https://echorand.me/posts/terraform-remote-state-consul/ In my new post on the CodeShip blog, I discuss configuring terraform with a consul remote backend. The entire aricle is available here. The accompanying git repository is here. Please file an issue if you have trouble following the setup. https://echorand.me/posts/fedora-29-scientific-rawhide-vagrant-box/ Mon, 18 Jun 2018 00:00:00 +0000 https://echorand.me/posts/fedora-29-scientific-rawhide-vagrant-box/ I am very excited to share that sometime back the Fedora project gave the go ahead on my idea of making Fedora Scientific available as Vagrant boxes starting with Fedora 29. This basically means (I think) that using Fedora Scientific in a virtual machine is even easier. Instead of downloading the ISO and then going through the installation process, you can now basically do: Download Fedora Scientific Vagrant box Initialize VM vagrant up Trying it out before Fedora 29 release As of a few days back, Fedora 29 rawhide vagrant boxes for Fedora Scientific are now being published. https://echorand.me/posts/py-ast-straightplugin-nox/ Sun, 13 May 2018 00:00:00 +0000 https://echorand.me/posts/py-ast-straightplugin-nox/ A few weeks back I wrote a analyser for Apache Thrift IDL in Python. We used it to enforce some code review guidelines. When we hooked it onto arcanist lint engine, we could give feedback to developers at the time they were proposing a code change. The thrift parsing was done using ptsd. The analyser was written as a single file which meant adding new rules meant changing the engine itself. https://echorand.me/posts/video-counter-gauge-oh-my/ Sun, 13 May 2018 00:00:00 +0000 https://echorand.me/posts/video-counter-gauge-oh-my/ The video for my PyCon talk - “Counter, Gauge, Oh My!” is available now. The accompanying git repository is here and has the demos, links to some of my related articles and other resources. If you have any feedback, please let me know. https://echorand.me/posts/pycon-counter-gauge-oh-my/ Wed, 09 May 2018 00:00:00 +0000 https://echorand.me/posts/pycon-counter-gauge-oh-my/ In a couple of days, on Saturday, I will be delivering my PYCON US talk titled Counter, Gauge, Upper 90 - Oh my!. This talk covers my journey into application monitoring in general and Python applications specifically. We will start with looking into the why of monitoring, learn the absolute basics of various monitoring terminologies and use pandas to play with some of the aggregation and summarizing operation usually associated with monitoring. https://echorand.me/posts/fedora-scientific-28/ Wed, 02 May 2018 00:00:00 +0000 https://echorand.me/posts/fedora-scientific-28/ Fedora 28 is out and I am happy to share that after a few releases of no Fedora Scientific, we are back. Thanks to all your emails and enquiring about whether Fedora Scientific is dead. Go Download it now! The Fedora Scientific Guide has been updated a bit as well. Thanks to all the packagers who fixed the issues we were having with some of the relevant packages. https://echorand.me/posts/my-own-ping/ Fri, 27 Apr 2018 00:00:00 +0000 https://echorand.me/posts/my-own-ping/ Introduction The ping program is one of the most common programs which is used to check the “aliveness” of a host and a typical execution looks as follows: $ ping 127.0.0.1 -c 1 -4 PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data. 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.062 ms --- 127.0.0.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.062/0.062/0.062/0.000 ms The -c switch indicates that we want to send a single “probe”. https://echorand.me/posts/cloudflare-dns-over-https/ Tue, 10 Apr 2018 00:00:00 +0000 https://echorand.me/posts/cloudflare-dns-over-https/ Introduction I recently learned about Cloudflare’s 1.1.1.1 DNS service. Let’s explore. Exploration One of the more interesting things that caught my attention there was DNS over HTTPS. That is, we can do this: 22:27 $ http 'https://cloudflare-dns.com/dns-query?ct=application/dns-json&name=echorand. me' HTTP/1.1 200 OK CF-RAY: 409535ca3b3765bd-SYD Connection: keep-alive Content-Length: 281 Content-Type: application/dns-json Date: Tue, 10 Apr 2018 12:27:53 GMT Server: cloudflare-nginx Set-Cookie: __cfduid=dfb12106907c3b55c52b27b8ea99e185a1523363273; expires=Wed, 10-Apr-19 12:27:53 GMT; path=/; domain=.cloudflare-dns.com; HttpOnly; Secure cache-control: max-age=285 { "AD": false, "Answer": [ { "TTL": 285, "data": "192. https://echorand.me/posts/fedora-scientific-28-beta/ Mon, 09 Apr 2018 00:00:00 +0000 https://echorand.me/posts/fedora-scientific-28-beta/ Fedora 28 beta was announced recently and I am happy to say Fedora Scientific is back. One note though, the following applications/libraries are not part of the release: networkx scilab sagemath rkward You can install them on your installation using: $ sudo dnf install python*-networkx scilab sagemath rkward The reason for the above software to be not installed was some failures to install the packages which is why I commented those out so that we can have a release. https://echorand.me/posts/aws-assume-iam-role-from-another/ Tue, 27 Feb 2018 00:00:00 +0000 https://echorand.me/posts/aws-assume-iam-role-from-another/ In this post, we will see how we can implement the AWS assume role functionality which allows an IAM role to be able to obtain temporary credentials to access a resource otherwise only accessible by another IAM role. We will implement the infrastructure changes using Terraform and see how to obtain temporary credentials and access an AWS resource (a S3 bucket) that the corresponding IAM role doesn’t have access to otherwise via the AWS CLI. https://echorand.me/posts/automate-deb-package-packagecloud/ Sat, 24 Feb 2018 00:00:00 +0000 https://echorand.me/posts/automate-deb-package-packagecloud/ In my earlier post, Quick and dirty debian packages for your Go application I shared a recipe building DEB packages for Go applications. We are going to see the following things in this post building upon our recipe in that post: Building the DEB packages in Travis CI Publishing the DEB package to packagecloud.io The primary assumption in my first post was using dep for dependency management. That still holds here. https://echorand.me/posts/python-rlo/ Mon, 19 Feb 2018 00:00:00 +0000 https://echorand.me/posts/python-rlo/ At work, I learned about how Right-to-Left Override was being used to make actually malicious files to look harmless. For example, a .exe file was being made to appear as .doc files. We didn’t want to allow uploading such files. This meant that I nedded to detect the presence of the RLO character in the filename. Then, I came across this post, where I learned about unicode bidirectional class and Python’s bidirectional() method. https://echorand.me/posts/redis-sorted-sets/ Mon, 19 Feb 2018 00:00:00 +0000 https://echorand.me/posts/redis-sorted-sets/ In this post, we will see a demo of sorted sets in redis. I just learned about them and I think they are really cool! This post shows how we can play with sorted sets first via the redis-cli, then from Python and Golang. ┌────────────┐ .───────────────. │ │ .─────────────. ( Redis CLI ) ───▶ │ Redis │ ◀───── ( Golang ) `───────────────' │ │ `─────────────' └────────────┘ ▲ │ │ .─────────────. ( Python ) `─────────────' We will first need a local redis server running. https://echorand.me/posts/python-using-specific-git-commits-of-packages/ Fri, 16 Feb 2018 00:00:00 +0000 https://echorand.me/posts/python-using-specific-git-commits-of-packages/ After a fair number of furious impatient attempts to try and use specific git commits of third party packages in my Python software, I finally have been able to make it work. I went back to the drawing board - basically reading setup-vs-requirement. Applications This is what I did: If you have a setup.py, add the library name to setup.py (abstract dependency) Add the git URL in a requirements.txt file (concrete dependency) When you create your deployment artifact, do pip install -r requirements. https://echorand.me/posts/rest-or-rpc-for-microservices/ Sun, 11 Feb 2018 00:00:00 +0000 https://echorand.me/posts/rest-or-rpc-for-microservices/ The Freelancer.com REST API is powered by a number of backend services. The API itself is powered by a Python HTTP server which communicates with the other services (Python, Golang and PHP) via RPC calls implemented using Apache Thrift. It is only during the past 2.5 years that I have been working with Apache Thrift or cross-language RPCs in general. The question often comes up especially when thinking about the future - why not just use HTTP throughout across all services? https://echorand.me/posts/c_cplus_scientific_programming/ Fri, 26 Jan 2018 00:00:00 +0000 https://echorand.me/posts/c_cplus_scientific_programming/ math.h provides basic mathematical functions as part of the C standard library and are also usable from C++. However, it needs to be supplemented with custom libraries when advanced numerical functionalities are desired. In this article, we shall take a look at two such libraries - the GNU Scientific Library and Blitz++. In the last section of this article, we take a look at Ch - a C/C++ interpreter which combines the power of C/C++ with the ease of use of an interpreter. https://echorand.me/posts/data-in-c/ Fri, 26 Jan 2018 00:00:00 +0000 https://echorand.me/posts/data-in-c/ In C, the data you use in your programs will usually fall into one of the three basic categories: int, char and float. Data in C has no existence without an associated memory location labeled by an identifier, usually referred to as a variable (the term variable is a bit misleading, since it essentially means that it must always vary, but you can have constant variables - i.e. variables whose values do not vary). https://echorand.me/posts/data-in-c-cpython/ Fri, 26 Jan 2018 00:00:00 +0000 https://echorand.me/posts/data-in-c-cpython/ In the last two articles (Data in CPython and Data in C), as it turned out, I discussed two fundamental points in each language: What happens in an assignment operation (such as a=2)? Are separate copies of data created or references passed to the original when they are sent as function parameters? In this article, I will summarize the findings by presenting a comparative analysis. Assignment The simplest way to explain the difference between an assignment operation in C and that in CPython is as follows. https://echorand.me/posts/data-in-cpython/ Fri, 26 Jan 2018 00:00:00 +0000 https://echorand.me/posts/data-in-cpython/ When writing programs in Python (CPython), you have access to data types such as a int, str, tuple, list and a dict. It is fairly obvious what each of these data types would be used to represent: an int data type would represent an integer and a list would represent a list of items - homeogeneous or heterogenous. As opposed to a language like C, the Python compiler automatically decides what type to use for your data without the need to be explicitly specified. https://echorand.me/posts/clean_up_variable_attribute/ Fri, 26 Jan 2018 00:00:00 +0000 https://echorand.me/posts/clean_up_variable_attribute/ GCC's C compiler allows you to define various variable attributes. One of them is the cleanup attribute (which you can also write as __cleanup__) which allows you to define a function to be called when the variable goes out of scope (for example, before returning from a function). This is useful, for example to never forget to close a file or freeing the memory you may have allocated. Next up is a demo example defining this attribute on an integer variable (which obviously has no practical value). https://echorand.me/posts/building-golang-deb-packages/ Thu, 25 Jan 2018 00:00:00 +0000 https://echorand.me/posts/building-golang-deb-packages/ In this post, we will learn about a quick and easy workflow for building and deploying your golang applications as Debian packages. The packages produced will not be official quality DEB packages. Assumptions I have been using dep for dependency management, and I assume that you are doing the same. Other dependency management solutions should work with only the specific bits of the workflow swapped out to suit the one you may be using. https://echorand.me/posts/python-prometheus-monitoring-options/ Wed, 24 Jan 2018 00:00:00 +0000 https://echorand.me/posts/python-prometheus-monitoring-options/ In an earlier article, Monitoring Your Synchronous Python Web Applications Using Prometheus, I discussed a limitation of using the Python client for prometheus. Limitation of native prometheus exporting prometheus was built with single process multi-threaded applications in mind. I use the term multi-threaded here to also include coroutine based concurrent applications such as those written in golang or using Python’s asynchronous primitives (Example: Monitoring Your Asynchronous Python Web Applications Using Prometheus). https://echorand.me/posts/linux-system-mining/ Mon, 22 Jan 2018 00:00:00 +0000 https://echorand.me/posts/linux-system-mining/ In this article, we will explore the Python programming language as a tool to retrieve various information about a system running Linux. Let's get started. Which Python? When I refer to Python, I am referring to CPython 2 (2.7 to be exact). I will mention it explicitly when the same code won't work with CPython 3 (3.3) and provide the alternative code, explaining the differences. Just to make sure that you have CPython installed, type python or python3 from the terminal and you should see the Python prompt displayed in your terminal. https://echorand.me/posts/systemd-supervisord/ Fri, 12 Jan 2018 00:00:00 +0000 https://echorand.me/posts/systemd-supervisord/ If you are running your server applications via supervisord on a Linux distro running systemd, you may find this post useful. Problem Scenario An example scenario to help us establish the utility for this post is as follows: systemd starts the shutdown process systemd stops supervisord supervisord stops your processes You see in-flight requests being dropped Solution What we want to do is prevent in-flight requests being dropped when a system is shutting down as part of a power off cycle (AWS instance termination, for example). https://echorand.me/posts/fedora-openvpn-systemd/ Fri, 12 Jan 2018 00:00:00 +0000 https://echorand.me/posts/fedora-openvpn-systemd/ First, I installed openvpn: $ sudo dnf -y install openvpn Then, I used the following systemd unit file from here to create a systemd service for creating a new VPN connection on Fedora 27: $ cat /etc/systemd/system/openvpn@.service [Unit] Description=OpenVPN service for %I After=syslog.target network-online.target Wants=network-online.target Documentation=man:openvpn(8) Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO [Service] Type=notify PrivateTmp=true WorkingDirectory=/etc/openvpn/client/%i/ ExecStart=/usr/sbin/openvpn --status %t/openvpn-server/status-%i.log --status-version 2 --suppress-timestamps --cipher AES-256-GCM --ncp-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC:BF-CBC --config /etc/openvpn/client/%i/%i.conf CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_DAC_OVERRIDE LimitNPROC=10 DeviceAllow=/dev/null rw DeviceAllow=/dev/net/tun rw ProtectSystem=true ProtectHome=true KillMode=process RestartSec=5s Restart=on-failure [Install] WantedBy=multi-user. https://echorand.me/posts/compilation-interpretation-c-python/ Fri, 05 Jan 2018 00:00:00 +0000 https://echorand.me/posts/compilation-interpretation-c-python/ It is common knowledge that programs written in high level languages have to be translated into a low level language using programs referred to as translators. This low level language is either in a native form, in the sense that it is understood by the operating system itself, or in an intermediate form which is understood by an intermediate program such as the bytecode interpreter. It is also known that C is a compiled language, whereas CPython is both first compiled and then interpreted. https://echorand.me/posts/vim-notes/ Fri, 05 Jan 2018 00:00:00 +0000 https://echorand.me/posts/vim-notes/ I use vim as my occasional text editor, especially when I am working on small projects or just editing a text document. Here’s some notes for myself, mostly created by searching the web and pointing to or copying other people’s answers. Modes Files Opening commands Cursor movement Line search Replacing a word in a block Changing a word at n positions Recording and playing macros Swap two characters Switch case Replace character Move block of text Delete Search Folds Movements Powerless verbs Miscellaneous movements/Scrolling External Resources Modes Insert Normal mode: In insert mode, press Ctrl-o to quickly execute a command and come back to insert mode Files Save current buffer as file. https://echorand.me/posts/tip-terraform-security-group-existing/ Fri, 05 Jan 2018 00:00:00 +0000 https://echorand.me/posts/tip-terraform-security-group-existing/ When using Terraform’s aws_security_group_rule with EC2 classic, you may get an error saying that the source security group doesn’t exist, even though it does. That’s probably because you (like me and others) used the source security group ID and not the security group name, like so: resource "aws_security_group_rule" "my_sg_rule" { type = "ingress" from_port = 11123 to_port = 11123 protocol = "tcp" security_group_id = "${aws_security_group.sg1.id}" source_security_group_id = "${aws_security_group.sg2.id}" } You should actually do this instead: https://echorand.me/posts/mysql-too-many-connection/ Thu, 04 Jan 2018 00:00:00 +0000 https://echorand.me/posts/mysql-too-many-connection/ If you are getting the “Too many connections” error, couple of things worth checking on the MySQL server: mysql> show processlist; .. The above will show the currently open connections. The second is: mysql> show variables like "max_connections"; .. The above will show the configured max_connections allowed. When you get this error, this value is less than or equal to the number of rows returned by show processlist. https://echorand.me/posts/github-pages-custom-domain-travis-ci/ Wed, 03 Jan 2018 00:00:00 +0000 https://echorand.me/posts/github-pages-custom-domain-travis-ci/ As of yesterday, this blog is automatically being published via Travis CI. When I push a new commit to my GitHub repository it triggers a new build in Travis CI. The build completes and the the git repository is then updated with the generated output (mostly HTML with some static CSS). The overall flow looks as follows: This is how I set it all up. Please see issue issue. Blog repository setup I use pelican as my blog engine. https://echorand.me/posts/python-grpc/ Mon, 01 Jan 2018 00:00:00 +0000 https://echorand.me/posts/python-grpc/ My article on getting started with gRPC in Python is up on the codeship blog: Using gRPC in Python I build upon it in the next article: Exploring Security, Metrics, and Error-handling with gRPC in Python The demos can be found in the python-grpc-demo repository. https://echorand.me/posts/imovie-rotate-video/ Mon, 01 Jan 2018 00:00:00 +0000 https://echorand.me/posts/imovie-rotate-video/ I had to rotate a few video frames using iMovie. The instructions for rotating the frames were working, but the rotation was being applied to “neighboring” frames as well. It turns out the trick is to use “split clip” to split a frame so that you can isolate the exact frame to be rotated. And that worked! Thanks to this forum reply. https://echorand.me/posts/consul-tags/ Fri, 01 Dec 2017 00:00:00 +0000 https://echorand.me/posts/consul-tags/ consul allows a service to associate itself with tags. These are arbitrary metadata that can be associated with the service and can be used for different purposes. Below I outline a few examples of making use of tags and discuss some related topics. Use case #1: Dedicated service instances based on requests Let’s say our service is a HTTP server (REST API) acting as a routing point for multiple independent resources with the following service definition: https://echorand.me/posts/docker-extra-hosts/ Sun, 29 Oct 2017 00:00:00 +0000 https://echorand.me/posts/docker-extra-hosts/ Problem Let’s say a program in a container should be able to resolve a custom hostname. Solution When using docker run: $ sudo docker run --add-host myhost.name:127.0.0.1 -ti python bash Unable to find image 'python:latest' locally latest: Pulling from library/python Digest: sha256:eb20fd0c13d2c57fb602572f27f05f7f1e87f606045175c108a7da1af967313e Status: Downloaded newer image for python:latest ... This will show up as an additional entry in the container’s /etc/hosts file: root@fee9aeccbc4b:/# cat /etc/hosts ... 127.0.0.1 myhost.name With docker compose, we can use the extra_hosts key: https://echorand.me/posts/python-prometheus/ Thu, 26 Oct 2017 00:00:00 +0000 https://echorand.me/posts/python-prometheus/ I recently wrote two articles on Python application monitoring with Prometheus: Monitoring Your Synchronous Python Web Applications Using Prometheus Monitoring Your Asynchronous Python Web Applications Using Prometheus The demos can be found in the python-prometheus-demo repository. Also checkout aiohttp-prometheus. https://echorand.me/posts/docker-user-defined-networks/ Thu, 26 Oct 2017 00:00:00 +0000 https://echorand.me/posts/docker-user-defined-networks/ Problem Let’s say a program in a container wants to communicate with a service running in another docker container on the same host. The current recommended approach to do so is using a user-defined network and avoid using links. Solution Create an user-defined network and run both (or as many you have) the containers in this network: (For reference, I am using docker 17.09.0-ce) $ sudo docker network create --driver bridge webapp1 The first container which we will launch in this network is a HTTP server listening on port 8000. https://echorand.me/posts/golang-dissecting-listen-and-serve/ Wed, 26 Apr 2017 00:00:00 +0000 https://echorand.me/posts/golang-dissecting-listen-and-serve/ Welcome to this blog post! Introduction What is DefaultServeMux? Registering handlers The HandleFunc() type Using your own Handler with ListenAndServe() Writing Middleware Rewrapping http.ResponseWriter References Introduction My aim in this post is to discuss three “concepts” in Go that I come across while writing HTTP servers. Through this post, my aim to get rid of my own lack of understanding (at least to a certain degree) about these. Hopefully, it will be of use to others too. https://echorand.me/posts/doing-math-humble-bundle/ Thu, 06 Apr 2017 00:00:00 +0000 https://echorand.me/posts/doing-math-humble-bundle/ Slug:humble-bundle Summary:Humble bundle No Starch Press has launched a "Pay what you want" Python Humble Bundle running from April 5th - April 19th! Your purchases will help support the Python Software Foundation and I am excited to announce that Doing Math with Python is part of it. For 1+ USD you can get three books including "Doing Math with Python"! For 15+ USD you get nine excellent Python books! https://echorand.me/posts/python-zipkin/ Tue, 28 Mar 2017 00:00:00 +0000 https://echorand.me/posts/python-zipkin/ Distributed tracing is the idea of tracing a network request as it travels through your services, as it would be in a microservices based architecture. The primary reason you may want to do is to troubleshoot or monitor the latency of a request as it travels through the different services. In this post we will see a demo of how we can introduce distributed tracing into a Python network stack communicating via HTTP. https://echorand.me/posts/golang-notes-gitbackup/ Sun, 26 Mar 2017 00:00:00 +0000 https://echorand.me/posts/golang-notes-gitbackup/ gitbackup is a tool to backup your git repositories from GitHub and GitLab. I wrote the initial version as a project for a go article which is in review for publication in a Linux magazine. It supports GitHub enterprise installations and custom GitLab installations in addition to repositories on github.com and gitlab.com. It's written in Golang, and built upon go-github and go-gitlab and other third party golang packages. Since the initial version, the project's code has seen number of changes which has been a learning experience for me since I am still fairly new to Go. https://echorand.me/posts/queue-logger-python-json-logging/ Wed, 01 Mar 2017 00:00:00 +0000 https://echorand.me/posts/queue-logger-python-json-logging/ Using QueueLogger with Python JSON Logger When logging from multiple processes (via multiprocessing module), using QueueHandler is one approach with Python 2. QueueHandler however sets exc_info attribute of a LogRecord to None since it is not "pickleable" (more on this later). This becomes a problem when you use python-json-logger to format your logs as JSON since it relies on exc_info being set. The result is you don't get exc_info in your logs. https://echorand.me/posts/golang-setup-1.8/ Wed, 01 Mar 2017 00:00:00 +0000 https://echorand.me/posts/golang-setup-1.8/ This guide will be how I usually setup and get started with Go development environment on Linux. By the end of this document, we will have seen how to: Install the Go 1.8 compiler and other tools (gofmt, for eaxmple), collectively referred to as go tools Install gb and the vendor plugin Vendoring third party packages Installing Go tools We can follow the official install guide to get the latest stable version of the Go tools: https://echorand.me/posts/golang-setup/ Thu, 23 Jun 2016 00:00:00 +0000 https://echorand.me/posts/golang-setup/ This guide will be how I usually setup and get started with Go development environment on Linux. By the end of this document, we will have seen how to: Install the Go compiler and other tools (gofmt, for eaxmple), collectively referred to as go tools Setup Go workspace Working with Go programs using third-party packages Installing Go tools The first option to install the Go compiler and other tools from your distro's package manager. https://echorand.me/posts/slides-jupyter-notebook/ Tue, 31 May 2016 00:00:00 +0000 https://echorand.me/posts/slides-jupyter-notebook/ I presented at the PyCon 2016 Education Summit on "Doing Math with Python" day before yesterday and a lightning talk yesterday. This is the first time, I prepared a slide deck using Jupyter Notebook + Reveal.js. I was pleased with the content creation process and the end result. So, here is what worked for me. Please note that I have basically taken quite a bit of liberty with HTML where I didn't want to search for markdown way of doing something https://echorand.me/posts/tee-in-rust/ Wed, 04 May 2016 00:00:00 +0000 https://echorand.me/posts/tee-in-rust/ My article "Write a program to tee output using Rust" is out in Linux Voice (Issue 27). When I first set out to learning Rust, I really liked how the Rust programming language book took the approach of introducing the language via small projects. In a recent "New Rustacean" bonus episode, Chris Krycho urges "to build anything" in Rust while one is learning the language. If you like both of those things, I think you will find my article useful. https://echorand.me/posts/libcontainer-runc-fedora-24/ Wed, 27 Apr 2016 00:00:00 +0000 https://echorand.me/posts/libcontainer-runc-fedora-24/ In this post, I will post my notes on how I got runC and then using libcontainer on Fedora. The first step is to install golang: $ sudo dnf -y install golang $ go version go version go1.6 linux/amd64 We will set GOPATH=~/golang/ and then do the following: $ mkdir -p ~/golang/github.com/opencontainers $ cd ~/golang/github.com/opencontainers $ git clone https://github.com/opencontainers/runc.git $ cd runc $ sudo dnf -y install libseccomp-devel $ make $ sudo make install At this stage, runc should be installed and ready to use: https://echorand.me/posts/go_image_cropping/ Sun, 17 Apr 2016 00:00:00 +0000 https://echorand.me/posts/go_image_cropping/ My article "Write an Image Cropping with Go" is out in Linux Voice (Issue 26). It is my first article to be published in Linux Voice and I am very excited about it. It is challenging to write an introductory article without being able to explain everything the reader needs to know so as to follow the article, while keeping the article concise. Therefore I have updated the GitHub repo of the final code with a README which points you to various resources and tips I found useful while I was learning myself. https://echorand.me/posts/fedora_koji_livecd_builds/ Wed, 17 Feb 2016 00:00:00 +0000 https://echorand.me/posts/fedora_koji_livecd_builds/ Update: Changed the Koji URL to reflect the change to livemedia As the Fedora Scientific maintainer, I have to make sure I look into whether the nightly build is failing so that I can look into the why. So far I have been doing that by going to the koji url. But I think there is a better way - make a program do it regularly. First, the program (a Python script): https://echorand.me/posts/mocking_s3_with_moto_in_python/ Mon, 25 Jan 2016 00:00:00 +0000 https://echorand.me/posts/mocking_s3_with_moto_in_python/ Let's say you have some Python application code which connects to Amazon S3 which retrieves the keys in a bucket. Very likely, the application would be using boto and the code would like this: import boto def get_s3_conn(): return boto.connect_s3('<aws-access-key', '<aws-secret-key>') def list_keys(): s3_conn = get_s3_conn() b = s3_conn.get_bucket('bucket_name') keys = b.list() return keys The corresponding test would presumably use some mocks and patching. Here is one way to write a test for the above code: https://echorand.me/posts/named_temporary_file/ Wed, 20 Jan 2016 00:00:00 +0000 https://echorand.me/posts/named_temporary_file/ In Python, when you need to create a temporary file with a filename associated to it on disk, NamedTemporaryFile function in the tempfile module is the goto function. Here are some use cases that I think one might use it for. Case #1: You simply need a named empty temporary file You just want a file object (pointing to an empty file) which has a filename associated to it and hence you cannot use a StringIO object: https://echorand.me/posts/mock_and_attribute_error/ Mon, 11 Jan 2016 00:00:00 +0000 https://echorand.me/posts/mock_and_attribute_error/ Updated: Fixed typo in the last paragraph. Today, I was curious to see this behavior of Mock() objects when using mock: >>> from mock import Mock >>> m = Mock() >>> m.i_dont_exist <Mock name='mock.i_dont_exist' id='139841609578768'> >>> m.i_dont_exist() <Mock name='mock.i_dont_exist()' id='139841609106896'> The above is expected, since I have not declared a spec when creating the Mock() object, so even when you call a non-existent method or get/set a non-existent attribute, you will not get a AttributeError. https://echorand.me/posts/data-containers-in-docker/ Sun, 13 Dec 2015 00:00:00 +0000 https://echorand.me/posts/data-containers-in-docker/ In this post, we shall take a look at the idea of data only containers - containers whose sole purpose is to exist on the docker host so that other containers can have portable access to a persistent data volume. Why do we need a persistent data volume? We will experiment with the jenkins image from the docker hub. Let's run a jenkins container using $ sudo docker run -p 8080:8080 jenkins. https://echorand.me/posts/virtualenv-first-python/ Mon, 30 Nov 2015 00:00:00 +0000 https://echorand.me/posts/virtualenv-first-python/ I have until the last few months (of my ~4 years of working with Python) always worked without virtualenv for all my Python projects. Why? I think I found the whole idea of having to do the following two steps before I work on something cumbersome: Remember the exact virtualenv name, and then Activate it That said, I was very much aware that it was certainly a good thing to do and would cause me less headaches someday. https://echorand.me/posts/python-hashseed/ Thu, 19 Nov 2015 00:00:00 +0000 https://echorand.me/posts/python-hashseed/ Recently at work, I wanted to test a string which was being created by the urllib.urlencode() function. My first attempt was simple - test my expected string with that being created by the function above using unittest's assertEquals() function. It passed all the times I ran the tests before I committed the code, but it started failing when the tests were ran as part of the deployment process. The input to the urllib. https://echorand.me/posts/docker-volume-mount-selinux/ Mon, 05 Oct 2015 00:00:00 +0000 https://echorand.me/posts/docker-volume-mount-selinux/ My workflow with docker usually involves volume mounting a host directory so that I can read and write to the host directory from my container as a non-root user. On a Fedora 23 host with SELinux enabled, this is what I have to do differently: Use: -v /var/dir1:var/dir1:Z Note the extra Z above? You can learn more about it this Project Atomic blog post https://echorand.me/posts/puppet-on-fedora-23/ Thu, 01 Oct 2015 00:00:00 +0000 https://echorand.me/posts/puppet-on-fedora-23/ My goal in this post is to show how to setup puppet in standalone mode on a Fedora 23 system. This setup will allow writing puppet modules and then you can apply them on your local system and check manually and via serverspec tests that they are doing what you intend them to be doing. Obviously, a VM would be the best test environment for this, or even a container. https://echorand.me/posts/fedora-22-scientific/ Thu, 11 Jun 2015 00:00:00 +0000 https://echorand.me/posts/fedora-22-scientific/ Fedora 22 was released recently which also means we have a new Fedora Scientific release. Fedora Scientific is now part of Fedora labs. As you can see, the new home for Fedora Scientific looks amazing. The "Featured Applications" section features the most important and useful tools in Fedora Scientific. I think that is a great idea. Everyone associated with it, thank you very much. I have also updated the Fedora Scientific Guide to reflect the changes. https://echorand.me/posts/hello/ Thu, 30 Apr 2015 00:00:00 +0000 https://echorand.me/posts/hello/ I changed the mapping of my domain to point to my GitHub pages. My old blog content is still available here. https://echorand.me/posts/early-stopping-demo-with-neural-network-pytorch/ Mon, 01 Jan 0001 00:00:00 +0000 https://echorand.me/posts/early-stopping-demo-with-neural-network-pytorch/ import torch from torch import nn from torch.utils.data import DataLoader from torchvision import datasets from torchvision.transforms import v2 from early_stopping import EarlyStopping MAX_EPOCHS = 100 training_data = datasets.FashionMNIST( root=“data”, train=True, download=True, transform=v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)]), ) test_data = datasets.FashionMNIST( root=“data”, train=False, download=True, transform=v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)]), ) train_dataloader = DataLoader(training_data, batch_size=64) test_dataloader = DataLoader(test_data, batch_size=64) class NeuralNetwork(nn.Module): def init(self): super().init() self.flatten = nn.Flatten() self.linear_relu_stack = nn.Sequential( nn.Linear(28 * 28, 512), nn.ReLU(), nn.Linear(512, 512), nn.