Ears to the ground…

Intrigued by the current AI trends across the industry, I was eager to try it out for something practical—to get a grasp on the concepts, terminologies, and see if I could leverage open source tools. The icing on the cake, I thought, would be improving on something I'd previously built.

..a simple RAG application

Let me talk about a very simple RAG1 application that I ended-up building. Few years ago, I had played-around with Strava2 APIs to fetch my activity data (from prehistoric era — when I was active) — the whole experience was fun as I’d experimented with a simple service (using (the now defunct) snips.ai , Raspberry Pi, and an Xmini). The application would:

• accept voice input like “how much did I run (/bike) last week (/month/year)”

• summarise my run and biking activities’ totals for a given duration, and then,

• respond to be via audio.

  (Here’s a demo video from back then)

In this new Strava-RAG project, the aim was to cash-in on that past experience with Strava APIs, and see how it can be made better. I got a simple project outline, and capabilities generated using ChatGPT.

Admittedly, since I have a soft corner for PostgreSQL, I was also keen on utilising its vector database3 extension.

The overall system goal was simple, and can be worded as:

A system that can answer question in plain English about my (or any athlete’s) activities like “What year was I the most active in”, or something more complex like “Show me the activities where I performed like my best activity in 2019”, and so on…

The following high-level functionalities were required:

1. Fetch: Authenticate and fetch activities from Strava

2. Persist: Store activities in the PostgreSQL

3. Query: Enable natural language querying on the stored activities using LLM

Let’s dive in!

Fetch: Authenticate, Authorize, and Fetch data from Strava

Strava uses OAuth protocol for authentication, which turned out to be a bit more involved4 than what it sounds in theory, so let me lay it down simply.

Here’s the general flow:

• You register your app with Strava and get a Client ID and Client Secret.

• You direct the user (yourself, in this case) to a special authorization URL that includes your Client ID, requested permissions (scopes), and a redirect URI.

• On approval, Strava redirects back to your specified URI with an authorization code.

• You exchange this code (along with your Client Secret) for an access token and refresh token.

• You use the access token to fetch actual activity data. When it expires (typically after 6 hours), you use the refresh tokento get a new access token without re-authorization.

This boiled down to two basic functions:

# Get the token
get_token(auth_code) -> access_token

# Use token to get activities
get_activities(access_token) -> activities[]

Note that the tokens expire - so this also involves an additional quirk of adding a refresh_token logic to the backend code.

The output of get_activities is an array of activities, which are then dumped to a JSON file.

Persist: Store activities in the PostgreSQL

Once the list of activities have been successfully fetched, the next step is to push them to the database. While doing that we also convert relevant information to embeddings to enable intelligent querying.

Following was the format of (would-be) embedded sentences.

{activity['name']} {activity['type']} {activity['distance']} meters in {activity['elapsed_time']} seconds"

All the activities were converted to the above format, and then I utilised the Python-provided sentence-transformer model to do the string → embedding transformation. For example, a sentence that is formed by replacing the placeholders in the above sentence would be:

morning cycling ride 10034 meters in 1934 seconds

which would then get converted via sentence-transformer model to a 384-dimension vector like so:

[0.034342  0.0534622  .0324592  0.0441267  …] 

each of which gets stored for each activity record.

It’s easy to write a simple iteration to do the same for the entire set of activities, and store it into a DB table. This table would then become our source of truth, and would be kept updated with new activities over time.

With this, I was done with the heart of the application. What remained was to enable querying in natural language, which could then leverage these embeddings.

Query: Enable natural language querying on the stored activities using LLM

I must admit that initially I took this step for granted, naïvely assuming that the LLM will take care of everything. It wasn’t to be.

The platform of choice was Ollama, since I wanted something lightweight, and local. Based on ChatGPT’s recommendation, I went ahead to install Mistral on Ollama. As per the default configuration, the LLM was available via a (local) API call, for basic querying.

Control Flow

The overall control flow can be envisioned as:

• Accept user’s query

• Process the query in the backend and return the query results

  • Encode the query using the sentence-transformer

  • Use the resultant embedding in the similarity-based query on the database

  • Return the query results

  These query results then become the additional (/augmented) context that has to be passed to the (Mistral) LLM

• Pass the additional context, and the base prompt to Mistral. This prompt was of the form:

  You are an assistant which summarizes Strava activity data, based on the retrieved activity information below.
  
  Retrieved Activity Context:
  {additional_context}
  
  Instructions:
  ...
  ... 

The high-level flow can be illustrated like so:

                              ┌─────────────────┐
                              │     USER        │
                              │  Asks Question  │
                              │"Best 2023 runs?"│
                              └─────────┬───────┘
                                        │
                                        ▼
                              ┌─────────────────┐
                              │   WEB FRONTEND  │
                              │ (HTML/CSS/JS)   │
                              └─────────┬───────┘
                                        │ HTTP POST
                                        ▼
                              ┌─────────────────┐
                              │  FLASK API      │
                              │   (Python)      │
                              │ /query endpoint │
                              └─────────┬───────┘
                                        │
                                        ▼
                    ┌─────────────────────────────────────┐
                    │           RAG ENGINE                │
                    │                                     │
                    │  ┌─────────────┐  ┌─────────────┐   │
                    │  │   RETRIEVE  │  │  GENERATE   │   │
                    │  │             │  │             │   │
                    │  │ Find best   │  │   Ollama    |   |
                    │  │             |  |  + Mistral  |   |
                    │  │ matching    │─▶│   (LLM)     │   │
                    │  │ activities  │  │             │   │
                    │  └─────────────┘  └─────────────┘   │
                    └─────────┬───────────────────────────┘
                              │
                              ▼
                    ┌─────────────────┐
                    │   POSTGRESQL    │
                    │   DATABASE      │
                    │                 │
                    │ • Activity data │
                    │ • Vector search │
                    │ • Smart filters │
                    └─────────┬───────┘
                              │
                              ▼
                        ┌─────────────────┐
                        │   RESPONSE      │
                        │                 │
                        │ "Here are your  │
                        │ best runs:      │
                        │ 🏃 Oct 1, 2023  │
                        │ 🏃 Sep 28, 2023"│
                        └─────────┬───────┘
                                  │
                                  ▼
                        ┌─────────────────┐
                        │     USER        │
                        │  Sees Results   │
                        └─────────────────┘

Prompt Refinement

• Was this enough? Nope — the prompt did need refining to handle cases where an any infrequent activity was queried, for example ‘Weight Training’, etc.

• Another refinement needed for the interpretation of ‘Best’. That is, while the ‘Best’ for a given Run/Ride activity would be determined based on Speed, Duration, etc. — the same cannot be said for other types of activities. In the case of ‘Hiking’ or ‘Weight Training’, for example, parameters to determine “Best” are quite different — e.g. ‘Elevation’ in the case of Hiking. Similarly, there were other edge cases that required instruction (prompt) refinement, and logic-based handling!

A Basic UI

Then I went ahead to add a simple, browser-based UI to the application, because until then it was all CLI-driven. This was trivial using Claude which created a basic (HTML/JS) UI, which could be seamlessly integrated to the existing Flask app.

Next steps..

In subsequent posts, I want to build-up on this basic project and to mimic a production system as I think that’s a topic which requires adequate attention. In other words I want to focus on: how does this all come together, and, more importantly, sustain over time, as would in a real-world system!

You can find the above project here.

Anomaly detection is not a novel problem in Data Science and has been in common vocabulary owing to its diverse applicability — in finance (fraud detection) to infrastructure monitoring (outages, cost spikes, ..). Early in my career, I had the opportunity to work in the Knowledge-based Computer Systems group at NCST, Mumbai. Our use case was to monitor quotation outliers in imports, and our systems took hours to analyse few months’ of data before flagging “undervalued” imports.

Two decades later, in a recent hackathon, we ended up processing millions of records in just minutes! Since we were dealing with timed events, and the initial hunch was that a multi-variate1 time-series-based anomaly detection would be a natural fit. However, it wasn’t to be!

To our surprise, common anomaly detection models like Isolation Forest significantly outperformed the time-series one both in training latency and prediction accuracy for multi-variate use cases.

This could of course be subjective, and it seemed that the training latency aspect was influenced by the fact that time-series-anomaly-detection model appeared to rely only on process-level concurrency, which limited its scalability in distributed environments like Spark.

The prediction accuracy (or the lack of it) aspect can be blamed on the data, which, even though was timestamped, but the events were sporadic — sometimes clustered, and on other occasions there was a lull. This was of course not in-line with the frequency that the time-series-based model would have expected.

Time-series-based models typically expect the data to be evenly distributed, and spikes would cause them to fumble. In other words, time-series based detection would work better in situations like sensor-based events, where one would expect a constant timestamped stream of data.

Since the data was granular to the second, clustering of events posed another interesting challenge — that of data loss! This was because in order to leverage the model, an index was needed. In our case, an index that was purely based on the timestamp field would have resulted in last-write-wins scenario because the system considers the last event (or row) it’s processing for the index. This is, of course, a side effect of not maintaining a millisecond or microsecond granularity while storing event timestamps.

To train a model using timestamped events for time-series anomaly detection, an important aspect is to maintain the data at the highest level of temporal granularity, to minimise data loss.

In parallel, we were also exploring other models, like Isolation Forest, LOF, PCA, etc., with various levels of contamination. We observed that Isolation Forest outperformed all other models both during training as well as prediction. For training, the time-series model we used became exponentially slower when the training feature count was increased even by a small number, while retaining the sample size.

Isolation Forest, in Spark environment, not only could leverage the distribution, but does it much more efficiently — and at negligible costs!

Based on these results, our next step would be to further refine the model with some amount of labeled data, allowing us to improve the precision.

Further reading

1. Isolation Forest: Paper

2. Time-series Anomaly Detection: Paper

“how many design patterns are you aware of … oh, those are just J2EE ones, what about the core ones”.

While it may sound superficial, knowledge of design patterns, or even the mere pursuit of it, did encourage thinking beyond the obvious, if I may say so. It nudged the critical thinker in you! Subsequently, enlightened individuals like myself would have appreciated aspects such as DB connection pools, choice of class hierarchy in Java collections, or something as simple as the ‘Undo’ command in any IDE that we were using, a bit better!

Anyhoo, the point of this post is to present an observation that modern day computing — say a typical web-based application development — doesn’t give you as many avenues to acknowledge the design patterns. Hence, I believe, a typical interview question of today, if it ventures into the archaic design patterns (GoF1 or otherwise), would be frowned upon!

Much of this shift can be attributed to the changing paradigms of software development. The introduction of tools and techniques that abstract away the intricate details has, on one hand, accelerated application development significantly, but on the other hand, it has diverted the modern-day developer from the avenues or even the necessity to pursue such knowledge.

However, as old school as I may sound, I still believe that the knowledge and appreciation of GoF patterns, in this age of architectural blueprints and reference architecture, would certainly come in handy in some way or another — be it cloud-native development, or in developing homegrown greenfield app! It may not provide leverage in the interviews — but does give a developer an edge while designing and developing a system, or a part of it!

One of the lessons I learned from a brief stint as a “Cloud Solution Architect” is that it’s easy to get away with murder! And murder being the (anti-)pattern of providing a quick cloud-native ‘solution’, using the latest and greatest (read: what the cloud vendor want you to sell) out there — and then not being there to face the music, when things go awry!
Of course, this varies from one organisation to another — but this is a general pattern observed in organisations that present (or make you perceive) cloud-native solutions as the knights in shining armour for the gullible damsels in distress organisations. The latter belong to two main categories:

1. Technical organisations who were doing great in the past, but their stacks have become stale over time, and hence want to join the cloud bandwagon, and,

2. Non-technical organisations, who have been sold the idea of cloud being a cheap and quick answer to a “happily ever after”!

In both these cases, sadly, by the time they realise that the picture is far less rosy than what it was sold for, it’s already too late! But the original intent of my post was more about the individuals who are bestowed with the responsibility of creating (and selling) their solutions. Often, the people at the receiving end are not savvy enough to understand the common pain points pertaining to cloud-based applications, to name a few:

• the hidden costs (e.g. bandwidth usage, fringe services, storage costs, etc.)

• the cost of maintenance (e.g. upgrading the instance types, vulnerability fixes, etc.)

• other convoluted arrangements — e.g. a little birdie told me about a public cloud vendor trying to sell a Java-based solution, where a simple node.js based application would do the job. Reason being: Java-based solution consume more compute!

So, then, the integrity of the solution architects in-charge becomes crucial! Are they true architects or mere salespersons? Ethics and authenticity define their role beyond mere sales. A few things that could characterise it are:

• seeing the solution through, and then taking a pulse after about three months or so of the solution going into production about how things are

• doing their due diligence before recommending the shiny new service on the cloud to your customers, and,

• asking about the non-functional requirements, and seeing that those are met in the solutions that they present. These should definitely include the running cost of the cloud-based application under peak load.

Now, since 'solution architects' may have limited influence on contracts and other legalities, the organisations engaging these cloud solution providers should ensure that their contracts include clauses expecting the following in a crisp and unambiguous language:

• A post go-live warranty period, and then follow-ups at regular intervals for pulse-checks and course-corrections, if required

• Agreed cost predictions, and then clauses on how much threshold breach would require the cloud solution provider to bear the cost

• Agreed performance expectations, and well-defined SLAs for breaches

Bringing accountability would definitely minimise escape routes and raise the bar expected of cloud solution organisations and indirectly, their architects. This, in turn, would pave the way for more reliable agreements!

Designing Data Intensive Applications (DDIA) is one of the most popular books in the present times1. It’s hardly a surprise that it is so!

Dr. Martin Kleppmann has been that wise man — by the medium of this book — to the many grasshoppers who, like yours truly, have been continuously witnessing the onslaught of technologies, frameworks, and techniques, over the years. While we were mostly awed by the functionality, we did not always get a chance to delve deeper. Even if we did, the knowledge gained was in pockets, and to fluctuating degrees.

It’s of course true, that with experience — active or passive knowledge gain — one does get an inkling of how things work (or would be working) in most of the cases. However, in almost all the cases, the hypothesis does require a validation, or a course-correction, or an “a-ha!” moment.
The beauty of DDIA is that it provides several avenues for the all of those.

Yes, the book doesn’t of course answer all the questions, but one feels good after having gone through a few chapters, mainly because:

• there seems to be, in some sense, a feeling of technical empathy, that a modern day application developer or designer can resonate with, and,

• the language used is easy enough, almost conversational — so the technical terms or jargons, wherever used, are more relatable and easy to consume.

For instance, the explanation of topics like data streaming, and how the usual messaging solution used to facilitate streaming work. The author has ensured that the aspects like event logs are discussed in an apt depth, while talking about technologies that enable event sourcing (e.g. Kafka, etc.).

Such an arrangement enables the reader to continue further research on the topics that she is curious about. It provides her just enough breadth, and a direction, to make judicious technical choices — be it during development, or while deciding on the modern-day software components.

Since this was my first week (first few days, to be precise) at work, and since they have a new location now -- I was apprehensive about the travel time (both to and fro). And, it so happens that TomTom offers pretty useful Traffic APIs -- not just for routing, but for real-time traffic updates as well!

Once this was in place, I used the rest of the space on the display to show the O2H and H2O travel times.

...and here's ze video proof!

Guess what, the humble Pi comes to the rescue here as well!

Something I realised while I was working on this project was that TOTPs are just a result of interplay between a unique seed string (e.g. J8hYUkig78) and your system's/device's time. Basically, that's all there's to it!

Finding the seed!

Usually this seed is embedded within the QR code that these apps want you to scan while setting up the MFA. So, as one can imagine, it's just a matter of interpreting the QR code, to extract this string.

The QR can be opened in the Google Photos app, and an Instagram-ish icon at the bottom invokes Google Lens. It would spit-out something like:

otpauth://totp/someunique%3identification%3string?secret=J8hYUkig78&issuer=AcmeTechCo

As could be evident, we're only interested in the secret from the above. Once you have this seed string, you may want to pat yourself on the back, as you've solved the crux of the issue. Or, not just yet -- the source application would also want you to enter the TOTPs thus generated (using the desired TOTP generation program/app/utility, so as to validate them, and tie them back to your account). So you may have to do that. Once that's done, you're all set!

Now, TOTP generation programs/apps/utilities are aplenty! A simple Google search led me to a Python-based CLI program. The same developer advised to use his updated Go-based utility instead, as it had a better local encryption (note that unencrypted local seeds are potential security risks!), apart from other features. The only thing missing was an ARM-based binary release of the utility (since I wanted to have it running on an RPi), and the developer was gracious enough to accept my contribution for it!

So, once the totp utility was installed, it was just a matter of setting up a new config using the seed, and giving it a name, like so:

totp config add target-app-name J8hYUkig78

(There are of course other ways, so that the secret isn't echoed on the CLI/bash history.)

Once this is in place, TOTPs can be generated using the command:

totp target-app-name

The final step is to show this on a RPi display (Adafruit SSD1306 in my case!). This was simple using the example programs provided by Adafruit. The only modification I did was enable Python-based invocation of the above totp command, and the refresh it every few seconds, to display the updated OTP.

Here's a short clip of the final outcome.

However, Plex seems to have upped the game several notches. Or maybe the people at Plex know how to impress this Netflix-addicted population — the ones who would want to be stream on any. device, support both app and browser based streaming, continue from where they left off, be able to load media directly, share their (in-house) media server with their friends/family, be able to restrict content per account, etc. (I am sure you see what I did there.)

Again, I will abstain from listing down the installation steps for Plex -- there are numerous websites that have those.

Loading the media just requires following a specific and simple directory format. A spare hard disk which could auto-mount could be attached to the RPi for it. Of course, the advantage of using portable media is that one could attach it to any other media source, and directly modify the media to be made available via Plex. Or, for the geeky ones -- a cron job could be written to rsync the media over ssh to this Pi.

There is however, restrictions on the "Free lunch". If one wants to use the Plex client app, one would have to pay a nominal fee. This, of course, enables a host of features not available otherwise.
The "Free" option, of course, is the browser based client, which can do everything that the app can do, albeit at a slightly less of a convenience. I am not complaining, though. ;)

Since my laptop's storage is limited, oftentimes it started complaining, as soon as any space-consuming operation started. At that point, I often had to make some hard life choices! :), you know, of the "shall I keep the big file or zap it!" kinds.

On top of that, I am not sure about others, I have realised that OSX has made working with an external HDD as painful as possible! A FAT32 formatted HDD takes forever to be recognised! To top it all, there's that eternal irk of having to "safely removing the drive". I mean c'mon. Windows has done it -- how long will OSX take??

Anyway, so those were the reasons. But, I guess, the basic reason was: because I wanted to. :-]

Once, your NFS is in the network, on Mac, it's just a few more steps to make your new storage available, and ready to use!

The HDD did need to be formatted, as the original format of FAT32 did not go too well with the Raspberry Pi OS (erstwhile called Raspbian Buster) -- in that the auto-mount used to fail. I formatted it as FAT to keep it OS-neutral, even though there are trade-offs but the benefits outweighed!

So, yes, that it.

Of course, the benefit of network-wide ad-blocking is that it does its job in all of your home devices. If however, your router does not allow configuring a DNS -- then you'd have to configure the DNS on a per-device level. While it may sound painful, but trust me, it's worth it!

Thanks to online advertising -- reading even a simple news article has become painful. While a lot of people use ad-block plugins, these plugins are, limited to browsers. How do you deal with ads on devices where one does not use the browser -- e.g. while playing games, etc? That's where a network-level ad-blocking gets an upper hand!

Okay, I just realised that I haven't talked about why is "RPi" there in this post's title. The thing is, I came across a post on Pi-hole as I was looking for a better (read: any) use for a legacy Pi2 -- which was lying about mostly unused -- thanks to it being trumped by newer Pis that I got later.

Seems being a Pi-hole server is one of the best uses that I could've put it to! The admin console is a rich and responsive UI, which allows your to further tweak the Pi-hole server as per your needs, for example: for explicitly allowing/denying any ad server, blocking specific keywords, etc.

Anyway, as Apache Indian would have put it: 'nuff said! Do go ahead and try it out this amazing project and may you bask in the glory of an ad-free world! And oh, btw, one doesn't really need a RaspberryPi for Pi-hole -- you can potentially install it on anything -- andd...there's a Docker image as well!

I came across this phrase sometime back, and was instantly intrigued by it. So, like any good samaritan, let me share what I could make of it for the larger good of humankind! While I present my assessment, I'll also highlight aspects that make it very viable in most of the computing contexts. We will also delve into how this phrase, at some level, dabbles with the aspects of modern day computing like eventual consistency, the tradeoff between performance and correctness, and Amdahl's law.

First, let's get to understand what the phrase means.

Let me make a claim: We come across apology-based computing in our day-to-day digital ongoings -- be it shopping on e-com websites, chatting using our phone apps, or general browsing.

So what is it?!

It merely points to the fact that in this age of highly distributed systems¹, in a majority of situations, it's ok for messages to be delayed, go undelivered, or, just go awry! Note that messages here imply any sort of communication between two or more systems¹.

If this sounds a bit overbearing and leads to cringing of eyes, you might get want to recall situations in which you had to "refresh the page" in order to see an update, or those times when your text or chat messages did not deliver and a cute little (!) sign appeared beside it, or, (for the technically inclined) you had to explicitly invalidate a cache, so that updated data is reflected quickly. Also, recall the fact that you were sort of okay with it, and in a majority of such scenarios, did not complain.

And, that's precisely it!

Over-time, as the systems have grown, and as the apps have proliferated into almost all aspects of our life -- we're becoming more and more OK, when every once in a while, the systems do not behave as expected. This is not just philosophical -- we humans did not become more patient overnight! Rather, over the years, something interesting has happened -- our response as intermediate or ultimate users of these systems has evolved. So much so, that a seemingly bureaucratic statement that,

It's easier to ask for forgiveness than to get permission.

https://en.wikiquote.org/wiki/Grace_Hopper

has found a benign presence in computing and forgiveness or apology aspects have become the order of the day!

Why did this happen, you ask? Because there's no other option!

Something of this sort is what David Ungar has deliberated upon, and proved in his iconic talk titled "Everything You Know (about Parallel Programming) Is Wrong!" (see embedded video).

In this talk, in the light of the above quote, David Ungar highlights how the bias in computing is leaning (or should lean) more towards something he refers to as "end-to-end nondeterminism", or "race-and-repair", rather than correctness.

Correctness or Determinism comes at a cost, and despite one's best efforts, we are limited by Amdahl's law -- when it's within the confines of a system, and aspects like CAP, and other distributed system vagaries, the moment a process (transaction) crosses the boundaries of one system.

So, what do we do?

Well, distributed systems engineers are well-aware of the phenomenon that,

Failure is a norm rather than an exception.

which, if you come to think of it, is a paradigm shift from the conventional thinking where we, as programmers or system architects, were told to treat it as catastrophic! However, treating failures as a norm in modern-day computing leads us to something very practical -- something we call "designing for failure".
That is to say, we need to build systems with better resiliency, quick failure detection, fault tolerance, and, with CAP in perspective -- be willing to compromise on correctness in favour of availability, by being eventually correct! That pretty much takes care of most of the scenarios. (Of course, we're not talking about mission critical systems or transactions -- where correctness and/or availability, whatever be the cost, is indispensable!)

So, going back to the aforementioned scenarios, what the systems are doing by making us 'refresh' the browser window or by making us log back in, is aligning to the correctness part of the application. The other option, of course, would have been a 503 or something like this, which leads to far more painful memories!

¹Systems = Processes

Although, maybe, I am obligated to be loyal to AWS for various reasons (my work is AWS-centric, plus the fact that it facilitated my foray in the cloud), but still, over the short span of time that I have been in this field -- I am getting increasingly impressed with Microsoft. Their documentation on aspects like cloud design patterns, microservices, etc., is simply impeccable -- to say the least!

Moreover, since I am a huge fan of MOOC websites like Coursera, edX, etc, not very long ago, I came across a very nice course on edX, being offered by Microsoft, called Architecting Distributed Cloud Applications, which, again, I loved!

Apart from cloud-specific courses, they have a lot of other, general CS courses in the offing, which are also pretty good.

Overall, I think MS has come of age under Mr. Nadela, and is delivering how it should. Kudos to the teams, who keep such useful artefacts updated, as well as open to the world.

After all, the age of silos is long gone, and collaboration is the key!

Much of what I write is a compilation of what I found on the Internet, esp. Pēteris Ņikiforovs's post. I am indebted to him, as I heavily benefitted from his post while trying to get WP working. However, since that article is a bit dated, and cannot be used verbatim -- I decided to jot this post down.

It is assumed that one would be having the domain name handy (e.g. myblogsite.com or foobar.io), before proceeding with the installation. In this post, let's go ahead with foobar.io.

So, foobar.io will be hosted on WordPress, with MySQL database, and PHP getting served via ngnix. To enable HTTPS access, we'd use a certificate generated via Letsencrypt.

On AWS, a t2.micro EC2 instance (free-tier eligible) would suffice for this set-up. Note that this instance is free only for 1yr, 750hrs a month, so, it would be a good idea to:

1. Not run any other (EC2) instance in the same AWS account

2. Create an AMI, once the installation is over, so that we can port our installation to a new AWS account after the 1yr period, and not have to do all this over again

Alright, so here we go!

I. Log-in to the instance, and start with creating environment variables, which'd used throughout the installation (Lines to modify are highlighted (optional), except. for line #1 (must-do!))

WP_DOMAIN="foobar.io" ## *** Change this domain *** 
WP_ADMIN_USERNAME="admin"
WP_ADMIN_PASSWORD="admin"
WP_ADMIN_EMAIL="no@spam.org"
WP_DB_NAME="wordpress"
WP_DB_USERNAME="wordpress"
WP_DB_PASSWORD="wordpress"
WP_PATH="/Applications/MAMP/htdocs/wordpress"
MYSQL_ROOT_PASSWORD="root"

II. Ensure that the environment is set (assuming that the above variables are put in a file called setEnv.sh)

source ./setEnv.sh

III. Install the software, viz. nginx, MySQL and PHP (but first remove the default apache server, and upgrade the packages)

sudo apt remove apache2 # we dont want the default server
sudo apt update && sudo apt upgrade -y # lets stay up to date
echo "mysql-server-5.7 mysql-server/root_password password $MYSQL_ROOT_PASSWORD" | sudo debconf-set-selections
echo "mysql-server-5.7 mysql-server/root_password_again password $MYSQL_ROOT_PASSWORD" | sudo debconf-set-selections
sudo apt install -y nginx php php-fpm php-mysql php-curl php-gd mysql-server

IV. Configure MySQL

mysql -u root -p$MYSQL_ROOT_PASSWORD <<EOF
CREATE USER '$WP_DB_USERNAME'@'localhost' IDENTIFIED BY '$WP_DB_PASSWORD';
CREATE DATABASE $WP_DB_NAME;
GRANT ALL ON $WP_DB_NAME.* TO '$WP_DB_USERNAME'@'localhost';
EOF

V. Configure nginx

sudo mkdir -p $WP_PATH/public $WP_PATH/logs
sudo tee /etc/nginx/sites-available/$WP_DOMAIN <<EOF

server {
  listen 80;
  server_name $WP_DOMAIN www.$WP_DOMAIN;
  root $WP_PATH/public;
  index index.php;
  access_log $WP_PATH/logs/access.log;
  error_log $WP_PATH/logs/error.log;
  location / {
   try_files \$uri \$uri/ /index.php?\$args;
  }
 location ~ \.php\$ {
 include snippets/fastcgi-php.conf;
 fastcgi_pass unix:/run/php/php7.2-fpm.sock;
 }
}
EOF

sudo ln -s /etc/nginx/sites-available/$WP_DOMAIN /etc/nginx/sites-enabled/$WP_DOMAIN
sudo systemctl restart nginx # if this fails, check the ngnix logs before continuing

VI. Install Letsencrypt, and auto-install certs (Note that in the DNS config, the domain should already be pointing to the current server public IP, as Letsencrypt would be performing domain validation)

cd ~
sudo apt install letsencrypt # we'll use it later, not right now
git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt
sudo ./letsencrypt-auto --nginx # respond to the prompts after this

VII. Finally, install WordPress

sudo rm -rf $WP_PATH/public/ # !!!
sudo mkdir -p $WP_PATH/public/
sudo chown -R $USER $WP_PATH/public/
cd $WP_PATH/public/

wget https://wordpress.org/latest.tar.gz
tar xf latest.tar.gz --strip-components=1
rm latest.tar.gz

mv wp-config-sample.php wp-config.php
sed -i s/database_name_here/$WP_DB_NAME/ wp-config.php
sed -i s/username_here/$WP_DB_USERNAME/ wp-config.php
sed -i s/password_here/$WP_DB_PASSWORD/ wp-config.php
echo "define('FS_METHOD', 'direct');" >> wp-config.php
sudo chown -R www-data:www-data $WP_PATH/public/

VIII. At this point we should be able to browse to foobar.io, to continue with the WP installation wizard, using the credentials we configured in the first step ($WP_ADMIN_USERNAME, $WP_USERNAME_PASSWORD).

IX. Since Letsencrypt certs expire after a given period of time, we might want to automate the renewal by creating a cron job

sudo tee /etc/cron.daily/letsencrypt <<EOF
letsencrypt renew --agree-tos && systemctl restart nginx
EOF
sudo chmod +x /etc/cron.daily/letsencrypt

That's it! One should be all set now!

Troubleshooting:

If, for any reason, the WP password needs to be reset, the following can be done

mysql -u $WP_DB_USERNAME -p$MYSQL_ROOT_PASSWORD -d$WP_DB_NAME <<EOF
UPDATE `wp_users` SET `user_pass`= MD5('yourpassword') WHERE `user_login`='yourusername';
EOF

In this age of wireless everything, I chose to stay 'wired' for a long time, especially when it came to music. Reason: although I don't (can't) claim to be an audiophile, I do appreciate hi-fidelity (hi-fi) music. Hi-fi audio is soothing even at high amplitudes, and I think good tracks deserve a listening, and not just a hearing! In other words, I am not a .mp3 guy, but more of a .wav (or .flac, if you please) person. Uncompressed/lossless audio rules!

My audio rig is a simple (non-wireless) amp and a pair of monitors, and it pretty much serves my purpose.

The main issue, however, was -- the amp is about 9' (9 ft.) away from my music source(s). Which means that I had to make do with a 10' 3.5mm to RCA audio cable, to stream audio from the laptop, phone, etc.

This, of course, worked like a charm in terms of music quality -- any loss in fidelity was too minor to be noticed -- but this arrangement wasn't very safe. I'd to be careful about myself, and more so for others, to avoid tripping over the lengthy cable that went almost diagonally across the room. (Honestly, I was more concerned about what a human, tripping over it, would take along, since the cable was attached to one of the many precious sources at any given point in time. One can easily infer that the priority wasn't on saving the human in such a scenario.)

Anyway, so a wire-free setup was, maybe not indispensable but good-to-have. I looked into a few options, the cheapest and quite common being a Bluetooth audio receiver. There are many available on Amazon, but from my previous experiences of Bluetooth receivers, I realized that one would have to compromise on sound quality. Now, there might be fancy receivers as well, but I did not want to have to spend a lot.

The rise of the Phoenix...

I did, however, have an AppleTV 2 which was gathering dust, mostly because over time smarter devices/options had replaced whatever little purpose it originally used to serve. The only use left for it was extending/mirroring the Mac screen wirelessly, which isn't really a jaw-dropping feature!

Point being, I was keen on making use of this, mostly useless, AppleTV in the eventual, wire-free setup. From previous experience, and a bit of Googling,  it came down to two options:

1. Use the HDMI output option of AppleTV and use the audio port of the target device

2. Use AppleTV's optical audio port, somehow.

The first option wasn't viable because even if there are devices which are capable of extracting the audio from an HDMI source, for example, display devices like monitors, televisions or projectors -- their sound processing is, as far as I expect, very rudimentary. So, again, there's a compromise on sound quality involved. I also found a bunch of cheap HDMI audio "extractors" -- but they did not look very different from the Bluetooth receivers I talked about earlier.

The second option seemed to be far more popular. If only the amp in question (or one's AV Receiver) had an optical audio in -- I'd have been all set. But there wasn't, and hence, I wasn't.

The process thus needed a "bridge" step -- a bridging gadget was required to accept this optical audio from AppleTV, and then somehow, magically, let me hook up the amp. Enter: Fiio D30K! This nifty little thing does exactly that. In other words, if the amp or AV Receiver has a simple RCA/3.5mm input, this optical-to-whatever converter device would do the trick of accepting the optical audio*, and providing the converted audio as RCA/3.5mm out. There are plenty of similar devices available on Amazon, but one might want to get a decent one. I have been happy with Fiio products over the years, so I went with this one.

Anyway, that pretty much completes the setup! Once this was in place, the audio could then be streamed to my audio rig, wirelessly! Yaay!

Just when you think you're all set...

There was a hitch! It was in the form of flaky audio, especially when the audio file was large. I correctly suspected that it was because now there was too much traffic on the Wifi network (streaming audio + regular Internet use), keeping in perspective the basic (Netgear WGR614) router that I had.

It was time to separate the concerns.

Fortunately, I had a spare Wifi router lying around, in which I set-up another network dedicated to audio streaming. This, however, posed two related challenges:

• How to stream audio from the Internet (as the Internet and audio streaming on were now on two difference WLANs)

• How to connect the laptop (one of the main sources) to more than one (Wifi?) networks.

The resolution was anyone's guess: one of the networks had to be wired!

I chose to make the streaming network wired, as the router was right next to my desk. On my laptop, this enabled me to stay on (Internet) Wifi and streaming network, at the same time. As indicated in the picture, I needed to specify that this (audio streaming ethernet) is NOT the network to seek Internet requests from, and hence I did not specify a DNS for it. [Note: IP address specification as 'manual' doesn't have anything to do with the setup -- it's just there for sentimental reasons :).]

Stirred, but not shaken...

A "few minor" issues still remain, but I am happy with the overall setup now. These "few minor" issues are:

• There's often a few moments of audio lag when playing streaming video -- but that's not any related to this arrangement. I remember observing this lag even in a normal setup

• Streaming from Andriod (or other non-iOS devices) requires special software/apps.

[* FiioD30K accepts coaxial input as well]

Uncle Bob's aptly coined SOLID Design Principles form the basis of a robust software application. Today, I want to talk about one of those principles, the Liskov's Substitution Principle (LSP) because it's easy to deviate from, and a few conscious design choices can prevent us from doing so.

In the simplest terms, LSP suggests that:

Any change that makes a subtype not replaceable for a supertype, should be avoided.

Suppose, we have a class hierarchy like so:

At the first glance, the relationships here seem fine, but if we carry out an IS-A test, the issue becomes obvious: that Tea isn't necessarily a CaffeinatedDrink (for instance: there's decaf!).

Thus, this design violates LSP, because it indicates that all Teas are Caffeinated Drinks. Now, a naïve approach would be to try to retrofit this design, to allow for decaf teas as well -- by adding a flag or suchlike -- but that would be clumsy!

There are several ways to deal with this anomaly, and the decision can be based on the stage of development we're at, along with other factors. So, let's continue with our example and see how can it be dealt with:

1. We know for sure that we'd need to pull Tea out of this hierarchy. Though Coffee looks more justified there, we can pull that out as well, to keep things crisp (and also because someone told you about 'Decaf Coffee' as well!).

   A better option, thus, seems to be:

   • For common behaviour of Teas & Coffees, introduce a Drink type

   • Both Tea and Coffee can then be subtypes of Drink

   • Caffeinated can just be an interface which is implemented as needed

Upon this change, we don't cringe anymore to say that Tea IS-A Drink, with Caffeinated behaviour. Whereas, a DecafTea differs from it. Another perspective could be, Coffee is substitutable both for Drink or Caffeinated, but DecafTea is substitutable ONLY for a Drink.

2. Another approach is to follow Effective Java [Bloch, 2017, Item 18]: Favor composition over inheritance. With this, Drink becomes a member of Tea and Coffee, and Caffeinated (interface) is implemented by all but, say, DecafTea.

Here, we do away with the class hierarchy, and directly use the concrete instances of individual drinks. However, we do keep the Caffeinated behaviour separated, and again, can safely say that Tea/Coffee IS-A Caffeinated drink. Moreover, we're also getting a more robust design because of disallowing (class-based-) inheritance.

How do we ensure we come-up with LSP-compliant design? Well, there are few simple things that can be borne in mind while working on class associations:

• Intuition: Is it sounding right? [Example: Should StudentEnrollment really extend Student, when all it wants is to access some Student properties?]

• Concatenation test: Do the Parent and Child types sound right upon concatenation? [Example: While Flyer+Bird may sound correct, a Flyer+Chicken may not. So does 'Flyer' need to be a class type or an interface type?], and finally and most importantly,

• IS-A test: Is the IS-A condition holding good?

Value Objects (VOs) are prevalent and needed in traditional Java programming. They're almost everywhere --  to hold information within a process, for message-passing, and various other areas.

Apart from having getters and setters for the properties, on several occasions, there's a requirement for these VOs to implement equals() and hashCode(). Developers usually hand-write these methods or use the modern IDE templates to generate them. This works fine initially or until there's a need to update the VOs with one or more additional properties.

With an update, the baggage that comes with new properties includes:

• a new set of getters and setters,

• updates required to equals(), hashCode(), and,

• update required to toString(),if needed

This is, of course, cumbersome, error-prone, and the simple VO soon starts looking like an airplane cockpit!

Google's AutoValue framework is a smart approach to address this issue. With just a couple of annotations, almost all of the "junk" is done away with, and the class becomes smarter -- any future property updates, including getters, setters, as well as equals()*, hashCode()** and toString() are all handled automagically!

The VO then just looks like a basic set of properties of the given type, like so:

import com.google.auto.value.AutoValue;
@AutoValue abstract class CartItem { abstract int itemCode();

abstract int quantity();

abstract int price();

static CartItem create(int itemCode, int quantity, int price) { 
return new AutoValue_CartItem(itemCode, quantity, price); 
} 
} 

Note the default presence of a static factory method create(), as suggested in Effective Java [Bloch, 2017], Item 2.

The use of this annotated VO would be no different from a typical one. For instance, the CartItem defined above would have a simple invocation like this:

@Test
public void create() throws Exception {
    CartItem item1 = CartItem.create(10,33, 12);
    CartItem item2 = CartItem.create(10,33, 12);

assertEquals(item1, item2); // this would be true } Apart from the default support for a static factory, AutoValue also supports Builder classes, within the VOs. Armed with this knowledge, let's take another jab at the example in my previous post on Builders. We continue with the same Cake example and add the required annotations and modifiers. The updated version of the class would then be:

import com.google.auto.value.AutoValue;
@AutoValue abstract class Cake { // Required params abstract int flour(); abstract int bakingPowder();

// Optional params abstract int eggs(); abstract int sugar(); abstract int oil();

static Maker builder(int flourCups, int bkngPwdr) { // return builder instance with defaults for non-required field return new AutoValue_Cake.Builder().flour(flourCups).bakingPowder(bkngPwdr).eggs(0).sugar(0).oil(0); }

@AutoValue.Builder abstract static class Maker { abstract Maker flour(int flourCups); abstract Maker bakingPowder(int bkngPwdr); abstract Maker eggs(int eggCount); abstract Maker sugar(int sugarMg); abstract Maker oil(int oilOz);

abstract Cake build(); } } 

Observe that:

• the member Builder class (named Maker here) just needs to be marked with @AutoValue.Builder annotation, and the framework takes care of everything else

• in the parent class, we could also have had a no-arg builder() method but we specifically want to have only one way of building this class -- with the required params

• as shown above, the optional parameters should be set to their default values since we want the flexibility of choosing only the relevant optional params. [With non-primitive members, @Nullable can be used.]

Just to complete the discussion, here is an example of the ease with which this new builder can be invoked:

@Test
public void makeCakes() {
// Build a cake without oil Cake cakeNoOil = Cake.builder(2, 3).sugar(2).eggs(2).build();

assertNotNull(cakeNoOil);

// Check that it has 0 oil assertEquals(0, cakeNoOil.oil()); // default

// Make cake with oil Cake cakeWOil = Cake.builder(2, 3).sugar(2).oil(1).eggs(2).build();

// Obviously, both the cakes are different assertNotEquals(cakeNoOil, cakeWOil); // valid

// Another cake that's same as cake w/ oil Cake anotherCakeWOil = Cake.builder(2, 3).sugar(2).oil(1) .eggs(2).build();

assertEquals(cakeWOil, anotherCakeWOil); // valid 
} 

There are many other fine-grained things that can be done while using AutoValue, like specifying getters for specific properties or customizing toString(), etc.

It's impressive how AutoValue facilitates writing and static factory methods and builders quickly -- taking the headache out of defining and updating VOs.

[Full implementation of the abovementioned example is here.]

Further reading:

1. AutoValue with Builders

2. Project Lombok also addresses the VO update issue, along with other things

* Effective Java [Bloch, 2017], Item 10 ** Effective Java [Bloch, 2017], Item 11

When I was writing the last post, I realized how much I used to be in awe of Effective Java [Bloch, 2017]. It was a book that covered what no other did. It wasn't just coding -- there're plenty of books where one could learn "Your first Java Program" and beyond, and throw an air punch. It wasn't also about language syntax and semantics -- Java Complete Reference [Schildt, 2017] fitted the bill there*, or OOPs (every other Java book started with OOPs concepts). Rather, Effective Java covered the basics of writing elegant Java code, and as a by-product, also underlined how easy was it to be swayed away by 'convention'. I wouldn't recommend it as a Java learner's first book. But it should very well be one's second Java book and the one that she/he would keep revisiting throughout the programming career.

Ever since I've picked it up again, it's become tough to keep it aside. With each of its topic, I realize, over time, how much have I drifted away from the delight of writing good code, and how much do I need to still learn. Go read it now if you haven't had a chance yet. What's more, the much-awaited 3/E, which covers Java 7, 8 and 9, is out now!

While on this topic, let me talk about another one of my favourite books on Java -- Thinking in Java [Eckel, 1998].

This is the book which I considered as a Java developer's Bible at one point in time. Since there are no new editions after the 4/E, the syntactical parts might be a bit obsolete now. But still, in my opinion, it's the best book to get one's (Java & OOPS) fundamentals in place.

* I have always found Java Complete Reference a bit too elaborate to my liking. Most of which is about the language syntax and semantics. All of that might have been useful in the early days of the Internet when it wasn't that easy to look up things online. But I doubt if that's needed now.

Paraphrasing Josh Bloch in Effective Java [Bloch, 2017, Item 2]:

While creating objects, in cases where the number of optional parameters of an object is considerable, say 4 or more, one might think of static factory methods [Bloch, 2017, Item 1] as a solution -- but they're more suitable for a small set of parameters. When there are several optional params, static factories cannot be used as it's cumbersome to imagine and cater to all possible parameter combinations. Another approach that's proposed in such cases is using JavaBeans but it has its own shortcomings.

Therefore, we usually go with multiple (telescoping) constructors for such requirements. For example:

public Cake(int oilTbsp, int flourMg){
  this(oilTbsp, flourMg, 0);
}
public Cake(int oilTbsp, int flourMg, int eggCount){ this(oilTbsp, flourMg, eggCount, 0); }

public Cake(int oilTbsp, int flourMg, int eggCount, int bakingPowderMg){ this(oilTbsp, flourMg, eggCount, bakingPowederMg); }

//...

Such implementations, although purpose-serving, are a bit contrived in that the class client needs to tally the parameters accurately. Consider a large parameter list, and this would be an overkill.

A variation of Builder pattern [Gamma, 1995], is what Bloch suggests, for such cases. In it, a builder class is a static member of the class it builds, for example:

public class Cake{
  //...
  private Cake(Builder builder){
    //...
  }

public static class Builder{ //... } } Since the original constructor is hidden, the client first gets a reference to the Builder class -- passing all the required params to its constructor or static factory. The client then calls setters on the returned builder object to set the optional parameters of interest. Finally, the client makes a call to the build() method to generate an immutable object. Since the builder setter methods return the builder itself, the invocations can be chained, like so:

// Set only the parameters of interest
Cake cake = new Cake.Builder(350, 45).egg(2).sugar(240).cocoa(35)...build();

As is apparent, this is intuitive as well as concise.

A builder can be further enhanced by enabling it to build more than one object, based on parameters. One has to be cautious, however, to disallow building an object of an inconsistent state. This can be ensured by validating the passed parameters as early as possible and throwing a suitable exception.

Builders can also be used to automate certain tasks and fill in the fields. For example, autoincrementing the object Id, etc.

As Josh Bloch advises, we should be using Builders as often as possible, especially in cases where the number of parameters is significant. They're a simple and elegant alternative to telescoping constructors or JavaBeans.

[Full implementation of the Cake builder example is here.]

A detailed post would follow. (And yes, as mentioned in the video description, kindly ignore the choice of LED colours :)).