Nabeel Valley

Looky, a wild HTML!

Sat, 12 Oct 2019 00:00:00 GMT

Looky, a wild HTML!

Here at last

So, after far too long, we are here. A mostly-functional website, that I have not tested on a single iOS device which may prove to be problematic

Note to self, test on an iPhone or something

For the past six months or so I've looked at myself with a full stack of irony knowing that I, a web developer, do not have my own website. Or well, not one I'd actually want anyone to see (*cough cough* I'm talking to you, Weebly)

The Design Process

Well, the design process, especially when you're trying to satisfy your unattainable standard of perfection, can be difficult. I actually ran through a shocking number of iterations and end-to-end redesigns but I just could not seem to find something that satisfied me. Until one day, it all just sort of hit me at once

None of this nonsensical "cards and shadows" drama, no random floopy animations, none of that, just a blank slate, with some text and imagery.

I think I have a fair stock of those

Design v1

The first iteration that I have saved is this one, I had big dreams, there were going to be things moving all over the place, I was going to break the norm with horizontal scrolling, and just basically turn the site into a slide-show. But I didn't like the lack of fluidity with the overall design. Regardless, here are the first few screens:

During this initial phase I had sort figured out what my key goals were:

Unique experience on every device.
Unobtrusive elements
Clean lines
I needed to take emphasis away from the design, and onto the content

Basically, I wanted something basic

By now I had decided on the three most important things, #E44D90, Montserrat, and Nova Mono, but there's still a lot I was unsure of

Design v2

This version mostly morphed out of the first one, I was trying to address two things primarily

How can I not confuse people
Actually yeah, just the first thing

Personally, I hate this design. But It was definitely valuable in telling me what I didn't want this site to be

I've always disliked cards, and abusing it the way I was in that further deepened my hate. I felt kind of like "Well if everyone else does it, maybe I should give it a shot?", to be fair it's probably not the cards, it's me

Design v3

This is what you're seeing now (Assuming you're reading this in October 2019) I love how this design places a strong emphasis on the imagery and text, there isn't really much else in the way of it aside from occasional underline. Simplicity

I addressed my remaining goals by sticking to a basic site layout with a fairly aggressive grid to deliver interesting layouts for different devices

That being said, and my heart being content. It was time to code

The "Bashing your head against a wall" Process

I used React - Why? Because I used React. Sure, I could have went the Server-Side route and made sure I was sending normal HTML to people, but I wanted a sustainable component library that I could develop as well as super-quick responsiveness on the client, and from experience, I knew I could knock this out fastest with React. That being said, I kind of wish I had looked at setting up pre-rendering before I started, because I feel like it's going to be admin to add after (will basically need to refactor out all the <Suspense/>) tags

Now, as any front-end developer should know in 2019, you gotta start with npx create-react-app my-app and after about three years of node_modules you're ready to go

Content Management System?

I'm not going to go into the details of the development because there isn't really anything too fancy going on here - the site is mostly static anyway. I will however just talk about the Content Management System

Well ..., I think it depends on your view of a CMS. I really didn't want to go overkill on this so I'm making use of a fairly simple system. Each post consists of a post.md and a post.json file

The .md file is the Markdown file, it's the file that I write in as I write the post, and potentially every post in future

The .json file is also very simple at the moment and contains some metadata for the post such as the title and the date. I made this separation so that I would not have to get funky with my file processing, and I'm fairly happy with it

To process and render the Markdown I'm using a library that I fell in love with just over a year ago called Showdown.js, and to add syntax highlighting I'm using some flavour of Highlight.js

I've then simply rendered the content by fetching it from the Folder-cms and passing it through a Markdown Component

import React from 'react'
import { Converter } from 'showdown'
import * as showdownHighlighter from 'showdown-highlight'

import './Markdown.css'

const convertMarkdownToHtml = (text) => {
  const converter = new Converter({
    headerLevelStart: 2,
    parseImgDimensions: true,
    extensions: [showdownHighlighter],
  })
  const html = converter.makeHtml(text)
  return html
}
const Markdown = ({ text }) => (
  <div
    className="Markdown"
    dangerouslySetInnerHTML={{ __html: convertMarkdownToHtml(text) }}
  ></div>
)

export default Markdown

And that's pretty much it. It gets rendered into the page you're looking at now and that's all it takes really. It's a bit of a shame that my syntax highlighting CSS theme doesn't really do justice on JS, another TODO I suppose

Conclusion

There were a couple of new things I learnt during this entire process, simple little things really like how to change the colour of the scrolly thing on your browser, or how to put together a mildly-reasonable file system to store the posts, but more than anything I enjoyed being able to just take something from pure concept to a final build that lines up pretty well with my initial vision

That's all for now, stay tuned, lots more to come. But in the mean time take a look around, let me know what bugs you find (as I said, very untested)

Nabeel Valley

Real-time Communication with MQTT

Tue, 12 Nov 2019 00:00:00 GMT

Real-time Communication with MQTT

Overview

MQTT is an open standard for communication and is especially useful for communication between IoT devices due to its low bandwidth requirements

Today we're going to be taking a look at using MQTT to easily communicate between multiple applications. We'll look at using a static website and Eclipse Mosquitto as a message broker to enable communication between two instances of the application, but the same principles can be extended to other programming languages and MQTT clients. For our clients we'll be using MQTT.js which runs in the browser via CDN and we interact with it using JavaScript, but can also be added from npm

If you'd like to follow along with the completed code you can get it here

Broker

Eclipse Mosquitto is an open-source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 5.0, 3.1.1 and 3.1. Mosquitto is lightweight and is suitable for use on all devices from low power single board computers to full servers. (Eclipse Mosquitto )

Message brokers are a method of intra-application communication, MQTT makes use of a publish/subscribe model for application communication. In this model different applications, or clients, can publish messages to a topic which can then be picked up and operated on by any other applications that are subscribed to the topic

Setting Up the Broker

You will first need to download Mosquitto from here , all 1.5mb of it, and run through the installer and then ensure that mosquitto is in your path. If you need to know how to add something to your path take a look at this Gist . For a Windows 64 bit installation of Mosquitto you should add the following to your path C:\Program Files\mosquitto

The MQTT installation should include the following three commands

mosquitto - This runs the MQTT server/broker
mosquitto_pub - A simple message publisher
mosquitto_sub - A simple message subscriber

We can get more information by running either of the above commands with the --help flag, e.g. mosquitto --help

Basic Messaging

Let's start the message broker in verbose mode by opening a new shell and running the mosquitto -v command, you should see some output like the following:

1573494107: mosquitto version 1.6.7 starting
1573494107: Using default config.
1573494107: Opening ipv6 listen socket on port 1883.
1573494107: Opening ipv4 listen socket on port 1883.

We can see from the above that the message broker is running on our local port 1883

In a new shell, we can create a client which is subscribed to a topic. We'll name this topic messages but it can be pretty much anything you want, to start the subscriber client we can run mosquitto_sub -t "messages" -v, you won't see any output in the subscriber shell as yet, but looking at the broker shell you should logging which says that a client was connected

1573494642: New connection from ::1 on port 1883.
1573494642: New client connected from ::1 as mosq-fwbsJxdXOnQW0LOaIn (p2, c1, k60).
1573494642: No will message specified.
1573494642: Sending CONNACK to mosq-fwbsJxdXOnQW0LOaIn (0, 0)
1573494642: Received SUBSCRIBE from mosq-fwbsJxdXOnQW0LOaIn
1573494642:     messages (QoS 0)
1573494642: mosq-fwbsJxdXOnQW0LOaIn 0 messages
1573494642: Sending SUBACK to mosq-fwbsJxdXOnQW0LOaIn
1573494702: Received PINGREQ from mosq-fwbsJxdXOnQW0LOaIn
1573494702: Sending PINGRESP to mosq-fwbsJxdXOnQW0LOaIn

To publish a message open another shell and run mosquitto_pub -t "messages" -m "This is my message!", in our subscriber shell we should see the following output:

messages This is my message!

And in the broker we'll see the following:

1573494844: New connection from ::1 on port 1883.
1573494844: New client connected from ::1 as mosq-MWbaa2TpZGV0FrTmcF (p2, c1, k60).
1573494844: No will message specified.
1573494844: Sending CONNACK to mosq-MWbaa2TpZGV0FrTmcF (0, 0)
1573494844: Received PUBLISH from mosq-MWbaa2TpZGV0FrTmcF (d0, q0, r0, m0, 'messages', ... (19 bytes))
1573494844: Sending PUBLISH to mosq-fwbsJxdXOnQW0LOaIn (d0, q0, r0, m0, 'messages', ... (19 bytes))
1573494844: Received DISCONNECT from mosq-MWbaa2TpZGV0FrTmcF
1573494844: Client mosq-MWbaa2TpZGV0FrTmcF disconnected.

When you're done with this you can close the open shell windows

Broker with WebSockets

For our usecase, we will make use of WebSockets so we can communicate directly from the browser. When starting up the message broker we have the option to pass in a configuration file. Let's create one that states the port and the protocol we would like to use. Create a new directory called mqtt, and in it create a new file called mosquitto.conf and place the following contents which simply tell it to listen on port 9001 and use the websockets protocol

mosquitto.conf

listener 9001
protocol websockets

If you'd like to know what else can be done in the configuration file you can read the documentation

We can start the broker again from the mqtt directory using mosquitto -c mosquitto.conf -v, this time we should see that it is running on 9001 with websockets

1573499281: mosquitto version 1.6.7 starting
1573499281: Config loaded from mosquitto.conf.
1573499281: Opening websockets listen socket on port 9001.

Now that we have configured the broker to use Web Sockets we can start connecting to it from a web page and publish some messages

Client

We'll be using MQTT.js to connect to our message broker. MQTT.js is a simple MQTT Client Library for connecting to message brokers, we'll be using it via CDN for the sake of simplicity and will also be using Materialize for our CSS because I'm not going to write a bunch of CSS and I don't want this to look completely terrible

HTML

Let's first setup the index.html. It consists of a two-column layout with the following:

Main Heading with a badge to indicate if we successfully connected to the broker
Inputs to Publish Message
Table of all received messages
Materialize and MQTT.js files via CDN
Script tag for all our JavaScript, because we don't need two files for this

index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <meta http-equiv="X-UA-Compatible" content="ie=edge" />
    <title>Real-time Communication with MQTT</title>

    <!-- MQTT.js -->
    <script src="https://unpkg.com/mqtt@3.0.0/dist/mqtt.min.js"></script>

    <!-- Materialize -->
    <!-- Compiled and minified CSS -->
    <link
      rel="stylesheet"
      href="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/css/materialize.min.css"
    />
  </head>

  <body>
    <div class="container">
      <!-- Page Content goes here -->
      <div class="row">
        <div class="col s12">
          <h1>
            Messages
            <span
              id="connect-badge"
              class="new badge orange"
              data-badge-caption=""
              >connecting</span
            >
          </h1>
        </div>
      </div>
      <div class="row">
        <div class="col s12 m6">
          <h2>Input</h2>
          <div class="input-field">
            <input id="name" type="text" class="validate" />
            <label class="active" for="name">First Name</label>
          </div>

          <div class="input-field">
            <textarea
              id="message"
              class="materialize-textarea"
              data-length="120"
            ></textarea>
            <label for="message">Message</label>
          </div>

          <button class="waves-effect waves-light btn" onclick="handleSubmit()">
            Publish Message
          </button>
        </div>
        <div class="col s12 m6">
          <h2>Output</h2>
          <table>
            <thead>
              <tr>
                <th>Name</th>
                <th>Message</th>
              </tr>
            </thead>
            <tbody id="messages"></tbody>
          </table>
        </div>
      </div>
    </div>

    <script>
      // Our Javascript will go here
    </script>
  </body>

  <!-- Materialize -->
  <!-- Compiled and minified JavaScript -->
  <script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/js/materialize.min.js"></script>
</html>

Running the Website

Since we are including resources from other domains your browser may have some issues with you using the 'double click' method of running the files, so we're going to need to use a web server

A list of quick one-line web servers can be found on this Gist and if you're using Visual Studio Code you can use the Live Server Extension

Connecting to the Message Broker

Now that we're up and running we can connect to the broker that we set up previously. The Mosquitto Broker should be running on 127.0.0.1:9001, using the mqtt object that is exposed in the global space by MQTT.js we can simply create an instance of the client in our <script> section with:

const client = mqtt.connect('mqtt://127.0.0.1:9001')

This gives us a client object that is the MQTT Client connected to our broker. Next, we need to set a handler for the connect event which is when the client successfully connects to the message broker, we do this with the following code:

client.on('connect', function () {
  client.subscribe('messages', function (err) {
    if (!err) {
      const badge = document.getElementById('connect-badge')
      badge.innerText = 'connected'
      badge.classList.remove('orange')
      badge.classList.add('green')
    }
  })
})

You can see that the client.on function takes a callback, in which we tell our client to subscribe to the messages topic as we did previously via the mosquitto_sub, and then once it has successfully subscribed we change the badge status to indicate that we are connected

Since we have successfully subscribed the client to the broker we should be able to publish messages. We'll do that using the data from the input fields in on the page and the Publish Message button's handler

Publishing Messages

Create a handler for the Publish Message button called handleSubmit, we've already referenced this in our button's HTML. The handler simply needs to read the values from the input fields and call client.publish with a stringified form of the data

function handleSubmit() {
  const nameInput = document.getElementById('name')
  const messageInput = document.getElementById('message')

  const data = {
    name: nameInput.value,
    message: messageInput.value,
  }

  const payload = JSON.stringify(data)

  client.publish('messages', payload)
}

The client.publish function above takes in a topic and message as its inputs, in our case we want to publish to the messages topic and we are publishing the JSON object as a string

Refreshing the page, entering some text into the fields and clicking the Publish Message button should publish the message to the broker, we should see this in the message broker's output

1573506416: Received PUBLISH from mqttjs_d5cdbc8e (d0, q0, r0, m0, 'messages', ... (34 bytes))
1573506416: Sending PUBLISH to mqttjs_d5cdbc8e (d0, q0, r0, m0, 'messages', ... (34 bytes))

Now that we can publish messages to the topic, we will need to create a handler for the message event which is triggered every time one of our subscribed topics has something published to it. We can do this again with the client.on function:

client.on('message', function (topic, message) {
  const data = JSON.parse(message)

  document.getElementById('messages').innerHTML += `<tr><td>${
    data.name
  }</td><td>${data.message.replace(/\n/g, '<br>')}</td></tr>`
})

Our message handler receives the topic, which in our case will always be messages and the message which is a buffer of the data published to the topic, this will be the same data that we published using the client.publish function, this time we read the JSON back into an object and add it to the messages table as HTML

You should be able to open a few different windows of the web page and they should all be able to Publish Messages to one another

What Next?

If you're not familiar with message brokers this could be a lot to take in and that's okay. I think playing around with the mosquitto_pub and mosquitto_sub tools can be helpful to get a more interactive feel for the way everything works. You can also just play with the MQTT.js code we used via your browser console and looking at the documentation on the website

If you want to take a look at a free instance of an MQTT Broker that you can play around with take a look at Cloud MQTT , and if you want to look at how you can take the concepts covered here today and play around with some IoT concepts as well then it may also be worth taking a look at this IBM Developer Tutorial using IBM Cloud and Watson IoT

If you're interested in spinning up your own MQTT broker with custom functionality you can take a look at Mosca or get an even deeper understanding of the MQTT Protocol on IBM Developer

Conclusion

We've used a couple of different technologies and depending on your background you may not be familiar with everything here. The best thing I'd say to get to understand what we've covered would be to play around with the code as well as look at some of the other services and use-cases for message brokers and get a feel for what's out there

Good luck

Nabeel Valley

Intro to F# Web APIs

Wed, 30 Oct 2019 00:00:00 GMT

Intro to F# Web APIs

So we're going to be taking a bit of a look on how you can go about building your first F# Web API using .NET Core. I'm going to cover a lot of the basics, a lot of which should be familiar to anyone who has worked with .NET Web Applications and F# in general.

Along the way I'm also going to go through some important concepts that I feel are maybe not that clear from a documentation perspective that are actually super relevant to using this F# in a real-life context

If you're totally new to F# though you may want to take a look at F# for Fun and Profit or my personal quick reference documentation over here

Getting Started

Assuming you've got the .NET Core SDK with F# installed, you can simply create a new project with the following:

dotnet new webapi --language F# --name FSharpWebApi

code .\FSharpWebApi

Alternatively, if you're feeling a little unexperimental you can use the Visual Studio project creation wizard, psshhtt

Once you have the project open you can run the following command to launch the application:

dotnet run

Which should start the application on https://localhost:5001 and http://localhost:5000, you can see the current existing endpoint at /weatherforecast, this is handled by the Controllers/WeatherForecastController.fs file

Looking Around

Looking at the structure of the project files you should see the following:

FSharpWebApi
│   appsettings.Development.json
│   appsettings.json
│   FSharpWebApi.fsproj
│   Program.fs
│   Startup.fs
│   WeatherForecast.fs
│
├───Controllers
│       WeatherForecastController.fs
│
└───Properties
        launchSettings.json

So, mostly we see the typical Web API stuff that we'd expect for a C# project such as the startup and program files. In F# they serve pretty much the same purpose.

Looking at the Program.fs file we can see that it contains the main function and configures the Web Host, next we can see that the Startup.fs file contains the usual configuration methods. We should note that the method calls within these functions are piped to an ignore so the the functions to not return their respective Builders as this will break the application

The Program.fs and Startup.fs files can be seen below

Program.fs

namespace FSharpWebApi

module Program =
    let exitCode = 0

    let CreateHostBuilder args =
        Host.CreateDefaultBuilder(args)
            .ConfigureWebHostDefaults(fun webBuilder ->
                webBuilder.UseStartup<Startup>() |> ignore
            )

    [<EntryPoint>]
    let main args =
        CreateHostBuilder(args).Build().Run()

        exitCode

Startup.fs

namespace FSharpWebApi

type Startup private () =
    new (configuration: IConfiguration) as this =
        Startup() then
        this.Configuration <- configuration

    // This method gets called by the runtime. Use this method to add services to the container.
    member this.ConfigureServices(services: IServiceCollection) =
        // Add framework services.
        services.AddControllers() |> ignore

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    member this.Configure(app: IApplicationBuilder, env: IWebHostEnvironment) =
        if (env.IsDevelopment()) then
            app.UseDeveloperExceptionPage() |> ignore

        app.UseHttpsRedirection() |> ignore
        app.UseRouting() |> ignore

        app.UseAuthorization() |> ignore

        app.UseEndpoints(fun endpoints ->
            endpoints.MapControllers() |> ignore
            ) |> ignore

    member val Configuration : IConfiguration = null with get, set

Next we have the FSharpWebApi.fsproj file which contains references to the relevant code files. It's important to note that the order of the files in the ItemGroup specifies the order that files depend on each other. Lower files depend on files higher up

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="WeatherForecast.fs" />
    <Compile Include="Controllers/WeatherForecastController.fs" />
    <Compile Include="Startup.fs" />
    <Compile Include="Program.fs" />
  </ItemGroup>

</Project>

Lastly, we have a controller that resides in the Controllers/WeatherForecastController.fs with its types defined in the WeatherForecast.fs file. Looking at the WeatherForecast.fs file we can see that the type has a few simple properties and one function

WeatherForecast.fs

namespace FSharpWebApi

open System

type WeatherForecast =
    { Date: DateTime
      TemperatureC: int
      Summary: string }

    member this.TemperatureF =
        32 + (int (float this.TemperatureC / 0.5556))

Next up, we can see the controller which contains a single GET endpoint which delivers a random array of weather forecasts. Here we can see a few different things. First, the namespace is FSharpWebApi.Controllers, this pretty much follows the .NET standard of the Namespace being related to the Folder name, we can also see the ApiController attribute that adds some useful functionality for basic API handling and the Route attribute that states the controller route

The WeatherForecastController type defines the controller and that it inherits from ControllerBase, additionally the constructor requires the ILogger service which will be provided by DependencyInjection

Lastly, looking at the __Get method we can see the HttpGet attribute that specifies that this is a Get Method, and the __ shows that we don't care about references to the function's this, and the return type for the function is an array of WeatherForecast

WeatherForecastController.fs

namespace FSharpWebApi.Controllers

open System
open Microsoft.AspNetCore.Mvc
open Microsoft.Extensions.Logging
open FSharpWebApi

[<ApiController>]
[<Route("[controller]")>]
type WeatherForecastController (logger : ILogger<WeatherForecastController>) =
    inherit ControllerBase()

    let summaries = [| "Freezing"; "Bracing"; "Chilly"; "Cool"; "Mild"; "Warm"; "Balmy"; "Hot"; "Sweltering"; "Scorching" |]

    [<HttpGet>]
    member __.Get() : WeatherForecast[] =
        let rng = System.Random()
        [|
            for index in 0..4 ->
                { Date = DateTime.Now.AddDays(float index)
                  TemperatureC = rng.Next(-20,55)
                  Summary = summaries.[rng.Next(summaries.Length)] }
        |]

Creating a Controller

Creating a new controller is not particularly complex given that we have the above as a starting point.

Get Handler

We're going to create a handler that is able to return a simple message for an even param, and a 404 for a odd param in order to look at how we can return actual response codes in cases where we aren't always able to return something of a constant type

First, we can create a Controllers/MessageController.fs file with just some basic scaffolding to start with. We'll define a Get controller that just returns the id it receives as a route param multiplied by two if the the result shouldDouble param is set to true. Additionally we can see the sprint function used to format the output as a string

Before we can add the data to the actual controller we need to add the <Compile Include="Controllers/MessageController.fs" /> to the ItemGroup in the FSharpWebApi.fsproj file, :

FSharpWebApi.fsproj

  <ItemGroup>
    <Compile Include="WeatherForecast.fs" />
    <Compile Include="Controllers/WeatherForecastController.fs" />
    <Compile Include="Controllers/MessageController.fs" />
    <Compile Include="Startup.fs" />
    <Compile Include="Program.fs" />
  </ItemGroup>

And then we can put together the controller in the MessageController.fs file:

MessageController.fs

namespace FSharpWebApi.Controllers

open Microsoft.AspNetCore.Mvc
open Microsoft.Extensions.Logging

[<ApiController>]
[<Route("[controller]")>]
type MessageController (logger : ILogger<MessageController>) =
    inherit ControllerBase()

    [<HttpGet("{id}")>]
    member __.Get (id : int, shouldDouble : bool) : string=
        logger.LogInformation("I am a controller")

        let result =
            match shouldDouble with
            | true -> id * 2
            | false -> id

        sprintf "Hello %d" result

From the function's signature we can see that it has an id and shouldDouble values as inputs and that it returns a string. We have made these explicit however if we were to leave them out it would be fine too as F# would be able to infer the types, see that below:

We can open the following URLs in our browser and should be able to open the /message/3 and /message/3?shouldDouble=true routes and see hello 3 and hello 6 respectively

Note that if not specified the handler inputs will try to be parsed from the body

Now, if we would want to update this to return some sort of general HTTP Response Code when a user sends some kind of input, for example if the result is 4, we will need to modify the function such that we are able to reference the this and the return type of the function is now an IActionResult

[<HttpGet("{id}")>]
member this.Get (id : int, shouldDouble : bool) : IActionResult =
    logger.LogInformation("I am a controller")

    let result =
        match shouldDouble with
        | true -> id * 2
        | false -> id

    match result with
    | 4 -> this.NoContent() :> IActionResult
    | _ ->
        sprintf "Hello %d" result
        |> this.Ok
        :> IActionResult

From this we can see that we are using an additional match to either return this.NoContent() as an IActionResult or this.Ok with the piped message as an IActionResult. Just note that the following matches are equivalent:



// call the `this.Ok` function with
match result with
| 4 -> this.NoContent() :> IActionResult
| _ ->
    this.Ok(sprintf "Hello %d" result) :> IActionResult

// pipe the result of the format through
match result with
| 4 -> this.NoContent() :> IActionResult
| _ ->
    sprintf "Hello %d" result
    |> this.Ok
    :> IActionResult

// pipe the result of the format through on a single line
match result with
| 4 -> this.NoContent() :> IActionResult
| _ -> sprintf "Hello %d" result |> this.Ok :> IActionResult

Post Handler

We can also create a POST handler that will pretty much do the same as the above handler, we can pretty much just take the values from the function body and pass it to the previous handler we put together

Before we can create the handler, we need to create a type called PostData that can be used by the method to receive data, we can define this towards the top of the file, above the type definition for our MessageController. The type also needs to have the CLIMutable attribute so that the JSON deserializer can parse the data from the post body into it correctly

[<CLIMutable>]
type PostData =
    { id : int
      shouldDouble : bool }

Next we simply need to define the Post method with an HttpPost attribute which will just call the this.Get using the input params. this can be done pretty simply too

[<HttpPost>]
member this.Post(data : PostData) : IActionResult =
    this.Get(data.id, data.shouldDouble)

And that's really all that's needed

Conclusion

So yeah, that's pretty much it - Not that bad right? I feel like there are a couple of things that feel a little bit weird because of the pieces of OOP running around from C# that add a bit more overhead than I'd like, but it's .NET, that's inevitable

Still a few more to posts on F# to come, so stay in tuned

Nabeel Valley

The Gatsby Migration, pt.2 - Dumb Pages

Sat, 01 Feb 2020 00:00:00 GMT

The Gatsby Migration, pt.2 - Dumb Pages

Introduction

In the last post we looked setting up an application with a few basic routes. These routes were all assigned to Components in the src/pages directory.

This post will be going throught the Gatsby Setup necessary in order to migrate our current site to Gatsby, we will be looking at the second step in the process that was outlined in the last post:

Creating the initial React App
Rendering the "Dumb" pages with Gatsby (This post)
Rendering the "Smart" page with Gatsby

Getting Ready

In order to Gatsbyify the application, there are three steps that we will need to take before we can start updating our pages to work with the new system

Update folder structure
Install Gatsby
Create the necessary Gatsby Config files

Update Folder Structure

Gatsby Builds are placed into the public directory. This currently is the output directory of a React build if we are using the standard React configuration

Before we do anything more we should rename our public directory to static as this is what Gatsby uses for static files, and then make a git commit before we add the public directory to our .gitignore

Now we need to add the following lines to the end of our .gitignore file so that we do not track the gatsby build files:

.gitignore

## gatsby
public
.cache

Install Gatsby

To add Gataby to our project we need to add the gatsby package from npm as a dependency to our project. From the project's root directory run:

yarn add gatsby

Next we'll add/update the following commands in our package.json file so that we can start the Gatsby Dev Server as well as build the application

package.json

"scripts": {
    "start": "gatsby develop",
    "build": "gatsby build",
    "clean": "gatsby clean",
    ...
},

Create Config Files

In order to enable gatsby we need to add the gatsby-config.js file to our root directory, we can use the starter file with the following content, as it currently stands this doesn't do anything

gatsby-config.js

module.exports = {
  plugins: [],
}

Next we'll create an html.js file in the src directory with any relevant content from your index.html file if any of it has been updated. Also be sure to remove the %PUBLIC_URL% stuff from the file content

The html.js file needs to be a React Component with the following basic structure for a standard CRA app

src/html.js

import React from 'react'
import PropTypes from 'prop-types'

export default function HTML(props) {
  return (
    <html {...props.htmlAttributes}>
      <head>
        <meta charSet="utf-8" />
        <link rel="shortcut icon" href="/favicon.png" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
        <meta name="theme-color" content="#ffffff" />

        <link rel="manifest" href="/manifest.json" />

        <title>React App</title>

        <script
          dangerouslySetInnerHTML={{
            __html: `
                        // Any scripts that need to be included in the HTML itself
                        // Like tracking code, etc.
                        console.log("Inline Javascript")
                    `,
          }}
        ></script>

        {props.headComponents}
      </head>
      <body {...props.bodyAttributes}>
        {props.preBodyComponents}
        <div
          key={`body`}
          id="___gatsby"
          dangerouslySetInnerHTML={{ __html: props.body }}
        />
        {props.postBodyComponents}
      </body>
    </html>
  )
}

HTML.propTypes = {
  htmlAttributes: PropTypes.object,
  headComponents: PropTypes.array,
  bodyAttributes: PropTypes.object,
  preBodyComponents: PropTypes.array,
  body: PropTypes.string,
  postBodyComponents: PropTypes.array,
}

You can add or remove any elements in that file as per your specific requirements but most of the time the above should be fine

Now that we have updated the index.js file delete the static/index.html file you can run the Gatsby Dev Server

yarn clean
yarn start

Your application should now be running on http://localhost:8000/, if started correctly you should see the Gatsby Development 404 Page:

Development 404 Page

Gatsby.js development 404 page

There's not a page yet at /

Create a React.js component in your site directory at src/pages/index.js and this page will automatically refresh to show the new page component you created.

If you were trying to reach another page, perhaps you can find it below.

Pages (4)

Search:

/Blog/
/Home/
/NotFound/
/Post/

Page Setup

From the Development 404 Page we can see that Gatsby has found our previously created pages - this is because Gatsby looks for pages in the pages directory, however when we click on a the links we will notice the following:

Home and 404 render correctly
Blog results in an error message
Post results in an error message (we will address this page in the next post)
Routes aren't aligned with our initial setup
Components no longer have the layout we setup in the App.js file

These are, for the most part, easy problems to solve

Fix the Blog Error Message

If we look at the Blog page we will see that there is an issue with the Page render, this is because we need to change the Link components to be imported from gatsby instead of react-router-dom because with Gatsby we are no longer using the React Router

Blog.js

import { Link } from 'gatsby'

Fix the Routes

We can also see that our routes are capitalized which is not what we want. Gatsby uses the file organization to do routing, so in order to correct our routes to align what we defined previously in our App.js we will first need to rename our files:

Component	Route	Old Name	New Name
Home	`/`	`Home.js`	`index.js`
Blog	`/blog`	`Blog.js`	`blog.js`
NotFound	`/*`	`NotFound.js`	`404.js`

We're not going to handle the Post component for now because this uses a dynamic route which is something I'll cover in the next part of this series

You should stop and restart the Gatsby Dev server with:

yarn clean
yarn start

When the page loads up we should now see our Home content rendered on the / route

Fix the Layout

When we were using the standard React Routing we made use of the App component to wrap our page routes as well as our navigation. We'll convert our App.js file into a component that we can use in our other pages

Remove the Router and Switch
Update the Links to use gatsby instead of react-router-dom
Add the children input parameter
Render the children in the main section
Remove unused imports

App.js

import React from 'react'
import './App.css'
import { Link } from 'gatsby'

const App = ({ children }) => (
  <div className="App">
    <header>
      <nav>
        <h1>My Website Title</h1>
        <Link to="/">Home</Link>
        <Link to="/blog">Blog</Link>
      </nav>
    </header>
    <main>{children}</main>
  </div>
)

export default App

Use the Layout

Now that we've essentially created a Layout component in the form of the App component we can use it to wrap the pages that we've exported. We can do this for the Home page like so:

index.js

import React from 'react'
import App from '../App'

const Home = () => (
  <App>
    <div className="Home">
      <h1>Home</h1>
      <p>This is the Home page</p>
    </div>
  </App>
)

export default Home

The same applies for the blog and 404 pages

Blog

blog.js

import React from 'react'
import { Link } from 'gatsby'
import App from '../App'

const Blog = () => (
  <App>
    <div className="Blog">
      <h1>Blog</h1>
      <p>This is the Blog page</p>
      <div>
        <Link to="/blog/post-1">Post 1</Link>
      </div>
      <div>
        <Link to="/blog/post-2">Post 2</Link>
      </div>
    </div>
  </App>
)

export default Blog

404

404.js

import React from 'react'
import App from '../App'

const NotFound = () => (
  <App>
    <div className="NotFound">
      <h1>404</h1>
      <p>Page Not Found</p>
    </div>
  </App>
)

export default NotFound

Fix the CSS

Before we can use the App component we need to add the gatsby-plugin-postcss and postcss-preset-env plugins so that Gatsby knows how to interpret the default create-react-app method of importing our CSS into a component

Stop the Dev Server and install the required packages:

yarn add gatsby-plugin-postcss postcss-preset-env

We then need to update the gatsby-config.js file to use the plugin we just added:

gatsby-config.js

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-postcss`,
      options: {
        postCssPlugins: [require(`postcss-preset-env`)({ stage: 0 })],
      },
    },
  ],
}

If you start the application now you'll notice that the font is not correct, this is because we specify the font styles in the index.css file. In order to fix this we need to import the index.css file into our application, in the default React application this is done via the src/index.js file, but since we don't use that to load up the application anymore we need to create another file to do that with Gatsby

In the root directory create a file called gatsby-browser.js, in this we just need to import the index.css file:

gatsby-browser.js

import './src/index.css'

You can now run the following commands and start up the application:

yarn clean
yarn start

Assuming everything went as planned the application should start up correctly and the styling from our index.css file will be correctly applied

Summary

By now we have completed the first part of the migration process - converting our static pages to use Gatsby - by taking the following steps:

Updating our file structure to better align with Gatsby
Create the relevant Gatsby config files
Update our Link usage
Update our Routing
Use a shared layout component
Fixing the CSS to work with Gatsby

In the next part we'll look at how we can go about providing the data needed for our Post component using GraphQL and show our posts. We'll also look at how we can pre-process the data that we provide to our component so that we can enhance our content while improving our content creation process

Nabeel Valley

Debug POSTs using an Express App

Fri, 04 Dec 2020 00:00:00 GMT

Debug POSTs using an Express App

Sometimes it's useful to have an endpoint that you can use to debug data that's being POSTed to an application

You can make use of the following express.js app to log your application's POST requests:

View Code

const express = require('express')
const app = express()

// parse json
app.use(express.json())

// GET endpoint to check uptime
app.get('/', (req, res) => {
  res.json({ data: 'hello' })
})

// POST endpointthat logs request body
app.post('/', (req, res) => {
  console.log(req.body)
  res.json(req.body)
})

// listen for requests
const listener = app.listen(process.env.PORT, () => {
  console.log('listening on port ' + listener.address().port)
})

Web APIs with AdonisJS and PostgreSQL

Sun, 06 Sep 2020 00:00:00 GMT

Web APIs with AdonisJS and PostgreSQL

In this post, we'll take a look at building an API using the AdonisJS framework. We'll specifically be looking at the AdonisJS v5 Release Preview and how we can create a simple REST API that handles some CRUD operations on a Database

Because we're going to be creating a REST API, you should have some experience making HTTP Requests, for our purposes we can use a website called Postwoman to interact with our API, but you can also use any other tool you prefer

About AdonisJS

AdonisJS is an opinionated, Typescript-first web framework that provides a lot more "out of the box" functionality than the traditional framework or library in the Node.js ecosystem and is more comparable to something like .NET's WebAPI or MVC Framework than things like Express or Next in Node.js

Some of the built-in features and decisions that stand out for me are:

First-class Typescript support
Class-based controllers
Class-Based SQL ORM
Dependency Injection
Request Body Validation
Authentication
Server-side View Rendering

There are a lot more features, and it would be impractical for me to talk about all of them in a single post. Overall, the framework seems to be very complete at this point

Prerequisites

We'll be using a Node.js and a SQL Database for this post, so if you're going to follow along with this post you will need to have a couple of things installed:

Node.js and NPM
Any Supported SQL Database:
- MySQL
- SQLite
- Microsoft SQL Server
- PostgreSQL
- MariaDB
- OracleDB

Alternatively, you can run a Docker image for the database which may be easier, I'll be using Postgres via Docker with VSCode

To learn more about VSCode Dev Containers you can look at my previous post on how to use Dev Containers

Initialize an Application

Now that we've got all our necessary dependencies installed, we can initialize an application using the npm init command:

During the initialization the CLI will ask you what type of project to initialize, be sure to select API Server

npm init adonis-ts-app my-api

The above command is made up of the following parts:

npm init which is the npm command that will run an initialization script from the provided package
adonis-ts-app which is the package to be used for initialization
my-api is the name of the folder/project we want to create

Once the project has been configured, navigate into the project directory:

cd my-api

And start the app:

npm start

The Ace CLI

AdonisJS makes use of a command-line application called ace, ace can be used to do common tasks as well as custom tasks that we define. By default, it can scaffold controllers, commands, and a bunch of other things as well as run and build an AdonisJS application

Environment Variables

AdonisJS Makes use of Environment Variables do set the application configuration, in your generated files you should see a file named .env, this file contains the environment variables used by the application. Looking at this file will give us an idea of our current configuration:

.env

PORT=3333
HOST=0.0.0.0
NODE_ENV=development
APP_KEY=mE8zN8V_7PzazKfv9_ds-8CGjVRLA2wo

At the moment, we can see that our application will be listening on host 0.0.0.0 and port 3333, this means that we can access our application from our browser at http://localhost:3333

If we open this page on our browser we will see the adonis logs kicked off in our command line, something like this:

ℹ  info      cleaning up build directory build
ℹ  info      copy .env,ace build
☐  pending   compiling typescript source files
✔  success   built successfully
ℹ  info      copy .adonisrc.json build
…  watch     watching file system for changes
ℹ  info      starting http server
✔  create    ace-manifest.json
[1595755199986] INFO  (my-app/1531 on 47f057d8722c): started server on 0.0.0.0:3333

The first request may take some time, this is because the server is still starting itself up. In the meantime, however, let's discuss how we'll be accessing our API

Making Requests

You will need to use Postwoman or something similar when making requests

Now that our application is running, we can start making requests to our API. By default, AdonisJS sets up a hello-world route at the base of our application (the / route`)

We can reach this endpoint by simply making a GET request to http://localhost:3333 which will return the following:

{
  "hello": "world"
}

AdonisJS can define routes using a method similar to libraries like Express.js, the "Hello World" route is defined in the start/routes.ts file and has the following:

routes.ts

import Route from '@ioc:Adonis/Core/Route'

Route.get('/', async () => {
  return { hello: 'world' }
})

The Route.get portion states that this is a GET route on the / path with an async handler function that returns an object

However, for this post, we won't be defining our route's handler functions like this. We're going to make use of controllers

Controllers

AdonisJS uses controllers to structure our API. This makes use of a class which contains functions intended to work as an interface between the HTTP request and the work we want to do via a provider

We will simply state which controller and method we want to use in our routes.ts file instead of containing all the handler logic in that file

To generate a controller we can use ace. We will use ace to create a UsersContoller.

First, stop your running application with ctrl + c in the terminal, then use the following ace command to generate a User controller:

node ace make:controller User

This will generate an app/Controllers/Http/UsersController.ts file with the following contents:

UsersController.ts

export default class UsersController {}

Which exports a class called UsersController with no default functionality

We can create any functions inside of here that we want to, and then map these functions to routes. For now, let's add a get function that will put some placeholder data into. We'll update this later to use our database

A few things to note about the function we're going to create:

The function is asynchronous, so if there are any long-running operations it won't block other things from running
The name of the function is get and it has no parameters
The function returns an array of objects that represents a User

UsersController.ts

export default class UsersController {
  public async get() {
    return [
      {
        id: 1,
        name: 'Bob Smith',
        email: 'bob@smithmail.com',
      },
    ]
  }
}

Now that we have defined our function, we can add a route that will cause this function to be called. We do this by adding the following in the routes.ts file:

Route.get('users', 'UsersController.get')

We can then run npm start from the command line and make a GET request to http://localhost:3333/users which should return our user:

[
  {
    "id": 1,
    "name": "Bob Smith",
    "email": "bob@smithmail.com"
  }
]

Database

Now that we've got some basic understanding of how AdonisJS maps routes to functionality, we can connect our application to a Database

To add database functionality, we first want to install the adonisjs/lucid package to our application. Behind the scenes, AdonisJS makes use of Lucid for connecting to and working with databases

npm i @adonisjs/lucid@alpha

And then run the following command to initialize it:

node ace invoke @adonisjs/lucid

Which should give the following output:

   create    config/database.ts
   update    .env
   update    tsconfig.json { types += @adonisjs/lucid }
   update    .adonisrc.json { commands += @adonisjs/lucid/build/commands }
   update    .adonisrc.json { providers += @adonisjs/lucid }
✔  create    ace-manifest.json

If you open your .env file you will see the configuration for a sqlite database

.env

## ... other config
DB_CONNECTION=sqlite
DB_HOST=127.0.0.1
DB_USER=lucid
DB_PASSWORD=lucid
DB_NAME=lucid

You can follow the general database setup information on the AdonisJS Docs to set yours up, but I'll be using Postgres as I've mentioned before

To use Postgress, you need to install the pg package from npm:

npm i pg

Then configure your application to use Postgres. We will first update our .env file to have our database credentials:

.env

DB_CONNECTION=pg
DB_HOST=db
DB_USER=user
DB_PASSWORD=pass
DB_NAME=data
DB_PORT=5432

Once you've configured your database connection information in the .env file, our database connection information will be taken care of by AdonisJS

These environment variables are used in the config/database.ts file, we can see them in the pg part of the file

database.ts

  ...
    pg: {
      client: 'pg',
      connection: {
        host: Env.get('DB_HOST', '127.0.0.1') as string,
        port: Number(Env.get('DB_PORT', 5432)),
        user: Env.get('DB_USER', 'lucid') as string,
        password: Env.get('DB_PASSWORD', 'lucid') as string,
        database: Env.get('DB_NAME', 'lucid') as string,
      },
      healthCheck: true,
    },
  ...

In the database.ts file above, look for the section for your relevant database, and set the healthCheck property to true

Now that we've set things up, we can create a health-check route that will enable us to view the current status of our database connection. We will add a handler for this in the routes.ts file:

import HealthCheck from '@ioc:Adonis/Core/HealthCheck'
// ... existing file content

Route.get('health', async ({ response }) => {
  const report = await HealthCheck.getReport()
  return report.healthy ? response.ok(report) : response.badRequest(report)
})

Once you've done all the above, start the development server again with npm start

Then make a GET request to http://localhost:3333/health to view your health-check information

{
  "healthy": true,
  "report": {
    "env": {
      "displayName": "Node Env Check",
      "health": {
        "healthy": true
      }
    },
    "appKey": {
      "displayName": "App Key Check",
      "health": {
        "healthy": true
      }
    },
    "lucid": {
      "displayName": "Database",
      "health": {
        "healthy": true,
        "message": "All connections are healthy"
      },
      "meta": [
        {
          "connection": "pg",
          "message": "Connection is healthy",
          "error": null
        }
      ]
    }
  }
}

In the lucid section we will see if our database connection is working or any applicable error information.

Defining Models

Once we've configured our database, we will want to interact with the data in it. Lucid makes use of models essentially as proxies for database tables. We can generate a model for our User with ace as follows:

Stop your development server with ctrl + c before running this:

node ace make:model User

The above script will have generated a User model class in the app/Models/User.ts file that extends BaseModel. All models must do this. The generated User.ts file looks like this:

User.ts

import { DateTime } from 'luxon'
import { BaseModel, column } from '@ioc:Adonis/Lucid/Orm'

export default class User extends BaseModel {
  @column({ isPrimary: true })
  public id: number

  @column.dateTime({ autoCreate: true })
  public createdAt: DateTime

  @column.dateTime({ autoCreate: true, autoUpdate: true })
  public updatedAt: DateTime
}

When viewing the above file your code editor may give you a warning saying that decorators are not supported, if you see this then set the experimentalDecorators property to true in your tsconfig.json file:

...
  "compilerOptions": {
    "experimentalDecorators": true,
...

Note the @column decorators, these are used to map columns in our database to our model fields. We'll add a field for our user's name and email as follows:

User.ts

export default class User extends BaseModel {
  // ... other stuff in class
  @column()
  public name: String

  @column()
  public email: String
}

Any properties or methods defined in a class without the @column decorator will not be mapped to the database, we can just use these as normal functions in the class and implement utilities from them

Migrating the Database

At this point, our database does not contain a user table, which will be used to store our data for the User model. We need to create a Database Migration which will add the required table and fields.

ace provides us with a method to scaffold a migration file. To create this file we will run the following command:

node ace make:migration users

Which will have created a database/migrations/SOME_ID_users.ts file with the following:

*_users.ts

import BaseSchema from '@ioc:Adonis/Lucid/Schema'

export default class Users extends BaseSchema {
  protected tableName = 'users'

  public async up() {
    this.schema.createTable(this.tableName, (table) => {
      table.increments('id')
      table.timestamps(true)
    })
  }

  public async down() {
    this.schema.dropTable(this.tableName)
  }
}

The generated file (above) contains an up function which will create a users table with an id as well as createdAt and updatedAt fields. We will need to modify the up function to add our new fields as well:

export default class Users extends BaseSchema {
  ...

  public async up () {
    this.schema.createTable(this.tableName, (table) => {
      table.increments('id')
      table.timestamps(true)

      // our added fields
      table.string('name').notNullable()
      table.string('email').unique().notNullable()
    })
  }

  ...

Once we've defined our migration script we need to build the application with:

node ace build

And then we can run the migration using ace as follows:

node ace migration:run

If there is an error in a migration script, we can rollback the migration with node ace migration:rollback which will run the down function in your migration

Interact with the Database

Now that we've got our database, we can interact with it using the User model we defined earlier

The first change we'll make is to modify the get function to return all users. To do this we need to import App/Models/User and use the User.all method:

import User from 'App/Models/User'

export default class UsersController {
  public async get() {
    // get all users
    return await User.all()
  }
}

We use await because the User.all function is asynchronous

Next, we'll add a function to create a User. We will call it post. To make this function work we need to do a couple of things:

Import HttpContextContract from @ioc:Adonis/Core/HttpContext
Retrieve the request from the HttpContextContract
Create a User
Return the created user

import { HttpContextContract } from '@ioc:Adonis/Core/HttpContext'
// ... other imports

export default class UsersController {
  // ... get user code above
  public async post({ request }: HttpContextContract) {
    // get the user from the request body
    const newUser = request.all() as Partial<User>

    // create a user using the object we received
    const user = await User.create(newUser)

    // return the created user object
    return user
  }
}

The request.all function combines the data from the request body and query string into a single object

When we've added that, our completed UsersController.ts file should look like this:

UsersController.ts

import { HttpContextContract } from '@ioc:Adonis/Core/HttpContext'
import User from 'App/Models/User'

export default class UsersController {
  public async get() {
    return await User.all()
  }

  public async post({ request }: HttpContextContract) {
    const newUser = request.all() as Partial<User>
    const user = await User.create(newUser)
    return user
  }
}

Now that we've added a new function to our controller, we need to expose a route to it in the routes.ts file:

routes.ts

...
Route.post('users', 'UsersController.post')
...

Consume the API

Now that we've created API endpoints for listing all users and creating a user we can restart our development server with npm start and consume our API from wherever we want to make some HTTP requests

Create a User

To create a User we need to make aPOSTrequest tohttp://localhost:3333/users`and a Content-Type ofapplication/jsonwith the following as thebody for our request:

{
  "name": "Bob Smith",
  "email": "bob@smithmail.com"
}

Which should return a created user like:

{
  "name": "Bob Smith",
  "email": "bob@smithmail.com",
  "created_at": "2020-07-26T14:35:25.987-00:00",
  "updated_at": "2020-07-26T14:35:25.988-00:00",
  "id": 4
}

We can also try to create a User with the same information but we will see that this fails due to the database constraints we added in our migration script

Next, we can get a list of all Users by making a GET request to http://localhost:333/users which should give us back our created users:

[
  {
    "id": 4,
    "created_at": "2020-07-26T14:35:25.987-00:00",
    "updated_at": "2020-07-26T14:35:25.988-00:00",
    "name": "Bob Smith",
    "email": "bob@smithmail.com"
  }
]

Summary

My overall impression of AdonisJS is pretty good. The framework feels very stable and I had much fewer issues in the process of learning it and writing this post than I have had using other more popular frameworks.

AdonisJS is very well documented with code samples for most traditional tasks and is backed by some solid libraries for things like database integration

Working with SQL using the framework has been fairly straightforward, and the ability to write database migrations using functionality provided by the framework instead of just tossing some wild SQL together makes it more approachable

Personally, however, I prefer no-SQL databases like MongoDB and tend to use them more often when using JavaScript or TypeScript, but I feel like if the need arises for a SQL database then AdonisJS is a really good option, especially if you're a JavaScript developer and don't want to have to learn Java or C# for this type of functionality

There is also a lot more functionality than what I've gone through in this post, so I'd recommend browsing the AdonisJS docs to get a broader sense of what the framework entails

If you feel like playing around with the code I've gone through in this post without having to write it all then I've got it all on GitHub and it should just be plug-and-play using Visual Studio Code Remote Containers. If you want to learn more about that, then you can check out my previous blog post on developing within a container

Colour in Black and White Photography

Tue, 07 Apr 2020 00:00:00 GMT

Colour in Black and White Photography

Let's talk about black and white shall we?. As a first-time monochromer there are a couple of aspects of shooting and editing a black and white picture that isn't immediately apparent

When shooting in Black and White (henceforth B&W) there are some things we need to be aware especially aware of, namely:

Brightness
Texture
Colour

Now, of course, other considerations can be made but most of these boil down to how you use these three elements. For the most part, I'll be discussing how colour comes into play from an editing perspective, but this should give you an awareness of how this applies when shooting too

Brightness

When I refer to brightness in this context I am not referring to the exposure of the image but rather the distribution of brightness across the image itself, where are your shadows and highlights, how bright is your subject in relation to your background, etc.

Typically when we are working with a B&W image we try to have our subject be a little brighter (or much brighter) than everything else, this helps us to draw the viewer's attention more easily, of course, this is only one of the factors

Texture

When shooting B&W we also look for texture, I would argue that texture is to a black and white picture what colour is to a colour one. Texture helps us to differentiate between different elements in a photo and is very evident when used a lot of strong contrast. Sometimes you want a lot of texture, sometimes you don't - it is, however, important that you acknowledge how it contributes to an image

Colour

My focus in this post is primarily colour because I think it's the easiest to play around with and can make the biggest impact to a picture when editing (how else does anyone get B&W these days?)

At a simplified level, the way we work with colour in a B&W picture is by manipulating the brightness with which different colours are interpreted which in turn allows us to emphasize (or de-emphasize) specific elements in an image

Over the remainder of this post I'll be discussing a lot of this in the context of the following image:

For our discussion there are a few important aspects of this image to note:

The image is darkest in the background (top left) and transitions to being brighter in the foreground (bottom right)
We primarily have the subject in red and the background element (leaves) in green
The brightness of our red and green elements in their respective sections of the image are approximately the same

If we were to simply convert this image to a B&W one we get the following:

From the above we see that the distinction between our red and green elements isn't really visible anymore, this is because of the impact that colour mixing has on our image. Mixing allows us to control the way the different colours are seen in the image and most photo editing tools have some version of a Black and White Mixer that allows us to apply a colour filter to a Black and White image

Neutral Filtering

This is the "natural" way that a B&W conversion works, using this method for any two colours at approximately the same brightness we should see the same brightness in the B&W version

If we look at the neutral version of our image we can observe that the brightness across our image is fairly consistent with that of the colour version with a darker background and brighter foreground. We can also see that there is very little differentiation between the red and green elements in our image

In photography, there are few well-known colour filters which can be placed in front of your lens to achieve what we are doing here digitally. Commonly we have red, green, blue, and yellow filters. We'll just be looking at the red and green filters that were used to edit this image

Red Filter

In a picture like this where our subject is red, we can use a red filter to increase the brightness of the red portions of our image. To do this we would bump up the red and orange sections of our colours while trying to maintain a smooth change from one colour to the next to prevent any sharp changes in the brightness of our image

Additionally, I've also made the green slightly darker to add some additional contrast. We can see the result of doing so along with the changes made to the brightness of the respective colours below:

Using a method like this we can draw our focus back to the red elements a little more than if we had left the brightnesses as they were in the Neutral image while pushing down the brightness of our green elements so we don't have the two competing for focus

Green Filter

We can also do something like the above using a filter that brightens the green and darkens the red

In the resulting image above we can see that the greens are now much brighter and the reds are almost black - however we still maintain a good level of contrast between our too resulting elements. We can see a comparison of the impact the different colour filters have on our image below. These are ordered from left to right as red, neutral and green

Using what we've looked at in the above we can see that the way we choose to filter a black and white image has a huge impact on how we direct the viewer's attention in our image as well as how clearly we can differentiate between different elements

Note that while we do have different common colour filters, when editing a B&W image it usually makes sense to play around with your Black and White Mixer Tool to see what works best for your image

Nabeel Valley

Package Management with Poetry

Wed, 07 Oct 2020 00:00:00 GMT

Package Management with Poetry

Poetry Docs

Python package management is typically quite a mess. Managing packages with pip often requires additional management of things like virtual environments and python version management

Poetry is a package manager that abstracts a lot of the typical Python dependency and environment management away from the user

Install Poetry

Before you can install poetry you need to have Python installed

To install poetry you can do one of the following depending on your OS:

Windows Powershell

(Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -UseBasicParsing).Content | python -

Bash

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

Initialize Poetry

Create a New Project

If you're starting a new project, you can run the following:

poetry new my-project

The above will generate a pyproject.toml file with the project settings. Alternatively you can use the following to add poetry to an existing project

Add to Existing Project

To add poetry to an existing project, run the following:

poetry init

Using Poetry

Add Dependency

To manage dependencies you can use the poetry add command. For example, if we would like to install flask

poetry add flask

Run Application

To run an application using the virtual environment created by poetry you can use the poetry run command, followed by the command you want to run:

poetry run python app.py

Running a flask app would look something like this:

poetry run flask run

Create a Shell

To create a shell in the poetry virtual environment run:

poetry shell

The above will open a poetry shell with the virtual environment. You can then do something like run the python command to open a python shell in the environment

Trust .NET Core Dev Certificates

Wed, 07 Oct 2020 00:00:00 GMT

Trust .NET Core Dev Certificates

When developing .NET Core web applications with HTTPS, to simplifiy your experience and avoid browser warnings you can trust the .NET Core dev certs on your local device with the followning commands:

dotnet dev-certs https --clean
dotnet dev-certs https --trust

Capture Fetch with Cypress

Tue, 10 Nov 2020 00:00:00 GMT

Capture Fetch with Cypress

To capture the result of a fetch request with Cypress you will need to make use of cy.route2

Setup

The cy.route2 command needs to be enabled in your cypress.json file before usage, to do so add the following:

cypress.json

{
  "experimentalNetworkStubbing": true
}

Usage

Next, using the command to capture requests to a route looks a bit like this in a test:

cy.route2('POST', '/do-stuff').as('data')

In the above you need to use the cy.route2 function. In the above we're capturing POST requests to the /do-stuff endpoint. The object which contains the request and response data is stored in the @data which can be retreived and worked with like so:

cy.wait('@data').then((data) => {
  // do stuff using the data object
  console.log(data)
})

The cy.wait function retreives the data object, we use the .then function with a callback function to do stuff with the fully resolved data object

Furthermore, the HTTP response from the data object can be parsed using JSON.parse, you can then also add assertions based on the response object. Adding this in, the above code would look more like this:

cy.wait('@data').then((data) => {
  console.log(data)

  // parse the response body
  const res = JSON.parse(data.response.body)

  // assertions on the response
  assert.isTrue(res.success)
  assert.isNotNull(res.message)
})

Automate Anything with GitHub Actions

Tue, 11 Aug 2020 00:00:00 GMT

Automate Anything with GitHub Actions

You can take a look at the Action we build in this post here on GitHub. I've also included an action that's responsible for building and updating the action dist which may also be of interest. It's all in this Repository

A few weeks ago I was playing around with GitHub actions and the recently introduced GitHub Account README functionality and wanted a way to make this "static" file a bit more dynamic

Enter GitHub Actions. GitHub actions are a way for you to define and run programmatic tasks. If you can put it in code, then you can run it as an action

Usually, you'll be using actions that have been defined by GitHub or another developer for common tasks, such as running your application CI or deployment processes. However, since actions are extremely generic, we're going to do something a little different - update our Twitter bio?

In this post, we're going create a GitHub action that uses the Twitter API to update our bio, as well as make the action we created run automatically on GitHub

Create a GitHub Repository for us to work in
Set up Twitter Developer Credentials
Write a Node.js script that uses the Twitter API
Configure our GitHub Action Metadata
Add our Twitter Secrets in GitHub
Configure a GitHub Action that will run our script (an action, within an action)

Prerequisites

Git
Node.js
Visual Studio Code or any other Code Editor

Create a GitHub Repo

For us to run our action we'll need a GitHub Repository to use, to create a Repo go to GitHub and sign in, thereafter go the 'Create a new repository page' and fill in the details, be sure to select Initialize this repository with a README, pick Node as the .gitignore file, and select a license if you'd like to

Once you've done that, click Create repository and you should see the initial files we added to the Repo. Next, click on the Code button and copy the URL in the text box

Now, from a terminal, you will need to clone the repository. Run the following command and make sure to paste the link you just copied in place of <YOUR URL> below:

git clone <YOUR URL>

Next, open the folder you just cloned in your code editor and create a new file called .env in the folder root directory. We'll keep our Twitter credentials in this file for testing. For now, just add the following to the file:

.env

TWITTER_CONSUMER_KEY=
TWITTER_CONSUMER_SECRET=
TWITTER_ACCESS_KEY=
TWITTER_ACCESS_SECRET=

In the next step, we're going to get the credentials from Twitter, we'll add them into the file above when we're done

Get Some Twitter Cred.

Twitter exposes the Twitter Developer API that allows us to do all kinds of useful and pointless things by interacting with Twitter's data

For us to consume the Twitter API we require credentials for the API. To set these up we'll need to do a few things

First, open your browser on the Twitter Developer Portal and click the sign-in button. Then, once you've signed in, you should see your name on the top right of the page, click on the dropdown arrow and select Apps to go to the App Dashboard

On the App Dashboard, click Create an app and fill in the required information. For the Website URL you can put the URL of your GitHub repository that you copied previously into this field. Once you've filled in all the details click Create at the bottom of the page

You should now see the Details Page for the app we just created. Next, click on the Keys and Tokens tab and click Generate then copy and paste each token after the = in the .env file we created without any spaces before or after the key

When you've pasted your tokens into their places, your .env file should look something like this:

Note that using the template for .gitignore that we chose, the .env file will be automatically ignored. Don't publically upload this data just anywhere as it can potentially give someone access to your Twitter account

.env

TWITTER_CONSUMER_KEY=xxxxxxxxxxxxxxxxxxxxxx
TWITTER_CONSUMER_SECRET=xxxxxxxxxxxxxxxxxxx
TWITTER_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxx
TWITTER_ACCESS_SECRET=xxxxxxxxxxxxxxxxxxxxx

(With the actual keys, and not just xxxx)

Now that we've got our credentials set up, we can start to work on the application

Using the Twitter API

We're going to be writing a script that runs on Node.js (JavaScript) and makes use of the Twitter API using the twit library for Node.js and the GitHub libraries for working with GitHub actions

To get started, run the following command from a terminal within your repository's directory to initialize a new Node.js project and select the defaults for all the questions:

npm init -y

Next, we'll add the dependencies that our application is going to need to run:

@actions/core allows us to interact with the data provided to our action by GitHub
twit is used to interact with the Twitter API
dotenv enables us to load in our .env file so our application can use it

npm install @actions/core twit dotenv

Once the application is done installing we'll create an index.js file inside of our repo folder and we can get started on our code

Inside of the index.js the first thing we'll want to do is import our environment variables from the .env file we configured, we'll do that using the dotenv NPM Package we installed previously:

index.js

require('dotenv').config()

Next, we'll configure a new Twitter API Client using our environment variables. The twitter Package exports a Twitter class that we can create an instance of, do to this we will use the Twitter constructor and get our environment variables that are all stored in the process.env variable:

index.js

const Twitter = require('twitter')

const client = new Twitter({
  consumer_key: process.env.TWITTER_CONSUMER_KEY,
  consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
  access_token_key: process.env.TWITTER_ACCESS_KEY,
  access_token_secret: process.env.TWITTER_ACCESS_SECRET,
})

Now that we've got an instance of a Twitter client that we can use to interact with the API we'll want to use it

The Twitter API Client runs asynchronously, for us to work with it correctly we have two options:

Using the callback method, which means that we give the client a function to run when it's done sending the data to Twitter. While this is easier in our specific circumstance it can become difficult to keep track of when we have many different callback functions
The second option is to use async and await syntax, which allows us to write our code more sequentially and makes it easier for us to control our sequence and flow

To use the async/await functionality, we need the code that we need to await to be inside of an async function. In this one function, we can await as many tasks as we want sequentially without worrying about callbacks

We'll create a main function that is async in which we will do any asynchronous interactions, in our case - interact with Twitter

After the code we've already got in our index.js file, define the main function as follows:

index.js

const main = async () => {
  // this is where we'll work with the twitter client
}

Next, we'll add the code to interact with the Twitter API inside of this function. To update our Profile Bio (or description as it's called in the Twitter API) we'll make use of the account/update_profile endpoint

The data we send the Twitter API will need to be an object with a description property for what we want to set our Twitter Bio as. We'll await this function call by adding it within our main function:

index.js

const main = async () => {
  const response = await client.post('account/update_profile', {
    description: 'Hello, World!',
  })

  console.log(response)
}

The above code will set our Twitter description to Hello, World! and then print our the response we get back from Twitter

The way we've written the above code is a little bit dangerous as we aren't handling any potential errors/failures that may happen when interacting with the Twitter API

When working with an API via an HTTP Request there's always the chance that there could be an error. Errors can be caused by anything ranging from poor network connections, incorrect credentials, or a system outage on the API host itself

For our application to give us a bit more information about what happened, we may want to handle the error before passing it on to the process that kicked off our script. We'll make use of a try/catch to handle this error:

index.js

const main = async () => {
  try {
    const response = await client.post('account/update_profile', {
      description: 'Hello, World!',
    })

    console.log(response)
  } catch (error) {
    console.error(error)
    throw new Error(
      `The Twitter API Responded with an Error: ${error[0].code}, ${error[0].message}`
    )
  }
}

You can see above that we're just doing some cleanup of the error message before throwing the exception to the process

Now that we've fully defined our client and main functions we can call the function as the last line of the script:

index.js

main()

This should make a request to the Twitter API and print our the response or error if there is one

So, the above function will always set our Twitter Bio to the same value, this isn't interesting. Next, we'd like to get some data from the workflow that's going to be running our action. The way we would do this is with an input

Our action is going to take an input called bio. We can use the @actions/core library to get the value of the input. We can do this with the following:

index.js

const core = require('@actions/core')

const bio = core.getInput('bio') || 'Hello, World!'

The above code also sets a default value of Hello, World! to the description, this will allow us to also run the action without throwing an exception on our local machine as well as if the bio is not provided to the action

Furthermore, instead of just throwing errors, we can rather set the error-status using the core.setFailed function. Updating our main function to do this we now have:

index.js

const main = async () => {
  try {
    const response = await client.post('account/update_profile', {
      description: bio,
    })

    console.log(response)
  } catch (error) {
    console.error(error)
    core.setFailed(
      `The Twitter API Responded with an Error: ${error[0].code}, ${error[0].message}`
    )
  }
}

Finally, our full index.js file should look like this:

index.js

re('dotenv').config()

const Twitter = require('twitter')
const core = require('@actions/core')

const bio = core.getInput('bio') || 'Hello, World!'

const client = new Twitter({
  consumer_key: process.env.TWITTER_CONSUMER_KEY,
  consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
  access_token_key: process.env.TWITTER_ACCESS_KEY,
  access_token_secret: process.env.TWITTER_ACCESS_SECRET,
})

const main = async () => {
  try {
    const response = await client.post('account/update_profile', {
      description: bio,
    })

    console.log(response)
  } catch (error) {
    console.error(error)
    core.setFailed(
      `The Twitter API Responded with an Error: ${error[0].code}, ${error[0].message}`
    )
  }
}

main()

Now that we've written the functionality for our action we'll want to turn it into an action

Configure the Action Metadata

For GitHub to recognise our code as an action, we need to create an action.yml file that contains a description of our action. Our action.yml file needs to have the following information:

name for our action
description of the action itself
an input parameter of bio that will be used by our application
runs which states the script to run

The action.yml file for our action looks like the following:

action.yml

name: 'Twitter Bio Update'
description: 'Update your Twitter Account Bio'
inputs:
  bio:
    description: 'Text that you would like to set as your Twitter Bio'
    required: true
    default: 'Hello, World!'
runs:
  using: 'node12'
  main: 'index.js'

The fields we've got above are all pretty much required. The above fields are the simplest configuration for a Node.js action. The GitHub Docs have a lot more information on more complex configurations and actions

Now that we've defined our action, we will want to configure it to run. But before we can do that, we'll want to set up our secrets

Setting Up Secrets in GitHub

Now that we've got our action defined, we're almost ready to write a Workflow that will use this action. However, our action requires environment variables (that we've got saved in our .env file) but we don't want these to be pushed to GitHub as part of our source code. The way to set up these environment variables in GitHub is called a secret

To add our environment variables in GitHub you'll need to open your Repo on GitHub and navigate to Settings > Secrets then click New secret and add your first secret. If we use our .env file as a reference we'll want to create a secret for each line in the file. To do this look at the name of the environment variable (everything before the =) and set this as the Name for the secret, then look at the value (everything after the =) and set this as the Value for the secret then click Add secret. Do this for every line in your .env file (every environment variable`

Create a Workflow

Workflows are GitHub's way of tying together a bunch of actions to run. Often, we will want to run multiple actions. We place these into what's called a step in a job. Each Workflow can have multiple steps and jobs and a repository can have multiple workflows

The structure of a workflow is something like this:

workflow
|-- Job 1
|   |-- Step 1
|   |-- Step 2
|-- Job 2
    |-- Step 1

A Workflow can have any number of jobs and steps

We're going to create a Workflow with a single job with the goal of running our script. For us to do this, the job will need to do the following:

Checkout our code
Configure Node.js and NPM
Install Dependencies
Run our Action

To do Steps 1 and 2 we're going to use actions that are defined by GitHub. We're going to go through our Workflow file working from the top-down as this is the order in which everything will be run

Firstly, we'll need to create a new directory in our Repository named .github and inside of this another called workflows. Inside of the .github/workflows directory create a file named main.yml (this can be any name so long as it's a yml file)

Next, on the first line of this file we will define a name for our workflow like so:

main.yml

name: Run Twitter Bio Action

This is the name that will be displayed when the workflow runs. Actions are run when a specific GitHub event happens (more about that in the Docs). We're going to configure our action to run manually only, so we will use the workflow_dispatch event. After the name in the main.yml file, add the following:

main.yml

on:
  workflow_dispatch:
    inputs:
      bio:
        description: 'Twitter Bio'
        required: true

In the above, we set an on event for workflow_dispatch with an input for bio that we're going to pass on to our action

Now that we've got the workflow metadata defined, we're going to add a job. To do this, we'll add a jobs object with a name of update-bio, a specification on what OS it needs to run on, and the steps that will be a part of it:

main.yml

jobs:
  update-bio:
    runs-on: ubuntu-latest
    name: Update Twitter Bio
    steps:
      # we'll add the steps here in a moment

We can see above that we're running on the ubuntu-latest OS. Next, we'll add the first step for checking out our code in the action. This will use actions/checkout:

main.yml

steps:
  - name: Checkout
    uses: actions/checkout@v2

In the above, we give our step a name which is for display purposes, and a uses which says what action this step should use. In our case, we're checking out our code using the actions/checkout@v2 action

Next, we'll want to configure Node.js and NPM because our action needs these to run. We can use the actions/setup-node@v1 for this:

main.yml

- name: Setup Node.js
  uses: actions/setup-node@v1
  with:
    node-version: 12.x

In this action, we use the with to state an input that the action needs. In this case, we specify that we want to use a node-version of 12.x

Now we've got Node.js and NPM, we need to install the dependencies for our action to run. We do this with the npm install command like so:

main.yml

- name: Install Dependencies
  run: npm install

And lastly, we will configure our action to run with the following:

main.yml

- name: Run Action
  uses: ./
  env:
    TWITTER_CONSUMER_KEY: ${{ secrets.TWITTER_CONSUMER_KEY }}
    TWITTER_CONSUMER_SECRET: ${{ secrets.TWITTER_CONSUMER_SECRET }}
    TWITTER_ACCESS_KEY: ${{ secrets.TWITTER_ACCESS_KEY }}
    TWITTER_ACCESS_SECRET: ${{ secrets.TWITTER_ACCESS_SECRET }}
  with:
    bio: ${{ github.event.inputs.bio }}

In this last step we're doing quite a few things:

We're setting a name for our action
We set a uses to be ./ which means we want to run the action at the root of our Repo (our action.yml)
We set the env, these are the different environment variables we want to pass to the application. We use ${\{ ... }\} to mean that it's a variable, and we use secrets.variable to access the variable from the secrets we configured in the repository
We use the with to set the bio from the input we give to the workflow on our workflow_dispatch event. GitHub exposes this in the github.event.inputs.bio variable

With the Run Action step added, we've got a full workflow. The overall main.yml file should have the following:

main.yml

name: Run Twitter Bio Action
on:
  workflow_dispatch:
    inputs:
      bio:
        description: 'Twitter Bio'
        required: true

jobs:
  update-bio:
    runs-on: ubuntu-latest
    name: Update Twitter Bio
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Node.js
        uses: actions/setup-node@v1
        with:
          node-version: 12.x
      - name: Install Dependencies
        run: npm install
      - name: Run Action
        uses: ./
        env:
          TWITTER_CONSUMER_KEY: ${{ secrets.TWITTER_CONSUMER_KEY }}
          TWITTER_CONSUMER_SECRET: ${{ secrets.TWITTER_CONSUMER_SECRET }}
          TWITTER_ACCESS_KEY: ${{ secrets.TWITTER_ACCESS_KEY }}
          TWITTER_ACCESS_SECRET: ${{ secrets.TWITTER_ACCESS_SECRET }}
        with:
          bio: ${{ github.event.inputs.bio }}

Run the Action

To run the action, we first need to get everything to GitHub. From Repo directory, in the terminal, run the following commands:

git add .
git commit -m "I made a GitHub Action!"
git push

Then, go to your repository on GitHub and click on the Actions tab. You should then see your Workflow listed. Click on your workflow name, and then the Run workflow dropdown. Fill in your Twitter Bio, wait for the workflow to complete and look at your Twitter profile!

From GitHub, you are also able to inspect and view any logs or errors from a Workflow run. If the workflow fails you can also take a look at the output that was thrown by the step that resulted in the failure

Summary

And that's about it. We've taken a look at how you can use GitHub actions to automate a pretty silly task, but there's a lot more to using GitHub Actions, and the sky's the limit in terms of what you can use them for

Overall, GitHub actions aren't too different from similar options like Azure Pipelines, Jenkins, or any other CI service. What makes them so useful is how easily they hook into the source control system and how well they work and are supported within the GitHub ecosystem

That all being said, they're not the easiest things to configure and it can take a while to get them right if you're doing something especially complex. Overall they're pretty cool and are a pretty good place to get started with automating tasks and working with things like continuous integration and deployments and I'd definitely recommend giving them a shot

Scripting with FSharp

Tue, 13 Oct 2020 00:00:00 GMT

Scripting with FSharp

Before you can use the following, you will need the .NET Core SDK installed

Open the F# Interactive Console

To open an F# interactive console using the dotnet CLI. You can run the following command:

dotnet fsi

Note that to end statements/execute in the F# Interactive console use ;; at the end of a line or section of code

Once in the console, running #help;; from the fsi console to view the help menu, and #quit;; to quit the interactive session

Additionally, you can write multi-line F# code as well as just single line expressions. Each expression should be terminated with ;;. For example, you can write a function that will print some data into the console:

let printer s =
    printfn s
;;

Next, you can call the function with:

printer "Hello World";;

Which will execute the above code and output Hello World

Run an F# Script

F# scripts can be run using the dotnet fsi command as well, followed by the path to an F# script file:

dotnet fsi ./myscript.fsx

The Gatsby Migration, pt.3 - Smart Pages

Sun, 15 Mar 2020 00:00:00 GMT

The Gatsby Migration, pt.3 - Smart Pages

Introduction

So far we've created the initial react application as with a few routes for our Home, Blog, and 404 pages. In this post we'll look at how we can set up our Post component to render our pages dynamically based on the JSON data we have. We'll also extend this so that we can have some more content in a markdown file that we'll parse and add to our Gatsby data

Creating the initial React App
Rendering the "Dumb" pages with Gatsby
Rendering the "Smart" page with Gatsby (This post)

Setting Up

We're going to make our data a little more complex by creating two additional markdown files in our static/posts directory to enable us to have more content with each post

Create the following markdown files in the application and align the names with our post-1.json and post-2.json files:

static/posts/post-1.md
static/posts/post-2.md

Gatsby Plugins

To read the data from our files we're going to do the following:

Use the gatsby-source-filesystem to read our files into the Gatsby Data Layer
Define our own plugin that can read the file content, parse the markdown, and add it into the data layer

Reading the File Metadata

To read our file data we will need to first install the gatsby-source-filesystem plugin. Plugins in Gatsby enable us to ingest or transform data in our application. We then make use of GraphQL to query the data from the relevant component

Install the gatsby-source-filesystem plugin with:

yarn add gatsby-source-filesystem

And then add the plugin configuration to the gatsby-node.js file into the plugins array:

gatsby-node.js

{
    resolve: `gatsby-source-filesystem`,
    options: {
        name: `content`,
        path: `${__dirname}/static/posts`,
    },
}

This will read all the data from our posts directory into the filesystem. We can now start the application back up with yarn start and navigate to http://localhost:8000/__graphql in our browser to view the GraphQL data. We should be able to see the GraphiQL interface

From the GraphiQL interface run the following query to see the data from the files in our directory:

query PostData {
  allFile {
    nodes {
      name
      extension
      absolutePath
    }
  }
}

This should yield the following JSON with our file meta data in it:

{
  "data": {
    "allFile": {
      "nodes": [
        {
          "name": "post-1",
          "extension": "json",
          "absolutePath": "C:/repos/cra-to-gatsby/static/posts/post-1.json"
        },
        {
          "name": "1",
          "extension": "jpg",
          "absolutePath": "C:/repos/cra-to-gatsby/static/posts/1.jpg"
        },
        {
          "name": "post-1",
          "extension": "md",
          "absolutePath": "C:/repos/cra-to-gatsby/static/posts/post-1.md"
        }
        // file 2 data
      ]
    }
  }
}

Processing the Files

Now that we have our metadata for each file in the file system, we're going to create a plugin that will allow us to read the file data and add it the GraphQL data layer

In order to do this, create a plugins directory in the root folder. Inside of the plugins directory create a folder and folder for our plugin.

Create a new folder in the plugins directory with another folder called gatsby-transformer-postdata

From this directory run the following commands to initialize and link the yarn package:

plugins/gatsby-transformer-postdata

yarn init -y
yarn link

We'll also add the showdown package which will allow us to convert the markdown into the HTML so we can render it with our Post component

yarn add showdown

And then create an gatsby-node.js file in this directory with the following content:

/plugins/gatsby-transformer-postdata/gatsby-node.js

const fs = require('fs')
const crypto = require('crypto')
const showdown = require('showdown')

exports.onCreateNode = async ({ node, getNode, actions }) => {
  const { createNodeField, createNode } = actions

  // we'll process the node data here
}

This exposes the onCreateNode Gatsby API for our plugin. This is what Gatsby calls when creating nodes and we will be able to hook into this to create new nodes with all the data for each respective post based on the created file nodes

From the onCreateNode function we'll do the following to create the new nodes:

Check if it is a markdown node
Check if the JSON file exists
Read file content
Parse the metadata into an object
Convert the markdown to HTML
Get the name of the node
Define the data for our node
Create the new node using the createNode function

gatsby-transformer-postdata/gatsby-node.js

const fs = require('fs')
const crypto = require('crypto')
const showdown = require('showdown')

exports.onCreateNode = async ({ node, actions, loadNodeContent }) => {
  const { createNodeField, createNode } = actions

  // 1. Check if it is a markdown node
  if (node.internal.mediaType == 'text/markdown') {
    const jsonFilePath = `${node.absolutePath.slice(0, -3)}.json`

    console.log(jsonFilePath)

    // 2. Check if the JSON file exists
    if (fs.existsSync(jsonFilePath)) {
      // 3. Read file content
      const markdownFilePath = node.absolutePath
      const markdownContent = fs.readFileSync(markdownFilePath, 'utf8')
      const jsonContent = fs.readFileSync(jsonFilePath, 'utf8')

      // 4. Parse the metadata into an object
      const metaData = JSON.parse(jsonContent)

      // 5. Convert the markdown to HTML
      const converter = new showdown.Converter()
      const html = converter.makeHtml(markdownContent)

      // 6. Get the name of the node
      const name = node.name

      // 7. Define the data for our node
      const nodeData = {
        name,
        html,
        metaData,
        slug: `/blog/${name}`,
      }

      // 8. Create the new node using the `createNode` function
      const newNode = {
        // Node data
        ...nodeData,

        // Required fields.
        id: `RenderedMarkdownPost-${name}`,
        children: [],
        internal: {
          type: `RenderedMarkdownPost`,
          contentDigest: crypto
            .createHash(`md5`)
            .update(JSON.stringify(nodeData))
            .digest(`hex`),
        },
      }

      createNode(newNode)
    }
  }
}

From the root directory you can clean and rerun the application:

yarn clean
yarn start

Now, reload the GraphiQL at http://localhost:8000/__graphql and run the following query to extract the data we just pushed into the node:

query AllPostData {
  allRenderedMarkdownPost {
    nodes {
      html
      name
      slug
      metaData {
        body
        image
        title
      }
    }
  }
}

This should give us the relevant post data:

{
  "data": {
    "allRenderedMarkdownPost": {
      "nodes": [
        {
          "html": "<p>Hello here is some content for Post 1</p>\n<ol>\n<li>Hello</li>\n<li>World</li>\n</ol>",
          "name": "post-1",
          "slug": "/blog/post-1",
          "metaData": {
            "body": "Hello world, how are you",
            "image": "/posts/1.jpg",
            "title": "Post 1"
          }
        },
        {
          "html": "<p>Hello here is some content for Post 2</p>\n<ol>\n<li>Hello</li>\n<li>World</li>\n</ol>",
          "name": "post-2",
          "slug": "/blog/post-2",
          "metaData": {
            "body": "Hello world, I am fine",
            "image": "/posts/2.jpg",
            "title": "Post 2"
          }
        }
      ]
    }
  }
}

Create Pages

Now that we've got all our data for the pages in one place we can use the onCreatePages API to create our posts, and the Post component to render the pages

Setting Up

Before we really do anything we need to rename the Blog.js file to blog.js as well as create the src/components directory and move the Post.js file into it, you may need to restart your application again using yarn start

Create Pages Dynamically

In our site root create a gatsby-node.js file which exposes an onCreatePages function:

gatsby-node.js

const path = require('path')

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions
}

From this function we need to do the following:

Query for the PostData using the graphql function
Create a page for each renderedMarkdownPost

gatsby-node.js

const path = require('path')

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions

  // 1. Query for the PostData using the `graphql` function
  const result = await graphql(`
    query AllPostData {
      allRenderedMarkdownPost {
        nodes {
          html
          name
          slug
          metaData {
            body
            image
            title
          }
        }
      }
    }
  `)

  result.data.allRenderedMarkdownPost.nodes.forEach((node) => {
    // 2. Create a page for each `renderedMarkdownPost`
    createPage({
      path: node.slug,
      component: path.resolve(`./src/components/Post.js`),
      context: node,
    })
  })
}

Render the Page Data

From the Post component we need to:

Export the query for the data
Get the data for the Post
Render the data

components/post.js

import { graphql } from 'gatsby'
import App from '../App'

const Post = ({ data }) => {
  // 2. Get the data for the Post
  const postData = data.renderedMarkdownPost

  // 3. Render the data
  return (
    <App>
      <div className="Post">
        <p>
          This is the <code>{postData.slug}</code> page
        </p>
        <h1>{postData.metaData.title}</h1>
        <div
          className="markdown"
          dangerouslySetInnerHTML={{ __html: postData.html }}
        ></div>
      </div>
    </App>
  )
}

export default Post

// 1. Export the `query` for the data
export const query = graphql`
  query PostData($slug: String!) {
    renderedMarkdownPost(slug: { eq: $slug }) {
      html
      name
      slug
      metaData {
        body
        image
        title
      }
    }
  }
`

From the Post component above we use the slug to determine which page to render, we also set the HTML content for the markdown element above using the HTML we generated. We also now have our pages dynamically created based on the data in our static directory

You can also see that we have significantly reduced the complexity in the Post component now that we don't need to handle the data fetching from the component

If you look at the site now you should be able to navigate through all the pages as you'd expect to be able to

Summary

By now we have completed the the entire migration process - converting our static and dynamic pages to use Gatsby. In order to bring the dynamic page generation functionality to our site we've done the following:

Used the gatsby-source-filesystem plugin to read our file data
Created a local plugin to get the data for each post and convert the markdown to HTML
Use the onCreatePages API to dynamically create pages based on the post data
Update the Post component to render from the data supplied by the graphql query

And that's about it, through this series we've covered most of the basics on building a Gatsby site and handling a few scenarios for processing data using plugins and rendering content using Gatsby's available APIs

Nabeel Valley

Code for Noobs

Sun, 17 May 2020 00:00:00 GMT

Code for Noobs

So I've been meaning to write an introductory post about the basics of JavaScript for quite some time. A few weeks ago I managed to finally get something going on this Twitter thread (@not_nabeel)

Let's get into it

0. What is Programming

Programming is our way of telling a computer to do things. This can be anything from checking our spelling to making trippy digital art

The basics of programming are the same in most languages and doesn't involve much math or binary (0010101) like a lot of people seem to think. Programming languages usually make use of a very small subset of words and concepts of a normal human language like English

The language I'm going to be using is called JavaScript but many of these concepts are the same in most other programming languages

1. What is JavaScript?

JavaScript is the language used to add interactivity to a website but it can also be used to do pretty much anything you like. JavaScript runs in all modern web browsers and can be accessed through your browser's developer tools, but it can also be used on other devices using something like @nodejs

For our purposes though, we don't need to worry too much about how that all works right now. To get started we'll be using tools like @CodePen, @replit and @glitch which give you text editor and a place to run your code

You'll see some code samples embeded in the

I'll be posting links to the code on these sites as I go along

2. Text Data

In a program, we can store different types of data (known as "data types")

When we store text we use a data type called string. In JavaScript a string is just text surrounded in either double quotes, single quotes, or `these things` (which are known as backticks):

'i am text'

'i am also text'

;`me too.`

;`unlike the others,
  i can b
multiple lines
lllooonnnggg
    and have random spaces`

3. Variables

If we want to keep our data for use at a later stage we need to give it a name, otherwise how do we know what data we're trying to use right? To give a piece of data (or text, in our case) a name we create a variable

We create a variable using let or const along with a variable name and the data that we want to give the name to :

View Code

// this code shouldn't do anything if you run it
// we are just creating some variables

let myText = 'Hello World'

const myOtherText = 'Bye World'

We use let for data whose data may change and const for data whose data won't you can remember this like: const = constant

Note that it is also possible to use the keyword var to create a variable. This is something that is left over from older versions of JavaScript and can have some effects that are better to just avoid (if you're interested you can read this article about how it impacts variable scope but it is a bit of a more challenging concept to understand)

When creating a variable, there are a few rules we need to follow:

Variable names must not contain spaces or stuff like !@#%%^&\*(), underscores and dollar signs are okay though
Variable names must not be a word the language has set aside for something else. e.g const or let
Variable names must not start with a number
Variable names should be descriptive. while names like "x" and "blah" are allowed, if they have no meaning in the context it's better to opt for something people will understand

4. Printing Data

Languages have ways we can show data to a user (or programmer). In JavaScript this is the "console.log" function. We'll discuss functions later on but for now know that when we give them data, and they do stuff

View Code

// print the data
console.log('Bob Smith') //prints -> Bob Smith

// print a variable
const jenny = 'Jenny Smith'
console.log(jenny) // prints -> Jenny Smith

5. Numbers

Numbers are another type of data in JavaScript, to store a number we can just write the number. using numbers we can also do things like getting fat or running from the po-lice

- is plus
- is minus
* is multiply
/ is divide

View Code

const myWeight = 100
const food = 100 + 50

const myNewWeight = myWeight + food

console.log(myNewWeight) // print -> 250

const myWeightAfterRunnin = myWeight - myWeight * 0.2

console.log(myWeightAfterRunnin) // print -> 80

But, JavaScript lets us add do maths with anything. so maybe like - don't do this:

View Code

const dontDoThis = 'apple' * 'pineapple'

console.log(dontDoThis) // print -> NaN (Not a Number)

6. Arrays

Arrays are how we store a set of data. an array can have different data types in it, but usually, we want to be storing the same stuff in an array. we make an array by wrapping our items in [ ] and separating each item with a comma (,)

View Code

const myFriends = ["i", "don't", "have", "any"]

const thingsToRemember = ["uhmm", 42, 12, "idk, i guess i forgot"]

const multipleLines = [
"arrays don't have to be",
"on a single",
"line"
]

If we want to get a specific element in an array we can use the variable name with the index (position) of the element. the index starts at 0. we will get undefined (I'll explain this in a bit) if we try to get a value at an index that does not exist

View Code

  const myData = [
  "zeroeth index",
  "first index",
  "second index",
  "third index"
]

console.log(myData[0]) // print -> zeroeth index
console.log(myData[1]) // print -> first index
console.log(myData[2]) // print -> second index
console.log(myData[3]) // print -> third index

// indexes that do not exist in our array
console.log(myData[-1]) // print -> undefined
console.log(myData[100]) // print -> undefined

7. Booleans

So far we've looked at strings and numbers, we have an even more basic data type called a boolean. a boolean is a value that can either be true or false. when creating a variable for a boolean we do not wrap the value in quotes

View Code

var myTrueBool = true

var myFalseBool = false

console.log(myTrueBool)
console.log(myFalseBool)

8. Complex Data

If we want to store data that are more complex than the ones we've seen above, we can make use of "objects". use { } around our data to group more basic data. these use keys as names for values in the object

View Code

const jenny = {
  id: 123,
  name: 'Jenny',
  surname: 'Smith',
  age: "idk, can't ask a woman that",
  favouriteBooks: [
    'To not kill a Mockingbird',
    'Why are books so voilent, yoh',
  ],
}

// we can get specific values using their keys
const jennyName = jenny.name
const jennyBooks = jenny.favouriteBooks

9. Undefined and Null

the values undefined and null have a special meaning.

undefined is a representation for a variable that does not have a value
null is a value of "nothing"

For example:

What's wrong with chocolate? null
What's a flobuir? undefined

10. Functions

Functions are a way we can group code for a specific set of instructions, usually with some end purpose. In JavaScript we define a function in one of two ways: using the word function or with the fat arrow syntax

View Code

// using the "function" keyword
// this creates a variable called "myFunc1"
function myFunc() {
  const stuff = 'I am Function'
  console.log(stuff)
}

// we can also create the variable using
// "let" or "const" like we do for other variables
const myLetFunc = function () {
  const stuff = 'I am another Function'
  console.log(stuff)
}

// fat arrow function
const myFatFunc = () => {
  console.log('Phat Funk')
}

// single line fat-arrow functions don't need the {}
const mySingleLineFunct = () => console.log('single line')

To use a function we simply write the name of the function and then add () at the end, provided the function doesn't take any data (also known as parameters)

View Code

const sayHello = function () {
  console.log('Hello')
}

// "call" the function
sayHello() // prints -> Hello

const sayBye = () => console.log('BYEE')

// "call" the function
sayBye() // prints -> BYEE

Remember earlier I said that functions take data and do stuff? For us to give a function data we need to tell it what variables to store that data as when we call it. We do this by having the data in the () when we create (or 'declare') our function

View Code

function sayHello(name) {
  console.log('Hello ' + name)
}

// use the function
sayHello('Bob') // print -> Hello Bob

const sayBye = (name) => console.log('Bye ' + name)

//use the function
const name = 'Jenny'
sayBye(name) // print -> Bye Jenny

When computers are following our code they do so from the top down, so the first line is handled by the computer before the second. the second before the third, etc. functions are a way for us to reuse code that was written on a line higher up in our code. You may notice that we use the variable name in a lot of different places above. this uses scope and broadly means any variable name created in a function (or pretty much anywhere between { }) is only available within that section

Scope complicated, you can read more about it here

Functions can also take multiple parameters, this is done by listing the parameters between the () and separating them by commas. the order that we list them when we create our function is the same as the order we need to put them when using the function

View Code

const sayBye = (firstName, lastName) => {
  console.log('Bye ' + firstName + ' ' + lastName)
}

// use the function
const jennyName = 'Jenny'
const jennySurname = 'Smith'
sayBye(jennyName, jennySurname) // print -> Bye Jenny Smith

Functions are also able to give us back a value after doing some stuff. They do this by using the return keyword which tells it what to give back. A function stops processing when it sees the keyword return - anything after it in a function is ignored

View Code

const addNumbers = (num1, num2) => {
  const result = num1 + num2
  return result
}

const mySum = addNumbers(1, 2)

console.log(mySum) // print -> 3

Summary

That's it for now, I'll definitely be updating this post/series as things are added

Nabeel Valley

Custom Attributes in C# Web Controllers

Thu, 17 Dec 2020 00:00:00 GMT

Custom Attributes in C# Web Controllers

Implementing an attribute for a WebAPI or class in C# can help to reduce duplication and centralize parts of the application logic. This could be used for a variety of tasks such as logging information when methods are called as well as managinng authorization

In this post I'm going to cover the following:

Attribute Types and Execution Order

There are a few different attribute types that we can handle on a WebAPI that provide us with the ability to wrap some functionality around our endpoints, below are some of the common attributes that we can implement and the order in which they execute (StackOverflow)

Authorization - IAuthorizationFilter
Action - IActionFilter
Result - IResultFilter
Exception - IExceptionFilter

IActionFilter

The IActionFilter executes before and after a method is executed and contains two different methods for doing this, namely the OnActionExecuting and OnActionExecuted methods respectively. A basic implemtation of IActionFilter would look like this:

namespace CSharpAttributes.Attributes
{
  public class LogStatusAttribute : Attribute, IActionFilter
  {
    public LogStatusAttribute()
    {
      Console.WriteLine("Attribute Initialized");
    }

    public void OnActionExecuting(ActionExecutingContext context)
    {
      Console.WriteLine("OnActionExecuting");
    }

    public void OnActionExecuted(ActionExecutedContext context)
    {
      Console.WriteLine("OnActionExecuted");
    }
  }
}

This can then be implemented on a controller method with a [LogStatus] attribute:

[LogStatus]
[HttpGet]
public IEnumerable<WeatherForecast> Get()
{
  Console.WriteLine("Executing Get");
  return data;
}

The order of logging which we see will be as follows:

Attribute Initialized when the controller is instantiated
OnActionExecuting when the controller is called
Executing Get when the controller is executed
OnActionExecuted when the controller is done executing

IAuthorizationFilter

The IAuthorizationFilter executes as the first filter on a controller's method call

namespace CSharpAttributes.Attributes
{
  public class CustomAuthorizeAttribute : Attribute, IAuthorizationFilter
  {
    public CustomAuthorizeAttribute()
    {
      Console.WriteLine("Attribute Initialized");
    }

    public void OnAuthorization(AuthorizationFilterContext context)
    {
      Console.WriteLine("OnAuthorization");
    }
  }
}

This can then be implemented on a controller method with a [CustomAuthorize] attribute:

[CustomAuthorize]
[HttpGet]
public IEnumerable<WeatherForecast> Get()
{
  Console.WriteLine("Executing Get");
  return data;
}

The order of logging which we see will be as follows:

Attribute Initialized when the controller is instantiated
OnAuthorization when the controller is called
Executing Get when the controller is executed

Modify Response Data

An attribute's context parameter gives us ways by which we can access the HttpContext as well as set the result of a method call so that it can be handled down the line. For example, we can implement our CustomAuthorize attribute with the following:

public void OnAuthorization(AuthorizationFilterContext context)
{
  if (!context.HttpContext.Request.Headers.ContainsKey("X-Custom-Auth"))
  {
    context.Result = new UnauthorizedResult();
  }

  Console.WriteLine("Attribute Called");
}

This will mean that if we set the context.Result in our method then the controller will not be executed and the endpoint will return the UnauthorizedResult early. You can also see that we're able to access things like the HttpContext which makes it easy for us to view the request/response data and do things based on that

Attribute on a Class

Note that it's also possible to apply the above to each method in a class by adding the attribute at the top of the class declaration:

[ApiController]
[LogStatus]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
  ...

Attributes with Input Parameters

We are also able to create attributes that enable the consumer to modify their behaviour by taking input parameters to the constructor, we can update our LogStatus attribute to do something like add a prefix before all logs:

namespace CSharpAttributes.Attributes
{
  public class LogStatusAttribute : Attribute, IActionFilter
  {
    private readonly string _prefix;

    public LogStatusAttribute(string prefix = "")
    {
      _prefix = prefix;
      Console.WriteLine("Attribute Initialized");
    }

    public void OnActionExecuted(ActionExecutedContext context)
    {
      Console.WriteLine(_prefix + "OnActionExecuted");
    }

    public void OnActionExecuting(ActionExecutingContext context)
    {
      Console.WriteLine(_prefix + "OnActionExecuting");
    }
  }
}

Then, applying to our controller method like so:

[LogStatus("WeatherController-Get:")]
[HttpGet]
public IEnumerable<WeatherForecast> Get()
{
  Console.WriteLine("Executing Get");
  return data;
}

So the new output will look like so:

Attribute Initialized when the controller is instantiated
WeatherForecast-Get:OnActionExecuting when the controller is called
Executing Get when the controller is executed
WeatherForecast-Get:OnActionExecuted when the controller is done executing

Attribute Setting at Class and Method Level

Since an attribute can be implemented at a class and method level it's useful for us to be able to implement it at a class and the override the behaviour or add behaviour for a specific method

We can do this by setting the attribute inheritence to false

Updating out LogStatusAttribute we can add the AttributeUsage Attribute as follows:

namespace CSharpAttributes.Attributes
{
  [AttributeUsage(AttributeTargets.All, Inherited = false)]
  public class LogStatusAttribute : Attribute, IActionFilter
  {
    ...

This means that we can independently apply the attribute at class and method levels, so now our controller can look something like this:

namespace CSharpAttributes.Controllers
{
  [ApiController]
  [Route("[controller]")]
  [LogStatus("WeatherForecast:")]
  public class WeatherForecastController : ControllerBase
  {
    [LogStatus("Get:")]
    [HttpGet]
    public IEnumerable<WeatherForecast> Get()
    {
      Console.WriteLine("Executing Get");
      return data;
    }
  }
}

Which will output the logs as follows:

Attribute Initialized when the controller is instantiated
WeatherForecast:OnActionExecuting when the class is called
Get:OnActionExecuting when the controller is called
Executing Get when the controller is executed
Get:OnActionExecuted when the controller is done executing
WeatherForecast:OnActionExecuted when the class is done executing

The Gatsby Migration, pt.1 - Setting the Scene

Tue, 21 Jan 2020 00:00:00 GMT

The Gatsby Migration, pt.1 - Setting the Scene

Introduction

Lately I've been a little concerned with my current SPA approach on my personal site as well as a few others. More specifically the high initial load time due to the calls to the backend to retrieve content

With the aim of solving this problem I've spent a lot of time looking at and playing with Static Site Generators.Foreword: they're all a lot more complicated than one would think

So for a static site the "static" content only changes so often, based on this we can generate page content with the data we're planning to load in - that's what we're going to try to do

Now we'll be starting off with a React app generated with create-react-app so that we can have a starting point for our Gatsby site as well as understanding how we can approach some of the challenges when switching over to a site generator like Gatsby

For the sake of completeness, this series will be broken into four posts covering the following:

Creating the initial React App (This post)
Rendering the "Dumb" pages with Gatsby
Rendering the "Smart" page with Gatsby

The React App

First we're going to be starting off with a new React app that we will work on changing into a Gatsby one parts 2 and 3, we'll then focus on using plugins to enhance our content in 4

For this part we'll build a basic React app that has the following:

Two "hard-coded" pages and a 404 page
A dynamic page with an API call to retrieve data
Overall app layout with child routes for 1 - 3

To get started, we'll create a fresh React App:

npx create-react-app gatsby-to-be

And add the react-router

yarn add react-router-dom

Running this command should set up the application, if you don't know much about React I'd suggest taking a look at the documentation

Next cd gatsby-to-be and run yarn start, you should be able to visit the application in your browser at http://localhost:3000/

Looking at the generated files we have a public directory with some icons, an index.html file into which our React application will run once built, and a src directory that has the application code. The index.js file is what loads the application into the DOM and the App.js file which is the main component for our application

Hard Coded Pages

We will create the following three hard-coded pages in the src/pages directory

These pages are just React components that we will assign Routes to.

The pages we are using are known as functional components because they are javascript functions that return JSX

If we intend to use JSX in a file we need to ensure that we import React. The other component we are importing is the Link component which is a lot like a normal HTML a tag but with some special functionality to make the client-side navigation work

Blog.js

import React from 'react'
import { Link } from 'react-router-dom'

const Blog = () => (
  <div className="Blog">
    <h1>Blog</h1>
    <p>This is the Blog page</p>
    <div>
      <Link to="/blog/post-1">Post 1</Link>
    </div>
    <div>
      <Link to="/blog/post-2">Post 2</Link>
    </div>
  </div>
)

export default Blog

Additionally we have the Home.js and NotFound.js files which are similar to the Blog.js file we created

Home

Home.js

import React from 'react'

const Home = () => (
  <div className="Home">
    <h1>Home</h1>
    <p>This is the Home page</p>
  </div>
)

export default Home

Not Found

NotFound.js

import React from 'react'

const NotFound = () => (
  <div className="NotFound">
    <h1>404</h1>
    <p>Page Not Found</p>
  </div>
)

export default NotFound

Dynamic Post Page

Next up we'll create a component that can render out content for a blog post. This will consist of a few hooks which are react functions that we can use to sort of control the data in a function

The Post component will:

Display a loading indicator initially
Figure out what post we're trying to render based on the URL
Retrieve a JSON file from the public directory based
Set the component state after reading the file
Display the content from the file in a JSX template

The useState hook is used to initialize the state the component, in this case using the hasError and data variables, as well as providing the functions necessary for updating those in the form of setHasError and setData respectively

We use fetch in the useEffect hook to retrieve the data from the public directory. The useEffect hook allows us to pass a function that will be called to update side effects. The second input, in our case [] is the array of objects that, when are changed, we want the hook to run - since we only want it to run once and don't care about any other state changes we pass in an empty array for this value

Post.js

import React, { useEffect, useState } from 'react'

const Post = ({ match }) => {
  const slug = match.params.slug

  const [hasError, setHasError] = useState(false)
  const [data, setData] = useState(null)

  useEffect(() => {
    const fetchData = async () => {
      try {
        const res = await fetch(`/posts/${slug}.json`)
        const json = await res.json()
        setData(json)
      } catch (error) {
        console.log(error.toString())
        setHasError(true)
      }
    }

    fetchData()
  }, [])

  return (
    <div className="Post">
      <p>
        This is the <code>{slug}</code> page
      </p>
      {data ? (
        <div className="content">
          <h1>{data.title}</h1>
          <p>{data.body}</p>
          <img src={data.image} alt="" />
        </div>
      ) : hasError ? (
        <div className="error">
          <h1>Error</h1>
          <p>{hasError}</p>
        </div>
      ) : (
        <p>Loading .. </p>
      )}
    </div>
  )
}

export default Post

In the above component we're making use of the following pattern to decide what to render conditionally:

If the data is loaded then show the data
Else if there is an error then show the error data
Otherwise show a loading message

The way we are rendering the Post component with a parameter for match. The match parameter will be passed in as a prop from the Router that we will configure next, this allows us to use the slug from the URL to retrieve the content for the page

The two data files we have to pull content from in the public/posts directory are post-1.json and post-2.json

Post 1 - JSON

post-1.json

{
  "title": "Post 1",
  "body": "Hello world, how are you",
  "image": "/posts/1.jpg"
}

Post 2 - JSON

post-2.json

{
  "title": "Post 2",
  "body": "Hello world, I am fine",
  "image": "/posts/2.jpg"
}

The images posts/1.jpg and posts/2.jpg can also just be any images in that directory

App Layout and Routes

Lastly, we'll specify our application layout with the relevant routes in the App.js file, referencing the components we have created, we do this using the BrowserRouter. When we switch the project over to a Gatsby one, the App.js file will be converted into our Layout component to wrap our different pages

We use the Router component and the page inside of it, this essentially handles Routing via the Link components. Next we have a div as a wrapper for our component as well as a header, nav, and main tags to organise the page

The Route component takes in the component that we would like to display for a given route, and the Switch helps us to ensure that only route is actively being fisplayed at a time. The Switch will navigate from the first to last Route and render the first one that matches the given path

Fow now, our App.js file is as follows:

import React from 'react'
import './App.css'
import { BrowserRouter as Router, Link, Switch, Route } from 'react-router-dom'
import Home from './pages/Home'
import Blog from './pages/Blog'
import Post from './pages/Post'
import NotFound from './pages/NotFound'

const App = () => (
  <Router>
    <div className="App">
      <header>
        <nav>
          <h1>My Website Title</h1>
          <Link to="/">Home</Link>
          <Link to="/blog">Blog</Link>
        </nav>
      </header>
      <main>
        <Switch>
          <Route exact path="/" component={Home}></Route>

          <Route exact path="/blog" component={Blog}></Route>

          <Route exact path="/blog/:slug" component={Post}></Route>

          <Route path="/*" component={NotFound}></Route>
        </Switch>
      </main>
    </div>
  </Router>
)

export default App

In the above component we can see that where we are rendering the Post component we have a parameter in the route called slug, this parameter will be passed in to the Post component as part of the match object

Summary

We have built a fairly simple application that makes use of both static and data-based pages, we have also tied all of these together using the App component and a Router with the following routes:

/ which will render the Home component
/blog which will render the Blog component
/blog/:slug which will render the Post component and retrieve the data based on the given slug
/* which will match any other routes and render the NotFound component

In the next post we're going to look at how to take what we have so far and transform this application into a Gatsby one, but for now it may be useful to think about what kind of steps we may need to take if we'd like to make this a static website based on the way our current routes work

Nabeel Valley

Mongo DB in Docker

Tue, 22 Sep 2020 00:00:00 GMT

Mongo DB in Docker

Considerations

To run MongDB in a Docker Container there are a few things to take note of:

You may need to configure auth
Volume storage to be set up

From Terminal

To run a Docker container using a single command in the terminal you can run the following command:

docker run -d -p 27017:27017

You can also specify additional information like the volumes you would like to use using flags when running:

docker run -d -p 27017:27017

From Compose

More info can be found here

You can also run a MongoDB Container with Compose which may be a bit easier:

docker-compose.yml

version: '3.3'

services:
  # mongonode0:27017
  mongo0:
    image: mongo
    hostname: mongo0
    container_name: mongo0
    restart: always
    ports:
      - '37000:27017'
    volumes:
      - ./mongo0-data:/data/db
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: password

Split an Array into Segments

Wed, 23 Sep 2020 00:00:00 GMT

Split an Array into Segments

It can sometimes be useful to split an array into a separated set of rows or columns, such as when trying to separate elements into buckets while maintaing order.

For example, given the following input:

const input = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

With the desired output array as follows, in which we split elements into different rows, while maintaing a top-down/left-right ordering:

const output = [
  [1, 4, 7, 10],
  [2, 5, 8],
  [3, 6, 9],
]

We can accomplish this using a function like so:

const transform = (arr, segments) =>
  [...new Array(segments)].map((_, outerPos) =>
    input.filter((el, innerPos) => innerPos % segments === outerPos)
  )

Which can then be used with:

const rows = 3
const transformedData = transform(input, rows)

Generate data for a Postman request

Tue, 24 Nov 2020 00:00:00 GMT

Generate data for a Postman request

When making requests with Postman to test an API it is often useful to have generated data, for example when creating users in a backend system you may want the ability to get custom user data

The method we will use to do this is by setting Postman environment variables in the Pre-request Script of a Postman request

Firstly, we can write some basic logging in the Pre-request script section on Postman and view the result in the Postman console:

const name = 'Nabeel'

console.log(`Hello ${name}`)

Additionally, postman gives us some functions to set and get environment variables, so we can set the name as an environment variable like so:

const name = 'Nabeel'

pm.environment.set('name', name)

And log it like so:

console.log(pm.environment.get('name'))

Aditionally, we can make HTTP Requests using pm.sendRequest which then takes a callback for what we want to do after the request, we can use the randomuser api to get a user:

pm.sendRequest('https://randomuser.me/api/', (err, res) => {
  const apiResponse = res.json()

  // do more stuff
})

Using this method, we can get some random user data and set it in the environment variables as follows:

pm.sendRequest('https://randomuser.me/api/', (err, res) => {
  const apiResponse = res.json()

  const user = apiResponse.results[0]

  const email = user.email
  const firstName = user.name.first
  const lastName = user.name.last

  pm.environment.set('email', email)
  pm.environment.set('firstName', firstName)
  pm.environment.set('lastName', lastName)
})

We can then use this in our request body using Postman's {\{}\} syntax. If we were making a JSON request, our body would look something like this:

POST: https://my-api.com/users

{
  "email": "{{email}}",
  "firstName": "{{firstName}}",
  "lastName": "{{lastName}}"
}

Postman will then populate the slots from the environment variables that we set automatically when making the request

Code in a Container

Sat, 25 Jul 2020 00:00:00 GMT

Code in a Container

Recently I'd started using Visual Studio Code's Remote Containers functionality for development and it's been really useful

The Remote Containers extension allows us to write code and develop applications within a virtualized environment that makes it easier for us to manage our development environment as well as more closely resemble our target deployment environment (if we're deploying to Docker or Kubernetes)

In this post, I'll take a look at what a Docker container is, why we would want to use one as a development environment, and how we can go about setting one up for VSCode

Prerequisites

If you intend to follow along with this post you'll need to have the following installed:

A Windows or Mac OS version capable of running Docker Desktop
Docker Desktop
Visual Studio Code
Visual Studio Code's Remote Containers Extension
Visual Studio Code's Docker Extension
Some familiarity with using the command line

Docker Containers

A Container, in this context, is a simple virtual machine that contains the code required to run an application with all its dependencies

A Docker container is built from a docker image and run by the docker command. I'll explain these as we go along

To check that Docker is installed correctly on your machine run the following command:

docker run hello-world

If your install is working correctly you should see something like this:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete

Digest: sha256:49a1c8800c94df04e9658809b006fd8a686cab8028d33cfba2cc049724254202
Status: Downloaded newer image for hello-world:latest

Hello from Docker

...

Docker Images

Docker images are typically used to run applications in a production-type environment, every Docker container we run needs to be based on an image, every running container is like an instance of an image - similar to how objects are an instance of a class

An image states what our container will need to be made of, what it depends on, and how it runs. We define how docker should build our image in a Dockerfile

We're going to go through some of the basics of Docker Images and Docker as would typically be done when creating a container to be run in production before we get into development containers so you've got an understanding of how this all works

To get started create a new folder and open it from Visual Studio Code and do the following:

Create an Application

We'll need a simple "hello-world" web server using Node.js, for the sake of example. You can, however, use any language (or Languages) you want when creating an application to run within Docker. You do not need to have any dependencies for the specific application or language installed on your computer, we will handle this using Docker

For our purpose, create a file called index.js with the following:

const http = require('http')

const requestListener = function (req, res) {
  res.writeHead(200)
  res.end('Hello, World!')
}

const serverListeningCallback = function () {
  console.log('Server started')
}

const server = http.createServer(requestListener)
server.listen(8080, serverListeningCallback)

You can see in the above on the last line that the application will listen on port 8080, just keep this in mind

We don't need to run this file as yet, but if we want, we can run this with the following command from our working directory:

node app.js

At this point our working directory should look like this:

working-directory
|__ index.js

Create a Dockerfile

There are a few steps that are the same for most Dockerfiles you'll be building:

A Base Image that your container/image should use, in our case node:12, which has node and npm preinstalled
Copy all the code in the current (.) directory
Define your runtime port/ports (in the case of a web application)
The command that will be run to start the application

Any line starting with a # is a comment, Docker will ignore these

Dockerfile

## step 1 - FROM baseImage
FROM node:12

## step 2 - COPY source destination
COPY . .

## step 3 - EXPOSE port
EXPOSE 8080

## step 4 - CMD stratupCommandArray
CMD ["node", "app.js"]

At this point our working directory should look like this:

working-directory
|__ index.js
|__ Dockerfile

We can build our image, based on the Dockerfile using the following docker command:

Note the . at the end of the command

docker build -t my-docker-app .

The above command can be broken down as follows:

docker build the command from the Docker CLI to build an image
-t my-docker-app says what we want our image to be called, in the above my-docker-app
. which is the directory in which the Dockerfile is located, in our case our current directory

We can then run the image we just built like so:

docker run -p 8080:8080 my-docker-app

docker run is the command from the Docker CLI to run a container
-p 8080:8080 is our port mapping, it is ordered as HOST_PORT:CONTAINER_PORT and allows us to say which port on our host we want to map to our container, the container port is the same port that our app listens on and is EXPOSEd in the Dockerfile
my-docker-app is the image tag we would like to run

Each time we change the app files for a container like above we need to rebuild the container before running, and that normally making changes to files during the image build or container run will not modify the original files on our computer

Now that the application is running on port 8080 you can open http://localhost:8080 in your browser and you should see your Hello World app running

When you're done with that you can go back to the terminal where the container was started and use ctrl + c to stop the container

If you've never used Docker before and have got everything running this far, congratulations! If you've got any questions you can comment below or hit me up on Twitter @not_nabeel

Moving swiftly along

Development Containers

So now that we understand a bit about containers and how we can go about using them in production, we'll look at why we may want to use them as a development environment

Why Develop in a Container

As developers, we are far too familiar with the "it runs on my machine" dilemma. Development environments can be wildly inconsistent between different developers or different operating systems, and ensuring that our development code runs easily on everyone's computer can be challenging

Containers can help us to explicitly define our development environment, our application dependencies, what networking relationships, and (potentially) what other sibling applications need to be running in development, like databases, or other application tiers

Visual Studio Code can help transport us into a container so that we work on our application in a well-defined environment, not just run our application within one while reducing the overall number of things we need to have installed on our computer

How to Develop in a Container

To develop in a Container using Visual Studio Code we will need to have:

What's important to note is that we don't need any of our application's runtime or development dependencies installed, like Node.js, these will all be handled by VSCode within our container

To configure our project for running in a container we need to first open the project folder (the folder we used previously) in Visual Studio Code

Use an Existing Dockerfile

Once open use the keyboard shortcut ctrl + shift + p to open the Command Palette and search for Remote-Containers: Add Development Container Configuration Files and click enter, you will then have an option to use the existing Dockerfile from Dockerfile which will generate a .devcontainer/devcontainer.json file

At this point our working directory should look like this:

working-directory
|__ .devcontainer
|   |__ devcontainer.json
|
|__ index.js
|__ Dockerfile

The .devcontainer/devcontainer.json file that was created will contain the following:

devcontainer.json

// For format details, see https://aka.ms/vscode-remote/devcontainer.json or this file's README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.128.0/containers/docker-existing-dockerfile
{
    "name": "Existing Dockerfile",

    // Sets the run context to one level up instead of the .devcontainer folder.
    "context": "..",

    // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
    "dockerFile": "..\\Dockerfile",

    // Set *default* container specific settings.json values on container create.
    "settings": {
        "terminal.integrated.shell.linux": null
    },

    // Add the IDs of extensions you want installed when the container is created.
    "extensions": []

    ...
}

The above file is the configuration for our development container, we can also allow VSCode to generate a Dockerfile which we'll look at later in the post

We'll stick to our simple Dockerfile for this post, but if you've got a different Dockerfile when running your application in Production and Development then you may need a different file in the dockerFile property below

Now that we've got a starting point we can add a little to our configuration so that everything is just right:

Change the name property to name our workspace (purely aesthetic)
Add a forwardPorts property to expose our application port to our localhost network, be sure to add the , after "extensions":[]

Once we make the above changes we should have this:

devcontainer.json

{
    "name": "My Workspace",

    // Sets the run context to one level up instead of the .devcontainer folder.
    "context": "..",

    // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
    "dockerFile": "..\\Dockerfile",

    // Set *default* container specific settings.json values on container create.
    "settings": {
        "terminal.integrated.shell.linux": null
    },

    // Add the IDs of extensions you want installed when the container is created.
    "extensions": [],

    // Use 'forwardPorts' to make a list of ports inside the container available locally.
    "forwardPorts": [
        8080
    ],
    ...
}

Now that we've configured our build container, use ctrl + shift + p to open the Command Palette again and search for Remote-Containers: Reopen in Container and click enter which will build the container and set up an image with the following setup for us:

Linked ports as defined in the forwardPorts property
Configure a VSCode development server inside the container so our editor can link to it
Mount our system's file directory into the container so we can edit our files
Does not run the CMD command from our Dockerfile
Open a VSCode window linked to the container so we can start working with our code

Now that you're in the container you can edit your files and run it by doing the following:

Use ctrl + shift + p and then search for Terminal: Create new Integrated Terminal and click enter
Type node app.js into the new Terminal window and click enter to run our app.js file
Navigate to http://localhost:8080 in your browser to view your running app

At this point we've created a container to use as a development file and run our application, you can stop the application with ctrl + c

You can switch from developing in a container back to your local environment with ctrl + shift + p and searching for Remote-Containers: Reopen locally and clicking enter

Now that we're back on our local environment (and not docker) we can look at the other way we can set up our project for VSCode

Using a Preconfigured Dockerfile

Visual Studio Code's Remote Containers Extension provides some pre-configured Dockerfiles for common application or application framework types. One of the available preconfigured Dockerfiles is for working on Node.js applications

The preconfigured files usually just provide a starting point for applications and often you will need to modify these to suit your application, we don't need this for the application we're working on however

To redefine our Docker development config, let's delete the .devcontainer directory in our application and regenerate this

We can regenerate the files needed with ctrl + shift + p, and searching for Remote-Containers: Add Development Container Configuration Files again, clicking enter and then selecting the From a predefined configuration definition option, and then selecting Node.js 12, this should now create a .devcontainer/devcontainer.json file as well as a new .devcontainer/Dockerfile that we did not have previously, our working directory will now look like so:

working-directory
|__ .devcontainer
|   |__ devcontainer.json
|   |__ Dockerfile         # predefined dev container Dockerfile
|
|__ index.js
|__ Dockerfile             # our self-defined Dockerfile

If we look at the devcontainer.json file we will see something similar to what we had before:

devcontainer.json

{
    "name": "Node.js 12",
    "dockerFile": "Dockerfile",

    // Set *default* container specific settings.json values on container create.
    "settings": {
        "terminal.integrated.shell.linux": "/bin/bash"
    },

    // Add the IDs of extensions you want installed when the container is created.
    "extensions": [
        "dbaeumer.vscode-eslint"
    ]

    ...
}

You may, however, note that the dockerFile property is missing, this just means that VSCode will use the default Dockerfile which has been created in the .devcontainer directory

We can go ahead and change the name if we want, we should also add the forwardPorts option as we did previously:

devcontainer.json

{
    ...

    "forwardPorts": [
        8080
    ],

    ...
}

Now looking at the Dockerfile which defines the base development container:

Dockerfile

FROM mcr.microsoft.com/vscode/devcontainers/javascript-node:0-12

This is a bit different to ours because Visual Studio Code will handle the file copying and port exposing on its own for the development container. Note that this configuration can only be used for development and can't really be deployed as a production container. This type of setup is necessary if our development image and production image will be different (which they usually are)

Now that the development container has been set-up, we can use ctrl + shift + p and Remote-Containers: Reopen in Container to open our development container, from here we can work on our application and run the application the same as we did before

Which Method to Use

We've looked at two different methods for configuring our development container, either of which can be used in any project. Below are my recommendations:

If you've got an existing Dockerfile and your development container can be the same as your production container, for things like simple node.js or python apps, and you don't want to maintain another Dockerfile then this may be a quick solution to opt for

Otherwise, if your development container needs to be different from your production one then it's probably easier to start with a predefined VSCode Container as a base and add in any development configuration you need to the .devcontainer/Dockerfile

Lastly, if you don't have an existing Dockerfile at all then I'd suggest using a predefined one so that even if it's not fully configured you've got a relatively good starting point, especially when working with more complex languages and frameworks as a custom Dockerfile for these can be some work to configure

Summary

In this post, we've covered the basics of using Docker to run your applications in a container as well as how to define and build your images. We also looked at why we may want to use a container for development and how we can do this using Visual Studio Code

Node.js Dirname vs Resolve

Fri, 25 Sep 2020 00:00:00 GMT

Node.js Dirname vs Resolve

Node.js has a few methods by which we can get the directory in which we are currently executing, and get paths relative to it

Get the Current Directory

Process Directory

We can get the working directory from where we started the node script with:

const processDir = process.cwd()

File/Module Directory

And we can get the directory in which the currently executing file is in with:

const fileDir = __dirname

Get Path to a Target Location

To get an absolute path to a specific target file/directory we have a few methods

Joining Paths

We can use the path module's join method to get a path given any path pieces you can go up or down a directory using the ../ notation

const { join } = require('path')

const path1 = join(basePath, './downdir/myfile.txt')
const path2 = join(basePath, '../updir')

Absolute Path from Process Directory

To get an absolute path from the process directory, you can use the path module's resolve function:

const { resolve } = require('path')

const absPath = resolve('./downdir/myfile.txt')

Like the join method you can also use the ../ notation to move up a directory

Using resolve is basically shorthand for using join with process.cwd()

const absPath = join(process.cwd(), './downdir/myfile.txt')

Absolute Path from File/Module Directory

If it makes more sense to get the path relative to the executing file, you can use a combination of __dirname and join like so:

const { join } = require('path')

const absPath = join(__dirname, './downdir/myfile.txt')

Backup SQL Server Database as Script

Thu, 26 Nov 2020 00:00:00 GMT

Backup SQL Server Database as Script

To create a scripted backup of a database on SQL Server do the following:

Right click on the database name > Tasks > Generate Scripts...
On the Choose Objects screen select Scriot entire database and all database objects
On the Set Scripting Options screen click on Advanced and select:
1. Script DROP and CREATE = Script CREATE
2. Types of data to script = Schema and data
Set the location to save the script
On the Summary page you can click Next which will start the generation
Thereafter you can view the generated database script from the location it was generated to

Highlight Specific Elements

Tue, 29 Sep 2020 00:00:00 GMT

Highlight Specific Elements

You can highlight all HTML elements which respond to a specific CSS selector with the following. This can be useful for debugging purposes:

document
  .querySelectorAll(selector)
  .forEach((el) => (el.style.border = 'solid 2px red'))

For example, the following [id] selector will find all elements with an id attribute:

document
  .querySelectorAll('[id]')
  .forEach((el) => (el.style.border = 'solid 2px red'))

URL Text Fragments

Tue, 29 Sep 2020 00:00:00 GMT

URL Text Fragments

When linking to a webpage it can be useful to link to a specific part of a page. Using a hash in the URl you can link to the id attribute in the HTML of a page

For example, https://my.website.com#overview would link to the element with id="overview" in the HTML

However, it may be useful for us to link to a general piece of text on a page, we can use text fragments to do so. These work similar to the way a hash does, but we can link to any text on a page even if it does not have an id. Additionally, the user's browser will highlight the text set to draw the user's attention to it

Note that at the moment only Chromium based browsers support this

To do this, we need to identifiy the text on the page we want to link to. Say we'd like to link to the first paragraph of this page, we can use the text directive (#:~:text=) param in the URL to link to specific text instead of just the # like we use for an ID

We would need to take the part of the text we want to focus, URL encode it, then add it to the text directive the final URL may look something like this:

https://my.website.com/#:~:text=When%20linking%20to%20a%20webpage

If you need to highlight a larger piece of text, you can use the start and end of your text segment, separated by a comma, for example selecting the entire first paragraph of this post:

https://my.website.com/#:~:text=when%20linking,HTML%20of%20a%20page

The link below should link to the first paragraph of this page if opened in a new tab (provided your browser supports text fragments):

ctrl + click me

You can read more about text fragments in URLs on web.dev

Serialize a JsonValue Array using F# and FSharp.Data's JsonProvider

Mon, 01 Mar 2021 00:00:00 GMT

Serialize a JsonValue Array using F# and FSharp.Data's JsonProvider

When working with the FSharp.Data.JsonProvider type provider you may encounter a need to serialize a JsonValue array

Let's take a look at an example where this may be necessary:

First, we'll define the type of our data using a type provider, in this case it's connected to the following:

data/api-response.json

{
  "data": [
    {
      "id": "id",
      "caption": "caption",
      "media_type": "IMAGE",
      "media_url": "url_to_image"
    }
  ],
  "paging": {
    "cursors": {
      "before": "hash",
      "after": "hash"
    },
    "next": "full_request_url"
  }
}

And the F# file that's using this as the basis for the type definition is as follows:

open FSharp.Data

type ApiResponse = JsonProvider<"./data/api-response.json", SampleIsList=true>

We can also create some sample data using the ApiResponse.Parse method that's now defined on our type thanks to the JsonProvider

// get some sample data
let sampleData = ApiResponse.Parse("""{
  "data": [
      {
          "id": "id",
          "caption": "caption",
          "media_type": "IMAGE",
          "media_url": "url_to_image"
      }
  ],
  "paging": {
      "cursors": {
          "before": "hash",
          "after": "hash"
      },
      "next": "full_request_url"
  }
}""")

If you were to use the sampleData object and try to parse it to JSON you could directly call the sampleData.JsonValue.ToString() method. As designed, this immediately returns the JSON representation of the object

However, when we take a look at the sampleData.Data property, we will notice this is a Datum array, the problem here is that the array type doesn't have JsonValue property

However, if we look at how JsonValue is defined we will see the following:

union JsonValue =
  | String of string
  | Number of decimal
  | Float of float
  | Record of properties : (string * JsonValue) array
  | Array of elements : JsonValue array
  | Boolean of bool
  | Null

Based on this, we can see that an Array of elements can be seen as a JsonValue array, using this information, we can make use of the JsonValue.Array constructor and the JsonValue property of each of the elements of the Data property to convert the JsonValue array to a JsonValue

let jsonValue =
    // get the property we want
    sampleData.Data
    // extract the JsonValue property from each element
    |> Array.map (fun p -> p.JsonValue)
    // pass into the JsonValue.Array constructor
    |> JsonValue.Array

let json = jsonValue.ToString()

Logging Aliases for Javascript

Tue, 02 Nov 2021 00:00:00 GMT

Logging Aliases for Javascript

I often find myself writing a function to JSON.stringify some data to log in either a pretty or flat structure

It's just more of a convenience method, and it's pretty much the same as doing console.log(JSON.stringify(data)) and looks like this:

const _ = (data: any, pretty: boolean = false) => {
  return console.log(JSON.stringify(data, null, pretty ? 2 : 0))
}

And then, when I need to log something:

_(myData)

Or, if I want to pretty print the JSON

_(myData, true)

Jenkins: A Git process may have crashed in the repository

Wed, 03 Feb 2021 00:00:00 GMT

Jenkins: A Git process may have crashed in the repository

Taken from this StackOverflow answer

Another git process seems to be running in this repository, e.g.
an edirot opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue

Sometimes when using Jenkins a Git process may crash or timeout resulting in an error message like the above one. This can result in issues when running later Git steps or running other Git steps in the repository in the Jenkins workspace

The simplest way I've found to fix this is to simply delete the .git/index.lock or .git/shallow.lock file from the workspace that's giving the issue and run again

Multi-module Python project using an init.py file

Thu, 06 May 2021 00:00:00 GMT

Multi-module Python project using an init.py file

When creating a multi-module project for Python, especially when coming from another language ecosystem, you may encounter issues importing code from other files, let's take a look at this

The Modules

In our project, we've got the following files:

package1
  |- module1.py
package2
  |- namespace2
       |- module2.py
main.py

Let's say each of our modules export a function, here's the code for them:

package1/module1.py

def hello_p1_m1():
  return "hello"

package2/subpackage1/module1.py

def hello_p2_s1_m1():
  return "hello"

The Main

Now that we've got our files with their functions separated, we try using them from the main.py file:

main.py

import package1.module1 as p1m1
import package2.namespace1.module1 as p2s1m1

print(p1m1.hello_p1_m1())
print(p2n1m1.hello_p2_s1_m1())

The Error

Now, we try to run this using python:

python main.py

And we get the following error:

Traceback (most recent call last):
  File "main.py", line 1, in <module>
    import package1.module1 as p1m1
ImportError: No module named package1.module1

The Solution

Now, this can be pretty annoying, but the solution is very simple. Python identifies a module as a single file, and the assumption that leads to the issue above is that we assume a package has the same convention of just being a directory

However, a packgage requires an __init__.py file to be defined so that they are recognized correctly

So, we need to add two files:

package1/__init__.py
package2/subpackage1/__init__.py

Additionally, these files are completely empty, and only serve as information. Note that we don't need to include a file in package2 directly as this directory does not contain any modules within it so is not really a package in itself and is simply a wrapper subpackage1

Setup HomeAssistant RaspberryPi with WiFi configured

Tue, 09 Mar 2021 00:00:00 GMT

Setup HomeAssistant RaspberryPi with WiFi configured

A short lil guide to setting up Home Assistant on a RaspberryPi using WiFi instead of Ethernet

Download Balena Etcher and install from here
Go to the HomeAssistant website and copy the relevant release URL from here
Connect your RaspberryPi's SD card to your computer
Open Balena Etcher and select Flash from URL and select the RaspberryPi's SD to flash to
Once flashing is complete, open the RaspberryPi's SD in File Explorer, and create the following file:

RASPBERRYPIDRIVE/CONFIG/network/my-network

!cat .\my-network
[connection]
id=my-network
uuid=72111c67-4a5d-4d5c-925e-f8ee26efb3c3
type=802-11-wireless
[802-11-wireless]
ssid=YOUR_WIFI_NETWORK_NAME
#hidden=true

[802-11-wireless-security]
auth-alg=open
key-mgmt=wpa-psk
psk=YOUR_WIFI_PASSWORD

[ipv4]
method=auto

[ipv6]
addr-gen-mode=stable-privacy
method=auto

Be sure to use LF line endings, additional information on this setup can be found on GitHub

XUnit with F#

Sat, 10 Apr 2021 00:00:00 GMT

XUnit with F#

An important part of writing any software is testing. Unit testing is an automated testing method in which we test individual components of our software to verify that their behaviour aligns with our expectations

This post will take a look at the process of setting up a new F# library and two methods of configuring XUnit to test your project's code

Create a Project

Before we can start testing we need a project that we can run tests on

First, we're going to create a folder that we can work in:

mkdir MyProject
cd MyProject

We can use the following command to create a new project in our MyProject directory:

dotnet new classlib -lang=f# -o MyProject.Lib

The project that we created will contain the following Library.ts file, this is the file that we'll write tests for. First, we want to update the hello function so that it returns a formatted string:

MyProject.Lib/Library.fs

namespace MyProject.Lib

module Say =
    let hello name =
        sprintf "Hello %s" name

Adding Tests

Depending on our preferred project structure we can either:

Add tests in a separate project
Add test files alongside lib files

Method 1: Create Tests in Separate Project

The standard method of .NET unit testing with XUnit is to make use of separate Project and Test solutions, so a normal test setup would look something like:

MyProject.Lib
MyProject.Lib.Tests

To add an new XUnit test project you can run:

dotnet new xunit -lang=f# -o MyProject.Tests

Then, so we're able to test the code from MyProject.Lib, we need to add a reference to it from the Test project we just created:

dotnet add MyProject.Tests reference MyProject.Lib

Then we can create a file in our test project called LibraryTests.fs which will contain our test code which we will cover in the last section, as well as adding a reference to this file in the MyProject.Tests.fsproj

MyProject.Tests.fsproj

...
  <ItemGroup>
    <Compile Include="Tests.fs" />
    <!-- Add the next line -->
    <Compile Include="LibraryTests.fs" />
    <Compile Include="Program.fs" />
  </ItemGroup>
...

Method 2: Create Tests Alongside Lib Files

The second method I'm going to discuss is keeping our test.fs files alongside the code that the file is testing. The structure of our project is something more like this:

MyProject.Lib
|- Library.fs
|- Library.test.fs

Overall I find this more manageable is the way I keep my test code in other languages and frameworks as well

To implement this method we need to add some dependencies to our project

cd MyProject.Lib
dotnet add package Microsoft.NET.Test.Sdk
dotnet add package xunit
dotnet add package xunit.runner.visualstudio

Then we can create a file in our project called Library.test.fs which will contain our test code which we will cover next, as well as a reference to this file in the MyProject.Lib.fsproj

MyProject.Lib.fsproj

...
  <ItemGroup>
    <Compile Include="Library.fs" />
    <!-- Add the next line -->
    <Compile Include="Library.test.fs" />
  </ItemGroup>
...

It's important to note that this file must be added below the Library.fs file as it will reference it for tests to run

Test Files

More detailed information on XUnit can be found in Unit Testing notes

Since we've configured XUnit it may be useful to understand how these tests work in the context of F#. XUnit tests are organized into modules. Regardless of which of the two methods above you're using the test files work the same

Generally, a test file will contain:

A top-level module definition
open statements to import XUnit
Test functions annotated with Fact or Theory

XUnit tests can be broken into 2 types:

Single-case tests without input parameters inputs are labelled Fact
Multi-case tests which make use of input parameters are labelled Theory and use InlineData

Fact

Let's add the following content into our test file into one that tests the hello function from our Lib code with the input "Name"

LibraryTests.fs/Library.test.fs

module LibraryTests

open Xunit
open MyProject.Lib.Say

[<Fact>]
let ``Say.hello -> "Hello name" `` () =
    let name = "name"
    let expected = "Hello name"

    let result = hello name

    Assert.Equal(expected, result)

F# allows us to name our functions using special characters provided they're enclosed in backticks as seen above. Naming test functions this way allows them to be more discriptive than more traditional variable names

Additionally, there's the normal XUnit test setup which includes calling our test function with some input and asserting something about it using Assert.Equal from XUnit

Theory

We can add a Theory to test our function with multiple different inputs:

LibraryTests.fs/Library.test.fs

[<Theory>]
[<InlineData("name", "Hello name")>]
[<InlineData("World", "Hello World")>]
let ``Say.hello -> concantenated string`` (name:string, expected: string) =
    let result = hello name

    Assert.Equal(expected, result)

In the above we add the InlineData attribute which allows us to provide inputs to our test, as well as specifying a name and expected argument for our function. The test framework will then call our test using the arguments as specified in InlineData

When we're done our test file should have the following content:

module LibraryTests

open Xunit
open MyProject.Lib.Say

[<Fact>]
let ``Say.hello -> "Hello name" `` () =
    let name = "name"
    let expected = "Hello name"

    let result = hello name

    Assert.Equal(expected, result)

[<Theory>]
[<InlineData("name", "Hello name")>]
[<InlineData("World", "Hello World")>]
let ``Say.hello -> concantenated string`` (name:string, expected: string) =
    let result = hello name

    Assert.Equal(expected, result)

Running Tests

In order to run tests we can use the dotnet-cli. Depending on the method used you can run your test from the project's root directory using the following command:

Method 1 - dotnet test MyProject.Tests
Method 2 - dotnet test MyProject.Lib

Alternatively, tests can also be run from your IDE or Visual Studio Code with the Ionide and .NET Core Test Explorer extensions installed

Additional Resources

If you'd like a deeper look into F# or XUnit here are some of my other posts which cover those:

Nabeel Valley

Template Literal Types with Typescript

Mon, 16 Aug 2021 00:00:00 GMT

Template Literal Types with Typescript

Template literal types provide us a way to combine string types or enums in Typescript.

In the below example, we can make use of a string type called Action and an enum called Resource to define a type which is a combination of an action combined with a resource

type Action = 'UPDATE' | 'CREATE' | 'DELETE'

enum Resource {
  Person = 'Person',
  Product = 'Product',
  Sale = 'Sale',
}

// the `Lowercase` type concerts all string type options to lowercase
type ResourceAction = `${Resource}.${Action}`

Such that the ResourceAction type is now defined as:

type ResourceAction =
  | 'Person.UPDATE'
  | 'Person.CREATE'
  | 'Person.DELETE'
  | 'Product.UPDATE'
  | 'Product.CREATE'
  | 'Product.DELETE'
  | 'Sale.UPDATE'
  | 'Sale.CREATE'
  | 'Sale.DELETE'

Now, if you're like me - you probably want your types to be consistent in some way. For this purpose, we can use the Lowercase type as follows:

type ResourceActionLowercase = Lowercase<`${Resource}.${Action}`>

Which results in the following types:

type ResourceActionLowercase =
  | 'person.update'
  | 'person.create'
  | 'person.delete'
  | 'product.update'
  | 'product.create'
  | 'product.delete'
  | 'sale.update'
  | 'sale.create'
  | 'sale.delete'

Building Serverless Apps using the Serverless Stack Framework

Thu, 17 Jun 2021 00:00:00 GMT

Building Serverless Apps using the Serverless Stack Framework

Prior to doing any of the below you will require your ~/.aws/credentials file to be configured with the credentials for your AWS account

Serverless Stack Framework

SST Framework is a framework built on top of CDK for working with Lambdas and other CDK constructs

It provides easy CDK setups and a streamlined debug and deploy process and even has integration with the VSCode debugger to debug stacks on AWS

Init Project

To init a new project use the following command:

npx create-serverless-stack@latest my-sst-app --language typescript

Which will create a Serverless Stack applocation using TypeScript

Run the App

You can run the created project in using the config defined in the sst.json file:

{
  "name": "my-sst-app",
  "stage": "dev",
  "region": "us-east-1",
  "lint": true,
  "typeCheck": true
}

Using the following commands command will build then deploy a dev stack and allow you to interact with it via AWS/browser/Postman/etc.

npm run start

Additionally, running using the above command will also start the application with hot reloading enabled so when you save files the corresponding AWS resources will be redeployed so you can continue testing

The Files

The application is structured like a relatively normal Lambda/CDK app with lib which contains the following CDK code:

Stack

lib/index.ts

import MyStack from './MyStack'
import * as sst from '@serverless-stack/resources'

export default function main(app: sst.App): void {
  // Set default runtime for all functions
  app.setDefaultFunctionProps({
    runtime: 'nodejs12.x',
  })

  new MyStack(app, 'my-stack')

  // Add more stacks
}

lib/MyStack.ts

import * as sst from '@serverless-stack/resources'

export default class MyStack extends sst.Stack {
  constructor(scope: sst.App, id: string, props?: sst.StackProps) {
    super(scope, id, props)

    // Create the HTTP API
    const api = new sst.Api(this, 'Api', {
      routes: {
        'GET /': 'src/lambda.handler',
      },
    })

    // Show API endpoint in output
    this.addOutputs({
      ApiEndpoint: api.httpApi.apiEndpoint,
    })
  }
}

And src which contains the lambda code:

src/lambda.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  return {
    statusCode: 200,
    headers: { 'Content-Type': 'text/plain' },
    body: `Hello, World! Your request was received at ${event.requestContext.time}.`,
  }
}

Add a new Endpoint

Using the defined constructs it's really easy for us to add an additional endpoint:

src/hello.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  const response = {
    data: 'Hello, World! This is another lambda but with JSON',
  }

  return {
    statusCode: 200,
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(response),
  }
}

And then in the stack we just update the routes:

lib/MyStack.ts

const api = new sst.Api(this, 'Api', {
  routes: {
    'GET /': 'src/lambda.handler',
    'GET /hello': 'src/hello.handler', // new endpoint handler
  },
})

So that the full stack looks like this:

lib/MyStack.ts

import * as sst from '@serverless-stack/resources'

export default class MyStack extends sst.Stack {
  constructor(scope: sst.App, id: string, props?: sst.StackProps) {
    super(scope, id, props)

    // Create the HTTP API
    const api = new sst.Api(this, 'Api', {
      routes: {
        'GET /': 'src/lambda.handler',
        'GET /hello': 'src/hello.handler',
      },
    })

    // Show API endpoint in output
    this.addOutputs({
      ApiEndpoint: api.httpApi.apiEndpoint,
    })
  }
}

VSCode Debugging

SST supports VSCode Debugging, all that's required is for you to create a .vscode/launch.json filw with the following content:

.vscode/launch.json

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug SST Start",
      "type": "node",
      "request": "launch",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["start"],
      "port": 9229,
      "skipFiles": ["<node_internals>/**"]
    },
    {
      "name": "Debug SST Tests",
      "type": "node",
      "request": "launch",
      "runtimeExecutable": "${workspaceRoot}/node_modules/.bin/sst",
      "args": ["test", "--runInBand", "--no-cache", "--watchAll=false"],
      "cwd": "${workspaceRoot}",
      "protocol": "inspector",
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen",
      "env": { "CI": "true" },
      "disableOptimisticBPs": true
    }
  ]
}

This will then allow you to run Debug SST Start which will configure the AWS resources using the npm start command and connect the debugger to the instance so you can debug your functions locally as well as make use of the automated function deployment

Add a DB

From these docs

We can define our table using the sst.Table class:

const table = new sst.Table(this, 'Notes', {
  fields: {
    userId: sst.TableFieldType.STRING,
    noteId: sst.TableFieldType.NUMBER,
  },
  primaryIndex: {
    partitionKey: 'userId',
    sortKey: 'noteId',
  },
})

Next, we can add some endpoint definitions for the functions we'll create as well as access to the table name via the environment:

const api = new sst.Api(this, 'Api', {
  defaultFunctionProps: {
    timeout: 60, // increase timeout so we can debug
    environment: {
      tableName: table.dynamodbTable.tableName,
    },
  },
  routes: {
    // .. other routes
    'GET  /notes': 'src/notes/getAll.handler', // userId in query
    'GET  /notes/{noteId}': 'src/notes/get.handler', // userId in query
    'POST /notes': 'src/notes/create.handler',
  },
})

And lastly we can grant the permissions to our api to access the table

api.attachPermissions([table])

Adding the above to the MyStack.ts file results in the following:

import * as sst from '@serverless-stack/resources'

export default class MyStack extends sst.Stack {
  constructor(scope: sst.App, id: string, props?: sst.StackProps) {
    super(scope, id, props)

    const table = new sst.Table(this, 'Notes', {
      fields: {
        userId: sst.TableFieldType.STRING,
        noteId: sst.TableFieldType.STRING,
      },
      primaryIndex: {
        partitionKey: 'userId',
        sortKey: 'noteId',
      },
    })

    // Create the HTTP API
    const api = new sst.Api(this, 'Api', {
      defaultFunctionProps: {
        timeout: 60, // increase timeout so we can debug
        environment: {
          tableName: table.dynamodbTable.tableName,
        },
      },
      routes: {
        // .. other routes
        'GET  /notes': 'src/notes/getAll.handler', // userId in query
        'GET  /notes/{noteId}': 'src/notes/get.handler', // userId in query
        'POST /notes': 'src/notes/create.handler',
      },
    })

    api.attachPermissions([table])

    // Show API endpoint in output
    this.addOutputs({
      ApiEndpoint: api.httpApi.apiEndpoint,
    })
  }
}

Before we go any further, we need to install some dependencies in our app, particularly uuid for generating unique id's for notes, we can install a dependency with:

npm install uuid
npm install aws-sdk

Define Common Structures

We'll also create some general helper functions for returning responses of different types, you can view the details for their files below but these just wrap the response in a status and header as well as stringify the body

src/responses/successResponse.ts

const successResponse = <T>(item: T) => {
  return {
    statusCode: 200,
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(item),
  }
}

export default successResponse

src/responses/badResuestsResponse.ts

const badRequestResponse = (msg: string) => {
  return {
    statusCode: 400,
    headers: { 'Content-Type': 'text/plain' },
    body: msg,
  }
}

export default badRequestResponse

src/responses/internalErrorResponse.ts

const internalErrorResponse = (msg: string) => {
  console.error(msg)
  return {
    statusCode: 500,
    headers: { 'Content-Type': 'text/plain' },
    body: 'internal error',
  }
}

export default internalErrorResponse

And we've also got a Note type which will be the data that gets stored/retreived:

src/notes/Note.ts

type Note = {
  userId: string
  noteId: string
  content?: string
  createdAt: number
}

export default Note

Access DB

Once we've got a DB table defined as above, we can then access the table to execute different queries

We would create a DB object instance using:

const db = new DynamoDB.DocumentClient()

Create

A create is the simplest one of the database functions for us to implement, this uses the db.put function with the Item to save which is of type Note:

const create = async (tableName: string, item: Note) => {
  await db.put({ TableName: tableName, Item: item }).promise()
}

Get

We can implement a getOne function by using db.get and providing the full Key consisting of the userId and noteId

const getOne = async (tableName: string, noteId: string, userId: string) => {
  const result = await db
    .get({
      TableName: tableName,
      Key: {
        userId: userId,
        noteId: noteId,
      },
    })
    .promise()

  return result.Item
}

GetAll

We can implement a getByUserId function which will make use of db.query and use the ExpressionAttributeValues to populate the KeyConditionExpression as seen below:

const getByUserId = async (tableName: string, userId: string) => {
  const result = await db
    .query({
      TableName: tableName,
      KeyConditionExpression: 'userId = :userId',
      ExpressionAttributeValues: {
        ':userId': userId,
      },
    })
    .promise()

  return result.Items
}

Define Lambdas

Now that we know how to write data to Dynamo, we can implement the following files for the endpoints we defined above:

Create

src/notes/create.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'
import { DynamoDB } from 'aws-sdk'
import { v1 } from 'uuid'
import internalErrorResponse from '../responses/internalErrorResponse'
import successResponse from '../responses/successResponse'
import badRequestResponse from '../responses/badRequestResponse'
import Note from './Note'

const db = new DynamoDB.DocumentClient()

const toItem = (data: string, content: string): Note => {
  return {
    userId: data,
    noteId: v1(),
    content: content,
    createdAt: Date.now(),
  }
}

const parseBody = (event: APIGatewayProxyEventV2) => {
  const data = JSON.parse(event.body || '{}')

  return {
    userId: data.userId,
    content: data.content,
  }
}

const isValid = (data: Partial<Note>) =>
  typeof data.userId !== 'undefined' && typeof data.content !== 'undefined'

const create = async (tableName: string, item: Note) => {
  await db.put({ TableName: tableName, Item: item }).promise()
}

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  if (typeof process.env.tableName === 'undefined')
    return internalErrorResponse('tableName is undefined')

  const tableName = process.env.tableName
  const data = parseBody(event)

  if (!isValid(data))
    return badRequestResponse('userId and content are required')

  const item = toItem(data.userId, data.content)
  await create(tableName, item)

  return successResponse(item)
}

Get

src/notes/get.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'
import { DynamoDB } from 'aws-sdk'
import badRequestResponse from '../responses/badRequestResponse'
import internalErrorResponse from '../responses/internalErrorResponse'
import successResponse from '../responses/successResponse'

type RequestParams = {
  noteId?: string
  userId?: string
}

const db = new DynamoDB.DocumentClient()

const parseBody = (event: APIGatewayProxyEventV2): RequestParams => {
  const pathData = event.pathParameters
  const queryData = event.queryStringParameters

  return {
    noteId: pathData?.noteId,
    userId: queryData?.userId,
  }
}

const isValid = (data: RequestParams) =>
  typeof data.noteId !== 'undefined' && typeof data.userId !== 'undefined'

const getOne = async (tableName: string, noteId: string, userId: string) => {
  const result = await db
    .get({
      TableName: tableName,
      Key: {
        userId: userId,
        noteId: noteId,
      },
    })
    .promise()

  return result.Item
}

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  const data = parseBody(event)

  if (typeof process.env.tableName === 'undefined')
    return internalErrorResponse('tableName is undefined')

  const tableName = process.env.tableName

  if (!isValid(data))
    return badRequestResponse(
      'noteId is required in path, userId is required in query'
    )

  const items = await getOne(
    tableName,
    data.noteId as string,
    data.userId as string
  )

  return successResponse(items)
}
import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'
import { DynamoDB } from 'aws-sdk'
import badRequestResponse from '../responses/badRequestResponse'
import internalErrorResponse from '../responses/internalErrorResponse'
import successResponse from '../responses/successResponse'

type RequestParams = {
  noteId?: string
  userId?: string
}

const db = new DynamoDB.DocumentClient()

const parseBody = (event: APIGatewayProxyEventV2): RequestParams => {
  const pathData = event.pathParameters
  const queryData = event.queryStringParameters

  return {
    noteId: pathData?.noteId,
    userId: queryData?.userId,
  }
}

const isValid = (data: RequestParams) =>
  typeof data.noteId !== 'undefined' && typeof data.userId !== 'undefined'

const getOne = async (tableName: string, noteId: string, userId: string) => {
  const result = await db
    .get({
      TableName: tableName,
      Key: {
        userId: userId,
        noteId: noteId,
      },
    })
    .promise()

  return result.Item
}

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  const data = parseBody(event)

  if (typeof process.env.tableName === 'undefined')
    return internalErrorResponse('tableName is undefined')

  const tableName = process.env.tableName

  if (!isValid(data))
    return badRequestResponse(
      'noteId is required in path, userId is required in query'
    )

  const items = await getOne(
    tableName,
    data.noteId as string,
    data.userId as string
  )

  return successResponse(items)
}

GetAll

src/notes/getAll.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'
import { DynamoDB } from 'aws-sdk'
import badRequestResponse from '../responses/badRequestResponse'
import internalErrorResponse from '../responses/internalErrorResponse'
import successResponse from '../responses/successResponse'

type PathParams = {
  userId?: string
}

const db = new DynamoDB.DocumentClient()

const parseBody = (event: APIGatewayProxyEventV2): PathParams => {
  const data = event.queryStringParameters

  return {
    userId: data?.userId,
  }
}

const isValid = (data: PathParams) => typeof data.userId !== 'undefined'

const getByUserId = async (tableName: string, userId: string) => {
  const result = await db
    .query({
      TableName: tableName,
      KeyConditionExpression: 'userId = :userId',
      ExpressionAttributeValues: {
        ':userId': userId,
      },
    })
    .promise()

  return result.Items
}

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  const data = parseBody(event)

  if (typeof process.env.tableName === 'undefined')
    return internalErrorResponse('tableName is undefined')

  const tableName = process.env.tableName

  if (!isValid(data)) return badRequestResponse('userId is required in query')

  const items = await getByUserId(tableName, data.userId as string)

  return successResponse(items)
}

Testing

Once we've got all the above completed, we can actually test our endpoints and create and read back data

create:

POST https://AWS_ENDPOINT_HERE/notes

{
  "userId": "USER_ID",
  "content": "Hello world"
}

Which responds with:

200

{
  "content": "Hello world",
  "createdAt": 1619177078298,
  "noteId": "NOTE_ID_UUID",
  "userId": "USER_ID"
}

get:

GET https://AWS_ENDPOINT_HERE/notes/NOTE_ID_UUID?userId=USER_ID

200

{
  "content": "Hello world",
  "createdAt": 1619177078298,
  "noteId": "NOTE_ID_UUID",
  "userId": "USER_ID"
}

getAll

GET htttps://AWS_ENDPOINT_HERE/notes?userId=USER_ID

200

[
  {
    "content": "Hello world",
    "createdAt": 1619177078298,
    "noteId": "NOTE_ID_UUID",
    "userId": "USER_ID"
  }
]

Creating Notes Using a Queue

When working with microservices a common pattern is to use a message queue for any operations that can happen in an asynchronous fashion, we can create an SQS queue which we can use to stage messages and then separately save them at a rate that we're able to process them

In order to make this kind of logic we're going to break up our create data flow - a the moment it's this:

lambda -> dynamo
return <-

We're going to turn it into this:

lambda1 -> sqs
 return <-

          sqs -> lambda2 -> dynamo

This kind of pattern becomes especially useful if we're doing a lot more stuff with the data other than just the single DB operation and also allows us to retry things like saving to the DB if we have errors, etc.

A more complex data flow could look something like this (not what we're implementing):

lambda1 -> sqs
 return <-

           sqs -> lambda2 -> dynamo // save to db
               -> lambda3 -> s3     // generate a report
           sqs <-

           sqs -> lambda4           // send an email

Create Queue

SST provides us with the sst.Queue class that we can use for this purpose

To create a Queue you can use the following in stack:

const queue = new sst.Queue(this, 'NotesQueue', {
  consumer: 'src/consumers/createNote.handler',
})

queue.attachPermissions([table])
queue.consumerFunction?.addEnvironment(
  'tableName',
  table.dynamodbTable.tableName
)

The above code does the following:

Create a queue
Give the queue permission to access the table
Add the tableName environment variable to the queue's consumerFunction

We will also need to grant permissions to the API to access the queue so that our create handler is able to add messages to the queue

api.attachPermissions([table, queue])

Which means our Stack now looks like this:

lib/MyStack.ts

import * as sst from '@serverless-stack/resources'

export default class MyStack extends sst.Stack {
  constructor(scope: sst.App, id: string, props?: sst.StackProps) {
    super(scope, id, props)

    const table = new sst.Table(this, 'Notes', {
      fields: {
        userId: sst.TableFieldType.STRING,
        noteId: sst.TableFieldType.STRING,
      },
      primaryIndex: {
        partitionKey: 'userId',
        sortKey: 'noteId',
      },
    })

    const queue = new sst.Queue(this, 'NotesQueue', {
      consumer: 'src/consumers/createNote.handler',
    })

    queue.attachPermissions([table])
    queue.consumerFunction?.addEnvironment(
      'tableName',
      table.dynamodbTable.tableName
    )

    // Create the HTTP API
    const api = new sst.Api(this, 'Api', {
      defaultFunctionProps: {
        timeout: 60, // increase timeout so we can debug
        environment: {
          tableName: table.dynamodbTable.tableName,
          queueUrl: queue.sqsQueue.queueUrl,
        },
      },
      routes: {
        'GET  /': 'src/lambda.handler',
        'GET  /hello': 'src/hello.handler',
        'GET  /notes': 'src/notes/getAll.handler',
        'POST /notes': 'src/notes/create.handler',
        'GET  /notes/{noteId}': 'src/notes/get.handler',
      },
    })

    api.attachPermissions([table, queue])

    // Show API endpoint in output
    this.addOutputs({
      ApiEndpoint: api.httpApi.apiEndpoint,
    })
  }
}

Update the Create Handler

Since we plan to create notes via a queue we will update our create function in the handler to create a new message in the queue, this is done using the SQS class from aws-sdk:

src/notes/create.ts

import { SQS } from 'aws-sdk'

const queue = new SQS()

Once we've got our instance, the create function is done by means of the queue.sendMessage function:

src/notes/create.ts

const create = async (queueUrl: string, item: Note) => {
  return await queue
    .sendMessage({
      QueueUrl: queueUrl,
      DelaySeconds: 0,
      MessageBody: JSON.stringify(item),
    })
    .promise()
}

Lastly, our handler remains mostly the same with the exception of some additional validation to check that we have the queue connection information in the environment:

src/notes/create.ts

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  // pre-save validation
  if (typeof process.env.queueUrl === 'undefined')
    return internalErrorResponse('queueUrl is undefined')

  const queueUrl = process.env.queueUrl

  const data = parseBody(event)

  if (!isValid(data))
    return badRequestResponse('userId and content are required')

  // save process
  const item = toItem(data.userId, data.content)
  const creatresult = await create(queueUrl, item)

  if (!creatresult.MessageId) internalErrorResponse('MessageId is undefined')

  return successResponse(item)
}

Implementing the above into the create handler means that our create.ts file now looks like this:

src/notes/create.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'
import { v1 } from 'uuid'
import internalErrorResponse from '../responses/internalErrorResponse'
import successResponse from '../responses/successResponse'
import badRequestResponse from '../responses/badRequestResponse'
import Note from './Note'
import { SQS } from 'aws-sdk'

const queue = new SQS()

// helper functions start

const toItem = (data: string, content: string): Note => {
  return {
    userId: data,
    noteId: v1(),
    content: content,
    createdAt: Date.now(),
  }
}

const parseBody = (event: APIGatewayProxyEventV2) => {
  const data = JSON.parse(event.body || '{}')

  return {
    userId: data.userId,
    content: data.content,
  }
}

const isValid = (data: Partial<Note>) =>
  typeof data.userId !== 'undefined' && typeof data.content !== 'undefined'

// helper functions end

const create = async (queueUrl: string, item: Note) => {
  return await queue
    .sendMessage({
      QueueUrl: queueUrl,
      DelaySeconds: 0,
      MessageBody: JSON.stringify(item),
    })
    .promise()
}

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  // pre-save validation
  if (typeof process.env.queueUrl === 'undefined')
    return internalErrorResponse('queueUrl is undefined')

  const queueUrl = process.env.queueUrl

  const data = parseBody(event)

  if (!isValid(data))
    return badRequestResponse('userId and content are required')

  // save process
  const item = toItem(data.userId, data.content)
  const creatresult = await create(queueUrl, item)

  if (!creatresult.MessageId) internalErrorResponse('MessageId is undefined')

  return successResponse(item)
}

Add Queue-Based Create Handler

Now that we've updated our logic to save the notes into the queue, we need to add the logic for the src/consumers/createNote.handler consumer function as we specified above, this handler will be sent an SQSEvent and will make use of the DynamoDB Table we gave it permissions to use

First, we take the create function that was previously on the create.ts file for saving to the DB:

src/consumers/createNote.ts

import { DynamoDB } from 'aws-sdk'

const db = new DynamoDB.DocumentClient()

const create = async (tableName: string, item: Note) => {
  const createResult = await db
    .put({ TableName: tableName, Item: item })
    .promise()
  if (!createResult) throw new Error('create failed')

  return createResult
}

We'll also need a function for parsing the SQSRecord object into a Note:

src/consumers/createNote.ts

const parseBody = (record: SQSRecord): Note => {
  const { noteId, userId, content, createdAt } = JSON.parse(record.body) as Note

  // do this to ensure we only extract information we need
  return {
    noteId,
    userId,
    content,
    createdAt,
  }
}

And finally we consume the above through the handler, you can see in the below code that we are iterating over the event.Records object, this is because the SQSEvent adds each new event into this array, the reason for this is because we can also specify batching into our Queue so that the handler is only triggered after n events instead of each time, and though this isn't happening in our case, we still should handle this for our handler:

src/consumers/createNote.ts

export const handler: SQSHandler = async (event) => {
  // pre-save environment check
  if (typeof process.env.tableName === 'undefined')
    throw new Error('tableName is undefined')

  const tableName = process.env.tableName

  for (let i = 0; i < event.Records.length; i++) {
    const r = event.Records[i]
    const item = parseBody(r)
    console.log(item)

    const result = await create(tableName, item)
    console.log(result)
  }
}

Putting all the above together our createNote.ts file now has the following code:

import { SQSHandler, SQSRecord } from 'aws-lambda'
import Note from '../notes/Note'
import { DynamoDB } from 'aws-sdk'

const db = new DynamoDB.DocumentClient()

const create = async (tableName: string, item: Note) => {
  const createResult = await db
    .put({ TableName: tableName, Item: item })
    .promise()
  if (!createResult) throw new Error('create failed')

  return createResult
}

const parseBody = (record: SQSRecord): Note => {
  const { noteId, userId, content, createdAt } = JSON.parse(record.body) as Note

  // do this to ensure we only extract information we need
  return {
    noteId,
    userId,
    content,
    createdAt,
  }
}

export const handler: SQSHandler = async (event) => {
  if (typeof process.env.tableName === 'undefined')
    throw new Error('tableName is undefined')

  const tableName = process.env.tableName

  for (let i = 0; i < event.Records.length; i++) {
    const r = event.Records[i]
    const item = parseBody(r)
    console.log(item)

    const result = await create(tableName, item)
    console.log(result)
  }
}

This completes the implementation of the asynchronous saving mechanism for notes. As far as a consumer of our API is concerned, nothing has changed and they will still be able to use the API exactly as we had in the Testing section above

Deploy

Thus far, we've just been running our API in debug mode via the npm run start command, while useful for testing this adds a lot of code to make debugging possible, and isn't something we'd want in our final deployed code

Deploying using sst is still very easy, all we need to do is run the npm run deploy command and this will update our lambda to use a production build of the code instead:

npm run deploy

Teardown

Lastly, the sst CLI also provides us with a function to teardown our start/deploy code. So once you're done playing around you can use this to teardown all your deployed services:

npm run remove

Note that running the remove command will not delete the DB tables, you will need to do this manually

Render Element by Tag Name in React

Fri, 19 Mar 2021 00:00:00 GMT

Render Element by Tag Name in React

When using React it can sometimes be useful to render a standard HTML element given the element name dynamically as a prop

React allows us to do this provided we store the element name in a variable that starts with a capital letter, as JSX requires this to render a custom element

We can do something like this by defining a generic element in React which just takes in the name of the tag in addition to it's usual props and children, something like this:

const GenericElement = ({ tagName: Tag, children, ...innerProps }) => (
  <Tag {...innerProps}>{children}</Tag>
)

We can then render this element by calling the GenericElement with the tagName prop and then any children we'd like:

<GenericElement tagName="h1">I am an h1</GenericElement>

Here's a short example using Replit for reference, all the important stuff is in the src/App.jsx file:

Custom Styles in Markdown

Tue, 23 Mar 2021 00:00:00 GMT

Custom Styles in Markdown

When working with markdown it can often be useful to be able to style elements using custom CSS

We can accomplish this by including a style tag. An example to illustrate this is changing the way a table renders in a specific document by changing the style of table rows to be striped, you would include the following CSS:

<style>
tr:nth-child(even) {
  background-color: #b2b2b2;
  color: #f4f4f4;
}
</style>

If the platform you're displaying the HTML in already has the styles included then you may need to add !important to override in the CSS:

<style>
tr:nth-child(even) {
  background-color: #b2b2b2!important;
  color: #f4f4f4!important;
}
</style>

my-sample-doc.md

## Custom CSS in Markdown Example

This is a document where tables are rendered with every second table row with a specific background and foreground colour

<style>
tr:nth-child(even) {
  background-color: #b2b2b2!important;
  color: #f4f4f4!important;
}
</style>

The above CSS will lead to the following table being rendered with alternating row colours:

| Col 1 | Col 2 | Col 3 |
| ----- | ----- | ----- |
| A     | B     | C     |
| 1     | 2     | 3     |
| A1    | B2    | C3    |

Note that depending on your markdown renderer the embeded style tags may be removed, you'll have to look at your specific renderer/converter/platform to see whether this works for you

With the converter I'm using, showdown.js, this is how the table above looks (with !important included to override my current table styles):

Col 1	Col 2	Col 3
A	B	C
1	2	3
A1	B2	C3

Control a Raspberry Pi GPIO with Python

Sat, 27 Mar 2021 00:00:00 GMT

Control a Raspberry Pi GPIO with Python

Circuit Diagram

This script will make use of the GPIO Pins to flicker an LED

The circuit diagram is below:

I've used a $330\Omega$ resistor for the resistor connected in series $R_LED$ , however the resistance for a given LED can be calculated with this equation (From Circuit Specialists):

R_{LED} = \frac{V_{source} - V_{LED}}{I_{LED}}

It's important to connect the LED in the correct direction on the circuit

Code

Below is a simple python script which will handle turning the GPIO pins on and off using the RPi.GPIO library

import RPi.GPIO as GPIO
import time

pin = 21
dur = 1

GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(pin, GPIO.OUT)

while True:
  GPIO.output(pin, GPIO.HIGH)
  print "LED ON"
  time.sleep(dur)

  GPIO.output(pin, GPIO.LOW)
  print "LED OFF"
  time.sleep(dur)

Javascript Range Function

Fri, 29 Oct 2021 00:00:00 GMT

Javascript Range Function

Something i often find myself needing is a way to create a range in javascript, similar to what python has

Here's a basic implementation of something that works like that:

const range = (start, end, count, includeEnd = false) => {
  const space = (end - start) / (count - includeEnd)
  return new Array(count).fill(0).map((_, i) => start + i * space)
}

The above will output something like:

const withoutEnd = range(20, 21, 5)
// withoutEnd === [ 20, 20.2, 20.4, 20.6, 20.8 ]

const withEnd = range(20, 21, 5, true)
// withEnd === [ 20, 20.25, 20.5, 20.75, 21 ]

React Top Level API

Tue, 01 Mar 2022 00:00:00 GMT

React Top Level API

For a reference on the React Top-Level API you can take a look at the React Docs

Introduction

React allows us to do lots of different things using concepts like composition and higher order components. Most of the time these methods are good enough for us to do what we want, however there are some cases where these methods can prove to be insufficient such as when building complex library components or components that need to allow for dynamic composition or do runtime modification of things like child component props, etc.

For the above purposes we can make use of the React Top-Level API. For the purpose of this writeup I'll be making use of the parts of this API that allow us to modify a component's children and modify their props as well as how they're rendered

Our Goal

For the purpose of this doc I'll be using the React API to get to do the following:

Show the count of children (Items) passed to a component (Wrapper)
Render each child in a sub-wrapper ItemWrapper
Modify the props of the children by adding a position prop

When this is done, we want to render a component that results in the following markup:

<Wrapper>
  <Count />
  <ItemWrapper>
    <Item name="" position="" />
  </ItemWrapper>
  <ItemWrapper>
    <Item name="" position="" />
  </ItemWrapper>
  <ItemWrapper>
    <Item name="" position="" />
  </ItemWrapper>
</Wrapper>

But a consumer can be used like:

<Wrapper>
  <Item name="">
  <Item name="">
  <Item name="">
</Wrapper>

Using `React.Children` to work with a component's `children`

The React.Children API (see docs) provides us with some utilities for traversing the children passed to a component

Before we can do any of the following, we need to define the structure of an item. Our Item component is defined as follows:

interface ItemProps {
  name: string
  position?: number
}

const Item: React.FC<ItemProps> = ({ name, position }) => (
  <div>
    {name}, {position}
  </div>
)

Use `React.Children.count` to get the count

The React.Children.count function counts the number of child nodes passed to a React component, we can use it like so:

const count = React.Children.count(children)

For our example, let's start off by creating a Count component that simply takes a count prop and displays some text:

interface CountProps {
  count: number
}

const Count: React.FC<CountProps> = ({ count }) => <p>Total: {count}</p>

Next, we can define our Wrapper which will take children and pass the count to our Count component:

const Wrapper: React.FC = ({ children }) => {
  const count = React.Children.count(children)

  return (
    <div>
      <Count count={count} />
    </div>
  )
}

Use `React.Children.map` to wrap each child

Next, the React.Children.map function allows us to map over the children of an element and do stuff with it, for example:

const items = React.Children.map(children, (child) => {
  return <ItemWrapper>{child}</ItemWrapper>
})

Based on the above, we can define an ItemWrapper as so:

const ItemWrapper: React.FC = ({ children }) => (
  <li
    style={{
      backgroundColor: 'lightgrey',
      padding: '10px',
      margin: '20px',
    }}
  >
    {children}
  </li>
)

And we can update the Wrapper to make use of React.children.map:

const Wrapper: React.FC = ({ children }) => {
  const count = React.Children.count(children)

  const items = React.Children.map(children, (child) => {
    return <ItemWrapper>{child}</ItemWrapper>
  })

  return (
    <div>
      <Count count={count} />
      <ul>{items}</ul>
    </div>
  )
}

Use `React.cloneElement` to change child props

Lastly, we want to append a position prop to the Item. To do this we can make use of the React.cloneElement function which allows us to clone an element and modify the props of it. Using this function looks like so:

const childProps = child.props
const newProps = { ...child.props, position: index }

const newChild = React.cloneElement(child, newProps)

Integrating this into the React.Children.map function above will result in our Wrapper looking like so:

const Wrapper: React.FC = ({ children }) => {
  const count = React.Children.count(children)

  const items = React.Children.map(children, (child, index) => {
    const childProps = child.props
    const newProps = { ...child.props, position: index }

    const newChild = React.cloneElement(child, newProps)

    return <ItemWrapper>{newChild}</ItemWrapper>
  })

  return (
    <div>
      <Count count={count} />
      <ul>{items}</ul>
    </div>
  )
}

Use `React.isValidElement`

We've completed most of what's needed, however if for some reason our child is not a valid react element our component may still crash. To get around this we can use the React.isValidElement function

We can update our map function above to return null if the element is not value:

const items = React.Children.map(children, (child, index) => {
  if (!React.isValidElement(child)) return null

  const childProps = child.props
  const newProps = { ...child.props, position: index }

  const newChild = React.cloneElement(child, newProps)

  return <ItemWrapper>{newChild}</ItemWrapper>
})

Which results in our Wrapper now being:

const Wrapper: React.FC = ({ children }) => {
  const count = React.Children.count(children)
  const items = React.Children.map(children, (child, index) => {
    if (!React.isValidElement(child)) return null

    const childProps = child.props
    const newProps = { ...child.props, position: index }

    const newChild = React.cloneElement(child, newProps)

    return <ItemWrapper>{newChild}</ItemWrapper>
  })

  return (
    <div>
      <Count count={count} />
      <ul>{items}</ul>
    </div>
  )
}

The Result

Lastly, we'll render the above using the App component, the API for the above components should be composable as we outlined initially. The App component will now look like so:

const App: React.FC = () => {
  return (
    <Wrapper>
      <Item name="Apple" />
      <Item name="Banana" />
      <Item name="Chocolate" />
    </Wrapper>
  )
}

And the rendered HTML:

<div>
  <p>Total: 3</p>
  <ul>
    <li style="background-color: lightgrey; padding: 10px; margin: 20px;">
      <div>Apple, 0</div>
    </li>
    <li style="background-color: lightgrey; padding: 10px; margin: 20px;">
      <div>Banana, 1</div>
    </li>
    <li style="background-color: lightgrey; padding: 10px; margin: 20px;">
      <div>Chocolate, 2</div>
    </li>
  </ul>
</div>

Total: 3

Apple, 0
Banana, 1
Chocolate, 2

And lastly, if you'd like to interact with the code from this sample you can see it in this Repl

Dev Tools Update

Wed, 03 Aug 2022 00:00:00 GMT

Dev Tools Update

Programming Languages

I've primarily switched over to Typescript as my language for building applications, but I've also been learning Rust for the purpose of building some low-level libraries

For Typescript, something I've been finding to be really informative is the Type Challenges Repository which has a lot of challenges along with solutions for modeling and solving complex problems using Typescript's type system

For Rust, the Rust Book along with the No Boilerplate YouTube Channel have been really informative

Frameworks

The current version of my website is built using 11ty which is a static site generator which uses Javascript and templates for generating HTML from a variety of data sources - in my case just JSON and Markdown files

A lot of the professional work I do revolves around the React and React-Native ecosystem, but I've made some changes to my stack for personal projects due to the overhead that comes with using React

For now I'm using 11ty for my website as well as Hugo for some smaller projects

I've moved on from Gatsby for my site due to the slow build times as a static site builder - and the unnecessary complexity of graphql for this purpose along with the overall burden of node modules that comes with it

For other projects I'd like to give the Remix Framework a shot for full stack apps. Remix is a full stack framework for building React applications with a pretty cool looking data fetching and project structure

I've also been playing around with Tauri for building desktop applications. Tauri uses a Rust backend with a bring-your-own client approach and pretty much uses any client side javascript framework - I'm looking towards Svelte for this since it keeps things a lot more vanilla which is a good break from working with the other major frameworks

Code Editor

I'm back to VSCode after a short stint with NeoVim.

My reasons for leaving NeoVim were mostly due to the amount of setup when switching devices as well as the issues I experienced on Windows where configuration management was a bit inconsistent as well as the dependency management for some plugins became a lot of admin. Even preconfigured solutions like LunarVim and AstroNvim didn't work well due to lots of compat issues on Windows

As far as VSCode is concerned, some things I've been using are:

The GitHub Online Editor
The GitHub Colorblind Theme (GitHub.github-vscode-theme)
Code Spell Checker Extension (streetsidesoftware.code-spell-checker)
GitLens Extension (eamodio.gitlens)
Hex Editor (ms-vscode.hexeditor)
Rust Analyzer Extension (rust-lang.rust-analyzer)
Fix Extension (withfig.fig)

Terminal

Some terminal tools I've been enjoying recently are:

Hyper - An alternative terminal that's so far been less sluggish and more customizable than the Windows Terminal application
Fig - Provides autocomplete for commands and integrates with loads of CLI tools to provide detailed suggestions and documentations
NuShell - A shell that works with the idea of command output as data that can be manipulated and transformed which adds a lot of cool data processing functionality on top of your normal shell comands

A Simple JSON Backed Database in Typescript

Wed, 06 Jul 2022 00:00:00 GMT

A Simple JSON Backed Database in Typescript

Define the database

The database is defined as a class which has an in-memory store data and a persist method that allows for persisting the database to a file as well as content when an instance is created from an existing JSON file

The code for the database can be seen below:

import { existsSync, readFileSync, writeFileSync } from 'fs'

export class Database<TData> {
  public data: TData

  constructor(private readonly dbPath: string, initial: TData) {
    this.data = this.load(initial)
  }

  public update = (data: Partial<TData>) =>
    (this.data = { ...this.data, ...data })

  public commit = () => this.persist(this.data)

  private persist = (data: TData) =>
    writeFileSync(this.dbPath, JSON.stringify(data))

  private read = (): TData =>
    JSON.parse(readFileSync(this.dbPath).toString()) as TData

  private load = (initial: TData): TData => {
    if (!existsSync(this.dbPath)) {
      this.persist(initial)
    }

    const current = this.read()

    return {
      ...initial,
      ...current,
    }
  }
}

Usage

The database can be used by creating an instance and modifying the data in it by using the update method, and can be written to a file using the commit method:

import { Database } from './db'

interface User {
  name: string
  age: number
}

interface DB {
  users: User[]
}

const initial: DB = {
  users: [],
}

const db = new Database<DB>('./data.json', initial)

db.update({
  users: [
    {
      name: 'Test',
      age: 20,
    },
  ],
})

db.commit()

Functional Example

The above code is found in the below REPL as a runnable example:

Localhost HTTP Proxy with Node.js

Tue, 08 Mar 2022 00:00:00 GMT

Localhost HTTP Proxy with Node.js

An localhost HTTP proxy is useful for debugging and can be easily defined using Node.js by installing http-proxy

yarn add http-proxy

And then adding the following to an index.js file:

index.js

const httpProxy = require('http-proxy')

const target = 'http://my-target-website.com:1234'

httpProxy.createProxyServer({ target }).listen(8080)

Which will create a server that listens on 8080 and will proxy requests to the target

A type for getting properties common in all objects from a union

Fri, 08 Jul 2022 00:00:00 GMT

A type for getting properties common in all objects from a union

Overview

Something that may come up in practice is a use for a type that allows us to enforce that an object has only the common properties for a given object

For example, given the two objects below:

type AThing = {
  name: string
  age: number
  email: string
  phone: number
}

type BThing = {
  businessName: string[]
  email: string
  phone: string
}

I would like a type that contains only the things that these objects have in common, namely phone and email

A Type for Common Object Keys

This isn't something that typescript has out-of-the-box, however it can be implemented by using some type juggling

First, we define a type called CommonKeys which gets us all the keys which are common in the two objects

type CommonKeys<T, R = {}> = R extends T
  ? keyof T & CommonKeys<Exclude<T, R>>
  : keyof T

The CommonKeys type makes use of a condition to check if R which is the recursive type extends T which is the input type. Based on this, we cut down T one type at a time until there is no more object that can extend R, then for an input type T which is the same as R (an empty object) the result of CommonKeys<{}> will be never since {} has no keys, and will end the recursion

Applying this to the above types, we get:

type ABCommonKeys = CommonKeys<AThing | BThing>
// type ABCommonKeys = "email" | "phone"

And as a sanity check, we can also apply this to {}:

type Basic = CommonKeys<{}>
// type Basic = never

A Type for Common Object

Next, we can use the CommonKeys type defined above to create a Common type which wne used with the intersection will result in a type that has all the keys common in all types from the intersection

type Common<T> = Pick<T, CommonKeys<T>>

We can apply this now to a type of AThing | BThing like so:

type ABCommonObject = Common<AThing | BThing>
// type ABCommonObject = {
//   email: string;
//   phone: string | number;
// }

And we can see that we have the desired result which is an object with the properties that are common between both input object types

Final Solution

We can put the code from above together into the final solution which is just the two above types:

/** Gets the keys common to any type/union of `T` */
type CommonKeys<T, R = {}> = R extends T
  ? keyof T & CommonKeys<Exclude<T, R>>
  : keyof T

/** Gets an object type containing all keys that exist within a type/union `T` */
type Common<T> = Pick<T, CommonKeys<T>>

Visualizations with React

Fri, 11 Feb 2022 00:00:00 GMT

Visualizations with React

Data Visualization with D3 and React

React is a library for building reactive user interfaces using JavaScript (or Typescript) and D3 (short for Data-Driven Documents) is a set of libraries for working with visualizations based on data

Before getting started, I would recommend familiarity with SVG, React, and D3

Some good references for SVG are on the MDN SVG Docs

A good place to start for React would be the React Docs or my React Notes

And lastly, the D3 Docs

Getting Stared

To follow along, you will need to install Node.js and be comfortable using the terminal

I'm going to be using a React App with TypeScript initialized with Vite as follows:

yarn create vite

And then selecting the react-ts option when prompted. Next, install d3 from the project root with:

yarn add d3
yarn add --dev @types/d3

Now that we've got a basic project setup, we can start talking about D3

Scales (`d3-scale`)

d3-scale Documentation

Broadly, scales allow us to map from one set of values to another set of values,

Scales in D3 are a set of tools which map a dimension of data to a visual variable. They help us go from something like count in our data to something like width in our rendered SVG

We can create scales for a sample dataset like so:

type Datum = {
  name: string
  count: number
}

export const data: Datum[] = [
  { name: '🍊', count: 21 },
  { name: '🍇', count: 13 },
  { name: '🍏', count: 8 },
  { name: '🍌', count: 5 },
  { name: '🍐', count: 3 },
  { name: '🍋', count: 2 },
  { name: '🍎', count: 1 },
  { name: '🍉', count: 1 },
]

Also, a common thing to do when working with scales is to define margins around out image, this is done simply as an object like so:
const margin = {
  top: 20,
  right: 20,
  bottom: 20,
  left: 35,
}
This just helps us simplify some position/layout things down the line

Scales work by taking a value from the domain (data space) and returning a value from range (visual space):

const width = 600
const height = 400

const x = d3
  .scaleLinear()
  .domain([0, 10]) // values of the data space
  .range([0, width]) // values of the visual space

const position = x(3) // position = scale(value)

Additionally, there's also the invert method which goes the other way - from range to domain

const position = x(3) // position === 30
const value = x.invert(30) // value === 3

The invert method is useful for things like calculating a value from a mouse position

D3 has different Scale types:

Continuous (Linear, Power, Log, Identity, Time, Radial)
Sequential
Diverging
Quantize
Quantile
Threshold
Ordinal (Band, Point)

Continuous Scales

These scales map continuous data to other continuous data

D3 has a few different continuous scale types:

Linear
Power
Log
Identity
Radial
Time
Sequential Color

For my purposes at the moment I'm going to be looking at the methods for Linear and Sequential Color scales, but the documentation explains all of the above very thoroughly and is worth a read for additional information on their usage

Linear

We can use a linear scale in the fruit example for mapping count to an x width:

const maxX = d3.max(data, (d) => d.count) as number

const x = d3
  .scaleLinear<number>()
  .domain([0, maxX])
  .range([margin.left, width - margin.right])

If we don't want the custom domain to range interpolation we can create a custom interpolator. An interpolator is a function that takes a value from the domain and returns the resulting range value

D3 has a few different interpolators included for tasks such as interpolating colors or rounding values

We can create a custom color domain to interpolate over and use the interpolateHsl or interpolateRgb functions:

const color = d3
  .scaleLinear<string>()
  .domain([0, maxX])
  .range(['pink', 'lightgreen'])
  .interpolate(d3.interpolateHsl)

Sequential Color

If for some reason we want to use the pre-included color scales

The scaleSequential scale is a method that allows us to map to a color range using an interpolator.

D3 has a few different interpolators we can use with this function like d3.interpolatePurples, d3.interpolateRainbow or d3.interpolateCool among others which look quite nice

We can create a color scale using the d3.interpolatePurples which will map the data to a scale of purples:

const color = d3
  .scaleSequential()
  .domain([0, maxX])
  .interpolator(d3.interpolatePurples)

These can be used instead of the scaleLinear with interpolateHsl for example above but to provide a pre-calibrated color scale

Ordinal Scales

Ordinal scales have a discrete domain and range and are used for the mapping of discrete data. These are a good fit for mapping a scale with categorical data. D3 offers us the following scales:

Band Scale
Point Scale

Band Scale

A Band Scale is a type of Ordinal Scale where the output range is continuous and numeric

We can create a mapping for where each of our labels should be positioned with scaleBand:

const names = data.map((d) => d.name)

const y = d3
  .scaleBand()
  .domain(names)
  .range([margin.top, height - margin.bottom])
  .padding(0.1)

The domain can be any size array, unlike in the case of continuous scales where the are usually start and end values

Building a Bar Graph

When creating visuals with D3 there are a few different ways we can output to SVG data. D3 provides us with some methods for creating shapes and elements programmatically via a builder pattern - similar to how we create scales.

However, there are also cases where we would want to define out SVG elements manually, such as when working with React so that the react renderer can handle the rendering of the SVG elements and we can manage our DOM structure in a way that's a bit more representative of the way we work in React

The SVG Root

Every SVG image has to have an svg root element. To help ensure that this root scales correctly we also use it with a viewBox attribute which specifies which portion of the SVG is visible since the contents can go outside of the bounds of the View Box and we may not want to display this overflow content by default

Using the definitions for margin, width and height from before we can get the viewBox for the SVG we're trying to render like so:

const viewBox = `0 ${margin.top} ${width} ${height - margin.top}`

And then, using that value in the svg element:

return <svg viewBox={viewBox}>{/* we will render the graph in here */}</svg>

At this point we don't really have anything in the SVG, next up we'll do the following:

Add Bars to the SVG
Add Y Labels to the SVG
Add X Labels to the SVG

Bars

We can create Bars using the following:

const bars = data.map((d) => (
  <rect
    key={y(d.name)}
    fill={color(d.count)}
    y={y(d.name)}
    x={x(0)}
    width={x(d.count) - x(0)}
    height={y.bandwidth()}
  />
))

We make use of the x and y functions which help us get the positions for the rect as well as y.bandWidth() and x(d.count) to height and width for the element

We can then add that into the SVG using:

return (
  <svg viewBox={viewBox}>
    <g>{bars}</g>
  </svg>
)

At this point, the resulting SVG will look like this:

Y Labels

Next, using similar concepts as above, we can add the Y Labels:

const yLabels = data.map((d) => (
  <text key={y(d.name)} y={y(d.name)} x={0} dy="0.35em">
    {d.name}
  </text>
))

Next, we can add this into the SVG, and also wrapping the element in a g with a some basic text alignment and translation for positioning it correctly:

return (
  <svg viewBox={viewBox}>
    <g
      fill="steelblue"
      textAnchor="end"
      transform={`translate(${margin.left - 5}, ${y.bandwidth() / 2})`}
    >
      {yLabels}
    </g>
    <g>{bars}</g>
  </svg>
)

The state of the SVG at this point is:

X Labels

Next, we can add the X Labels over each rect using:

const xLabels = data.map((d) => (
  <text key={y(d.name)} y={y(d.name)} x={x(d.count)} dy="0.35em">
    {d.count}
  </text>
))

And the resulting code looks like this:

return (
  <svg viewBox={viewBox}>
    <g
      fill="steelblue"
      textAnchor="end"
      transform={`translate(${margin.left - 5}, ${y.bandwidth() / 2})`}
    >
      {yLabels}
    </g>
    <g>{bars}</g>
    <g
      fill="white"
      textAnchor="end"
      transform={`translate(-6, ${y.bandwidth() / 2})`}
    >
      {xLabels}
    </g>
  </svg>
)

And the final SVG:

Final Result

The code for the entire file/graph can be seen below:

Fruit.tsx

import React from 'react'
import * as d3 from 'd3'
import { data } from '../data/fruit'

const width = 600
const height = 400

const margin = {
  top: 20,
  right: 20,
  bottom: 20,
  left: 35,
}

const maxX = d3.max(data, (d) => d.count) as number

const x = d3
  .scaleLinear<number>()
  .domain([0, maxX])
  .range([margin.left, width - margin.right])
  .interpolate(d3.interpolateRound)

const names = data.map((d) => d.name)

const y = d3
  .scaleBand()
  .domain(names)
  .range([margin.top, height - margin.bottom])
  .padding(0.1)
  .round(true)

const color = d3
  .scaleSequential()
  .domain([0, maxX])
  .interpolator(d3.interpolateCool)

export const Fruit: React.FC = ({}) => {
  const viewBox = `0 ${margin.top} ${width} ${height - margin.top}`

  const yLabels = data.map((d) => (
    <text key={y(d.name)} y={y(d.name)} x={0} dy="0.35em">
      {d.name}
    </text>
  ))

  const bars = data.map((d) => (
    <rect
      key={y(d.name)}
      fill={color(d.count)}
      y={y(d.name)}
      x={x(0)}
      width={x(d.count) - x(0)}
      height={y.bandwidth()}
    />
  ))

  const xLabels = data.map((d) => (
    <text key={y(d.name)} y={y(d.name)} x={x(d.count)} dy="0.35em">
      {d.count}
    </text>
  ))

  return (
    <svg viewBox={viewBox}>
      <g
        fill="steelblue"
        textAnchor="end"
        transform={`translate(${margin.left - 5}, ${y.bandwidth() / 2})`}
      >
        {yLabels}
      </g>
      <g>{bars}</g>
      <g
        fill="white"
        textAnchor="end"
        transform={`translate(-6, ${y.bandwidth() / 2})`}
      >
        {xLabels}
      </g>
    </svg>
  )
}

Ticks and Grid Lines

Note that D3 includes a d3-axis package but that doesn't quite work given that we're manually creating the SVG using React and not D3's string-based rendering

We may want to add Ticks and Grid Lines on the X-Axis, we can do this using the scale's ticks method like so:

const xGrid = x.ticks().map((t) => (
  <g key={t}>
    <line
      stroke="lightgrey"
      x1={x(t)}
      y1={margin.top}
      x2={x(t)}
      y2={height - margin.bottom}
    />
    <text fill="darkgrey" textAnchor="middle" x={x(t)} y={height}>
      {t}
    </text>
  </g>
))

And then render this in the svg as:

return (
  <svg viewBox={viewBox}>
    <g>{xGrid}</g>
    {/* previous graph content */}
  </svg>
)

The result will look like this:

Building a Line Graph

We can apply all the same as in the Bar Graph before to draw a Line Graph. The example I'll be using consists of a Datum as follows:

export type Datum = {
  date: Date
  temp: number
}

Given that the X-Axis is a DateTime we will need to do some additional conversions as well as formatting

Working with Domains

In the context of this graph it would also be useful to have an automatically calculated domain instead of a hardcoded one as in the previous example

We can use the d3.extent function to calculate a domain:

const dateDomain = d3.extent(data, (d) => d.date) as [Date, Date]
const tempDomain = d3.extent(data, (d) => d.temp).reverse() as [number, number]

We can then use this domain definitions in a scale:

const tempScale = d3
  .scaleLinear<number>()
  .domain(tempDomain)
  .range([margin.top, height - margin.bottom])
  .interpolate(d3.interpolateRound)

const dateScale = d3
  .scaleTime()
  .domain(dateDomain)
  .range([margin.left, width - margin.right])

Create a Line

The d3.line function is useful for creating a d attribute for an SVG path element which defines the line segments

The line function requires x and y mappings. The line for the graph path can be seen as follows:

const line = d3
  .line<Datum>()
  .x((d) => dateScale(d.date))
  .y((d) => tempScale(d.temp))(data) as string

We also include the Datum type in the above to scope down the type of data allowed in the resulting function

Formatting

D3 includes functions for formatting DateTimes. We can create a formatter for a DateTime as follows:

const formatter = d3.timeFormat('%Y-%m')

We can then use the formatter like so:

formatter(dateTime)

Grid Lines

We can define the X Axis and grid lines similar to how we did it previously:

const xGrid = dateTicks.map((t) => (
  <g key={t.toString()}>
    <line
      stroke="lightgrey"
      x1={dateScale(t)}
      y1={margin.top}
      x2={dateScale(t)}
      y2={height - margin.bottom}
      strokeDasharray={4}
    />
    <text fill="darkgrey" textAnchor="middle" x={dateScale(t)} y={height}>
      {formatter(t)}
    </text>
  </g>
))

And the Y Axis grid lines:

const yGrid = tempTicks.map((t) => (
  <g key={t.toString()}>
    <line
      stroke="lightgrey"
      y1={tempScale(t)}
      x1={margin.left}
      y2={tempScale(t)}
      x2={width - margin.right}
      strokeDasharray={4}
    />
    <text fill="darkgrey" textAnchor="end" y={tempScale(t)} x={margin.left - 5}>
      {t}
    </text>
  </g>
))

Final result

Using all the values that have been defined above, we can create the overall graph and grid lines like so:

return (
  <svg viewBox={viewBox}>
    <g>{xGrid}</g>
    <g>{yGrid}</g>
    <path d={line} stroke="steelblue" fill="none" />
  </svg>
)

The final code can be seen below:

Temperature.tsx

import React from 'react'
import * as d3 from 'd3'
import { data, Datum } from '../data/temperature'

const width = 600
const height = 400

const margin = {
  top: 20,
  right: 50,
  bottom: 20,
  left: 50,
}

const tempDomain = d3.extent(data, (d) => d.temp).reverse() as [number, number]

const tempScale = d3
  .scaleLinear<number>()
  .domain(tempDomain)
  .range([margin.top, height - margin.bottom])
  .interpolate(d3.interpolateRound)

const tempTicks = tempScale.ticks()

const dateDomain = d3.extent(data, (d) => d.date) as [Date, Date]

const dateScale = d3
  .scaleTime()
  .domain(dateDomain)
  .range([margin.left, width - margin.right])

const dateTicks = dateScale.ticks(5).concat(dateScale.domain())

const line = d3
  .line<Datum>()
  .x((d) => dateScale(d.date))
  .y((d) => tempScale(d.temp))(data) as string

const formatter = d3.timeFormat('%Y-%m')

export const Temperature: React.FC = ({}) => {
  const viewBox = `0 0 ${width} ${height}`

  const xGrid = dateTicks.map((t) => (
    <g key={t.toString()}>
      <line
        stroke="lightgrey"
        x1={dateScale(t)}
        y1={margin.top}
        x2={dateScale(t)}
        y2={height - margin.bottom}
        strokeDasharray={4}
      />
      <text fill="darkgrey" textAnchor="middle" x={dateScale(t)} y={height}>
        {formatter(t)}
      </text>
    </g>
  ))

  const yGrid = tempTicks.map((t) => (
    <g key={t.toString()}>
      <line
        stroke="lightgrey"
        y1={tempScale(t)}
        x1={margin.left}
        y2={tempScale(t)}
        x2={width - margin.right}
        strokeDasharray={4}
      />
      <text
        fill="darkgrey"
        textAnchor="end"
        y={tempScale(t)}
        x={margin.left - 5}
      >
        {t}
      </text>
    </g>
  ))

  return (
    <svg viewBox={viewBox}>
      <g>{xGrid}</g>
      <g>{yGrid}</g>
      <g>
        <path d={line} stroke="steelblue" fill="none" />
      </g>
    </svg>
  )
}

And the resulting SVG will then look something like this:

Expanding Bottom Navigation with CSS Transitions

Sun, 13 Nov 2022 00:00:00 GMT

Expanding Bottom Navigation with CSS Transitions

So I was looking around on YouTube and came across a video about this library called google_nav_bar for flutter and I really liked the idea of how it works and wanted to implement something similar in an app I've been working on

The basic functionality of the of the library can be seen on google_nav_bar package page

I thought the main challenge of this would be implementing the fade-in of the text with the expanding content section, I took a first shot with the result below:

<script>

	let selected = 'home';

	let items = ['home', 'search', 'archive', 'settings'];
	let colors = {
		home: 'lightpink',
		search: 'violet',
		archive: 'skyblue',
		settings: 'lightgrey'
	};
</script>

<svelte:head>
	<link rel="stylesheet" href="https://unpkg.com/mono-icons@1.0.5/iconfont/icons.css" />
</svelte:head>

<nav class="wrapper">
	<ul class="list">
		{#each items as item (item)}
			<li class="item" class:selected={selected === item} on:click={() => (selected = item)}>
				<div class="icon" style={`--bg: ${colors[item]}`}>
					<i class={`mi mi-${item}`} />
				</div>
				<div class="text">
					{item}
				</div>
			</li>
		{/each}
	</ul>
</nav>

<style>
	.mi {
		font-size: 24px;
	}

	.wrapper {
		display: flex;
		width: 100%;
	}

	.list {
		flex: 1;
		display: flex;
		flex-direction: row;
		align-items: center;
		justify-content: space-between;
		padding: 10px 20px;
	}

	.item {
		display: flex;
		flex-direction: row;
		align-items: center;
		gap: 8px;
		justify-content: flex-start;
	}

	.item .text {
		width: 0px;
		opacity: 0;
		overflow: hidden;
		transition: width 0.15s 0s, opacity 0.15s 0s;
	}

	.item.selected .text {
		width: 100px;
		opacity: 1;
		transition: width 0.15s 0s, opacity 0.15s 0.075s;
	}

	.icon {
		display: flex;
		align-items: center;
		justify-content: center;
		color: var(--bg);
		line-height: 0;
	}

	/* resets	 */
	ul {
		padding: 0;
		margin: 0;
	}

	li {
		margin: 0;
		list-style: none;
	}
</style>

I like the overall feel and I think the finicky bits of using the transition are done, but I'd still like to play around a bit more to see how close to the original library I can get it

For now though, here's the REPL with the current working state of the component:

Typescript Utilities

Tue, 13 Dec 2022 00:00:00 GMT

Typescript Utilities

Here's a collection of some utility types I've put together:

Functions

ts-utils/functions.ts

import { Reverse } from './arrays'

/**
 * A function with defined input and output
 */
export type Fn<TParams, TResult> = (params: TParams) => TResult

/**
 * A function that returns a result or undefined
 */
export type OptionFn<TParams, TResult> = Fn<TParams, TResult | undefined>

/**
 * A function that returns void
 */
export type VoidFn<TParams> = (params: TParams) => void

/**
 * Convert a function type into an async version of that function
 */
export type Async<TFn extends (...args: any[]) => any> = (
  ...args: Parameters<TFn>
) => Promise<ReturnType<TFn>>

/**
 * Create an async version of `Fn`
 */
export type AsyncFn<TParams, TResult> = Async<Fn<TParams, TResult>>

/**
 * Create an async version of `OptionFn`
 */
export type AsyncOptionFn<TParams, TResult> = Async<OptionFn<TParams, TResult>>

/**
 * Create an async version of `VoidFn`
 */
export type AsyncVoidFn<TParams> = Async<VoidFn<TParams>>

/**
 * Create a version of a function that may either be sync or async
 */
export type SyncOrAsync<TFn extends (...args: any[]) => any> = (
  ...args: Parameters<TFn>
) => ReturnType<TFn> | Promise<ReturnType<TFn>>

/**
 * Create a version of `Fn` that may either be sync or async
 */
export type SyncOrAsyncFn<TParams, TResult> = SyncOrAsync<Fn<TParams, TResult>>

/**
 * Create a version of `OptionFn` that may either be sync or async
 */
export type SyncOrAsyncOptionFn<TParams, TResult> = SyncOrAsync<
  OptionFn<TParams, TResult>
>

/**
 * Create a version of `VoidFn` that may either be sync or async
 */
export type SyncOrAsyncVoidFn<TParams> = SyncOrAsync<VoidFn<TParams>>

/**
 * Returns a function with the arguments reversed
 */
export type ReverseArguments<T extends (...args: any[]) => any> = (
  ...args: Reverse<Parameters<T>>
) => ReturnType<T>

Arrays

ts-utils/arrays.ts

/**
 * An array with at least 1 item in it
 */
export type NonEmptyArray<T> = [T, ...T[]]

/**
 * Array filter type, used to filter an array via the `.filter` method
 */
export type ArrayFilter<T> = (value: T, index: number, array: T[]) => boolean

/**
 * Array map type, used to map an array via the `.map` method
 */
export type ArrayMap<T, U> = (value: T, index: number, array: T[]) => U

/**
 * Array reduce type, used to reduce an array via the `.reduce` method
 */
export type ArrayReducer<T, U> = (
  previousValue: U,
  currentValue: T,
  currentIndex: number,
  array: T[],
) => U

/**
 * Defines a step type that may contain different data types as an array such
 * that each step adds or removes from the current input but should always
 * result in the same final object that can be validated by just verifying the
 * final length assuming all steps have been validated prior
 */
export type StepsOf<Final extends Readonly<any[]>> = Final extends [
  ...infer Next,
  infer _,
]
  ? StepsOf<Next> | Final
  : []

/**
 * Reverses the input tuple
 */
export type Reverse<T extends Array<unknown>> = T extends [
  infer First,
  ...infer Rest,
]
  ? [...Reverse<Rest>, First]
  : []

/**
 * Remove first value from array
 */
export type Eat<T extends unknown[]> = T extends [infer _, ...infer R]
  ? R
  : never

/**
 * Remove last value from array
 */
export type EatLast<T extends unknown[]> = T extends [...infer R, infer _]
  ? R
  : never

/**
 * Gets all possible indexes of A as an array
 */
export type Indexes<A extends any[]> = A extends [unknown, ... infer I] ? [...Indexes<I>, I['length']] : []

/**
 * Gets all possible indexes of A as a union
 */
export type Index<A extends any[]> = Indexes<A>[number]

/**
 * Exclude the given T from Arr
 */
export type ArrExclude<Arr, T, Acc extends unknown [] = []> = 
  Arr extends [infer F, ...infer R]
  ? T extends F
    ? ArrExclude<R, T, Acc>
    : ArrExclude<R, T, [...Acc, F]>
  : Acc

Objects

ts-utils/objects.ts

/**
 * Definition for a type guard that checks if a value is of a specific type
 */
export type Guard<TResult, TOther = unknown> = (
  value: TResult | TOther,
) => value is TResult

/**
 * Create a type where the provided keys are optional
 *
 * @param T the base type
 * @param O the keys to make optional
 */
export type WithOptional<T extends {}, O extends keyof T> = Omit<T, O> &
  Partial<Pick<T, O>>

/**
 * Create a type where the provided keys are required
 *
 * @param T the base type
 * @param R the keys to make required
 */
export type WithRequired<T extends {}, R extends keyof T> = T &
  Required<Pick<T, R>>

/**
 * Create a type where all all properties and sub-properties are recursively partial unless they are
 * of the type specified in TKeep
 *
 * @param T the base type
 * @param TKeep types to not make partial
 */
export type DeepPartial<T, TKeep = never> = T extends TKeep
  ? T
  : T extends object
    ? {
        [P in keyof T]?: DeepPartial<T[P], TKeep>
      }
    : T

/**
 * Returns a specific subset of `keyof T`
 *
 * The resulting types can be used with utilities like `Omit` or `Pick` in a reusable manner
 *
 * @param T the base type
 * @param K keys of T
 */
export type Keys<T, K extends keyof T> = K

export type Primitive = string | number | boolean | Symbol | Date

/**
 * Create a type where all direct properties are optional if they're Primitive otherwise it will be
 * a partial of the property
 *
 * @param T the base type
 * @param TKeep the types to not partialize
 */
export type FlatPartial<T, TKeep = Primitive> = {
  [K in keyof T]: T[K] extends TKeep ? T[K] | undefined : Partial<T[K]>
}

/**
 * Gets all keys of an object where the property at that key in the object extends a condition
 *
 * @param Cond the condition that properties need to  extend
 * @param T the object with the keys of interest
 */
type ConditionalKeys<Cond, T> = {
  [K in keyof T]: T[K] extends Cond ? K : never
}[keyof T]

/**
 * Get the primitive keys of a given object
 */
type PrimitiveKeys<T> = ConditionalKeys<Primitive, T>

/**
 * Pick the keys of an object where the property value is a primitive
 */
type PickPrimitive<T> = Pick<T, PrimitiveKeys<T>>

/**
 * Pick the keys of an object where the property value is not primitive
 */
type ObjectKeys<T> = Exclude<keyof T, PrimitiveKeys<T>>

/**
 * Join two keys using a separator
 */
type JoinKeys<
  TSep extends string,
  P extends string,
  K extends string,
> = `${P}${TSep}${K}`

/**
 * Create an object with all keys prepended with some prefix
 *
 * @param T the type with the keys of interest
 * @param P the prefix to prepend keys with
 * @param Sep the separator to use for nesting keys
 */
type ExpandPathsRec<T, P extends string, Sep extends string> = {
  [K in keyof T & string]: T[K] extends Primitive
    ? {
        [key in JoinKeys<Sep, P, K>]: T[K]
      }
    : ExpandPathsRec<T[K], JoinKeys<Sep, P, K>, Sep>
}[keyof T & string]

/**
 * Create the resultant nested object using the given keys of the input object
 *
 * @param T the object to un-nest
 * @param P the keys to keep when un-nesting
 */
type ExpandPaths<T, P extends keyof T, Sep extends string> = P extends string
  ? ExpandPathsRec<T[P], P, Sep>
  : {}

/**
 * Bring all object properties to the top-level recursively
 *
 * @param T the object to expand
 * @param Sep the separator to use when expanding
 */
type Pathify<T, Sep extends string> = PickPrimitive<T> &
  ExpandPaths<T, ObjectKeys<T>, Sep>

/**
 * Get object with only keys that are an array
 */
export type PickArrays<T> = {
  [K in keyof T as T[K] extends Array<any> ? K : never]: T[K]
}

/**
 * Get the keys of the object `Obj` that start with the given string `P`
 *
 * @param P the string that a key should be prefixed with
 * @param Obj the object from which to take the matching keys
 */
export type KeysStartingWith<P extends string, Obj> = keyof Obj &
  `${P}${string}`

/**
 * Get the keys of the object `Obj` that end with the given string `S`
 *
 * @param S the string that a key should be suffixed with
 * @param Obj the object from which to take the matching keys
 */
export type KeysEndingWith<S extends string, Obj> = keyof Obj & `${string}${S}`

/**
 * Get a type or else a fallback if it is `never`
 *
 * @param T the base type
 * @param F the fallback to use if `T extends never`
 */
export type OrElse<T, F> = T extends never ? F : T

/**
  * Check if two types are exactly equal
  *
  * This is a direct copy from type-challenges:
  * https://github.com/type-challenges/type-challenges/blob/main/utils/index.d.ts
  */
export type Equal<X, Y> =
  (<T>() => T extends X ? 1 : 2) extends
  (<T>() => T extends Y ? 1 : 2) ? true : false

Strings

ts-utils/strings.ts

import { NonEmptyArray, Reverse } from './arrays'

/**
 * Types that can be cleanly/predictably converted into a string
 */
export type Stringable = Exclude<string | number, ''>

/**
 * String type that will show suggestions but accept any string
 *
 * @param TSugg the strings that you want to appear as IDE suggestions
 */
export type SuggestedStrings<TSugg> = TSugg | (string & {})

/**
 * Joins stringable members into a single, typed, string
 */
export type Join<TSep extends string, T extends Array<Stringable>> = T extends [
  infer El,
  ...infer Rest,
]
  ? El extends Stringable
    ? Rest extends NonEmptyArray<Stringable>
      ? `${El}${TSep}${Join<TSep, Rest>}`
      : El
    : ''
  : ''

/**
 * Split a strongly typed string into its parts
 */
export type Split<
  TSep extends string,
  TStr extends string,
  TBase extends string[] = [],
> = TStr extends `${infer El}${TSep}${infer Rest}`
  ? [...TBase, El, ...Split<TSep, Rest>]
  : [TStr]

/**
 * Get the chars of a strongly typed string as an array
 */
export type Chars<
  TStr extends string,
  TBase extends string[] = [],
> = TStr extends `${infer El}${''}${infer Rest}`
  ? Rest extends ''
    ? [TStr]
    : [...TBase, El, ...Chars<Rest>]
  : never

/**
 * Get the length of a string
 */
export type Length<TStr extends string> = Chars<TStr>['length']

/**
 * Get the first character of a string
 */
export type First<T extends string> = T extends `${infer F}${string}`
  ? F
  : never

/**
 * Get the last character of a string
 */
export type Last<TStr extends string> = Reverse<Chars<TStr>>[0]

/**
 * Get a string with the first character omitted
 */
export type Eat<T extends string> = T extends `${string}${infer E}` ? E : never

JSON Schema

Type for inferring a simple JSON Schema into a TypeScript type. I think this will become super useful once Typescript adds support for importing JSON values as const. This will mean that using the schema would be something like:

// currently the `const` part doesn't exist, but would be super cool if we could get here
import schema from './my-schema.json' with { type: 'json', const: true };

export type MySchema = FromJSONSchema<typeof schema>;

The open issue that would make the above possible is this: https://github.com/microsoft/TypeScript/issues/32063

It's relatively trivial to implement some kind of pre-build script that could do this for all JSON schemas but I think it's worth waiting till the TS part of this solution exists because it would be great to be able to infer this statically

ts-utils/json-schema.ts

/**
 * Relevant schema structure for inferring type definition
 */
interface PartialJSONSchema<T extends Properties = Properties> {
  type: 'object'
  properties: T
  required?: Array<keyof T>
}

type TypeMap = {
  string: string
  boolean: boolean
}

type Type = TypeMap[keyof TypeMap]

type Properties = Record<string, { type: keyof TypeMap; default?: unknown }>

/**
 * Create a type where the provided keys are required
 *
 * @param T the base type
 * @param R the keys to make required
 */
type WithRequired<T, R extends keyof T> = T & Required<Pick<T, R>>

/**
 * Keys that have been defaulted by the
 */
type DefaultedKeys<T extends Properties> = {
  [K in keyof T]: T[K]['default'] extends Type ? K : never
}[keyof T]

/**
 * A field is required if its default value is in the list of required keys
 */
type Structure<T extends PartialJSONSchema> = WithRequired<
  {
    [K in keyof T['properties']]?: TypeMap[T['properties'][K]['type']]
  },
  DefaultedKeys<T['properties']>
>

/**
 * Extracts inner resultant type to make IDE inferrence/display better
 */
type Simplify<T> = {
  [KeyType in keyof T]: T[KeyType]
}

/**
 * Get the typescript type inferred by a JSON Schema
 */
export type FromJSONSchema<T extends PartialJSONSchema> = Simplify<
  T['required'] extends Array<keyof T['properties']>
    ? WithRequired<Structure<T>, T['required'][number]>
    : Structure<T>
>

Podcast and RSS Reader App Design

Wed, 14 Dec 2022 00:00:00 GMT

Podcast and RSS Reader App Design

I've been developing an app for reading RSS feeds and listening to podcasts for personal use recently, and I finally had some inspiration as far as the design goes. Though it's ever-changing, here's a preview of the design at this point

The app is meant to provide a reading space for RSS feeds as well as allow a user to bookmark articles and tweets for later reading. In addition, it provides a way to listen to podcasts while browsing other content that's been added in a way that feels consistent and easy to use across all platforms

Mobile

I've tried to remove anything that's not absolutely essential from the overall app with the goal of preventing clutter on the mobile app. In order to do this I've made the bottom navigation something that adapts and shows the user relevant actions based on the screen they're on as opposed to providing a combination of local and global menus as is normally seen in mobile apps

For the navigation I'm liking the idea of a static bottom navigation that transforms and adapts as the user moves through the app - this is a decision that's persisted with the desktop design as well because of how it aids in delivering a cohesive cross-platform experience

Light

For the light theme I've settled on a warm-ish background to link up with the idea of "highlighting" when reading as one might do on paper

Dark

For the dark color scheme I wanted to bring something interesting to the color palette since the typical "black" style was feeling boring. I figured color was a way to add some interest to the relatively text-heavy UI and the green was chosen for the feeling of comfort it brings which should bring a softer feel for reading

Desktop

For the desktop view my focus was to keep things pretty minimal and reduce the visual noise, focusing on the reader experience while being consistent with the mobile view, offering some simple style and layout modifications to better make use of the available space

As far as the inbox view goes, there are some simple tabs at the top that allow switching between the different article types as in the mobile case but aside from some spacing and font size tweaks everything else is pretty much aligned to the mobile style

For the article view I'm taking advantage of the added space in the bottom to add some of the actions that are at the top of the screen in the mobile view - this helps make better use of the space and keep the reading canvas clean as well as isolate actions into a single place which should make it easier to navigate around the app using a mouse

Light

Dark

Smooth Bottom Navigator with Secondary Actions

Thu, 15 Dec 2022 00:00:00 GMT

Smooth Bottom Navigator with Secondary Actions

Not dissimilar from my previous post on the expanding bottom nav this expands on the idea of animating between states in the navigator in a more global way, this version makes use of a similar animation/transition pattern but does so by modifying the heights as well as the absolute positioning of different components

The implementation below takes the overall concept to the simplest possible state, however some refinements that can still be made include being responsive to the size of the additional content as well as doing a more accurate calculation on how large the sliding tab should be and how it's placed

The values for the padding/positions are very hardcoded, but in practice you'd likely want to make these respond to data provided and size appropriately in regards to rest of the component

Here's the svelte code below:

<script>
  import {
    InboxIcon,
    HomeIcon,
    DatabaseIcon,
    MessageCircleIcon,
    MicIcon,
    MusicIcon,
  } from "svelte-feather-icons";

  let selected = "home";

  const icons = [
    {
      id: "home",
      icon: HomeIcon,
    },
    {
      id: "database",
      icon: DatabaseIcon,
    },
    {
      id: "message",
      icon: MessageCircleIcon,
    },
    {
      id: "mic",
      icon: MicIcon,
    },
  ];

  $: selectedIndex = icons.findIndex((item) => item.id === selected);
  $: expanded = selected === "mic";
</script>

<h1>{selected}</h1>

<div class="wrapper">
  <div class="background" class:expanded class:collapsed={!expanded}>
    <div
      class="slider"
      style="--left: {(selectedIndex / icons.length) * 100}%;--right: {((selectedIndex+1) / icons.length) * 100}%;"
    />
    <div class="content" class:expanded class:collapsed={!expanded}>
      <MusicIcon />
      <p>Recording: 00:01:23</p>
    </div>
  </div>

  <div class="items">
    {#each icons as icon}
      <div
        class="item"
        on:click={() => (selected = icon.id)}
        on:keypress={console.log}
      >
        <svelte:component this={icon.icon} />
      </div>
    {/each}
  </div>
</div>

<style global>
  /* uses fixed postion in order to lock it to lock the component to the bototom of the screen */
  .wrapper {
    position: fixed;
    width: 100vw;
    bottom: 0px;
  }

  /* ensure the background and items are all placed the same since they need to overlap	 */
  .items,
  .background {
    height: 0px;
    position: absolute;
    left: 0px;
    bottom: 20px;
    right: 0px;
    max-width: 80vw;
    margin-left: auto;
    margin-right: auto;
  }

  .background {
    border: solid 1px black;
    border-radius: 16px;
    background-color: white;
    height: 52px;

    transition: height 300ms ease-in-out;
  }

  .background.collapsed {
    transition-delay: 150ms;
  }

  .background.expanded {
    height: 100px;
    transition-delay: 0ms;
  }

  .content {
    overflow: hidden;
    display: flex;
    flex-direction: row;
    gap: 16px;
    height: 24px;
    opacity: 1;
    padding: 20px 40px;
    transition: all 300ms ease-in-out;
  }

  .content.collapsed {
    height: 0;
    opacity: 0;
    padding: 0px 40px;
    transition-delay: 0ms;
  }

  .content.expanded {
    height: 24px;
    opacity: 1;
    transition-delay: 150ms;
  }

  .content p {
    margin: 0;
  }

  .slider {
    position: absolute;
    left: calc(var(--left) + 6px);
    right: va(--left);
    bottom: 6px;
    height: 40px;
    width: calc(var(--right) - var(--left) - 12px);
    border-radius: 12px;
    background-color: #87b5eb70;
    transition: left 300ms ease-in-out;
  }

  .items {
    display: flex;
    flex-direction: row;
    justify-content: space-around;
    height: 40px;
  }

  /* set the icon color	 */
  .item {
    color: black;
  }

  /* select a better default font */
  * {
    font-family: Arial, Helvetica, sans-serif;
    margin: 0;
    padding: 0;
  }
</style>

And the current version of the component can be seen here:

Using React.memo for Controlling Component Rendering

Tue, 16 Aug 2022 00:00:00 GMT

Using React.memo for Controlling Component Rendering

React Top Level API

The React library contains some functions at it's top level scope. Amongst these are the built-in hooks (like useState, useCallback, etc.) as well as some other functions for manipulating React Elements directly - which I've covered in a previous post on The React Top Level API

Component Rendering

By default, React will trigger a component render whenever there is a change to its state or props. React.memo allows us to take control of the props triggered render by giving us a way to look into the prop-change process

React.memo is a higher order component (HOC) that allows us to wrap a component and control whether or not it is updated/re-rendered by defining a function that tells react whether or not it's props are different - and effectively whether this should trigger a new render

Doing the above is useful for complex components that don't necessarily need to be rendered every time their props are changed

API Definition

The React Docs give us the following example for the React.memo HOC:

const MyComponent = (props) => {
  /* render using props */
}

const areEqual = (prevProps, nextProps) => {
  /*
  return true if passing nextProps to render would return
  the same result as passing prevProps to render,
  otherwise return false
  */
}

const MyComponentMemo = React.memo(MyComponent, areEqual)

The component MyComponent will be rendered whenever props are changed, however, using React.memo lets us define a function called areEqual that we can can use to tell React.memo whether or not the new props would render a different result to the old props

We can then use MyComponentMemo in place of MyComponent to take control of when the component is rendered

Rendering On a Specific Type of Change

Say we have the specific component TimeDisplay which shows the time that's being passed into it from App:

import './App.css'
import React, { useState, useEffect } from 'react'

interface TimeDisplayProps {
  time: number
}

const TimeDisplay: React.FC<TimeDisplayProps> = ({ time }) => {
  const display = new Date(time).toString()

  return <h1>{display}</h1>
}

export default function App() {
  const [time, setTime] = useState(Date.now())

  useEffect(() => {
    const handle = setInterval(() => {
      setTime(Date.now())
    }, 100)

    return () => {
      clearInterval(handle)
    }
  }, [])

  return (
    <main>
      <TimeDisplay time={time} />
    </main>
  )
}

The TimeDisplay component in our case only displays time to the second, so any millisecond-level changes don't matter to the component and so we can save on those renders by checking if the difference in time is similar to the previous render's time

Let's assume for our purpose that it's acceptable for the time to be delayed by about 5 seconds, we then can define a function called areTimesWithinOneSecond which compares the next render's props with the previous and returns if they are within 5 seconds of each other:

const areTimesWithinFiveSeconds = (
  prev: TimeDisplayProps,
  next: TimeDisplayProps
): boolean => {
  const diff = next.time - prev.time

  return diff < 5000
}

We can use the above function in a React.memo to define a version of the TimeDisplay component which will prevent unnecessary renders:

const TimeDisplayMemo = React.memo(TimeDisplay, areTimesWithinFiveSeconds)

And it can then be used as a drop-in replacement for the TimeDisplay component:

export default function App() {
  const [time, setTime] = useState(Date.now())

  useEffect(() => {
    const handle = setInterval(() => {
      setTime(Date.now())
    }, 100)

    return () => {
      clearInterval(handle)
    }
  }, [])

  return (
    <main>
      <TimeDisplayMemo time={time} />
    </main>
  )
}

Conclusion

From the above implementation we can see that it's possible to delay rendering of a component using React.memo if the component doesn't need to be re-rendered, hence improving performance by decreasing the number of renders react needs to carry out

REPL

The REPL with the example above can seen below:

References

The React Docs

Un-editable sections inside of a content editable

Wed, 16 Nov 2022 00:00:00 GMT

Un-editable sections inside of a content editable

I was working on an EditorJS plugin and needed to have a way of creating a block that would restrict certain user interactions

Initially, I tried to make use of event listeners to block the undesirable events, but the behavior on mobile was inconsistent and I wanted to avoid the mobile keyboard popping up as well when needed

In addition to the above, I also noticed that even when blocking the events using JavaScript, though the debugger didn't register the event, the user was still able to make changes to the sections being blocked

The solution proved to be fairly simple, though only tested on chrome so it may not be robust.

In my case, what worked was the following:

<div contenteditable="true">
  <div contenteditable="false">
    I should be uneditable, and on mobile won't even trigger a keyboard popup
  </div>
</div>

I should be uneditable, and on mobile won't even trigger a keyboard popup

Now, more generally, we can have other editable content while still having uneditable sections

<div contenteditable="true">
  I continue to be editable

  <div contenteditable="false">
    But I should be uneditable, and on mobile won't even trigger a keyboard
    popup
  </div>
</div>

I continue to be editable

But I should be uneditable, and on mobile won't even trigger a keyboard popup

Binary Data and File Formats

Sat, 20 Aug 2022 00:00:00 GMT

Binary Data and File Formats

Introduction

Computers work with data stored in binary formats. Reading and interpreting binary data is an important part of understanding file formats so their data can be read and interpreted

Bits and Bytes, and Binary

Binary files store data using bits

A bit can be either a 1 or 0. When working with bit-data, it's useful to group them into sets of 8, known as bytes

A byte is a sequence of 8-bits which can contain a value ranging from 0 to 255 - these are referred to as the decimal representation

Bytes consist of 8-bit, with each position representing a power of 2 from 2^0 to 2^7, as seen below:

0 0 0 0 0 0 0 0
| | | | | | | |
| | | | | | | |__ 2^0 - 0 or 1
| | | | | | |____ 2^1 - 0 or 2
| | | | | |______ 2^2 - 0 or 4
| | | | |________ 2^3 - 0 or 8
| | | |__________ 2^4 - 0 or 16
| | |____________ 2^5 - 0 or 32
| |______________ 2^6 - 0 or 64
|________________ 2^7 - 0 or 128

total:                  0 to 255

The byte above represents the value for 0, this is because all the bits have a value of 0

Using the above explanation, the number 1 is represented using the following:

0 0 0 0 0 0 0 1
| | | | | | | |
| | | | | | | |__ 2^0 - 1
| | | | | | |____ 2^1 - 0
| | | | | |______ 2^2 - 0
| | | | |________ 2^3 - 0
| | | |__________ 2^4 - 0
| | |____________ 2^5 - 0
| |______________ 2^6 - 0
|________________ 2^7 - 0

total:                  1

Where the position for 2^0 is the only bit with a value (1)

Similarly, 2 is represented as:

0 0 0 0 0 0 2 0
| | | | | | | |
| | | | | | | |__ 2^0 - 0
| | | | | | |____ 2^1 - 2
| | | | | |______ 2^2 - 0
| | | | |________ 2^3 - 0
| | | |__________ 2^4 - 0
| | |____________ 2^5 - 0
| |______________ 2^6 - 0
|________________ 2^7 - 0

total:                  2

Where the bit for 2^1 has a value

Or the number 5 with bits 2^0 and 2^2 having a value:

0 0 0 0 0 1 0 1
| | | | | | | |
| | | | | | | |__ 2^0 - 1
| | | | | | |____ 2^1 - 0
| | | | | |______ 2^2 - 4
| | | | |________ 2^3 - 0
| | | |__________ 2^4 - 0
| | |____________ 2^5 - 0
| |______________ 2^6 - 0
|________________ 2^7 - 0

total:                  5

Which is calculated by adding 2^0 + 2^2 = 1 + 4 = 5

A larger number, like 234 is:

1 1 1 0 1 0 1 0
| | | | | | | |
| | | | | | | |__ 2^0 - 0
| | | | | | |____ 2^1 - 2
| | | | | |______ 2^2 - 0
| | | | |________ 2^3 - 8
| | | |__________ 2^4 - 0
| | |____________ 2^5 - 32
| |______________ 2^6 - 64
|________________ 2^7 - 128

total:                  234

The calculation for the above value is:

2^1 + 2^3 + 2^5 + 2^6 + 2^7 = total = 234

When substituting the powers of 2:

2 + 8 + 32 + 64 + 128 = total = 234

The numbers discussed above are all 1-byte (8-bit) numbers, which have a range between 0 and 255, adding bits to the value will allow the representation of bigger numbers, for example, a 2-byte (16-bit) number can have a value from 0 to 65,535

Hexadecimal (Hex)

In the above example, numbers are represented in binary format (e.g. 000000020), or decimal format (e.g. 2)

When looking at binary data, it can be a bit easier to navigate around by representing data in hexadecimal (hex) format - which represents every 4 bits as a value ranging from 0-15, so, similar to the byte example above, but instead using 4-bits:

0 0 0 0
| | | |
| | | |__ 2^0 - 0 or 1
| | |____ 2^1 - 0 or 2
| |______ 2^2 - 0 or 4
|________ 2^3 - 0 or 8

Using what's already been discussed, the number 12 can be represented in bits as 1100,

Hex numbers additionally convert each of these values into a value from 0-9 or A-F, as seen in the following table:

Decimal	Bits	Hex
0	`0000`	`0`
1	`0001`	`1`
2	`0010`	`2`
3	`0011`	`3`
4	`0100`	`4`
5	`0101`	`5`
6	`0110`	`6`
7	`0111`	`7`
8	`1000`	`8`
9	`1001`	`9`
10	`1010`	`A`
11	`1011`	`B`
12	`1100`	`C`
13	`1101`	`D`
14	`1110`	`E`
15	`1111`	`F`

Using the binary representation, a byte can be represented using 2 Hex values which are taken by using the first 4-bits as the first hex value, and the second 4-bits as the second hex value

For example, the value for 234, represented as bits: 11101010 can be split into 2 sets of 4-bits 1110 1010, using the table above, this becomes EA in hex

Binary Files

Binary files encode data using bits, when viewing them, it's convenient to view the data in them using bytes represented as hex values as was shown above

Binary files usually require some knowledge of how their data is structured to correctly interpret the information. This is usually described in the specification for the file format

Text Files

Plain text files are usually in the UTF-8 or UTF-16 format - this means that they use 8-bits or 16-bits to represent each character, but there are lots of other formats that a text file can use

UTF-8 data can be read by converting the binary data to text data using a table which maps the byte/hex value to the character -for the UTF-8 format, the character "A" is encoded in hex as 41 and "z" is 7a

The Hexadecimal representation for a text file that contains the following UTF-8 data:

Hello World!

Would be stored as a binary file which contains:

48 65 6C 6C 6F 20 57 6F 72 6C 64 21

Putting the hex below the text content, the hex to character mapping can be seen:

H  e  l  l  o     W  o  r  l  d  !
48 65 6C 6C 6F 20 57 6F 72 6C 64 21

Binary files can be viewed in a hex editor to see the raw binary data, but interpreting these files depends on the format used and will differ between file formats

Conclusion

Computers store data using bits. Bits can be structured into sets of 8-bits, called a byte

Data can be represented using bits, bytes, decimal values, or hexadecimal values

Files store data using binary. Binary data can be represented as decimal or hex depending and can be viewed as whichever is appropriate

References

Virtual Machine vs Containers

Tue, 20 Dec 2022 00:00:00 GMT

Virtual Machine vs Containers

Virtual Machines

The below diagram illustrates a VM as it would traditionally be run along with it's relevant applications and dependencies:

Containers

The below diagram shows conainers running on a Container runtime in which they share dependencies as well as shows the relationship between an image and container instances

Read Metadata from Images using Rust

Thu, 25 Aug 2022 00:00:00 GMT

Read Metadata from Images using Rust

The complete Rust code discussed in this post can be found in the exiflib GitHub repo

Introduction

Image files, such as JPEG, PNG, and RAW formats from digital cameras and software, contain metadata about the image. This metadata can contain information ranging from the make and model of the camera used to the specific shooting conditions under which a picture was taken

Reading this data depends on the image format used. This post looks at specifically reading metadata from images that use the Exchangeable Image File Format (EXIF) for storing metadata

The Rust Programming Language

The Rust programming language is used to read and process the image files. Rust is a general-purpose programming language with an emphasis on performance and type safety

While this post doesn't cover the specifics of programming in Rust, any code samples are accompanied by a description of what the code does but it's useful to have a basic understanding of programming for understanding exactly what the code is doing

It's also worth noting that this post covers a lot of bit-level processing of image files, to get a basic understanding of binary data works take a look at the previous post on Understanding Binary File Formats

The Exchangeable Image File Format (EXIF)

The Exchangeable Image File Format (EXIF) is based on the Tag Image File Format (TIFF) specification for storing metadata. This data is organised into Image File Directories (IFDs) within an image file

The EXIF section in an image file is structured as follows:

Section	Subsection	Number of Bytes
Header
	EXIF Marker (Exif00)	6 bytes
IFD
	Byte Order (II or MM)	2 bytes
	Magic Number (42)	2 bytes
	Data Start Location	4 bytes
	Data Count	2 bytes
	Data Entries	Data Count x 12 bytes/entry
Additional Data Section

Reading EXIF Data

Reading EXIF data is done by reading the bytes in the file. The following examples will use a JPEG from a Fujifilm X-T200 as a reference, though the same concepts can be applied to understanding data from any file format that stores metadata using the EXIF structure

Under the Hood of an Image File

Below is a snippet of the Hex data for a JPEG file alongside the bytes decoded as text:

Hex Data                                           Decoded Text                       Approximate EXIF Subsections

FF D8 FF E1 57 FE 45 78 69 66 00 00 49 49 2A 00    . . . . W . E x i f . . I I * .    Header, Byte Order, Magic Number
08 00 00 00 0C 00 0F 01 02 00 09 00 00 00 9E 00    . . . . . . . . . . . . . . . .    Data Count, Data Start Location
00 00 10 01 02 00 07 00 00 00 A8 00 00 00 12 01    . . . . . . . . . . . . . . . .    |
03 00 01 00 00 00 01 00 00 00 1A 01 05 00 01 00    . . . . . . . . . . . . . . . .    |
00 00 B0 00 00 00 1B 01 05 00 01 00 00 00 B8 00    . . . . . . . . . . . . . . . .    | Data Entries
00 00 28 01 03 00 01 00 00 00 02 00 00 00 31 01    . . ( . . . . . . . . . . . 1 .    |
02 00 1E 00 00 00 C0 00 00 00 32 01 02 00 14 00    . . . . . . . . . . 2 . . . . .    |_
00 00 DE 00 00 00 13 02 03 00 01 00 00 00 02 00    . . . . . . . . . . . . . . . .    |
00 00 98 82 02 00 05 00 00 00 F2 00 00 00 69 87    . . . . . . . . . . . . . . i .    |
04 00 01 00 00 00 14 01 00 00 A5 C4 07 00 1C 00    . . . . . . . . . . . . . . . .    | Additional Data Section
00 00 F8 00 00 00 EC 29 00 00 46 55 4A 49 46 49    . . . . . . . ) . . F U J I F I    |
4C 4D 00 00 58 2D 54 32 30 30 00 00 48 00 00 00    L M . . X - T 2 0 0 . . H . . .    |
01 00 00 00 48 00 00 00 01 00 00 00 44 69 67 69    . . . . H . . . . . . . D i g i    |

The above snippet shows the hex data, in here the EXIF marker can be found on the first line - 45 78 69 66 00 00 which decodes to EXIF\0\, followed by the byte order 49 49 - II which means that the byte order for the file is Little Endian - which means that the smallest value in a sequence is the first byte - this can be used to decode 2A 00 to 42 if the byte order was Big endian the bytes representing 42 would be flipped

The byte order section is the most important thing to note on this first line as it tells an application how to read the data in the IFD as well as it is what any byte offsets should be calculated relative to

Additionally, the data entries section and the additional data section are broadly marked off, to understand where data is located in this file

Reading a File as Bytes

Rust provides a method for reading a file in the standard library is fs::read which can be used by providing it with a path to the file to read, the code for this looks like so:

let file = fs::read("./sample.jpg").unwrap();

The .unwrap() at the end tells rust to either get the file data or exit the program with an error if it could not read the file

The result of this is a Vec[u8] which means a vector (or list) of bytes - the bytes in the list are represented as integer values between 0 and 255, these are equivalent to the hex values in the snippet above

The file is what is used to read the bytes from and will be the data source for reading the EXIF data

Finding the EXIF Starting Point

To find the starting point of the EXIF data we can scan through the file until we find the Exif\0\0 pattern, a function can be defined for searching for a pattern in a list of bytes:

pub fn get_sequence_range(bytes: &[u8], pattern: &[u8]) -> Option<Range<usize>> {
    let start = bytes
        .windows(pattern.len())
        .position(|window| window == pattern)?;

    let end = start + pattern.len();

    Some(start..end)
}

The function above called get_sequence_range searches the bytes for a pattern. This uses the windows function in rust which creates a bunch of smaller lists and finds the position where the window, which is a section of bytes that's the same length as the search pattern and checks if the value is equal to the pattern

If the pattern can be found, then the function will return a range (basically, a start and end point) that goes from the start of the found pattern until the end of the pattern, which is simply the start value plus the length of the pattern

The Option indicates that the function returns either a value if it finds one (denoted by Some) or will return nothing if no value is found, denoted by None. The above function uses the shorthand for the None case which is done by placing a ? at the end of the check for the pattern - which will cause the function to end early if it could not find the pattern

The above function for finding the starting point can be used by passing it the file's bytes like so:

const EXIF_MARKER: &[u8] = "Exif\0\0".as_bytes();

let exif_range = get_sequence_range(file, EXIF_MARKER)?;

Note that in the above function the EXIF_MARKER is defined as the Exif\0\0 text converted to bytes, this is passed as the search pattern to the get_sequence_range function. This gets the EXIF header location which is used to find the Byte order (Endian) marker

Getting the Byte Order

Once we know the location of the EXIF marker, the byte order values go from the 6th and 7th byte after the start of the marker. Since this is done using a range, this means that the range goes from 6 to 8, since the end value is not included in the range, this can be see defined below:

const ENDIAN_RANGE_FROM_EXIF_MARKER: Range<usize> = 6..8;

The bytes for the endian value can be found relative to the exif_range like so:

let start = exif_range.start + ENDIAN_RANGE_FROM_EXIF_MARKER.start;

let bytes = file.get(start..)?;
let endian_bytes = bytes.get(0..2)?;
let endian = get_endian(endian_bytes)?;

The above makes use of a get_endian function to get the byte order which can be defined as follows:

fn get_endian(endian_bytes: &[u8]) -> Option<Endian> {
    let endian = parsing::full_bytes_string(endian_bytes)?;

    match endian.as_str() {
        "MM" => Some(Endian::Big),
        "II" => Some(Endian::Little),
        _ => None,
    }
}

The above function takes the bytes which start at the endian location and converts them to a string (text) value

These values are then compared using a match. If it is II or MM the function returns Big Endian (Endian::Big) or Little Endian (Endian::Little) respectively. If no matching value is found, then None is returned instead

The code above also finds the bytes, which defines the file's bytes but trims off all the bytes that are before the endian marker - this is important since any data in the IFD needs to be read relative to the this location

Following the Endian bytes are 2 bytes which specify the Magic Number (42) as mentioned above - this can also be checked to verify the byte order of the file but is not covered in this post

Getting the IFD Data Start Location and Count

Immediately after the Magic Number is four bytes that specify where the IFD data starts, in the snippet above, these bytes are 08 00 00 00 which convert to the value of 8, this informs us that the IFD data starts from 8 bytes from the Endian marker

By following the offset value, the number of entries in the IFD can be found at the 8 bytes from the Endian marker, in this case, bytes 0C 00 which convert to the value of 12, which indicates that there are 12 entries in the IFD

The code applying the above logic can be seen below:

let ifd0_offset = get_ifd_offset(&endian, ifd)? as usize;
let ifd0_entry_offset = ifd0_offset + 2;

let ifd0_count = u16::from_offset_endian_bytes(&endian, ifd, ifd0_offset)?;

The function which gets the ifd0_offset does the lookup of bytes from the range 4 to 8, relative to the Endian marker

Reading Entries in the IFD

As a reference example, the bytes for the first entry in the IFD above will be used to understand the data and how it's stored

After the bytes indicate the count, the next section consists of the entries. Each entry consists of 12 bytes and is structured like so:

Tag	Data Format	Component Length	Data
2 bytes	2 Bytes	4 Bytes	4 Bytes
`0F 01`	`02 00`	`09 00 00 00`	`9E 00 00 00`

The Tag is an identifier that specifies what the value of the entry represents
The Data format states how the data should be read
The Component Length states how many bytes the data for the entry consists of
The data can either be the actual data, or a value that gives the offset to the data, depending on the Component Lenght

Tag ID

Reading the Tag is done by parsing the first two bytes of an entry - This converts the value into a 16-bit unsigned integer (a positive integer)

The TagID is read like so:

let tag = u16::from_endian_bytes(endian, entry)?;

The value of the tag is a 16-bit unsigned integer, but it's more commonly represented as Hex value in the tag lookup tables, a lookup table for these can be found at the EXIF Tool Tag Names Doc

The value of the tag above 0F 01 can be converted to hex for the Little Endian notation resulting in 0x010F, the lookup table states that this tag identifies the Make property in the Exif data

Data Format

The data stored in an entry can be of 12 different formats, each of these associated with a format value - the format value can be read by reading from byte index 2 in the entry, like so:

let format_value = u16::from_offset_endian_bytes(endian, entry, 2)?;

The format value is a 16-bit unsigned integer, the same as the Tag, though this is used as an integer value and not hex. Each integer value maps to a specific format type, as seen in the below table:

Format Value	Format	Bytes per Component	Data Type	Description
1	Unsigned Byte	1	u8	8-bit positive integer
2	ASCII String	1	String	Text/String value
3	Unsigned Short	2	u16	16-bit positive integer
4	Unsigned Long	4	u32	32-bit positive integer
5	Unsigned Rational	8	u32, u32	positive fraction - numerator and denominator
6	Signed Byte	1	i8	8-bit integer
7	Undefined	1	[u8]	list of bytes
8	Signed Short	2	i16	16-bit integer
9	Signed Long	4	i32	32-bit integer
10	Signed Rational	8	i32, i32	fraction value - numerator and denominator
11	Single Float	4	f32	floating point/decimal
12	Double Float	8	f64	double precision floating point

The above table is implemented in code by first defining a type that states all the possible format types:

pub enum TagFormat {
    UnsignedByte,
    AsciiString,
    UnsignedShort,
    UnsignedLong,
    UnsignedRational,
    SignedByte,
    Undefined,
    SignedShort,
    SignedLong,
    SignedRational,
    SingleFloat,
    DoubleFloat,
}

Each type of value can also be described in terms of the data type it stores as follows:

pub enum ExifValue<'a> {
    UnsignedByte(u8),
    AsciiString(String),
    UnsignedShort(u16),
    UnsignedLong(u32),
    UnsignedRational(u32, u32),
    SignedByte(i8),
    Undefined(&'a [u8]),
    SignedShort(i16),
    SignedLong(i32),
    SignedRational(i32, i32),
    SingleFloat(f32),
    DoubleFloat(f64),
}

Thereafter, a function to go from the given Format Value to the type of the tag being used:

fn get_tag_format(value: &u16) -> Option<TagFormat> {
    match value {
        1 => Some(TagFormat::UnsignedByte),
        2 => Some(TagFormat::AsciiString),
        3 => Some(TagFormat::UnsignedShort),
        4 => Some(TagFormat::UnsignedLong),
        5 => Some(TagFormat::UnsignedRational),
        6 => Some(TagFormat::SignedByte),
        7 => Some(TagFormat::Undefined),
        8 => Some(TagFormat::SignedShort),
        9 => Some(TagFormat::SignedLong),
        10 => Some(TagFormat::SignedRational),
        11 => Some(TagFormat::SingleFloat),
        12 => Some(TagFormat::DoubleFloat),
        _ => None,
    }
}

As done previously, if the correct value can't be found, the function returns None

So, adding to the above code, the code for reading the tag value is:

let format_value = u16::from_offset_endian_bytes(endian, entry, 2)?;
let format = get_tag_format(&format_value)?;

Component Length

The Component length specifies the number of components for the tag format being read - for most tag formats this will be 1, however, for specific values like AsciiString or Undefined, this may be different in which case it specifies the length of the string or how many bytes are required respectively

The value for the component length can be found by reading the relevant bytes in the entry and converting them to a 32-bit unsigned integer, starting from byte index 4, like so:

let component_length = u32::from_offset_endian_bytes(endian, entry, 4)?;

Data

Once the component length is known, getting the total length of the data to be read is done by multiplying the component length by the bytes per component - since different components need different amounts of data

A function that gets the bytes per component for a given tag format can be seen below:

fn get_bytes_per_component(format: &TagFormat) -> u32 {
    match format {
        TagFormat::UnsignedByte => 1,
        TagFormat::AsciiString => 1,
        TagFormat::UnsignedShort => 2,
        TagFormat::UnsignedLong => 4,
        TagFormat::UnsignedRational => 8,
        TagFormat::SignedByte => 1,
        TagFormat::Undefined => 1,
        TagFormat::SignedShort => 2,
        TagFormat::SignedLong => 4,
        TagFormat::SignedRational => 8,
        TagFormat::SingleFloat => 4,
        TagFormat::DoubleFloat => 8,
    }
}

This is based on the table shown previously on component formats

Next, the total length can be defined as the component length multiplied by the bytes per component which can be seen in the code below:

let component_length = u32::from_offset_endian_bytes(endian, entry, 4)?;

let bytes_per_component = get_bytes_per_component(&format);

let total_length = component_length * bytes_per_component;

The data can be read from the data bytes, which start at index 8 of the entry

let data = entry.get(8..12)?;

The data value must be the raw bytes because depending on the resulting length it needs to be processed differently

If the total_length is less than or equal to 4, the data can be read directly from the data bytes, this can be done using a function that converts a TagFormat and data to the relevant value as defined in the table:

fn parse_tag_value<'a>(
    format: &TagFormat,
    endian: &'a Endian,
    bytes: &'a [u8],
) -> Option<ExifValue<'a>> {
    match format {
        TagFormat::UnsignedByte => parsing::bytes_to_unsigned_byte(endian, bytes),
        TagFormat::AsciiString => parsing::bytes_to_ascii_string(bytes),
        TagFormat::UnsignedShort => parsing::bytes_to_unsigned_short(endian, bytes),
        TagFormat::UnsignedLong => parsing::bytes_to_unsigned_long(endian, bytes),
        TagFormat::UnsignedRational => parsing::bytes_to_unsigned_rational(endian, bytes),
        TagFormat::SignedByte => parsing::bytes_to_signed_byte(endian, bytes),
        TagFormat::Undefined => parsing::bytes_to_undefined(bytes),
        TagFormat::SignedShort => parsing::bytes_to_signed_short(endian, bytes),
        TagFormat::SignedLong => parsing::bytes_to_signed_long(endian, bytes),
        TagFormat::SignedRational => parsing::bytes_to_signed_rational(endian, bytes),
        TagFormat::SingleFloat => parsing::bytes_to_single_float(endian, bytes),
        TagFormat::DoubleFloat => parsing::bytes_to_double_float(endian, bytes),
    }
}

And the data value can be obtained using the function above like so:

let value = parse_tag_value(&format, endian, data)

However, if the total_length is greater than 4, the data value needs to be read as an offset from the IFD which is then converted, again, using the parse_tag_value function above

// the value needs to be checked at the offset and used from there
let offset = u32::from_endian_bytes(endian, data)?;

let start = (offset) as usize;
let end = start + (length) as usize;

let range = start..end;

let value_bytes = bytes.get(range)?;

let result = parse_tag_value(&format, endian, value_bytes)

Putting all the above together, reading the tag above will give:

Tag	Data Format	Component Length	Data
2 bytes	2 Bytes	4 Bytes	4 Bytes
`0F 01`	`02 00`	`09 00 00 00`	`9E 00 00 00`
`0x010f`	ASCII String	9	FUJIFILM\0

Reading Additional Entries

Once a single entry can be read - reading additional entries follows the same pattern. Since the bytes per entry is fixed - always 12 - and the number of entries is known from the IFD count, each entry can be iterated over by going 12 bytes at a time and reading their data individually. A more detailed implementation of this as well as the rest of the code can be found on the exiflib GitHub repo

Conclusion

This post provides a basic outline on reading EXIF data from an image, as well as covers the byte structure for reading EXIF entries from an image file. There's a lot more to reading EXIF data from images, but at a high level the parsing covered here should form a basic grounding in how reading this data works

For further reference and inspiration take a look a the reference list at the end of this post as well as the exiflib GitHub repo mentioned previously

References

Implementation details and guidance for reading metadata from:

Some reference implementations and libraries:

Education App Design Ideas

Tue, 27 Sep 2022 00:00:00 GMT

Education App Design Ideas

General App Overview

Content Creation and Editing Experience

Alternate Themes

Type Narrowing in Typescript

Tue, 31 May 2022 00:00:00 GMT

Type Narrowing in Typescript

Type Narrowing allows us create conditions under which an object of one type can be used as if it is of another type. We usually use this in conjunction with union types to allow us to specify different handling of the types based on the resulting value

Using `typeof`

We can use the typeof keyword in javascript to find out whether what type an object is. This is useful if we have an object that can take on different structures, for example

type Data = string[]
type GetData = Data | (() => Data)

In the above example, we have a type called GetData which can be either some data or a function to get data. Using this, we can can create a function which fetches data like so:

const fetchData = (getData: GetData): Data => {
  if (typeof getData === 'function') {
    return getData()
  }

  return getData
}

Using `in`

Javascript also has the in operator which can be used to infer types by us checking a property of an object

type SimpleData = {
  name: string
}

type ComplexData = {
  name: {
    first: string
    last: string
  }
  isComplex: true
}

type AnyData = SimpleData | ComplexData

We can then use the in operator to check the existence of a property of an object by using it along with a key that we expect to be in one object but not another

const getComplexName = (data: AnyData): string => {
  // isComplex is the name of the key that we expect in `ComplexData` but not `SimpleData`
  if ('isComplex' in data) {
    return [data.name.first, data.name.last].join(' ')
  }

  return data.name
}

Using `is`

We can use the typescript is keyword to specify that the return of a boolean means that a variable satisfies a specific condition

For example, we can create a function that basically does what the in operator in the above function does:

const isComplex = (data: AnyData): data is ComplexData => {
  return (data as ComplexData).isComplex
}

This can be used in place of the in check in the above example like so:

const getComplexName2 = (data: AnyData): string => {
  // isComplex is the name of the key that we expect in `ComplexData` but not `SimpleData`
  if (isComplex(data)) {
    return [data.name.first, data.name.last].join(' ')
  }

  return data.name
}

References

Generic Object Property Formatter and Validator using Typescript

Tue, 09 May 2023 00:00:00 GMT

Generic Object Property Formatter and Validator using Typescript

At times there's a need for transforming specific object properties from an input format to some specific output, this can be done using the following code in some generic means

The Concept of a Schema

The concept of a schema to be used for formatting some data can be defined as follows

export type Primitive = string | number | boolean | Symbol | Date

/** * Define a formatter that can take in a generic object type and format each entry of that object */
export type FormatterSchema<T> = {
  [K in keyof T]?: T[K] extends Primitive
    ? // if the value is primitive then we can just transform it using a simple function
      (val: T[K]) => T[K]
    : // if it's an object then it should either be a formatter schema or a transformer function
      FormatterSchema<T[K]> | ((val: T[K]) => T[K])
}

An Interpreter

The implementation of an interpreter that satisfies the above can done as follows:

type Formatter<T> = (val: T) => T
const isFormatterFn = <T>(
  formatter: FormatterSchema<T>[keyof T]
): formatter is Formatter<T[keyof T]> => typeof formatter === 'function'

const isFormatterObj = <T>(
  formatter: FormatterSchema<T>[keyof T]
): formatter is Exclude<Formatter<T[keyof T]>, Function> =>
  typeof formatter === 'object'

export const interpret =
  <T>(schema: FormatterSchema<T>): Formatter<T> =>
  (val) => {
    const keys = Object.keys(schema) as Array<keyof T>
    return keys.reduce<T>((prev, key) => {
      const keySchema = schema[key]
      const keyVal = val[key]

      const isSchemaFn = isFormatterFn(keySchema)
      if (isSchemaFn) {
        return {
          ...prev,
          [key]: keySchema(keyVal),
        }
      }

      const isSchemaObj = isFormatterObj(keySchema)
      if (isSchemaObj) {
        return {
          ...prev,
          [key]: interpret(keySchema)(keyVal),
        }
      }

      return prev
    }, val)
  }

Transformers

We can define some general transformers, for example:

const trunc = Math.trunc

const round = Math.round

const length = (len: number) => (val: string) => val.slice(0, len)

const trim = (val: string) => val.trim()

const constant =
  <T>(val: T) =>
  () =>
    val

const optional =
  <T>(fn: (val: T) => T) =>
  (val?: T) =>
    val === undefined ? undefined : fn(val)

export const formatter = {
  trunc,
  round,
  length,
  trim,
  constant,
  optional,
}

Usage

For the sake of example we can define a data type to use

type User = {
  name: string
  location: {
    address: string
    city: string
    gps?: [number, number]
  }
}

const user: User = {
  name: 'Bob Smithysmithson',
  location: {
    city: 'Somewhere secret',
    address: '123 Secret street',
    gps: [123, 456],
  },
}

Using as a Transformer

We can use the above by defining the transformer and using it to transform the input data according to the defined schema

const format = interpret<User>({
  name: formatter.length(10),
  location: {
    city: (val) => (['Home', 'Away'].includes(val) ? val : 'Other'),
    gps: (val) => (isValidGPS(val) ? val : undefined),
  },
})

const result = format(user)

console.log(result)
// {
//   name: 'Bob Smithy',
//   location: {
//     city: 'Other',
//     adddtess: '123 Secret street',
//     gps: undefined,
//   }
// }

Using as a Validator

Using the same principal as above we can use it for validaing data. Validation can be done by throwing in the transformer function

const validate = interpret<User>({
  name: (val) => {
    if (val.length > 5) {
      throw 'Length too long'
    }

    return val
  },
})

// throws with Length too long
validate({
  name: 'Jack Smith',
})

Exploration

It may be worth taking the concept and expanding it into a more fully fledged library with a way to handle validations more simply or to provide more builtin transformers and validators but for the moment it can be stated that there are enough javascript libraries for validation and the formatter alone is likely of limited value but may be worth exploring further if gaps in the existing technology are found

Regardless of the utility of the particular type above I think the concept and types give some insight into how one can build out a library as the ones discussed above

Working Example

A working playground with the code from here can be found at The Typescript Playground

Interacting with Kafka with Kotlin Coroutines

Sat, 11 Nov 2023 00:00:00 GMT

Interacting with Kafka with Kotlin Coroutines

Overview

The purpose of this post is to illustrate a method of interacting with Kafka using Kotlin in a functional programming style while using Kotlin coroutines for a multi-threading. We will be interacting with the Kafka Client for Java and will be building a small library on top of this for the purpose of simplifying communication and handling tasks like JSON Serialization

If you would like to view the completed source code, you can take a look at the kotlin-kafka GitHub repository

Kafka

According to then Kafka Website:

"Apache Kafka is an open-source distributed event streaming platform:

Generally we can think of Kafka as a platform that enables us to connect data producers to data.

Kafka is an event platform that provides us with a few core functions:

Publishing and subscribing event data
Processing of events in real-time or retrospectively
Storage of event streams

From a more detailed perspective, Kafka internally handles storage of event streams, but we are given control over the means of data production, consumption, and processing via the Kafka API, namely:

The Producer API for production
The Consumer API for subscription
The Streams API for processing stream data

Kotlin

Kotlin is a statically typed programming language built on the Java Virtual Machine that provides interop with Java code

The Code

Config

To get some admin stuff out of the way, before you can really do any of this you will to have a .env file in the project that you can load which contains some application configuration, for the purpose of our application we require the following config in this file - below is some example content

.env

BOOTSTRAP_SERVERS=my-server-url:9092
SASL_JAAS_CONFIG=org.apache.kafka.common.security.scram.ScramLoginModule required username="someUsername" password="somePassword";

Additionally, we have some non-sensitive config in our application.properties file in our application resources folder which contains the following:

resources/application.properties

sasl.mechanism=SCRAM-SHA-256
security.protocol=SASL_SSL
key.serializer=org.apache.kafka.common.serialization.StringSerializer
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
auto.offset.reset=earliest
group.id=$GROUP_NAME
application.id=example-app

Next, we need to load this in our application to create a Properties object along with all the other application config we require. We can create the Properties object using the application.properties and .env files as follows:

App.kt

package example

import io.github.cdimascio.dotenv.Dotenv
import java.io.FileInputStream
import java.util.*

fun loadProperties(): Properties {
    val props = Properties()
    val resource = ClassLoader.getSystemResource("application.properties")
    println("File  path: ${resource.path}")
    FileInputStream(resource.path).use { stream ->
        props.load(stream)
    }

    val dotenv = Dotenv.load()
    props["bootstrap.servers"] = dotenv["BOOTSTRAP_SERVERS"]
    props["sasl.jaas.config"] = dotenv["SASL_JAAS_CONFIG"]

    return props
}

The above example uses the io.github.cdimascio:dotenv-java:3.0.0 package for loading the environment variables and some builtin Java utilities for loading the application properties file

Next, for the purpose of using it with our library we will create a Config class that wraps the properties file we defined so that we can use this a little more elegantly in our consumers. Realistically we probably should do some validation on the resulting Properties that we load in but we'll just keep it simple and define Config as a class that contains the properties as a property:

Config.kt

package za.co.nabeelvalley.kafka

import java.util.Properties

open class Config(internal val properties: Properties) {}

Working with JSON Data

An important part of what we want our client to handle is the JSON serialization and deserialization when sending data to Kafka. Sending JSON data is not a requirement of Kafka as a platform, but it's the usecase that we're building our library around and so is something we need to consider

Serialization

Serialization in this context refers to the process of converting our Kotlin classes into a string and back to a Kotlin class. For this discussion we will refer to a class that is able to do this bidirectional conversion as a Serializer.

We can define generic representation of a serializer as a class that contains a method callsed serialize that takes in data of type T and returns a string, and contains a method called deserialize that takes in a string and returns an object of type T

Not that at this point we're not considering that the serializer needs to return JSON. In our context a JSON serializer is just a specific implementation of the serialization concept that we have defined

An interface that describes the Serializer we mentioned above can be seen as follows:

Serializer.kt

package za.co.nabeelvalley.kafka

interface ISerializer<T : Any> {
    fun serialize(data: T): String
    fun deserialize(data: String): T
}

JSON Serialization

Given the definition of a serializer we can define a JSON serializer that uses the kotlinx.serialization library and implements our ISerializer as follows:

JsonSerializer.kt

package za.co.nabeelvalley.kafka

import kotlinx.serialization.KSerializer
import kotlinx.serialization.json.Json
import kotlinx.serialization.serializer
import kotlin.reflect.KClass

class JsonSerializer<T : Any>(type: KClass<T>) : ISerializer<T> {
    private val serializer: KSerializer<T> = serializer(type.java) as KSerializer<T>

    override fun serialize(data: T): String = Json.encodeToString(serializer, data)

    override fun deserialize(data: String): T = Json.decodeFromString(serializer, data)
}

The above code is a little funky since we're using reflection on the actual class of the input data to define our serializer, other than we're just using the kotlinx serializer to handle the data transformation. The thing that matters in this context is that we are able abstract the reflection aspect of the serializer, this will help make the final interface we provide to the user for working with Kafka simpler

Serde Serializer

Now that we have defined a simple representation of a serializer that provides some interop with the Kotlin data types, we need to implement the other side of this which is a SerdeSerializer which is what the Kafka Clients need to work with. The requirements of this serializer are a little different to the one we defined above. This serializer needs to:

Have a separate Serializer and Deserializer interfaces that need to be implemented
Return a ByteArray instead of String

We can define these serializers such that they can be constructed from and ISerializer interface that we defined previously. This will make it possible for consumers of our library to swap our their serialization strategy to enable other usecases than the simple JSON communication we are considering

As mentioned above, we need to implement a separate Serializer and Deserializer respecively as:

SerdeSerializer.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.common.serialization.Serializer

class SerdeSerializer<T : Any>(private val serializer: ISerializer<T>) : Serializer<T> {
    override fun serialize(topic: String?, data: T): ByteArray {
        val result = serializer.serialize(data)
        return result.toByteArray()
    }
}

And

SerdeDeserializer.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.common.serialization.Deserializer

class SerdeDeserializer<T : Any>(private val serializer: ISerializer<T>) : Deserializer<T?> {
    override fun deserialize(topic: String?, data: ByteArray?): T? {
        try {
            val string = String(data!!)
            return serializer.deserialize(string)
        } catch (error: Error) {
            println("Error Deserializing Data: $error")
            return null
        }
    }
}

Our implementation is a little basic and will just ignore any data that we can't serialize, however depending on our usecase we may need to handle this differently

Lastly, we define the actual Serde Serializer implementation using the above implementations:

Serializer.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.common.serialization.Serde

class Serializer<T : Any>(private val serializer: ISerializer<T>) : Serde<T> {
    override fun serializer() = SerdeSerializer<T>(serializer)

    override fun deserializer() = SerdeDeserializer<T>(serializer)
}

As far as serialization and deserialization goes, this should be everything we need for working with JSON data

Producing Data

Producing data is a method by which a client sends data to a Kafka topic. We can define this as a type as follows:

Producer.kt

typealias Send<T> = (topic: String, message: T) -> Unit

Now, to provide a functional library interface we will want to provider application code a space in which they will be able to work with the producer that we populate without needing to create a new producer for each message we want to send

We'll codify this intent as a type as follows:

Producer.kt

typealias Produce<T> = suspend (send: Send<T>) -> Unit

Note that we define this as a suspend function that will enable users to send messages from within a coroutine context

Next, we define the type of our producer as method with a way to create a producer instance for users who may want to manage the lifecycle of the KafkaProducer on their own. This however also means they lose access to the automatic serialization and deserialization that we will provide via our producer method

This interface is defined as follows:

Producer.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.clients.producer.KafkaProducer
import java.util.*

interface IProducer<T> {
    /**
     * Returns the raw producer used by Kafka
     */
    fun createProducer(): KafkaProducer<String, String>
    fun produce(callback: Produce<T>)
}

For the purpose of our implementation we can define some functions ourside of our class that will provde the implementation we require

For the createProducer function, we simply provide a wrapper around the KafkaProducer provided to us by the Java Kafka Client Library:

Producer.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.clients.producer.KafkaProducer
import org.apache.kafka.clients.producer.ProducerRecord
import java.util.*
import kotlinx.coroutines.runBlocking

fun createProducer(properties: Properties) =
    KafkaProducer<String, String>(properties)

For the sake of consistency, we will do the same for the concept of a ProducerRecord which will be used by the produce function:

Producer.kt

fun createRecord(topic: String, message: String) =
    ProducerRecord<String, String>(topic, message)

Next, the produce function can be defined. The role of this function is to handle serialization of data and provide a means for a user to send data to a topic

The producer will take a callback which is the context in which any usage of the send function should be used before the producer is disposed:

Producer.kt

fun <T : Any> produce(properties: Properties, serializer: ISerializer<T>, callback: Produce<T>) {
    createProducer(properties).use { producer ->
        val send = fun(topic: String, message: T) {
            val payload = serializer.serialize(message)
            val record = createRecord(topic, payload)
            producer.send(record)
        }

        runBlocking {
            callback(send)
        }
    }
}

We have also added the properties and serializer values as an input to the producer as this is needed by Kafka, lastly, we will define our actual Producer implementation which builds on the functions we defined above

Note that our Producer class implements IProducer and extends Config, this is because we use the Config class as the source of truth of the configuration to be used for our Kafka instance and we want to able to access this config

Producer.kt

class Producer<T : Any>(
    properties: Properties,
    private val serializer: ISerializer<T>
) : Config(properties),
    IProducer<T> {
    override fun createProducer() = createProducer(properties)

    override fun produce(callback: Produce<T>) = produce<T>(properties, serializer, callback)
}

At this point we have a complete implementation of a producer

Using the Producer

In our application code we can instantiate and use the producer as follows:

Firstly, we need to define the type of data we are goind to send with the @Serializable annotation

App.kt

import kotlinx.coroutines.*
import kotlinx.serialization.Serializable
import za.co.nabeelvalley.kafka.*
import java.util.*


@Serializable
data class ProducerData(val message: String, val key: Int)

Next, we can define a function for producing data, this will require the properties we loaded previously:

App.kt

fun instantiateAndProduce(properties: Properties): Unit {
    val serializer = JsonSerializer(ProducerData::class)
    val producer = Producer(properties, serializer)

    runBlocking {
        producer.produce { send ->
            val data = ProducerData("Hello world", 1)
            send("my-topic", data)
        }
    }
}

We use runBlocking since our producer needs a coroutine scope in which to send data. Sending data us used within the produce method in which we create some data and call the send method provide by the produce function

An interesting to note is that we are passing the class of our data to the serializer to create an instance - this is the usage of the funky reflection thing we saw previously

Consuming Data

Our code for consuming data will follow a similar pattern to what we use to consume the data in the previous section

For consuming data, Kafka relies on the concept of polling for records from the part of the consumer, for our client, we will expose using the following type which defines a poll as a method that takes nothing and returns a list of data of type T

Consumer.kt

typealias Poll<T> = () -> List<T>

Next, we can define the type that defines how we want our data to be consumed. For our sake, this is a suspend function that will receive a poll method that it can call to get data

Consumer.kt

typealias Consume<T> = suspend (poll: Poll<T>) -> Unit

Next, as before, we can define an interface for a Consumer in which we have a method to create a KafkaConsumer and a method for actually consuming the data. In the case of consuming we need a list of topics to read from as well as the polling frequency duration.

Consumer.kt

package za.co.nabeelvalley.kafka

import kotlinx.coroutines.runBlocking
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.kafka.clients.consumer.KafkaConsumer
import java.time.Duration
import java.util.*

interface IConsumer<T> {
    /**
     * Returns the raw consumer used by Kafka
     */
    fun createConsumer(): KafkaConsumer<String, String>
    fun consume(topics: List<String>, duration: Long, callback: Consume<T>)
}

Next, we can define our createConsumer method quite simply as:

Consumer.kt

fun createConsumer(properties: Properties) =
    KafkaConsumer<String, String>(properties)

And we can define our consume method such that it takes in the properties and serializer as with the producer, but will also take som patterns to be used for subscribing to and the duration above, and finally the callback Consume function:

Consumer.kt

fun <T : Any> consume(
   properties: Properties,
   serializer: ISerializer<T>,
   patterns: List<String>,
   duration: Long,
   callback: Consume<T>
) {
   createConsumer(properties).use { consumer ->
      consumer.subscribe(patterns)
      val poll = fun(): List<T> {
         val records = consumer.poll(Duration.ofMillis(duration))
         val data = records.toList()
               .map(ConsumerRecord<String, String>::value)
               .map(serializer::deserialize)

         return data
      }

      runBlocking {
         callback(poll)
      }
   }
}

The consume function is very similar to the produce function we defined previously, however now instead of being provided a function to send data we now have a function that will return that data

Lastly, we can finish off the definition of our Consumer using what we have above:

Consumer.kt

class Consumer<T : Any>(
    properties: Properties,
    private val serializer: ISerializer<T>
) : Config(properties), IConsumer<T> {
    override fun createConsumer() = createConsumer(properties)

    override fun consume(topics: List<String>, duration: Long, callback: Consume<T>) =
        consume(properties, serializer, topics, duration, callback)
}

Using the Consumer

Using the Consumer follows a very similar pattern to the producer, however we need to create a loop that will poll for data and handle as necessary when data is received:

App.kt

@Serializable
data class ConsumerData(val message: String, val key: Int)

fun instantiateAndConsume(properties: Properties): Unit {
   val serializer = JsonSerializer(ConsumerData::class)
   val consumer = Consumer(properties, serializer)

   runBlocking {
       consumer.consume(listOf("my-topic"), 1000) { poll ->
         while (true) {
            val messages = poll()
            println("Received ${messages.size} messages")
            messages.forEach(fun(message) {
               println("Received: $message")
            })
         }
      }
   }
}

In the above, we use a while(true) loop to re-poll continuously but this can freely change on the implementation, similar to with the producer code

Stream Processing

In Kafka, we can think of a stream process as a combination of a consumer and producer such that data comes in from a topic and is sent to a different topic

The thing that makes streams interesting is the builder API that the Kafka Java Library provides to us for defining the operations to be done on the stream data. For our implementation we'll be referring to this as a TransformProcessor, this processor needs to take in some data of type TConsume and return data of type TProduce, however, since we want to provide users complete flexibility in working with this data, we will instead more generally allow a user to convert a stream between the predefined data types, using the underlying library this is called a KStream

From a type perspective, we can define a TransformProcessor as follows:

SerializedStream.kt

typealias TransformProcessor<TConsume, TProduce> = (stream: KStream<String, TConsume>) -> KStream<String, TProduce>

Now, we're going to be starting this implementation from what we want, assuming that it is possible for us to in some way define a KStream that is instantiated to work with our connection and the respective TConsume and TProduce data.

We will also be using a type called Produced which is what the Kafka Client uses to represent the data that the stream will return since this is what we need in order to send data to a processor

Our implementation will be called a SerializedStream and this looks like the following:

SerializedStream.kt

package za.co.nabeelvalley.kafka

import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.Job
import kotlinx.coroutines.launch
import org.apache.kafka.streams.KafkaStreams
import org.apache.kafka.streams.StreamsBuilder
import org.apache.kafka.streams.kstream.KStream
import org.apache.kafka.streams.kstream.Produced
import java.util.*

typealias TransformProcessor<TConsume, TProduce> = (stream: KStream<String, TConsume>) -> KStream<String, TProduce>
typealias Close = () -> Unit
typealias Process = suspend (close: Close) -> Unit

class SerializedStream<TConsume : Any, TProduce : Any>(
    private val properties: Properties,
    private val builder: StreamsBuilder,
    private val producer: Produced<String, TProduce>,
    private val stream: KStream<String, TConsume>
) {
   fun startStreaming(
      topic: String,
      processor: KStream<String, TProduce>,
      process: Process
   ): Job {
      processor.to(topic, producer)

      val streams = KafkaStreams(
         builder.build(),
         properties
      )

      val scope = CoroutineScope(Dispatchers.IO)
      return scope.launch {
         streams.start()
         process()
         streams.close()
      }
   }

   fun getProcessor(
       processor: TransformProcessor<TConsume, TProduce>
   ): KStream<String, TProduce> = processor(stream)
}

In the above implementation we have an input to our startStreaming function called process, the process function is a callback that needs to call close once it is done running. When the process function returns the processing will stop, the scope of this function also defines lifecycle of the stream processor and is used for that purpose

So we have defined the processing methodology using a KStream but have not provided a way to create a KStream. Since the stream can be defined in many different ways, we can define this using a builder class called StreamBuilder. This class will be instantiated with the Kafka connection properties and input/output serializers, thereafter it can produce methods for instantiating the SerializedStream instance that we can use for data processing

For the sake of our example we will provide a method called fromTopic which returns a SerializedStream that is configured to work on a single topic, and a fromTopics method which will return a SerializedStream that listens to multiple topics:

StreamBuilder.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.common.serialization.Serdes
import org.apache.kafka.streams.StreamsBuilder
import org.apache.kafka.streams.kstream.Consumed
import org.apache.kafka.streams.kstream.Produced
import java.util.*

interface IStreamBuilder<TConsume : Any, TProduce : Any> {
    fun fromTopic(topic: String): SerializedStream<TConsume, TProduce>
    fun fromTopics(topics: List<String>): SerializedStream<TConsume, TProduce>
}

An implementation of this interface is as follows:

StreamBuilder.kt

class StreamBuilder<TConsume : Any, TProduce : Any>(
    properties: Properties,
    consumeSerializer: ISerializer<TConsume>,
    producerSerializer: ISerializer<TProduce>,
) : Config(properties), IStreamBuilder<TConsume, TProduce> {
    private val inputSerde = Serializer<TConsume>(consumeSerializer)
    private val consumed = Consumed.with(Serdes.String(), inputSerde)

    private val outputSerde = Serializer<TProduce>(producerSerializer)
    private val produced = Produced.with(Serdes.String(), outputSerde)

    override fun fromTopic(topic: String): SerializedStream<TConsume, TProduce> {
        val builder = StreamsBuilder()
        val stream = builder.stream(mutableListOf(topic), consumed)

        return SerializedStream(properties, builder, produced, stream)
    }

    override fun fromTopics(topics: List<String>): SerializedStream<TConsume, TProduce> {
        val builder = StreamsBuilder()
        val stream = builder.stream(topics.toMutableList(), consumed)

        return SerializedStream(properties, builder, produced, stream)
    }
}

The above class makes use of the produced and consumed properties which are what Kafka will use for serializing and deserializing data in the stream

And that's about it as far as our implementation for streaming goes

Using the Stream Processor

We can use the stream processor code:

App.kt

fun initializeAndProcess(properties: Properties): Job {
    val producedSerializer = JsonSerializer(ProducerData::class)
    val consumedSerializer = JsonSerializer(ConsumerData::class)
    val streamBuilder = StreamBuilder(properties, consumedSerializer, producedSerializer)
    val stream = streamBuilder.fromTopic("input-topic")

    val processor = stream.getProcessor { kStream ->
        kStream.mapValues { key, value ->
            ProducerData("Message processed: $key", value.key)
        }
    }

    val scope = CoroutineScope(Dispatchers.IO)
    return scope.launch {
        stream.startStreaming("output-topic", processor) { close ->
            coroutineScope {
                println("Processor starting")
                // Non-blocking loop as long as the coroutine is active
                while (isActive) {
                    delay(10_000)
                }

                // close when no longer active
                close()
                println("Processor closed")
            }
        }
    }
}

Most of this is just the normal construction that you will have for any instance of the stream client, what is interesting is the part where we define the processor:

App.kt

val processor = stream.getProcessor { kStream ->
    kStream.mapValues { key, value ->
        ProducerData("Message processed: $key", value.key)
    }
}

In the above example we are simply mapping a single record using mapValues, this is very similar to the Collection methods available in Kotlin but is instead used to define how data will be transformed in the stream

The processor we define is what will be executed on records or groups of records depending on how we want to handle the resulting data

Conclusion

In this post we've covered the basic implementation of how we can interact with Kafka using the Kotlin programming language and built a small library that takes us through the basic use cases of Serializing, Producing, Consuming, and Processing stream data

References

Nvim Error when using Plugins

Tue, 13 Jun 2023 00:00:00 GMT

Nvim Error when using Plugins

When trying to use an NVim plugin you may run into the following error message:

Error executing lua ...path/to/some/vim/file.lua

This can happen due to a plugin update that was run or another plugin being installed that caused the working tree of the plugin to be out of date or in an invalid state

To fix this, you may want to try updating and cleaning your plugins. In my case, I am using VimPlug so the commands are:

:PlugUpdate
:PlugClean

Thereafter, quitting and reopening vim seems to usually fix the issue, if not then you may need to investigate the plugin itself

JPA Queries without the Magic

Fri, 20 Oct 2023 00:00:00 GMT

JPA Queries without the Magic

If you're just looking for the solution you can just skip ahead

The go-to way to build web applications with Java or Kotlin is using Spring. Spring is a pretty decent web framework that provides a lot of utilities for handling common web application related tasks

However I find that Spring does a fair amount of compiler magic to make it work the way it does - of particular concern to this post is how Spring generates implementations for database queries using Query kewords as part of your function names which spring will resolve to a query on the underlying database

Below is a short example of how we would use Spring to query our database to illustrate the problem:

The Example

Let us consider an application in which we have some entity in our system that we would like to store in a database and be able to search over

In order to do this, we need to define a couple of things things:

An Entity which represents that data structure for our User

In Spring we can define entities, like a User as a class with some annotations

import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;

@Entity
@Table(name = "User")
public class User {

    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private long id;

    @Column(unique = true)
    private String email;
}

A Repository which is the class through which we will interact with the database for the specific entity

import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;

import java.util.List;
import java.util.Optional;

@Repository
public interface UserRepository extends JpaRepository<User, Long> { }

For the repository all we really need is to use the @Repository annotation which will register the repository with the Spring dependency injection, and we need to extend JpaRepository which provides some base methods for our repository such as findAll, save, or delete among others

Up until this point, we haven't seen anything too weird and the code thus far is fairly easy to reason about - however, this becomes interesting when we want to add a method like findByEmail which we define in the interface as follows:

@Repository
public interface UserRepository extends JpaRepository<User, Long> { 
    List<User> findByEmail(String email);
}

That's it, we don't implement this anywhere - Spring will generate this implementation for us - weird - but okay. We can use this function anywhere we have a repsitory instance and it will behave as expected

Growing Pains

But now, what if we want the findByEmail function to be case incensitive? Well seemingly we can just name this however so let's try:

@Repository
public interface UserRepository extends JpaRepository<User, Long> { 
    List<User> findByEmailCaseInsensitive(String email);
}

You can try using that, but it won't work - turns out the magic is documented at least, and Spring tells us that we actually need to use IgnoreCase in the name - this makes use of the Query Keywords I mentioned above

So sure, we can change this:

@Repository
public interface UserRepository extends JpaRepository<User, Long> { 
    List<User> findByEmailIgnoreCase(String email);
}

And also go and change everwhere we're using that method, since it's name now dictates its implementation we will need to keep in mind that if we ever want to change the implementation we will need to update all our clients

But wait - that's very UN-Java like isn't it? This seems to break the entire purpose of using functions - if my function's behaviour is directly dependant on it's name then how is this any different to just repeating the implementation everywhere need this? We're typing the same code either way

Now, I'm not all complaints - using a half competent IDE we should be able to do this refactor fairly painlessly and move on with our day - this isn't the main issue yet

Where we start running into problems is when our queries need to get more complex, for example when we decide that we want the search to extend to something like a secondary email filed, for example if we add a new field to our entity, like companyEmail:

@Entity
@Table(name = "User")
public class User {
    // ... existing code

    // new field
    @Column()
    private String companyEmail;
}

The Limit Does Exist

And we now need to update our email function to:

@Repository
public interface UserRepository extends JpaRepository<User, Long> { 
    List<User> findByEmailIgnoreCaseOrCompanyEmailIgnoreCase(String email);
}

That's a bit rough right? There's really no reason our method name should be that long - oh, and it won't work either - even though this follows the JPA naming convention it's just a little too complicated for the code generator to get right

Okay, so now we're stuck right? No - Why would I go on this rant if I don't have a solution in my head

Well, time to share it I guess

The Solution(s)

Query Annotations

Spring gives us two escape hatches for handling the problem we just ran into - we can use the @Query annotation which allows us to define a custom JQL query (Not SQL) that is defined inline and will do the appropriate value substitutions as needed:

@Repository
public interface UserRepository extends JpaRepository<User, Long> { 
    @Query("SELECT u FROM User u WHERE LOWER(u.email) LIKE LOWER(:query) OR LOWER(u.comanyEmail) LIKE LOWER(:query)")
    List<User> findByEmail(String email);
}

Now, if this were the only option, I could live with it, it takes away some of the magic and gives me a decent amount of control over what I'm doing

... but ... how can i say ... It's ugly

Even if we assume that we don't somehow have any errors in the above string, it's just a little odd - like other ORMs have lovely fluent interfaces like LINQ in C# or knex or prisma in Javascript/TypeScript that understand SQL and fit naturally into our programming languge

Specifications

Specifications provide us with a way to define our query within our programming language, and I think that's nice, so here's how they work

In order to use specificiations we need to do a few things:

Define the specification method

It doesn't matter too much where we define the specification, but for our example we'll put it in a static class so we can use it wherever:

import org.springframework.data.jpa.domain.Specification;

public class UserSpecification {

    /**
     * Search for an User using a free text search on the email and name fields
     *
     * @param search text to be searched
     * @return a specification that can be used with the `repository.findAll()` on a `JpaSpecificationExecutor`
     */
    public static Specification<User> searchForUserByEmail(String search) {
        var likeSearch = search.toLowerCase();

        return (root, query, criteriaBuilder) -> {
            return criteriaBuilder.or(
                    criteriaBuilder.like(criteriaBuilder.lower(root.get("email")), likeSearch),
                    criteriaBuilder.like(criteriaBuilder.lower(root.get("companyEmail")), likeSearch)
            );
        };
    }
}

The structure of the specification is a little verbose but you could refactor this to be a bit more reusable if you wanted

Extend the JpaSpeficationExecutor on our repository

We can delete all the methods in the repository and add the JpaSpecificationExecutor<User> to the list of things we're extending

// ... existing imports

import org.springframework.data.jpa.repository.JpaSpecificationExecutor;

@Repository
public interface UserRepository extends JpaRepository<User, Long>, JpaSpecificationExecutor<User> { }

The above definition will provide us with two nice method on the repository, namely findAll and findOne which both take a Speficication

Use the speficication with our repository

Now that we have everything defined, we can use this in some part of our application as such:

var spec = UserSpecification.searchForUserByEmail("bob@email.com");
var result = userRepository.findOne(spec);

// or for findAll
var results = userRepository.findAll(spec)

And that's it, In my opinion the Specification solution is a bit easier to manage within the context of the greater codebase without having to worry about too much compile time magic and possible typos in the JQL query

Conclusion

I think either of the the provided solutions are fair and give us with a good way to manage more complex queries more flexibly in a way that isolates our implementation from where we intend to use the code

Oh, and since you made it this far - I'm SO sorry you're writing Java and I wish your sanity all the best

Postman Flows

Thu, 23 Mar 2023 00:00:00 GMT

Postman Flows

Postman is a tool used for making HTTP requests during development that enables easy testing and experimentation with HTTP/REST APIs

Recently Postman released something called Flows which is a visual editor for building API flows between different applications and stitching them together into some kind of output view

Flows can be done to DO things but can also be used to SHOW information based on the data that passes through them

Below is an example of a Postman Flow that reads the posts from my website and lists/counts them when run from postman

Another example of a Postman Flow is of their Stock dashboard example

References

For more information on how to build an use a flow you can take a look at the Postman Flows Overview

Postman flows seems targeted at more technical users. In addition to this I think IFTTT is worth taking a look at if you're interested in doing some small flow-style automations

For a bit of a more self-hosted alernative, you can also take a look at Node-RED

Let's talk about Feeds

Tue, 24 Jan 2023 00:00:00 GMT

Let's talk about Feeds

Order Up

Life's busy and disastrous and trying to pop in and talk to myself for a few hours is unattainable most of the time

Recently I've gotten really into RSS. RSS is short for Really Simple Syndication and it's basically just a format/standard that enables websites to share their content in a way that's easy for other websites or applications to consume

Starter

An RSS Feed is really just an XML file that contains information on a website's content, and depending on the nature of the website this is usually something like a blog or podcast feed, but I've also seen RSS used for things like project statuses or website uptime information

Main Course

Here's a sample RSS Feed that I found on RSS Tools that I modified a bit to represent something you might see when working with websites or blog content:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
  <channel>
    <title>RSS Example</title>
    <description>This is an example of an RSS feed</description>
    <link>http://www.domain.com/link.htm</link>
    <lastBuildDate>Mon, 28 Aug 2006 11:12:55 -0400</lastBuildDate>
    <pubDate>Tue, 29 Aug 2006 09:00:00 -0400</pubDate>
    <item>
      <title>Item Example</title>
      <description>This is an example of an Item</description>
      <link>http://www.domain.com/link.htm</link>
      <guid isPermaLink="false">1102345</guid>
      <pubDate>Tue, 29 Aug 2006 09:00:00 -0400</pubDate>
      <content:encoded>This is some content</content:encoded>
    </item>
  </channel>
</rss>

In the above example we can see a channel which defines a source for content

Next up we've got the title, description, link, lastBuildDate, and pubDate which are the metadata for the overall feed and are required for readers that may be accessing the feed

Now, all of that is really cool, but we're not into the juicy bits yet - diving into the content itself we see in the item section.

A feed can define multiple item tags - each of these would have some metadata as well as the actual content associated with the post

In the example, each item has the following:

title
description
link
guid
content:encoded

The above fields are pretty so-so, but the part I find intriguing is the guid which is any unique identifier for an item in a feed, this doesn't have a specific format but just needs to be unique to the feed.

Next, is the content:encoded section which is what the feed uses to share it's content - this can either be plain text or HTML, and it's up to the client to figure it out - based on this I suppose you could really share any content or data which is a pretty cool concept

Dessert

Overall, RSS is pretty simple as a format and has some issues and limitation - versions prior to RSS 2.0 can also be a bit difficult to work with as it's not totally standardised

Another issue I've encountered when working with RSS is the size of the file itself. Some feeds post frequently and may have very long posts. This can lead to very long posts

I've noticed that some blogs with large RSS feeds choose to only store their latest items on the RSS feed, this means that consumer interested in reading older items need to have some cache of their own which was made when the items were initially posted - you could use the guid for this or some kind of diffing method but pretty much anything outside of the XML file delivered is the job of a consumer to work with

The Bill

I've seen a lot of revived interest in the RSS format but there are also competitors to the format like ATOM which is a more complete standard and supports features like paged feeds and explicit content type definitions

ATOM seems to solve the few issues I have with RSS but RSS is still literally everywhere which is tough to fight

My latest side project is an RSS reader called Articly which is built with Svelte/SvelteKit since I really wanted to get a feeling for the framework to build a full application and this seemed like a fun opportunity to do that

Structuring HTML Content

Thu, 26 Jan 2023 00:00:00 GMT

Structuring HTML Content

Isn't HTML a structure?

HTML content is structured as a tree - while this is useful for the medium, this structure isn't very convenient for transforming into data that can be used outside of the web or with libraries that use content in a more flat structure

While building Articly I wanted to use a library called EditorJS for displaying and making text content interactive. An immediate problem I ran into was importing content into the editor since it requires the data in a specific format - which is not HTML but rather an array of objects with simplified content

StackOverflow is helpful sometimes

In order to get the content into the EditorJS format, I needed to find a way to transform the HTML that I had from scraping the web and reading RSS feeds into something I could use as some kind of base

After a little bit of searching, I found this handy function on StackOverflow:

//Recursively loop through DOM elements and assign properties to object
function treeHTML(element, object) {
  object['type'] = element.nodeName
  var nodeList = element.childNodes
  if (nodeList != null) {
    if (nodeList.length) {
      object['content'] = []
      for (var i = 0; i < nodeList.length; i++) {
        if (nodeList[i].nodeType == 3) {
          object['content'].push(nodeList[i].nodeValue)
        } else {
          object['content'].push({})
          treeHTML(nodeList[i], object['content'][object['content'].length - 1])
        }
      }
    }
  }
  if (element.attributes != null) {
    if (element.attributes.length) {
      object['attributes'] = {}
      for (var i = 0; i < element.attributes.length; i++) {
        object['attributes'][element.attributes[i].nodeName] =
          element.attributes[i].nodeValue
      }
    }
  }
}

Using the above as a guideline, I concluded that the main thing that I need was to iterate over the HTML content in a way that would allow me to build a content array which is recursive - the idea at this point is not to remove the tree structure, but rather to transform it into something that's a bit easier to work with

A different tree

Now that I had an idea of how to approach the problem, the next step was to define the structure I wanted, below is the final structure I decided on:

The source HTML element object needs to be stored as this may useful to downstream processors
The tagName of the element will be needed to determine how a specific element needs to be handled
The textContent and innerHTML of the element to be stored - this is the core data of the element
The attributes of the HTML element should be unwrapped to make it easier for downstream code to read
The children of the element need to be converted into the the same type by recursively following steps 1-5

Based on the above, the type definition can be seen below along with the implementation:

type TagName = Uppercase<keyof HTMLElementTagNameMap>

type HTMLStringContent = string

type TransformResult = {
  element: Element
  tagName: TagName
  textContent?: string
  htmlContent?: HTMLStringContent
  attrs: Record<string, string>
  children: TransformResult[]
}

const transform = (el: Element): TransformResult => ({
  element: el,
  // tag names are strings internally but that's not very informative downstream
  tagName: el.tagName as TagName,
  textContent: el.textContent || undefined,
  htmlContent: el.innerHTML,
  children: Array.from(el.children).map(transform),
  attrs: Array.from(el.attributes).reduce(
    (acc, { name, value }) => ({ ...acc, [name]: value }),
    {}
  ),
})

Now that I have the data as a tree structure, we can pass in some simple HTML to see what pops out the other end.

Given the following HTML content:

<div>
  <p>Hello World</p>

  <section>
    <img src="hello.jpg" alt="this is an image" />
  </section>
</div>

We can transform it like so:

// use the DOM to parse it from a string
const el = new DOMParser().parseFromString(html, 'text/html').body

// the convertHtmlToBlocks function takes an HTML ELement
const transformed = transform(el)

The transformed data looks something like this:

{
  "element": {},
  "tagName": "BODY",
  "textContent": "\n  Hello World\n\n  \n    \n  \n",
  "htmlContent": "<div>\n  <p>Hello World</p>\n\n  <section>\n    <img src=\"hello.jpg\" alt=\"this is an image\">\n  </section>\n</div>",
  "children": [
    {
      "element": {},
      "tagName": "DIV",
      "textContent": "\n  Hello World\n\n  \n    \n  \n",
      "htmlContent": "\n  <p>Hello World</p>\n\n  <section>\n    <img src=\"hello.jpg\" alt=\"this is an image\">\n  </section>\n",
      "children": [
        {
          "element": {},
          "tagName": "P",
          "textContent": "Hello World",
          "htmlContent": "Hello World",
          "children": [],
          "attrs": {}
        },
        {
          "element": {},
          "tagName": "SECTION",
          "textContent": "\n    \n  ",
          "htmlContent": "\n    <img src=\"hello.jpg\" alt=\"this is an image\">\n  ",
          "children": [
            {
              "element": {},
              "tagName": "IMG",
              "htmlContent": "",
              "children": [],
              "attrs": { "src": "hello.jpg", "alt": "this is an image" }
            }
          ],
          "attrs": {}
        }
      ],
      "attrs": {}
    }
  ],
  "attrs": {}
}

Not too dissimilar to the structure raw HTML we would have had if we just used DOMParser directly, however it now has a lot less noise

Deforestation

As I mentioned earlier, we need to transform the data into a flat array of items - so the question that comes up now is - how do we do that?

Looking at the input HTML we used, we can break things up into two types of elements - containers and content

Containers are pretty much useless to the content structure we're trying to build - the content is all we really care about

This is an important distinction because it tells us what we can throw away

Secondly, we can think of the content inside of a container as an array of content, once we remove all the containers, this content will become flat

So for example, this HTML:

<div>
  <p>Hello World</p>

  <section>
    <img src="hello.jpg" alt="this is an image" />
  </section>
</div>

When we remove the containers can be thought of as:

<p>Hello World</p>

<img src="hello.jpg" alt="this is an image" />

Which can be thought of as an array like so:

;[paragraph, image]

This is our end goal. In order to get here we still have to figure out two things:

How can we separate out the containers from the content
How can we transform the content into the structure that's useful to us

Separating the leaves from the wood

If we think as containers as having no meaningful data, and just being containers for content, then we can conclude that a way to view the data structure is as an array of content - the container is the array, and the content is the items in the array

So, we can write a transformer for a container as something that just returns an array of content

const removeContainer = (data: TransformResult) => data.children

The above is pretty useful, since this means that the following HTML:

<div>
  <p>Hello World</p>

  <section>
    <img src="hello.jpg" alt="this is an image" />
  </section>
</div>

Will be essentially be converted to this:

<p>Hello World</p>

<section>
  <img src="hello.jpg" alt="this is an image" />
</section>

Now, we still see that there's a section leftover since we only returned one layer of children. Since we already have a way to get rid of the wrapper, we can just apply that to the child that's a section,

So we could have something like this:

const removeContainer = (data: TransformResult) =>
  data.children.map((child) =>
    isContainer(child) ? removeContainer(child) : child
  )

Now the function is a bit weird because the inner map is either returning an array if the child is a container, or a single child if it's not - for consistency, let's just always return an array:

const removeContainer = (data: TransformResult) =>
  data.children.map((child) =>
    isContainer(child) ? removeContainer(child) : [child]
  )

Much better, but now we've introduced something weird - instead of just returning a TransformResult[] we're now returning a TransformResult[][] - let's just leave this here for now, we can always unwrap the arrays later - importantly we now know that we've eliminated the wrappers, so the content we're left with now represents:

<p>Hello World</p>

<img src="hello.jpg" alt="this is an image" />

So this is pretty great, and is the general idea of how we can unwrap things - next up we can talk about transforming the specific elements into useful data blocks

Building blocks

EditorJS has different sections - blocks as it calls them - of content. These are simple Javascript objects that represent the data for the block

For the sake of our discussion, we're going to consider two blocks - ParagraphBlock and ImageBlock

The simplified types that represent their data can be seen below:

export type ParagraphBlock = {
  type: 'paragraph'
  data: {
    text: string
  }
}

export type SimpleImageBlock = {
  type: 'image'
  data: {
    url: string
    caption: string
  }
}

We can look at the the TransformResult for each of the above elements from when we passed the HTML into our transform function previously, we can see

For the paragraph:

{
  "element": {},
  "tagName": "P",
  "textContent": "Hello World",
  "htmlContent": "Hello World",
  "children": [],
  "attrs": {}
}

Which can be translated to the ParagraphBlock data as:

{
  type: "paragraph",
  data: {
    text: "Hello World"
  }
}

A function for doing this could look something like:

const convertParagraph = (data: TransformResult): ParagraphBlock | undefined =>
  data.textContent
    ? {
        type: 'paragraph',
        data: {
          text: data.textContent,
        },
      }
    : undefined

Cool, this lets us transform a paragraph into some structured data.

We can do something similar for images:

const convertImage = (data: TransformResult): ImageBlock | undefined =>
  data.attrs.src
    ? {
        type: 'image',
        data: {
          url: data.attrs.src,
          caption: data.attrs.alt || '',
        },
      }
    : undefined

Putting it all together

So now that we know how to transform the HTML into something useful, remove the wrappers, and represent individual HTML sections as content blocks, we can put it all together into something that lets us convert a section of HTML fully:

First, we can update the removeContainer function to call the converter on the type of tag that it finds:

// note that we need a handler for BODY since the `DOMParser` will always add a body element when parsing
const isContainer = (data: TransformResult) =>
  data.tagName === 'BODY' ||
  data.tagName === 'DIV' ||
  data.tagName === 'SECTION'

const removeContainer = (data: TransformResult) =>
  data.children.map((child) => {
    if (isContainer(child)) {
      return removeContainer(child)
    } else {
      if (child.tagName === 'IMG') {
        const block = convertImage(child)

        return block ? [block] : []
      } else if (child.tagName === 'P') {
        const block = convertParagraph(child)

        return block ? [block] : []
      }
    }
  })

Now, you can probably see a pattern that's going to arise as we add more and more elements that we want to handle - so it may be better to create a list of handlers for different tag types:

type Block = ParagraphBlock | ImageBlock

const handlers: Partial<Record<TagName, (data: TransformResult) => Block[]>> = {
  // content blocks
  IMG: convertImage,
  P: convertParagraph,

  // container blocks - we will always want to remove these
  DIV: removeContainer,
  SECTION: removeContainer,
  BODY: removeContainer,
}

Using the above structure, we can tweak the convertImage and convertParagraph functions a bit so that they return the Block[] consistently:

const convertParagraph = (data: TransformResult): ParagraphBlock[] =>
  data.textContent
    ? [
        {
          type: 'paragraph',
          data: {
            text: data.textContent,
          },
        },
      ]
    : []

const convertImage = (data: TransformResult): ImageBlock[] =>
  data.attrs.src
    ? [
        {
          type: 'image',
          data: {
            url: data.attrs.src,
            caption: data.attrs.alt || '',
          },
        },
      ]
    : []

And we can update the removeContainer function to handle things a bit more genericallly:

const removeContainer = (data: TransformResult): Block[] => {
  const contentArr = data.children.map((child) => {
    const handler = handlers[child.tagName]

    if (!handler) {
      return []
    }

    return handler(child)
  })

  return contentArr.flat(1)
}

If you're really attentive, you'll notice the contentArr.flat(1) that was added in the above snippet, this flattens the Block[][] into a Block[]

Once we've got that, we can define a convert function that will take the HTML and output the structured blocks like so:

const convert = (el: Element): Block[] => {
  const transformed = transform(el)

  const initialHandler = handlers[transformed.tagName]

  if (!initialHandler) {
    throw new Error('No handler found for top-level wrapper')
  }

  return initialHandler(transformed)
}

Adding more content types

That's about it, to handle more specific types of content or other HTML elements is just a matter of following the recipe that we did above:

Define if an element is a container or content
If it's a container, just use the removeContainer handler
If it's content then define a handler for the specific kind of content

If you roll this out for loads of elements you'll eventually have a pretty robust converter

Conclusion

That's it! We've covered the basics for building a transformer like this, and once you have a good feel for how this works, the concepts can be applied to loads of different usecases

If you'd like to see the completed version of my converter, you can take a look at the html-editorjs GitHub repo and if you'd like to look at it in action in an application then take a look at Articly

Using Type Guards and Unions to prevent bugs and increase maintainability

Mon, 26 Jun 2023 00:00:00 GMT

Using Type Guards and Unions to prevent bugs and increase maintainability

When working with a dynamic language, like Javascript, a problem that we can often run into is one where a variable may be of multiple possible types in a given place in our code. Due to this, we often run into a need to check the type of an object

The Scene

For the sake of introducing the need for type guards, we're going to consider an imaginary news site in which users can read articles posted to the site. For our site, we want to enable users to add comments and interact with articles. For this purpose, we require that each user is logged in and exists in our database

The object we use for representing a user who just reads our site is as follows:

const reader = {
  username: 'john',
  email: 'john@email.com',
}

After some time, we decide to make it such that certain users can create articles on the website. To do this, we add a value on the user that states that the user is a writer:

const writer = {
  username: 'smith',
  email: 'smith@email.com',
  isWriter: true,
}

This is great, our data now reflects that we have a user type that can post articles to the website.

The site is growing well and the content available is flourishing, by we seem to be getting a frequent complaints from users that some articles are not appropriate for the site or may be incorrect or misleading

So, to mitigate this issue, we decide that for our website we want to add a new type of permission that defines users that are responsible for moderating content. However, to prevent a conflict of interest - moderators are not allowed to write articles for the site

So we choose to add a field for that indicates a user is a moderator:

const moderator = {
  username: 'bob',
  email: 'bob@email.com',
  isWriter: false,
  isModerator: true,
}

Based on this, we update the application types to account for the new field, and we have the following:

interface User {
  username: string
  email: string
  isWriter: boolean
  isModerator: boolean
}

We update the application to handle the above solution. We even add a few helper methods in our code that allow us to determine if a user is a writer or moderator

const isWriter = (user: User): boolean => user.isWriter

const isModerator = (user: User): boolean => user.isModerator

Fantastic! We're happy with our implementation and we release the code into production

Everything is going great, our moderators are keeping the content quality up and our writers are writing great content

The Bug

After a few weeks though, readers start to notice a problem - some articles that are taken down by moderators seem to be reappearing - it looks as if some malicious moderator is re-enabling bad articles

Upon some further investigation, the team notices that all these articles are ones in which the moderator is the writer of the article - but this shouldn't have happened right? We said that each moderator should not be able to also function as a writer

Eventually, we track down the bug that made this possible, it was in the function called convertUserToModerator:

const convertUserToModerator = (user: User): User => {
  user.isModerator = true

  return user
}

The issue was caused by us not setting the isWriter field to false in the above function, cool - so we just fix that:

const convertUserToModerator = (user: User): User => {
  user.isModerator = true
  user.isWriter = false

  return user
}

Great - we deploy the fix and migrate to ensure that isWriter is set to false for all existing moderators. The issue seems to be fixed, but after a while users start reporting that the problem is happening again

The cause seems to be the same - it looks like users are somehow still having isWriter and isModerator simultaneously set to true and ending up with a permission that shouldn't be possible in our system

Impossible States

So we decide to investigate the problem and determine that the reason we are running into this bug seems to be due to some fundamental way that the data has been defined

Since each user can have true or false for any combination of isWriter and isModerator we end up representing something unintended which can be seen in the table of possible combinations below:

User Type	`isWriter === true`	`isWriter === false`
`isModerator === true`	Unknown	Moderator
`isModerator === false`	Writer	Reader

From the above, we can see that we have a state that is not handled in the current code - this "Unknown" state. In this state, the isModerator and isWriter functions return true

We want this state to be impossible in our code. This is not a state that our application should ever have to consider. From a type-design perspective, it's not enough to handle these impossible states, we need to prevent them from existing in the first place

The Solution

The problem above stems from the fact that we're using two different states to represent something that should be a single state. From the above diagram, we can resolve the following valid types of users in our system

Reader - reads and adds comments to articles
Writer - writes articles
Moderator - moderates articles, does not write articles

Using these user types, we can update the User definition from above:

interface User {
  type: 'reader' | 'writer' | 'moderator'

  username: string
  email: string
}

The type field allows us to determine the type of a user. We can then use this value to check the type of a user and use this to inform how we use the object down the line

Extending the Moderator

Our solution is running well and we have managed to fully eliminate the bug.

But now, the moderators are a little bored with their duties and would like some way that they can compete with each other on the quality of their content moderation - so they decide to create a leaderboard in which they look at how many articles they have read and approved or denied - as interest goes however, the moderators reach out to the development team and ask if it would be possible to add the number of approvals and denials so that the moderators don't need to calculate this manually every month for their leaderboard

The development team has the idea to add these scores into the User definition type:

interface User {
  type: 'reader' | 'writer' | 'moderator'

  username: string
  email: string

  approvedArticles: number
  deniedArticles: number
}

During a PR review session, the team decides that this is a bit weird since it implies that any user can have some approved or denied articles - this doesn't make sense, so someone suggests to use a Discriminated Union in which only users of the moderator type can have these properties - the developer also says that this will make it easier later to add additional fields to any role without having to worry about the implication of this on other roles

The team decides that they want to have a Base user which has some shared properties and a union which adds additional fields

The BaseUser user has a generic parameter called Type which is the type for the user, since we want to ensure that each possible user type has this:

interface BaseUser<Type extends string> {
  type: Type

  username: string
  email: string
}

Each specialized user type can inherit from BaseUser with a specific type applied, for the writer and reader types, these can just be aliases to the base interface

type ReaderUser = BaseUser<'reader'>
type WriterUser = BaseUser<'writer'>

The moderator however, will need to extend this to add the additional properties:

interface ModeratorUser extends BaseUser<'moderator'> {
  approvedArticles: number
  deniedArticles: number
}

Lastly, the three types above can be used to define a single type called User which has the same functionality as the User interface we've been using until now:

type User = ReaderUser | WriterUser | ModeratorUser

The code above is the Typescript syntax for defining a Union which is a type that can be one of multiple types. Each type in the Union is separated by the |. The union we have above is a special type of union called a Discriminated Union which is a special type of Union that has a field called a discriminator which uniquely identifies each element of the union - in our case this is the type field of BaseUser which used the generic Type parameter

In our backend code, we can now use the type property of our user to conditionally fetch the data needed to populate the moderator object

Checks, Checks

As the codebase grows, the team writes some functions to check the type of the user so that specific computations can be done, for example checking the total number of articles a moderator has been reviewed. An example of one of these checks can be seen below:

const isModerator = (user: User): boolean => user.type === 'moderator'

The above check is used in quite a few places, like when counting the total number of articles reviewed:

const getTotalArticlesReviewed = (user: User): number | undefined => {
  if (isModerator(user)) {
    const moderatorUser = user as ModeratorUser

    return moderatorUser.approvedArticles + moderatorUser.deniedArticles
  }

  return undefined
}

Similar casting patterns as above seem to be cropping up in a lot of places, another example is when listing the moderators that a user follows:

const moderatorUsers = users.filter(isModerator) as ModeratorUser[]

Type Guards

The above solutions use Type Assertions which allow a user to override the Typescript Compiler's type system to manually state the type of an object - this is dangerous because it essentially means that we give up type checking at that point in the code

The type assertions around user types are leading to lots of bugs, the team has a discussion about how this problem can be solved and someone suggests using a Type Guard. Type Guards are functions that can be used to check if a given variable meets a specific criterion under which it can be considered a narrower type that the input type. The difference between a Type Guard in this context and a regular function is that a Type Guard returns type information that can be used to inform the Typescript compiler about the state of a variable

The proposed example is to convert the isModerator function into an Assertion Function by converting the return type from boolean to user is ModeratorUser - this assertion tells the compiler that within the scope that the check is true, the value is of the stated type and not the original type

For example, isModerator can be updated as such:

const isModerator = (user: User): user is ModeratorUser =>
  user.type === 'moderator'

This means that when using the code above, we no longer need to cast since Typescript can infer the variable appropriately

const getTotalArticlesReviewed = (user: User): number | undefined => {
  if (isModerator(user)) {
    // within the scope of this condition the `user` type is known to be `ModeratorUser`
    return user.approvedArticles + user.deniedArticles
  }

  return undefined
}

// the `.filter` function accepts the type guard and can now infer that moderatorUsers is `ModeratorUser[]`
const moderatorUsers = users.filter(isModerator)

Conclusion

Discriminated Unions and type Guards are a powerful methods that can be used for working with complex object types and encoding application rules into your type system to minimize bugs and make code easier to extend

Update or Append to DynamoDB Attributes

Tue, 28 Feb 2023 00:00:00 GMT

Update or Append to DynamoDB Attributes

DynamoDB Overview

DynamoDB is AWS's No SQL Database Service. Dynamo uses partition keys and sort keys to uniquely identify and partion item in the database which allows for high scalability and throughput

When working with traditional SQL databases a common operation is to update the value of a specific column without having to first fetch the entire row. DynamoDB offers us similar functionality using the AWS SDK

The Update Command

DynamoDB commands are simple objects that consist of a few different parts. When looking to update an item the following are relavant:

Key - which is an object that uniquely identifies an item in the database
UpdateExpression - which is an expression that describes the upadate operation to be done using placeholders for attribute names and values (AWS Documentation - Update Expressions)
ExpressionAttributeNames - generic placehodlders for attribute names
ExpresssionAtributeValues - generic placeholders for values

When defining UpdateExpression we use placeholders for the attribute names and values to prevent any escaping/unsupported character related issues. A common convention for this is to use # at the start of attribute names and : at the start of attribute values to make expressions a bit easier to negotiate, for example, if I wanted to set name to bob I would do '#name': 'name' in the ExpressionAttributeNames and '#name': 'bob' in the ExpressionAttributeValues

DynamoDB uses an object representation that is a bit inconvenient to work with, we can use the marshall and unmarshall functions from @aws-sdk/util-dynamodb to simplify things a bit but if you'd like to know more about the data format used you can look towards the end of this post

Our Example

For our example, imagine we have a table of user check-ins with data structured as follows:

type Item = {
  // partition key
  group: number
  // sort key
  username: string

  status: string
  checkIns: {
    place: string
    time: number
  }[]
}

Update an Attribute

We can use a bit of a generic structure to outline our data that we plan to update a single attribute. To do this, we can use the SET command

The update expression for setting a value will look like so:

SET #attr = :value

Yup, that's pretty much it as far as the expression goes, the actual names of the fields we want to update aren't relevant here, instead they're mentioned in the ExpressionAtrributeNames and ExpressionAttributeValues

The overall command with all of that is as follows:

import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
import { marshall, unmarshall } from '@aws-sdk/util-dynamodb'

const client = new DynamoDBClient({})

const pk = 'bob'
const sk = 12
const updateValue = 'available'

const command = new UpdateItemCommand({
  TableName: 'my-table-name',
  Key: marshall({
    username: pk,
    group: sk,
  }),
  UpdateExpression: 'SET #attr = :value',
  ExpressionAttributeNames: {
    '#attr': 'status',
  },
  ExpressionAtrributeValues: marshall({
    ':value': updateValue,
  }),
})

await client.send()

And that's pretty much the process for updating an attribute witha specific value

Append to a List Attribute That Exists

In the above data structure we have the checkIns field which is a list of objects

We can use the same method as above to define the UpdateExpression, however this time we can use the list_append function in our expression to state that we would like to append a value to

The list_append function appends one list to another - this means that it needs two lists to operate on. To update a list we can just use itself as the first input and the new list as the second. The UpdateExpression looks like this:

SET #attr = list_append(#attr, :value)

For our sake we only want to append a single item, so we can just wrap it in an array when we pass it on in the command. The command for the above update looks like so:

import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
import { marshall, unmarshall } from '@aws-sdk/util-dynamodb'

const client = new DynamoDBClient({})

const pk = 'bob'
const sk = 12
const updateValue = {
  place: 'home',
  time: Date.now(),
}

const command = new UpdateItemCommand({
  TableName: 'my-table-name',
  Key: marshall({
    username: pk,
    group: sk,
  }),
  UpdateExpression: 'SET #attr = list_append(#attr, :value)',
  ExpressionAttributeNames: {
    '#attr': 'checkIns',
  },
  ExpressionAtrributeValues: marshall({
    // array since the update expression works on two lists
    ':value': [updateValue],
  }),
})

await client.send()

Append to a List Attribute That May Not Exist

In some cases, we can end up trying to append to an attribute that may not exist, under these circumstances we can use the is_not_exists function that takes an attribute name and a fallback value and will return the fallback if the attribute does not exist in the item

We can change the UpdateExpression to make use of this as follows:

SET #attr = list_append(if_not_exists(#attr, :fallback), :value)

And the full command looks like so:

import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
import { marshall, unmarshall } from '@aws-sdk/util-dynamodb'

const client = new DynamoDBClient({})

const pk = 'bob'
const sk = 12
const updateValue = {
  place: 'home',
  time: Date.now(),
}

const command = new UpdateItemCommand({
  TableName: 'my-table-name',
  Key: marshall({
    username: pk,
    group: sk,
  }),
  UpdateExpression:
    'SET #attr = list_append(if_not_exists(#attr, :fallback), :value)',
  ExpressionAttributeNames: {
    '#attr': 'checkIns',
  },
  ExpressionAtrributeValues: marshall({
    ':value': [updateValue],
    // fallback to an empty array if the value does not exist before appending
    ':fallback': [],
  }),
})

await client.send()

The above expresson helps us append an item to the list while also providing a fallback for the case where the list item may not exist

A Note on Marshalled/Unmarshalled data

DynamoDB works with data in the "marshalled" form, which is an object representation for primitive data types (AWS Documentation - Attribute Value). Some examples of marshalled and unmarshalled data can be seen below:

// STRING
// Unmarshalled
"hello"

// Marshalled
{
  "S": "Hello"
}

// NUMBER
// Unmarshalled
25

// Marshalled
{
  "N": "25"
}


// MAP
// Unmarshalled
{
  name: "bob"
}

/// Marshalled
{
  "M":{
    "name":{
      "S":"hello"
    }
  }
}

Additional Resources

Speaking of DynamoDB updates, it looks like there's a library for building queries which seems promising and may bwe worth taking a look at called ElectroDB

Draggable Div with XState and React

Tue, 31 Jan 2023 00:00:00 GMT

Draggable Div with XState and React

Why State Management

Recently I've been interested in understanding non-standard user interactions and about how different applications develop this functionality. A particularly good example for me has been looking into the codebase for TLDraw where I ran into state machines

Now, I've played around with a few state machine libraries and I'm fairly familiar with them and have implemented some fairly simple examples for working with multi-step forms and the like, but I haven't really used them in the specific context of designing state

The State Designer Library mentions the idea of Designing State which suggests that the design of how the user interface should work and the implementation of it should be treated as independent, in order to do so, we should design the states that we would like to work with separately from the UI that implements the state

This provides us with a decent abstraction which should make things about state a lot less tangled and easier to reason about

XState

State management libraries provide us with a set of abstractions and tools for designing state. There are a few that are commonly used, of which I'll be using XState in this post. XState has pretty good TypeScript support as well as some great tooling for visualizing and designing state on Stately

The Problem

For this post I've chosen to build a simple draggable div component that makes use of XState. The idea here was to get a feel for the library and see how it can be used to tackle problems around UI interaction

Visualizing the State

Defining the state can be done using the XState visual editor that can be used in VSCode or Stately, this is useful because it lets you visualize different ways that the state can be represented in a relatively low-friction setting

The structure I've decided on for the representing my component's state can be seen below:

State Machine Code

The code for the above state machine, with some added type information, can be seen below:

Draggable.machine.ts

import { createMachine, assign } from 'xstate'

type Position = {
  x: number
  y: number
}

type Delta = {
  dx: number
  dy: number
}

type Focus = {
  focused: boolean
}

export const machine = createMachine(
  {
    schema: {
      context: {} as Position,
      events: {} as
        | { type: 'MOVE'; position: Delta }
        | { type: 'FOCUS' }
        | { type: 'BLUR' },
    },

    initial: 'inactive',
    states: {
      inactive: {
        on: {
          FOCUS: 'active',
        },
      },

      active: {
        on: {
          MOVE: {
            target: 'active',
            internal: true,
            actions: 'updatePosition',
          },
          BLUR: 'inactive',
        },
      },
    },
  },
  {
    actions: {
      updatePosition: assign((context, event) => ({
        x: event.position.dx + (context.x || 0),
        y: event.position.dy + (context.y || 0),
      })),
    },
  }
)

In the above, we can also see the following:

An initial state that is inactive with an event of FOCUS which sets the state to active
An active state that has an internal event of MOVE which will trigger the updatePosition action and an event of BLUR which sets the state to inactive
There is a context that will store the coordinates of the dragged element
An actions object which has an action called updatePosition which assigns the context to the new position

We can also see that in the schema the type of events is specified. This makes it so that XState can infer the type of event passed to the updatePosition function

Attach the State to the UI

In order to move from one state to another we use the send method from XState

To use a state machine in React we use the useMachine hook from XState:

Draggable.tsx

import { useMachine } from "@xstate/react";
import React from "react";
import { machine } from "./Draggable.machine";

interface DraggableProps {
  children: React.ReactNode;
}

export const Draggable = ({ children }: DraggableProps) => {
  const [current, send] = useMachine(machine);


  // rest of component

Next, we can use the current to figure out if we are in an active state so that we can do some styling based on that:

Draggable.tsx

const isActive = current.matches('active')

And lastly we can hook up the UI Events to the state machine:

Draggable.tsx

export const Draggable = ({ children }: DraggableProps) => {
  const [current, send] = useMachine(machine)

  const isActive = current.matches('active')

  return (
    <div style={{ position: 'relative' }}>
      <div
        onMouseDown={() => send('FOCUS')}
        onMouseUp={() => send('BLUR')}
        onMouseLeave={() => send('BLUR')}
        onMouseMove={(ev) => {
          send({
            type: 'MOVE',
            position: {
              dx: ev.movementX,
              dy: ev.movementY,
            },
          })
        }}
        style={{
          position: 'absolute',
          backgroundColor: isActive ? 'skyblue' : 'red',
          userSelect: 'none',
          top: current.context.y,
          left: current.context.x,
          padding: 20,
        }}
      >
        {children}
      </div>
    </div>
  )
}

The above will result in a draggable div like so:

Repl.it link

Possible Improvements

If you play around with the above example you'll probably notice that it's not perfect. Though the states we've defined are correct,the complexity in mapping the UI events becomes apparent, as well as the various edge cases that may arise around how DOM events fire in response to user interaction

Though I don't aim to solve all of these points for the sake of the example, it's worth pointing out as a source of exploration. Some possible solutions that may help are things like changing where we listen to specific events, for example listening to the mouseUp events on only the wrapper and not the draggable component, or changing which specific events we're listening to

The idea is that the handling of the UI events is now separated from the actual state management which should make fixing interaction bugs simpler while also decoupling our state from any specific implementation of the UI

week 35, year 2024 - datetimes and logic programming

Sun, 01 Sep 2024 00:00:00 GMT

week 35, year 2024 - datetimes and logic programming

what i'm working on

after many months i spent some time on a project i'm working with a friend to build a sort of "home cloud" platform. it's always weird coming to a project having no clue where you left it and trying to stitch some context from whatever happens when you run pnpm start

i also found a handy little way to open links from the vscode terminal

what i found

i spent a little time learning about logic programming and came across an interesting talk about some of the challenges when writing libraries

prolog

a logic programming language - super weird but this talk on prolog by Michael Hendricks helped shed some light on how this is something that's actually usable

standard libraries

i also came across a great talk about some challenges when maintaining something like the kotlin standard library by Vsevolod Tolstopyatov

week 31, year 2024 - webgpu and parsers

Fri, 02 Aug 2024 00:00:00 GMT

week 31, year 2024 - webgpu and parsers

welcome

since this is my first post in this format i figured i should just say hey! this series will be a bit of a braindump for project updates and links to things i found this week

what i'm working on

right now i'm the concept of a photo editing app that uses shaders for applying image effects, the initial concept seems to be promising but there's still a fair amount of work to be done before i can have anything even remotely usable

i also had an excuse to use git bisect in a new and interesting way and made some additions to my git notes about using the —first-parent and replay functionality

what i found

`ts-parsec`

i've been pretty invested in parser combinators recently and have been playing around with the ts-parsec library and it's been a pretty fun time. for a general idea of what these things are all about you can take a look at my blog post on parser combinators

`WebGPU`

webgpu is the "new" way to create shaders on the web and comes with a new javsacript API and a new language called wgsl

the api is a bit more complex than the one for webgl but still reasonably manageable if you take a read through the webgpu fundamentals site

additionally, i found a few great resources for using webgpu for image filters, namely work by Alain Galvan and Maxim McNair

Conditionally Protect Properties in Typescript

Tue, 03 Dec 2024 00:00:00 GMT

Conditionally Protect Properties in Typescript

Type Guards

So, type guards are really handy in Typescript as they let us check if something meets a certain requirement before moving along, for example, given the following user type:

type User = {
  active: boolean;
  name: string;
  age: number;
};

We can define a type guard that checks if the user is fully is active before allowing certain things. To do this we usually use a type guard, that looks like this:

type ActiveUser = User & { active: true };

const isActive = (base: Partial<User>): base is ActiveUser =>
  !!(base.active && base.age && base.name);

The important thing in a type guard is that it takes something of one type and asserts something about that type, e.g. that it is an ActiveUser in the above. This is done by returning a boolean, if it is true then the assertion applies, otherwise it does not

And normally, we would use it like so:

if (isActive(user)) {
    // do stuff that can only be done with an active user
}

// outside of this scope users are not active

Gaurded Class

The above solution is usually good enough. But it's also possible to couple the types of these checks while not providing direct access to the underlying object. This is useful in cases where we may want to restrict access to some functionality unless a certain set of checks pass

To do this, we can encapsulate the value and check into a class, e.g. the Guarded class below:

class Guarded<Unsafe, Safe extends Unsafe> {
  constructor(
    protected readonly value: Unsafe,
    private readonly safe: (value: Unsafe) => value is Safe,
  ) {}

  /**
   * Type guard that grants access to the wrapped `value`
   */
  isSafe(): this is { value: Safe } {
    return this.safe(this.value);
  }
}

In the above, we define a Guarded class htat has a type of a Safe and Unsafe value. These generics can be inferred by the arguments provided to constructor

Next, we define the isSafe method on the class that is a gaurd that says something about this which is the instance itself.

So, we can use Guarded to wrap something that we want to only make available under certain scenarios:

const guarded = new Guarded(value, isActive);

if (guarded.isSafe()) {
  console.log('Can access value here', guarded.value);
}

// guarded.value; // error: Property 'value' is protected and only accessible within class 'Guarded<Unsafe, Safe>' and its subclasses.ts(2445)
console.log("Can't access value here");

What's great is that this safe check is re-run whenever we want to access the value, this means that we can make certain properties available based on some other state that can be checked in the safe method

This approach means that at compile time the Typescript compiler will check that we are only accessing value within a context that we have defined to be okay by way of our isSafe method. Additionally, since we can specify the behavior of isSafe we end up with a a check that works at the intersection of compile-time and runtime

Note

The first idea of Type Guards is quite often used and should be good enough for normal use

Withr egards to the Guarded idea, use with caution. The idea here it to create a certain safety for consumers of some code, but this is at the expense of complexity and should be used with care. Outside of the "can it be done" discussion I haven't really had much use for something like this since usually a normal type guard is good enough

Show Children when Parent is Hidden with CSS

Thu, 04 Apr 2024 00:00:00 GMT

Show Children when Parent is Hidden with CSS

When working with HTML we sometimes have a case where we want to view child items of an HTML element while the parent element itself is hidden

Normally when trying to hide or show some elements, we would apply display:none and move on. However, we may run into a case where we want to make a specific child visible while keeping the remaining content hidden, so, we might try to add the following to the child display:block.

We will quickly see that this solution doesn't work. Another thing we might trying to play with the opacity of the items but this doesn't really solve our problem either

The solution to this is the visibility property

According to the MDN Documentation on Visibility: "The visibility CSS property shows or hides an element without changing the layout of a document."

The MDN definition brings up an important note - the visibility property does not change the layout of the document - this means the hidden elements will still take up space on the screen, which is different to the behaviour of display:none. However, we can mitigate that in other ways if we need to deepending on what we're trying to do

We can use the visibility property to get what we want. For our example let's use a ol with some children, however, we want anything inside of our list to be hidden other than a specific child

The HTML we have is as follows:

css-visibility/css-visibility.css

.hide {
  display: var(--hide-display);
  visibility: var(--hide-visibility, hidden);
}

.show {
  display: var(--show-display);
  visibility: var(--show-visibility, visible);
}

.example {
  padding: 1rem;
  background-color: var(--color-on-base);
  color: var(--color-base);
}

css-visibility/CSSVisibility.html

<div class="example">
  <div>before hidden</div>

  <ol class="hide">
    <li>Item 1</li>
    <li class="show">Item 2</li>
    <li>Item 3</li>
  </ol>

  <div>after hidden</div>
</div>

before hidden

after hidden

It's also important to note that the content before and after the list are not visible but they still take up space

You can play around with the CSS properties below to see how the influence the overall styling:

css-visibility/css-visibility.css

.hide {
  display: var(--hide-display);
  visibility: var(--hide-visibility, hidden);
}

.show {
  display: var(--show-display);
  visibility: var(--show-visibility, visible);
}

.example {
  padding: 1rem;
  background-color: var(--color-on-base);
  color: var(--color-base);
}

css-visibility/CSSVisibility.html

<div class="example">
  <div>before hidden</div>

  <ol class="hide">
    <li>Item 1</li>
    <li class="show">Item 2</li>
    <li>Item 3</li>
  </ol>

  <div>after hidden</div>
</div>

before hidden

after hidden

View Transitions and an Astro Presentation Framework

Wed, 06 Mar 2024 00:00:00 GMT

View Transitions and an Astro Presentation Framework

Well, since this is a post about building a presentation framework within your Astro site, it may be worth mentioning that you can view this page as a presentation using the below button:

The Problem

I'm kind of lazy.

I had to put together a little presentation based on something I've written about previously and wanted a lazy way to reuse my existing content while also making the resulting presentation available on my website

Overall, these are the requirements I had in mind:

My Requirements

Not require any additional build process
Work with Markdown or MDX so I can include it in my website easily
Have a small learning curve
Integrate flexibly with my existing content - pages should be able to be very easily converted to slides

So I investigated a few solutions:

Existing Slide Solutions

Just a markdown doc
Reveal.js
MDX Deck
Spectacle
Plain' ol' HTML
PowerPoint??

The existing solutions just don't work for my case

Now, it's not that they're not good - most of them are pretty great and have some features that I would like to use if this were some once-off throwaway presentation, but since I would like to refer back to and manage the way I want they're not really suitable

I also didn't want to style everything from scratch or write lots of HTML everywhere

The Solution

Build it myself. Obviously

Instead of just looking at the existing options, I instead chose to build a library/framework that would work with my existing Astro site while keeping the implementation relatively minimal and just depending on plain CSS, Javascript, MDX, and Astro to get the job done

Use Existing Tooling

HTML
CSS
Javascript (Typescript)
MDX

Facilitated by - not coupled to - Astro

Code

So, since I'm building it myself, that means code - and that's what we're going to look at

The API

I wanted to keep the API relatively simple. It should work with existing markdown content and allow me to delineate a slide in a way that can be easily read from the DOM so as to minimize the amount of build-time processing I need to do as well as minimize how much markup I need to write. For this purpose, I decided that I want it to fit into an MDX document like so:

The API

Button to launch the presentation

<site-presentation />

... Existing page content

<site-presentation-slide>

## Heading for Slide

Some content for the slide

```js
console.log('I am a code block')
```

> And literally any other markdown content

</site-presentation-slide>

Hiven the above, I had a high level idea that I would need two parts to this - firstly I want to use the Slide component to give me something to latch onto in the HTML that I can manipulate, secondly I know I would need some kind of component that would control the overall presentation state, I called that Presentation

The `Slide` Component

The Slide component is simply a wrapper that includes content in an HTML section with a class presentation-slide which will contain the contents of a slide

The `Slide` Component

components/Slide.astro

<section class="presentation-slide>
  <slot />
</section>

The `Presentation` Component

The Presentation component needs to do a few different things:

The `Presentation` Component

Hide presentation until enabled
Allow navigation of slides
Render slide content above existing page
Manage transitions between slide pages

Firstly, we can just take a look at the HTML that we will render contains a few basic elements as well as a script tag that grabs a reference to these elements. The presentation-hidden class is used for hiding or showing the presentation when active/inactive:

Basic Elements

components/Presentation.astro

<button id="presentation-button" class="presentation-hidden" type="button">
  Start Presentation
</button>

<div
  id="presentation-container"
  class="presentation-hidden presentation-overflow-hidden"
>
  <main id="presentation-content">
    <h1>No slides found on page</h1>
  </main>
</div>

<script>
  const button = document.getElementById('presentation-button') as HTMLButtonElement
  const container = document.getElementById('presentation-container') as HTMLDivElement
  const content = document.getElementById('presentation-content') as HTMLDivElement
</script>

Next, we can grab the actual slide content by using the presentation-slide class we defined earlier:

Get Slides

components/Presentation.astro

let slides = Array.from(document.querySelectorAll('.presentation-slide')).map(
  (el) => el.outerHTML,
)

let slide = 0

Once we have the content of the slides and a variable to track which slide we are on, we can define a function that will set the slide content. This will set the innerHTML of the content element to the slide that is active. We can handle this by first defining some utilities for grabbing the next and previous slides as well as mapping a key code to the function that will resolve the next slide

Slide Utilities

components/Presentation.astro

const nextSlide = () => {
  if (slide === slides.length - 1) {
    return slide
  }

  return slide + 1
}

const prevSlide = () => {
  if (slide === 0) {
    return slide
  }

  return slide - 1
}

const keyHandlers: Record<string, () => number> = {
  ArrowRight: nextSlide,
  ArrowLeft: prevSlide,
}

Next, we can define what it means for us to start and end a presentation. For this example, starting a presentation will remove the presentation-hidden class from the main wrapper so we can make the presentation visible on the page as well as set the content to the current slide index (we initialized this to 0 above)

Start and End Presentation

components/Presentation.astro

const startPresentation = () => {
  container.classList.remove('presentation-hidden')
  if (slides.length) {
    content.innerHTML = slides[slide]
  }
}

const endPresentation = () => {
  container.classList.add('presentation-hidden')
}

We set the content to slide instead of 0 so that we can pause and continue the presentation if we wanted to

Next, hook up some event handlers so that we can have a method for controlling our presentation:

Wiring things up

components/Presentation.astro

// If there is no presentation on the page then we don't initialize
if (slides.length) {
  button.addEventListener('click', startPresentation)

  window.addEventListener('keyup', (ev) => {
    const isEscape = ev.key === 'Escape'
    if (isEscape) {
      endPresentation()
      return
    }

    const getSlide = keyHandlers[ev.key]

    if (!getSlide) {
      return
    }

    const nextSlide = getSlide()
    if (slide === nextSlide) {
      return
    }

    slide = nextSlide
    content.innerHTML = slides[slide]
  })
}

In the above, the left and right arrows are used to navigate slides and the escape key is used to end the presentation

Next up, we need to add some CSS to make the slides pin to the root of our application above everything else so that you can actually use this:

Styling

components/Presentation.astro

<style is:global>
  .presentation-overflow-hidden {
    overflow: hidden;
  }

  .presentation-hidden {
    display: none;
  }

  #presentation-container {
    z-index: 10;
    position: fixed;
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
    overflow: auto;

    backdrop-filter: blur(50px);
    background-color: #0000007d;
  }

  #presentation-content {
    display: flex;
    flex-direction: column;

    background-color: black;
    color: white;

    box-sizing: border-box;
    min-height: 100vh;
    width: 100%;
    padding: 4rem;
  }
</style>

And that's pretty much it for the core implementation. One other piece of fanciness that I wanted to add was the ability to make an actual slide transition. To do this I decided to use the View Transitions API and found a few nice references on the Unnecessary View Transitions API List

In order to use this you need to have the feature enabled in your browser at the moment but it should be stable soon (I hope)

For this implementation, we will need to have different animations for the case where we are moving forwards or backwards. In order to do this, we will define some classes as part of our keyboard handler resolution that we will append to the presentation-container:

View Transitions

components/Presentation.astro

const nextClass = 'presentation-next'
const prevClass = 'presentation-prev'

const transitionClasses = [nextClass, prevClass]

const keyHandlers: Record<string, [string, () => number]> = {
  ArrowRight: [nextClass, nextSlide],
  ArrowLeft: [prevClass, prevSlide],
}

Then, we will update our event handling logic to set these classes on the contaienr

Setting the Classes

components/Presentation.astro

const [transitionClass, getSlide] = handler

content.classList.remove(...transitionClasses)
content.classList.add(transitionClass)

const nextSlide = getSlide()
if (slide === nextSlide) {
  return
}

Then, instead just setting the content.innerHTML directly, we do it within the document.startViewTransition callback which will be what handles the state transition between the content leaving the DOM and the new content that is entering the DOM

Note that the startViewTransition API is experimental and typescript may complain, you will need to install @types/dom-view-transitions which will provide the type definition you need to use this API

Starting the Transition

components/Presentation.astro

document.startViewTransition(() => {
  slide = nextSlide
  content.innerHTML = slides[slide]
})

The last thing we need to do is define the view transitions for when the content enters and exists the DOM. The transitions are defined in the style tag of our component as follows:

Firstly we need to

Animations

components/Presentation.astro

@keyframes slide-out-right {
  0% {
    transform: translateX(0) scale(1);
  }
  15% {
    transform: translateX(0) scale(0.8) translateY(0%);
  }
  85% {
    transform: translateX(100%) scale(0.8) translateY(0%);
  }
  100% {
    transform: translateX(100%) scale(1);
  }
}

@keyframes slide-out-left {
  0% {
    transform: translateX(0) scale(1);
  }
  15% {
    transform: translateX(0) scale(0.8) translateY(0%);
  }
  85% {
    transform: translateX(-100%) scale(0.8) translateY(0%);
  }
  100% {
    transform: translateX(-100%) scale(1);
  }
}

The above defines two basic animations that we will use for our transitions. We define an animation that moves an element off the screen to the right called slide-out-right and another to move it to the left called slide-out-left. These animations can also be reversed to slide content in from the right or left respectively

For the "Next" animation we need to do the following:

Slide the old content to the left
Slide the new content from the right

In the below we set that the presentation-next class defines a view-transition-name called next. Then, we define the transitions for the old and new content that applies to the next transition name as an Animation. We are referencing a slide-out-right and slide-out-left animations which we defined previously:

CSS Transitions

Below is the transition for moving to the next slide

components/Presentation.astro

.presentation-next {
  view-transition-name: next;
}

::view-transition-old(next) {
  animation: slide-out-left 0.5s linear;
}
::view-transition-new(next) {
  animation: slide-out-right 0.5s linear reverse;
}

view-transition-old refers to content that is being removed from the DOM, view-transition-new refers to the content that is being added to the DOM

Lastly, the implementation for the presentation-prev we can reuse the same animations for moving left or right as we defined previously, but change the directions as needed for the relevant section

components/Presentation.astro

.presentation-prev {
  view-transition-name: prev;
}

::view-transition-old(prev) {
  animation: slide-out-right 0.5s linear;
}
::view-transition-new(prev) {
  animation: slide-out-left 0.5s linear reverse;
}

And yes, that's a fair amount of code. All-in it's about 200 lines - most of which is the CSS for the transition though. Generally the implementation is pretty straightforward and should be relatively easy to tweak to match the vibe of your website without adding any dependency bloat.

As it stands right now the implementation is pretty simple but leaves a lot of space to be extended

Added since this post was written

Presenter mode with some kind of synchronized state for multiple monitors (LocalStorage?)
Progress tracking
Support for non-static components

Future Ideas

Presenter notes and preview of next slide
Make this a library so other people can use it with less copy pasta
More transitions and styling possibilities
Dynamic code blocks/customizable transitions (Code Surfer)
Automatic zooming

Conclusion

We used some interesting CSS here and overall we can see that it's not always a huge amount of work to write your own implementation of something.

Update 17 June 2026

This component has since been massively updated and now uses web components on the side - take at the source in your browser for the latest iteration!

References

week 40, year 2024 - paper and pictures

Sun, 06 Oct 2024 00:00:00 GMT

week 40, year 2024 - paper and pictures

firstly, a small sadness - i missed the past two posts due to my vacation, but on the other hand i guess i had a vacation so that works out i suppose

what i'm working on

spent the past week mostly working on a talk that i'm (hopefully) going to be doing towards the end of the month, i also gave a little intro presentation on nushell that I think went pretty well

what i found

this week i'm going to be sharing a few sites i've come across of people doing interesting things

paper engineering

Julia Yus makes really interesting things out of paper and her portfolio is a really window into the things that you wouldn't have thought were possible with paper

fashion photography

George Morris is a London based photographer and director with a portfolio of images and videos focused on some campaigns that he's done

street photography

Eric Anderson is a photographer and web designer with a particularly great portfolio of street photography

Use localStorage for Tab Synchronization

Thu, 07 Mar 2024 00:00:00 GMT

Use localStorage for Tab Synchronization

localStorage

localStorage is a method for persistent data storage in the browser. The storage itself is a simple key-value store with a synchronous API

We can write data to a given key using the following:

window.localStorage.setItem('my-key', 'my-data')

And we can get that data back using:

const myData = window.localStorage.getItem('my-key')

Pretty neat right?

For the purpose of this post we'll look into how we can use this as a method for sending data between different browser tabs of the same website

Storage Event

Aside from just data storage - localStorage (as well as sessionStorage) provides us with a notification when the data in a storage is changed - this is called a StorageEvent. What's particularly interesting to us is that it's fired in all other tabs of a particular site when any a tab of that site modifies the data

We can listen for this event using window.addEventListener

window.addEventListener('storage', () => {
  console.log('Something has changed in storage')
})

Now, if you think about it for a short while, you may come to an interesting conclusion - we can talk to other tabs

Speaking to Tabs

Open this page in two different windows side-by-side for this example

Since we can talk between tabs, we can synchronize parts of our UI using the messages we recieive

Below we have a button that uses this - each time it is pressed in one tab - it updates the count in all tabs

localstorage-sync/SynchronizedButton.html

<button id="sync-button">Count</button>

<script>
  (function () {
    const button = document.getElementById('sync-button')
    const syncKey = 'button-example-key'

    const getCount = () => {
      const count = window.localStorage.getItem(syncKey) || '0'

      return parseInt(count)
    }

    const render = () => {
      const count = getCount()
      button.innerText = `Count: ${count}`
    }

    window.addEventListener('storage', () => render())

    button.addEventListener('click', () => {
      const count = getCount()
      const nextCount = count + 1

      // Since the storage event only fires in other tabs we must also render our own tab
      render()
      window.localStorage.setItem(syncKey, nextCount.toString())
    })

    render()
  })()
</script>

Generic Implementation

So overall, this is a pretty neat method for sharing data between pages, but we end up having to do some repetitive work when setting this up in many places or when working with anything that isn't a string (as you can see with the parseInts required in the above example)

To make this a little more generic we're first going to extract the getting and setting of data into their own functions:

const getValue = <T>(key: string, initial: T) => {
  try {
    const existing = window.localStorage.getItem(key)
    if (!existing) {
      return initial
    }

    return JSON.parse(existing) as T
  } catch {
    return initial
  }
}

const setValue = <T>(key: string, value: T) =>
  window.localStorage.setItem(key, JSON.stringify(value))

These provide relatively simple wrappers around the reading and writing of data and making it possible for us to work with objects

Next, we can provide an easy way to define a data reader and writer - the reason we define these separately is to make the interface a little easier for consumers to use without needing to rearrange all their code:

export type SetValue<T> = (value: T) => void

export const createSyncReader = <T>(
  key: string,
  initial: T,
  onChange: (value: T) => void
) => {
  window.addEventListener('storage', () => {
    const value = getValue(key, initial)
    onChange(value)
  })

  return () => getValue(key, initial)
}

export const createSyncWriter =
  <T>(key: string): SetValue<T> =>
  (value) =>
    setValue(key, value)

Our reader function will call the provided onChange callback that is provided to it when the data is changed with the parsed data. We can implement the same button as above using this new code as can be seen below:

localstorage-sync/SynchronizedButtonRefactor.html

<button id="sync-button-refactor">Count</button>

<script>
  (function () {
    const getValue = (key, initial) => {
      try {
        const existing = window.localStorage.getItem(key);
        if (!existing) {
          return initial;
        }
        return JSON.parse(existing);
      }
      catch (_a) {
        return initial;
      }
    };
    const setValue = (key, value) => window.localStorage.setItem(key, JSON.stringify(value));

    const createSyncReader = (key, initial, onChange) => {
      window.addEventListener("storage", () => {
        const value = getValue(key, initial);
        onChange(value);
      });
      return () => getValue(key, initial);
    };

    const createSyncWriter = (key) => (value) => setValue(key, value);

    const button = document.getElementById('sync-button-refactor');
    const syncKey = 'button-example-key-refactor';
    const render = (count) => {
      button.innerText = `Count: ${count}`;
    };
    const readValue = createSyncReader(syncKey, 0, render);
    const writeValue = createSyncWriter(syncKey);
    button.addEventListener('click', () => {
      const count = readValue();
      const nextCount = count + 1;
      // Since the storage event only fires in other tabs we must also render our own tab
      render(nextCount);
      writeValue(nextCount);
    });
    render(readValue());
  })()
</script>