Daniel Michaels | Developer | Entrepreneur | Veteran

HTTPX is the new Requests with Async

Thu, 25 Jun 2020 00:00:00 +0000

Long live requests welcome HTTPX

HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.

Far and away, requests stands as the most notable third party package. It’s almost considered part of what makes up python these days. Unfortunately, little forward progress has been made in recent times to incorporate python’s async capabilities.

This is where HTTPX is stepping up to the plate. And it aims to be compatible with most of requests API.

Why use HTTPX over Requests?

In addtion to async, HTTPX also supports:

HTTP/2
Timeouts by default
Full type annotation (yay, code editors)
Direct requests to WSGI and ASGI applications
Plus, all the standard features of requests - list here

Simple example - sync

As HTTPX supports the core requests API, to do a simple get of a webpage we can use httpx.get()

>>> import httpx

>>> url = 'http://httpbin.org/get'
>>> resp = httpx.get(url)
>>> resp.json()
# json output cleaned up for ease of reading.
{'args': {},
 'headers': {'Accept': '*/*',
 'Accept-Encoding': 'gzip, deflate',
 'Host': 'httpbin.org',
 'User-Agent': 'python-httpx/0.13.3',
 'X-Amzn-Trace-Id': 'Root=1-5f014d41-5cef343010793f4686d7c258'},
 'origin': '19.13.3.36',
 'url': 'http://httpbin.org/get'}
}

Client is the new Session

When using requests, if you needed to do anything more than receieve simple data then the Session instance was necessary.

HTTPX does not have a Session API instead it has Client and AsyncClient for obvious use cases.

From the HTTPX docs:

If you do anything more than experimentation, one-off scripts, or prototypes, then you should use a Client instance.

For more reasons why you should use Client read the docs. But the core reasons are:

Reduced latency across requests (no handshaking).
Reduced CPU usage and round-trips.
Reduced network congestion.
Without Client you don’t get HTTP/2 support

To use a the Client interface, its recommended to use a context manager.

>>> with httpx.Client() as client:
 ...

To make a request with the Client its as simple as:

>>> with httpx.Client() as client:
 resp = httpx.get('https://httpbin.org/get')

>>> resp
<Response [200 OK]>

One of the really cool features is base_url which allows for URL prepending.

# base_url example
>>> with httpx.Client(base_url='http://httpbin.org') as client:
 resp = client.get('/headers')

>>> resp.request.headers
Headers({'host': 'httpbin.org', 'user-agent': 'python-httpx/0.13.3', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive'})

AsyncClient simple

To make an async call using the Client interface is as simple as adding the async keyword to the context manager and swapping Client with AsyncClient.

# AsyncClient example
>>> async with httpx.AsyncClient() as client:
 resp = await client.get('http://httpbin.org/get')
>>> resp
<Response [200 OK]>

There is some API differences when using the AsyncClient that are worth being aware of.

AsyncClient real example

I am not a python async expert so I did have some issues getting it to function as expected. When I needed to concurrently get several json endpoints worth of data a lot of experiementation was needed on my part.

Florimond Mancas post set out the ground work for solving my problem; I needed a way to call a list of url’s looping over the endpoint from a list id’s.

# Florimond's example
>>> import httpx
>>> import asyncio
>>> # We're going to fetch tag pages concurrently...
>>> async def fetch(tag, client):
 return await client.get(f'https://dev.to/t/{tag}')

>>> async with httpx.AsyncClient() as client:
 responses = await asyncio.gather(
 fetch('hacktoberfest', client),
 fetch('python', client),
 fetch('opensource' client),
 )

>>> responses
[<Response [200 OK]>, <Response [200 OK]>, <Response [200 OK]>]
>>> urls = [r.url for r in responses]
>>> urls
[URL('https://dev.to/t/hacktoberfest'),
 URL('https://dev.to/t/python'),
 URL('https://dev.to/t/opensource')]

This worked perfectly in an Ipython terminal but I needed to integrate into a sync class (swapping out the slow synchronous api call’s for a faster async version).

My version

I needed three things:

A method to call each endpoint,
AsyncClient for executing the requests, and
Something to kick off these actions.

The following is a snippet which meets all three critieria. It takes Florimond’s demonstration example and shows how it could be implemented into a real piece of callable code.

# 1
async def fetch(item_id, client):
 """
 Return async response object for HackerNews item.
 :param item_id: json id for HackerNews item
 :param client: httpx.AsyncClient method
 :return: httpx.AsyncClient response object
 """
 return await client.get(
 f"https://hacker-news.firebaseio.com/v0/item/{item_id}.json"
 )
# 2
async def story_metadata():
 """
 Return a list of responses from the given list of HackerNews item id's.
 :return: list of HackerNews response objects
 """
 _list = [23709004, 23717964, 23723433, 23725506, 23713605]
 async with httpx.AsyncClient() as client:
 responses = await asyncio.gather(*[fetch(item_id, client) for item_id in _list])
 return responses
# 3
def run_async_task(coro):
 """
 A helper method to create async coroutines.
 :return: result set from the async function
 """
 results = asyncio.run(coro)
 return results

Adding async here led to a speed increase of 50% over a relatively small selection of items - it only grabs the top ten articles. The return time went from 3+ seconds to ~1.1-1.5, which is consistent with network latency rather than python synchronicity.

Wrap up

Over the last month or so, whenever I have needed a requests like interface I have elected to use HTTPX instead. In this time there has been only one thing missing; a caching adapter. Previously, I would lean heavily upon requests-cache.

Apart from this, the experience has been seemless in fact I find HTTPX easier to use. It has a very active community, great leadership and highly active development. Being a relatively new package it is very possible to get involved and contribute as well.

Gist

A full copy of the working code is below as well as here

DNS the easy parts

Sun, 16 Jun 2019 00:00:00 +0000

DNS

The Internet’s success is predicated on the Domain Name System or DNS. We’ve all heard of it, many of us loosely understand it but, like all things, when you need to fix it knowledge gaps become painfully obvious.

When it comes to diagnosing network issues, understanding adblocking, speeding up queries or protecting privacy, you can bet DNS will be mentioned. Unlike other protocols such as Network Address Translation , or Open Shortest Path First, DNS is a network fundamental that every engineer should have down. Sadly, during my CCNA studies, it did not get much more than the cursory explanation.

Overview

DNS is a protocol that can map IP addresses to human readable names. When a user types example.com into their browser bar and hits enter, the machine does not know how to reach example.com without first translating it into an IP routable address.

Before any network activity takes place between the client and the destination, a DNS lookup must first be conducted. Luckily, DNS is fast but it does add latency to the request. If this transaction fails the client will be unable to make a connection to the server. A user could, alternatively manually enter the servers IP address and this would work - as long as the server is up. But if we could remember IP addresses as readily as we do names, DNS would be redundant.

It is worth noting that DNS also allows us to clump many IP addresses under one namespace. For instance, we can reach hosts on our local network using domain names instead of IP addresses. We use DNS for more than just browsing the web; emails, IoT devices and many other services that make networks invisible to the majority of users are powered by this system.

From ten thousand feet, lets examine how a simple name to IP address translation takes place.

DNS Resolution

In the below diagram we have a series queries starting with the browser asking for the address to example.com and ending with the server delivering content back to the browser.

Figure 1. Simple DNS Lookup - attrib

When our above client types example.com into the browser and hits enter, a series of steps are taken by the client and external servers to translate the human readable name into a machine readable IP address.

Each step broken down:

After the user hits enter, the browser will check to see if the address has been cached, if not the operating systems cache will queried, and if both fail, the system will send out a request to a DNS resolver.
Our resolver will parse its own cache and if nothing matches, will then forward a request to one of the DNS root name servers. Root name servers are at the top of the DNS hierarchy and are represented by the right most . of our websites name.
Once received, the root name server will check its zone file for the Top-Level Domain (TLD) matching the request - in this case com. It returns the address of com’s name server back to our resolver.
Our resolver now forwards a new request, this time to the com TLD authoritative name server.
The com name server checks its zone file for the subdomain example and returns that address to the resolver.
Now our resolver can ask the DNS server responsible (authoritative) for the example domains address.
The server responds with the address to example.com.
Our resolver, having sent out three requests up to this point, will now forward the correct IP address to the client. It will also save the address in its cache for future lookups.
The browser will now initiate a connection using HTTP with example.com.
example.com having received the connection request, begins transferring data to the clients browser.

This is a very simplistic resolution. It works well in common household networks where your wireless router is not a DNS server. If you are instituting a firewall or managing your own name server, the below diagram is a more likely reflection of what is happening.

Figure 2. Forwarding Queries

In the above example, the local DNS server sends a query to a recursive resolver from a large DNS provider. As stated in Figure 2, this can also be called a Forward DNS Server. The forward resolver will conduct all the iterative requests on our behalf before returning the answer back to our local DNS server, who forwards it to the client.

Why Root?

DNS works in a hierarchical structure with the root or . being at the top of the tree - very similar to the Linux file structure.

So why does the recursive resolver go straight to the root name server, and not say the com domain?

Looking at the above example we can see that everything is connected to root. So, the resolver must start with the only known address it has; root. Logically, it is the only server that can reach all possible domains.

Top-Level Domains

After querying the root server, it will return the TLD name server for the domain being queried. The TLD server contains all the information for domains that share a common extension, such as edu, com, or whatever comes directly after the last dot in the url. More simply, the name server for net will have the address for every website that ends with net.

So, when the user requests example.com, the TLD name server of the com domain will reply with the address of the authoritative server for the example domain. Simply put, the name server who looks after the com domain knows where to find every single domain that uses a .com and will return the websites address back to the resolver, completing the lookup.

Top-level domains can be broken into two groups:

Generic (gTLD):
- Addresses such as com, net, and gov.
Country code (ccTLD):
- Country specific, jp, au, ca.

With an address such as example.com.au the resolver will still query the root name servers but it will send the resolver to the country specific top level domain of au. The domain to the right of the last . will always be queried first.

We can demonstrate this by running drill bbc.co.uk -T or dig bbc.co.uk +trace.

Subdomains

Technically, after the TLD, each domain down the tree can be referenced as a subdomain; example in example.com being a subdomain of the com TLD.

Typically, subdomains more often refer to addresses such as api.github.com with api being a subdomain of github.

You can see the resolution by running drill m.facebook.com -T or dig m.facebook.com +trace.

At the date of publication, the facebook lookup provided a good example of subdomain and CNAME resolution. I have included a truncated output below.

>> drill m.facebook.com -T

...snip...
.	518400	IN	NS	l.root-servers.net.
.	518400	IN	NS	i.root-servers.net.
com.	172800	IN	NS	l.gtld-servers.net.
...snip...
com.	172800	IN	NS	m.gtld-servers.net.
facebook.com.	172800	IN	NS	a.ns.facebook.com.
facebook.com.	172800	IN	NS	b.ns.facebook.com.
m.facebook.com.	3600	IN	CNAME	star-mini.c10r.facebook.com.
c10r.facebook.com.	3600	IN	NS	a.ns.c10r.facebook.com.
c10r.facebook.com.	3600	IN	NS	b.ns.c10r.facebook.com.
star-mini.c10r.facebook.com.	60	IN	A	157.240.8.35

Terms

Essentially all technical efforts wallow in acronyms and special “terms of art” […] with several non-obvious terms to confuse those who have not been involved for a while.

RFC 4144 - How to Gain Prominence and Influence in Standards Organizations

Sums up many of the terms used to express components of the domain name system. A brief description of the main offenders follows and how to examine them using drill. If drill is unfamiliar to you, dig will work just the same. nslookup is not covered in this post, but will essentially do the same thing.

DNS resolver

The ‘resolver’ is the agent between the client and the name servers. A resolver will start the queries that eventually return a valid address translation for the client.

A recursive resolver knows how to traverse the tree to deliver a response to a query.

Authoritative Server

Owns the domain, or has all the records for that domain. E.g. the com name server, knows the whereabouts of every single domain that uses it. Popular sites such as google.com, yahoo.com and github.com are all domains under the com authoritative server, it knows the address to each of those sites DNS server within their network.

Top-level Domain (TLD)

Domains that are one level below the root are known as top level domains.

Examples: com, jp, edu

Country Code TLD’s such as au, uk and nz are examples. taste.com.au will be resolved in the following order:

. - Root
-> .au
-> com.au
-> taste.com.au

This can be seen via drill -T taste.com.au which will trace (what -T does) through each root name server all the way to the A record of taste.com.au.

Recursive request

When the client needs to get the address of a URL, it first checks its cache and if not found will send a request to its recursive resolver asking for the address. Often, the resolver being queried will be your ISP’s default name server (please don’t do this; they monetize it), or something like Google’s public DNS server; 8.8.8.8. My opinion: use 1.1.1.1 or some other provider - Google are using you, too. The recursive resolver will then begin a series of iterative requests on your behalf.

For the most part recursive queries begin with the client and end at the recursive resolver in a kind of A->B, B->A relationship. This is because the resolver is doing iterative queries on your behalf and here lies the big difference between recursive and iterative requests.

Recursive queries require an answer

As Microsoft puts it:

recursive query indicates that the client wants a definitive answer to its query. The response to the recursive query must be a valid address or a message indicating that the address cannot be found.

Meaning a recursive can only reply with either the address, or an error (NXDOMAIN) but not a referral to another name server.

Iterative request

Generally, the queries from a recursive resolver to other name servers are iterative. They leave the resolver and start at the top of the DNS tree via the root name servers. The root replies with the appropriate domains name server. The resolver then makes another query to that server. The iterations continue down the hierarchy until an address is resolved.

Unlike recursive queries, iterative requests will accept any answer; address, error or a referral to someone who knows more.

Microsoft again:

An iterative query indicates that the server will accept a referral to another server in place of a definitive answer to the query.

Non-Iterative request

When a query is made and the resolver has the address mapping in its cache, it will refer the client immediately to it. No further queries are made and this is known as a non-iterative request.

Record Types

A:

IPv4 address associated to domain name.
Running drill example.com defaults to returning an A record.

AAAA:

IPv6 address association.
drill aaaa example.com will explicitly return the IPv6 address, if it has one.

Mail Exchanger (MX):

Mail exchanger records containing the emails associated to the domain.
drill mx google.com will return all the mail exchangers, and each ones priority.

Name Server (NS):

A list of name servers connected to the domain.
drill ns cloudflare.com outputs their name server domains.

Pointer (PTR):

Pointer file for reverse DNS lookups
For example, do the following: drill defence.gov.au. Take the IP address - 203.6.74.5 and do a reverse lookup using only the IP address. drill -x 203.6.74.5. It should return www.defence.gov.au.
When both the forward (name-to-address) and reverse (address-to-name) entries match, this is known as a forward-confirmed reverse DNS. Its a weak form of authentication - it proves the IP and domain are owned by the same entity. Sender Policy Framework and MX records are often better indicators of authenticity. Nonetheless, it can be a method used to whitelist a domain, or a reason for other mail exchangers to blacklist, or drop mail to that domain, if the lookup fails.

Canonical Name (CNAME):

Can be used to alias one name to another, most often to an A record. It operates a bit like a treasure map with clues pointing to the treasure - the A record.
drill blog.cloudflare.com returns CNAME cloudflare.ghost.io which directs to the A record.

As a example of how to read a zone file, or response with CNAME information:

 NAME TYPE VALUE
--------------------------------------------------
bar.example.com. CNAME foo.example.com.
foo.example.com. A 192.0.2.23

This should be interpreted as:

bar.example.com. is an alias for the CNAME foo.example.com.. A client request for bar.example.com. will be returned foo.example.com.

The next query to the example domain will be asking for the foo subdomain. Which will be returned an A record with address 192.0.2.23. This adds latency as another round trip must be conducted and generally chaining CNAME’s is considered bad form.

Text record (TXT):

Contains extra data such a sender policy framework and encryption information.
Often contain valuable information about the domain, for instance:

>> drill telstra.com.au txt

;; ANSWER SECTION:
telstra.com.au. 1303 IN TXT "v=spf1 include:_spf.telstra.com.au ~all"
telstra.com.au. 1303 IN TXT "google-site-verification=GeZnWLAmVtLSHbjW5VEawrbtjQzhSY-e4wsT9VBaJkU"
telstra.com.au. 1303 IN TXT "7EC66119-3393-46B7-9DA9-9AD7643925FA"

The TXT returned some information about the telstra.com.au domain; they use Google Apps in their systems.

A better one is vodafone.com.au who I know for certain use Salesforce (I saw it on their systems in store recently) and that is readily confirmed in their TXT response.

>>> drill vodafone.com.au txt -t

;; ANSWER SECTION:
vodafone.com.au.	506	IN	TXT	"MS=ms76000886"
vodafone.com.au.	506	IN	TXT	"407D-EB23-4E8F-A96D-CA1D-27F0-1FD5-B8DC"
vodafone.com.au.	506	IN	TXT	"google-site-verification=s_5FhJrgc0EtEXFv6eikEih2i1fjH18PIz2e93qxOLE"
vodafone.com.au.	506	IN	TXT	"NrkgBWZQJfU+vkg4E7z1RchofBb+WL+Ab0NdlXM6OZ4lo5KJLtj7U8LVEV+iASwT/7Zrn4OqBQ5ETPOrPn9cwA=="
vodafone.com.au.	506	IN	TXT	"v=spf1 ip4:54.171.206.163 ip4:101.119.57.42 ip4:119.11.1.42 ip4:101.119.57.43 ip4:119.11.1.43 ip4:101.119.57.14 ip4:119.11.1.14 ip4:216.9.247.4 ip4:216.9.247.48 ip4:216.9.247.49 ip4:216.9.247.68" " include:_spf.salesforce.com include:herald.responsetek.com include:srs.bis3.ap.blackberry.com include:taleo.net include:production.fxdms.net a:mail1.hutchison.com.au a:mail2.hutchison.com.au a:mail3.hutchison.com.au a:mail4.hutchison.com.au ~all"
vodafone.com.au.	506	IN	TXT	"BR6ST3VIM"
vodafone.com.au.	506	IN	TXT	"MS=ms13634633"

The -t flag indicated the response needed TCP as the message size was too large to fit in one UDP packet.

drill lets you know:

;; WARNING: The answer packet was truncated; you might want to
;; query again with TCP (-t argument), or EDNS0 (-b for buffer size)

Start of Authority (SOA):

Contains administrative info about a zone such as domain owner, contact details and serial number.
Serial number is used to determine the currency of the zone’s data.

drill SOA taste.com.au

>>> ;; QUESTION SECTION:
>>> ;; taste.com.au. IN SOA
>>> 
>>> ;; ANSWER SECTION:
>>> taste.com.au. 176 IN SOA dns0.news.com.au. hostmaster.news.com.au. 2019060600 900 600 604800 300

Each tabbed section in detail:

IN and SOA; Internet and Start of Authority.
dns0.news.com.au is the primary name server.
The email contact for the domain; hostmaster.news.com.au is equal to hostmaster@news.com.au. The first . should be replaced by an @ symbol.
2019060600 is the current serial number for the domain. Serial numbers are used by other DNS servers to check its trustworthiness in delivering up to date information.
900 The time in seconds the secondary name server should wait before checking the master’s SOA record for changes. Also known as the refresh rate. More aptly this means; how long am I willing to accept my secondary server having out-of-date information.
600 is the retry time it should wait before trying another refresh if the last one failed.
604800 is the expire counter in seconds. It lets the secondary name server know how long to hold their information before it is no longer considered authoritative. Generally, this is a large number, and should always be greater than refresh and retry counters.
300 is the minimum time-to-live in seconds before the records in the zone are considered invalid.

Today, the refresh, retry, expire and minimum are still important but used much less frequently. RFC’s 1996 and 2136 have included the NOTIFY and UPDATE opcodes to allow the master DNS server to send updates to its slaves, rather than waiting for them to poll the master for changes.

Wrapping Up

The domain name system is far too technically rich to even scratch the surface in one short post, but hopefully it helps to cover off a few confusing points. If nothing else, it may serve as a launch pad for some to go out and read more on this incredible and interesting protocol. One which we use hundreds of times a day.

If you want to use a state of the art DNS service that respects privacy, is customisable, easy to use and secure then check out NextDNS. It’s free to sign up, can be configured on routers, browsers and devices. My android devices all override their system DNS and block ads so effectively it feels like internet is free of them! If you’re interested, try it here at my affliate link.

NATS as C2

Wed, 13 May 2026 00:00:00 +0000

NATS as C2

https://www.sysdig.com/blog/nats-as-c2-inside-a-new-technique-attackers-are-using-to-harvest-cloud-credentials-and-ai-api-keys

I’m using NATS as an outbound connector/transport between my control server and pfSense firewalls - looks like they’re also leveraging this same technique. In my usecase its flawless!

I’ve been telling security friends that they are sleeping on NATS. Maybe after reading this they’ll actually read up on it instead of default to what they know; rabbit and kafka.

“Tasks are queued centrally, workers pull and explicitly ack, and a dropped worker returns its in-flight tasks to the queue for redelivery. This matches the architectural argument earlier in this writeup: NATS-as-C2 gives operators durability and at-least-once delivery without bespoke client code.”

NATS servers provide three properties that scanner-pool operators historically had to engineer themselves:

Wire-level authorization: Per-subject ACLs are enforced by the broker, not by client-side checks that a captured node can disable.

One-to-many fan-out: A single publish to result.scan reaches every aggregator without the worker enumerating peers, which improves OPSEC and simplifies horizontal scaling.

First-class auth and durability: Username/password, TLS, and nkey auth are native, and JetStream provides durable queues so a worker can drop offline without losing its work.

“The KeyHunter operator that the Sysdig TRT discovered is using NATS for the same reasons engineering teams adopt it: subject-scoped authorization, native fan-out, and durable queues. None of those properties alone are unique to legitimate workloads, and applying them to a credential-hunting worker pool produces a botnet that is more liable and scalable than the typical HTTP-panel architecture”

Take away the C2 nerfarious nature of this application and you’re left with a legitimate tool that is incredibly powerful and sadly overlooked by many.

I implore people to give NATS a try.

Tags:

#nats #c2

Shunt docs page is live

Sun, 26 Apr 2026 00:00:00 +0000

Shunt docs page is live

I created a mkdocs page for my NATS based rule router; [shunt].

I use this extensively in my homelab. For instance,

Making all my Zigbee2MQTT events available in VictoriaMetrics
Notifications automations by making HTTP requests into my Knative handlers
Telling me when my plants need watering
Various other

Some examples use cases:

Telegram hooks for Jellysearr so I know that I need to approve kids requests (and when they are done)
Synadia Control Plane alert webhooks into Knative which trigger telegram
Motion sensors lights for my office

An example shunt rule:

- name: z2m-metrics
 trigger:
 nats:
 subject: "zigbee2mqtt.*"
 action:
 http:
 url: "http://fn-z2m-metrics.knative-functions.svc.cluster.local/ingest"
 method: POST
 headers:
 X-Z2M-Device: "{@subject.1}"
 passthrough: true
 retry:
 maxAttempts: 3
 initialDelay: "2s"
 maxDelay: "10s"

This will listen for every message on the zigbee2mqtt.* subject and make a POST with the * subject token as the header. Internally, my knative function will receive the request stripping the header for our metrics endpoint.

For the uninitiated, knative functions are HTTP based and thats why we POST into the handler rather than listen on another subject and have shunt publish to it. Knative eventing works basically the same way but shunt means we can avoid the overhead of eventing completely and rely solely on serving.

This is a noisy endpoint and shunt works pretty well here.

Tags:

#shunt #nats #knative

Zed's new Agent first panel is good

Thu, 23 Apr 2026 00:00:00 +0000

Zed’s new Agent first panel is good

I really like Zed’s new Agent first panel.

Bummer that I just spent time setting up my hotkeys to open the agent panel e.g. <leader>ccc - kind of moot now. But, this is a better change!

I still prefer using JetBrains products (especially for debugging) but for pure “agentic” style work (such as project I don’t care that much about) it’s a better, faster experience.

There might be a future where I’d switch solely to Zed over JetBrains - but I still think that future is a way off yet.

Love seeing teams try new things like this too - the “agentic” world is in so much flux right now so who knows if they’ll even keep this change. I hope they do.

Tags:

#zed #ide

Agents should use config directories properly

Mon, 13 Apr 2026 00:00:00 +0000

Agents should use config directories properly

Why do all the agents/LLM’s use ~/.$foo instead of ~/.config/$foo.

Its lazy programming, teaches others that it’s okay to do this (it’s not) and basically ensures for the rest of time we have our home directories filled up with shit.

Do a ls -la on your $HOME and look at home much junk there is. $XDG_CONFIG_HOME exists for a reason.

I feel these companies have a duty of care because everyone will follow them blindly.

Thing is we can plainly see none of them give a shit. Look at Anthropic stealing IP. Scam Altman (say no more).

Talk about grinding my gears.

At least OpenCode gets right - literally one of the few shining lights in this shitshow.

Culprit list:

ollama
openai (.codex)
anthropic (.cluade)
pi
amp
crush
google (.gemini)

Tags:

#rant #ai

TIL: raycast for mac file search

Fri, 10 Apr 2026 00:00:00 +0000

TIL: raycast for mac file search

Geez I feel like a right snapper head.

I hate finder, in fact I legitimately hate most of MacOS defaults like what hell is wrong with that file explorer - utter trash.

Anyway TIL that I can use Raycast to actually search like a sane person.

If I could delete finder and the poor excuse for a file explorer from MacOS I would.

It’s a POS and anyone who likes it is a victim of some Stockholm syndrome like disease also known as the “cult of mac”.

Otherwise super impressed with my MBP!

Tags:

#rant #raycast

I built a Tauri app for my home lab

Mon, 23 Mar 2026 00:00:00 +0000

I built a Tauri app for my home lab

I’ve been leaning a lot more into AI coding lately.

It’s made me rethink a couple things:

I can use any language
Reliability and error detection at compile time is more important than ever

Rust is a trade-off between compile-time and memory safety at the cost of developer experience and cognitive load. It takes a lot longer to develop a Rust app than a Go one and the results for most app’s (that I would built) aren’t worth the “Rust cost” IMO.

AI kind of changes the game here.

It iterates so fast that Rust on-boarding and learning can be “just-in-time” or “just-enough”. Now you can “write” Rust and learn it as you go - it’s like learning to code by reading someone else’s code except its your code for your idea.

And, now you get the sweet compile time safety. It also has the benefit of keeping AI in check - it writes something and breaks an contract somewhere, well it’ll detect that and the knock out the fix super fast.

I still love Go, I still opt for it (for 95% of things) but think AI is changing the way we look at certain objectives.

So I wrote a Tauri app. It’s a MacOS dock based app for monitoring all my Knative applications status. It connects to your kube context (so technically could monitor any Knative stack).

It can:

ping it (to wake it up)
view its ingress (if its serving a page/API etc will show it)
view logs

I don’t need too much more than that.

I found the Tauri experience to be top-notch compared to Wails (a Go Tauri like app builder).

Repo: https://github.com/danielmichaels/knative-dash

Rust, cap.so and capturing login links

Mon, 23 Mar 2026 00:00:00 +0000

I recent spun up a cap instance on my Coolify instance. It’s free and open source which is great but it has some limitations:

Only sends magic login/sign up links
Only supports Resend as the Email provider

Whilst Resend is great and has a free tier I don’t want to use it. Firstly, it only supports a single domain and it cannot be a wildcard. Secondly, I have a SMTP provider already that I like to use for this sort of stuff.

I noticed that if you don’t set the Resend provider it will output the magic link code to the containers stdout. Not very helpful when you want to login on your phone.

It got me thinking - could I trap the login code and forward to my phone somehow?

Which is when I set Claude to work building a app to do exactly that.

I recorded a quick video of it in action and have the repo available on github.

But basically I created a Rust app that listens to the Cap container stdout for the text containing the code and when found it sends it to my telegram channel.

Coolify runs Cap as a docker compose file so I built a container and added it to that compose file. Now it exists within that compose network giving easy access to the docker logs.

Pretty simple and now I can login from anywhere!

Tags:

#rust #coolify

TIL: Remarkable can share to screen

Wed, 11 Mar 2026 00:00:00 +0000

If you are on the same network and have the Remarkable software installed it will sync from the tablet to your app on the computer.

Which is awesome for screen sharing and recording videos on OBS etc.

Such a great find.

Tags:

#til #remarkable

Dokploy Clourdflared Tailscale and Proxmox

Sun, 08 Mar 2026 00:00:00 +0000

Dokploy Clourdflared Tailscale and Proxmox

I am pretty happy with this little experiment:

Can I host Dokploy inside my LAN using Proxmox and have an application auto-deploy on git push from GitHub?

Well, yes.

Installing Dokploy isn’t covered here but I just following the PVE Helper Scripts and installed it into a Ubuntu LXC (and then snapshotted it).

I hooked up GitHub and a particular repo which has a docker-compose file. Then deployed it using that fille. Note that I don’t build this but instead pull from <ghcr.io> - this saves a lot of compute/memory on the host Dokploy and circumvents build issues.

I also setup a Cloudflared container in another Dokploy project using a compose file as well - passing in the API key from Cloudflare. All apples.

The trickiest and more painful part when setting up the tunnels was the subdomain limitations I found on Cloudflare.

These did not work:

app.dev.domain.com

These did:

app-dev.domain.com

I got the app.dev.domain.com to work initially but then I needed river.dev.domain.com but it would not allow another subdomain. I removed the app.dev. domain and then could using any nested subdomains at all. Instead had to resort to using $app-dev.$domain which looks dumb IMO. But, it works so moving on.

Now I got all that working pretty easily. Could access the applications from my LAN on a domain I controlled. But, Dokploy only offers a URL for re-deploying when using a Compose project which means it needs internet access.

Here comes Tailscale and GitHub actions to the rescue.

Secret	Source
`TS_OAUTH_CLIENT_ID`	Tailscale admin → OAuth clients
`TS_OAUTH_SECRET`	Tailscale admin → OAuth clients
`DOKPLOY_API_KEY`	Dokploy → Settings → Profile → API
`DOKPLOY_COMPOSE_ID`	One-time API query above
`DOKPLOY_URL`	`http://<tailscale-ip-or-hostname>:3000`

Github Action snippet:


# this step comes after build and push to GHCR
deploy:
 name: Deploy to Dokploy
 runs-on: ubuntu-latest
 needs: merge
 if: success()
 steps:
 - name: Connect to Tailscale
 uses: tailscale/github-action@v2
 with:
 oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
 oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
 tags: tag:ci

 - name: Trigger Dokploy deployment
 run: |
 curl -fsSL -X POST \
 "${{ secrets.DOKPLOY_URL }}/api/compose.deploy" \
 -H 'Content-Type: application/json' \
 -H "x-api-key: ${{ secrets.DOKPLOY_API_KEY }}" \
 -d "{\"composeId\": \"${{ secrets.DOKPLOY_COMPOSE_ID }}\"}"


I like the OAuth approach as its ephemeral and really wasn't that much extra work to get set up.

So now, instead of deploying Dokploy to a VPS costing me money I now get all benefits (and trade-offs too) of running it in my homelab but having it internet accessible. 

Also it was my first time using Dokploy (I run Coolify in another VPS) and I think its got some nice
features but for my usecase I think they're equally as good. I do like that Dokploy supports multi-instance containers due to its native `docker compose` support. With that also comes the really nice ability to run NATS in an isolated compose setup. 

Tags:

 #dokploy #tailscale #cicd

GHA build and push faster

Fri, 06 Mar 2026 00:00:00 +0000

GHA build and push faster

I’ve been using the same GHA build and push for my local/private/personal projects for years and they take forever.

I asked claude to make it faster and knocked a 15m build to 4m with this:

name: Build and Push to GHCR

on:
 push:
 branches: [main]
 tags: ["v*.*.*"]
 workflow_dispatch:

env:
 REGISTRY: ghcr.io
 IMAGE_NAME: ${{ github.repository }}

jobs:
 build:
 name: Build ${{ matrix.platform }}
 runs-on: ${{ matrix.runner }}
 permissions:
 contents: read
 packages: write
 strategy:
 fail-fast: false
 matrix:
 include:
 - platform: linux/amd64
 runner: ubuntu-latest
 - platform: linux/arm64
 runner: ubuntu-24.04-arm

 steps:
 - name: Checkout
 uses: actions/checkout@v4

 - name: Set up Docker Buildx
 uses: docker/setup-buildx-action@v3

 - name: Log in to GHCR
 uses: docker/login-action@v3
 with:
 registry: ${{ env.REGISTRY }}
 username: ${{ github.actor }}
 password: ${{ secrets.GITHUB_TOKEN }}

 - name: Extract metadata
 id: meta
 uses: docker/metadata-action@v5
 with:
 images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

 - name: Build and push by digest
 id: build
 uses: docker/build-push-action@v6
 with:
 context: .
 platforms: ${{ matrix.platform }}
 push: true
 labels: ${{ steps.meta.outputs.labels }}
 outputs: type=image,name=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }},push-by-digest=true,name-canonical=true
 cache-from: type=gha,scope=${{ matrix.platform }}
 cache-to: type=gha,mode=max,scope=${{ matrix.platform }}

 - name: Export digest
 run: |
 mkdir -p /tmp/digests
 echo "${{ steps.build.outputs.digest }}" > /tmp/digests/${{ matrix.platform == 'linux/amd64' && 'amd64' || 'arm64' }}

 - name: Upload digest
 uses: actions/upload-artifact@v4
 with:
 name: digest-${{ matrix.platform == 'linux/amd64' && 'amd64' || 'arm64' }}
 path: /tmp/digests/
 if-no-files-found: error
 retention-days: 1

 merge:
 name: Merge manifests
 runs-on: ubuntu-latest
 needs: build
 permissions:
 contents: read
 packages: write

 steps:
 - name: Download digests
 uses: actions/download-artifact@v4
 with:
 path: /tmp/digests
 pattern: digest-*
 merge-multiple: true

 - name: Set up Docker Buildx
 uses: docker/setup-buildx-action@v3

 - name: Log in to GHCR
 uses: docker/login-action@v3
 with:
 registry: ${{ env.REGISTRY }}
 username: ${{ github.actor }}
 password: ${{ secrets.GITHUB_TOKEN }}

 - name: Extract metadata
 id: meta
 uses: docker/metadata-action@v5
 with:
 images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
 tags: |
 type=ref,event=branch
 type=semver,pattern={{version}}
 type=semver,pattern={{major}}.{{minor}}
 type=sha,prefix=sha-,format=short

 - name: Create and push manifest
 run: |
 DIGESTS=$(cat /tmp/digests/amd64 /tmp/digests/arm64 | xargs -I{} echo "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@{}")
 TAGS=$(echo "${{ steps.meta.outputs.tags }}" | xargs -I{} echo "--tag {}")
 docker buildx imagetools create $TAGS $DIGESTS

Instead of using QEMU.

Tags:

#til #cicd

Remove firefox autocomplete history

Fri, 27 Feb 2026 00:00:00 +0000

Remove firefox autocomplete history

Because I constantly forget how to do this:

Click form
Press Down arrow key
Should be highlighted, if so:
Shift+Backspace

Tags:

#til

Codex has a pretty slick UI

Thu, 26 Feb 2026 00:00:00 +0000

Codex has a pretty slick UI

Contrasting Claude Code against Codex its plainly obvious that Codex’s rust implementation is blazingly fast.

Whether its better or not is moot. In the age of AI I do not get why you would use node to build ~~(anything..)~~ a CLI app. Granted Claude Code kicked off the terminal AI revolution but still like…why?

Really impressed by how smooth Codex works in comparison and I hope many more tools adopt Go, Rust, Zig as their UI’s (like OpenCode have as well)

Tags:

#ai #rust #cli

I suddenly feel compelled to start a factory at home

Tue, 24 Feb 2026 00:00:00 +0000

I suddenly feel compelled to start a factory at home

https://youtu.be/hqGFcwyXYI0

I don’t know what I’d create, build or sell but I do know that I need a homemade factory!

Amazing what some cheap plywood and Arduino’s can net you. More evidence that I need to keep going with my OT learning and to keep setting up my IoT home network but with more physical controls.

Some random things I would like to build:

Water tank overflow regulator
Garden water tank fill control system (guttering fed with active cut over when full)

Random little projects that would be useful around the house and get me more hands on with control systems and hardware.

Tags:

#factory #iot #youtube

Singleton row pattern in application design

Thu, 19 Feb 2026 00:00:00 +0000

Singleton row pattern in application design

Anti-pattern? It depends.

I’m using it in a situation where only one instance of a key can ever exist. New key? We update the existing. It is a global state mechanic. The alternative is an Entity-Attribute-Value table - also a “it depends” anti-pattern.

In this case I’m creating an identifier with the same value, always. All INSERT and DELETE operations target the same known id. Has a nice benefit of making upsert behaviours deterministic.

Whilst I have this feeling of it being “wasteful” to use a single table for a single row - I think it works (we’ll see what my workmates say!).

Some alternatives; is_active BOOLEAN. This is nice because we have a historical log. We just insert new keys with is_active=1 and set all the others to is_active=0. Or, ORDER BY created DESC LIMIT 1. But, it adds some complexity like deactivating old rows when activating the new one, queries are a little more complex (I guess not too complex - WHERE is_active == 1). I think the downside here is AFAIK the database doesn’t provide a guarantee only a single row is active.

It depends

Is the right call here but for my current use case it works well enough. Should I ever need more historical analysis then moving to the is_active model is a no-brainer.

Also, I think you could have the singleton and a separate history table. I have done this in the past as a cheap timeseries like purview of the changes - it that example it was tracking DNS records over time; current versus historical.

Tags:

#database #design

SCP nats-box creds k8s

Fri, 13 Feb 2026 00:00:00 +0000

SCP nats-box creds k8s

Note to self, for k8s local testing a quick way to do some debugging with access to nats is via nats-box. In control plane you need creds so I just cp a creds file into the container.

kubectl cp ~/Downloads/my.creds nats-box-664bcb786c-6wt22:/tmp/my.creds -n nats

Then use it from inside the container.

If you have network path to the cluster/control plane server then you can just use nats CLI as is. E.g. nats -s $NATS_URL --creds my.creds rtt

Tags:

#nats #debug #k8s

FluxCD soldiers 5

Thu, 12 Feb 2026 00:00:00 +0000

FluxCD soldiers 5

1. Force Reconcile (most common)

# Reconcile a kustomization and pull latest git source
flux reconcile kustomization <name> --with-source

# Reconcile just the git source
flux reconcile source git flux-system

# Reconcile a specific HelmRelease
flux reconcile helmrelease <name> -n <namespace>

# Reconcile a HelmRepository (force re-fetch chart index)
flux reconcile source helm <name> -n <namespace>

When: You pushed a commit and don’t want to wait for the poll interval. --with-source on a kustomization reconciles its referenced GitRepository first.

2. Suspend / Resume (reset failure state)

flux suspend helmrelease <name> -n <namespace>
flux resume helmrelease <name> -n <namespace>

When: A HelmRelease has exhausted its retry budget (e.g. 10 failures) and is stuck. Even after pushing a fix, Flux won’t retry because the failure counter is maxed out. Suspend/resume resets the counter and forces a fresh reconciliation with current values.

Also works for kustomizations:

flux suspend kustomization <name>
flux resume kustomization <name>

Why not just reconcile? flux reconcile only triggers a new attempt if retries remain. Once retries are exhausted, the controller stops trying. Suspend/resume is the reset.

3. Check Status

# Overview of all Flux resources
flux get all

# Specific resource types
flux get kustomizations
flux get helmreleases -A
flux get sources git
flux get sources helm -A

# Detailed events on a HelmRelease
kubectl describe helmrelease <name> -n <namespace>

# Watch for changes
flux get helmreleases -A -w

4. Debugging

# Check Flux controller logs
flux logs --kind=HelmRelease --name=<name> --namespace=<namespace>

# Check all Flux controller logs
flux logs --all-namespaces

# Verify local values against chart schema before pushing
helm repo add <repo> <url>
helm repo update
helm template <release> <repo>/<chart> -f values.yaml

# Check what revision Flux sees
flux get sources git

Tags:

#flux #kubernetes

AI on the cow

Mon, 02 Feb 2026 00:00:00 +0000

AI on the cow

I’m AI exhausted. Most AI-first products/business are junk (clawdbot being an temporal example at time of writing).

This on the other hand is a real, tangible use case. They’re running a tiny model on constrained embedded controllers attached to a cows neck.

It runs inference on the telemetry of the cow in 7ms which (as they state) significantly reduces battery draw. If anomoly is detected then it will wake the board and send packets of BLE-LE to a base station. The base station then forwards it to the (Go) backend for processing and storing into TimescaleDB.

And, I’m glossing over all the embedded work they’ve done like design their own chips and their own pick-n-place machine to prototype boards!

Honestly amazing and reminds that I love embedded stuff - might whip out the arduino’s and esp32’s!

Link: https://www.youtube.com/watch?v=E4Ng1iW_tfs

Tags:

#iot #cow #embedded #ai

TIL: Go's fnv hash

Wed, 14 Jan 2026 00:00:00 +0000

TIL: Go’s fnv hash

A colleague introduced me to hash/fnv from the Go stdlib today.

It’s a very simple and fast hashing lib when you don’t need cryptographic hashing.

Some use cases for me to remember:

Hash map keys
Deduplication

demo

Tags:

#til #hashing

Tailscale with pfSense

Tue, 30 Dec 2025 00:00:00 +0000

Tailscale with pfSense

I finally set up Tailscale on pfSense and what a massive improvement to my life it is!

Being able to connect to Tailscale and then get instant access to everything running on my LAN is massive.

Now I can do things like host apps in my public Coolify and have that container connected to TS giving it access to non-public assets running in my home network. It also doesn’t consume my tailscale machine limits because I’m just exposing the subnets - that being said those services do not get a TS routable address. I don’t mind though as the direct subnet connection is fine as these services are on static IPs.

I dislike the Cloudflare tunnels approach - this is much better for this use case. That being said, I do host Pangolin for other access patterns.

Tags:

#tailscale

Github is the Microsoft of VCS

Tue, 16 Dec 2025 00:00:00 +0000

Github is the Microsoft of VCS

And the irony of Microsoft owning Github is not lost on me.

Github is the de facto, first to market hosted git repo.

We all know it got market share because it was first to market, in SV and gave you free private repo’s from the get go.

People equate github with git - I’m sure some even think Github created git!

Anyway, after the new pricing for Github Actions self-hosted runners I think its become so painfully obvious that its a dying product.

GitLab is superior. Fact. End of story.

If you haven’t used GitLab in a commerical production environment you really don’t know what you’re missing out on. GitLab CE/free-tier just ain’t the same.

Gitea, Forejo aren’t it either - they’re just GitHub dressed up in a new suit. I mean Gitea modelled their “actions” after GHA - what the hell.

I swear I’m becoming more GitLab pilled every day!

Tags:

#git #rant #github #gitlab

I miss being a sniper sometimes

Fri, 12 Dec 2025 00:00:00 +0000

I miss being a sniper sometimes

https://www.youtube.com/watch?v=MvvBqcWNW_s

I don’t watch much of this stuff any more but they got with that thumbnail!

Some times I do miss that life. Honing your craft, really getting good at something. Pushing the limits and being relied upon to make decisions that matter.

Also having an absolute blast doing it too!

Being paid to become world class at long range shooting isn’t normal. I need to remember I lived a life that was abnormal - did a job that was a lifestyle and all consuming.

That time is over but geez it’s hard not to miss the good parts.

Also so good to see Nightforce absolutely dominating - I think I saw a Leupold once the rest were NF’s.

Tags:

#random

Huma API demo repo

Mon, 24 Nov 2025 00:00:00 +0000

Huma API demo repo

I created and pushed a Huma demo application to GitHub.

I’ve been using Huma in my personal projects for a while now and so far am very happy with it. It “just works”.

Unless you are writing a trivial API I think you must have an OpenAPI specification available to your users and your future self.

Huma takes a code first approach - similar to FastAPI. I really like its validation model and how nicely it outputs errors.

When leveraged with restish it is a great pairing.

Repo is at https://github.com/danielmichaels/huma-demo and show cases unauthenticated routes alongside authenticated.

API key and Cookie authentication is demonstrated via the API, API docs and a HTML index page.

Claude did a great job slopping out the HTML/JS to demo the cookie auth.

Tags:

#huma #go #api

TIL about fmt.Appendf

Mon, 24 Nov 2025 00:00:00 +0000

TIL about fmt.Appendf

Turn this:

request := []byte(fmt.Sprintf(`{"request_id":%d}`, rand.Int63()))

Into

request := fmt.Appendf(nil, `{"request_id":%d}`, rand.Int63())

I really like this; much simpler to type and quite clean in mu opinion.

Another benefit is reduced allocations. It allows us to extend a byte slice instead of generating new ones.

The source is quite easy to understand IMO

// ref: https://cs.opensource.google/go/go/+/master:src/fmt/print.go;l=247

// Appendf formats according to a format specifier, appends the result to the byte
// slice, and returns the updated slice.
func Appendf(b []byte, format string, a ...any) []byte {
	p := newPrinter()
	p.doPrintf(format, a)
	b = append(b, p.buf...)
	p.free()
	return b
}

Tags:

#go #til

Ziglings is an engaging way to learn zig basics

Sat, 08 Nov 2025 00:00:00 +0000

Ziglings is an engaging way to learn zig basics

I’m learning zig (enough to get by) and it’s an engaging and simple way to get started.

To learn a new language I’d typically find a blog or series of articles and follow along. This can be a little tedious because I like to type things out and run the code myself.

Often I’d sick of the tedium of writing the boilerplate, or find articles that miss all the context or supporting code the scaffolds the learning point.

Ziglings (like Rustlings, which I haven’t tried) instead sets up all of that for you in a single repo.

Each “zigling” is an intentionally broken program with comments on how to get it working. This is great because you can use whatever IDE you’re comfortable with and test things quickly.

I found a couple exercises didn’t explain clearly for me which is great because it forces me to research why. For me that improves my learning outcomes.

Anyway I’m 30+ ziglings in and the language has some cool features. So far I think my favourite is errdefer which is like defer but only executes on an error.

Snippet from the exercise:

fn makeNumber() MyErr!u32 {
 std.debug.print("Getting number...", .{});

 // Please make the "failed" message print ONLY if the makeNumber()
 // function exits with an error:
 errdefer std.debug.print("failed!\n", .{});

 var num = try getNumber(); // <-- This could fail!

 num = try increaseNumber(num); // <-- This could ALSO fail!

 std.debug.print("got {}. ", .{num});

 return num;
}

Outputs:

Getting number...got 5. Getting number...failed!

Whilst zig is potentially more low level than I typically need I am interested in learning it because of that. I need a reason to go deeper in my computer science fundamentals (I am self-taught).

After the exercises I think I’ll write a simple JWT decoder CLI - it’s something I use a lot in my $dayjob.

Tags:

#zig #learning

Claude Code update

Fri, 07 Nov 2025 00:00:00 +0000

Claude Code update

What cannot be understood will be rewritten - Bill Kennedy

Okay I’ve come around on this tool a bit.

Used correctly with intent and under firm guidance, it can be very powerful.

It’s still a massive idiot often enough that its brilliance becomes offset to the point where its untrustworthy.

I am not convinced anyone blindly accepting this code won’t be bitten in the arse badly in the future. It writes excellent legacy code.

That being said, I believe it is a massive help more often than not (when wielded correctly). My main use cases:

Rubber ducking
Architectural conversations
Writing code when I know exactly what I want, and ensuring it does exactly that
Questioning my assumptions/code (basically rubber ducky as well)

Where I really love it is writing throwaway code.

Like a lot of dev’s there is literally not enough time to do all the fun/interesting things you can think up. Writing code takes time and I have a life outside of my chair. But, with Claude I can speed run some of these and get it to write pure legacy code because I literally don’t give two hoots if it blows up later - it’s an experiment, fun, exploration, single-shot script etc

For that, Claude is amazing and worth the money.

As an example, I have a Brother thermal printer and hate that I have to plug it into a laptop/desktop to print stuff. So I decided I’d turn it into a always available service on my network. Plug it into a NUC/pi, write a flask server with a couple endpoints, use those to talk over pyusb and print the labels my wife needs. Tell claude to create the HTML forms and then use the forms submit data to the flask server. Put both on tailscale. Dockerise so I can run the flask server on my LAN (connected to printer), and let other services interact with it.

I could absolutely do all this without AI but it went from a weekends work to a couple hours (80% of that burnt just trying to get the printer to work because honestly f*ck printers bro).

So yeah, I’m warmed to Claude Code but its still just a tool to be used. My next big challenge is to learn zig without just relying on AI - if I can learn zig with the help of AI but not be so reliant on it that I don’t actually learn it, then I’ll chalk that up as a big win too.

Tags:

#claude #ai

restish is a great API CLI tool

Mon, 29 Sep 2025 00:00:00 +0000

restish is a great API CLI tool

I really like huma. I think every API no matter how small should aim to have a OpenAPI document. It future proofs you, and with tools like Huma or goa its built in.

In the Huma docs the creator recommends a tool called restish - of which he is also the author. I played with it but to be honest didn’t really explore well enough to “get” it. Now I do.

If you have a public OpenAPI spec/doc you can add it to a restish namespace. Doing this saves the spec and lets restish “see” all of it including its request and response bodies.

Heres an example of Synadia Clouds API (it’s long):

API for Synadia Cloud

Usage:
 restish prod [flags]
 restish prod [command]

Signing Key Groups Commands:
 copy-account-sk-group Copy Account SK Group
 create-account-sk-group Create Account Signing Key Group
 delete-account-sk Delete Account Signing
 delete-account-sk-group Delete Account Signing Key Group
 get-account-sk Get Account Signing
 get-account-sk-group Get Account Signing Key Group
 list-account-sk-group List Account Signing Key Groups
 list-account-sk-group-keys List Signing Keys
 rotate-account-sk Roate Active Signing Key
 update-account-sk Update Account Signing
 update-account-sk-group Update Account Signing Key Group

NATS Users Commands:
 assign-nats-user-team-app-user Assign Team App User to NATS User
 copy-nats-user Copy nats user
 create-user Create NATS User
 delete-nats-user Delete NATS User
 download-nats-user-bearer-jwt Get Bearer JWT
 download-nats-user-creds Get Creds
 download-nats-user-http-gw-token Get HTTP Gateway Token
 get-nats-user Get NATS User
 list-account-sk-group-users List NATS Users
 list-nats-user-connections List NATs User Connections
 list-nats-user-issuances List nats user issuances
 list-nats-user-team-app-users List Team App Users
 list-users List NATS Users
 rotate-nats-user Rotate nats user nkey and seed
 un-assign-nats-user-team-app-user Unassign Team App User from NATS User
 update-nats-user Update NATS User

Accounts Commands:
 create-account Create Account
 create-or-update-nats-user-revocation Create or Update Revocation for a NATS User NKey
 delete-account Delete Account
 delete-nats-user-revocation Delete a for a NATS User NKey
 get-account Get Account
 get-account-export Export Account Seeds
 get-account-info Get Account Info
 get-account-metrics Get Account Metrics
 get-nats-user-revocation Get Revocation for a NATS User NKey
 list-account-connections List Account Connections
 list-accounts List Accounts
 list-accounts-overview-metrics List Accounts overview metrics
 list-agent-tokens List Agent Tokens
 nats-core-websocket-viewer Subscribe to a NATS Core subject over websockets
 unmanage-account Unmanage Account
 update-account Update Account

Alerts Commands:
 acknowledge-alert Acknowledge Alert
 create-alert-rule Create Account Alert Rule
 create-system-alert-rule Create System Alert Rule
 delete-alert-rule Delete Account Alert Rule
 delete-system-alert-rule Delete System Alert Rule
 get-alert Get Alert
 get-alert-rule Get Account Alert Rule
 get-system-alert-rule Get System Alert Rule
 list-alert-rules List Account Alert Rules
 list-alerts List Alerts
 list-system-alert-rules List System Alert Rules
 run-alert-rule Run Account Alert Rule
 run-system-alert-rule Run System Alert Rule
 update-alert-rule Update Account Alert Rule
 update-system-alert-rule Update System Alert Rules

App Users Commands:
 assign-account-team-app-user Assign Team App User to Account
 assign-system-team-app-user Assign Team App User to System
 assign-team-app-user Assign App User to Team
 create-app-user Create App User
 delete-app-user Delete App User
 get-app-user Get App User
 get-current-agent-token Get Current Agent Token
 list-account-team-app-users List Account Team App Users
 list-app-user-roles List Roles
 list-app-users List App Users
 list-system-team-app-users List System Team App Users
 list-team-app-users List App Users
 un-assign-account-team-app-user Unassign Team App User from Account
 un-assign-system-team-app-user Unassign Team App User from System
 un-assign-team-app-user Unassign App User from Team
 update-app-user Update App User
 update-team-app-user Update App User Team Assignment

JetStream Commands:
 delete-pull-consumer Delete Pull Consumer
 delete-push-consumer Delete Push Consumer
 get-jet-stream-placement-options Get JetStream Placement Options
 get-pull-consumer-info Get Pull Consumer
 get-push-consumer-info Get Push Consumer
 list-jet-stream-assets List JetStream Assets
 update-pull-consumer Update Pull Consumer
 update-push-consumer Update Push Consumer

JetStream: KV Buckets Commands:
 create-kv-bucket Create KV Bucket
 create-kv-pull-consumer Create Pull Consumer
 create-kv-push-consumer Create Push Consumer
 delete-kv-bucket Delete KV Bucket
 get-kv-bucket Get KV Bucket
 list-kv-buckets List KV buckets
 list-kv-consumers List Consumers
 update-kv-bucket Update KV Bucket

JetStream: Mirrors Commands:
 create-mirror Create Mirror
 create-mirror-pull-consumer Create Pull Consumer
 create-mirror-push-consumer Create Push consumer
 delete-mirror Delete Mirror
 get-mirror Get Mirror
 list-mirror-consumers List Consumers
 list-mirrors List Mirrors
 update-mirror Update Mirror

JetStream: Object Buckets Commands:
 create-obj-pull-consumer Create Pull Consumer
 create-obj-push-consumer Create Push Consumer
 create-object-bucket Create Object Bucket
 delete-object-bucket Delete Object Bucket
 get-object-bucket Get Object Bucket
 list-obj-consumers List Consumers
 list-object-buckets List Object buckets
 update-object-bucket Update Object Bucket

JetStream: Streams Commands:
 create-pull-consumer Create Pull Consumer
 create-push-consumer Create Push Consumer
 create-stream Create Stream
 delete-stream Delete Stream
 get-stream-info Get Stream
 list-streams List Streams
 purge-kv-bucket Purge KV Bucket
 purge-mirror Purge Mirror
 purge-obj-bucket Purge Object Bucket
 purge-stream Purge Stream
 update-stream Update Stream

Sharing Commands:
 create-stream-export Create Stream Export
 create-stream-import Create Stream Import
 create-stream-shares Create Stream Shares
 create-subject-export Create Subject Export
 create-subject-import Create Subject Import
 create-subject-shares Create Subject Shares
 delete-stream-export Delete Stream Export
 delete-stream-import Delete Stream Import
 delete-stream-share Delete Stream Share
 delete-subject-export Delete Subject Export
 delete-subject-import Delete Subject Import
 delete-subject-share Delete Subject Share
 get-stream-export Get Stream Export
 get-stream-import Get Stream Import
 get-subject-export Get Subject Export
 get-subject-import Get Subject Import
 list-stream-exports List Stream Exports
 list-stream-exports-shared List Shared Stream Exports
 list-stream-imports List Stream Imports
 list-stream-shares List Stream Shares
 list-subject-exports List Subject Exports
 list-subject-exports-shared List Shared Subject Exports
 list-subject-imports List Subject Imports
 list-subject-shares List Subject Shares
 update-subject-export Update Subject Export
 update-subject-import Update Subject Import

Agent Tokens Commands:
 delete-agent-token Delete Agent Token

Auth Callout Commands:
 add-auth-callout-target-account Configure Target Account
 add-auth-callout-user Create Auth Callout User
 delete-auth-callout Delete Auth Callout Config
 delete-auth-callout-target-account Delete Target Account
 delete-auth-callout-user Delete Control Account User
 get-auth-callout Auth Callout Config
 list-auth-callout-authenticators Get List of Available Authenticators
 list-auth-callout-target-accounts Get Target Account List
 list-auth-callout-users Get Target Account List

Authorization Commands:
 check Check Authz Decisions
 list-policies Get Policy List
 list-roles Get Authz roles List

Component Versions Commands:
 get-available-component-versions Get Available Component Versions
 get-available-component-versions-by-type Get Available Component Versions By Type

Invitations Commands:
 decide-invitation Accept or reject team invitation
 invite-app-user Invite App Users
 list-invitations List of pending invitations

JetStream: Consumers Commands:
 list-consumers List Consumers

NATS User Issuances Commands:
 get-nats-user-issuance Get nats user issuance

Session Commands:
 accept-terms Accept terms
 create-app-user-access-token Create Personal Access Token
 delete-personal-access-token Delete Personal Access Token
 get-personal-access-token Get Personal Access Token
 get-version Get Version
 leave-team Leave Team
 list-app-user-access-tokens List Personal Access Tokens
 update-personal-access-token Update Personal Access Token

Platform Components Commands:
 platform-component-connect Connect a Platform Component

App Service Accounts Commands:
 create-app-service-account Create App Service Account
 create-app-service-account-token Create Acess Token for App Service Account
 delete-app-service-account Delete App Service Account
 delete-app-service-account-token Delete App Service Account Token
 get-app-service-account Get App Service Account
 get-app-service-account-token Get App Service Account Token
 list-app-service-account-tokens List Access Tokens for App Service Account
 list-app-service-accounts List App Service Accounts
 update-app-service-account Update App Service Account
 update-app-service-account-token Update App Service Account Token

Team Service Accounts Commands:
 create-team-service-account-token Create Acess Token for Team Service Account
 delete-team-service-account Delete Team Service Account
 delete-team-service-account-token Delete Team Service Account Token
 get-team-service-account Get Team Service Account
 get-team-service-account-token Get Team Service Account Token
 list-team-service-account-tokens List Access Tokens for Team Service Account
 update-team-service-account Update Team Service Account
 update-team-service-account-token Update Team Service Account Token

Systems Commands:
 bulk-share Share assets across accounts
 create-system Create System
 delete-system Delete System
 enable-auth-callout Enable Auth Callout For System
 get-component-token Get a component access token
 get-system Get System
 get-system-audit-job-result Get Audit Job Result
 get-system-audit-job-status Get Audit Job Status
 get-system-export Export System Seeds
 get-system-limits Get System Limits
 get-system-prometheus-metrics Get Prometheus Metrics
 import-account Import Account
 import-user Import User
 list-auth-callout-configs List Auth Callout Configs
 list-clusters List Clusters
 list-connections List Connections
 list-servers List Servers
 list-system-audit-jobs List Audit Jobs
 list-system-info-accounts List System Accounts Info
 list-system-info-servers List System Servers info
 list-team-systems List Systems
 rotate-agent-token Rotate Agent Token
 run-system-audit-check Run System Audit Check
 system-jwt-sync Re-sync JWTs of all accounts in this system
 unmanage-system Unmanage System
 update-platform-components Update Platform Components for System
 update-system Update System

Teams Commands:
 create-team Create Team
 create-team-service-account Create Team Service Account
 delete-team Delete Team
 get-team Get Team
 get-team-limits Get Team Limits
 import-system Import a System
 list-team-info-app-users List info of App Users in Team
 list-team-service-accounts List Team Service Accounts
 list-teams List Teams
 update-team Update Team

Workloads Commands:
 get-workload-limits Get Workloads limits

Yep, that is every single endpoint from the doc.

If I want to list all consumers but don’t remember what the request body is?

Just use restish <namespace> list-consumers and it’ll print this.

List Consumers
## Argument Schema:

{
 stream-id: (string)
}


## Response 200 (application/json)

Success


{
 items*: [
 allOf{
 {
 ack_floor*: {
 consumer_seq*: (integer min:0 max:1.8446744073709552e+19)
 last_active: (string nullable:true)
 stream_seq*: (integer min:0 max:1.8446744073709552e+19)
 }
 cluster: allOf{
 {
 leader: (string)
 name: (string)
 replicas: [
 allOf{
 {
 active*: (integer format:int64)
 current*: (boolean)
 lag: (integer min:0 max:1.8446744073709552e+19)
 name*: (string)
 offline: (boolean)
 }
 }
 ]
 }
 }
 config*: {
 ack_policy*: (string enum:none,all,explicit) enums have been changed to match UnmarshalJSON in https://github.com/nats-io/jsm.go/blob/main/api/consumers.go
 ack_wait: (integer format:int64)
 backoff: [
 (integer format:int64)
 ]
 deliver_group: (string)
 deliver_policy*: (string enum:all,last,new,by_start_sequence,by_start_time,last_per_subject)
 deliver_subject: (string)
 description: (string)
 direct: (boolean) Don't add to general clients.
 durable_name: (string)
 filter_subject: (string)
 filter_subjects: [
 (string)
 ]
 flow_control: (boolean)
 headers_only: (boolean)
 idle_heartbeat: (integer format:int64)
 inactive_threshold: (integer format:int64)
 max_ack_pending: (integer)
 max_batch: (integer)
 max_bytes: (integer)
 max_deliver: (integer)
 max_expires: (integer format:int64)
 max_waiting: (integer)
 mem_storage: (boolean)
 metadata: {
 <any>: (string)
 }
 name: (string)
 num_replicas*: (integer)
 opt_start_seq: (integer min:0 max:1.8446744073709552e+19)
 opt_start_time: (string nullable:true)
 rate_limit_bps: (integer min:0 max:1.8446744073709552e+19)
 replay_policy*: (string enum:instant,original)
 sample_freq: (string)
 }
 created*: (string format:date-time)
 delivered*: {
 consumer_seq*: (integer min:0 max:1.8446744073709552e+19)
 last_active: (string nullable:true)
 stream_seq*: (integer min:0 max:1.8446744073709552e+19)
 }
 name*: (string)
 num_ack_pending*: (integer)
 num_pending*: (integer min:0 max:1.8446744073709552e+19)
 num_redelivered*: (integer)
 num_waiting*: (integer)
 push_bound: (boolean)
 stream_name*: (string)
 ts*: (string format:date-time)
 }
 }
 ]
}


## Responses 400/401/403/404
# truncated...

To me this makes using API’s using a CLI actually possible without consulting the docs.

Walkthrough

Let’s walkthrough how to set all this up by using the restish example API.

restish api configure example
# next steps are interactive
restish api configure example https://api.rest.sh
> ? Select option Save and exit

We’ve told restish to associate https://api.rest.sh with the namespace of example. This is how we can segresgate many different API’s from one another.

Now we can auto detect what endpoints exist.

restish example

Which returns,

Usage:
 restish example [flags]
 restish example [command]

Echo Commands:
 delete-echo 
 get-echo 
 patch-echo 
 post-echo 
 put-echo 

Books Commands:
 delete-book 
 get-book 
 list-books 
 patch-book Patch book
 put-book 

Caching Commands:
 get-cached 

Example Commands:
 get-example 

Images Commands:
 get-image 
 list-images 

Status Commands:
 get-status 

Types Commands:
 get-types-example 
 put-types-example

I want to put-book but don’t know what I need to provide in the request body.

# restish example put-book -h
# truncating some output for brevity

## Argument Schema:

{
 book-id: (string)
}

## Option Schema:

{
 --if-match: [
 (string)
 ]
 --if-none-match: [
 (string)
 ]
 --if-modified-since: (string format:date-time-http) Succeeds if the server's
resource date is more recent than the passed date.
 --if-unmodified-since: (string format:date-time-http) Succeeds if the server's
resource date is older or the same as the passed date.
}

## Input Example

{
 "author": "string",
 "published": "2020-05-14T23:44:51-07:00",
 "rating_average": 1,
 "ratings": 1,
 "recent_ratings": [
 {
 "date": "2020-05-14T23:44:51-07:00",
 "rating": 1
 }
 ],
 "title": "string"
}

Now that we know what the request requires we can put a book into the collection

 echo '{
 "author": "Someone",
 "published": "1900-05-14T00:00:00-12:00",
 "rating_average": 4,
 "ratings": 30,
 "title": "My book is good"
 }' | restish example put-book good-book

# returns HTTP/2.0 204 No Content

And restish example list-books shows it in the collection

[
 {
 modified: 2025-09-29T03:36:11.74478507Z
 url: "/books/the-demon-haunted-world"
 version: "yWWo/Pq0gMgjMNgWw+K+Wg=="
 }
 {
 modified: 2025-09-29T03:36:11.74478507Z
 url: "/books/the-fabric-of-the-cosmos"
 version: "qhTFCaoJg92NnpRTMJzsEA=="
 }
 {
 modified: 2025-09-29T03:42:34.560280084Z
 url: "/books/good-book"
 version: "3Bvek8xG1BB3i+g4kvDzFg=="
 }
]

Of course you can also just use it like you would curl, here’s an example:

 restish api.github.com/users/rs | jq '{name, company, location, login, bio}'
> {
> "name": "Olivier Poitrey",
> "company": "Netflix",
> "location": "Silicon Valley, California, USA",
> "login": "rs",
> "bio": "Director of Engineering at Netflix\r\nCo-Founder & ex-CTO of Dailymotion\r\nCo-Founder of NextDNS"
> }

There is a lot more to cover but thats some of the parts I really like.

restish is a tool that makes it much easier to stay in the terminal whilst interacting with a unfamiliar or complex API without needing the docs.

Tags:

#restish #api

TIL about Go's json.Indent and when to use it

Fri, 19 Sep 2025 00:00:00 +0000

TIL about Go’s json.Indent and when to use it

I needed to json.Unmarshal then json.MarshalIndent a large amount of JSON. In this was a set of very large numbers.

I learnt that this causes precision issues because unmarshalling to an interface{} will use float64 for numbers.

This caused a lot of head scratching, and all I can say is thank god for the debugger! Those who “don’t use debuggers”, well, I tip my hat to you because print debugging would not have go me through this pickle.

Example JSON

"account_details": [
 {
 "reserved_memory": 18446744073709551615,
 "reserved_storage": 18446744073709551615
 }
]

Was raising:

decode 0001.json: json: cannot unmarshal number 18446744073709552000 into Go struct field Foo.account_details.Bar.reserved_memory of type uint64

To achieve my aims and keep the numbers exactly the same, json.Indent can be used.

My takeaway:

json.Indent is for pure text manipulation
json.MarshalIndent is for parsing

I made a quick https://go.dev/play/p/qFKb23mUYuz link showing it in action.

Tags:

#go #json

ROG PG278Q Won't start - fix

Wed, 17 Sep 2025 00:00:00 +0000

ROG PG278Q Won’t start - fix

This monitor is a POS. Whenever we get a power outage this thing has issues starting back up.

I think the capacitors are faulty and the only way I’ve been able to semi-reliably get it back up and going is to turn it off for 30+ seconds. This seems to discharge the capacitors and then it will start.

It happens infrequently enough that I forget what to do to fix it.

Still, it doesn’t fix the issue where this monitor loses connectivity with my Dell dock requiring me to pull the cable out and switch display ports. That happens 3-6x a day.

I’ll never buy another ASUS monitor again.

Tags:

#asus #rog #rant

Remarkable 3.22 SSH

Tue, 16 Sep 2025 00:00:00 +0000

Remarkable 3.22 SSH

I finally got access to 3.22 which supports handwriting search! This is great because I only use it for hand writing.

It also allows the (for lack of proper wording) side panel to be snapped to the top bar instead. This unlocks a little extra writing room on the edges as I like to have the panel always visible.

Annoyingly, 3.22 brings a new security feature; SSH disabled over Wifi by default.

To enable it you have to plug it into a internet capable device (e.g. laptop), wait for it to get a new IP address and the SSH into it. Once in, run rm-ssh-over-wlan on. SSH over Wifi will now be enabled.

Tags:

#remarkable

Disable Firefox AI summary

Thu, 04 Sep 2025 00:00:00 +0000

Disable Firefox AI summary

I don’t want your summary or crappy AI force fed into my browser.

To disable, navigate to about:config and paste in browser.ml.chat.enabled. Set it to false.

Fixed.

I’m getting AI fatigue.

Tags:

#ai #firefox

Working across timezones

Wed, 03 Sep 2025 00:00:00 +0000

Working across timezones

This week I’ve had meetings across several timezones, delivered a demo at 0200 (my time) and still got heaps of work done.

In the lead up to the demo I’ve been chatting with colleagues in Europe and the US. Being part of a distributed global team means some times out of hours calls are required.

When I explain that I’m talking to someone at 2100 or delivering a demo at 0200 they think two things; I’m crazy and that I have no “work life balance”.

I think they’re wrong.

For me, it works. I can hand off to others, or wake up to find x project is no longer blocked freeing me up to start working on it immediately.

As for the late night calls, they aren’t so bad. I have a lot of experience working long hours at odd times often cold, wet and hungry.

Going to bed at 2200, waking up at 0145 to deliver a demo from the comfort of my warm house, then going back to my comfy bed.. that is a privilege!

That is life on easy mode. Some times, perspective is all you need.

Being woken for picket at 0200 after only getting to bed at 0000, doing an hour on watch, then waking again at 0500 before first light - only then to do another full day’s patrolling. That’s hard! Software development is easy in comparison.

An easy, enjoyable trade-off. I love this career!

Tags:

#rant

Uses

Sun, 17 Aug 2025 00:00:00 +0000

This is a list of software, tools, and stuff that I use on a regular basis.

Development

Editors

GoLand - I write a lot of Go
PyCharm - I used to write a lot of Python
Webstorm - I write typescript because I have to
Zed for light scripting/reading code
NeoVim or just Vim for everything else

Terminal & Shell

Ghostty - For my Mac
WezTerm - On Linux
ddterm - Dropdown terminal for linux
Zsh with Oh My Zsh - Shell with useful plugins
Starship cross-shell prompt
zellij a better TMUX experience

Development Tools

task a better makefile. Use it in every project.
chezmoi for managing dotfiles, scripts and more across machines backed by git
arkade easily install kubernetes tools (useful more so on Linux as brew is pretty good)
curlie I alias this to curl as its a dropline replacement with better defaults
lazygit for when I’m not using JetBrains in built git tools
k9s; I can’t use kubectl anymore

System & Productivity

Operating System

MacOS - Macbook M4 Pro
- Only recently started using this. Mac’s are a lot better since my last one in around 2015-16 ish.
Ubuntu - Linux desktop
- Simple, just works. Containers as a native construct is far superior to all MacOS workarounds.

Productivity and General Apps

Obsidian - Note-taking and knowledge management. Pay for cloud sync.
Remarkable2 - All my rough work, notes, sketches start in here before transcribing to Obsidian. Pay for cloud.
Zet - My personal zettlekasten CLI which gets published to this site
raycast if you aren’t using this on Mac what is wrong with you! Its amazing
bitwarden Password Vault. I pay for premium at $10 USD a year; it’s a steal

Hosting & Services

Hosting

Hetzner Any time I need a VPS that doesn’t need to be in my geo
Coolify Self-hosted application platform
Proxmox - VM hypervisor for my home lab

Networking

pfSense firewall
unifi I use their switches and access points. They’re the Apple of networking

Last updated: August 2025

Ghostty quick-terminal makes Mac terminals useful again

Sat, 26 Jul 2025 00:00:00 +0000

Ghostty quick-terminal makes Mac terminals useful again

A while back I whinged hard about terminals on Mac. At the time I was unaware, or it didn’t exist - not sure - of Ghostty’s “quick terminal”.

This for my interests works exactly the same as how I’ve setup iTerm to dropdown using F12. Which is/was the only way to achieve what many linux terminals offer out of the box. Yakuke, Guake and ddterm being examples of this.

I think now I can safely transition from iTerm and WezTerm (still love the lua configs..) and make Ghostty my daily.

Some references:

Note, I’ve not tested quick term on linux. I have read conflicting reports; x11 is okay but wayland might not be supported. Honestly, I think it’ll eventually get support there but it would be a real shame if it doesn’t get implemented. Still, linux has many great offerings and I largely use Zellij inside Ghostty anyway which works in Guake/Yakuke/ddterm as-is

Tags:

#ghostty #terminal

Claude Code findings

Wed, 09 Jul 2025 00:00:00 +0000

Claude Code findings

After using it for a few days, some initial thoughts and observations.

It uses *.bak files despite having git. Obviously it’s learnt this practice from us, failed humans. Often it forgets to clean them up.
Sometimes you really have to fight it to get it to do what you want no matter how much “planning” or “context engineering” you do.
- E.g. using NATS JetStream legacy over the new standard. Several times it reverted from “new” to “legacy”. In the end I took over and fixed it after much wasted tokens.
AI is really a “shovels and picks” moment IMO - the model vendors are absolutely killing it with things like claude code. Forget MAX they are chasing enterprise pay-as-you-go. I, rather cynically, believe it’s possible they don’t want models that are too smart else they’ll lose money on credits; akin to why we’ll never get a cure for cancer - no money in it.
It’s insanely good for scaffolding projects but mediocre at filling in the complex centre of most big projects.
Very easy to lose your way if you let it auto-edit - I can’t use it for work and hand on heart I fully understand it; actually better off writing the code myself and asking AI questions about my intent rather than letting it take the wheel (so no change from status quo for me).
Reviewing code is boring and you’ll do A LOT of that using these tools.
Asking questions, clarifying how things work and talking through your thought process are amazing quality of life improvements from these tools.
It’s pretty average at logging what it’s done in any markdown file; often does the work then I have to request it updates documents.
Super easy to get lazy.
Voice prompting is really cool because I can ramble on and it’ll understand me - sometimes when I start talking I realise my question/prompt is silly or I refine it more than I would just typing alone. Rubber ducky, if you will.

Now, could I do better at “context engineering”? Most likely. Do I think throwing shade on anyone who has anything negative to say via a “probably not prompting it right” is a very bad pattern, yes I do. IMO, it’s another form of character assassination we’ve seen time and time again in the last few years; “trust the science”, “right winger”, “MAGA” and so on as a quick way to dismiss someone who is not “on your side”. So yeah, I think we do ourselves an injustice here by immediately dismissing valuable feedback by labeling someone by virtue of “not prompting correctly”! Bit of a philosophical rant but I can’t help but feel it in this current era of AI.

Regardless, I can absolutely see how these tools are a real boon for developers. Do I think running 6 agents at the same time on a single codebase is going to yield amazing results - no I don’t, at least not long term.

Tags:

#ai #claude #rant

NATS, ADS-B and my blog

Sun, 06 Jul 2025 00:00:00 +0000

NATS, ADS-B and my blog

With the help of Claude Code, I (mostly) “vibed” a new page on my static Hugo blog.

The page connects to a server over websockets and that server is connected to Synadia Cloud over NATS. At my house I have a Raspberry Pi 2 with an RTL-SDR configured to listen for ADS-B beacons 1090Mhz.

The program running on the Pi is called dump1090 (created by the Antirez who wrote Redis btw). It outputs raw AVR data onto port 30003 which is what I collect.

So I wrote Go monolith with two entrypoints; agent and server. The agent is installed/ran on a sensor (rpi2) and its configured to listen on TCP 30003 for the raw data. It also connects to a NATS server and then publishes the data to NATS. The agent uses core NATS so its fire and forget - no persistence. I set it up to reconnect using a backoff mechanism incase my local network or its wifi dongle goes down. It’s connected to my IOT VLAN which is firewalled from the rest of the network but because it can make outbound requests, such as a NATS server, we get around all the complexities of firewalls, network address translation etc. Go NATS!

The server is also connected to NATS - though it can be configured to run as an embedded NATS server. It’s subscribing to the subjects that contain the 1090 raw data. It then does some parsing and munging to turn this into useful info. ADS-B packets are transmit frequently and have a structure but not every packet needs to fill all the “fields” - you can get packets with a mix match of these fields and we have to construct them in order to build out useful data structures. Once each “packet” is rolled up into a valid go struct we save that into a NATS JetStream Key Value store. The aircrafts ICAO is the key and it has a TTL of 30 minutes.

A websocket is made available on the server which serves as a proxy for its NATS connection. Messages can be sent up the socket and as long as they match the whitelist they’ll be sent to NATS and the reply returned down the socket to the caller. I found this to be the safest way to do it in a static site which cannot hide secrets.

I’ve also created a couple of NATS micro endpoints which allow users to lookup aircraft via their callsign or ICAO number. Another great feature of NATS.

Access to this is being exposed (and here’s where Claude came in clutch) on my Hugo blog at https://danielms.site/nats-adsb. It uses a Hugo shortcode which then contains a lot of plain javascript to connect to the servers websocket and render the responses on the page.

On the site you can “watch” all aircraft in real-time (as long as their are planes fly near my house). And lookup their callsign/ICAO number in the table, or do a search of any planes data via a form field. I’m leveraging a free ADSB API in the server to do these requests over NATS micro - the API sometimes has issues but, hey, its free.

Tags:

#nats #websockets #hugo

NATS ADS-B

Thu, 03 Jul 2025 00:00:00 +0000

What you are seeing here is live ADS-B data from a sensor located at my house. It is connected to a server which then connects to Synadia Cloud. The server proxies data to and from NATS over a websocket.

Usage

Clicking on Watch Aircraft will start a live tail of aircraft seen in the last 30 minutes. There may be times when no aircraft are flying due to noise, typically 0000-0500 AEST.

There are two function that can be made to look up aircraft by their Callsign or mode s number. These make use of NATS services/micro framework.

Disconnected

Messages Received

Active Aircraft

Errors

Ollama on the plane

Wed, 02 Jul 2025 00:00:00 +0000

Ollama on the plane

I was doing some work offline on a flight today and (sadly) reached for AI only to realise my usual wasn’t available whilst offline.

Then I remembered I setup ollama with qwen3 the other day. Well it worked really well in helping me figure out how to use a library. I haven’t used ollama in a long time and found the local models quite inferior to online but qwen3 was really quite good.

I think having access to an offline model is a real boon for productivity, especially when you just want to consult docs and ask questions about simple stuff like that.

Tags:

#ai #ollama

NATS and Fly Gateway proxy

Tue, 24 Jun 2025 00:00:00 +0000

NATS and Fly Gateway proxy

Woodpecker CI, NATS and Fly.io

This was posted in my work chat today. It’s a simple but easy to understand and briefly touches on some of the power of NATS.

NATS small footprint, and ability to use leaf nodes really blows up the possibilities for the tool. The fact that John is leveraging Synadia Cloud with his home leaf node is really powerful. I love reading about how people are using these technologies and platforms to build what they want rather than shoehorning ideas into the requirements and limitations of other tools.

Also, echo’s my distain for GitHub Actions. I think Dagger is the future of CI these days but Woodpecker/Drone is nice too.

Tags:

#nats #synadia #cicd

Go vendor for debugging dependencies

Sun, 22 Jun 2025 00:00:00 +0000

Go vendor for debugging dependencies

TIL how to debug dependencies in Go easily.

One of my deps have a client timeout of 2 * time.Second which meant debugging the connections was nearly impossible.

I ran go mod vendor and I was then able to make changes to the vendored code. I increased the timeout to something reasonable and was able to find the issue.

I could use the replace directive in my go.mod and clone down the repo but that seemed like a lot of work. If vendoring didn’t work it would of been my next step but vendor is perfect for simple changes like this.

Tags:

#go

Github UI needs vimium extension to be useful

Thu, 19 Jun 2025 00:00:00 +0000

Github UI needs vimium extension to be useful

<rant>

Long issues/PR’s etc in Github UI are a real PITA to navigate around. For instance, if I want to see the status of the branches actions I have to go all the way to the bottom of the page.

If you don’t have vimium installed I don’t know what you do here except scroll like a maniac (I’m sure there’s other hotkeys). <Shift>-G is a saviour here.

I think GitLab’s experience is far superior here. Firstly, the important stuff is at the top of the fold; Merge and pipeline statuses. Secondly, all comments can be nested in GitLab; something thats not possible in Github. Having to create another entry on the page just to reply to a question (which could be several comments back) is really bad UX.

I feel like Github exists this way solely to be different to GitLab and because its the incumbent and they have zero incentive to change. Possibly, a lot of developers don’t know there’s a better experience waiting for them as they’ve never used GitLab properly.

</rant>

Tags:

#rant #github #gitlab

Went to my local SecTalks today

Tue, 03 Jun 2025 00:00:00 +0000

Went to my local SecTalks today

In an effort to be more involved in the tech community I ventured off the local SecTalks chapter for a talk on application security.

The talk was pretty good, covered a lot of ground from basic defensive programming to supply chain management. The target audience was not software developers but security folk and I think Lochie did a good job from that regard.

In my opinion, security engineers sometimes forget software engineering is just as vast and difficult as their trade. For red-team types, security failures only need to happen once to achieve their intent whereas application defence is a constant and ongoing process. This can lead to a misalignment of realities between the two professions and animosity from my experience. Throwing shade is always so easy!

The speakers intent was to help remind everyone that we, application/software developers, do try our best, for the most part, to create secure products. I think he did that.

I should probably get up and give a software related talk one day though I’d have to find a topic that’s suitable for the SecTalk crowd. Maybe security scanning at scale with NATS as the backbone! I was scheming up some ideas in the drive home; probably a network scanner that hits bug bounty targets, collects results stores in KV then aggregates data into an ObjectStore for historical analysis or something like that.

Tags:

#sectalks #meetup

Go: errors.As versus type assertions

Mon, 26 May 2025 00:00:00 +0000

Go: errors.As versus type assertions

I prefer to use error unwrapping via errors.As and here is my reasoning.

Examples

Type assertion:

// fails if error is wrapped
if newErr, ok := err.(*new.Error); ok {
 // only works for direct new.Error instances
}

Error unwrapping

// works with wrapped errors like fmt.Errorf("foo: %w", newErr)
var newErr *new.Error
if errors.As(err, &newErr) {
 // finds new.Error even if wrapped
}

Reasoning

Introduced in 1.13 alongside errors.Is and fmt.Errorf("%w",err) to fix type assertion issues.
Type assertion is brittle (I personally try to avoid it when possible, sometimes unavoidable).
Recommended by Go team, https://go.dev/blog/go1.13-errors
If, in future, we decided to wrap this error, a type assertion will break; errors.As will still work.

Realworld Example

I use this (sometimes) in my projects to enforce some sane error handling when processing JSON. Standard json errors can be hard to decern the true issue.

This block uses several error handling techniques where they make sense with heavy use of errors.As throughout. Much of this was taken from Alex Edwards who provides many great resources and tools for Gophers.

func readJSON(w http.ResponseWriter, r *http.Request, dst interface{}) error {
	// Set a max body length. Without this it will accept unlimited size requests
	maxBytes := 1_048_576 // 1MB
	r.Body = http.MaxBytesReader(w, r.Body, int64(maxBytes))

	// Init a Decoder and call DisallowUnknownFields() on it before decoding.
	// This means that JSON from the client will be rejected if it contains keys
	// which do not match the target destination struct. If not implemented,
	// the decoder will silently drop unknown fields - this will raise an error instead.
	dec := json.NewDecoder(r.Body)
	dec.DisallowUnknownFields()

	// decode the request body into the target struct/destination
	err := dec.Decode(dst)
	if err != nil {
		// start triaging the various JSON related errors
		var syntaxError *json.SyntaxError
		var unmarshallTypeError *json.UnmarshalTypeError
		var invalidUnmarshallError *json.InvalidUnmarshalError

		switch {
		// Use the errors.As() function to check whether the error has the
		// *json.SyntaxError. If it does, then return a user-readable error
		// message including the location of the problem
		case errors.As(err, &syntaxError):
			return fmt.Errorf(
				"body contains badly-formed JSON (at character %d)",
				syntaxError.Offset,
			)

		// Decode() can also return an io.ErrUnexpectedEOF for JSON syntax errors. This is
		// checked for with errors.Is() and returns a generic error message to the client.
		case errors.Is(err, io.ErrUnexpectedEOF):
			return errors.New("body contains badly-formed JSON")

		// Wrong JSON types will return an error when they do not match the target destination
		// struct.
		case errors.As(err, &unmarshallTypeError):
			if unmarshallTypeError.Field != "" {
				return fmt.Errorf(
					"body contains incorrect JSON type for field %q",
					unmarshallTypeError.Field,
				)
			}
			return fmt.Errorf(
				"body contains incorrect JSON type (at character %d)",
				unmarshallTypeError.Offset,
			)

		// An EOF error will be returned by Decode() if the request body is empty. Use errors.Is()
		// to check for this and return a human-readable error message
		case errors.Is(err, io.EOF):
			return errors.New("body must not be empty")

		// If JSON contains a field which cannot be mapped to the target destination
		// then Decode will return an error message in the format "json: unknown field "<name>""
		// We check for this, extract the field name and interpolate it into an error
		// which is returned to the client
		case strings.HasPrefix(err.Error(), "json: unknown field "):
			fieldName := strings.TrimPrefix(err.Error(), "json: unknown field ")
			return fmt.Errorf("body contains unknown key %s", fieldName)

		// If the request body exceeds maxBytes the decode will fail with a
		// "http: request body too large".
		case err.Error() == "http: request body too large":
			return fmt.Errorf("body must not be larger than %d bytes", maxBytes)

		// A json.InvalidUnmarshallError will be returned if we pass a non-nil pointer
		// to Decode(). We catch and panic, rather than return an error.
		case errors.As(err, &invalidUnmarshallError):
			panic(err)

		// All else fails, return an error as-is
		default:
			return err
		}
	}

	// Call Decode() again, using a pointer to anonymous empty struct as the
	// destination. If the body only has one JSON value then an io.EOF error
	// will be returned. If there is anything else, extra data has been sent
	// and we craft a custom error message back to the client
	err = dec.Decode(&struct{}{})
	if err != io.EOF {
		return errors.New("body must only contain a single JSON value")
	}
	return nil
}

Tags:

#go

NATS Micro example

Tue, 20 May 2025 00:00:00 +0000

NATS Micro example

NATS Micro is a super simple way to create microservices using NATS as the communication layer.

We’re using them a lot at Synadia internally but I’d never used them before.

It only takes about 5 minutes to go from an empty file to running service that is handling load. It’s really impressive.

Over the weekend I wrote a small (imperfect) example application to learn enough to get started. It fetches the top posts from HackerNews over the last 24 hours, saves them to NATS KV, sorts by points, saves that days sorted posts to Object Store and exposes a service that prints the top n posts.

Once the service is running you can inspect most things with just three commands:

nats micro ls
nats micro info HackerNews - HackerNews being the service I’m running
nats micro stats HackerNews

One thing I did come across was mixing of publish and request semantics. In one of my endpoints I am publishing items as they come in. This allows the consumer to execute on each item instead of a large list - typical work queue behaviour. But, if I nc.Publish inside of a req and don’t respond it will cause the service to report an error. This is more about how I structured the code, and it’s easy to fix. I just created an two endpoints; one for publishing/subscribing and one that will respond. The endpoint which only exists to pub/sub could probably be excluded from the services framework but I included it because it gives great discoverability and statistics out of the box.

Going forward I see myself using nats micro/nats service a lot. It’s such a simple, yet powerful construct around NATS.

One thing I thought of was how this could be a OpenFaaS-like replacement, which also uses NATS, where you could bootstrap “functions” and do it all in micro. It would be easy to expose a single HTTP gateway like OpenFaaS’s which proxies/transforms the data into NATS protocol. These things kind of already exist but not quite the same.

References:

Tags:

#nats #micro #services #openfaas

Projects

Mon, 12 May 2025 00:00:00 +0000

check-redirects.com

A web application to help find out where your short links are taking you. Entering a URL will return each hop’s status code, response headers, and the entire redirect chain.

Written in Go, with react and sqlite embedded using PocketBase.

tars.run

A dead simple, hobbyist, no guarantees URL shortener. Written in Go, rendered with Next.js and persisted with SQLite through the magic of Litestream.

RFC.py

A simple python client that offers users the ability to search for, read and bookmark RFC’s from the Internet Engineering Task Force whilst offline.

http-tracer

A python CLI tool for ascertaining the redirection path taken when accessing a given URL.

Mudmap.io

Archived

Cloud monitoring and configuration for pfSense firewalls. Secure, safe and easy to configure and use. Take the hassle out of multisite firewall maintenance.

DBeaver wordwrap and autoformat by default

Thu, 08 May 2025 00:00:00 +0000

DBeaver wordwrap and autoformat by default

Make the values panel auto-format and wordwrap by default. This requires changes to the config files in DBeaver’s workdirectory.

Make sure DBeaver is closed before doing this.

Edit these files:

workdirectory/workspace6/.metadata/.plugins/org.eclipse.core.runtime/.settings/org.eclipse.ui.workbench.prefs

# add these to bottom of file 
lineNumberRuler=true
wordwrap.enabled=true

workdirectory/workspace6/.metadata/.plugins/org.jkiss.dbeaver.ui.editors.data/dialog_settings.xml

<?xml version="1.0" encoding="UTF-8"?>
<section name="Workbench">
 <section name="AbstractTextPanelEditor">
 # these two lines
 <item key="content.text.editor.auto-format" value="true"/>
 <item key="content.text.editor.word-wrap" value="true"/>
 </section>
</section>

ref: https://superuser.com/a/1827163/862055

Tags:

#sql

iterm2 dropdown tips

Mon, 05 May 2025 00:00:00 +0000

Setting up dropdown like ddterm, guake or yakuke.

This explains it well with pictures:

https://dev.to/vikbert/drop-down-iterm2-in-macos-2od

To allow it to dropdown into the “desktop” you currently have focused:

Profiles -> $profile_name -> Keys -> Configure Hotkey WindowFloating Window -> Floating Window -> restart iTerm.
ref: https://gitlab.com/gnachman/iterm2/-/issues/10695#note_1267078045,

Prevent it from disappearing when you change focus:

https://superuser.com/questions/1100918/how-can-i-prevent-the-iterm2-hotkey-window-from-losing-focus

Tags:

#macos #iterm

OpenFaaS golang-middleware go.work fix

Mon, 21 Apr 2025 00:00:00 +0000

OpenFaaS golang-middleware go.work fix

This is an annoying problem where golang-middleware has a go.work referencing go 1.23 but I’m using 1.24 and it want versioning like 1.23.0

| 2025/04/21 12:11:59 stdout: go: module function listed in go.work file requires go >= 1.23.0, but go.work lists go 1.23; to update it:
| 2025/04/21 12:11:59 stdout: go work use
| 2025/04/21 12:11:59 stdout: failed to build, error: exit status 1

To fix this I had to go into the template/golang-middleware/go.work file and change 1.23 to 1.23.0 or it won’t build locally.

This is janky but it works for now.

Note: I use docker compose for local development with openfaas but even using faas-cli local-run this issue occurs. In fact, my fix didn’t fix this issue using local-run.

Tags:

#openfaas

Homeassistant MQTT (and NATS)

Thu, 17 Apr 2025 00:00:00 +0000

Homeassistant MQTT (and NATS)

I have a couple of automations (pool pump being most important) that run over Zigbee2MQTT.

I was using the Homeassistant built in MQTT broker but decided to test out NATS MQTT implementation and see if it was compatible. It works a treat!

However, when doing this to make it work I didn’t realise that I need to update the Z2M configuration and point it at the NATS cluster and reconfigure HA’s MQTT service to use it as well.

When I triggered things from the HA Z2M panel everything worked as expected and I could see messages being routed but my automations wouldn’t work.

The fix is Settings -> Devices & services -> MQTT -> Reconfigure then change the broker address. Once I did that my automations started working again.

Tags:

#mqtt #nats #Homeassistant

Coolify basic auth

Sun, 13 Apr 2025 00:00:00 +0000

Coolify basic auth

Setting basic auth on Coolify requires editing the traefik labels.

I’ve only needed to do it in docker compose files. Heres my snippet:

 labels:
 - 'traefik.http.middlewares.$UNIQUE_TO_PROJECT.basicauth.users=$USERNAME:$PASSWORD_GENERATED_BY_HTPASSWD'

Link to docs

Tags:

#coolify #traefik

My Thinkpad is dead, back on the (old) macbook

Mon, 07 Apr 2025 00:00:00 +0000

My Thinkpad is dead, back on the (old) macbook

My trusty Thinkpad is finally so slow that I’ve calling it dead.

This is bringing great shame to me as I know am resorting to using my partners old Mac.

I think I’ve been a linux master race user since leaving mac around 2017-2018. Despite mac being based on linux it still sucks to use.

No docker (docker destop/orbstack/colima are all workarounds), inexplicable usage of the “Command” key in the “Alt” key position, weird security prompts requiring clicky clicky.

But, I do get it, people enjoy this because some things “just work”. I mean they work because millions and millions are poured into its development.

Now I need to figure out how to manage Chezmoi with two different operating systems. Should I keep using mise, or just leverage brew. I feel like the answers to these questions are only a footgun away!

On a good note, rebuilding this machine showed me that my zet tool no longer builds as an underlying package has been deleted upstream. I’m guessing its cached on my other machines so I didn’t notice - some learnings in there. I rewrote the tool using https://github.com/alecthomas/kong over the weekend to get it back online.

https://github.com/danielmichaels/zet-cmd

Edit: adding apps that have similar ergonomics on Mac as they do linux:

Terminals

Dropdown:

iterm2 is (so far) the only terminal I can configure to have dropdown like functionality similar to guake or ddterm on Linux. This really breaks my flow. I have f12 mapped to dropdown a terminal meaning I can keep my IDE/Browser whatever in view whilst I have terminal access. Iterm2 supports this BUT if you use mac’s desktop’s it’ll yank you from a fullscreen in desktop 3 back to desktop 1 if thats where you “main” desktop is. Iterm won’t dropdown into the current desktop - only the “main”, typically desktop 1. I’m sure theres a mac way of explaining this but its a massively negative experience for my flow.

Edit: I think I’ve fixed this by doing this, which is to set Profiles -> <profile> -> Keys -> Configure Hotkey WindowFloating Window -> Floating Window then restart iTerm. Works well so far..

Terminal:

wezterm and Iterm2 so far are the only terminals that support the Zellij option key (alt rebinding).

Terminals that do not support my workflow:

ghostty: doesn’t work well with Zellij, no alt key alternative
alacrity: as above
kitty: as above

Tags:

#mac #linux

Dagger CI is best CI

Sun, 23 Mar 2025 00:00:00 +0000

Dagger CI is best CI

I’ve switched to using dagger for CI and couldn’t be happier.

Without pushing code to the origin I can run my entire suite of CI tools locally. This means I know everything will run when I push and CI will pass should I want to know.

More importantly, I can debug/fix broken CI without needless commits. This saves me a lot of time and if I was in a team, money wasted on compute/CI run minutes.

Whats more, you can “checkout” someones branch using dagger without git pulling/git switching. Dagger isn’t just limited to running CI - it can run entire apps including supporting services. This means you can run the entire project in the PR and check it works as it should with all the expected services. Obviously, this is only as good as the effort that goes into writing the Dagger files.

Doing it this way makes running complicated applications easier, for instance, if you needed NATS, Postgres and Redis to run E2E/Integration tests. You can now do that locally or in your CI, e.g. GH Actions or Gitlab Runners.

All that your CI runner does now is start your Dagger process and begin everything inside Dagger. It’s a far better user experience.

I need to sit down and write more about this amazing technology.

Tags:

#dagger #ci

xclip helper

Mon, 03 Mar 2025 00:00:00 +0000

xclip helper

I do a lot of json manipulation/reading in my day job. Often I need to copy contents into the clipboard for putting into MR/Issues or documentation.

Today I finally automated that after years of typing xclip -sel clipboard

#!/usr/bin/env bash

if [ $# -gt 0 ]; then
 # read a file and pipe to xclip
 cat "$1" | xclip -sel clipboard
else
 # read from stdin; "text" | clip
 xclip -sel clipboard
fi
echo "copied to clipboard!"

Tags:

#scripts #linux

Taskfile dynamic command variables

Wed, 29 Jan 2025 00:00:00 +0000

Taskfile dynamic command variables

Taskfile is amazing and I use it as the task runner for all my projects.

Why?

no .env pollution between operations; define envs per task
concatenate many commands together, for instance, regenerate tailwind when files change and restart server
simplify my life by reducing the amount of typing

There more reasons but those are the highlights.

One thing that I only learned a couple months ago is that you can supply dynamic variables to tasks.

As example, I have a bunch of python files that need various environment variables - each file needing a different one to the next. Thats really annoying to configure manually. Here’s a simplistic version using Taskfile.

Here’s how I’d do that in a Taskfile

version: "3"

vars:
 PG_PORT: 5672

tasks:
 benchmark:*:
 env:
 PG_HOST: foo.bar
 TRACEMALLOC: 1
 vars:
 ARG_1: "{{index .MATCH 0}}"
 cmds:
 - python -m {{.ARG_1}}

 production:*:
 env:
 PG_HOST: prod.foo.bar
 TRACEMALLOC: 0
 vars:
 ARG_1: "{{index .MATCH 0}}"
 cmds:
 - python -m {{.ARG_1}}

Now I can run my benchmark or production code for any python module by passing the name of the module like so:

task benchmark:pg_connection
# or
task production:pg_connection

Thats a really silly example, heres one I actually use.

version: "3"

tasks:
 nats:cname:*:
 desc: Publish a message to NATS to trigger a CNAME lookup
 vars:
 ARG_1: "{{index .MATCH 0}}"
 cmds:
 - nats pub scanners.dangling_cname.{{.ARG_1}}

Now I can publish messages to NATS easily in development to the queue I want with any domain name I want to target.

task nats:cname:google.com
task nats:cname:tesla

This has been a massive quality of life improvement. Its possible to use more than one ARG here too but for simplicity I’ve left it at just one.

Tags:

#taskfile

Mermaid.js for Hugo blogs

Fri, 17 Jan 2025 00:00:00 +0000

Mermaid.js for Hugo blogs

Just implemented a mermaid.js shortcode in my Hugo blog.

All credit to: https://navendu.me/posts/adding-diagrams-to-your-hugo-blog-with-mermaid/

This will only render on my Hugo blog and serves as a reference only.

ChatGPT generated this - I don’t care if it makes sense its just a demo

graph TD; A[Start] --> B{Is it raining?}; B -- Yes --> C[Take an umbrella]; B -- No --> D[Go outside]; C --> E[Proceed with your day]; D --> E;

Note: this isn’t perfect and sometimes complex mermaid diagrams fail to render when uploaded as a zet. WIP.

Tags:

#mermaid #hugo

PocketBase used in Beszel server monitoring

Fri, 17 Jan 2025 00:00:00 +0000

PocketBase used in Beszel server monitoring

TIL: beszel uses PocketBase as its server backend!

Lately, I’ve been semi obsessing over PB due to its rich featureset. For a Go dev, it’s a game changer in terms of speed and ease of use.

My side projects will never amount to anything needing more than a single instance. Instead of burning a lot of my limited (2 kids) free time with boilerplate before even getting to the core logic of whatever I’m trying to build, I usually run out of steam.

PocketBase goes a long way to fixing that. It gives so much out-of-the-box. IMO its the Django of Go. Of course, unlike Laravel, Django, Ruby on Rails, it only supports SQLite but.. do you really need PG or MySql? Probably not. Just get a bigger box and vertically scale that bad boy!

So today when I found out beszel uses PB, I was kinda over the moon. This isn’t a toy its a legit useful project. Even TailScale dev rel have made a video on it.

If nothing else its made me even more adamant that PB is a underutilized tool in the Go dev’s toolkit.

I think the big reason for this is its kinda marketed as a JS dev’s tool. Effectively just a FireBase alternative - and it can be. But, its written in Go and can be extended as little or as much as you like.

Want a message queue hanging off it - Done! Want a custom API - Done! The sky is the limit. Only limitation is your imagination.

Tags:

#til #go #pocketbase

PocketBase learned me a browser caching

Wed, 15 Jan 2025 00:00:00 +0000

PocketBase learned me a browser caching

Im writing a file sharing tool to explore pocketbases abilities. The files are stored in PB and retrieved by creating their unique url. This is easy using the JS SDK and works well.

I needed to extend this im go by using PBs hooks. Particularly the OnRequestFileDownload hook. Everytime a file is downloaded we can create custom logic before returning the file object

In this case, im creating a entry in the analytics table to track each download. I also have options like download limits, expiry and password protection. So I need to use the hook.

This is where the problem began. I could trigger the download by not the hook.

It took me a number of hours to figure it out. Browsers cache things. After the first download that url is cached and therefore it won’t fetch the contents from the server again. Which is perplexing to me, but it wouldn’t even show up in my dev tools tab. This means no hook called, no analytics. Not helpful

The workaround is to cache bust by adding a querystrinh to the url. I used Unix time which means every second the cache is busted and contents pulled freshly from server. Perfect.

Tags:

#til #browsers #pocketbase

Using Github Actions to publish my zettelkasten notes

Tue, 14 Jan 2025 00:00:00 +0000

I publish my zettelkasten notes to this website using GitHub Actions.

What is a zettelkasten?

A Zettelkasten is a personal tool for thinking and writing that creates an interconnected web of thought

From the zettelkasten website, a true zet should have the following properties:

Atomic - Each note contains one main idea
Unique identifiers - Every note has a unique ID for referencing
Linking - Notes are connected to other related notes via links
Emergence - Knowledge and insights emerge from the network of connected notes

It’s meant to ensure that over time, the notes become more interconnected and the network of knowledge grows.

My zettelkasten isn’t completely true to this model but its good enough for my use case.

Why do I use a zettelkasten?

When I learn something, figure out a new way or approach, ill try put it in my zet. Keeping them short and to the point makes this more likely to happen. I found with a blog I finessed it too much which built up a resistance.

My zets can be as polished or sloppy as I choose. I am the only person I write for.

Unlike obsidian or other tools my approach is just text. Linking beyond a simple search, is on me to do at the time of search. This works better than an automated tool because it’s the act of re-reading notes even if unrelated that builds your mental model of your own knowledge base (spaced repetition)

My zet-cmd tool

To write, read and edit my notes I have a custom CLI written in Go, called zet-cmd.

It uses GitHub as the backend to store the zets and your $EDITOR to write them. It’s a simple tool that does one thing and does it well enough for me.

Here’s the output of zet:

NAME
zet - zettelkasten commander

SYNOPSIS
zet COMMAND

COMMANDS
help - display help similar to man page format
conf - manage conf in /home/danielmichaels/.cache/zet/config.yaml
var - cache variables in /home/danielmichaels/.cache/zet/vars
new|c|create - Create a new zet
l|latest|last - Get the most recent zet isosec and print it screen
e|edit - edit a zet
g|get - Retrieve a zet for editing
q|query - create a searchable URL with a query string
f|find - Find a zet title by search term
check - check environment variables and configuration
t|tags - Find zet(s) by tag'
git - run git commands over the zet repo
v|view - view command for zet entries.

DESCRIPTION
The zet command is Zettelkasten Bonzai branch used to create small slips of knowledge.Those slips are then uploaded to Github for public search-ability and ease of use.

CONTACT
Site: danielms.site
Source: git@github.com:danielmichaels/zet-cmd.git
Issues: github.com/danielmichaels/zet-cmd/issues

LEGAL
zet (v0.5.0) Copyright 2022-2024 Daniel Michaels
License Apache-2.0

The tool is written in Go and uses Bonzai for the CLI. Though, it’s using an older version and will need updating to the latest version - which has several breaking changes. Could equally be rewritten in kong another favourite of mine.

How I use it

When I want to create a new zet I run zet new. This will create a new entry in my zet repo using a timestamp as a folder name, and inside that folder a new README.md. The zet new "my new zet" will generate the README.md with my new zet as the # (h1) heading and drop into the editor, in my case, (n)vim.

Saving and exiting will then prompt you to commit the changes to the repo, this will then push the changes to GitHub.

Editing a zet is as simple as running zet edit. This accepts last for the most recent zet, or a search term.

For example, zet edit kong returns:

# zet edit kong
0) 20230107005542 Kong is an amazing CLI for Go apps
1) 20240801073758 How I write Golang CLI tools today (using Kong)

Entering one of the numbers will open the zet in your editor.

Viewing a zet is as simple as running zet view. This accepts last for the most recent zet, or a search term. Same functionality as zet edit. zet view all will list all zets in the repo.

Zets are rendered in the terminal using glow to make them visually appealing.

Adding zet’s to a hugo site

Sometimes I am not at my computer but want to reference something I’ve written about before. All my zets are public, so I figured why not add them to my site. The added bonus is others can also view them, and it takes the pressure off of me to write a blog post.

To do that I use a GitHub action to fetch the zets from my repo and add them to my site. The action runs every hour and fetches the latest zets only triggering a rebuild if a new zet is added. This is done via the commit functionality of stefanzweifel/git-auto-commit-action which just commits directly to master. It’s my blog, so I don’t care if it commits straight to master.

All the tools and actions are available on GitHub in the scripts and .github directories. ./scripts/fetch-zet.go is the script this action uses to fetch the zets.

How it works

A picture paints a thousand words. This graph shows the broad flow of events. Refer to ./scripts/fetch-zet.go for the full flow.

graph TD subgraph "Github Actions hourly schedule" A[Start fetch-zet.go] --> E[Fetch Contents from GitHub API] E --> F{Check if New Zets} F -->|Yes| G[Write JSON to assets/zet.json] G --> H[Create zet entry if needed] F -->|No| L[Exit - No Updates Needed] H --> M[Commit changes to repo] end M --> N[Netlify rebuild on commit to Master]

Each hour a GitHub action will run the fetch-zet binary. This will fetch the contents of the zet repo from the GitHub API. The API returns a list of all the files in the repo.

Once a new zet is identified, the script will write the new zet to a JSON file in the assets directory. This is then compared against the existing zet entries, if a new one is found the go program will create a new zet markdown file.

The markdown file’s top level h1 tag is used to create the slug and title for the hugo page. The GitHub API provides the title, but we don’t use that because if the h1 changes after the original commit, the API will still return the original commits title.

Go templating is used to create the markdown file with hugo’s frontmatter.

Once done, the GitHub action will commit any new files to the repo. Netlify will then detect that a commit has been made to master and trigger a redeployment of the site.

The result

So far, this simple process has allowed me to continue to write zets using my CLI tools but be able to refer to them from anywhere. Before this I was locked out of viewing them without setting up my zet CLI locally.

As you can see, it works well enough.

Migrating from CapRover to Coolify

Fri, 03 Jan 2025 00:00:00 +0000

Migrating from CapRover to Coolify

I think I’ve been running a CapRover instance since 2019 or 2020. So long that my instance is still running Ubuntu 18.04. And, thats why I’m migrating.

Recently, I blew away all my machines and installed Ubuntu 24.04. In doing this I accidentally deleted my old CapRover SSH keys. The machine is SSH access only so I don’t have anyway to manage it, or upgrade CapRover.

I decided to spin up a new instance on Hetzner and try the backup/restore process that CapRover recommends. The process is quite clunky IMO and ultimately didn’t work. I have a number of services running on the CapRover and knowing I can’t migrate them over gave me the impetus to move completely over to Coolify.

Side note, one thing that annoyed me about CapRover is when services went down due to node issue, such as running out of RAM (underprovisioning), all the services would not automatically come back up (I miss kuberenetes!). This meant I had to go into each app and manually restart them - theres no API or CLI for this. Coolify has an API and I’m hopeful I could write scripts to do this.

So far I’m really enjoying Coolify. I like the approach to deploying services from GitHub. CapRover used webhooks but each service needed to manually sign in and then copy/paste the webhook address into the repo. Coolify instead creates a GitHub App so you can add whichever repo’s you want and it automatically has access to them. Much better DX.

I use basic auth a lot on my CapRover services (e.g. NTFY and ChangeDetection.io). Coolify doesn’t have a button/automation for this. You have to manually add a label to the service file (compose file) which Traefik then enforces. Its not too hard but its also not as easy as a button and form with username/ password fields for the auth like CapRover does. Also I’m not sure how it handles websockets which, again, CapRover sets up with a single click. I suspect this might another Traefik config I’ll have to set.

So far, so good. Glad I made the switch.

Tags:

#caprover #coolify #paas

PocketBase and Litestream - very unreliable

Wed, 01 Jan 2025 00:00:00 +0000

PocketBase and Litestream - very unreliable

After most of a day I’ve given up on a pure PocketBase Litestream setup.

It’s really flaky and I experienced several instances where the restored database was subseqently overritten by PocketBase. I couldn’t ascertain why and don’t have the time to understand the problem.

Frankly, I’m disappointed by it. I think PocketBase will be my new MVP/PoC stack with Go. Its an amazing tool and offers so much out of the box.

I created <pocketshare.infra.ptco.rocks> with it in about a week. Its a file sharing app - which I felt pushed all the PocketBase buttons. It was an excellent thing to build for learning PB.

For now, I’m resorting to volume mounting the DB but running it in a container. I’ll rely on automated backups and manual intervention in case of emergency rather than automation via Litestream.

This will work great but I’m still down about not figuring out Litestream!

ref: - https://pocketbase.io - https://github.com/pocketbase/pocketbase/discussions/3080

Tags:

#go #pocketbase

pycharm-goland broke my GPG verified commits

Mon, 16 Dec 2024 00:00:00 +0000

pycharm-goland broke my GPG verified commits

Open ~/.gnupg/gpg-agent.conf
Comment the line that uses pinentry-program <path>/pinentry-ide.sh
Reload gpg agent with gpgconf --reload gpg-agent

Hopefully they fix this in the next release (they usually are very responsive to these issues) because each time you open the IDE it’s going to prompt you to install this script again.

ref: https://youtrack.jetbrains.com/issue/IJPL-173525/Git-GPG-signing-fails-with-errors-like-Bad-CA-certificate-or-failed-to-write-commit-object

Tags:

#ide #gpg #bug

structured logging; convincing a team to use it

Tue, 10 Dec 2024 00:00:00 +0000

structured logging; convincing a team to use it

My $DAYJOB deals with millions of messages through the message bus, each messsage kicking off many more IO bound operations.

We use Loki and Grafana to monitor the logs.

Our python code relies heavily on threading and its hard to know whats happening in each thread.

And we don’t use structured logging. Which ever since I started left a sour taste in my mouth as I’ve seen it used very well in other environment which 10x less load and events.

Today, after someone made a big change which included what I consider bad changes to the plain loggers I decided to make the effort to convince them that structured logging is worth the effort.

My core driver is the time wasted finding issues in our threaded code which we could easily have detected/debugged with a trace ID spanning the events. OpenTelemetry in this team would be a very very hard sell. But a trace ID? Thats doable and has a lot of upside.

I know I will get a lot of pushback. Pretty much all the 12 factor type improvements I’ve made have received little interest.

So this time I made a huge effort to push this. Incredibly detailed reasoning, issues, example PoC’s in the core libraries. Tomorrow, I’ll start making the merge requests to get the initial work done. I think if I can get our core rabbit library cut over and then our two “hot path” projects 80% of the way there I’ll have the majority interested.

And, if not. Well, I’ll just document how I did it and keep all those learnings for future $DAYJOB if needed.

Tags:

#logging #software #team

DNS Enumeration summary

Wed, 20 Nov 2024 00:00:00 +0000

DNS Enumeration summary

Testing out fabric summary of youtube clips

From https://www.youtube.com/watch?v=GRv-O-Hq9Io which is about DNS enumeration and other techniques.

Using fabric -sp extract_wisdom -y=$URL

SUMMARY

Stoke Frederick hosts Bounty Thursdays, discussing tools, techniques, and insights about DNS in cybersecurity.

IDEAS:

DNS is a foundational element in cybersecurity, often blamed when issues arise.
The evolution of DNS tools reflects ongoing advancements in cybersecurity techniques.
Subdomain enumeration tools like Amass and Subfinder are vital for passive discovery.
Active DNS enumeration requires robust tools for effective brute forcing and validation.
Historical tools continue to influence current methodologies in DNS penetration testing.
Community knowledge sharing enhances understanding of DNS vulnerabilities and tools.
Subdomain takeovers represent a lucrative opportunity in bug bounty hunting.
Utilizing various DNS resolvers is crucial to avoid false positives during testing.
Understanding DNS records can reveal security weaknesses in an organization’s infrastructure.
Effective DNS enumeration involves both passive and active techniques for comprehensive testing.
Wildcard DNS responses can obscure vulnerabilities during penetration testing.
Tools like DNS Validator help maintain clean resolver lists for accurate testing.
Leveraging historical DNS data can provide insights into vulnerabilities and attack surfaces.
The importance of community collaboration is highlighted in the development of DNS tools.
Anomalies in DNS responses can indicate potential security issues or misconfigurations.
Continuous improvement and adaptation of tools are essential in the evolving cybersecurity landscape.
Properly configured DNS records can prevent unauthorized access and reduce attack vectors.
Rebinding attacks demonstrate the complexity of DNS vulnerabilities and their exploitation.
The DNS space requires ongoing education and exploration to uncover new techniques.
Experimenting with custom tools can lead to innovation in DNS testing methodologies.
Effective communication among cybersecurity professionals enhances the collective knowledge base.

INSIGHTS:

DNS remains central to cybersecurity; understanding its nuances is essential for effective testing.
The evolution of tools and techniques reflects the dynamic nature of the cybersecurity landscape.
Community-driven development leads to innovative solutions addressing complex cybersecurity challenges.
Subdomain enumeration is a key aspect of reconnaissance, influencing overall penetration testing success.
Active and passive techniques must be combined for a comprehensive approach to DNS security.
The ability to adapt and refine tools is crucial for maintaining effectiveness in testing.
Historical data can provide valuable insights into vulnerabilities and attack patterns in DNS.
Collaboration within the cybersecurity community fosters knowledge sharing and tool improvement.
Understanding DNS vulnerabilities can lead to more effective defense strategies in organizations.
Continuous learning and exploration are vital for staying ahead in the cybersecurity field.

QUOTES:

“DNS is the staple of everything and when things break we can always blame it on DNS.”
“There’s just so many tools out there.”
“The essence of why we write a tool is because a lot of people starting out need good resolvers.”
“The first two core spaces are passive and active.”
“Community is so important; the more you got in community, the more you learn.”
“You can learn to code in one language and better translate yourself to another.”
“It’s a classic catch-all; you will send an email to a server and it will respond.”
“You don’t always have to get a result; you just want to know.”
“If you see a few people suddenly starring a project, it’s a good indication they’re onto something.”
“Speed, reliability, and the ability to pipe into other tools are key metrics for evaluating tools.”
“You can get more insight by running it yourself.”
“The DNS space is probably the best example of why community is important.”
“The essence of DNS vulnerabilities is that they’re often overlooked.”
“Rebinding attacks can reveal internal resources previously thought secure.”
“Subdomain takeovers have historically been the most profitable bug in bug bounty hunting.”
“Every person who goes out there and writes a tool deserves recognition.”
“You need to introduce other resolvers if you’re scaling.”
“DNS is a foundational element in cybersecurity.”
“Effective DNS enumeration involves both passive and active techniques.”
“Historical DNS data provides insights into vulnerabilities and attack patterns.”

HABITS:

Regularly engage with the cybersecurity community to share knowledge and learn new techniques.
Experiment with different DNS tools to determine which best fits your workflow.
Maintain a clean resolver list for accurate DNS enumeration during testing.
Continuously update and refine personal coding skills to adapt to evolving tools.
Deploy personal domains for hands-on practice with DNS configurations and vulnerabilities.
Use a VPN when conducting DNS tests from home to avoid ISP issues.
Schedule regular practice sessions with new tools to familiarize yourself with their functionalities.
Engage in collaborative projects to enhance tool development and understanding of DNS.
Stay informed about the latest trends and updates in cybersecurity tools and techniques.
Document and share personal experiences with tools to contribute to community knowledge.

FACTS:

DNS is often blamed for network issues, reflecting its foundational role in internet infrastructure.
Historical tools for DNS enumeration have shaped current methodologies and practices.
Subdomain takeovers can yield significant financial rewards in bug bounty programs.
Many DNS tools emphasize passive discovery to minimize impact on target systems.
Active DNS enumeration requires a robust understanding of brute forcing techniques.
Community collaboration leads to the development of innovative cybersecurity tools.
Proper DNS configuration can significantly reduce an organization’s attack surface.
Wildcard DNS responses can obscure potential vulnerabilities during penetration testing.
Continuous learning is crucial in adapting to the evolving landscape of cybersecurity.
The DNS space is rich with opportunities for discovering vulnerabilities and attack vectors.

REFERENCES:

Amass
Subfinder
DNS Validator
DNS Dumpster
ctfr
Pure DNS
DNSX
Nuclei
Plunder
Security Trails
ByNine
TryHackMe
Singularity Framework
Hack Luke’s Tools
Project Discovery Tools
DNS Cool

ONE-SENTENCE TAKEAWAY

Understanding DNS vulnerabilities and employing effective tools are essential for successful penetration testing and bug bounty hunting.

RECOMMENDATIONS:

Engage in community forums to share insights and learn from experienced cybersecurity professionals.
Regularly practice with various DNS tools to enhance your skills in enumeration and testing.
Experiment with setting up your own domains to gain practical experience with DNS configurations.
Stay updated on the latest trends and tools in the cybersecurity landscape for effective testing.
Combine passive and active DNS enumeration techniques for comprehensive vulnerability assessments.
Leverage historical DNS data to identify potential vulnerabilities and attack vectors.
Document your experiences with tools to contribute to the broader cybersecurity community.
Collaborate with others on projects to foster innovation in DNS testing methodologies.
Develop a habit of evaluating new tools based on speed, reliability, and integration capabilities.
Participate in workshops or training sessions focused on advanced DNS techniques and tools.

Tags:

#fabric #dns

Deploy to Netlify using Github Actions

Tue, 22 Oct 2024 00:00:00 +0000

Deploy to Netlify using Github Actions

After switching some projects from my personal account into a new GitHub organisation my Netlify automated deploys started failing.

To use a private organisation you have to pay $19/USD per month. Considering its the 22nd and I’ve used 5 of my 300 free build minutes they can get stuffed.

So I had to switch from automated to triggering the deployment using their CLI in GitHub actions.

Honestly I just ask CodyAI (sourcegraph’s AI tool) to convert my hugo.toml file into a GitHub actions template and it got me 90% done. I had to tweak a couple things (switch to pnpm for example)

Here’s the file,

name: Netlify Deploy

on:
 workflow_dispatch:
 pull_request:
 push:
 branches:
 - main

jobs:
 build-and-deploy:
 runs-on: ubuntu-latest
 steps:
 - name: Checkout Repository
 uses: actions/checkout@v4

 - name: Setup Node.js
 uses: actions/setup-node@v4
 with:
 node-version: 20

 - name: Install pnpm
 uses: pnpm/action-setup@v2
 with:
 version: 8

 - name: Get pnpm store directory
 shell: bash
 run: |
 echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV

 - name: Setup pnpm cache
 uses: actions/cache@v3
 with:
 path: ${{ env.STORE_PATH }}
 key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
 restore-keys: |
 ${{ runner.os }}-pnpm-store-

 - name: Setup Hugo
 uses: peaceiris/actions-hugo@v2
 with:
 hugo-version: "0.134.3"
 extended: true

 - name: Setup Go
 uses: actions/setup-go@v4
 with:
 go-version: "1.23"

 - name: Install Dependencies
 run: pnpm install

 - name: Project Setup and Build
 run: |
 pnpm project-setup
 pnpm build

 - name: Deploy to Netlify
 uses: nwtgck/actions-netlify@v2.0
 with:
 publish-dir: "./public"
 production-branch: main
 github-token: ${{ secrets.NETLIFY_GITHUB_TOKEN }}
 deploy-message: "Deploy from GitHub Actions"
 enable-pull-request-comment: false
 enable-commit-comment: true
 overwrites-pull-request-comment: true
 env:
 NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
 NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
 timeout-minutes: 1

And its working perfectly!

Tags:

#netlify #github #cicd

git log to PR message

Sun, 20 Oct 2024 00:00:00 +0000

git log to PR message

git log <sha>..HEAD | mods "write a concise pull request for these changes"

And it produced a really decent summary of all the changes. I think the key is create meaningful commit messages.

I like to add a lot of context to my commit bodies and it pays off here.

Example

## Summary

This pull request introduces multiple updates and refactors across the dashboard
templates, email account management, and Slack integration. These changes aim to
enhance performance, usability, and maintainability.

## Key Changes

1. Dashboard Enhancements:\
 • Simplified, restructured, and cleaned up dashboard templates, notably\
 accounts.html and connections.html ([Commit 5bd7454]).\
 • Added accounts listing with filtering, sorting, and pagination ([Commit\
 a2b906d]).\
 • Implemented detailed connection display and profile updates ([Commit\
 d7cdda8]).
2. Account and Connection Management:\
 • Introduced delete functionality with confirmation for accounts and\
 connections ([Commit 72818d1]).\
 • Improved URL routing and messaging for account actions in views.py\
 ([Commit d968d7e]).\
 ..truncated

Tags:

#ai #git

docker compose up --watch

Fri, 27 Sep 2024 00:00:00 +0000

docker compose up –watch

I used to volume mount to get “hot-reload” behaviour. Now I’ve switched to docker compose watch.

Its much simpler to understand. Here’s an example of syncing code and restart a celery worker container:

services:
 worker:
 build:
 context: .
 network: host
 network_mode: host
 command: celery
 env_file:
 - .env
 depends_on:
 db:
 condition: service_healthy
 rabbit:
 condition: service_healthy
 volumes:
 - worker_data:/usr/src/app
 develop:
 watch:
 - action: sync+restart
 path: ./
 target: /app
 ignore:
 - .venv
volumes:
 worker_data:

A simplified compose.yml file but the core part I want to showcase is the develop key.

Using develop.watch lets you define the path to “watch”, target to sync the changes to and the action to take.

For this container I have to restart it but its very robust and simple to follow. You can also just sync which would work well for some applications.

It’s possible to achieve similar things with other tools (watchfiles is a good one) and methods but I prefer to lean on a docker builtin these days.

Tags:

#docker

Slack Bolt Python oauth install fix

Thu, 26 Sep 2024 00:00:00 +0000

Slack Bolt Python oauth install fix

https://github.com/slackapi/bolt-python/issues/492

It was not apparent to me that the slack-bolt python app requires you to start the oauth flow from /slack/install

At that address it will load a page with a “Install” button (not your own) and only clicking that to start the flow will work.

Otherwise it don’t work! Super confusing and wasted a number of precious hours on this.

Tags:

#slack #python

NF20MESH bridge mode

Mon, 23 Sep 2024 00:00:00 +0000

NF20MESH bridge mode

This thing is a POS.

To set bridge mode do the following:

disable wifi 2.4/5g and stop SSID being broadcast
remove all connections
disable DHCP on v4 and v6
add ptm and eth connections back as bridge mode

Now kick the connection on ABB and power off for a couple of minutes

Goa websocket with subprotocols (auth)

Sat, 07 Sep 2024 00:00:00 +0000

Goa websocket with subprotocols (auth)

How I figured out to pass an API key to my websocket endpoint using goa and React. It wasn’t straight forward!

Firstly, you have to setup an websocket.Upgrader{} and assign CheckOrigin and Subprotocols values.

upgrader := &websocket.Upgrader{}
upgrader.CheckOrigin = func(r *http.Request) bool { return true } // setup authorised origins; this is a demo

// You have to tell it that Sec-Websocket-Protocol and your custom auth header are valid subprotocols
// or it won't work! and you will get an error like this:
// Error during WebSocket handshake: Sent non-empty 'Sec-WebSocket-Protocol' header but no response was received
upgrader.Subprotocols = []string{"Sec-Websocket-Protocol", "x-api-key"}

Then you have to add some headers to your request. But before we do that, in Goa we need a custom middleware to read the subprotocol. Using Authorization: Bearer asdadad, or in my case x-api-key: 123 like a traditional HTTP request doesn’t work.

# authorization (subprotocol) header on a websocket 101 Switching
Sec-WebSocket-Protocol: x-api-key, key_000000000000

Because its x-api-key, key_000000000000, you have to split the string and do some manipulation. Things you don’t have to do in normal HTTP requests.

Here’s how I do it using a Goa middleware

// set custom context keys
type xApiKeyWSType string
const apiKeyCtxKey xApiKeyWSType = "x-api-key"

func WebsocketConnection() func(http.Handler) http.Handler {
	return func(h http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			req := r
 // if not a websocket connection, continue
			if upgrade := r.Header.Get("Connection"); upgrade != "Upgrade" {
				// Not a websocket connection
				h.ServeHTTP(w, req)
			}
 // Get the header with the key
			if auth := r.Header.Get("Sec-WebSocket-Protocol"); auth != "" {
 // the header will return a string: x-api-key, key_000000000000
 // we need to split, then trim white space
				header := strings.Split(auth, ",")
				header[1] = strings.TrimSpace(header[1])
				ctx := context.WithValue(r.Context(), apiKeyCtxKey, header[1])
				req = r.WithContext(ctx)
			}
			h.ServeHTTP(w, req)
		})
	}
}

Then in your security scheme, you’ll need to get that context key we just set.

func (a *ApiKey) Validate(
	ctx context.Context,
	key string,
	scheme *security.APIKeyScheme,
	db *repository.Queries,
) (context.Context, error) {
	if key == "" {
		key = GetXApiKeyWS(ctx)
	}
 // truncated but do something with the key.
 // a typical HTTP request will have the key available already but
 // we need to grab it via the GetXApiKeyWS function
	return ctx, nil
}

It took me a few hours of reading issues, code, websocket specs to figure out why wscat and insomnia could connect but not React. Inspecting the headers was a big clue - no response headers were set from the server. But, if I connected to a globally available unauthenticated server such as wss://echo.websocket.events they were.

It’s all working now and apart from this I’m really happy with how easy it is to get going with Goa and websockets. From a design.go/DSL prespective its simple. I just replaced Result with StreamingResult in my Services' Method. Had to plumb in the upgrader but it was simple. The hard part was making sure access was still secure behind my API keys (and soon to be JWTs).

Tags:

#goa #websockets #react #go

Hire me

Thu, 05 Sep 2024 00:00:00 +0000

Hello! I’m a professional developer specializing in Go, Python, and JavaScript. I can take your ideas and designs and turn them into web applications, ready for deployment.

Skills

Languages:

Go
Python
JavaScript

Areas of Expertise:

Transforming design into operational web applications
Building with event systems and microservices
API development using OpenAPI, specifically with a design-first approach
Deployment using Kubernetes, fly.io, Render, or a Virtual Private Server running containers
IaC using terraform and ansible to keep your infrastructure consistent

What Can I do for You?

Whether you’re an individual with a great idea or a business looking to improve your web presence, I can help you. I am not only a developer but also a consultant who can provide you with solutions tailored to your unique requirements. I go beyond just coding to understand your goals and develop customized solutions that help you achieve them.

Consulting:

I’ll delve deep into understanding your idea or problem, offering tech-based, innovative solutions. I can help you understand complex concepts and tech jargon and work with you to create a web application that meets all your requirements.

Building:

I love challenging projects that make me think and push boundaries. Whether it’s building a web application from scratch or redesigning an existing one to make it better, I am up for the task. I’m proficient in Go, Python, and JavaScript, and am comfortable building with containerisation, event systems, and microservices.

Why Hire Me?

In a world full of generic solutions, I offer personalized service. From deployment to designing strategies for growth, I work closely with my clients to ensure they’re not just getting a product, but a roadmap to success.

I know that technology is ever-evolving, and I strive to be at the forefront of it. My passion for continuous learning ensures your project isn’t just relevant for now, but future-ready.

Let’s Connect!

If you’re looking for a dedicated, passionate developer who is great at solving problems and loves taking on challenges, reach out to me. I’d love to hear about your idea and explore how we can turn it into a reality.

How to invalidate tanstack query key in same component

Tue, 03 Sep 2024 00:00:00 +0000

How to invalidate tanstack query key in same component

I am new to TanStack query, router, form etc.

I wanted to have a table and a form in the same component (or page containing many components).

Create a thing in the form, invalidate the cache and have the table re-fetch table data rendering the table with the new data.

At the same time I am using Orval to generate all my TanStack Query (TQ from now on) methods.

It took me a while to figure this out but AFAICT, you do not invalidate a cache from Orval’s generated spec. You instead use TQ directly.

export function Foo() {
 const queryClient = useQueryClient()
 const { status, data } = useTQListMethod()
 const { mutate } = useMutateTQListMethod({
 mutation: {
 onSuccess: () => {
 // this is how I invalidated the cache
 queryClient.invalidateQueries({
 queryKey: ["/formdata"] // matches the queryKey from useTQListMethod
 })
 }
 }
 })

 if (status === "pending") return <p>Loading...</p>;
 if (status === "error") return <p>Error :(</p>;


 const handleSubmit = async (e: React.FormEvent<HTMLFormElement>) => {
 const formData = new FormData(e.currentTarget)
 // truncated logic

 // does the mutation (sending POST data) but does not invalidate the cache!
 mutate({data: myData})
 }
 return (
 <>
 <form onSubmit={handleSubmit}>This is truncated and not that important</form>
 <MyTable data={data.data}
 </>
 )
}

Useing queryClient.invalidateQueries let me force my cache query to refetch and successfully re-render the table (MyTable) with the new entry.

Tags:

#react #tanstack

How I write Golang CLI tools today (using Kong)

Thu, 01 Aug 2024 00:00:00 +0000

How I write Golang CLI tools today (using Kong)

I’ve ditched cobra for alecthomas/kong for good. Smaller, easier to grok, great interface design.

This is a short snippet on how I layout a basic CLI (which I do for every project, nearly). CLI’s power most of my app’s, including web servers.

Here is the most simple layout of a useless but instructive example CLI.

.
├── cmd
│ └── app
│ └── main.go
├── go.mod
├── go.sum
└── internal
 └── cmd
 ├── cmd.go
 └── echo.go

Each *.go file in detail.

// cmd/app/main.go

package main

import (
 "fmt"
 "me/my-cli/internal/cmd"
 "os"

 "github.com/alecthomas/kong"
)

const appName = "my-cli"

var version string

type VersionFlag string

func (v VersionFlag) Decode(_ *kong.DecodeContext) error { return nil }
func (v VersionFlag) IsBool() bool { return true }
func (v VersionFlag) BeforeApply(app *kong.Kong, vars kong.Vars) error {
 fmt.Println(vars["version"])
 app.Exit(0)
 return nil
}

type CLI struct {
 cmd.Globals

 Echo cmd.EchoCommand `cmd:"" help:"Example of an Echo command"`
 Version VersionFlag ` help:"Print version information and quit" short:"v" name:"version"`
}

func run() error {
 if version == "" {
 version = "development"
 }
 cli := CLI{
 Version: VersionFlag(version),
 }
 // Display help if no args are provided instead of an error message
 if len(os.Args) < 2 {
 os.Args = append(os.Args, "--help")
 }

 ctx := kong.Parse(&cli,
 kong.Name(appName),
 kong.Description("My new CLI"),
 kong.UsageOnError(),
 kong.ConfigureHelp(kong.HelpOptions{
 Compact: true,
 }),
 kong.DefaultEnvars(appName),
 kong.Vars{
 "version": string(cli.Version),
 })
 err := ctx.Run(&cli.Globals)
 ctx.FatalIfErrorf(err)
 return nil
}

func main() {
 if err := run(); err != nil {
 os.Exit(1)
 }
}

// internal/cmd/cmd.go

package cmd

type Globals struct {
 Format string `help:"Output format" default:"console" enum:"console,json"`
}

// internal/cmd/cmd.go

package cmd

import (
 "encoding/json"
 "fmt"
)

type EchoCommand struct {
 Text string `arg:"" help:"text to echo"`
}

type Formatter interface {
 Output(text string) string
}

type OutputFunc func(text string) string

func (o OutputFunc) Output(text string) string {
 return o(text)
}

func echoMessage(text string, format Formatter) {
 fmt.Println(format.Output(text))
}

func (e *EchoCommand) Run(g *Globals) error {
 ConsoleFormatted := OutputFunc(func(text string) string {
 return text
 })
 JSONFormatted := OutputFunc(func(text string) string {
 jsonData, _ := json.Marshal(map[string]string{"message": text})
 return string(jsonData)
 })

 if g.Format == "json" {
 echoMessage(e.Text, JSONFormatted)
 }
 if g.Format == "console" {
 echoMessage(e.Text, ConsoleFormatted)
 }

 return nil
}

Once these files exist, from the root level, run go run cmd/app/main.go and it will output:

Usage: my-cli <command> [flags]

My new CLI

Flags:
 -h, --help Show context-sensitive help.
 --format="console" Output format ($MY-CLI_FORMAT)
 -v, --version Print version information and quit ($MY-CLI_VERSION)

Commands:
 echo Example of an Echo command

Run "my-cli <command> --help" for more information on a command.

The echo command can be called with:

go run cmd/app/main.go echo "this is going to be written to the console"
# this is going to print to the console

and to print JSON:

go run cmd/app/main.go echo --format=json "this is going to print out some json"
# {"message":"this is going to written to the console"}

Tags:

#go #cli #kong

RabbitMQ missed heartbeat BlockingConsumer

Thu, 25 Jul 2024 00:00:00 +0000

RabbitMQ missed heartbeat BlockingConsumer

Fix missed heartbeats from client, add the following line of code:

connection.process_data_events()

This is for when using a basic_consume, start_consuming.

ref: https://anands.me/blog/pika-missed-heartbeats-rabbitmq

Go's peterbourgon ff CLI package snippet

Thu, 11 Jul 2024 00:00:00 +0000

Go’s peterbourgon ff CLI package snippet

This snippet will create a simple CLI using https://github.com/peterbourgon/ff.

It will read CLI arguments in the following order:

flag
environment variable
config file

This isn’t perfect (its a POC) but is a good starting point.

package main

import (
	"context"
	"fmt"
	"github.com/peterbourgon/ff/v4"
	"github.com/peterbourgon/ff/v4/ffhelp"
	"github.com/peterbourgon/ff/v4/ffyaml"
	"log"
	"os"
	"os/signal"
	"path/filepath"
	"text/template"
)

const appName = "pvenotify"

func main() {
	ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, os.Kill)
	defer cancel()

	ret := run(ctx)
	os.Exit(ret)
}

type config struct {
	username string
	password string
	host string
	noTLSVerify bool
	verbose bool
	config string
	help bool
}

func run(ctx context.Context) int {
	var cfg config

	userConfigDir, err := os.UserConfigDir()
	if err != nil {
		log.Fatalf("could not determine user config dir: %v", err)
	}
	pveConfigDir := fmt.Sprintf("%s/%s", userConfigDir, "pvconfig")
	err = CreateDirectoryIfNotExist(pveConfigDir)
	if err != nil {
		_, _ = fmt.Fprintf(os.Stderr, "could not create pveconfig dir: %v\n", err)
	}
	fd := FileData{cfg}
	tfile, err := os.ReadFile("./config.yaml")
	tmpl := template.Must(template.New("config").Parse(string(tfile)))
	err = generateDefaultConfigFile(pveConfigDir, tmpl, fd)

	fs := ff.NewFlagSet(appName)
	fs.StringVar(&cfg.username, 'u', "username", "un", "username for authentication")
	fs.StringVar(&cfg.password, 'p', "password", "pw", "password for authentication")
	fs.StringVar(&cfg.host, 'a', "api", "http://localhost:8006/api2/json", "PVE api host address")
	fs.BoolVarDefault(&cfg.verbose, 'v', "verbose", false, "verbose logging")
	fs.BoolVarDefault(&cfg.noTLSVerify, 't', "no-tls-verify", false, "do not verify TLS connections")
	fs.StringVar(&cfg.config, 'c', "config", fmt.Sprintf("%s/config.yaml", pveConfigDir), "location of config file")
	fs.BoolVarDefault(&cfg.help, 'h', "help", false, "show help information")

	root := &ff.Command{
		Name: appName,
		Flags: fs,
		Usage: fmt.Sprintf("%s [OPTIONS]", appName),
		Subcommands: nil,
		Exec: func(_ context.Context, args []string) error {
			if cfg.help {
				_, _ = fmt.Fprintln(os.Stderr, ffhelp.Flags(fs))
				return nil
			}
			fmt.Printf("Config: %+v\n", cfg)
			return nil
		},
	}

	if err := root.ParseAndRun(
		ctx,
		os.Args[1:],
		ff.WithEnvVarPrefix("PVE"),
		ff.WithEnvVars(),
		ff.WithConfigFileFlag("config"),
		ff.WithConfigAllowMissingFile(),
		ff.WithConfigFileParser(ffyaml.Parser{}.Parse),
	); err != nil {
		log.Fatalf("failed to parse config: %v", err)
	}
	return 0
}
func doesNotExist(path string) bool {
	if _, err := os.Stat(path); os.IsNotExist(err) {
		return true
	}
	return false
}

func CreateDirectoryIfNotExist(dirPath string) error {
	if doesNotExist(dirPath) {
		if err := os.Mkdir(dirPath, 0755); err != nil {
			return err
		}
	}
	return nil
}

type FileData struct {
	config
}

func generateDefaultConfigFile(dirPath string, tmpl *template.Template, data FileData) error {
	defaultFileName := "config.yaml"
	filePath := filepath.Join(dirPath, defaultFileName)
	if doesNotExist(filePath) {
		file, err := os.Create(filePath)
		if err != nil {
			return err
		}
		defer file.Close()

		if err := tmpl.Execute(file, data); err != nil {
			return err
		}
	}
	return nil
}

And the config file would look like this:

no-tls-verify: true
username: username_via_config
password: password_via_config
host: http://localhost_via_config:8006/api2/json
verbose: true

Tags:

#go #cli

minio for local s3 testing

Thu, 27 Jun 2024 00:00:00 +0000

minio for local s3 testing

How to setup minio for use in local development.

services:
 minio:
 image: quay.io/minio/minio
 container_name: minio-local
 volumes:
 - minio_data:/data
 ports:
 - "9090:9090"
 - "9000:9000"
 environment:
 MINIO_ROOT_USER: root
 MINIO_ROOT_PASSWORD: password
 command: "server /data --console-address :9090"
volumes:
 minio_data: {}

The minio profile will point any s3 data to the s3-specific service which uses our local minio container as the endpoint_url.

You’ll also need to aws configure --profile minio and enter the API keys from minio. To create them in the UI go to Access Keys.

[profile minio]
services = s3-specific
[services s3-specific]
s3 =
 endpoint_url = http://localhost:9000

In the code we have to create and load some environment variables so that it won’t use the defaults

AWS_PROFILE=minio
AWS_ENDPOINT_URL_S3=http://localhost:9000

And finally in we need to create a bucket in minio called my-bucket or whatever you like. I just do that in the UI.

AWSWrangler

Once those items have been established, its possible to use the minio profile for awswrangler.

The best way to do this without making any code changes is to use environment variables.

In addition to the variables above also add AWS_ENDPOINT_URL=http://localhost:9000.

All together it will look like:

AWS_PROFILE=minio
AWS_ENDPOINT_URL=http://localhost:9000
AWS_ENDPOINT_URL_S3=http://localhost:9000

This will work on most s3 projects

Tags:

#s3 #aws #development

golang ssh client snippet

Sat, 27 Apr 2024 00:00:00 +0000

golang ssh client snippet

Here’s an SSH snippet I’ve used successfully across projects.

package ssh

import (
	"github.com/melbahja/goph"
	"golang.org/x/crypto/ssh"
)

type Auth []ssh.AuthMethod

type Config struct {
	Username string
	HostAddress string
	Auth goph.Auth
	Port int32
}

// Client is an SSH client capable of sending and receiving through a tunnel.
func Client(sc *Config) (*goph.Client, error) {
	client, err := NewClientWithPort(sc)
	if err != nil {
		return nil, err
	}
	return client, nil
}

// NewClientWithPort returns a client containing an SSHConfig struct and allows
// for a custom port to be set on the goph.Client.
func NewClientWithPort(sc *Config) (*goph.Client, error) {
	cf := goph.Config{
		Auth: sc.Auth,
		User: sc.Username,
		Addr: sc.HostAddress,
		Port: uint(sc.Port),
		Timeout: goph.DefaultTimeout,
		Callback: ssh.InsecureIgnoreHostKey(),
	}
	cl, err := goph.NewConn(&cf)
	// todo(dan) need to handle dead servers otherwise 60s + timeout before EHOSTUNREACH(113)
	if err != nil {
		return nil, err
	}
	return cl, nil
}

type ReturnedError struct {
	Status string
	Reason string
	Detail string
	Code int32
}

Tags:

#go #ssh

goa.design custom zerolog logger with HTTP Info, Warn and Error codes

Sun, 21 Apr 2024 00:00:00 +0000

goa.design custom zerolog logger with HTTP Info, Warn and Error codes

By default the goa logger isn’t great IMO. I like zerolog and have created a custom logger.

You could also use slog or zap, basically anything as long as it has these methods to implement the Logger interface.

package logger

import (
	"fmt"
	"os"

	"github.com/rs/zerolog"
)

// Logger is an adapted zerologger
type Logger struct {
	*zerolog.Logger
}

func New(serviceName string, isDebug, isConsole bool) *Logger {
	logLevel := zerolog.InfoLevel
	if isDebug {
		logLevel = zerolog.DebugLevel
	}
	zerolog.SetGlobalLevel(logLevel)
	svclogger := zerolog.New(os.Stderr).With().Timestamp().Str("service", serviceName).Logger()
	if isConsole {
		svclogger = zerolog.New(zerolog.ConsoleWriter{Out: os.Stdout}).
			With().
			Timestamp().
			Str("service", serviceName).
			Logger()
	}
	return &Logger{&svclogger}
}

// Log is called by the log middleware to log HTTP requests key values
func (logger *Logger) Log(keyvals ...interface{}) error {
	fields := FormatFields(keyvals)
 // If this isn't here, all HTTP requests get a zerolog.Info() block
 // which is super confusing when a request is a 5xx so I've overridden it
	if status, ok := fields["status"].(int); ok {
		s := status / 100
		switch s {
		case 4:
			logger.Warn().Fields(fields).Msgf("HTTP Request")
			return nil
		case 5:
			logger.Error().Fields(fields).Msgf("HTTP Request")
			return nil
		}
	}
	logger.Info().Fields(fields).Msgf("HTTP Request")
	return nil
}

// FormatFields formats input keyvals
// ref: https://github.com/goadesign/goa/blob/v1/logging/logrus/adapter.go#L64
func FormatFields(keyvals []interface{}) map[string]interface{} {
	n := (len(keyvals) + 1) / 2
	res := make(map[string]interface{}, n)
	for i := 0; i < len(keyvals); i += 2 {
		k := keyvals[i]
		var v interface{}
		if i+1 < len(keyvals) {
			v = keyvals[i+1]
		}
		res[fmt.Sprintf("%v", k)] = v
	}
	return res
}

Tags:

#zerolog #go #goa

Client-go; kubernetes deployment,service and ingress

Tue, 16 Apr 2024 00:00:00 +0000

Client-go; kubernetes deployment,service and ingress

How to create a simple deployment exposed with an ingress in Kubernetes using the client-go SDK.

Preconditions:

k3s
traefik
echo.k3s.lcl mapped to local IP in /etc/hosts (if demo purposes only)

package main

import (
	"context"
	"fmt"
	"log"
	"log/slog"
	"os"

	"k8s.io/apimachinery/pkg/util/intstr"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	networkingv1 "k8s.io/api/networking/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/tools/clientcmd"
)

func main() {
	client := k8sClient()

	err := createDeployment(
		client,
		"default",
		"echo",
		2,
		"ealen/echo-server",
	)
	if err != nil {
		log.Fatal(err)
	}
	err = createService(
		client,
		"default",
		"echo-svc",
		3000,
	)
	if err != nil {
		log.Fatal(err)
	}
	err = createIngress(
		client,
		"default",
		"echo-ingress",
		3000,
	)
	if err != nil {
		log.Fatal(err)
	}
}

func createDeployment(
	client *kubernetes.Clientset,
	ns string,
	deploymentName string,
	replicas int32,
	image string,
) error {
	dc := client.AppsV1().Deployments(ns)

	dp := &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Name: deploymentName,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &replicas,
			Selector: &metav1.LabelSelector{
				MatchLabels: map[string]string{
					"app": deploymentName,
				},
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: map[string]string{
						"app": deploymentName,
					},
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Name: deploymentName,
							Image: image,
							Ports: []corev1.ContainerPort{
								{
									Name: deploymentName,
									Protocol: corev1.ProtocolTCP,
									ContainerPort: 8888,
								},
							},
						},
					},
				},
			},
		},
	}

	slog.Info("deployment", "deployment", deploymentName, "status", "creating")
	result, err := dc.Create(context.Background(), dp, metav1.CreateOptions{})
	if err != nil {
		return err
	}
	slog.Info("deployment", "deployment", result.GetObjectMeta().GetName(), "status", "created")
	return nil
}

func createService(client *kubernetes.Clientset, ns string, serviceName string, port int32) error {
	svc := &corev1.Service{
		ObjectMeta: metav1.ObjectMeta{
			Name: serviceName,
			Labels: map[string]string{
				"app": "echo",
			},
		},
		Spec: corev1.ServiceSpec{
			Ports: []corev1.ServicePort{
				{
					Port: port,
					TargetPort: intstr.IntOrString{Type: intstr.Int, IntVal: 80},
				},
			},
			Selector: map[string]string{
				"app": "echo",
			},
		},
	}

	slog.Info("service", "service", serviceName, "status", "creating")
	service, err := client.CoreV1().Services(ns).Create(context.TODO(), svc, metav1.CreateOptions{})
	if err != nil {
		panic(err)
	}
	slog.Info("service", "service", service.GetObjectMeta().GetName(), "status", "created")
	return nil
}
func Ptr[T any](v T) *T {
	return &v
}

func createIngress(client *kubernetes.Clientset, ns string, ingressName string, port int32) error {
	ingress := &networkingv1.Ingress{
		ObjectMeta: metav1.ObjectMeta{
			Name: ingressName,
			Namespace: ns,
		},
		Spec: networkingv1.IngressSpec{
			IngressClassName: Ptr("traefik"),
			Rules: []networkingv1.IngressRule{
				{
					Host: "echo.k3s.lcl",
					IngressRuleValue: networkingv1.IngressRuleValue{
						HTTP: &networkingv1.HTTPIngressRuleValue{
							Paths: []networkingv1.HTTPIngressPath{
								{
									Path: "/",
									PathType: Ptr(networkingv1.PathTypePrefix),
									Backend: networkingv1.IngressBackend{
										Service: &networkingv1.IngressServiceBackend{
											Name: "echo-svc",
											Port: networkingv1.ServiceBackendPort{
												Number: port,
											},
										},
									},
								},
							},
						},
					},
				},
			},
		},
	}
	slog.Info("ingress", "ingress", ingressName, "status", "creating")
	result, err := client.NetworkingV1().
		Ingresses(ingress.Namespace).
		Create(context.TODO(), ingress, metav1.CreateOptions{})
	if err != nil {
		panic(err)
	}
	slog.Info("ingress", "ingress", result.GetObjectMeta().GetName(), "status", "created")
	return nil
}

func k8sClient() *kubernetes.Clientset {
 // If you are using $HOME/.kube/config uncomment this and remove the
 // os.Getenv("KUBECONFIG") line
	//userHomeDir, err := os.UserHomeDir()
	//if err != nil {
	//	fmt.Printf("error getting user home dir: %v\n", err)
	//	os.Exit(1)
	//}
	//kubeConfigPath := filepath.Join(userHomeDir, ".kube", "config")
	kubeConfigPath := os.Getenv("KUBECONFIG")
	fmt.Printf("Using kubeconfig: %s\n", kubeConfigPath)

	kubeConfig, err := clientcmd.BuildConfigFromFlags("", kubeConfigPath)
	if err != nil {
		fmt.Printf("Error getting kubernetes config: %v\n", err)
		os.Exit(1)
	}

	clientset, err := kubernetes.NewForConfig(kubeConfig)
	if err != nil {
		fmt.Printf("error getting kubernetes config: %v\n", err)
		os.Exit(1)
	}
	return clientset
}

Run with go run . and if you have a connection to the cluster, it should output:

Using kubeconfig: /etc/rancher/k3s/k3s.yaml
2024/04/16 22:07:07 INFO deployment deployment=echo status=creating
2024/04/16 22:07:07 INFO deployment deployment=echo status=created
2024/04/16 22:07:07 INFO service service=echo-svc status=creating
2024/04/16 22:07:07 INFO service service=echo-svc status=created
2024/04/16 22:07:07 INFO ingress ingress=echo-ingress status=creating
2024/04/16 22:07:07 INFO ingress ingress=echo-ingress status=created

If your /etc/hosts file has an entry that has your local IP pointing at echo.k3s.lcl then you should be able to curl the pod.

# example of what /etc/hosts should look like on your host machine
192.168.0.1 echo.k3s.lcl

Output from curl’ing echo.k3s.lcl:

# curl echo.k3s.lcl | jq .
{
 "host": {
 "hostname": "echo.k3s.lcl",
 "ip": "::ffff:10.42.0.1",
 "ips": []
 },
 "http": {
 "method": "GET",
 "baseUrl": "",
 "originalUrl": "/",
 "protocol": "http"
 },
 "request": {
 "params": {
 "0": "/"
 },
 "query": {},
 "cookies": {},
 "body": {},
 "headers": {
 "host": "echo.k3s.lcl",
 "user-agent": "curl/7.81.0",
 "accept": "*/*",
 "x-forwarded-for": "10.42.0.1",
 "x-forwarded-host": "echo.k3s.lcl",
 "x-forwarded-port": "80",
 "x-forwarded-proto": "http",
 "x-forwarded-server": "traefik-5df5cdc88d-b9tsn",
 "x-real-ip": "10.42.0.1",
 "accept-encoding": "gzip"
 }
 },
 "environment": {
 "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
 "HOSTNAME": "echo-68cd9fb7bd-zxjmd",
 "NODE_VERSION": "20.11.0",
 "YARN_VERSION": "1.22.19",
 "ECHO_SVC_PORT_3000_TCP": "tcp://10.43.41.1:3000",
 "ECHO_SVC_PORT_3000_TCP_PORT": "3000",
 "KUBERNETES_PORT": "tcp://10.43.0.1:443",
 "KUBERNETES_SERVICE_PORT_HTTPS": "443",
 "KUBERNETES_PORT_443_TCP_PROTO": "tcp",
 "KUBERNETES_PORT_443_TCP_PORT": "443",
 "KUBERNETES_PORT_443_TCP_ADDR": "10.43.0.1",
 "ECHO_SVC_PORT_3000_TCP_PROTO": "tcp",
 "KUBERNETES_SERVICE_HOST": "10.43.0.1",
 "ECHO_SVC_SERVICE_HOST": "10.43.41.1",
 "ECHO_SVC_PORT_3000_TCP_ADDR": "10.43.41.1",
 "KUBERNETES_PORT_443_TCP": "tcp://10.43.0.1:443",
 "ECHO_SVC_SERVICE_PORT": "3000",
 "ECHO_SVC_PORT": "tcp://10.43.41.1:3000",
 "KUBERNETES_SERVICE_PORT": "443",
 "HOME": "/root"
 }
}

Updated

For resources which are not kubernetes primitives such as Traefik, you can use a Dynamic Client to create unstructured.Unstructired{}.

In this example we’re replacing the networking.k8s.io Ingress with a traefik.io/v1alpha1 IngressRoute.

func k8sDynClient() *dynamic.DynamicClient {
	kubeConfigPath := os.Getenv("KUBECONFIG")
	fmt.Printf("Using kubeconfig: %s\n", kubeConfigPath)

	kubeConfig, err := clientcmd.BuildConfigFromFlags("", kubeConfigPath)
	if err != nil {
		fmt.Printf("Error getting kubernetes config: %v\n", err)
		os.Exit(1)
	}

 // Everything is the same as k8sClient except this line.
	clientset, err := dynamic.NewForConfig(kubeConfig)
	if err != nil {
		fmt.Printf("error getting kubernetes config: %v\n", err)
		os.Exit(1)
	}
	return clientset
}


func createDynamicIngressRoute(client *dynamic.DynamicClient, ns string, ingressName string, port int32) error {
	ingress := &unstructured.Unstructured{
		Object: map[string]interface{}{
			"apiVersion": "traefik.io/v1alpha1",
			"kind": "IngressRoute",
			"metadata": map[string]interface{}{
				"name": ingressName,
				"namespace": ns,
			},
			"spec": map[string]interface{}{
				"entryPoints": []interface{}{
					"web",
				},
				"routes": []interface{}{
					map[string]interface{}{
						"match": "Host(`echo.k3s.lcl`) && PathPrefix(`/`)",
						"kind": "Rule",
						"services": []interface{}{
							map[string]interface{}{
								"name": "echo-svc",
								"port": port,
								"namespace": "default",
								"kind": "Service",
							},
						},
					},
				},
			},
		},
	}
	slog.Info("ingress", "ingress", ingressName, "status", "creating")
	result, err := client.Resource(schema.GroupVersionResource{
		Group: "traefik.io",
		Version: "v1alpha1",
		Resource: "ingressroutes",
	}).Namespace(ns).Create(context.TODO(), ingress, metav1.CreateOptions{})
	if err != nil {
		return err
	}
	slog.Info("ingress", "ingress", result.GetName(), "status", "created")
	return nil
}

Tags:

#kubernetes #go

Golang: Time.In panic missing Location

Mon, 08 Apr 2024 00:00:00 +0000

Golang: Time.In panic missing Location

This worked fine in development on my machine.

func isToday(d Date) bool {
	t := time.Now().UTC()
	tz := "Australia/Brisbane"
	localTZ, err := time.LoadLocation(tz)
	if err != nil {
		slog.Error("timezone", "error", err)
		t = t.In(localTZ)
	}
	if d.DayMonth.Day() == t.Day() && d.DayMonth.Month() == t.Month() {
		slog.Info("date match", "date", d.DayMonth, "tz", tz)
		return true
	}
	return false
}

But in production produced:

2024-04-08T08:33:35Z 2024/04/08 08:33:35 ERROR timezone error="unknown time zone Australia/Brisbane"
2024-04-08T08:33:35Z 2024/04/08 08:33:35 http: panic serving 127.0.0.1:46164: time: missing Location in call to Time.In

Turns out, if Go can’t retrieve timezones from the machine it’ll panic. You can force Go to embed timezones into the application though by importing time/tzdata.


import _ "time/tzdata"


func isToday(d Date) bool {
	t := time.Now().UTC()
	tz := "Australia/Brisbane"
	localTZ, err := time.LoadLocation(tz)
	if err != nil {
		slog.Error("timezone", "error", err)
		t = t.In(localTZ)
	}
	if d.DayMonth.Day() == t.Day() && d.DayMonth.Month() == t.Month() {
		slog.Info("date match", "date", d.DayMonth, "tz", tz)
		return true
	}
	return false
}

Now it works!

Tags:

#TIL #time #go

Caddy and CORS

Sat, 30 Mar 2024 00:00:00 +0000

Caddy and CORS

A quick Caddy CORS snippet


(cors) {
	@cors_preflight method OPTIONS
	@cors header Origin {args.0}

	handle @cors_preflight {
		header Access-Control-Allow-Origin "{args.0}"
		header Access-Control-Allow-Methods "GET, POST, PUT, PATCH, DELETE"
		header Access-Control-Allow-Headers "Content-Type, Authorization"
		header Access-Control-Max-Age "3600"
		respond "" 204
	}

	handle @cors {
		header Access-Control-Allow-Origin "{args.0}"
		header Access-Control-Expose-Headers "Link"
	}
}

server1 {
	encode gzip
	tls internal
	import cors https://server2
	reverse_proxy https://192.168.20.10:5000 {
		transport http {
			tls_insecure_skip_verify
		}
	}
}

Tags:

#caddy #cors #til

nextdns rewrites

Sat, 30 Mar 2024 00:00:00 +0000

nextdns rewrites

I was humbled today; I thought nextdns rewrites meant you could point all queries to a DNS server.

For instance, *.foo.bar -> 192.168.20.20 (private LAN DNS server) and that would then pass on the request to the DNS server.

It doesn’t work like this. Instead you have to map each host to the LAN host address/IP (or external).

There’s no API for this either so for my 15 internal domains I have to enter them in the GUI.

I have several profiles and its a massive PITA. Now I have one profile for all nextdns enabled devices so that I only have to change one place.

A little disappointed about this not being exposed via an API (and thus scriptable).

Tags:

#TIL #dns #nextdns

Home server notes

Fri, 29 Mar 2024 00:00:00 +0000

Home server notes

Collecting issues that I hit along the way to bootstrapping my new proxmox homelab.

Paperless-ngx

For a reverse proxy to work you need to override the PAPERLESS_URL variable. To update this, open /opt/paperless/paperless.conf in the LXC container.

Caddy

Caddy doesn’t come with a convenient LXC container. Instead I created a ubuntu 22.04 VM and installed Caddy as a systemd unit. This makes saving the Caddyfile hard so I recommend saving the output once sorted and then configuring it to be API driven thereafter.

To make a server not request a HTTPS certificate automatically then use http:// in the server name block.

Proxmox Networking

Make sure IPv6 is set to Static and empty. If set to DHCP and your router doesn’t provide it boot times increase dramatically.

Set static IPv4, especially for Cloudflared LXC or /etc/resolv.conf will get the gateway instead of DNS server. The server will fail to connect to Cloudflare and /etc/resolv.conf will always be overwritten by DHCP.

Also make sure all services/VMs have a static IP so when the server reboots it doesn’t give new IPs rendering the proxy useless. Do this in the GUI and not in the service as it’ll be overwritten by cloud-init.

Goa openapi embedded openapi documentation

Tue, 26 Mar 2024 00:00:00 +0000

Goa openapi embedded openapi documentation

A snippet for how I am storing an OpenAPI documentation page within a Goa project docker image.

`design/openapi.go`

// design/openapi.go
package design

import . "goa.design/goa/v3/dsl"

var _ = Service("openapi", func() {
	Description("OpenAPI endpoints for debugging and demonstration")
	HTTP(func() {
		Path("/openapi")
	})
	Method("file", func() {
		Result(func() {
			Attribute("length", Int64, "Length is the downloaded content length in bytes.", func() {
				Example(4 * 1024 * 1024)
			})
			Attribute("encoding", String, func() {
				Example("application/json")
			})
			Required("length", "encoding")
		})

		Error("invalid_file_path", ErrorResult, "Could not locate file for download")
		Error("internal_error", ErrorResult, "Fault while processing download.")

		HTTP(func() {
			GET("/openapi3.json")
			SkipResponseBodyEncodeDecode()
			Response(func() {
				Header("length:Content-Length")
				Header("encoding:Content-Type")
			})
			Response("invalid_file_path", StatusNotFound)
			Response("internal_error", StatusInternalServerError)
		})
	})
	Method("documentation", func() {

		Result(func() {
			Attribute("length", Int64, "Length is the downloaded content length in bytes.", func() {
				Example(4 * 1024 * 1024)
			})
			Attribute("encoding", String, func() {
				Example("application/json")
			})
			Required("length", "encoding")
		})

		Error("invalid_file_path", ErrorResult, "Could not locate file for download")
		Error("internal_error", ErrorResult, "Fault while processing download.")

		HTTP(func() {
			GET("/docs")
			SkipResponseBodyEncodeDecode()
			Response(func() {
				Header("length:Content-Length")
				Header("encoding:Content-Type")
			})
			Response("invalid_file_path", StatusNotFound)
			Response("internal_error", StatusInternalServerError)
		})
	})
})

`openapi.go`

The example generated from the above DSL

package goagithub

import (
	"context"
	"embed"
	"github.com/danielmichaels/goa-github/gen/openapi"
	"io"
	"log"
)

//go:embed gen/http/openapi3.json
var openapijson embed.FS

//go:embed assets/static/docs.html
var assets embed.FS

// openapi service example implementation.
// The example methods log the requests and return zero values.
type openapisrvc struct {
	logger *log.Logger
}

// NewOpenapi returns the openapi service implementation.
func NewOpenapi(logger *log.Logger) openapi.Service {
	return &openapisrvc{logger}
}

func (o openapisrvc) File(ctx context.Context) (res *openapi.FileResult, body io.ReadCloser, err error) {
	f, err := openapijson.Open("gen/http/openapi3.json")
	if err != nil {
		return nil, nil, openapi.MakeInvalidFilePath(err)
	}
	fi, err := f.Stat()
	if err != nil {
		return nil, nil, openapi.MakeInternalError(err)
	}
	return &openapi.FileResult{
		Length: fi.Size(),
		Encoding: "application/json",
	}, f, nil
}

func (o openapisrvc) Documentation(ctx context.Context) (res *openapi.DocumentationResult, body io.ReadCloser, err error) {
	f, err := assets.Open("assets/static/docs.html")
	if err != nil {
		return nil, nil, openapi.MakeInvalidFilePath(err)
	}
	fi, err := f.Stat()
	if err != nil {
		return nil, nil, openapi.MakeInternalError(err)
	}
	return &openapi.DocumentationResult{
		Length: fi.Size(),
		Encoding: "text/html",
	}, f, nil
}

`assets/static/docs.html`

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <title>Open API</title>
 <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swagger-ui@5.10.0/dist/swagger-ui.css" integrity="sha256-IgNmYaATqY6z6AuD6hvz9XN0OyeAc94gsTa+lK8ka1Y=" crossorigin="anonymous">
 <style>
 /* Fast dark mode https://github.com/swagger-api/swagger-ui/issues/5327 */
 @media (prefers-color-scheme: dark) {
 body {
 background: #1f1f1f;
 }
 .swagger-ui {
 filter: invert(88%) hue-rotate(180deg);
 }
 .swagger-ui .microlight {
 filter: invert(100%) hue-rotate(180deg);
 }
 }
 </style>
</head>
<body>
<div id="swagger-ui"></div>
<script src="https://cdn.jsdelivr.net/npm/swagger-ui@5.10.0/dist/swagger-ui-bundle.js" integrity="sha256-i050FsZ0MSwm3mVMv7IhpfCdK90RKaXPS/EmiWxv8vc=" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/swagger-ui@5.10.0/dist/swagger-ui-standalone-preset.js" integrity="sha256-IGoJVXW7MRyeZOsKceWVePAShfVpJhnYhDhOQp+Yi38=" crossorigin="anonymous"></script>
<script>
 // Initialize Swagger UI
 window.onload = function() {
 const ui = SwaggerUIBundle({
 url: "/openapi/openapi3.json", // Provide the URL to your OpenAPI spec file
 dom_id: '#swagger-ui',
 deepLinking: true,
 presets: [
 SwaggerUIBundle.presets.apis,
 SwaggerUIStandalonePreset
 ],
 layout: "BaseLayout"
 });
 window.ui = ui;
 };
</script>
</body>
</html>

Tags:

#goa #dsl #go

GH Actions versus GitlabCI

Sun, 24 Mar 2024 00:00:00 +0000

GH Actions versus GitlabCI

I am convinced that people persist with GH actions because its simply available on Github.

It’s objectively worse in everyway than GitlabCI. I’ve used Drone and Woodpecker which are also ten times the product that GH Actions is.

Can’t use custom container images! What a PITA.

I make heavy use of “god” images for CI. These are images that have everything you need for all your CI activities. Think terraform, packer, taskfile, python, golang and so on.

It creates a huge image but it also means I only have to pull the image and then everything is available.

In Github, I have to run a step and then install that stuff (go and python, are available though). This means my pipelines are spent installing things like golines, betteralign and go-task - for every single invocation.

In Gitlab I pull the image once, its cached on the runner and boom, it fires up the pipeline and have access to all those things immediately.

Gitlab is better, as are any other providers who don’t work the same as Github.

Tags:

#rant #ci #github

Low effort issues and tickets, a cultural problem

Mon, 18 Mar 2024 00:00:00 +0000

Low effort issues and tickets, a cultural problem

Teams who lack any form a control by seniors seem to develop members who offer low effort and poor quality issues, tickets and PR’s - especially in young males.

I try to always set a high standard - which is actually just the standard.

Issues have at a minimum what, why, when and how. It will always have an end state.

Pull requests will always have at least:

What this is doing or why this exists, linking to tickets if available
How it can be run/tested
If required, diagrams/mermaid.js and so on.

I always act as though every ticket and PR could be:

Reviewed by someone unfamiliar with the code (new team member, for instance)
Used as reference in the future, rationale or searching for when bugs where introduced

Unfortunately, in several teams, I come across a culture of laziness where these “non-code” tasks are taken for granted.

Now I refuse to review, or pick up tickets unless they have more details. I’ll just comment (nicely) that more information is needed and when that is provided I’ll move forward with this task.

This works for me because I am typically afforded that level of leniency within the roles I chose.

But, for juniors, they don’t or can’t and will start to emulate poor performers who don’t exhibit the standard expected of all developers. Couple this with a team where no one is given guidelines or expectations and its a recipe for disaster.

A team culture is everything and it must be set by the seniors.

A couple military gem’s which are corny but ultimately true when it comes to standards and leadership:

It’s called leadership not like-a-ship
The standard you walk past is the standard you accept

I’ll do better to encourage people into making an effort with these small but significant pieces of the software development teamwork puzzle. I hope you all do too.

Tags:

#leadership #teams

MITM proxying for request inspection

Mon, 04 Mar 2024 00:00:00 +0000

MITM proxying for request inspection

I was listening to an old Running in Production podcast episode in which the developer reversed engineered API’s using Fiddler.

I’d used Fiddler briefly and found it cumbersome, along with mitmproxy.

So on a whim I searched and found two competing but excellent alternatives:

Both offer very comparable features for my use case. For simple use its splitting hairs and likely comes to down to the number of methods they support and their ergonomics.

For instance, Requestly is better to look at in my opinion but HTTP-Toolkit is no frills and just works.

Listening to the podcast was eye opening because I’d been working on a problem in my day job which required request inspection. Wireshark was too heavy handed and all the HTTP traffic was encrypted plus I can’t filter easily by only trapping certain processes, e.g. my current terminal.

With a MITM proxy I was instantly able to see every request unencrypted. Within seconds I could see that requests that we thought we going out weren’t.

If you need to track network requests unencrypted from a process you control be it a browser or terminal, you should be using a MITM proxy to intercept it.

Tags:

#TIL #mitm

Golang: title casing first character only

Sat, 17 Feb 2024 00:00:00 +0000

Golang: title casing first character only

Technically I think title case means all major words have their first letter capitalised.

This isn’t that; its capitalising the first work only.

I use this when rendering error fields to users. I like to keep all my errors consistent and lower cased. This means some extra work when presenting errors to users, for instance form field errors.

package main

import (
	"fmt"
	"strings"
)

func WithTitleCase(s string) string {
	return strings.ToUpper(s[:1]) + s[1:]
}

func main() {
	sentance := "today was a good day"
	newSentance := WithTitleCase(sentance)
	fmt.Println(newSentance)
}

go play: https://go.dev/play/p/r2auIbxsiRH?v=

Tags:

#go

AsyncClient: python quick example

Fri, 26 Jan 2024 00:00:00 +0000

AsyncClient: python quick example

I have started a new job in another python shop.

Over the last 6-12 months, I wrote a lot more Go and yaml than python as my primary role in my last job was kubernetes and infra management. So I am reacquainting myself in python.

This is a quick async snippet for future reference, stolen mostly from Rednafi’s blog post and changed to suit my needs.

This code will fetch multiple product_id’s from an API using asyncio and httpx. Pydantic is used to marshall things nicely.

import httpx
from pydantic import BaseModel
import asyncio

urls = [1, 2, 3, 4, 5, 6, 7, 8, 9]


async def fetch_data(product_id: int):
 async with httpx.AsyncClient() as client:
 response = await client.get(f"https://dummyjson.com/products/{product_id}")
 response.raise_for_status()
 data = response.json()
 product = Product(**data)
 print(f"{product.id}: {product.title}, {product.description}")


class Product(BaseModel):
 id: int
 title: str
 description: str


async def main() -> None:
 tasks = []
 for _id in urls:
 tasks.append(fetch_data(_id))
 await asyncio.gather(*tasks)


if __name__ == "__main__":
 asyncio.run(main())

Example output:

2: iPhone X, SIM-Free, Model A19211 6.5-inch Super Retina HD display with OLED technology A12 Bionic chip with ...
4: OPPOF19, OPPO F19 is officially announced on April 2021.
3: Samsung Universe 9, Samsung's new variant which goes beyond Galaxy to the Universe
6: MacBook Pro, MacBook Pro 2021 with mini-LED display may launch between September, November
9: Infinix INBOOK, Infinix Inbook X1 Ci3 10th 8GB 256GB 14 Win10 Grey – 1 Year Warranty
1: iPhone 9, An apple mobile which is nothing like apple
8: Microsoft Surface Laptop 4, Style and speed. Stand out on HD video calls backed by Studio Mics. Capture ideas on the vibrant touchscreen.
7: Samsung Galaxy Book, Samsung Galaxy Book S (2020) Laptop With Intel Lakefield Chip, 8GB of RAM Launched
5: Huawei P30, Huawei’s re-badged P30 Pro New Edition was officially unveiled yesterday in Germany and now the device has made its way to the UK.

Tags:

#python #async #httpx

HTMX and Echo; rendering HTML

Thu, 18 Jan 2024 00:00:00 +0000

HTMX and Echo; rendering HTML

I am experimenting with replacing a Next.js frontend with HTMX (and Templ). Typically, I use chi as my router but this time I’ve opted for echo.

For one reason, by default it uses context and all handlers must return an error. This prevents any missed returns when error handling (which can be caught with linters but its an extra step).

I’m used to html/template but getting it to work with echo was a different.

Following the guide and adding my own spin I came up with the following.

type Template struct {
	templates *template.Template
}

func NewTemplate(patterns []string) *Template {
	for i := range patterns {
 // assets/view/ contains all my templates
		patterns[i] = "view/" + patterns[i]
	}

	ts, err := template.New("").Funcs(nil).ParseFS(assets.EmbeddedFiles, patterns...)
	if err != nil {
		log.Fatal().Err(err).Msg("template.New err")
	}

	return &Template{templates: ts}
}

func (t *Template) Render(w io.Writer, name string, data interface{}, c echo.Context) error {
	return t.templates.ExecuteTemplate(w, name, data)
}

func main() {
 e := echo.New()
	e.Use(middleware.Recover())
 // this is how Echo learns about our templates.
 // I can pass in multiple locations for templates but I only care about
 // the HTMX fragments for this renderer - everything else is handled by Templ
	e.Renderer = NewTemplate([]string{"fragments/*.tmpl"})
 // all routes are loaded here
 app.routes(e)
	srv := &http.Server{
		Addr: fmt.Sprintf(":%d", app.Config.Server.Port),
		Handler: e,
		IdleTimeout: app.Config.Server.TimeoutIdle,
		ReadTimeout: app.Config.Server.TimeoutRead,
		WriteTimeout: app.Config.Server.TimeoutWrite,
	}
	app.Logger.Info().Msgf("Started Webserver on '%d'", app.Config.Server.Port)
	e.Logger.Fatal(srv.ListenAndServe())

Now I can render HTML (with HTMX) using Echo.

Tags:

#til #go #echo #templates

qemu kvm: operation not permitted fix

Thu, 18 Jan 2024 00:00:00 +0000

qemu kvm: operation not permitted fix

When starting a bridged VM and hit with:

Error starting domain: internal error: /usr/libexec/qemu-bridge-helper --use-vnet --br=virbr0 --fd=20: failed to communicate with bridge helper: Transport endpoint is not connected
stderr=failed to create tun device: Operation not permitted

It is likely that qemu has been updated and changed the bridge helper permissions.

Fix it with:

sudo chmod u+s /usr/lib/qemu/qemu-bridge-helper

Tags:

#qemu #vm #kvm

GoLand and templ with autocomplete

Tue, 16 Jan 2024 00:00:00 +0000

GoLand and templ with autocomplete

To use templ with autocomplete via its Plugin you must do the following:

Go to settings>templ and enter the path to the templ binary (e.g. $HOME/.local/bin/templ)
Close all GoLand instances
Launch GoLand via the terminal inside the project directory (e.g. goland .)

It works but still needs some refining at the project level.

The alternative is to use neovim for the templ files or VSCode. As a GoLand subscriber these are sub par alternatives but may be required in the short term.

Tags:

#go #templ

Design First API Development with Goa

Tue, 09 Jan 2024 00:00:00 +0000

Goa.design is a golang tool for developing APIs using a design first approach. By leveraging Goa it is possible to generate server and client code automatically, documentation through OpenAPI (version 2 and 3 are supported) as well as gRPC code.

This blog post introduces Goa, it’s concepts and showcases some short examples. It does not walk through the installation, or intend to supersede its tutorial/walkthrough which you should look over, here

Goa is broken into three parts; the design language (DSL), code generation and the Go package itself.

What is a DSL?

A DSL (Domain-Specific Language) is a programming language or a set of rules and syntax specifically designed to solve problems within a particular domain or industry. It provides a higher-level abstraction that allows programmers to express solutions in a more concise and declarative manner.

The above statement sums it up succinctly. In our case the domain or industry specifics relate to HTTP and RPC communication via network calls. A lot of API work is boilerplate and can be hard to implement all the things considered best practise. Goa strives to reduce the amount of programmer work required to build out well crafted APIs.

Instead of writing out an OpenAPI document and then converting it to Go code such as oapi-codegen, Goa uses its DSL to generate it (and the code).

Personally, I find the DSL quite powerful and easy to understand. What’s more, it starts simple but provides rich ways to extend it as your requirements dictate. The creator (raphael) is also very active on github and golang’s slack.

Here’s a snippet of the DSL.

package design

import . "goa.design/goa/v3/dsl"

// API describes the global properties of the API server.
var _ = API("check-redirects", func() {
	Title("HTTP Redirection Detection Service")
	Description("HTTP service detecting and reporting any and all redirects in a HTTP request")
	Server("server", func() {
		Host("localhost", func() { URI("http://localhost:9090") })
	})
})

var _ = Service("health", func() {
	Description("endpoints for determining service uptime and status")
	HTTP(func() {
		Path("/")
	})
	Method("healthz", func() {
		HTTP(func() {
			GET("/healthz")
			Response(StatusOK)
		})
		Result(Empty)
	})
	Method("version", func() {
		HTTP(func() {
			GET("/version")
			Response(StatusOK)
		})
		Result(AppVersion)
	})
})
var AppVersion = Type("version", func() {
	Description("Application version information")
	Attribute("version", String, "Application version", func() {
		Example("1.0")
		Example("6b51bebe0f965a5fffa8ff9db5aa702c76ec47f2")
	})
})

Without going too deep right now, this will create an API called server, a service called health and within the health service build two endpoints; healthz and version. A custom type called version will also be created and used in the /version endpoint.

The DSL provides an abstraction which lets you craft your API in a declarative way. One big benefit is its just Go code meaning you can easily extend, simplify or create generic functions when working with it.

Code generation

Writing the DSL by itself does not generate code or documentation. The goa CLI does that. Specifically, goa gen.

After installing Goa, you can generate the client and server code.

To generate the code run goa gen. Typically, Goa suggests a pattern of placing all DSL files into a directory called design. You don’t have to do this, but I think it makes sense for most use cases.

For the above code snippet, if it were at this path design/design.go you would run goa gen github.com/danielmichaels/checkredirects/design. In this example github.com/danielmichaels/checkredirects is my module that I used during go mod init.

Example generation

With this directory structure:

.
├── design
│   └── design.go
├── go.mod
└── go.sum

After I run goa gen github.com/danielmichaels/checkredirects/design, it will create a directory called gen with the following files:

├── design
│   └── design.go
├── gen # New
│   ├── health
│   │   ├── client.go
│   │   ├── endpoints.go
│   │   └── service.go
│   └── http
│   ├── cli
│   │   └── server
│   │   └── cli.go
│   ├── health
│   │   ├── client
│   │   │   ├── client.go
│   │   │   ├── cli.go
│   │   │   ├── encode_decode.go
│   │   │   ├── paths.go
│   │   │   └── types.go
│   │   └── server
│   │   ├── encode_decode.go
│   │   ├── paths.go
│   │   ├── server.go
│   │   └── types.go
│   ├── openapi3.json
│   ├── openapi3.yaml
│   ├── openapi.json
│   └── openapi.yaml
├── go.mod
└── go.sum

Now we have our Go code which we can import into our services. Note, this is all auto generated and should not be edited as it’ll be overwritten whenever we run goa gen.

Creating the services

Now that the package code has been generated we can create the entrypoint, and service files automatically with another command; goa example. Running this results in some new files:

.
├── cmd # New
│   ├── server
│   │   ├── http.go
│   │   └── main.go
│   └── server-cli
│   ├── http.go
│   └── main.go
├── design
│   └── design.go
├── gen # Truncated for brevity
├── go.mod
├── go.sum
└── health.go # New

Unlike goa gen this is a one-shot deal; if the generated files already exist it will not re-create them. This is because all your business logic will be inside these files and it may override things you don’t want overridden.

If we peek at health.go it will have stubbed out all the handlers, ready to be populated which your business logic.

package checkredirects

import (
	"context"
	"log"

	health "github.com/danielmichaels/checkredirects/gen/health"
)

// health service example implementation.
// The example methods log the requests and return zero values.
type healthsrvc struct {
	logger *log.Logger
}

// NewHealth returns the health service implementation.
func NewHealth(logger *log.Logger) health.Service {
	return &healthsrvc{logger}
}

// Healthz implements healthz.
func (s *healthsrvc) Healthz(ctx context.Context) (err error) {
	s.logger.Print("health.healthz")
	return
}

// Version implements version.
func (s *healthsrvc) Version(ctx context.Context) (res *health.Version2, err error) {
	res = &health.Version2{}
	s.logger.Print("health.version")
	return
}

So after writing only 40 lines (design/design.go) we were able to auto generate a complete and working web server with two endpoints.

The responses as expected do not return anything but work as demonstrated below.

$ curlie :9090/healthz
HTTP/1.1 200 OK
Date: Tue, 09 Jan 2024 04:44:21 GMT
Content-Length: 0

$ curlie :9090/version
HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 09 Jan 2024 04:44:28 GMT
Content-Length: 3

{
 
}

Documentation

Another great feature of Goa is how well it documents the code and that it can generate valid OpenAPI documents.

This is the document it created:

openapi: 3.0.3
info:
 title: HTTP Redirection Detection Service
 description: HTTP service detecting and reporting any and all redirects in a HTTP request
 version: 0.0.1
servers:
 - url: http://localhost:9090
paths:
 /healthz:
 get:
 tags:
 - health
 summary: healthz health
 operationId: health#healthz
 responses:
 "200":
 description: OK response.
 /version:
 get:
 tags:
 - health
 summary: version health
 operationId: health#version
 responses:
 "200":
 description: OK response.
 content:
 application/json:
 schema:
 $ref: '#/components/schemas/Version'
 example:
 version: 6b51bebe0f965a5fffa8ff9db5aa702c76ec47f2
components:
 schemas:
 Version:
 type: object
 properties:
 version:
 type: string
 description: Application version
 example: 6b51bebe0f965a5fffa8ff9db5aa702c76ec47f2
 example:
 version: 6b51bebe0f965a5fffa8ff9db5aa702c76ec47f2
tags:
 - name: health
 description: endpoints for determining service uptime and status

Goa also provides powerful constructs to enhance the documents. For example to define a field on a schema we can use Attribute or Field type. For this post we’re only focusing on HTTP which uses Attribute whereas Field is for both HTTP and gRPC.

Example of a Goa Payload and how we can add more context to it.

// Truncated
Payload(func() {
 Attribute("username", String, "Username", func () {
 Example("MyUsername")
		Pattern("^user_[a-zA-Z0-9]{12}$")
 })
 Required("password")
})
// Truncated

This will create an OpenAPI document which provides an example of MyUsername. The Pattern will also enforce the regex and automatically handle payload validation without the need to write any logic.

Conclusion

This post sought to introduce Goa in the most simple of terms. It hardly scratches the surface of its capabilities. I chose the most simplistic example I could because it will lay the groundwork for some follow-up posts which show how to add more realistic endpoints. In the next post I will create a service which accepts and returns JSON leveraging Goa’s Type, Error and Payload DSL primitives. This will be published with source code.

Daniel Michaels

Fri, 05 Jan 2024 00:00:00 +0000

Hey I’m Dan. I am now a software developer but in a past life was a soldier and served nearly a decade in the special forces. This is my outlet away from the day to day of working at a big corporation as a developer.

I mostly work with Python, Go and Javascript. These days I’ll reach for Go before anything else.

Most days, I can be found either trying to launch a profitable online business or hanging out with my family and friends. I love ice hockey (go bruins!), hunting and in general trying to be a little bit better than yesterday.

Why I love gjson

Thu, 04 Jan 2024 00:00:00 +0000

Why I love gjson

Easy parsing. That’s it. That’s the reason.

I have these structs

type ApiResponse struct {
	Message string `json:"message"`
	Status string `json:"status"`
	Code int `json:"code"`
	Return int `json:"return"`
}
type SystemDNSResponse struct {
	Data *pfsense.DNS `json:"data"`
	ApiResponse
}

When I make a request to the client I’ll get a response that looks something like this:

{
 "status": "ok",
 "code": 200,
 "return": 0,
 "message": "Success",
 "data": {
 "dnsserver": [
 "1.1.1.1",
 "8.8.8.8"
 ],
 "dnsallowoverride": false,
 "dnslocalhost": true
 }
}

I can marshall this into that struct easily using the built in encoding/json package.

But, I want a generic methods which can be passed in a struct and do the unmarshalling further down the line.

This is where gjson really shines; it can inspect []byte and output the string which I can then do something with. I don’t need any struct just the field name I want.

For example, I care about the code field. And have functions which will return errors based on its value:

// checkRawJsonStatusCode ensures that any non-200 status codes return an error
// from the firewall. The response from the firewall is not a http/net object
// so, we must manually inspect the code int and derive their meaning here.
func (s *pfsensesrvc) checkRawJsonStatusCode(code int) error {
	switch {
	case code >= 200 && code <= 299:
		break
	default:
		return fmt.Errorf("failed client validation of pfsense api: %d", code)
	}
	return nil
}

This function takes a string which would require unmarshalling to extract.

With gjson I can pass in the []byte and grab it out easily, like this:

// apiResponseCode returns the pfSense-api 'code' value from a successful API
// call to the client API.
func apiResponseCode(jsn []byte) int {
	const key = "code"
	c := gjson.GetBytes(jsn, key)
	return int(c.Num)
}

// MarshallAPIResponse parses weakly typed responses from the client device and marshall's
// it into a struct.
func (s *pfsensesrvc) MarshallAPIResponse(b []byte, result any) error {
 // truncated the rest of this method
	err = s.checkRawJsonStatusCode(apiResponseCode(b))
	if err != nil {
		return err
	}
	return nil
}

You can see that I grab the code field as an int which is passed in as an argument to checkRawJsonStatusCode.

I do it this way because in MarshallAPIResponse I only have to pass in []byte whereas if I used encoding/json I would need to be more explicit about the type.

Personally, I’ve found it to be invaluable, especially when creating an API which has many tens of endpoints, each returning their own types.

Tags:

#json #go #gjson

Linux captive portal - ubuntu

Sun, 31 Dec 2023 00:00:00 +0000

Linux captive portal - ubuntu

Whenever I fly I always have captive portal issues with my Linux laptop. The initial connection pops a captive portal but if I close the laptop and reopen it, it’ll kill the connection.

So far this has managed to get a reconnection.

sudo dhclient

Then navigate to https://captive.apple.com or neverssl.com so that it can re-open the captive portal.

Tags:

#qantas #captive-portal

New year, new challenges

Tue, 26 Dec 2023 00:00:00 +0000

New year, new challenges

Starting off 2024 by embracing the unknown.

We’re moving interstate, just bought our first home, having a baby and as of last week, now unemployed!

It’s interesting times in the household with a lot of exciting and daunting changes ahead. We can’t wait to move into our new house - it’s a nice house up the mountain. Lots of space for the kiddo’s, a pool and surrounded by nature.

The lack of job is a curveball though it should only be temporary (a contract error) but nonetheless is a blemish on the whole experience.

My family and friends think we’re crazy for buying a house whilst technically unemployed and moving and expecting another child. But, you gotta risk it for the biscuit!

Mind you, if anyone reading this needs some short term Go or python work - hit me up!

Otherwise, bring on 2024 as I can’t wait to move up to the Sunshine state (again).

Tags:

#life

Laravel DX is incredible

Sun, 17 Dec 2023 00:00:00 +0000

Laravel DX is incredible

Today, on a whim, I ran through the laravel tutorial.

I know zero PHP so I didn’t understand some of the syntax but the developer experience is absolutely first rate.

If I were to choose between Django and Laravel I would definitely reach for Laravel first. I didn’t even get into all the add-ons yet and I’ll already blown away.

The PHP itself didn’t seem so bad; people bemoan it hard but on the surface it seems pretty decent.

Might knock up a little app in it over the holidays and see how the full dev->deployment cycle goes.

One thing that blew me away is sail. It lets you run laravel and all of its ecosystem from within docker. I initially tried setting up PHP but hit various roadblocks before I found sail. I removed all PHP components and installed sail (which is really just a bash script and docker container). After that all of laravel’s CLI tools where magically embedded into sail. I don’t know another framework/language which does this or does it this well.

Tags:

#php #laravel

goa (goland) API design is better than oapi-codegen

Mon, 04 Dec 2023 00:00:00 +0000

goa (goland) API design is better than oapi-codegen

I’m a weekend in switching from deepmap/oapi-codegen to <goa.design> and couldn’t be happier.

So far it ticks the boxes the oapi-codegen provided but is much simpler to implement.

It uses a DSL instead of writing the openapi.yaml file. This DSL generates openapi version 2 and 3. It also generates gRPC protobuf files. I find the DSL quite idiomatic to use and beautiful to look at - unlike yaml.

Using goa so far has been much smoother and more expressive. Not to mention it forces better architecture and abstractions from the get go. I typically use a “god” struct for my application. Instead goa uses a service orientated approach except it generates 80% of it for you. It’s the opinionated yet open framework Go lacks.

So far really happy. Early days but will be recommending it to everyone.

Tags:

#go #goa #api

Looping over embedded structs in html template

Tue, 28 Nov 2023 00:00:00 +0000

Looping over embedded structs in html template

I am using the slack-go package and needed to pull out information from slacks Conversations API.

slack-go uses the following structs to store this data:

// Channel contains information about the channel
type Channel struct {
	GroupConversation
	IsChannel bool `json:"is_channel"`
	IsGeneral bool `json:"is_general"`
	IsMember bool `json:"is_member"`
	Locale string `json:"locale"`
}
// GroupConversation is the foundation for Group and Channel
type GroupConversation struct {
	Conversation
	Name string `json:"name"`
	Creator string `json:"creator"`
	IsArchived bool `json:"is_archived"`
	Members []string `json:"members"`
	Topic Topic `json:"topic"`
	Purpose Purpose `json:"purpose"`
}

Getting the conversation is easy;

s := slack.New(botToken)
	channels, _, _ := s.GetConversationsContext(ctx, &slack.GetConversationsParameters{
		TeamID: "id",
	})

And it returns []slack.Channel

In a html template I wanted to get all the available channels and put them into a <select>. It took a long time before I realised that GroupConversation is an embedded struct and thus you cannot reference it by a struct value.

To get the GroupConversation.Name from a slice of Channel in .tmpl I used the following loop:

<select class="select select-bordered w-full max-w-xs">
 <option disabled selected>Slack Channel</option>
 {{with .Channels}}
 {{range .}}
 <option>{{.Name}}</option>
 {{end}}
 {{end}}
 </select>

The {{ range . }} seems a little off putting to me; magical even. But, it works. I’m hoping there is a better way but this works quite well for now.

Tags:

#TIL #go

air.toml interrupts and kill_delay

Mon, 27 Nov 2023 00:00:00 +0000

air.toml interrupts and kill_delay

TIL that you can make air send system interrupt to the process when it reloads and have a waiting period.

This is really important for long running or async tasks which need to complete. e.g. sending a notification then writing to the DB.

# full air.toml for completeness.
root = "."
tmp_dir = "tmp"

[build]
bin = "./tmp/app serve"
cmd = "go build -o ./tmp/app ./cmd/app"
delay = 1000
exclude_dir = ["node_modules", "tmp", "vendor"]
exclude_file = []
exclude_regex = []
exclude_unchanged = false
follow_symlink = false
full_bin = ""
include_dir = []
include_ext = ["go", "tpl", "tmpl", "html", "css", "js"]
kill_delay = "2s" # here
send_interrupt = true # here
stop_on_error = true
log = "build-errors.log"

[color]
app = ""
build = "yellow"
main = "magenta"
runner = "green"
watcher = "cyan"

[log]
time = false

[misc]
clean_on_exit = false

Tags:

#TIL

Users don't read the manual

Thu, 09 Nov 2023 00:00:00 +0000

Users don’t read the manual

In my workplace I’ve written a couple of tools. Now its more than just me committing code to the project but I still do 90% of the ongoing work.

It bootstraps a local kubernetes cluster with all our teams resources and applications. We’re a dev team, not platform or infra. Most of them don’t know kubernetes which is why I created this tool.

It’s a pip install and within 3 minutes you’re entire dev env is stood up.

Yet, I get constant questions. Some of the answers are literally printed to the screen once the bootstrap step completes.

It’s hard not to get frustrated. On one hand, you can’t expect users to understand everything. On the other, there’s team members who do read the notices and take the time to investigate the tools --help before throwing their hands up saying “it doesn’t work”.

All I can do is try to educate and keep working on it to make it better.

Still, its a real epidemic in the open source world. I see so many issues on GitHub which are borderline aggressive towards project maintainers when their problem was literally solved in the --help text.

Tags:

#oss #people

Forking and contributing to a Go project

Mon, 06 Nov 2023 00:00:00 +0000

Forking and contributing to a Go project

I am using http://github.com/slack-go/slack for a project and it does not support Sign in with Slack OpenID connect.

After writing the connector manually I decided I’d try contributing upstream.

After forking and making the changes I thought would work I needed to test them against my working application and didn’t know how to use my fork instead of the actual package.

Thankfully, it’s pretty easy.

go mod edit -replace "github.com/slack-go/slack=github.com/danielmichaels/slack@openid-connect"

Doing this and a go mod tidy pulled in my branch and a minute later I’d proven my changes worked.

Hopefully it gets merged sometime. pr

Edit: My contribution got merged!

Tags:

#oss #til #go

Code AI teaches with examples

Sun, 29 Oct 2023 00:00:00 +0000

Code AI teaches with examples

I use AI probably at least once a day. I use it a lot to help with kubectl --output, jq and shell scripts.

But, where it really shines, at least for me, is teaching me through examples.

Here’s a snippet which I really like. I was thinking about a better way to initialise a bunch of NATS subscriptions. AI gave me a good solution that was different to what I was thinking. And much better; I was considering a map of funcs…

My ever growing list of subscriptions.

func (n *Nats) InitSubscribers() error {
	err := n.inboundMailReceived()
	if err != nil {
		return err
	}
	err = n.processInboundMail()
	if err != nil {
		return err
	}
	err = n.eventUserAssignedToEmail()
	if err != nil {
		return err
	}
	err = n.actionPostMessage()
	if err != nil {
		return err
	}
	err = n.eventChannelMessage()
	if err != nil {
		return err
	}
	return nil
}

What I asked JetBrains AI: Refactor this code for me. Here’s its output:

func (n *Nats) InitSubscribers() error {
	functions := []func() error{
		n.inboundMailReceived,
		n.processInboundMail,
		n.eventUserAssignedToEmail,
		n.actionPostMessage,
		n.eventChannelMessage,
		// add more function calls here if necessary
	}

	for _, fn := range functions {
		if err := fn(); err != nil {
			return err
		}
	}

	return nil
}

I love this because I had a problem. Thought through it myself and then asked AI. It showed me a way that I wasn’t initially considering, and I prefer it to my idea.

Now I have a good mental model for approaching this problem in the future. It’s like my own little peer reviewer or rubber ducky.

Tags:

#til #jetbrains #ai

kagi.com is superior to google

Sat, 14 Oct 2023 00:00:00 +0000

kagi.com is superior to google

I am on the Kagi free plan and sparingly using my 100 free queries whenever I feel like google is failing me.

So far, its been 100% effective. The most recent failed google search:

cheap or free databases

Within the first fold are listicles, reddit posts and here’s the kicker, relevant search results!

Given the simplistic question you’d think google would nail this. Instead it contained both obvious paid adverts (scyallaDB - how is that cheap or free!!) and those dumb “comparison” sites which are just SEO traps; g2.com for instance.

Kagi is probably worth paying for and if it keeps smoking google like this, I’ll pay for it.

It’s hard to prove it but google searches are getting worse over time.

Tags:

#google #kagi

Go's Functional Options pattern is great

Mon, 02 Oct 2023 00:00:00 +0000

Go’s Functional Options pattern is great

Some consider this a spicy take but I think it is a fantastic and idiomatic way to build functions/methods which are backwards compatible.

Consider this code from go-gitlab. By using the ...ClientOptionFunc it is possible to add new options without breaking compatibility.

// NewClient returns a new GitLab API client. To use API methods which require
// authentication, provide a valid private or personal token.
// ref: https://github.com/xanzy/go-gitlab/blob/2692fa8f0c4e16c36af8ebdc571da3a0d4ce2d19/gitlab.go#L240-L250
func NewClient(token string, options ...ClientOptionFunc) (*Client, error) {
	client, err := newClient(options...)
	if err != nil {
		return nil, err
	}
	client.authType = PrivateToken
	client.token = token
	return client, nil
}

Here is how I’ve extended it with my own custom HTTP client in a project I am working on.

func NewGitlab(token string, url string, insecure bool, timeout time.Duration, options ...gitlab.ClientOptionFunc) (*Gitlab, error) {
	tr := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: insecure}}
	hc := &http.Client{
		Transport: tr,
		Timeout: timeout,
	}
	options = append(options, gitlab.WithBaseURL(url))
	options = append(options, gitlab.WithHTTPClient(hc))
	cl, err := gitlab.NewClient(
		token,
		options...,
	)
	if err != nil {
		return nil, err
	}
	return &Gitlab{Client: cl}, err
}

With this pattern in one caller I can pass in options which others shouldn’t have.

// I do not want the default retries so I disable it.
glab, err := providers.NewGitlab(
 form.ClientToken,
 form.GitLabURL,
 insecure,
 providers.GitlabClientDefaultTimeout, // a const from internal
 gitlab.WithoutRetries(),
)

In every other instance I do not pass in the gitlab.WithoutRetries because the defaults are what I want.

A simple example but to me it makes life a lot easier and I’ve started adopting it more in my own work.

A good reference: https://michalzalecki.com/golang-options-pattern/

Tags:

#go #code

Sourcegraph is awesome

Mon, 02 Oct 2023 00:00:00 +0000

Sourcegraph is awesome

I’ve yet to find a way to control my snippets. I tried snips.sh but found it a little too finicky. It is a cool project but sometimes I need to capture more context than just a snippet. Like how does x fit into y and it might be several lines long or be across modules (Go specific here).

Sourcegraph lets me do that. I can just add my project and then search for a term and voila. It’s possible to navigate forward and backward through interfaces, methods and even commit history. It can do much more but that covers what I wanted so far. What I also like is that it runs on my Caprover instance and I can now login to it from anywhere. This is a cool feature as sometimes I’ll be dreaming up some idea and need to check something. Now I just login and search around from any computer, or phone.

Well worth checking out. Now I just need to convince my team we need it because searching around on GitLab when you have multiple microservices and you can’t remember which one does xyz is an absolute PITA.

Tags:

#sourcegraph #code

http status code mistakes: 303 versus 307

Tue, 19 Sep 2023 00:00:00 +0000

http status code mistakes: 303 versus 307

I just wasted a lot of time trying to debug an issue with my template driven web app.

When redirected after making a database change to another page, it kept deleting one of the fields and setting it back to the NOT NULL DEFAULT 0 as defined in my table schema.

I was questioning my understanding of SQL and the Post/Redirect/Get pattern.

Well it turns out I have fat fingered a tab complete and instead of http.StatusSeeOther I had inadvertently used http.StatusTemporaryRedirect.

Quote from rfc7231:

The 307 (Temporary Redirect) status code indicates that the target
resource resides temporarily under a different URI and the user agent
MUST NOT change the request method if it performs an automatic
redirection to that URI. Since the redirection can change over time,
the client ought to continue using the original effective request URI
for future requests.

The upper cased MUST NOT precending change the request method is a strong signal about what was happening to my requests.

TIL: 303 and 307 are very different and I should pay more attention to the redirect status code when redirecting.

Tags:

#http #redirects #til

Flush linux DNS cache

Wed, 13 Sep 2023 00:00:00 +0000

Flush linux DNS cache

TIL how to flush my local DNS cache in Ubuntu.

# check if cache is populated
resolvectl statistics
# flush cache
resolvectl flush-caches
# confirm, cache should say 0
resolvectl statistics

I was doing some local testing with cloud providers, I switched from one VPS provider to another but kept the same subdomain. This was causing curl and Go.http errors. Flusing the cache fixed it.

Tags:

#go #dns #curl

oapi-codegen: chi router groups and middleware

Tue, 08 Aug 2023 00:00:00 +0000

oapi-codegen: chi router groups and middleware

Today I learned a valuable lession in assumptions.

I assumed that putting the oapivalidator with the router.Group where the endoints live was how it worked. Not realising that zero validation was being done. Not until I started implemented JWT authentication and noticing that no errors were propagating.

Here is the code that works.

router := chi.NewRouter()
// middlewares
router.Use(oapimiddleware.OapiRequestValidator(openapiSpec))
// endpoints
router.Group(func(api chi.Router){
 server.HandlerFromMuxWithBaseURL(s, router, "/api/v1")
})

Tags:

#go #til #aar

VHS: your CLI home video recorder

Sat, 05 Aug 2023 00:00:00 +0000

VHS: your CLI home video recorder

I just removed my ‘peek’ recorded demo.gif from my gpt wrapper CLI tool’s README.md and replaced it with a vhs tape.

It’s so much better looking but can seriously prevent doxing information. Sometimes when recording something you can accidently include identifying information. Not with vhs, it records in its own custom terminal. Yet still honours your terminals bindings.

See the demo: https://github.com/danielmichaels/gpt

Tags:

#charmbracelet #vhs #go

charmbracelet gum is awesome

Thu, 03 Aug 2023 00:00:00 +0000

charmbracelet gum is awesome

By adding gum confirm "are you sure?" I removed so much read -p code and case statements.

Highly recommend

Tags:

#gum #go #bash

Go generic: non-ptr to ptr

Thu, 03 Aug 2023 00:00:00 +0000

Go generic: non-ptr to ptr

I’m using oapi-codegen and any required fields are set as pointers. This is great for doing nil checks. Initialising structs was more difficult because you cannot create a struct with *string fields like this:

type Foo struct {
 ID *string
}

func main() {
 f := Foo{ID: "123"}
}

This fails because its not a pointer. Usually I’d create a variable like var s *string and then populate the struct using s.

Then I found this great use of generics and am now using it everywhere.

// Ptr takes in non-pointer and returns a pointer
func Ptr[T any](v T) *T {
	return &v
}

// in use 
// a snippet from my tests
{name: "200: OK", method: "GET", url: "/api/v1/devices/device_000000000000", body: "", status: 200,
 want: repository.DeviceResponseJSON{
 Data: oapi.DeviceResponse{
 ApiVersion: Ptr("not provided"),
 CreatedBy: Ptr("guest"),
 DeviceActivated: Ptr(false),
 DeviceId: Ptr("device_000000000000"),
 DeviceName: Ptr("test_device"),
 DeviceVersion: Ptr("2.7"),
 HostAddress: Ptr("192.168.0.1"),
 HttpPort: Ptr(int32(8080)),
 SshPort: Ptr(int32(22)),
 TeamId: Ptr("team_000000000000"),
 },
 },
},

Just made writing tests so much easier.

Tags:

#TIL #generics #golang

Remote and voltage; a fix

Mon, 31 Jul 2023 00:00:00 +0000

Remote and voltage; a fix

Happy that I fixed a suddenly “broken” TV remote. The TV was reporting that the batteries had gone flat, so I replaced them. Still didn’t work and said it’s flat.

Got out the multimeter to test old and new.

Old: 1.4~V New: 1.6V

So I put an old and new together in series and tested, came to ~3.0v

Put them in the remote and it works perfectly.

Now I know you’re not meant to mix batteries but considering how sensitive this remote is, I don’t care!

Tags:

#debugging #electronics

Custom zerolog logger

Sun, 30 Jul 2023 00:00:00 +0000

Custom zerolog logger

Zerolog is my favourite logger for Go.

Chi comes with httplog which creates a very simple yet powerful http logger. For non-http apps which need logging I’ve copy pasted my own logger below:

package main


import (
	"fmt"
	"github.com/rs/zerolog"
	"github.com/rs/zerolog/log"
	"os"
	"strings"
	"time"
)

func NewLogger(serviceName string, opts ...Options) zerolog.Logger {
	if len(opts) > 0 {
		Configure(opts[0])
	} else {
		Configure(DefaultOptions)
	}
	logger := log.With().Str("service", strings.ToLower(serviceName))
	if !DefaultOptions.Concise && len(DefaultOptions.Tags) > 0 {
		logger = logger.Fields(map[string]interface{}{
			"tags": DefaultOptions.Tags,
		})
	}
	return logger.Logger()
}

var DefaultOptions = Options{
	LogLevel: "info",
	LevelFieldName: "level",
	JSON: false,
	Concise: false,
	Tags: nil,
	SkipHeaders: nil,
	TimeFieldFormat: time.RFC3339Nano,
	TimeFieldName: "timestamp",
}

// Configure will set new global/default options for the httplog and behaviour
// of underlying zerolog pkg and its global logger.
func Configure(opts Options) {
	if opts.LogLevel == "" {
		opts.LogLevel = "info"
	}

	if opts.LevelFieldName == "" {
		opts.LevelFieldName = "level"
	}

	if opts.TimeFieldFormat == "" {
		opts.TimeFieldFormat = time.RFC3339Nano
	}

	if opts.TimeFieldName == "" {
		opts.TimeFieldName = "timestamp"
	}

	// Pre-downcase all SkipHeaders
	for i, header := range opts.SkipHeaders {
		opts.SkipHeaders[i] = strings.ToLower(header)
	}

	DefaultOptions = opts

	// Config the zerolog global logger
	logLevel, err := zerolog.ParseLevel(strings.ToLower(opts.LogLevel))
	if err != nil {
		fmt.Printf("httplog: error! %v\n", err)
		os.Exit(1)
	}
	zerolog.SetGlobalLevel(logLevel)

	zerolog.LevelFieldName = strings.ToLower(opts.LevelFieldName)
	zerolog.TimestampFieldName = strings.ToLower(opts.TimeFieldName)
	zerolog.TimeFieldFormat = opts.TimeFieldFormat

	if !opts.JSON {
		log.Logger = log.Output(zerolog.ConsoleWriter{Out: os.Stderr, TimeFormat: opts.TimeFieldFormat})
	}
}

type Options struct {
	// LogLevel defines the minimum level of severity that app should log.
	//
	// Must be one of: ["trace", "debug", "info", "warn", "error", "critical"]
	LogLevel string

	// LevelFieldName sets the field name for the log level or severity.
	// Some providers parse and search for different field names.
	LevelFieldName string

	// JSON enables structured logging output in json. Make sure to enable this
	// in production mode so log aggregators can receive data in parsable format.
	//
	// In local development mode, its appropriate to set this value to false to
	// receive pretty output and stacktraces to stdout.
	JSON bool

	// Concise mode includes fewer log details during the request flow. For example
	// excluding details like request content length, user-agent and other details.
	// This is useful if during development your console is too noisy.
	Concise bool

	// Tags are additional fields included at the root level of all logs.
	// These can be useful for example the commit hash of a build, or an environment
	// name like prod/stg/dev
	Tags map[string]string

	// SkipHeaders are additional headers which are redacted from the logs
	SkipHeaders []string

	// TimeFieldFormat defines the time format of the Time field, defaulting to "time.RFC3339Nano" see options at:
	// https://pkg.go.dev/time#pkg-constants
	TimeFieldFormat string

	// TimeFieldName sets the field name for the time field.
	// Some providers parse and search for different field names.
	TimeFieldName string
}

This needs more tweaking but serves as a good starting point.

Tags:

#go #logging

Instant Motivation

Sat, 29 Jul 2023 00:00:00 +0000

Instant Motivation

Sat down to do some coding and was really struggling. Motivation was low, mood felt off and defeatist.

Got off my arse and jumped on the assault bike for twenty.

What a motivator; did I want to do it? No but I knew it would be what got be out of the func I was in.

Just workout people, its easy!

Tags:

#gym #motivation #mindset

zalando API guidelines

Fri, 21 Jul 2023 00:00:00 +0000

zalando API guidelines

https://opensource.zalando.com/restful-api-guidelines/

Looks like a decent guide for writing API’s

Tags:

#api

golang map references

Tue, 27 Jun 2023 00:00:00 +0000

golang map references

The CS naming for this type of structure escapes me. But for now, I’ll call it “dot notation”. I use this all the time in typescript (and in python).

I wanted a way to have const’s in Go but structured so that I could call them elsewhere in the following manner; Foo[Bar]. This is much cleaner than Foo["bar"] as the compiler prevents typo’s such as Foo["baz"].

type MapKey string

const (
 Key1 MapKey = "foo"
 Key2 MapKey = "bar"
 Key3 MapKey = "baz"
)

var MyMap = map[MapKey]string{
 Key1: "foo",
 Key2: "bar",
 Key3: "baz",
}

fmt.Println(MyMap[Key1]) // "foo"

It’s a little repetitious but works well for me.

Tags:

#go

ngroks newer cousin

Fri, 23 Jun 2023 00:00:00 +0000

ngroks newer cousin

I just stumbled upon jprq a Ngrok alternative.

https://github.com/azimjohn/jprq

To use it you need to authenticate using your Github account. This has the added side affect of using your github username as the jprq.live subdomain.

For instance, mine is danielmichaels.jprq.live. This is much more user friendly than Nrgoks randomly generated one.

Its written in Go as well.

Running it with --debug exposes a nice web UI as well. The only thing it lacks which I miss when compared to Ngrok is the ability to replay requests.

Tags:

#ngrok #tunnels #go

Bad Request v Unprocessible Entity

Thu, 01 Jun 2023 00:00:00 +0000

Bad Request v Unprocessible Entity

I think this Stack Overflow answer sums it up perfectly.

Case study: GitHub API

https://docs.github.com/en/rest/overview/resources-in-the-rest-api#client-errors

Maybe copying from well known APIs is a wise idea:

There are three possible types of client errors on API calls that receive request bodies:

Sending invalid JSON will result in a 400 Bad Request response:

HTTP/1.1 400 Bad Request
Content-Length: 35
{"message":"Problems parsing JSON"}
Sending the wrong type of JSON values will result in a 400 Bad Request response:

HTTP/1.1 400 Bad Request
Content-Length: 40

{"message":"Body should be a JSON object"}
Sending invalid fields will result in a 422 Unprocessable Entity response:

HTTP/1.1 422 Unprocessable Entity
Content-Length: 149

{
 "message": "Validation Failed",
 "errors": [
 {
 "resource": "Issue",
 "field": "title",
 "code": "missing_field"
 }
 ]
}

Tags:

#http #error

sqlite JSON and Go

Sun, 28 May 2023 00:00:00 +0000

sqlite JSON and Go

snips.sh and zet

Fri, 26 May 2023 00:00:00 +0000

snips.sh and zet

I like to capture snippets but have found that incorporating well formatted, context aware and runnable examples in this zet challenging.

Today I found snips.sh and its really awesome. I’m going to try it out as a way to enrich my zets.

Here is an example: https://snips.infra.ptco.rocks/f/32pTsJRvI_

This is a real file from a project I have which uses NATS as its message service.

To create that snippet all I needed to do was cat <file>.go | ssh snips.

This then saved my snippet to my snips.sh instance.

Because snips.sh takes input from stdin I think it’ll make a great partner to this type of notes capture.

Tags:

#snippets #go #ssh

oapi-codegen with Gin and Echo

Thu, 25 May 2023 00:00:00 +0000

oapi-codegen with Gin and Echo

Don’t work with anything other than the default base URL.

For example,

I want everything served at /api to be routed through the oapi-codegen generated ServerInteface. Anything else is not routed via oapi-codegen routes.

Doing this allows me to have an API backend and run a traditional web application.

In go-chi this is achievable and works really well.

In both gin and echo this is not possible. There are issues for echo in oapi-codegen’s github repo. The gin examples and google searches proved fruitless.

Don’t waste time on this. Sure I have to write a lot more to get chi working but it is pretty simple and highly extensible.

Tags:

#echo #gin #oapi-codegen #golang

API project design

Fri, 19 May 2023 00:00:00 +0000

API project design

A rough outline of how I’ll implement an API for a project I’m developing using oapi-codegen.

Endpoints:

/healthz:
 - GET
/users:
 - GET
 - POST
 - PATCH
 - DELETE
/monitors:
 - GET
/monitors/{monitor-type}:
 - GET
 - POST
 - PATCH
 - DELETE
/monitors/{monitor-type}/{monitor:id}
 - GET
 - POST
 - PATCH
 - DELETE

The barebones are there but implementation needs some work.

Tags:

#planning #openapi #go

Docker layers and ARG

Fri, 19 May 2023 00:00:00 +0000

Docker layers and ARG

I spent a lot of today debugging a PICNIC* error related to Docker’s multi-stage builds.

PICNIC: Problem In Chair, Not In Computer

Here is the example

# heavily truncated
FROM python:latest as base
ARG CERTIFICATES=certificates

FROM python:latest as builder

COPY $CERTIFICATES/* /usr/local/share/ca-certificates/

This example (in a more full examle at least) does not fail. It’ll copy. Except its going to copy /* into /usr/local/share/ca-certificates/ because $CERTIFICATES does not exist in the builder stage.

I spent a lot of time trying to figure out why my certificates/ directory was getting the certs and the source code during CI resulting in an image full of junk in /usr/../ca-certificates.

Pretty obvious in hindsight.

The fixed example Dockerfile:

FROM python:latest as base

FROM python:latest as builder
ARG CERTIFICATES=certificates # moved from base

COPY $CERTIFICATES/* /usr/local/share/ca-certificates/

Also worth noting, when installing custom certs in a multistage you’ll need to copy the certs into the final stage and re-run update-ca-certificates.

# example of multistage with certs
FROM python as base
# do stuff
FROM python as builder

ARG CERTIFICATES=certificates
COPY $CERTIFICATES/* /usr/local/share/ca-certificates/
RUN update-ca-certificates
# do stuff that needs custom cert

FROM base

COPY --from=builder /usr/local/share/ca-certificates/ /usr/local/share/ca-certificates/
# copy whatever else
RUN update-ca-certificates

Tags:

#til #docker #cert

Quick math temp conversions

Fri, 19 May 2023 00:00:00 +0000

Quick math temp conversions

Commit to memory; converting to and from Celsius and Farenheit.

Close enough approximations which are good enough for field work.

C to F

(Cx2) + 30

10 celsius X 2 = 20
20 + 30 = 50

Exact is formula is:
(10*1.8) + 32

so comparing the two methods the difference is
quick: 50
exact: 50

If the temp is 30C
the delta between the methods grows slightly
quick: 90
exact: 86

F to C

(F-30)/2

55 - 30 = 25 
25/2 = 12.5

Exact formula is:
(F-32)/1.8

comparing them:
quick: 12.5
exact: 12.7

This cowboy math is basically all you need in life.

Calculatingusing decimals and fractions is dumb when you can approximate it using year 3 math skills.

Hardest part is remembering the formula.

Tags:

#math #til #skills

Docker catalog and digest using crane

Wed, 17 May 2023 00:00:00 +0000

Docker catalog and digest using crane

I’ve written a number of bash scripts over my time to check container registry catalogs and then parse them for certain tags.

TIL that google’s crane supports this out of the box.

Install crane with arkade:

arkade get crane

Here’s the output of ls and digest:

# list all tags
crane ls danielmichaels/sitesearch
915194c0cd6350048a4b6c3a24eb555bae7e0a11
9c7d76991e79a8dd0fa1f081e70b5c2daf6838e1
a12e6f4326f0a3397df42cade3e4382dd1915f83
ab658d594ab00b36eb7a66d9493654d2d0f7d971
d7a22105650736aaa3f3fbf8a81a063824be6d0a
ed22c15b0057c1a5ad93a836e1793b01c7f2e195
latest
okteto
staging

# get a tag digest
crane digest danielmichaels/sitesearch:latest
sha256:2fdca04e110e75df31c22a33fcd6db9edda6fb36cc86c44bece01d0725346e72

Tags:

#til #crane #docker #registry

Running a business with OpenFaaS, Stripe and Sendgrid

Mon, 08 May 2023 00:00:00 +0000

I did some work recently to set up a payments and notification backend for a Carrd website. The website integrates with some booking services which are linked to payment providers.

The frontend needed two more services added that could not integrate with the booking service, and that got me thinking about how to level up the site without too much effort.

I have some experience using OpenFaaS and its smaller cousin, Faasd.

Faasd? OpenFaaS?

OpenFaaS is an easy to host Functions-as-a-Service platform. It is a very capable platform which lets you write functions in almost any language, on any cloud without any vendor lock-in. It has great tooling including its own CLI and UI. It is often deployed to Kubernetes which I’ve used to achieve interesting things in the past.

This project needed something a lot smaller and lightweight enough to run on a small and cheap VPS. That’s where faasd comes in.

It’s a single Go binary with enough firepower to easily run this app. Here’s the minimum system requirements:

512MB-1GB RAM
1-4 vCPU cores
10-25GB of disk space

Functions

The way I think about functions is as handlers for requests. Each function serves a route and that’s about it. Though I don’t split functions up by HTTP method, only by the url they’re responsible for.

In OpenFaaS speak each function gets its own url path such as https://fn.faas.url/function/<fn-name>/.

How it works is pretty impressive and well detailed across all the OpenFaaS blogs and documentation.

But, I’ll do my best to give a run down on how OpenFaaS works.

OpenFaaS takes a lower level concept like container orchestration and kubernetes and abstracts away all the complexity by exposing an application surface. This application surface is in the form of functions. Each function is its own container and for the most part accepts something from stdin and outputs stdout or stderr to the caller. Functions can be invoked via HTTP, the CLI or through event bridges such as Kafka, NATS, MQTT and so on. It also has a cron trigger which is a great builtin feature.

Since each function is its own container OpenFaaS makes it easier to scale up your functions. Just add more containers for that function. Isolation between functions is also a benefit. One bad function won’t crash your entire app.

Being that everything is containers means I can have functions in Go, python, node, C# or even bash. I typically use Go these days but have used a python function to augment my Go functions because it was the right choice. Had that project been a server written in Go more than likely I would of had to stand up a microservice and then setup NATS or HTTP endpoints for them to interact. Using OpenFaaS made it a cinch.

Although it’s not perfect. Sometimes function codebases can have a lot of repeated code. As an example, both my functions use the same logger and database packages which are identical. It’s not so bad because I have scripts that autogenerate new functions and handle these things. I will point out that you can create your own function template which could include any required modules effectively removing this hiccup.

Another beautiful addition is prometheus and Alert Manager out of the box. Whilst I am not using these for this project it is nice knowing I can easily hook up grafana in the future.

For a really succinct summary of OpenFaaS with great explanatory images please read Ivan Velichko’s blog post.

Function: Stripe

The primary function is the Stripe webhook handler which serves as the most common entrypoint.

When a customer makes a purchase the webhook handler consumes the response and based on what it is, acts on it.

As an example, when a checkout.session.complete is received, it is marshalled into a struct. Then inserted into a database, in this case MongoDB Atlas. The function then calls another function passing along the key for the recently inserted data.

I’ve found this works excellently. It quickly writes to the database and send the POST to the next function - both done in goroutines.

Function: Sendgrid/Email

Once a payment intent is received and dealt with in the Stripe function the email function is triggered. In my opinion this is one of the powerful parts of OpenFaaS; function chaining.

Once the payload is received the function checks which product was purchased and triggers a series of emails to the customer and website administrators.

Leveraging SendGrid here made sense because it has a generous free tier of 100 emails per day. Even then once you start paying it is still quite reasonable. It also has a decent email template generator which is used to deliver specific emails based on the product purchased.

After all that, the functions, third-party integrations and how its invoked look something like this.

Final thoughts

To integrate Stripe with Carrd I used payment links It was my first time using them, and they are fantastic. Hooking up the URL to Carrd is as easy as creating a button which directs the customer to the payment link.

Initially, I thought Carrd was only good for a simple landing page but with payment links you can really stretch it quite far.

It was the realisation that this project was really just the confluence of several other API’s that made me think about a different way to deal with this problem.

Is it better than a long-running server? I think it depends on what you want or need out of it. Do Functions-as-a-Service replace servers? No.

rRght tool for the right job and I think this is the right tool for now. Knowing that when this gets bigger I can lift and shift into kubernetes is nice. Nicer still, I don’t have to write any deployment manifests to make it all work because OpenFaaS handles everything under the hood.

I think in the future I will start all designs with a simple question; do I really need a backend, or do I just need some functions?

Helm upgrade dynamically with Go

Sun, 07 May 2023 00:00:00 +0000

Helm upgrade dynamically with Go

I wrote a python file to create helm upgrade commands dynamically based on environment variables.

Not happy with that I also wrote it in Go. I much prefer it because its easier to distribute and has zero dependencies.

// Auto generate a helm deployment for use across multiple projects in CI.
// To use this the following is required:
//
// - Environment variables set with the values needed. Read carefully for how to use each
// type of variable.
//
// - GitlabCI, this is primarily designed to be used in GitlabCI using its includes and
// artifact generation functionality
//
// See comments for how to use local environment variables to set values for helm.
//
// To use this in GitlabCI you only need to set the 'variables' block with your required values.
// Typically, this job will generate an artifact with the script as its output which a deployment
// job will have a needs relationship with and will then execute the script. YMMV
package main

import (
	"html/template"
	"log"
	"os"
	"strings"
)

// Environment variables, to use HELM_SET_FILE_PREFIX and HELM_SET_PREFIX
// the following considerations must be taken into a account.
// helm's '--set' and '--set-file' are used to override yaml fields inside a values.yml file.
// Using the following yaml as a example we'll illustrate how to use these envvars.
//
// example values.yml
// db
//
//	name: myDatabase
//	password: ""
//	caCert: {}
//
// Shell environment variables do not allow for "." notation. To get around this limitation
// it is expected that any "_" after the prefix will be translated into a "."
// For instance, HELM_SET_db_password=myPass would become '--set db.password=myPass'
// HELM_SET_FILE_db_caCert=/tmp/path/0/ca.crt becomes '--set-file db.caCert=/tmp/path/0/ca.crt'
//
// The outlier here is HELM_VALUES which is always provided as a string such as HELM_VALUES=helm/values.yml,helm/values2.yml
// which generates '--values helm/values.yml,helm/values2.yml'
const (
	HELM_SET_PREFIX = "HELM_SET_VALUE_"
	HELM_SET_FILE_PREFIX = "HELM_SET_VALUE_"
	HELM_VALUES = "HELM_VALUES"
)

type Values struct {
	Key string
	Value string
}

func extractFileAndSetFile(re_prefix string) []Values {
	var values []Values
	for _, env := range os.Environ() {
		if strings.HasPrefix(env, re_prefix) {
			trimmedValues := strings.SplitN(env, re_prefix, 2)
			splitOnEquals := strings.SplitN(trimmedValues[1], "=", 2)
			dotValue := strings.ReplaceAll(splitOnEquals[0], "_", ".")
			value := Values{
				Key: dotValue,
				Value: splitOnEquals[1],
			}
			values = append(values, value)
		}
	}
	return values
}

func extractValuesFiles(re_prefix string) []string {
	var values []string
	for _, env := range os.Environ() {
		if strings.HasPrefix(env, re_prefix) {
			kv := strings.SplitN(env, "=", 2)
			v := kv[1]
			values = append(values, v)
		}
	}
	return values
}

type TemplateData struct {
	Release string
	ChartPath string
	SetValue []Values
	SetFile []Values
	ValuesFiles []string
	// Allow for --dry-run to be passed in the script; usefule for debugging
	// or when wanting to use helm to template but kubectl to execute/apply
	DryRun any
}

func main() {
	setValues := extractFileAndSetFile(HELM_SET_PREFIX)
	setFiles := extractFileAndSetFile(HELM_SET_FILE_PREFIX)
	valuesFiles := extractValuesFiles(HELM_VALUES)
	release := os.Getenv("HELM_RELEASE")
	chartPath := os.Getenv("HELM_CHART_PATH")

	data := TemplateData{
		Release: release,
		ChartPath: chartPath,
		SetValue: setValues,
		SetFile: setFiles,
		ValuesFiles: valuesFiles,
		DryRun: os.Getenv("DRYRUN"),
	}

	temp := template.Must(template.New("helm").Parse(helmTemplate))

	err := temp.Execute(os.Stdout, data)
	if err != nil {
		log.Fatal(err)
	}

}

var helmTemplate = `#!/bin/bash

helm upgrade --install {{ .Release }} {{ .ChartPath }} \
	{{ if .SetValue -}}
		{{- range .SetValue -}}
	--set '{{ .Key }}={{ .Value }}' \
		{{- end -}}
	{{- end }}
	{{ if .SetFile }}
		{{- range .SetFile -}}
	--set-file '{{ .Key }}={{ .Value }}' \
		{{- end -}}
	{{- end }}
	{{ if .ValuesFiles }}
		{{- range .ValuesFiles -}}
	--values '{{ . }}' \
		{{- end -}}
	{{- end }}
	--atomic --timeout 300s {{ if .DryRun }}--dry-run{{ end }}
`

To build this as a distributable binary is as simple as go build main.go. You can name the binary with go build -o <name> main.go. The run the binary and pipe its output to a script for execution when required. e.g. go run main.go > deploy-helm.sh.

It is a simple solution and would be keen to here how others solve it.

Tags:

#helm #go #templates

Dynamically generate helm commands in CI

Fri, 05 May 2023 00:00:00 +0000

Dynamically generate helm commands in CI

We run helm for our deployments (eventually switching to Argo thankfully) and it works OK. What it doesn’t do well in is re-usability in CI.

We have a project which all other projects inherit from for CI and it works pretty well. It takes some organisation but once its done you can basically set and forget your deployments. Except that helm sucks for that and we ended up with each project needing a bespoke helm-deploy job.

This week I sat down and designed a better solution using python templating. Next week I think I’ll re-do it in Go because its in the standard lib.

Here’s how I’m doing it python.

import os
import re

from jinja2 import Template

helm_set_prefix="HELM_SET_VALUE_"
helm_set_file_prefix="HELM_SET_FILE_"
helm_values_prefix="HELM_VALUES"

def prefix(re_prefix: str):
 return re.compile(r"{prefix}\w".format(prefix=re_prefix))

helm_set = {key.replace(helm_set_prefix,"").replace("_","."):val for key, val in os.environ.items() if prefix(helm_set_prefix).match(key)}
helm_set_file = {key.replace(helm_set_file_prefix,"").replace("_","."):val for key, val in os.environ.items() if prefix(helm_set_file_prefix).match(key)}
# this is just an array
helm_values = [v for k,v in os.environ.items() if k.startswith(helm_values_prefix)]

template = """
#!/bin/bash 

helm upgrade --install {{ release }} {{ chart_path }} \
 {%- if helm_set -%}
 {%- for k,v in helm_set.items() -%}
 --set {{ k }}={{ v }} \
 {% endfor %}
 {% endif %}
 {%- if helm_set_file -%}
 {%- for k,v in helm_set_file.items() -%}
 --set-file {{ k }}={{ v }} \
 {% endfor %}
 {% endif %}
 {%- if helm_values -%}
 {%- for v in helm_values -%}
 --values {{ v }} \
 {% endfor %}
 {% endif %}
 --atomic --timeout {{ timeout }}
"""

data = {
 "timeout": os.getenv("TIMEOUT", "300s"),
 "release": os.getenv("RELEASE"),
 "chart_path": os.getenv("CHART_PATH", "helm"),
 "helm_set": helm_set,
 "helm_set_file": helm_set_file,
 "helm_values": helm_values,
 }

output = Template(template)

# run me like so:
# python main.py > script.sh
print(output.render(data))

I’ll bypass the how of getting this going in CI jobs but to pass values its as simple as providing environment variables with prefixes.

job:
 variables:
 RELEASE: MyRelease
 HELM_SET_db_name: foobar
 HELM_SET_FILE_cacert: ca.crt
 HELM_VALUES: helm/values.yml

Which would create a bash script which looks like

helm upgrade --install MyRelease helm \
 --set db.name='foobar' \
 --set-file cacert='ca.crt' \
 --values 'helm/values.yml' \
 --atomic --timeout 300s

This script can then be executed in another job as needed when output as an artifact which makes its highly extensible.

Update: I also wrote this in Go which is nicer because you don’t need to install anything and can build it as a binary and pass around easier.

Tags:

#python #ci #helm

Go: simple background task

Wed, 26 Apr 2023 00:00:00 +0000

Go: simple background task

A really simple background worker snippet. Far from perfect!

func background(fn func()) {
	// Launch a background goroutine.
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		// Recover any panic.
		defer func() {
			if err := recover(); err != nil {
 // zerolog
				logger.Error().Err(fmt.Errorf("%s", err)).Msg("background-task")
			}
		}()

		// Execute the arbitrary function that we passed as the parameter.
		wg.Done()
		fn()
	}()
	wg.Wait()
}

Tags:

#go #snippet

zerolog: reference article

Sat, 22 Apr 2023 00:00:00 +0000

zerolog: reference article

zerolog is my go to logger for all Go applications.

I stumbled upon this article and had to capture it somewhere for future reference. It covers every thing I’ll ever need. Already it’s given me some inspiration for cleaning up some loggers.

Tags:

#go #zerolog #logging

garbage collecting nix

Sun, 16 Apr 2023 00:00:00 +0000

garbage collecting nix

nix-collect-garbage -d

Also deleting nix profiles is done by nix profile list and then selecting the profile index.

For instance, nix profile 1

Tags:

#nix

kubernetes UnexpectedAdmissionError

Wed, 05 Apr 2023 00:00:00 +0000

kubernetes UnexpectedAdmissionError

Today we lost our storage from underneath us. This spawned hundreds of pods which failed with UnexpectedAdmissionError.

To fix this, all we needed to do was restart our nodes after the storage issue was fixed.

To delete all those pods I ran:

for pod in $(kubectl get pods -n <namespace> | grep -i UnexpectedAdmissionError | cut -d' ' -f 1);
do kubectl delete pod $pod -n <namespace>; done

Tags:

#kubernetes

Qantas Wifi and Docker

Sat, 01 Apr 2023 00:00:00 +0000

Qantas Wifi and Docker

I’m currently in the air and spent ages trying to figure out why I could connect to the in-flight wifi but not trigger the captive portal.

It was Docker.

The in-flight wifi shares an address space with docker’s defaults.

172.18.0.0/23

It did not occur to me that the br network sharing this was from Docker. Instead I spent a bit of time trying to manually remove it, and more time still curl‘ing and pinging the gateway to reverse engineer the portal URL.

In the end a docker system prune -af did the trick and immediately fired the captive portal.

Tags:

#captive-portal #qantas #networking #docker

A viable openapi documentation platform

Wed, 15 Mar 2023 00:00:00 +0000

A viable openapi documentation platform

I’ve been toying with how best to deploy my openapi spec file so that users can integrate with the backend. Initially, I thought about hosting it on the API server itself. But, it seemed quite complicated and sub optimal for a go server to do this.

Instead I think docusarus (which I already use in other projects) has first class support for this exact use case.

Demo: https://docusaurus-openapi.netlify.app/docs/intro

Tags:

#openapi #documentation

Erik Prince: The Rise and Fall of Blackwater

Mon, 27 Feb 2023 00:00:00 +0000

Erik Prince: The Rise and Fall of Blackwater

Today I listened to a podcast with Erik Prince the founder of Blackwater.

It was honestly the most inspiring and interesting podcast I’ve listened to in a long time. Some key takeaways,

Bias for action: Erik gets it done.
Pragmatic and practical. He does not overthink (or underthink)
Men, Mission and then Me; a true leader putting the men first
A strong believer in acting within the spirit of A letter to Garcia.
- I hadn’t thought about this in a few years and it made me sad.
Better to employ your rivals than fight them
Re-cock and go again; don’t let failure get in the way just take another angle.
A why can’t we do that guy - things aren’t just “that’s how it is” with Erik
A fantastic businessman with a good appreciation for history.
Military bureaucracy breeds zampolit behaviour.
- Case in point we had lawyers in our organisation that vetted each job; not to protect us but to make sure it conformed to the will of the political power!

So much of what Erik spoke about absolutely resonated with me. I fought in Iraq and Afghanistan and what a fucking joke those wars became through pathetic leadership at the top and in parliament.

Link to the episode, here

Tags:

#pmc #podcast

Mongorestore notes

Mon, 20 Feb 2023 00:00:00 +0000

Mongorestore notes

Taskfile is my new Makefile

Fri, 17 Feb 2023 00:00:00 +0000

Taskfile is my new Makefile

For every project one of the first things I do is create a Makefile. Generally this works well except when I need to do some sort of validation, for example checking the existence of a directory. It’s possible but I have to google it every time.

Enter Taskfile a yaml based Makefile written in Go. I am very familiar with Gitlab pipelines so writing yaml like this comes naturally. To me, this is so much easier to read and write.

Example

# https://taskfile.dev

version: '3'

vars:
 REGISTRY: docker.io/me
 IMAGE: myimage

tasks:
 default:
 desc: |
 List all available tasks
 cmds:
 - task --list
 silent: true

 env:
 desc: |
 Print all environment variables sorted alphabetically
 cmds:
 - env | sort
 silent: true

 dev:
 deps:
 - check
 desc: |
 Run the local development environment
 cmds:
 - air

 docker-build:
 desc: |
 Build the docker image
 cmds:
 - docker build . -f deploy/Dockerfile -t "{{.REGISTRY}}/{{.IMAGE}}"

 docker-push:
 deps:
 - docker-build
 desc: |
 Push the docker image
 cmds:
 - docker push "{{.REGISTRY}}/{{.IMAGE}}"

This has been incredibly useful in the development of a Django-channels application where I needed a lot of setup. I was able to pass this over to my team and the concept well received.

Tags:

#taskfile #go #makefile #til

reproducible random strings in python

Thu, 16 Feb 2023 00:00:00 +0000

reproducible random strings in python

import hashlib
import uuid

seed = 'Type your seed_string here' #Read comment below

m = hashlib.md5()
m.update(seed.encode('utf-8'))
new_uuid = uuid.UUID(m.hexdigest())

I used this to create kube resources programatically whilst keeping their names consistent for easier deletion and management.

Tags:

#python #kubernetes

Makefile check args are passed in

Sun, 12 Feb 2023 00:00:00 +0000

Makefile check args are passed in

TIL how to get a Makefile to abort if arguments are not supplied.

## sops/decrypt
.PHONY: sops/decrypt
sops/decrypt:
	@test $(file) || (echo "file= not set" ; exit 1)
	@test $(regex) || (echo "regex= not set" ; exit 1)
	@echo "decrypting ${file} with regex (data|${regex})"
	@sops --decrypt --age $$(cat "${SOPS_AGE_KEY_FILE}" | grep -oP "public key: \K(.*)") --encrypted-regex "^(data|${regex})$$" --in-place "${file}"

This will error with regex= not set if not supplied.

A simple solution that works good enough

Tags:

#TIL #Makefile

debug failed pod

Wed, 08 Feb 2023 00:00:00 +0000

debug failed pod

I was trying to debug a python job which created a pod using a bunch of argparse arguments.

This is how I figured out how to run all the python args and if it failed throw a sleep in bash. This then let me jump in and see what was actually happening on the container.

appreviated

# truncated
 command: ["bash", "-c"]
 args:
 - >-
 python script.py -l foo -b far || sleep 10000;
# truncated

ref: https://devops.stackexchange.com/a/15184/34192

Tags:

#kubernetes #til #debug

k3s hostmount

Tue, 07 Feb 2023 00:00:00 +0000

k3s hostmount

Using a host mount for local k3s development. This negates the need for skaffold, okteto etc for 80% use cases.

PROGNAME="hostmount"
usage="${PROGNAME} <container> -- Host mount local files into k3s

Must be located in the root directory of the git repo"

test ! -n "${1}" && echo "$usage" && exit 1

service="${1:-}"
jsonfile=/tmp/patch.json

json='
{
 "spec": {
 "template": {
 "spec": {
 "containers": [
 "name": "%s",
 "command": [
 "sh",
 "-c",
 "apk add py-watchdog && watchmedo auto-restart -d $PWD -R -- entrypoint.py"
 ],
 "volumeMounts": [
 "mountPath": "/usr/src/app",
 "name": "host-mount"
 }
 ]
 }
 ],
 "volumes": [
 {
 "name": "host-mount",
 "hostPath": {
 "path": "%s"
 }
 }
 ]
 }
 }
 }
 }
}
'

printf "$json" "$service" "$(pwd)" > $jsonfile 
cat $jsonfile

kubectl patch deployment "${service}" -n "${namespace:-default}" --patch-file $jsonfile

This needs a bit of massaging for each project but after that this works great for python.

I’ll create Go version using air when I have a project that needs it.

Tags:

#k3s #development

Go-colly meta tags

Tue, 31 Jan 2023 00:00:00 +0000

Go-colly meta tags

TIL how to scrape meta tags, specifically og:title and so on using go-colly.

c.OnHTML("html", func(e *colly.HTMLElement) {
 // note the use of backticks and double quotes - it has to be exactly like this
 title := e.ChildAttr(`meta[property="og:title"]`, "content")
 fmt.Println(title)
})

Tags:

#scraping #go #colly

Go oapi-codegen chi subrouters

Wed, 25 Jan 2023 00:00:00 +0000

Go oapi-codegen chi subrouters

I’m experimenting with open-api code generation. So far its pretty positive; write some yaml and it generates a bunch of interfaces. Those interfaces are where you write the handler business logic. It’ll handle missing data in and out and error if so.

As a chi user I’ve started making a simple todo app to test it all out. I was disappointed to learn that oapi-codegen doesn’t create the swagger UI for you but it will render the yaml file for you.

Except it doesn’t work out of the box for chi. In fact no other endpoint works - only those specified the specification.

To get around this you must create a subrouter and use that to do the specification validation else it’ll always fail for other routes. It makes sense in the end.

This is how I’ve done it in a simple way.

func main() {
	logger := httplog.NewLogger("mudmap", httplog.Options{
		JSON: false,
		Concise: true,
		LogLevel: "DEBUG",
	})
	logger = logger.With().Caller().Logger()
	swagger, err := spec.GetSwagger()
	if err != nil {
		_, _ = fmt.Fprintf(os.Stderr, "Error loading swagger spec\n: %s", err)
		os.Exit(1)
	}
	swagger.Servers = nil
	si := spec.NewTodoStore()

	r := chi.NewRouter()
	r.Use(middleware.RealIP)
	r.Use(middleware.Compress(5))
	r.Use(httplog.RequestLogger(logger))
	r.Get("/docs", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		json.NewEncoder(w).Encode(swagger)
	})
 // Add n number of non-specification endpoints. But use sparingly.

	r.Mount("/", subRouter(swagger))

	spec.HandlerFromMux(si, r)

	s := &http.Server{
		Handler: r,
		Addr: fmt.Sprintf("0.0.0.0:9090"),
	}

	log.Println("starting server")
	log.Fatal(s.ListenAndServe())
}


// subRouter holds the oapi-codegen validators which are 
// mounted to the parent chi router.
func subRouter(s *openapi3.T) http.Handler {
	r := chi.NewRouter()
	r.Use(oapimiddleware.OapiRequestValidator(s))
	return r
}

Tags:

#chi #go #openapi #codegen

docker run helper

Tue, 24 Jan 2023 00:00:00 +0000

docker run helper

I get tired of running docker run --rm -it <image>:<tag> sh

PROGNAME="dockerr"

usage="${PROGNAME} <image> [-h] [-t] [-c] -- docker run helper 

where:
 -h, --help Show this help text
 -t, --tag Image tag to use (default: latest)
 -c, --command Command to execute when container runs (default: none)
 --entrypoint Override containers entrypoint. This is mutually exclusive with
 --command and will remove it if applied

examples:
 ${PROGNAME} python
 ${PROGNAME} python -t buster -c bash
 ${PROGNAME} hashicorp/vault -t dev --entrypoint sh

tag=latest executable="" entrypoint=""

if [[ ${i} != -* ]] then image="${i}" fi

if [ “$#” -ne 0 ]; then while [ “$#” -gt 0 ]; then do case “$i” in -h|–help) usage ;; -t|–tag) tag="$2" ;; -c|–command) executable="$2" ;; –entrypoint) entrypoint="–entrypoint $2" executable="" ;; –) break ;; -*) echo “Invalid option ‘$1’. Use –help to see arguments” >&2 exit 1 ;; *) ;; esac shift done else echo “$usage” exit 1 fi

echo “running: docker run –rm -it $entrypoint $image:$tag $executable” docker run –rm -it $entrypoint $image:$tag $executable


This saves me a lot of needless typing. It doesn't handle <image>:<tag> 
but that can be achieved. This works good enough.

Tags:

 #bash #docker

Python's distutil strtobool replacement

Mon, 16 Jan 2023 00:00:00 +0000

Python’s distutil strtobool replacement

From 3.10 distutil.util.strtobool is deprecated with removal slated for 3.11.

This is a trivial function to replace its functionality and useful for kubernetes helm values.


def strtobool(value: str) -> bool:
 value = value.lower()
 if value in ("y", "yes", "on", "1", "true", "t"):
 return True
 return False

Tags:

#til #python #helm

hstr: bash history fuzzy finder

Sat, 14 Jan 2023 00:00:00 +0000

hstr: bash history fuzzy finder

I usually opt for history | grep -i <term> or history | ag <term> depending on which system I’m on.

For local development, I’ve started using hstr.

ref: https://github.com/dvorka/hstr

Fedora: dnf install hstr

Ubuntu 22.04 apt instll hstr

Or, just use nix-env -i hstr

Personally, I’ve opted for nix because I’m still on 20.04 and I am tired of installing ppa’s only to have their keys expire and break all updates on the system.

Tags:

#nix #tools

pfSense on Vultr (and a rant)

Sat, 14 Jan 2023 00:00:00 +0000

pfSense on Vultr (and a rant)

I run test pfSense firewalls on Vultr. It is a pain in ass hand cranking them each time.

I have a terraform module which can create the instance but not fully. Two issues arise using terraform.

Can only create an instance with an OS or an ISO
pfSense ISO instances break (it cannot detect the disk) and OS installs don’t have pfSense

So even though I can create the instance using the OS, then switch it to ISO via two terraform apply’s (there is probably a better one-shot way). It still requires even more manual steps.

Next problem is once its started you have to manually step through the TUI installer. After its installed you need to reboot and manually remove the ISO in the Vultr admin dashboard (settings>iso).

Yet another step, after that you have to configure the interfaces via the TUI.

And finally, then go to https://, login with admin:pfsene and run through another installer.

Now you’re ready to actually configure it the firewall itself. Such as SSH rules etc.

A massive pain the arse.

Also a pain the arse; poor terraform errors like:

// this error is actually related to the network block about 30 lines
// below the resource at line 1
// this is up there with ansibles useless 'syntax error here or maybe somewhere else'
// errors
 Error: vm '101' not found
│ 
│ with proxmox_vm_qemu.pfsense_node[1],
│ on main.tf line 1, in resource "proxmox_vm_qemu" "pfsense_node":
│ 1: resource "proxmox_vm_qemu" "pfsense_node" {
│ 
╵

Here is a really helpful guide: https://jarrodstech.net/project-pfsense-setup-on-vultr-with-private-lan/

All this BS makes me wish their was a great infrastructure as code firewall which is open source. A lot of people will say they’d never want their firewall managed using IaC but I say they’re living in the past.

If it ain’t in code, it ain’t tracked - spreadsheets, device42 etc aren’t the same.

If someone can log on and add or delete a rule and something can’t detect that then its, in my opinion, a liability.

If anyone reads this and wants to educate me, please (and I’m not joking) email me and let’s talk.

Tags:

#pfsense #rant #terraform #iac

ArgoCD: refresh v sync

Wed, 11 Jan 2023 00:00:00 +0000

ArgoCD: refresh v sync

Sync: Reconciles the current cluster state with the target state in git.

Refresh: Fetches the latest manifests from git and compares the diff.

Hard Refresh: Clears any caches and does a refresh.

Tags:

#til #kubernetes #argocd

fc is a hidden linux gem

Sun, 08 Jan 2023 00:00:00 +0000

fc is a hidden linux gem

TIL about fc or fix command.

Typing fc and hitting enter will drop you into your $EDITOR and let you edit the last command. Saving will execute it.

For long curl requests its amazing. Now I can send a command and fc into to alter the endpoint data etc using vim whilst keeping all the authentication fields intact.

It’s far easier than re-issuing the command via the terminal prompt even with vim bindings.

Tags:

#til #linux #tools

Mudmap updates Jan

Sun, 08 Jan 2023 00:00:00 +0000

Mudmap updates Jan

Mudmap updates for my end of month Retro because I always forget what I did during the month.

OpenVPN status endpoint (backend) deployed Sun 08 Jan 2023

Tags:

#retro #mudmap

December 2022 Retrospective

Sat, 07 Jan 2023 00:00:00 +0000

Summary

Two releases for Mudmap and two weeks in Perth

Mudmap

System update available widget
Installed packages table

After finishing up the above features I took an extended break from all things code. I still took my laptop away but never really felt the need to open it up. My holiday back home was far too relaxing!

System Update Available

In pfSense, on the main dashboard, it tells if the version is up-to-date or if a newer version is available.

Mudmap now does this too. A small quality of life improvement but another step in replicating the pfSense experience within Mudmap.

This was simple for customers who are on the API version 1.5.0 and above. However, it needed to be backwards compatible for users that aren’t. Quite a few are using the older version and I did not want to prompt them to upgrade. This meant checking their currently installed version and providing a fallback for them. This fallback indicates that they should update and provides a link with how to do so.

Installed Packages

Another quality of life improvement. Users can now check which third party packages are installed and their versions. It is read-only, as in they cannot upgrade from Mudmap, but it is a start.

The inclusion of the system update and installed packages means auditing devices got a little easier.

Random Mobility Workout of the Day (MWOD)

This was something I knocked up in two evenings to service my own simple needs.

I love watching and following along with Kelly Starret. When I was in the military the MWOD gave me a lot of relief from pain. Its tough work doing cool shit! Since I’ve become a desk warrior I sort of let it go but that has brought its own problems. I have new musculoskeletal pains.

Anyway, I built this to prompt me each day with a random workout to perform. It’s a Go app using templates to pull a list of 160+ videos from YouTube and embed them in the site. Go plus Sqlite makes life pretty easy. Codebase

Check it out, here

Cookiecutter’s

I also knocked up a couple of [cookiecutter] application templates. This was the fist time I’ve created my own. I took inspiration from Simon Willison’s Click-App cookiecutter approach.

For most of my CLI applications I use bonzai, which supports nested composition meaning I can include multiple Bonzai apps in one. The hard part is creating a new Bonzai app - it required cloning, or using a template from GitHub. I didn’t like this because the template has a load of boilerplate I don’t need.

Instead, I now create new Bonzai apps using cookiecutter https://github.com/danielmichaels/bonzai-app. It’s pretty handy for my use case and I think most people could knock up a unique cookiecutter for themselves in a couple of hours.

Taking heed to my advice above, and just as I was about to start building out Random MWOD I decided this was a good opportunity to create a Go web app cookiecutter.

I like my Go web apps as a starting point to use go-chi, tailwind and alpine. So I created the cookiecutter to build a minimal app with these tools. It also uses air for hot-reloading and goose for database migrations.

This highly opinionated cookiecutter can be found, here.

Recommendations

Book: 100 Go Mistakes and How to Avoid Them (link)

I write a decent amount of Go but there are various parts of the language which have tripped my up in the past. This book covers all the issues I’ve ever footgunned myself with. Each one explained well with not just why it happened but also the methods to fix it.

For any Go developer I think its worthwhile reading. So far I’ve learned a lot more about slices (under the hood), strings (particularly the importance of Runes) and solidified my knowledge in control structures.

There is usually a few ways of achieving things in programming. Something I really like about this book is that the author is opinionated about why x thing is the best method. When there are caveats those are well explained, too.

For instance, when checking if a slice is empty the best way is to check its length. This will trap both nil and empty slices.

// checks for nil and empty
// empty slices have a len of zero and nil slices are always empty
if len(foo) == 0 { 
	return false
}

I’m only about 40% of the way through, and I’ve referenced it a couple of times already. Another thing which is really neat; succinct little sections for each Go Mistake. This means you can easily pick it up whenever you’ve got a spare moment and it reads well on Kindle.

Beach Mode

The rest of this month was me soaking up rays at Perth’s beautiful beaches.

Kong is an amazing CLI for Go apps

Sat, 07 Jan 2023 00:00:00 +0000

Kong is an amazing CLI for Go apps

Firstly, I want to acknowledge that I love Bonzai for command composition and most CLI’s. It’s my goto and is well designed.

However, for complex applications in a team (who need get-opt like tooling) Bonzai doesn’t quite fit the bill.

Why not Cobra? It bloated as hell. The code generator creates in my opinion code that quickly gets messy and is hard to test.

I was using urfave/cli for quite a while and like it but started hitting some issues when I structured it how I like it. Funny formatting issues, global flags not working correctly, or overwriting subcommand flags.

Now I use alecthomas/kong. Here’s an example from its git repo using a struct based approach (it’s kind of similar to mitchellh/cli in this way).

This is a Docker like CLI.

// ref: https://github.com/alecthomas/kong/blob/master/_examples/docker
package main

import (
	"fmt"

	"github.com/alecthomas/kong"
)

type Globals struct {
	Config string `help:"Location of client config files" default:"~/.docker" type:"path"`
	Debug bool `short:"D" help:"Enable debug mode"`
	Host []string `short:"H" help:"Daemon socket(s) to connect to"`
	LogLevel string `short:"l" help:"Set the logging level (debug|info|warn|error|fatal)" default:"info"`
	TLS bool `help:"Use TLS; implied by --tls-verify"`
	TLSCACert string `name:"tls-ca-cert" help:"Trust certs signed only by this CA" default:"~/.docker/ca.pem" type:"path"`
	TLSCert string `help:"Path to TLS certificate file" default:"~/.docker/cert.pem" type:"path"`
	TLSKey string `help:"Path to TLS key file" default:"~/.docker/key.pem" type:"path"`
	TLSVerify bool `help:"Use TLS and verify the remote"`
	Version VersionFlag `name:"version" help:"Print version information and quit"`
}

type VersionFlag string

func (v VersionFlag) Decode(ctx *kong.DecodeContext) error { return nil }
func (v VersionFlag) IsBool() bool { return true }
func (v VersionFlag) BeforeApply(app *kong.Kong, vars kong.Vars) error {
	fmt.Println(vars["version"])
	app.Exit(0)
	return nil
}

type CLI struct {
	Globals

	Attach AttachCmd `cmd:"" help:"Attach local standard input, output, and error streams to a running container"`
	Build BuildCmd `cmd:"" help:"Build an image from a Dockerfile"`
 // omitted all the other options
}

type AttachCmd struct {
	DetachKeys string `help:"Override the key sequence for detaching a container"`
	NoStdin bool `help:"Do not attach STDIN"`
	SigProxy bool `help:"Proxy all received signals to the process" default:"true"`

	Container string `arg required help:"Container ID to attach to."`
}

func (a *AttachCmd) Run(globals *Globals) error {
	fmt.Printf("Config: %s\n", globals.Config)
	fmt.Printf("Attaching to: %v\n", a.Container)
	fmt.Printf("SigProxy: %v\n", a.SigProxy)
	return nil
}

type BuildCmd struct {
	Arg string `arg required`
}

func (cmd *BuildCmd) Run(globals *Globals) error {
	return nil
}
func main() {
	cli := CLI{
		Globals: Globals{
			Version: VersionFlag("0.1.1"),
		},
	}

	ctx := kong.Parse(&cli,
		kong.Name("docker"),
		kong.Description("A self-sufficient runtime for containers"),
		kong.UsageOnError(),
		kong.ConfigureHelp(kong.HelpOptions{
			Compact: true,
		}),
		kong.Vars{
			"version": "0.0.1",
		})
	err := ctx.Run(&cli.Globals)
	ctx.FatalIfErrorf(err)
}

Example usage:

# go run . --help
Usage: docker <command>

A self-sufficient runtime for containers

Flags:
 -h, --help Show context-sensitive help.
 --config="~/.docker" Location of client config files
 -D, --debug Enable debug mode
 -H, --host=HOST,... Daemon socket(s) to connect to
 -l, --log-level="info" Set the logging level (debug|info|warn|error|fatal)
 --tls Use TLS; implied by --tls-verify
 --tls-ca-cert="~/.docker/ca.pem" Trust certs signed only by this CA
 --tls-cert="~/.docker/cert.pem" Path to TLS certificate file
 --tls-key="~/.docker/key.pem" Path to TLS key file
 --tls-verify Use TLS and verify the remote
 --version Print version information and quit

Commands:
 attach Attach local standard input, output, and error streams to a running container
 build Build an image from a Dockerfile

Run "docker <command> --help" for more information on a command.

Calling attach

# go run . attach --sig-proxy=false container123
Config: /home/dan/.docker
Attaching to: container123
SigProxy: false

What I love about it is the ease of use when structing your application. I like to put things into internal and Kong makes it easy to place my CLI command structs outside of the main/run command within cmd/cli/main.go. These CLI structs are then placed into the above examples CLI struct as fields and everything just works.

I swapped out urfave/cli for Kong in about 40 minutes - about 7 commands in total.

Highly recommend it.

Additional content (11 July 2024)

Here is another example:

package main

import (
	"fmt"
	"github.com/alecthomas/kong"
	kongyaml "github.com/alecthomas/kong-yaml"
	"os"
	"path/filepath"
	"text/template"
)

const appName = "pvenotify"

type Globals struct {
	ConfigFile kong.ConfigFlag `short:"c" help:"Location of client config files" type:"path" default:"${config_path}"`
	Version VersionFlag `name:"version" help:"Print version information and quit"`
	Username string `name:"username" help:"Username to authenticate with"`
}

var version string

type VersionFlag string

func (v VersionFlag) Decode(_ *kong.DecodeContext) error { return nil }
func (v VersionFlag) IsBool() bool { return true }
func (v VersionFlag) BeforeApply(app *kong.Kong, vars kong.Vars) error {
	fmt.Println(vars["version"])
	app.Exit(0)
	return nil
}

type CLI struct {
	Globals

	Watch WatchCmd `cmd:"" help:"Watch Proxmox for updates"`
}

type WatchCmd struct {
}

func (a *WatchCmd) Run(globals *Globals) error {
	fmt.Println("Executing ", globals)
	return nil
}

func main() {
	if version == "" {
		version = "development"
	}
	cli := CLI{
		Globals: Globals{
			Version: VersionFlag(version),
		},
	}
	// Display help if no args are provided instead of an error message
	if len(os.Args) < 2 {
		os.Args = append(os.Args, "--help")
	}

	configDir, err := os.UserConfigDir()
	if err != nil {
		_, _ = fmt.Fprintf(os.Stderr, "Failed to get user config dir: %v\n", err)
	}
	defaultConfigPath := filepath.Join(configDir, appName)
	defaultConfigFile := filepath.Join(defaultConfigPath, "config.yaml")
	err = initialiseConfigFile(defaultConfigPath, defaultConfigFile, cli.Globals)
	if err != nil {
		fmt.Fprintln(os.Stderr, "failed to initialise config file:", err)
		os.Exit(1)
	}

	ctx := kong.Parse(&cli,
		kong.Name(appName),
		kong.Description("A Proxmox Notification Service"),
		kong.UsageOnError(),
		kong.ConfigureHelp(kong.HelpOptions{
			Compact: true,
		}),
		kong.Configuration(kongyaml.Loader, defaultConfigFile),
		kong.DefaultEnvars(appName),
		kong.Vars{
			"version": string(cli.Globals.Version),
			"config_path": defaultConfigFile,
		})
	err = ctx.Run(&cli.Globals)
	ctx.FatalIfErrorf(err)
}

func initialiseConfigFile(configPath, configFileName string, globals Globals) error {
	if !doesNotExist(configFileName) {
		return nil
	}
	_, _ = fmt.Fprintln(os.Stderr, "config file does not exist. attempting to create it. ")
	err := CreateDirectoryIfNotExist(configPath)
	if err != nil {
		return err
	}
	fd := FileData{globals}
	tfile, err := os.ReadFile("./config.yaml")
	tmpl := template.Must(template.New("config").Parse(string(tfile)))
	err = generateDefaultConfigFile(configFileName, tmpl, fd)
	if err != nil {
		return err
	}

	_, _ = fmt.Fprintf(os.Stderr, "config file created at: %s\n", configFileName)
	return nil
}

func doesNotExist(path string) bool {
	if _, err := os.Stat(path); os.IsNotExist(err) {
		return true
	}
	return false
}

func CreateDirectoryIfNotExist(dirPath string) error {
	if err := os.Mkdir(dirPath, 0755); err != nil {
		return err
	}
	return nil
}

type FileData struct {
	Globals
}

func generateDefaultConfigFile(fp string, tmpl *template.Template, data FileData) error {
	if doesNotExist(fp) {
		file, err := os.Create(fp)
		if err != nil {
			return err
		}
		defer file.Close()

		if err := tmpl.Execute(file, data); err != nil {
			return err
		}
	}
	return nil
}

This example will read from (and in this exact order):

flag
config file
environment variables

This is a little surprising to me. I would expect:

flags
environment variables
config file

Hopefully I am doing it wrong and theres a way to enforce that. Until then, I would have to rethink how to best configure such a CLI/tool and whether or not to leverage a config file/env/flag mixture.

In contrast, https://github.com/peterbourgon/ff does work as I expect. I have another zet with a working code snippet showcasing it.

Still, I much prefer the style of kong. The interface approach works well for my mental model, as does the struct tags approach. It is very easy to follow the code especially when the structs and/or methods are split across multiple files.

Tags:

#cli #go #kong

Why twitter beats mastodon

Wed, 21 Dec 2022 00:00:00 +0000

Why twitter beats mastodon

Mastodon is the HackerNews version of Twitter. Only technical people navigate to the site, and even less contribute.

One of Twitter’s downsides is it is an echo chamber with zero-bar for entry. It’s designed to draw engagement out from every single user. From the follower counts, retweets, likes and impressions - literally everything encourages maximum engagement.

This can breed interactions akin to the child who seeks attention from their parents but doesn’t care if its good or bad attention.

But, you can bypass all this stuff through curation, muting, blocking and ignoring. Between stimulus and response lies the freedom to choose your on way - it’s you who chooses to engage or enrage.

Some will say the Mastodon is the answer here. I strongly disagree. It is a walled garden that you build yourself. You design your own cage based on the thoughts and feelings you like. I thought Twitter was an echo chamber and then I found Mastodon.

There is limited discovery of new ideas. Very few reasons to expand your own horizons. And a corpus of mostly one-sided political and societal views. Tech Twitter is a vacuum but the majority of new Tooters are either leaving because they’re following the latest trend, or disagree with the current leadership.

To the last point, that’s a valid reason. But now you’re just attracting a whole bunch of folks who look different but think alike. Hardly an environment to generate discourse with anything more than thinly veiled like-a-ship seeking behaviour.

Couple all of this with the overall difficulty in finding a server, an app to use and the finding good folks to follow (who are often disparately spread across several servers) its a steep hill to climb. For the tech crowd it’s a badge of honour that you figured it all out. For the sports, educational, political, societal, environmental and the list goes on, others who don’t have the inclination, time or prowess to figure it all out? Well too bad dummy! iTs NoT ThAT hArD :sponge_bob_meme:

My parents can get on Twitter. They won’t make it to Mastodon in this lifetime.

But maybe that’s what the new tooters kind of want - a walled garden to call their own.

Tags:

#rant #mastodon #twitter

k3s: tuning local development

Mon, 19 Dec 2022 00:00:00 +0000

k3s: tuning local development

A couple of tunables for single node master/agent k3s.

Fix high CPU on gnome

Higher usage on Gnome/KDE can be caused by the large number of overlay volumes.

systemctl stop --user gvfs-udisks2-volume-monitor

Disable leader election

For a single node development environment removing leader election can reduce CPU usage.

cat << EOF | sudo tee /etc/rancher/k3s/config.yaml
kube-controller-manager-arg:
 - "leader-elect=false"
 - "node-monitor-period=60s"

Tags:

 #k3s #kubernetes #TIL

Go:embed reading individual files

Sun, 18 Dec 2022 00:00:00 +0000

Go:embed reading individual files

I was having trouble accessing a particular file (json, in this case). In the past, I’ve had no drama getting at templates and the like but was clearly missing something here.

To access a single file, you can use the ReadFile method on the embedded filesystem.

For instance,

# tree .
├── main.go
└── embeds
 └── stuff.json

I want to explicitly unmarshal stuff.json in my main.go

An abridged version of how to do that.

package main

import "embed"

//go:embed files
var jsonFiles embed.FS

fun main() {

 js, _ := jsonFiles.ReadFile("embeds/stuff.json")
 
 var p map[string]any
 err := json.Unmarshal(js, &p)
 if err != nil { log.Fatalln("failed to read embedded file") }

This opens the file for reading and gets all of its contents. A perfect solution for my needs.

To read the entire directory just swap out ReadFile with ReadDir("embeds").

Tags:

 #go #embed #TIL

Python rich progress bar and httpx

Thu, 15 Dec 2022 00:00:00 +0000

Python rich progress bar and httpx

How to create a progress bar using rich and httpx:

path = "/tmp/file"
url = httpx.URL("https://releases.ubuntu.com/20.04/ubuntu-20.04.3-desktop-amd64.iso")

with httpx.stream("GET", url=url, verify=False, follow_redirects=True) as r:
 with Progress() as progress:
 size = int(r.headers["Content-Length"]
 with open(path, "wb") as dest:
 dl = progress.add_task("[red]Downloading Ubuntu[/red]", total=size)
 for data in r.iter_raw():
 dest.write(data)
 progress.update(dl, advance=len(data), description="[yellow]Downloading...[/yellow]")
 progress.update(dl, description="[green]Download Complete[/green]")

Tags:

#python #httpx

Reminiscing

Tue, 13 Dec 2022 00:00:00 +0000

Reminiscing

Listening to a workmate talk about shooting MP5’s and other guns whilst at re:Invent in Vegas last week brought back fond memories.

He was talking about how wild it was shooting an MP5 on fully automatic.

Meanwhile, I’m sitting at my computer watching my Gitlab pipelines run thinking back to the last time I shot one. It was before breakfast on a random weekday down at the range practicing vehicle break contacts. We rolled up in the car to a (fake) checkpoint which had some drop head target dummies. On initiation I dropped a whole mag straight through a windscreen then made a fighting withdrawal throwing smoke, grenades and shooting off 40mm. The MP5k has some kick but it don’t matter too much when you trying to kill anything in front of you with it.

Fond memories from a life I used to lead and also took for granted. We truly lived most men’s dreams.

10/10 do not regret my time in the Regiment.

Tags:

#sas #shooting #reminisce

Golang: JSON API returning different responses

Sun, 11 Dec 2022 00:00:00 +0000

Golang: JSON API returning different responses

I am interfacing with an API which returns different responses from the endpoint dependant on a success or error. I want to be able to handle every response and this is how I did that.

A successful response looks something like:

[
 {
 "id": 1,
 "hop": 1,
 "url": "https://danielms.site",
 "http_version": "HTTP/1.1",
 "status_code": {
 "code": "200",
 "phrase": "OK"
 }
 }
]

And an error:

{
 "detail": {
 "error": "The URL could not be resolved.",
 "url": "https://danielms.site2",
 "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.101 Safari/537.36"
 }
}

In order to handle both events I created response structs for each and then added those a generic response struct. The generic response struct I could then create a UnmarshalJSON method for which fulfils the Unmarshaller interface.

Here is the struct and method.

type ResponseTypes struct {
	Response []RedirectResponse
	ErrorResponse ErrResponse
}

func (d *ResponseTypes) UnmarshalJSON(data []byte) error {
	fmt.Println(string(data))
	var m map[string]any
	var unmarshalErr error

	if err := json.Unmarshal(data, &m); err != nil {
		unmarshalErr = err
	}

	if _, ok := m["detail"]; ok {
		var errData ErrResponse
		err := json.Unmarshal(data, &errData)
		if err != nil {
			return err
		}
		d.ErrorResponse = errData
		return nil
	}

	var arr []map[string]any
	if err := json.Unmarshal(data, &arr); err != nil {
		unmarshalErr = err
	}

	if _, ok := arr[0]["status_code"]; ok {
		var respData []RedirectResponse
		err := json.Unmarshal(data, &respData)
		if err != nil {
			return err
		}
		d.Response = respData
		return nil
	}
	return unmarshalErr
}

The hard part was handling both slice and single element structs at the same time. Initially I had issues because in order to inspect the response I needed to unmarshal it. There might be a better way but above I’ve just unmarshalled again if there isn’t any error.

Because Go is awesome, I don’t have to do anything out of the ordinary to use the ResponseTypes.UnmarshalJSON method. The interface is fulfilled so I just call it as I would any other UnmarshalJSON method call.

// used in another function
r, _ := io.ReadAll(response.Body)
var result ResponseTypes
err = json.Unmarshal(r, &result)
if err != nil {
 return err
}
fmt.Println(result)

refs:

https://www.calhoun.io/how-to-parse-json-that-varies-between-an-array-or-a-single-item-with-go/

Tags:

#go #TIL

Automatic Pet Door

Sat, 10 Dec 2022 00:00:00 +0000

Automatic Pet Door

I have a dog and recently got a cat. The cat keeps going out the dog door which poses a problem; how do I allow the dog to go out freely but prevent the cat from going out?

Enter a DIY automatic pet door.

Requirements:

Dog can go freely
Cat must not be able to exit freely
Anything on animals must be passive i.e. not battery powered
No circuitry must live on the pet door (a flapper)

Idea: tag based access control:

the dog door locks when a tag is presented (the cats).

I think the door being default to unlocked works better than the inverse. This way if the cat is at the door whilst the dog it outside the dog would be locked out. This is a design consideration which should prevent the cat from tail gating or ambushing the door when the dog enters or exits.

My current dog door is a flap based system which uses light magnets to keep the flap in place. I want to keep using this as its a pet door which can be used in any sliding door (great for rentals).

The locking mechanism would be stronger magnets which are energised whenever the cat is present preventing the flap from opening.

A few options:

Acousto Magnetic detection
RF detection

Tags:

#rfid #rf #diy #iot

Hobby as a job has its downsides

Sat, 10 Dec 2022 00:00:00 +0000

Hobby as a job has its downsides

When I was in the army my mental stimulation hobby was coding. I’d work odd hours, sometimes away for weeks at a time with no downtime. So whenever I could I would be coding up something. This process was hugely rewarding to me.

Now that my day job is coding, and my hobby is coding it’s less rewarding.

Not in a bad way - I still love getting home to work on my stuff. Instead in a way that I now know how hard it can be to build something from start to finish, and to maintain it. It makes me less keen on building things that maybe a few years ago I would of jumped at. This means I’m giving up things that I’d otherwise do simply because my hobby is also my job. At least that’s how I see it.

Tags:

#rant #life

RF based detection: terms and definitions

Sat, 10 Dec 2022 00:00:00 +0000

RF based detection: terms and definitions

A few definitions for radio frequency detection tools.

Acousto-Magnetic

Emits a signal at 58kHz in bursts which energise any tag caught in the surveillance zone. These are very effective at tracking items which may contain metals or metallic paint. Tags are quite small though I don’t think they are coded meaning you cannot track items with this alone.

Tags:

#rf #definitions

My site has RSS

Tue, 06 Dec 2022 00:00:00 +0000

My site has RSS

The other day I was trying to figure out if my site; danielms.site has support for RSS.

Hugo supports RSS out of the box which means my site does too but I was looking in the wrong location.

RSS: https://danielms.site/index.xml

Tags:

#til #hugo #rss

November 2022 Retrospective

Fri, 02 Dec 2022 00:00:00 +0000

Summary

Three releases for Mudmap! 🎉

Mudmap

Switched data model to sqlc
Added multi-stage installer
pfSense logs

Switching to sqlc

This isn’t something I needed to do so why do it? Two reasons; type safety for the data model and reduce friction when adding or updating models.

I experimented with gRPC for another project a couple of months ago and was awestruck by how much the gRPC compiler does for you. It autogenerates thousands of lines of type safe code which you can then call without needing to write anything yourself. As an average at best developer I loved this. Updating the yaml definition rebuilds the interface with a single command and all the callers get updated. Of course, you might have to change your business logic depending on what you’ve updated, but you don’t need to fiddle with any of the service layer. It’s a huge time saver.

sqlc gives you this same experience but with sql. If tomorrow my Device model needs to include a Serial field I would have to update every sql statement, all the sql.Scan method calls and then worry about the implementation within handler and service layers. Instead, by using sqlc I just update my sql queries and model definitions and the entire data model will be auto generated for me with those changes. Now all I need to do is focus on business logic changes (which the go type system will catch).

It was a bit of learning curve at first especially with uuid fields and null’s but that speaks more to my flawed database schema design choices. Rant alert: for any product/project I produce in the future I will never use another uuid. A Stripe-like id (e.g. pi_123abc456) or Terraform’s random_pet (e.g. epic_sawfish_123) would be more than sufficient.

Multi-Stage Modals

Perhaps my biggest cross the bear in Mudmap is the flaky installer. It really gets me down, and I’ve still yet to find a good way to fix this issue - its churned one of my biggest customers too.

This feature does not fix the problem, instead it’s part of my pathway to making the installation process more pleasant.

How it works (starts at Password Confirmation):

A crude drawing

This change introduced a multi-stage modal instead of a single step. Using a multi-stage allowed for better error handling and branching when prompting users for input.

Additionally, the Device’s root password is now being stored in an in-memory database. I’ve made an ideological stand against storing this password in Mudmap’s actual database. I feel that storing the password in a memory datastore is a good practice and does not break the promise of not storing it in the database. go-memdb was an easy choice for this. The detractor is that in the future I may need to switch to an external key value store such as Redis if I spin up multiple services.

By storing a Device’s root password in memory it can now be retrieved during the installation process across multiple stages. Without doing this, each stage would require the user to re-enter the password. This is not only inconvenient but also introduces needless complexities like what happens if they re-enter it wrong - it is another error handler and conditional branch that needs to be dealt with.

Adding this also opens up future enhancements such as a proper Deletion event. When a Device is deleted or the installation fails, Mudmap must be removed from the firewall. This requires the removal of the Mudmap service account before removing the API. Once the Mudmap service account is removed every action taken must be done as the root user. This requires root access which with this new addition does not require user intervention.

A bad practice which I am trying to fix (and this fixes most of it) is not deleting Mudmap artifacts when an installation fails. It is possible that these artifacts on subsequent re-installation attempts may even cause issues. But, the biggest problem is Mudmap isn’t being a good citizen and reverting a user’s device back to the state it was when they first tried the platform out.

I do not expect this will fix failed installations. What I might try next is to let users install it manually. This would require a few steps and may be too much friction, but it gives the user power to do something when an issue is encountered. Another thing I am contemplating is storing some metadata about each device in an attempt to draw some correlations between successful and unsuccessful installation attempts. I am unsure about this though.

pfSense Logs

This month I also added a few more endpoints and user-facing pages for viewing the following logs:

Firewall
System
DHCP
pfSenseAPI

A small-ish update but added functionality all the same. Users can now view the logging of the above from within Mudmap. The pfSenseAPI audit logger is particularly handy. Eventually I will need to augment this with the user controlling the API, not just the API itself.

Writing this feature required a series of regex parsers to be created. The response from the API returns a single string per event. By creating custom parsers it was possible to create useful struct fields which could then be returned as json objects. A methodology which will likely serve other endpoints as they are integrated. Not captured in this work (scope creep) but planned is reading, updating and deleting system packages. Installing packages will be its own ticket but the methodology to do that is already there.

These small features when stacked up actually make Mudmap much more useful. Having to SSH into every device just to do a system and third party package audit is painful, soon Mudmap will remove that burden.

Zettelkastens

I use a custom implementation of a zettelkasten which I use to store snippets of information. The storage backend is GitHub and each Zet is just a single markdown file stored in a timestamped directory within my zet repo. Sometimes I want to review something I’ve written, but I’m not at my computer and searching for it on GitHub is inefficient. So I wrote a Go tool which would embed it into this site.

At first, I wrote it to just link directly to the Zet’s markdown file on GitHub. I found this wasn’t a great experience as it would take a few seconds to transition and the layout shift was jarring. I did learn how to render a json file using Hugo’s custom shortcodes though.

Instead of this I decided host the markdown files directly on this site. I enjoyed writing this because it leverages one of Go’s truly great strengths; text/template. If you’ve come from python you’ve probably heard of or used Jinja2 as the templating engine. It’s a sweet module but its another dependency and having an inbuilt templating engine within Go is one of its underappreciated features.

How I leverage Go’s text/template to auto generate Hugo’s frontmatter can be found here. The code is pragmatic and works but don’t look to it as an exemplar of good but instead practical. file1, file2 and GitHub actions.

Recommendations

FIFA Uncovered

Shock horror FIFA is corrupt as hell. A well crafted overview of how FIFA quickly became a money under the table organisation and how it continues to do so today. When people can wield absolute power (without any oversight) they will abuse it for their own gain.

What’s next

I’m testing out a new work/fun balance; 3-4 days on mudmap 3-4 days on whatever makes me happy per week. Going too hard too soon on Mudmap has been the thing that burns me out. Also, once I release a feature or update I take a couple of days off to build whatever as a reward. Doing this has actually lead to days where I’ve just jumped on Excalidraw and explored a Mudmap feature to build in the future. I’ll play with this a see where it takes me.

Goals

I want to release another Mudmap update before Christmas.

Analytics

mudmap.io

danielms.site

Golang: find uniques

Tue, 29 Nov 2022 00:00:00 +0000

Golang: find uniques

Coming from python where getting a Set from a list is as simple as calling s = set(array) the go version is quite verbose.

Here is how I am finding unique values from an array of integers.

func dedupeInts(ints []int) []int {
 // create a map of int:bool to track found int's
 all := make(map[int]bool)
 // create an array of int's which we'll add any unique entries to
 var list []int
 
 for _, item := range ints {
 // if value (int) is not in the map marked as true then add it the
 // the map and assign true
 // this is how we check for unique entries
 if _, value := all[item]; !value {
 all[item] = true
 list = append(list, item)
 }
 }
 return list
}

Tags:

#golang

Golang: case statements with an OR clause

Sat, 26 Nov 2022 00:00:00 +0000

Golang: case statements with an OR clause

TIL how to use Go’s switch and case statement with || (logical OR) like functionality:

switch foo {
 case "thing", "things":
 println("captured both")
 default:
 println(foo)
}

Reference:

https://go.dev/ref/spec#Switch_statements

Tags:

#go

Mudmap: installer fixes

Fri, 25 Nov 2022 00:00:00 +0000

Mudmap: installer fixes

This week I’ve been working on fixing Mudmap’s installer.

I’ve done the math and nearly 50% of installs have failed inexplicitly in the last 3 months. For users which installation worked without issue they’re happy and never complain. Obviously, the inverse has a lot of complaints. I am no closer to diagnosing exactly what causes this. A consistent (>60%) of failures trigger at the user account install step.

Given this I am exploring that step in detail but also changing the way the installer is handled on the front end. The error users receive just confuses them and that needs to go.

I’m also focusing on revert the device back to its pre-mudmap state with better consistency. This is important for users, and for future installs. To do this without storing users root password, I’ve elected to use an in memory data store - go-memdb - which is super lightweight and easy to understand.

Tags:

#mudmap #go

Slack apps

Fri, 25 Nov 2022 00:00:00 +0000

Slack apps

I’m thinking about making a slack app. I’ve been playing around with it on/off for the last week and it is an impressive API surface. Documentation is really good. Perhaps the hardest part is that most examples and tutorials centre around Node, Python or Java. Slack provides bolt in those languages but I basically refuse to write anything in python or node considering their memory bloat and difficult package management.

Writing a Slack app using Go has not been as easy as it would be using python or node though. This is due to the way Slack’s BlockKit is designed. Being able to dynamically update json is much easier in those languages. Stil, the slack-go library is pretty good. It’s only detractor is documentation. I’ve had to read source, and just try various things to get parts working.

The app I want to explore, and if viable write is a cloud platform manager. Render, Fly, Digital Ocean, Vultr etc all provide API access and some a full CLI. I think being able to get a read out of services, logs, billing etc via slack would be pretty useful. This means you can check your cloud assets via slack from anywhere - the CLI is limited to your machine. It also means you need an API on the device you’re using.

A stretch goal should this be viable is to create is for several clouds. I think at first it would be for Render or Fly.

Lots more ideation and overall design is needed but at worst this would be something I’m keen on using (I use both Render and Fly).

Tags:

#ideation #slack #go

inlets troubleshooting

Mon, 14 Nov 2022 00:00:00 +0000

inlets troubleshooting

inlets.dev troubleshooting.

So far, I’ve had some issues getting to work out of the box. I think this mostly comes down to the documentation drifting out of sync with the project.

inletsctl create \
--access-token-file ~/.doctl-token \
--provider digitalocean \
--region sgp1 \
--letsencrypt-domain inlets.cupscanteen.com \
--letsencrypt-domain t.cupscanteen.com \
--letsencrypt-domain slack.cupscanteen.com \
--letsencrypt-email webmaster@cupscanteen.com

Sometimes this doesn’t work. So to get it working, I jump on the host. For DO the password is emailed when you create the host (which the above step does for you).

vim /etc/default/inlets-pro
# add any extra domains with --letsencrypt-domain <sub>.<domain>.<tld>
# after the DOMAINS=

The documentation states its --letsencrypt-domain=foo.com,bar.com but this does not seem to work.

systemctl restart inlets-pro and see if the process has started correctly. It can take time to get the cert from Let’s Encrypt.

If this isn’t working, try changing /etc/default/inlets-pro ISSUER from prod to staging and restart the service. This should connect and if so, you know its a Let’s Encrypt issue. Potentially the IP has been temporarily rate limited, or the DNS entry does not exist so it cannot do a HTTP01 interrogation

Tags:

#inlets #tunnelling #troubleshooting

Golang *pq.Error handler for postgres

Sat, 12 Nov 2022 00:00:00 +0000

Golang *pq.Error handler for postgres

Since switching over to sqlc I no longer have access to the models. This means custom error message returns are not possible (at least to my knowledge). This caught me out as a duplicate key error which used to get trapped was slipping through my validation in Mudmap.

Here’s how to catch postgres errors using github.com/lib/pq.

Note: apparently pgx has much better handling for this out of the box.

// from inside a Mudmap handler.
dd, err := app.Db.DeviceOrgInsert(context.Background(), orgArgs)
		if err != nil {
 // here we crate a variable of the pq.Error type
			var pgErr *pq.Error
			switch {
 // Using errors.As we inspect the error type
			case errors.As(err, &pgErr):
 // I chose to be explicit and call .Name() rather than
 // case over the Code which in this case is 25302 or something
				switch pgErr.Code.Name() {
				case "unique_violation":
					v.AddError("devices_host_address_key", "a device with this host address already exists")
					app.failedValidationResponse(w, r, v.Errors)
				default:
					app.Logger.Error().Err(err).Msg("pg-error")
					app.serverErrorResponse(w, r, err)
				}
				return
 }
 return
 }

Tags:

#go #postgres #errors

Typescript: basics

Fri, 11 Nov 2022 00:00:00 +0000

Typescript: basics

Return type on an arrow function

const greet = (name: string): string => {
 return `Hi ${name}`
}
console.log(greet("Dan"))

Prefer type or interface when return types are more complicated.

type Player = {
 name: string;
 number: number;
}

const getPlayer = (): Player => ({
 name: "Jordan"
 number: 23
})

If your function could return different types, use a union

function getNameOrNumber(name: string): string|number {
 if name === "Jordan" {
 return 23
 }
 return "who cares"
}

Tags:

#typescript #programming

October 2022 Retrospective

Sat, 05 Nov 2022 00:00:00 +0000

Summary

A month of working on only things I find fun.

Contract Hour Tracking

I am a full-time employee on a hourly contract basis. It’s a funny position because I’m effectively paid casual rates but full-time. Unlike most other contracts the possibility of my contract being cancelled is nearly zero due to the extreme staff shortages in my workplace.

With that, I am given a set number of hours I can work within the contract period. I cannot go over but any hours remaining at contract end is simply money I’m robbing myself of. Unfortunately, I’m pretty lazy about tracking it all and when I did, I realised that I’ve under paid myself over this contract period. A project idea was spawned.

Enter a rather simple contract timesheet calculator; timesheet-calculator. It’s pretty simplistic in the output it provides but so far has been pretty helpful. Every day it sends an email with some stats to keep an eye on such as my mean daily required hours to reach contract zero. The most useful emails are my weekly and monthly ones. The monthly email is the most useful as it spits out the daily log in tabular format making filling out my monthly timesheet much easier.

A trivial project but for those with my level of laziness automation is a solution worth the time investment.

Nomad

At work, we’re pretty tied to Kubernetes and are actively looking at replacing our OpenShift cluster. The reasons are varied, but it mostly comes down this fact; it’s too much for us. We’re a small team of developers who also maintain our own Kubernetes cluster which all of our workloads run on. Without a dedicated Ops team its starting to drain our resources.

Our initial investigations into alternatives mostly centred around other Kubernetes platforms. One of the key metrics is it must be on-prem without the need for an active internet connection - we live and die by the proxy.

At first, we dismissed Nomad, mostly because of the time we’d spent on Kubernetes and the significant amount of work that’s gone into creating its supporting assets; manifests and pipelines as the exemplars.

Sunk cost fallacy aside, Nomad, on the surface looks like it suits our needs more adeptly. I wrote a couple of zets about Nomad; some basics and what I think is interesting about it. I spent a bit of time playing around with the nomad agent -dev mode but couldn’t commit too much more time to getting a home lab up and running. Though I think that is something I need to do in November, or over the Christmas holiday period.

Mudmap

I have not done any work on Mudmap this month.

Honestly, I am still unsure what to do with it. People are still interested, but I am unsure if it’s still right for me.

Recommendations

Andor

This is the first Disney spin-off that I have enjoyed. I wrote an entire rant about it here. It feels like the first real storyline to emerge from the Star Wars universe that isn’t just a rehash of existing lore, or absolute garbage (Mandalorian and Obi-Wan).

What’s next

I’ll keep on making things that are useful to me whilst I figure out the bigger picture of what I want to do.

Goals

I think my main thing will be to continue this year seeking enjoyment in my downtime.

Analytics

mudmap.io

danielms.site

Nomad Homelab

Thu, 03 Nov 2022 00:00:00 +0000

Nomad Homelab

I use kubernetes at work as a developer and as a maintainer. This includes OpenShift and k3s. For big org’s or teams this is great but I am starting to come around to the idea that it’s too much for us. Perhaps Nomad is a better more easy to administer solution.

Things I am interested in about Nomad:

qemu driver and docker driver
HCL (we use HCL for packer and terraform already)
Simpler, single binary setup
Native integration into Vault
Service discovery in-built (or can use Consul)
Great UI

Things I’ll miss:

Terminal interface like k9s (thouse damon exists as a Nomad version)
K8s ecosystem
Much less internet documentation
More work to get some apps going (compared to the plethora of helm charts out there)

What I worry about (for org, not pers use):

Enterprise paywall

Over this holidays I will be bringing up my own cluster to play around with.

Tags:

#hashicorp #kubernetes #nomad #infra

Workout finishers

Thu, 03 Nov 2022 00:00:00 +0000

Workout finishers

Some cheeky workout finishers:

Strength w/ Cardio

# repeat 3-4 times
# choose your adventure:
# 12, 16 or 20KG buyers choice
8 static lunges (each side)
5 clean and press (each side)
12 swings
# rest 1 min

# 3 rounds
100KG sled push @ 15m
5 24" box jump
5 Turkish Getup @ 5kn ea side
# no rest

Strength

# repeat 3-4 times
12 push-ups (banded)
20sec hollow hold
12 single-leg calf raise @ 32KG KB
# rest 1 min

Cardio

# assault bike
3 x 15s on / 15s off
1min pedal
# repeat 2-3 times

Rehab

hips

6 single-leg bulgarian split squat (12kg)
5 arabesques (6kg)
# 3 rounds 
6m each way, banded side walk with 10kg plate above head
# 3 rounds (plate keeps you from leaning)
20s copenhagen
6e/side seated straight leg lift over kettlebell (don't lean back)
6e/side matching side plank
# circuit 3x through

Tags:

#kettlebell #workout #fitness

Format JSON on The CLI Easily

Tue, 01 Nov 2022 00:00:00 +0000

Format JSON on The CLI Easily

python -m json.tool <file>

This can then be chained to overwrite an existing file, effectively formatting it.

python -m json.tool <file> | tee /tmp/file.json && cat /tmp/file.json > <file>

This gives the command multiple out’s without breaking the originating file. Probably a far better way but this works.

jq would work but it isn’t installed on remote Unix systems whereas python -m json.tool always works.

Tags:

#linux #json

Nomad: basics

Tue, 25 Oct 2022 00:00:00 +0000

Nomad: basics

Nomad is a scheduler/orchestrator for tasks. It can handle containers, VMs, .NET and Java application through its scheduler.

Highlights

Single binary; distributed; leader election and replication for HA
Plugin support, and GPU, FPGA and TPU out of the box support
Multi-region federation
Highly scalable (10k+ nodes in production in real-world examples)
Native integration into Hashicorp products like Vault, Consul and Terraform
Batch processing support (Kubernetes doesn’t do this well)

Versus Kubernetes

Kubernetes is an orchestration system for containers originally designed by Google, now governed by the Cloud Native Computing Foundation (CNCF) and developed by Google, Red Hat, and many others. Kubernetes and Nomad support similar core use cases for application deployment and management, but they differ in a few key ways. Kubernetes aims to provide all the features needed to run Linux container-based applications including cluster management, scheduling, service discovery, monitoring, secrets management and more. Nomad only aims to focus on cluster management and scheduling and is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery/service mesh and Vault for secret management.

source

Installing

Ubuntu:

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install nomad
# grab all Hashicorp tools if needed with
# sudo apt-get install packer terraform consul vault nomad

Definitions

Cluster:

agent - An agent is a Nomad process running in server or client mode. These are the building blocks of a Nomad cluster.
dev agent - Developer agent with defaults for running experiments or a dev cluster. It does not persist any state to disk.
server - Server mode agents are the “Control Plane” for the cluster. There is a cluster of servers per region and they manage all the jobs and clients, run evaluations, and create the task allocations.
leader - This is the server which performs the majority of the cluster management. It is in charge of applying plans, deriving vault tokens for workloads and administering the cluster state.
follower - Followers submit plans to the leader to provide more scheduling capacity to the cluster. The leader executes plans from the followers.
client - An agent running in client mode. These agents watch for any work and execute tasks. Client connections are multiplexed to servers.

Nomad Objects:

job - Defines one or more task groups which contain one or more tasks.
job spec - Schema definition for a Nomad job. Describes the type, tasks and resources required to run the job.
task group - A set of tasks which must be run together. Example, a web server may require a log shipping co-process is always running as well. A task group ensures that these two tasks are always run together.
task - The smallest unit of work in Nomad. These are executed by task drivers which allow Nomad to be flexible in the type of tasks it supports.
allocation - A mapping between a task group and a client node. A job can have hundreds or thousands of task groups which means an equivalent number of allocations must be mapped to client machines. Allocations are created from scheduling decisions known as an evaluation.
evaluation - The way in which Nomad makes scheduling decisions. When the state changes a new evaluation is triggered to determine the next action.

Scheduling:

bin packing - This is how Nomad maximises allocations on a device.
spread scheduling - The opposite of bin packing; this attempts to evenly spread work across all the machines in the fleet.

Starting Nomad

Nomad relies upon agent; an agent must be on every machine in the cluster. The server is responsible for managing the cluster and all other agents should be in client mode. Clients register the machine, conduct heart beats and run tasks when assigned from servers.

In development:

sudo nomad agent -dev -bind 0.0.0.0 -log-level INFO

Tags:

#hashicorp #nomad

Andor: finally a great disney spin off

Sat, 22 Oct 2022 00:00:00 +0000

Andor: finally a great disney spin off

As a Star Wars fan who hated the Mandalorian and all the other crap Disney+ money grabs, I am pretty happy with Andor.

The plot is interesting, Cassian’s character is realistic and gritty. Diego Luna does a great job playing him unlike the half-arsed acting in the other spin-offs. That villain woman from Obiwan was so unbelievable and over the top that it totally ruined it for me.

Star Wars should focus more on these plot driven, unrelated to to the typical story line shows. I mean, c’mon Luke Skywalker in the Mandalorian was just.. bad! Also, the Mandalorian literally had no storyline. Just crap.

Tags:

#rant

kubernetes: connection to the server localhost:8080 error

Thu, 20 Oct 2022 00:00:00 +0000

kubernetes: connection to the server localhost:8080 error

This annoying error is the result of a bad $KUBECONFIG file/ reference. To fix this you will need to get a new copy from the server.

For k3s this will mean scp’ing /etc/rancher/k3s/k3s.yaml to your client and then changing the server IP from 127.0.0.1 to the actual IP of the remote.

oc helpfully does all this for you but k3s does not.

Tags:

#kubernetes #troubleshooting

goland: highlight error without mouse hover

Mon, 17 Oct 2022 00:00:00 +0000

goland: highlight error without mouse hover

In Goland (and probably all intellij products; keymap dependant) to display the squiggly underline tooltip for a problem/error all you need is Alt+t

Couple this with displaying a function/methods comments (Shift+k) and I now rarely need to use my mouse at all.

Tags:

 #golang #productivity #intellij

create cli reminder script

Sun, 16 Oct 2022 00:00:00 +0000

create cli reminder script

I need to create a simple CLI interface that creates email reminders for things I need to do.

Use Bonzai to do this and send messages with a datetime to the cloud

Tags:

#project #bonzai #go

go parsing seconds into time.Duration

Sun, 16 Oct 2022 00:00:00 +0000

go parsing seconds into time.Duration

If you have time in seconds and need to turn that into a duration such as 2h30m use time.ParseDuration

# https://go.dev/play/p/6C_eVWyTeAD
var seconds = 86900

h, _ := time.ParseDuration(fmt.Sprintf("%ds", seconds))
fmt.Println(h)
// 24h8m20
fmt.Println(h.Hours())
// 24.138888888888889
fmt.Println(h.String()
// 24h8m20s

Now you can do operations on this as a time object or cast it to an float using h.Hours() and do math operations.

Tags:

#go #time

I need to track contract time

Sun, 16 Oct 2022 00:00:00 +0000

I need to track contract time

In a given contract year I am entitled to work 1920 hours. I cannot go over but I can go under. Going under is money lost for me - my contract owner gets the total allotment of their share.

I’ve done the math and over the last two and I’ll be pulling up short by ~300 hours (probably 295). This equates to several tens of thousands of dollars in income. Income I’ve left on the table due to my own ignorance of my worked hours.

I’m lazy - literally the epitome of Just-In-Time everything. Learning, chores, presents for family any topic you can think of. So whilst I track every hour I do using an app on my phone I rarely do any tracking long term. Meaning only today (with 32 working days left on my current contract) am I figuring out how far behind I am.

1920 / 8
> 240 # days at 8 hrs per day to complete contract hours at zero

So what can I do about this? Build a tool to prompt me every day of course.

contract-timetracker (terrible placeholder name). It should:

report hours remaining on contract
report mean hours per day required to achieve zero balance
calculate number of possible working days left in contract - taking public holidays in to account
produce a end of month report with total hours worked and number of days (possibly with each day’s total)
(optional) write to my contract timesheet excel file with the datetimes and email to myself
run using OpenFaaS on a cronjob so I don’t need to manage the service

Considering how JIT I am this seems like a decent 80% solution for keeping me on track for my next contract which starts in November. I got a pay rise and want to capitalise on it and not leave money I’m entitled to on the table.

Tags:

#project #contract #openfaas

Contract time countdown and hours remaining

Tue, 11 Oct 2022 00:00:00 +0000

Contract time countdown and hours remaining

I am useless and tracking my remaining hours for my contract year. Typically, I do my timesheets at the end of the month which isn’t helpful for forecasting how many hours I need to do over the course of the month to keep pace with my contract. I would also like a way to see how many hours I could achieve in a month and/or how many I need to do per day to achieve a set number, such as 160.

Example 1: Remaining hours in contract

This would calculate how many hours I have done since contract start date until today and then output how many I have remaining

Example 2: Given a target output how I have remaining for the month

Subtract hours done from target each day, returning a mean for each working day remaining in the month.

I think this could be achieved via my current timesheet accounting app. I can export the database each day and upload it to the cloud. A cron could fire and do all the calculations. A daily email, or notification could be sent with the figures presented.

Extra bonus: apply the timesheet data into my official excell spreadsheet which I need to submit to management each month. This is a timesink I could easily automate.

Tags:

#accounting #project

pop-os shell over i3

Tue, 11 Oct 2022 00:00:00 +0000

pop-os shell over i3

I have been an i3 user for maybe two years now. It is a great experience, or I should say, the good parts make for a great experience. The bad parts make for a terrible experience (for me).

I’m not a ricer, I don’t care about gaps. I just want to switch between panes and windows easily. In i3, I put golang in window 6, webstorm in 5 and pycharm in 4. All my browsers go in 2 (I usually run two browsers at the same time - Vivaldi and Opera). Email in 10, VirtualBox in 9, and so on.

This works so much better than Gnome or whatever’s tabbing experience. Yes, I can use workspaces but its really not the same.

In saying all this, there are some things I don’t like:

learning all the esoteric short keys
it needs a lot of customisation
sometimes things don’t work well
resizing things can be a pain
I use a mouse 30-40% of the time so it needs to work

Whereas, I like Gnome’s “it just works” approach. Running some applications work much better in a Gnome environment, for instance, OBS.

Today I found pop-os shell. It seems like a great alternative. At work, I use Gnome because I need to work, not fiddle with i3 customizations - the juice ain’t worth the squeeze. So, I installed it pop-os/shell at work and instantly fell in love with it. It’s not perfect but it basically covers my needs - I don’t think I need all that i3 provides, shell is enough.

The only big thing that sucks and its not shell its stupid as Windows. At home I’m a linux native but at work I recently had to switch from a linux workstation to a beefy windows box where I use VMWare to run my fedora VM. The thing that fucking sucks is that windows hardcodes the Super/Win key shortcuts, such as WIN+L (and WIN+G for XBOX crap). This totally breaks shell’s vim bindings.

Update: After nearly a month of running pop-os shell, I really love it. Its super simple and meets my needs. I do miss the switching between workspaces using the i3 number system but other than that, everything else Just Works.

Tags:

#linux #i3 #tiling #windows

SQLite select between dates

Tue, 11 Oct 2022 00:00:00 +0000

SQLite select between dates

SQLite has spartan types and today I learnt how to select between a date range.

SELECT date1,date2,amount,ROUND((JULIANDAY(date2) - JULIANDAY(date1))*86400) 
AS diff 
FROM times 
WHERE date1 between '2022-09-01' AND '2022-09-30';

Outputs

# .mode column
# .headers on
date1 date2 amount diff
---------------- ---------------- ---------- ----------
2022-09-02 08:25 2022-09-02 17:11 1008.17 31560.0
2022-09-05 08:22 2022-09-05 17:05 1002.42 31380.0
2022-09-06 08:56 2022-09-06 16:30 870.17 27240.0
2022-09-07 08:34 2022-09-07 17:05 979.42 30660.0

Using JULIANDAY and multiplying by 86400 returns the difference in seconds between date1 and date2.

Last seven days

SELECT date1,date2,amount,ROUND((JULIANDAY(date2)-JULIANDAY(date1))*86400)
AS diff
FROM times
WHERE date1 > datetime('now', '-7 days');

Since start of the month

SELECT date1,date2,amount,ROUND((JULIANDAY(date2)-JULIANDAY(date1))*86400)
AS diff
FROM times
WHERE date1
BETWEEN datetime('now', 'start of month') AND datetime('now', 'localtime');`

Tags:

#sql

Hugo sort json

Sun, 09 Oct 2022 00:00:00 +0000

Hugo sort json

Sorting JSON in hugo is quite easy.

Take this JSON object

[ 
 {
 "name": "three",
 "date": "7/02/2022",
 },
 {
 "name": "one",
 "date": "5/01/2022",
 },
 {
 "name": "two",
 "date": "7/01/2022",
 }
]

To sort this by date in Hugo you can use sort. In the HTML page to sort by desc;

{{ $items := getJSON "sort.json" }}
{{ range $item := sort $items "date" "desc" }}
<p>{{ $item.name}}</p>
<p>{{ $item.date }}</p>
{{ end }}

Tags:

#hugo #json #web

Sept 2022 Retrospective

Fri, 07 Oct 2022 00:00:00 +0000

Summary

It’s been two months since I last wrote a retrospective. This is why.

Work

I’m really enjoying my day job.

The team

The work is tough at times but the team is excellent. None of us are rockstar engineer’s, but we all muck in, learn what we need and get it done. It’s a long way from my time in the special forces but for a development team it’s pretty darn good. I’m proud of them and what we’re achieving with all thats stacked against us. And, from my experience, it’s in battle or in times of hardship that you really become a team, and also become friends. Smooth sailing never made a skilled sailor.

The work

I’m not going to dox my employer. But the work we do makes a difference. You likely can’t see it, and if we do our job right, never will but it matters. At the micro, I do stuff that any developer does daily; code, debug, fix, review, and deploy. Except that in this job I have numerous business and environmental constraints that 99% of other engineers would never come across.

It makes work slow, error-prone and brittle at times. Local development is hard. Sometimes impossible - we’ll do it live is a pretty normal! This sounds like a whinge but it’s actually a binding force for us in our mission. Embracing the suck has made us invent ingenious ways to overcome some of these constraints and exposed me to so much more than other developers in my position.

Life

I only have so much time each day to:

Work my day job
Be present with the family
Work on after hours pursuits
Exercise
Chores

Something has to give but also energy has to be dispersed across all.

Mudmap

Mudmap still lives but I’ve given it less attention than it deserves for the reasons stated above. I’d love to do more, and be able to build the things users ask for. Maintaining it is an energy drain that isn’t refilled by my interest in working on it. Though at times, I have had bursts of inspiration because of its potential, and its that potential which prevents me from shutting it down.

Possibly the biggest gripe is the difficulty in getting everything to install correctly on each device. I’ve yet to discover a repeatable or diagnosable reason as to why it works for some but not others. I think this is a problem many similar bring-your-own-hardware platforms have to deal with. Each time I get a customer email stating they couldn’t install correctly my heart sinks a little. It’s also hard not to be your code and blame yourself (a distinct possibility) for the issues. Energy sinkhole.

Pursuit of joy

Like all developers, I love creating and trying out new toys and techniques. Building a new application is great fun and is a great way to procrastinate. It is also a great way to try new things and bring them in to your day job, or gain better understanding of how something in your day job works. Lots of my colleagues don’t do this and seem to learn everything on the job without any additional work. That ain’t me, never has been. In the army, I’d go home and read the manual so that things sunk in. I have to do that in tech too - I’m a hard worker not a smart guy. My school aptitude tests put me in the “labour hire” job prospects category not the “attend university” one. So, sometimes I get engrossed in learning something at the detriment of other things like Mudmap.

On the flip side, my after hours learning has earned me a reputation as the Kubernetes guy at work. I’m also the Go guy now too. I can now say I’ve a working familiarity with gRPC and NATS as well. Doing these things helped me make several interview rounds (and get offers) in the last three months for 20-40% pay rises without having to do a single LeetCode session or other Code riddle bullshit. That being said, I acknowledge my CompSci fundamentals are a limiting factor so I’m now reading up on those things to round out my knowledge.

Lots of words to say; I enjoy my craft and like to jump around at the detriment of long-term projects such as Mudmap.

What’s next

I love how well mtlynch and czue write their retro’s. I took most of my inspiration from Michael’s, but I think pigeonholing myself into something that works for him is a misstep.

For instance, Michael is doing a fantastic job of documenting his many projects but mostly his successful business, TinyPilot. For me though, I work 40 hours a week in a day job I can’t talk much about and approximately 10-20 hours outside of that broken into 1-2 hour chunks over nights and weekends. Setting myself ambitious goals was actually causing me some anxiety, especially if my month’s work took a big hit from external factors.

What I’m planning to do now is just be more retrospective about what I did rather than only talk about how well I did on x goal. At the end of the day, I write on here for me not an audience. So my writings should reflect that and help guide future me when looking back.

Does this mean I’ll forsake any thought of setting goals for each month? No but I’m not going to be dogmatic about how I layout the retrospective and instead let it flow. Some months I might only do 10 hours on Mudmap but do 30 hours on something that teaches me a hell of a lot about <insert tech>.

Goals

Okay, I’m setting one goal for next month.

Write Octobers retrospective. Time to get back on the retro horse.

Analytics

mudmap.io

danielms.site

September Retro

Fri, 07 Oct 2022 00:00:00 +0000

September Retro

It’s been two months since I last wrote a retrospective. This is why.

Work

I’m really enjoying my day job.

The team

The work is tough at times but the team is excellent. None of us are rockstar engineer’s but we all muck in, learn what we need and get it done. It’s a long way from my time in the special forces but for a development team it’s pretty darn good. I’m proud of them and what we’re achieving with all thats stacked against us. And, from my experience, its in battle or in times of hardship that you really become a team, and also become friends. Smooth sailing never made a skilled sailor.

The work

It makes work slow, error prone and brittle at times. Local development is hard. Sometimes impossible - we’ll do it live is a pretty normal! This sounds like a whinge but it’s actually a binding force for us in our mission. Embracing the suck has made us invent ingenious ways to overcome some of these constraints and exposed me to so much more than other developers in my position.

Life

I only have so much time each day to:

Work my day job
Be present with the family
Work on after hours pursuits
Exercise
Chores

Something has to give but also energy has to be dispersed across all.

Mudmap

Pursuit of joy

Lots of words to say; I enjoy my craft and like to jump around at the detriment of long-term projects such as Mudmap.

What’s next

I love how well mtlynch and czue write their retro’s. I took most of my inspiration from Michael’s but I think pigeon holing myself into something that works for him is a misstep.

Goals

Okay, I’m setting one goal for next month.

Write a Octobers retrospective. Time to get back on the retro horse.

Job tasks I do these days

Mon, 26 Sep 2022 00:00:00 +0000

Job tasks I do these days

A list of all the types of things I am doing in my day job, and in my own time.

CI/CD
- gitlab pipelines
- GitLab API access to cross system boundaries with runners in different tenancies (no access outside certain operations)
- Pipeline maintenance
Kubernetes
- Building applications for kubernetes
  - Using non-greedy operations because scaling horizontally is better
- Administering production and staging clusters
- Administering user access and control for the team
- Fixing bad deployments and broken helm charts
- Attempting to transition the environment over to ArgoCD (we have little oversight when things drift)
- Stood up and introduced k3s to the team
  - we now run k3s locally with verified fakes for our third party API’s
Bash Scripting
- I use Bash for so many things and feel like I’m getting decent at it
Python
- I code review major deployments
- I write python most days using Rabbit and websocket backends
Golang
- Fairly competent having written a large code base entirely in it
- Reach for it instead of python for CLI or automations which bash isn’t suitable for
- Experienced somewhat in writing gRPC code for Go

Tags:

#checkin #career #skills

Ballbag quote of the day

Wed, 21 Sep 2022 00:00:00 +0000

Ballbag quote of the day

A question posted on a Slack I am in:

Question - on behalf of one of my students - does working in IT for a defence company (think one of the defence contractors out Williamtown) look “bad” on your resume in the long term considering defence is a “touchy” subject ? I’ve never heard of such a thing, but have had it suggested to me. What are the thoughts of the group ?

This response absolutely got me raging.

Interesting conversation. There’s no one answer and it depends on the person and the specifics of the defense role. I’d be curious about what they did and why they left and ethics would be part of that. It’s not a blanket no - for sure. People can change. Some ex-bankers even turn out ok. BUT I am surprised at the number of people here that seem to think what a person can offer is always far more important than the type of human they are. And often (not exclusively) that’s reflected in choices they’ve made including who they are willing to work for. (edited) Short answer - it should probably be a flag to learn more about the individual and their ethics

What an absolute dickhead way of thinking about someones choice of employer. This kind of idiot probably thinks working for Google or Facebook is ethical. I’m literally at a loss that someone could think this way, honestly I’d hate to work with this kind of moron. He’d hate the fact that I’ve fought in two wars for my country and also worked in defence industry. I can only imagine what he thinks I am based on what this opinion.

Honestly, I’m fuming and this kind of thinking makes you lose faith in people as a collective.

Tags:

#rant

WGD Fri 2022-09-23

Mon, 19 Sep 2022 00:00:00 +0000

WGD Fri 2022-09-23

Deployed Jellyfin in my house.
- Can’t believe how good it is and why I didn’t do this sooner
Refactoring Mudmap sql to use SQLc
- It’s a lot of work but so far is totaly worth it. Building new queries is so easy
- The only thing I don’t like is how it handles enums in Postgres
Spent all week at work refactoring a series of Gitlab pipelines
- Nested includes are not a lot of fun.
Started learning a bit more about gRPC and Protobuf’s for a project
- It’s a different way of thinking when you’ve mostly done REST using JSON.

Tags:

#wgd

docker buildkit breaks 'docker build -o'

Tue, 13 Sep 2022 00:00:00 +0000

docker buildkit breaks ‘docker build -o’

I wasted a day only to discover that enabling DOCKER_BUILDKIT=1 will break the docker build -o <bla.tar.gz>.... Instead of creating a tar file, it instead creates a directory! The only reason I figured this out was by running it manually and seeing that it was a directory called bla.tar.gz. Debugging this when it was used in our runners was a nightmare.

I turned off that feature entirely out of spite, rather than fix it. Besides, our pipelines are doing well without buildkit.

Tags:

#rant #docker #spite

programmer quote of the day

Sun, 04 Sep 2022 00:00:00 +0000

programmer quote of the day

You know they say economists know the price of everything and the value of nothing? Well programmers know the benefits of everything and the trade-offs of nothing. - Rich Hickey

This has to be the greatest programming quote I’ve ever seen.

self-hosted gitlab-runner queue slow

Sun, 04 Sep 2022 00:00:00 +0000

self-hosted gitlab-runner queue slow

Gitlab self-hosted runners seem to take a long time to pick up jobs from the queue. In many cases its was over 3 minutes before the pipeline would even start and then have lengthy pauses between stages.

Changing two lines in the /etc/gitlab-runner/config.toml file seems to have fixed this issue.

concurrent = 5
check_interval = 5

link

Also, as my runner is a docker-runner inside a container I could not edit this file. The Dockerfile is locked down so I had to docker cp the file out of the container, edit it, and then docker cp it back in.

docker cp <Container-ID>:/etc/gitlab-runner/config.toml .
vim config.toml
docker cp config.toml <Container-ID>:/etc/gitlab-runner/config.toml

Tags:

#cicd #gitlab #runners #docker

kubernetes ingress and metallb

Thu, 01 Sep 2022 00:00:00 +0000

kubernetes ingress and metallb

For my homelab, I’m teaching myself how to setup a bare metal cluster using k3s, nginx-ingress and metallb. It’s show soo many holes in my knowledge and understanding of k8s. When you work on established systems such as OpenShift at your day job, you take for granted all the things it does for you.

I misunderstand Ingress vs Services

At work, we use Ingress and NodePorts to allow traffic to our applications and I thought I understood it. But, now that I’m throwing Metallb into the mix, I clearly don’t.

My understanding at the minute

Create a metallb static IP for my nginx-ingress
Define a Service for my app
Use the container port on the Ingress and set a host
Create a DNS host override in pfSense
Use the metallb static IP of the nginx-ingress service for all the hostnames
‘argocd.home.labandtekton.home.labmap to192.168.20.199` and nginx will resolve it

So far that works. It took me a long time to figure that out and this is just for the LAN. Exposing applications later will be another thing I likely misunderstand.

Tags:

#kubernetes #homelab #til

get private ssl cert

Mon, 29 Aug 2022 00:00:00 +0000

get private ssl cert

This is how I get private SSL certs which is really useful when using private gitlab and argocd

openssl s_client -showcerts \
-servername gitlab.homelab.com \
-connect gitlab.homelab.com:443 \
</dev/null 2>/dev/null | sed -n -e \
'/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' \
> /tmp/sslkey.pem

Tags:

 #openssl #ssl #argocd

microservice learning task

Fri, 26 Aug 2022 00:00:00 +0000

microservice learning task

I am building out my Proxmox server and local kubernetes cluster. I don’t have a huge workload to put on either, nor a sophisticated arrangement. So, I figured this is a good opportunity to replicate the type of work I do in my day job. Unfortunately, despite working on a distributed microservice I don’t get to work on all parts so I have some knowledge gaps - this should remediate that.

Here’s what I’m thinking:

Weather God:

a simple weather service

It should:

weather now for location x
weather historical for location x
snow falls AU
barometric
weather chart

This is a very simple application but could contain many small microservices, and/or functions.

Spitballing;

temperature
barometric
chart generation
database
event bus
user interface

All deployed on to my local cluster

Tags:

#project #kubernetes #microservices #proxmox

TIL: bash flags

Fri, 26 Aug 2022 00:00:00 +0000

TIL: bash flags

TIL how to create flags/ optional arguments in bash scripts. Previously I just used “${1} ${2}” etc.


error()
{
 echo "$arg0: $*" >&2
 exit 1
}

flags()
{
 while test $# -gt 0
 do
 case "$1" in
 (-s|--source)
 shift
 [ $# = 0 ] && error "No source directory specified"
 export SOURCE="$1"
 shift;;
 (-d|--destination)
 shift
 [ $# = 0 ] && error "No destination specified"
 export DESTINATION="$1"
 shift;;
 (-c|--credentials)
 shift
 [ $# = 0 ] && error "No credentials specified"
 export CREDENTIALS="$1"
 shift;;
 (-b|--bandwidth)
 shift
 [ $# = 0 ] && error "No bandwidth specified"
 export BANDWIDTH="$1"
 shift;;
 (-t|--timeout)
 shift
 [ $# = 0 ] && error "No timeout specified"
 export TIMEOUT="$1"
 shift;;
 (-p|--port)
 shift
 [ $# = 0 ] && error "No port specified"
 export PORT="$1"
 shift;;
 (-l|--compression-level)
 shift
 [ $# = 0 ] && error "No compression level specified"
 export COMPRESS_LEVEL="$1"
 shift;;
 (-h|--help)
 help;;
# (-V|--version)
# version_info;;
 (*) usage;;
 esac
 done
}

flags "$@"

Tags:

#bash #til

k3s nginx ingress over traefik

Tue, 23 Aug 2022 00:00:00 +0000

k3s nginx ingress over traefik

I have elected to use nginx-ingress over the k3s standard of Traefik. This was mainly driven because OpenFaaS doesn’t yet support it for custom domain names. I also do not really use Traefik features so I am unsure what the big trade-offs are yet.

Tags:

#kubernetes #traefik #openfaas

kubernetes tips

Mon, 22 Aug 2022 00:00:00 +0000

kubernetes tips

Random helpers

Create Namespace if not Exists

kubectl create namespace test --dry-run -o yaml | kubectl apply -f -

Wait for deployment to rollout

This will wait until the deployment has completed and very useful in bash scripting.

kubectl rollout status deployment argocd-repo-server -n argocd

Tags:

#kubernetes #bash #kubectl

Getting into the back country

Tue, 16 Aug 2022 00:00:00 +0000

Getting into the back country

Over the last weekend I undertook a snow survival course in the back country near Threadbo. What a great weekend, and an awakening. I think I should do some more stuff down there and maybe even get some skiing in.

In the meantime here are the things I need to buy, and learn.

Buy

z-fold mat
better hard shell (pants and jacket)
synthetic jacket pref with water repellent outer
75L+ pack

Learn

weather systems
layering
climbing knots
climbing anchors

k3s custom resolv.conf

Wed, 03 Aug 2022 00:00:00 +0000

k3s custom resolv.conf

Setting up a local k3s just taught me a valuable lesson in networking.

It’s always DNS

And my own networking knowledge gaps. So I wrongly assumed that k3s was using my /etc/resolv.conf - wrong. It uses core-dns defaults (AFAIK) which means my pfSense DNS server which has host overrides for my local cluster was not being respected.

This resulted in my CI (drone and woodpecker, in testing) not being able to communicate with my git server. This makes sense, it cannot resolve the address.

To rectify, I used this process. It is probably a bandaid when I should be using something else. But, it works for now.

echo "nameserver 192.168.1.1" | sudo tee /etc/k3s-resolv.conf

echo 'kubelet-arg:' | sudo tee -a /etc/rancher/k3s/config.yaml
echo '- "resolv-conf=/etc/k3s-resolv.conf"' | sudo tee -a /etc/rancher/k3s/config.yaml

sudo systemctl restart k3s

Then kill the all the coredns pods so that they reload with the new resolv.conf.

kubectl get pod -n kube-system -l k8s-app=kube-dns --no-headers | awk '{print $1}' | xargs -I{} kubectl delete pod -n kube-system {}

Tags:

#kubernetes #coredns #dns

thisisunsafe Chrome ssl bypass

Wed, 03 Aug 2022 00:00:00 +0000

thisisunsafe Chrome ssl bypass

Note to self.

I wasted over an hour on this shit.

Firefox lets me bypass self-signed certificates by clicking on a button.

Chrome/Opera do not. Instead I have to type thisisunsage like a fucking dickhead to get past self-signed certs.

Honestly, what a joke.

Tags:

#rant #chromesucks

Razors worth remembering

Sun, 31 Jul 2022 00:00:00 +0000

Razors worth remembering

Here is a few really useful razors worth remembering.

Luck

When choosing between two paths, choose the path with the largest luck surface area

Arena

When faced with two options, choose the one that puts you in the arena.

Or, more simply, put yourself out there and be prepared to fail in public. And never listen to anyone who is on the side lines

Rooms

If you have the choice between two rooms, choose the room in which you are the dumbest.

This means, stretch yourself by sacrificing ego in the short term and reap long term growth.

Tags:

#life #advice

WGD Fri 2022-07-29

Sun, 31 Jul 2022 00:00:00 +0000

WGD Fri 2022-07-29

Mudmap

Sent out a number of emails to clients who have been having difficulties getting devices onboarded to get feedback
Updated some of the documentation
Ruled out an error I was seeing in testing as the cause of onboarding issues

Misc

Applied to renew my contract, which is actually a new contract so I will have to interview for it.
Helped out a mate build a wails application using Svelte

Continuing to get more and more deep into kubernetes at work. I totally understand why the CNCF landscape has a market cap of 10T USD. The thing is a massive pain in the arse but insanely useful.

python Protocols are Go interfaces

Fri, 22 Jul 2022 00:00:00 +0000

python Protocols are Go interfaces

This week I successfully used python’s Protocol to fake a dependency on Hashicorp’s Vault.

Background:

we need Vault access to save some data
init and sidecar must be running for the pod to be created
in development we don’t need access to the real vault
in development access to the real vault crosses network boundaries via indirection

Instead, we have decided to use a verified fake and build an interface which we can use to swap out the real and fake Vault when required.

After a suggestion from a colleague about Protocol, and some light reading, this process was super simple. It works almost exactly the same as a Go interface whereby you define the interface and other classes implement it.

Example:


class SecretStore(Protocol):
 def save(self):
 pass

class VaultStore:
 def save(self):
 print("vault store save")

class FailStore:
 def no_save(self):
 print("not implemented")

class Saver:
 def do_save(self, obj: SecretStore):
 return obj.save()

def main():
 saver = Saver()
 saver.do_save(VaultStore())
 saver.do_save(FailStore()) # <-- AttributeError: BrokenStore object has no attribute 'save'

if __name__ == "__main__":
 main()

Tags:

#python #interfaces #programming

wails as a standalone app

Fri, 22 Jul 2022 00:00:00 +0000

wails as a standalone app

The other day a friend suggested we build a tool but he wanted it to be a standalone GUI application. It needed to be a single binary and usable across Windows, MacOS and Linux.

After some research, including C++, Java and Kotlin, we stumbled upon wails. After about an hours of playing around it was obvious it met all those requirements, and it was familiar to web development.

My demo is bare bones example of hitting a 3rd party API with it. It uses svelte.

On a side note, this is first and only svelte app I’ve ever developed and it was very easy to use with great documentation. I see big things for svelte in the future.

Tags:

#wails #svelte #gui

WGD Fri 2022-07-22

Fri, 22 Jul 2022 00:00:00 +0000

WGD Fri 2022-07-22

Mudmap

Released my Multiple Accounts feature that I’ve been working on for over two months! It is now in production but the documentation and official announcement emails are yet to go out. I actually deployed it to production sooner than I initially planned. Nonetheless, all is well so far and can’t wait to get some feedback.

I think I’ve also enabled the right level of logging to identify why only some customers are having issues installing the API on their device. In my testing I noticed a CSRF error which caused the install to fail. No workaround yet but keen to see if its the same thing others are experiencing.

Misc

Played around with Wails and got a little demo done up using svelte. So far, its basically the same process as building a typical SPA except that its self-contained and no network latency between requests. That’s actually doing it a massive injustice, it is a great tool to build standalone applications especially if you have web development experience.

The last few weeks have been pretty hectic both in office hours, at home and then the ongoing Mudmap dev. Keeping my WGD streak fell away as I prioritised other things. Feeling back on track now and kind of missed cataloging what I’ve been up to.

June 2022 Retrospective

Wed, 06 Jul 2022 00:00:00 +0000

Summary

Moved house and still haven’t quite finished my one goal from last month.

Goal Performance

A review of last months three goals. See May’s Retrospective.

Deploy the multi-account feature

Appraisal: Some hiccups slowed development
Rating: B

All of this month’s Mudmap development was dedicated to multi-account feature. Unfortunately, that work is still ongoing but at least right now its down to fleshing out the user interface. So far, users can invite members to join their organisation, join other organisations, remove members from their own, update org details, and manage devices within their organisation based on simple permissions.

At the beginning of the month I spent a lot of time building on top of the Auth0 Authorization extension. This worked well in theory as it matched much of my vision for what I wanted to achieve. Unfortunately, it did not fit well into the application of that vision; simple actions require many API calls. For instance, I could find no easy method to get the _id of a group without iterating over every group and searching for its mutable name. To do most operations against the group you need this _id, meaning lots of API calls and then loops to find the _id. This means latency but also requires a fair bit of error handling. In all, it felt bad and without storing that data in the database was slow.

Luckily, I went back to the much simpler and easier to use Auth0 Management API. This in hindsight does everything I wanted the Auth0 authorization extension to do but with 1/10th the complexity. Building, integrating, and testing the authorization extension took me about a week of evenings but replacing it with the management API took about three hours, and it removed a significant amount of code.

In hindsight, the toil and wasted effort building out the organisations using Auth0’s authorization extension wasn’t wasted. I think it actually helped to design a much simpler and easier to maintain product. The effort helped shape the how and what of its current form. Still, it definitely slowed the deployment of the feature.

I also started on a side project to service my own needs - storeman - which cut into the feature development.

Recommendations

If you can afford it, get removalists to lift and shift your house. My hubris and tightwad-ism meant I moved my entire house on my own minus a mate who helped lift the real heavy stuff. The entire process absolutely cooked my lower back, and I’m only now starting to recover.
Peaky Blinders, the latest season. I watch about 30-60 minutes of television a day so I’m picky about what to tune into on the idiot box. This season of Peaky Blinders was the best by far, really hoping they make another season.

Wrap up

This is the second month when I’ve bombed on my targets. Still, I’m happy to be continually making progress on my objectives. It might be costing me some business in the short term, as user-facing features aren’t being developed quick enough. There isn’t much more time I can steal from my days to get more done, but I think I could be smarter in my planning - hindsight, of course.

What can I do better?

Plan more, code less - meaning, write more documentation of what I want to achieve before I try to achieve it.

What have I done well?

Accomplished a lot in terms of functionality especially when I lost a number of days due to the move.

Next month’s goals

Deploy the organisations feature
Allow users to back up their device from within Mudmap

Analytics

mudmap.io

check-redirects.com

danielms.site

WGD Mon 2022-07-01

Sun, 03 Jul 2022 00:00:00 +0000

WGD Mon 2022-07-01

[Mudmap]

After a month of working with the Auth0 Authorization extension, I ripped it out Instead, I’ve replaced it with the much simpler Management API and have achieved the same thing.

Misc

Moved house
Started seeing my EP twice a week

A light week in terms of work but very busy packing, moving and unpacking for the move. Moving is such a painful experience but on the whole we’re really happy we did it. This house is much nicer.

Ansible check exists with register

Mon, 20 Jun 2022 00:00:00 +0000

Ansible check exists with register

In building my Ubuntu disaster recovery ansible script, I discovered the register condition.

To prevent needless tasks from running, for instance, shell commands which would be redundant. It is possible to check a condition exists and only execute the task based on the response.

- name: Download GH cli install script
 stat:
 path: gh
 register: gh_exists

- name: Install gh cli
 become: true
 when: gh_exists.stat.exists == False
 ansible.builtin.get_url:
 url: https://raw.githubusercontent.com/danielmichaels/dot/master/installers/install-gh
 dest: /tmp/gh.sh
 tags:
 - install
 - productivity

- name: Install GH Cli
 become: true
 when: gh_exists.stat.exists == False
 ansible.builtin.shell:
 cmd: cat /tmp/gh.sh | sh -s -- -y
 tags:
 - install
 - productivity

The above code block installs the gh CLI tool. It executes a script from my github dot repo only if the gh binary does not exist in the PATH.

I also achieved the same thing for installing Go. Using this method I didn’t need to install a ansible-galaxy third party role.

# vars:
# go_version: "1.18.3"
- name: Check go exists and version
 stat:
 path: /usr/local/go/bin/go
 register: go_exists

- name: Remove existing /usr/local/go
 when: go_exists.stat.exists == False
 become: true
 ansible.builtin.file:
 path: /usr/local/go
 state: absent

- name: Install Golang
 when: go_exists.stat.exists == False
 become: true
 ansible.builtin.unarchive:
 src: "https://go.dev/dl/go{{ go_version }}.linux-amd64.tar.gz"
 dest: /usr/local
 remote_src: yes

It might not be elegant but it works well for me.

Tags:

#ansible #productivity

Kubernetes local development musings: work edition

Mon, 20 Jun 2022 00:00:00 +0000

Kubernetes local development musings: work edition

At work we use K8s for development, staging and production. We have to, our external dependencies are inaccessible without it.

This makes local development painful. We need to shift left and move more of the stack closer to our local systems.

Possible ideas:

minikube as the local cluster
k3s as a local cluster
kind/k3d as a dockerised local cluster

TBC

Performance review notes Q1 2022

Sat, 18 Jun 2022 00:00:00 +0000

Performance review notes Q1 2022

Keeping track of my last performance review.

Skill Levels:

Level: 4

Comments:

Dan has exceeded expectations. He is a high performing member whose contribution has been critical to the delivery of XXXX and XXX’s ability to meet its strategic objectives. He has, without doubt, developed and implemented complex programs that have achieved an excellently engineered result. He built the persistence functionality for XXXX and drove this essential requirement through to production. Additionally, he took over the development of another highly specialised software component for the project at short notice. He not only delivered the highly specialised features and integrations, but improved upon the original design.

Dan has consistently contributed to XXX’s software engineering processes and mentored members of XXX’s XXXX team in integrating and streamlining code to enable XXXX. He has taken on work to professionalise the XXX XXX structure and permissions. He has enabled a self-service process which automates the creation and launching of resources without the need for manual intervention by XXX staff. He has been responsible for running user acceptance testing and XXXX familiarisation sessions for stakeholders. Dan is a very resonpsive and valued member of XXX and an asset to the section.

Tags:

#performance #appraisal #brag

WGD Fri 2022-06-17

Sat, 18 Jun 2022 00:00:00 +0000

WGD Fri 2022-06-17

Storeman

Finished the basic endpoints for Containers and Items, meaning users can add a container and then put items in it. QR code and image support WIP
Started on the QR code support branch

Mudmap

Admin users can now invite users by email to join their Organisation
Started work on the invite token storage and endpoints on the backend
Fielded some questions about pricing for non-for-profit foundations - something I had not considered.

Misc

Preparations for moving house continue
Signed up for a three day adventure training course in the alpine regions where we’ll learn some basic snow camping and survival skills.

Mudmap org and new user registration

Thu, 16 Jun 2022 00:00:00 +0000

Mudmap org and new user registration

The rough registration process for adding a user to an Organisation.

Admin account enters email into Org settings tab of user they wish to invite
Email is sent to them with a short lived key
User opens email and either signs up or logs into Mudmap via Auth0
User navigates to Settings -> Organisation and clicks Join Org
User enters key from email
If key is expired, alert user and admin
Otherwise, add user to Org and remove them from existing

Scope of work (no particular order)

Invite form
Invite Email (sendgrid template & Go code)
Registration flow documentation with video
Token/Key table for tracking purposes, auth’ing Invites on backend
Token auth endpoint for checking validity
Endpoint ^ add user to Org in DB, and in Auth0 Groups
Email alerting Admins user has joined
Email alerting Admins user attempted to join but token was out of date?
Or, just allow user to get new token?
Email welcoming User to Org

Tags:

#mudmap #architecture

Go database and sql tips

Tue, 14 Jun 2022 00:00:00 +0000

Go database and sql tips

Note to self.

func (s store) ItemUpdate(id int, name, description string, image []byte) error {
	stmt := `
	UPDATE items
	SET
		name = $1,
		description = $2,
		image = $3,
		updated_at = datetime('now')
	WHERE
		id = $4
	RETURNING id`
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	var c Item
	args := []interface{}{name, description, image, id}
	err := s.DB.QueryRowContext(ctx, stmt, args...).Scan(&c.ID)
	if err != nil {
		switch {
		case errors.Is(err, sql.ErrNoRows):
			return errors.New("item does not exist")
		default:
			return err
		}
	}
	return nil
}

This code works but this really tripped me up…

The stmt block uses $1 numbering, instead of the usual ?. I stupidly thought the $n numbering meant something when executing the args... inside the QueryRowContext method.

It does not. It is simply there for you, to help count the number of variable statements inside the query, AFAIK. This cost me an hour or so because I set the id = $1 so as to map the ItemUpdate parameters with the query.

I hope I remember this in the future! I don’t like ORM’s but this definitely would not have tripped me up.

Tags:

#go #database #failure

Go template iterators

Fri, 10 Jun 2022 00:00:00 +0000

Go template iterators

How to use Go templates range and $.

range

This loops over the length of a struct setting the dot value to element being looped over.

type Example struct {
 Id string
 Name string
}

{{ range Example }}
ID: {{.Id}} & Name: {{.Name}}
{{end}}

If you need to access a value via the . notation and that value is outside the current iteration, you need to use $..

// example 
{{.Container.ID}}
// range over an Item struct to build a URL
{{range .Items}}
// the $ is required so that the template can inspect global state
// not just the current iteration. Item does not have a .Container.ID 
// field and will fail without $
<a href="/containers/{{$.Container.ID}}/items/{{.ID}}">url</a>
{{end}}

Tags:

#go #templates

WGD Fri 2022-06-10

Wed, 08 Jun 2022 00:00:00 +0000

WGD Fri 2022-06-10

[Mudmap]

Work continues with the multi-account setup.

UI changes to the Settings page which is now broken into smaller subsections:
- User
- Organisation
- Billing
- Membership

Had some user reported errors which I suspect are being caused by sshguard being triggered. sshguard, as it sounds, is an application which blacklists SSH brute forcing. When users attempt to set their initial password wrong too many times, sshguard might be blacklisting Mudmap’s server IP addresses. This can lead to unexplained errors, things work then suddenly don’t but after a period time it resolves. I’ve added instructions to the the documentation with a fix, which entails whitelisting Mudmap’s IP’s from sshguard.

Also, I realise that I am having issues with devices more often than I like but have no metrics on potential issues. I have started investigating a way to log device details so I can determine if issues are more common for certain setup’s, say VM versus physical hardware.

Storeman

Started on a project for cataloguing my many storage boxes and their contents using QR codes.

Its a pretty simple premise, instead of writing or using your memory, you’ll put a QR code on a storage container. The QR links to a page with a list of its contents including images for future reference. I’m writing it in Go using basic templates and a little bit of Alpine.js.

We’ve moved across three states and four homes in the last four years. Moving sucks but not knowing what’s in the many boxes we have laying around the garage sucks even more.

Scaffolded basic structure
Created database layer and migrations
Views for home and viewing containers done at a basic level
Templating and form helpers integrated into the Server struct

Authentication and session management is up next

Misc

Landed a new lease, moving in at the end of the month.
Bought a nice second hand car to replace our old one - makes getting to work a lot easier
Deployed my own PicoShare instance at share.danielms.site using fly.
Published May’s retro

Breadcrumbs for Go templates

Tue, 07 Jun 2022 00:00:00 +0000

Breadcrumbs for Go templates

How I am dealing with breadcrumbs for Go templates.

Create a BreadCrumb struct
Attach the BreadCrumb to the template, in my case I use a TemplateData struct in every template.
Pass the BreadCrumb to the view and then reference it within the template.

type BreadCrumb struct {
 Name string
 Href string
}
type TemplateData struct {
 BreadCrumbs []BreadCrumb
 // ... trunc
}

func (s *Server) handleHome() http.HandlerFunc {
 return func(w http.ResponseWriter, r *http.Request) {
 // trunc
 crumbs := []BreadCrumb{
 {Name: "Page One", Href: "/one"},
 {Name: "Page Two", Href: "/two"},
 }
 s.render(w, https.StatusOK, "home.tmpl", TemplateData{BreadCrumbs: crumbs})

Then reference that in templates with

{{ if .BreadCrumbs }}
 {{range .BreadCrumbs }}
 <li>
 <div class="flex items-center">
 <svg class="flex-shrink-0 h-5 w-5 text-gray-400"
 xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20"
 fill="currentColor" aria-hidden="true">
 <path fill-rule="evenodd"
 d="M7.293 14.707a1 1 0 010-1.414L10.586 10 7.293 6.707a1 1 0 011.414-1.414l4 4a1 1 0 010 1.414l-4 4a1 1 0 01-1.414 0z"
 clip-rule="evenodd"/>
 </svg>
 <a href="{{.Href}}" class="ml-4 text-sm font-medium text-gray-500 hover:text-gray-700">{{.Name}}</a>
 </div>
 </li>
 {{end}}
{{end}}

Tags:

#go #templates

fly.io CNAME setup

Tue, 07 Jun 2022 00:00:00 +0000

fly.io CNAME setup

I started using fly.io today and so far it is pretty awesome.

tl;dr share.danielms.site

I setup everything without much issue but coming from Render, setting up the CNAME was not as easy (in hindsight, its super simple).

After deploying my PicoShare app using the fantastic flyctl CLI, I could easily access it from its fly domain but I wanted to put it on my personal website as a subdomain.

Netlify is my DNS provider, so I setup a CNAME over there but it would not resolve - OpenSSL error. Turns out you need to manually generate the certificate for the subdomain.

To do that:

fly certs create share.danielms.site

> The certificate for share.danielms.site has not been issued yet.
> 
> Hostname = share.danielms.site
> 
> DNS Provider = nsone
> 
> Certificate Authority = Let's Encrypt
> 
> Issued =
> 
> Added to App = 35 seconds ago
> 
> Source = fly
> 
> Your certificate for share.danielms.site is being issued. Status is Awaiting certificates. Make sure to create another certificate for www.share.danielms.site when the current certificate is issued.

Check the status with fly certs show share.danielms.site. It only takes a few seconds to deploy anyway but this will show if there are any errors.

Viola share.danielms.site

Tags:

#fly #dns

May 2022 Retrospective

Sat, 04 Jun 2022 00:00:00 +0000

Summary

Goal Performance

A review of last months three goals. See April’s Retrospective.

Deploy pfSense+ capable Mudmap

Appraisal: Did not roll this out.
Rating: D

All of this month was dedicated to multi-account feature. This is still a great feature to roll out by I decided early on in the month that it would come after.

There are some compatibility issues remaining with the API’s support for pfSense+, namely with ARM devices. This has only become apparent in the last few days, and is not something I can test either. So, in hindsight waiting this one has saved Mudmap from some potentially unhappy customers. Hoping a fix is released for this in due course. When this does release it will have to come with that caveat.

Get started on multi-account support (a big customer request)

Appraisal: Still ongoing but should be finished this month.
Rating: C

It is not finished yet, but at least it is work I am excited to do and release for my customers. I’ve given this a high priority as numerous people have requested it. They have staff who they want to manage devices but its currently not possible without sharing a single account. The sub-accounts must also have some permission settings, like payment details for instance.

So, I’ve settled on the following structure;

Organisation
 |
 |
 ---------
 | |
Users Devices

Each user has a permission applied which will grant them access to certain views and API operations. Permissions haven’t been started yet.

I’ve had to migrate Users to Organisations along with their subscriptions and Devices.

Considering the changes I’m probably going to do an outage window over a weekend, rather than roll it out during a weekday and potentially catch someone in a weird state. Not ready for that yet but these are thing considerations I’ve been thinking about along with writing the code.

I am still working on the user interface, which as always, is the most tedious and slow part for me. The other hard part, is managing how users add new members to their Organisation. To be a member they have to sign up via Auth0 - to get a valid id - and then they can be added to the Org. So far, I’ve settled on admin users being able to invite members by email. Then they sign up and enter a code to be assigned to the Org. It feels a little contrived and suboptimal but works for now.

I feel like I accomplished a lot towards achieving this goal.

Recommendations

If you want an excellent security tool that scans and identifies everything on your network, check out Rumble. It is a company that was started by H.D Moore who created the Metasploit project. I ran it to see what is on my network but also to see how it identifies my many pfSense VM’s. It found all of my assets with very detailed info within 3 minutes across two /24 networks. Definitely worth a look, and you can run it from a raspberry Pi.

Wrap up

A great month from a productivity standpoint. I’ve managed to work out a decent balance for work and out-of-work work, in addition to my life. Much of last year I was burning the candle from both ends; late night and early hours. Honestly, I got more done, but it wasn’t as good as the work I’m doing now. I’ve settled into a sleep in until 6:30-7 schedule, and I feel much more rested with more clarity of thought. In the past, I was up at 5, and it took me 30 minutes just to wake up!

We also got out for an overnight camping trip before it starts to get cold. Camped right next to a river watching the fire is a luxury I don’t take for granted. What I take for granted is how quickly time (i.e. our life) flies by between such adventures. Keeping grounded and getting away from the house has been another good move over the last couple of months - if I’m at home, I can’t help but be plugged in.

What can I do better?

I need to get in front of issues sooner, and have those hard conversations with people more often. At times I’m too passive or indifferent.

What have I done well?

Took several days to design and proof-of-concept the multi-account architecture and it made the development so much faster.

Next month’s goals

Deploy the multi-account feature

Analytics

mudmap.io

check-redirects.com

danielms.site

WGD Fri 2022-06-03

Fri, 03 Jun 2022 00:00:00 +0000

WGD Fri 2022-06-03

[Mudmap]

Refactored codebase to make better use of the internal package. 1
Started work on the frontend; Organisations and Users are now displayed in the settings pages. These pages are restricted according to the user privileges.
Added gzip supported middleware. I haven’t profiled it but the content size is significantly reduced.

Misc

Watched Top Gun 2. It’s an entertaining movie with some cool action shots. Definitely worth watching if you enjoyed the first one.

Saw a really cool idea whilst at bunnings over the weekend. Side note, I fixed my tallboys draw runners like a real man. It was a QR code system for notating what’s in storage containers. You slap a QR stick on it, scan it and add its contents to the web page.

As a developer, I immediately thought I could write that. Which, I actually might but as a self hosted open source project. You could deploy it to heroku or fly and never pay a cent. How often and how fast does something like this need to be, especially when you are the only user.

Also, I got to start on a Go project at work. It’s replacing a often used compliance tool which scans the contents of packages written in Java. The only caveat, it’s a side project and can’t get in the way of my day to day work. I’ll take it though as I get to work on an interesting problem in Go and get paid handsomely for it.

Tags:

#wgd #mudmap

Go features repo for future reference

Mon, 30 May 2022 00:00:00 +0000

Go features repo for future reference

I need to write a simple web app that exposes some of the more useful Go features such as Mutexes, gorotinues, select and NewRequestWithContext.

The rough outline:

webserver
each endpoint with a single feature

This way I, or anyone else can pull down the repo and run the server. Using curl they can hit each endpoint to see how it works whilst reading over the code. Each endpoint should have good logging to showcase what is happening.

Tags:

#golang #development

WGD Sat 2022-05-28

Sat, 28 May 2022 00:00:00 +0000

WGD Sat 2022-05-28

Mudmap

Continuing the Mudmap Organisations feature
Migrating users to organisations is done
Users in the same organisation now share devices and can create, read, update, and delete devices.
Payment webhooks is still being ported over but is about half way done
Finished the Auth0 Management API client, allowing Mudmap to add to app_metadata fields. This allows Mudmap to store some data in the JWT which is then used in the request context.
Working race conditions where the Auth0 data is out of sync with the database, leaving mixed state. This only happens on state change, i.e. a user joins a new organisation.
Investigated why my logs have disappeared from New Relic. Nothing has changed on my end, so I suspect its Render not forwarding the application logs correctly. They appear to be coming through again - again, nothing changed on my end.
Still toying with the idea of ripping out Postgres for Litestream and have been researching and reading over the many HN threads about it lately. It works for tailscale, so it’ll probably work for this app.

Misc

Sold one of our cars this week, leaving us with only one until we can find a new one.
Getting kicked out of our rental because the owners are selling so we’re scrambling to find a new house within the next 6 weeks. Canberra rent has exploded so we’re looking at an extra ~100 per week for a comparable house.

Ansible localhost setup

Wed, 25 May 2022 00:00:00 +0000

Ansible localhost setup

As I’ve now committed myself to Ubuntu, setting up an Ansible playbook for my personal machine is probably a good idea. I already have several several playbooks for my cloud VM’s.

Things which will need automating:

Dotfiles
ds, zet
github repo’s (mudmap, zet, ds, dot)
oh-my-zsh
starship
i3 (i3status-rust) & xandr
opera, vivaldi
pycharm, golang, webstorm
curlie, exa, air
Go (latest)
Docker (podman, too)

Apt packages:

psql
vim
zsh
curl
python3
npm, yarn, node

Tags:

#ansible #development

zerolog Objects

Wed, 25 May 2022 00:00:00 +0000

zerolog Objects

zerolog is so good. Today I learnt that it can use marshal structs into an object for logging. All you need to do is add the MarshalZerologObject method to the struct. Then it will be accessible on the logger via the .Object method.

Example.

// user.go
type User struct {
 Name string `json:"name"`
 Email string `json:"email"`
}

func (u User) MarshalZerologObject(e *zerolog.Event) {
 e.Str("user", u.Name).Str("email", u.Email)
}

// where its called
u := User{Name: "test", Email: "test@test.com"}
logger.Info().Object("user", u).Msg("user logged")
// outputs:
// INF user logged user={"name":"test","email":"test@test.com"}

Note the above example output is using a ConsoleLogger, not the JSONLogger which you should use in production.

Tags:

#go #zerolog

WGD 2022-05-20

Fri, 20 May 2022 00:00:00 +0000

WGD 2022-05-20

Still working on Mudmap’s switch from single user accounts to multiple accounts per organisation.

Mudmap

I’m making a lot of progress on the transition from the one user one account model. I’ve had to create extra tables, migrate users and devices between them whilst ensuring backwards compatibility. I think the changes to the database models are done for this section of changes - permissions will come next.

I learned how to write Postgres subqueries this week and it’s pretty cool seeing a complex query execute repeatedly without issue.

User and organisation (what I’m calling accounts) data is stored in the database but a couple of items are being used in context.Values. Doing this prevents Mudmap from having to do queries for user or organisation data when accessing other related data. I understand that it is a slight anti-pattern but I feel it’s actually a good use case for it. Using context also allows for storing certain keys in the authentication token. Again, it saves extra queries.

I’ve been using go-resty for making and receiving HTTP requests inside Mudmap. In the past I’ve written HTTP clients using just the standard lib but resty provides a few things I need almost out of the box. Namely, retries and interceptors. When a token expires, resty is configured to automatically re-authenticate by sending a POST to the auth backend. I could write that myself but I’m running a business and don’t have time for that!

Also got to play with some Go Mutex’s this week too. Have not really had a need to implement any until now, and it’s pretty easy. Three lines of code will get you pretty far.

// contrived example
var mu sync.Mutex
mu.Lock()
defer mu.Unlock

Work

Had some huge wins at work this week. Our team got a lot of kudo’s for the work we’ve been putting it too. No rest though, straight from one fire to the next. So far, I’ve worked on Django, Angular and Android this week alone. Its enough to scatter a man’s brain.

Voyager-1 an engineering feat

Thu, 19 May 2022 00:00:00 +0000

Voyager-1 an engineering feat

I am blown away that the Voyager-1 spacecraft is nearly 45 years old and still functional. Engineers are still administering this craft, most of whom were not born when it launched! We have multi-billion dollar companies experiencing days of downtime with more engineers than you can shake a stick at, and then we have this beast.

It takes 20 hours to receive data from the vessel! It is also the only spacecraft we have in deep space.

What an absolute marvel of modern engineering.

Engineers Investigating NASA’s Voyager 1 Telemetry Data

Tags:

#space #engineering #nasa

Go-resty: A Simple HTTP Client

Wed, 18 May 2022 00:00:00 +0000

Go-resty: A Simple HTTP Client

When you need a decent HTTP client and don’t want to craft your own, then go-resty is a good choice. It is great when you need retries or an way to send requests based on the response, i.e. token expiration and renewal.

How I implement a JWT re-authentication check before each request.

import (
	"github.com/go-resty/resty/v2"
 "github.com/form3tech-oss/jwt-go"
)

func NewAuth0() Auth0 {
	c := resty.New()
	c.OnBeforeRequest(func(client *resty.Client, request *resty.Request) error {
		fmt.Println(c.Token)
		if !IsTokenValid(c.Token) {
			r, err := Authenticate()
			if err != nil {
				return err
			}
			request.Token = r.AccessToken
		}
		return nil
	})
	return Auth0{
		Groups: GroupApiStruct{
			client: RestClient{*c, *config.AppConfig(), c.Token},
		},
	}
}
func IsTokenValid(token string) bool {
	println(token)
	if token == "" {
		return false
	}
	claims := &jwt.StandardClaims{}
	p := &jwt.Parser{}
	_, _, err := p.ParseUnverified(token, claims)
	if err != nil {
		return false
	}
	err = claims.Valid()
	if err != nil {
		return false
	}
	return true
}

func Authenticate() (AuthResponse, error) {
	c := config.AppConfig()
	client := resty.New()

	var result AuthResponse
	_, err := client.R().
		SetHeader("Content-Type", "application/json").
		SetBody(AuthPayload{
			GrantType: "client_credentials",
			ClientId: c.Secrets.Auth0ApiClientId,
			ClientSecret: c.Secrets.Auth0ApiClientSecret,
			Audience: c.Secrets.Auth0ApiClientAudience,
		}).
		SetResult(&result).
		Post("https://mudmap.au.auth0.com/oauth/token")
	if err != nil {
		fmt.Println(err.Error())
		return AuthResponse{}, err
	}
	return result, nil
}

type AuthResponse struct {
	AccessToken string `json:"access_token,omitempty"`
	ExpiresIn int `json:"expires_in,omitempty"`
	TokenType string `json:"token_type,omitempty"`
	Scope string `json:"scope,omitempty"`
}

type AuthPayload struct {
	GrantType string `json:"grant_type,omitempty"`
	ClientId string `json:"client_id,omitempty"`
	ClientSecret string `json:"client_secret,omitempty"`
	Audience string `json:"audience,omitempty"`
}

This lets me check that the JWT is valid and if not, to request a new one from the service.

Coding this myself using just the standard library while possible is not a pragmatic use of my time. I vendor my code so even if resty went away tomorrow, it’ll keep working long enough to replace it.

Tags:

#go #rest

People don't listen and I don't have time for them

Sat, 14 May 2022 00:00:00 +0000

People don’t listen and I don’t have time for them

Last night I was at a house warming and did some light mingling with people who I mostly don’t know. Only two people, the host and another bloke, actually conversed with me. Everyone else, was having a one-way conversation where they were talking at me or waiting for their chance to speak. It is a common thread in my interactions with people these days.

Honestly, they are doing me a favour by exposing themselves quickly as someone to not waste any more time with.

Life is too short for these kind of interactions.

Tags:

#rant #people #courtesy

WGD Fri 2022-05-13

Fri, 13 May 2022 00:00:00 +0000

WGD Fri 2022-05-13

This week has been all about research, proof-of-concept’s and database migrations.

Mudmap

My biggest customer has been asking for Mudmap to support multiple users under a single account. Last week, they really put a fire under my feet saying they need it, and won’t be pushing the rest of their devices onto the platform until its done. In terms of numbers, it equates to about a 900% increase in billable units. So, it is worth dropping everything else to make this a reality.

I spent this week testing my assumptions, writing documents (mostly fleshing out my ideas) and some of the migration files needed. This also includes the initial models files on the backend.

zet-cmd

This week I pushed some updates and fixes to my zettelkasten CLI tool. I typically read the zet’s using vim but sometimes it is nice to format them to the terminal nicely instead. As a regular user of gh, I noticed their markdown renderer was quite beautiful. So, I looked through the go.mod and found the package by charmbracelet called glamour. The pull request was pretty simple. I also tidied up some things that after some use made zet-cmd feel unnatural.

Old friends

Also caught up with a mate I’ve not seen in a couple of years. Now that the travel restrictions are lifting, more and more people I know are filtering through Canberra. It was a great evening reminiscing.

Backups to Cloud

I had over 250GB of files which have been sitting on a single external HDD just waiting to get corrupted. I’ve wanted to push them to my cloud storage for a while but lacked a good way of doing a. This week I found rclone. What an easy and fool-proof experience. It took a couple of days to sync but was largely set and forget. It was cool showing my girlfriend GoPro video of use jumping out of planes into the ocean too. I did that stuff for nearly ten years and she still has really no idea of what I did day to day.

Tags:

#wgd

Developer Experience is Everything

Wed, 11 May 2022 00:00:00 +0000

Developer Experience is Everything

In my day job, our development velocity is a crawl. One year ago we were a high performing team knocking out updates and smashing sprints. Today, due to corporate desire to streamlime multiple development teams we are effectively useless. They wasted millions of dollars building a system which will never meet our needs - it literally can’t. The sad part? The new platform is actually really slick (OpenShift).

For any future job, developer experience must be spoken about during the interview. It’s a tough thing to do because everyone lies during interviews or they don’t talk about it because of non-disclosure. I.e. everything is falling apart but we need to bluff this bloke in to taking this code monkey job, and for cheap.

If you cannot develop on your machine (either on device or remotely into container etc) and see your changes in real-time, the developer environment is broken. The business will atrophy good developers because of it, meaning the people who can fix it will leave. Life is far too short for that sort of corporate bullshit.

Tags:

#rant #development

Old skills, new skills

Wed, 11 May 2022 00:00:00 +0000

Old skills, new skills

Where I’ve been is not where I am today. I’ve been uploading some old photos from a hard drive to the cloud. Looking over them with some nostalgia and a level of pride about how I transitioned from one life to the next. For a guy that never finished high school and was told I’d never amount to anything, I’ve done alright.

Old Skills:

Sniper
MOE
Diver
Driver
Parachuting
FOFO
Submarines
Improvised Explosives
Clandestine MOE
Surveillance
Surveillance Detection
Navigation
CQB Instructor/TL
Water Ops Supervisor
Boating
Visual Tracker
Roping
PAFA
Survival

New Skills:

Software Dev
Networks
k8s (beginner)
Python, Go, JS
Shell Scripting
Linux Admin
CI Pipelines
Testing
Documentation
PR/Code reviews
SaaS Owner

Tags:

#life #skills

Mudmap Org Register Proposed Flow

Mon, 09 May 2022 00:00:00 +0000

Mudmap Org Register Proposed Flow

The biggest issue I’ve come to experience with the new multi-account design is re-arranging how Users fit.

I’ve decided organisation is a better name than account or business.

Flow

User signs ups (entered in DB)
Org is created, User assigned to Org as Admin (Org name is user.email)
Users are still tracked in DB using their user_id from Auth0
Org’s will now get the Stripe sub_id (Not the User)
User’s are added to Org’s by invitation
User’s must accept the invite, triggering the move.
Devices belong to Org’s
All members of an Org can see Devices
Existing devices will be moved to each User’s (new) Org
Org delete will CASCADE

Everything is tracked in the database but “synced” with Auth0 using the Authorization Extension. This allows Mudmap to review a User’s access to an Org and Device (permissions are omitted for now) by inspecting the JWT. If the JWT has some limitation that I’ve not accounted for then each request will require a DB lookup. For latency reasons I’d prefer not to do that. If, that is the case it’s probably a prime reason to transition to Litestream (assuming replication is working well).

This is a ten thousand foot view and still needs a few things ironed out.

Tags:

#mudmap #research #planning

Twitch streams I follow

Mon, 09 May 2022 00:00:00 +0000

Twitch streams I follow

I watch Twitch at work but not logged in so I sometimes forget which accounts I like to view.

Tags:

#twitch

Cheatsheet: psql and pg_dump

Sat, 07 May 2022 00:00:00 +0000

Cheatsheet: psql and pg_dump

In my day job I use MariaDB so I often forget the syntax for Postgres and I’m trying not to leverage graphical DB explorers as often. When jumping on to containers from a k8s orchestration platform, you don’t often have access to anything but the CLI.

Commands

Login

In the container: psql -U <username> -d <database> On local host to db: psql -h localhost -p 5432 -U dbuser -d db

Not specifying the database will cause an error, unlike MariaDB which just wants a username and password.

Show Tables

\dt

Desc Table

\d <table-name>

Dump DB into File:

pg_dump -U <user> -d <database> >> sqldump.sql then run docker cp if its inside a container.

Restore DB from file:

psql -f sqlfile.sql -U dbuser -p 5432 -h localhost -d db

**Install psql without Postgres

sudo apt install postgresql-client

Still stuck?

Instead of reaching for google, run curl cht.sh/psql it is fantastic.

Tags:

#database #psql #postgres

Mudmap sub-account overview

Sat, 07 May 2022 00:00:00 +0000

Mudmap sub-account overview

A rough idea of what this change should achieve.

organisation is has a collection of users
users has a relation to permissions
device belong to organisation, not users
Existing users’s must be attached to a organisation (probably by creating a organisation using the user.Name field as group.Name and adding them as a users to that group)
organisation owners can update organisation info
permissions should be a Many-to-Many. users can have many permissions and the same permission can belong to many users.
Auth0 should also be aware of users and organisation
Auth0 should also be aware of permissions
Auth0 should place permissions within the token (if possible)
Updates to the permisions table must also be reflected in Auth0
Updates to the organisation table must be reflected in Auth0

Next Steps

Test assumptions with Auth0’s Authorization and Group tooling
Create permissions and organisation table
Update users table to have relation with organisation
Create change devices relation from users to organisation
Write sync methods so that the DB and Auth0 remain in sync
Middleware for checking a user can access a device based on JWT token fields

This is a decent sized change with a few moving parts. I can possibly remove the Auth0 lock-in but I see no good way of doing this without forcing a DB lookup for each request (check permission and group). If I use Auth0 I can hopefully keep it sync’d with the DB and embed that data in the app_metadata field of the token. A assumption I need to test fully before too much investment in code.

20220505024039 Mudmap sub-accounts

Updates

I’ve added app_metadata to the JWT. This can now be plucked off on the backend.

Its possible to use M2M API key to make changes to the Group, Permissions and Roles of Users and organisation. However, the free-tier only gives you 1000 calls per month (from what I can tell)

API Docs: https://auth0.com/docs/api/authorization-extension Getting a token: https://auth0.com/docs/api/authorization-extension?shell#get-an-access-token

Some calls examples:

# get all groups
curlie https://mudmap.au12.webtask.io/<URL>/api/groups -H "authorization: Bearer <TOKEN>"

Returns

{
 "groups": [
 {
 "_id": "8931fb3a-8163-4c8a-8534-b5cc43381172",
 "name": "Test-Group",
 "description": "For Testing Only",
 "members": [
 "auth0|61de4f3e8e3c6000710b8c0d"
 ],
 "mappings": [

 ]
 },
 {
 "description": "test-description",
 "name": "new-test",
 "_id": "0fdf83fe-32db-463c-b362-4e29c0817781",
 "mappings": [

 ],
 "members": [

 ]
 }
 ],
 "total": 2
}

Tags:

#mudmap #planning

WGD Fri 2022-05-06

Fri, 06 May 2022 00:00:00 +0000

WGD Fri 2022-05-06

Preparing Mudmap for a big change and researching account grouping functionality.

Mudmap

Mudmap’s underlying API has seen a big update and I spent a fair chunk of time testing it to make sure no breaking changes are introduced.

I can now create a Go client from the openapi.json the API provides. But, I find the auto generated code that gets created really hard to follow. I guess that is the point, its a contract between the API and the client. Nonetheless, I’m not going to use the tools I’ve tried but instead just use it to influence how I build each endpoint.

Also, I started researching and thinking through how I am going to create an account hierarchy within Mudmap. Users want to have teams or groups with RBAC policies. Since I use Auth0 as my authentication provider, I am somewhat pigeon holed to use their methods. From what I can tell so far, their Organisations product is not what I want. It is also too expensive. Instead I am looking at creating some custom wrapping code to extend their groups extension with my own database. Research continues as there are some considerations such a latency here.

Updated Mudmap’s goreleaser builds to use 1.18 as I’ve started using some generics where it makes it easier to grok that piece of code.

Noticed that Mudmap has a very small memory leak somewhere. Over the course of a week the memory increases about 5mb which is nothing. Still, I hate that it is there but have decided put it on the backburner while I go after more critical items.

Mudmap Sub-Accounts

Wrote the April retrospective but haven’t published it yet. Still need to proof it and make sure the standard grammar issues are fixed up.

Released a number of small updates for my zet-cmd package. Really enjoying working on it and using it to catalogue my thoughts. People can freely read it which is fine but I feel zero regrets from the poor wording or brain spew that gets placed there.

Conclusion

Worked nearly 50 hours this week on a big deployment. It’s bloody frustrating working in the constrained environment I call work. Aside from that, it has been a good pairing with a much more senior engineer during the process.

Tags:

#whatgotdone

Mudmap Org Permissions

Thu, 05 May 2022 00:00:00 +0000

Mudmap Org Permissions

Mudmap must be able to provide user accounts under an account root. This account root will be called an organisation. All users will belong to an organisation.

RBAC policies are applied per-user according to their level of access.

As a general guide, a root account is created from which all accounts for that company or group are then underneath. This account is the organisation owner for lack of better wording. It might be prudent to allow multiple root account holders.

This means devices are attached to an account or at least an account user. Users associated to that user can see any device in the hierarchy. At the moment, only a single user account can have and see its own devices. This will need to be radically altered to accommodate these changes.

RBAC Policies

Account Type	Payments	Privileges	Read	Write
Root	y	y	y	y
Privileged	n	y	y	y
Manager	n	n	y	y
User	n	n	y	n
Custom	?	?	?	?

managers cannot elevate privileges (only privileged account or root)

Questions

How to create this type of account setup with Auth0? Do I need to hack something together
How to grant RBAC - JWT or DB look up on each request? JWT best case, DB worse but either way RBAC should be stored in a DB for reference in admin panels
How to associate users to the root account programmatically with backward compat

Tags:

#mudmap #research

April 2022 Retrospective

Wed, 04 May 2022 00:00:00 +0000

Summary

Recharged after a wonderful holiday spent visiting family and friends who we haven’t seen since Covid started.

Highlight

Went on a relaxing holiday in Alice Springs.

Goal Performance

A review of last months three goals. See March’s Retrospective.

Push two Mudmap features to production

Appraisal: I did push some very small fixes but nothing substantial
Rating: C+

From the outset, I over committed myself this month. Nearly ten days of this month was spent on holiday where I did very little active development on Mudmap. Although, I did resolve some customer inquiries, analyse bug reports and speak with customers.

A substantial update was released for the API that is installed on the firewalls. It was officially released at the end of the month but has been available in beta for several months. In preparation for this, I’ve been getting some things in order to make the transition easy. Included in the release is support for pfSense+ v22.01 - something I did not expect to make it out of beta.

In prep for this, I’ve been working on building a development environment and acquiring the authorizations to deploy pfSense+ testing devices. This includes local virtual machines and cloud instances.

Other than that, I made some fixes to the user interface such as typo’s on buttons. Mudmap’s frontend will also ask users to refresh their page if a newer version is available whilst they are actively using it. Though I have found that caching on Next.js seems to intelligent enough to detect changes to the assets pretty quickly.

Walk to Alice Springs telegraph station

Appraisal: Easy
Rating: A

A silly goal but something I set purely to ensure I took my little girl on a decent walk through some of Australia’s most beautiful country.

I’ve had a few little holiday breaks here and there over the last year or so. None of them hold a candle to how relaxed and re-charged I feel after this one. Alice, is a remarkable little place of only about 25,000 people, and stands in the centre of Australia. The nearest cities are Darwin and Adelaide both being about a 15-hour drive (click the links). Australia’s vastness is hard to contemplate for many - driving 15 hours in Europe, Asia or the America’s will have you passing by many large cities. In Coober Pedy, on the way to Adelaide, people live underground as it’s literally too bloody hot on the surface!

The picture below is from a short overnighter out bush - my daughters first night in a swag. She was absolutely thrilled to be sleeping under the stars and to wake up as the sun rises. In sum, it was a fantastic holiday with friends and family.

Recommendations

Learn Golang

If you’re a developer and looking to pick up a new language then you should have a look at Go. I had toyed around with it on and off for a while but started writing a lot more stuff in it towards the end of last year. Now, I reach for it whenever I’m starting something that is more than a simple bash script. There is a lot to love but for me, it’s Go’s ability to be compiled to any operating system and how easy it is to bundle assets (such as HTML) into the binary. Go is truly portable and after years with Python and Javascript, it is such a weight off my shoulders. Understanding the language can also be really helpful when figuring out errors with cloud tools too. After all, basically every cloud project these days is written in it.

Wrap up

A quiet month on the retrospective front but a big month for me personally. Looking forward to writing a more lengthy post in June.

What can I do better?

Be more time aware when making plans, i.e. I did not really factor in how little development work would get done one Mudmap during my holiday.
Update the roadmap more frequently as this is something users are asking about often.
Release the monthly retro in a more timely manner.

What have I done well?

Actually took a proper amount of time off and feel recharged
Spent sometime exploring a few other programming projects

Next month’s goals

Deploy pfSense+ capable Mudmap
Get started on multi-account support (a big customer request)

Analytics

mudmap.io

check-redirects.com

danielms.site

Microservice contracts can be coupled

Mon, 02 May 2022 00:00:00 +0000

Microservice contracts can be coupled

Microservices aren’t evil or the saviour, but they are often touted as the solution to a lot of problems. One of the biggest microservice misconceptions is decoupling.

A service which sends stuff to a queue to be consumed by another service is not magically decoupled. It takes work and consistency to do that. Lately, I’ve been seeing a lot of service code with references to things that are only a concern for the consuming services. I.e. they’re coupled.

Further, if service A needs more data from another service, now I need to touch two code bases to make that change. If I deploy one without updating the other, it could result in a crash (though not always).

I like writing services when it makes sense but I do not agree its the panacea of all coding. For a lot of things, monolith’s or large services are more efficient. This holds especially true if your team size is small.

Tags:

#rant #microservices #design

Go test -race requires CGO_ENABLED=1

Sun, 01 May 2022 00:00:00 +0000

Go test -race requires CGO_ENABLED=1

To run go test -race CGO_ENABLED=1 must be set. I do not use CGO as I do not want any linking.

The workaround is simple, export it for that command.

CGO_ENABLED=1 go test -race ./...

Tags:

#golang #cgo #testing

pfSense-api v1.4.0

Sat, 30 Apr 2022 00:00:00 +0000

pfSense-api v1.4.0

Some excellent news, pfsense-api has been updated to v1.4.0 which brings several fantastic changes.

Highlights:

Support for pfSense+ (this could be a catalyst for many new customers)
Basic Auth instead of sending in the body* (see notes)
OpenAPI swagger endpoint (previously it was a markdown doc)
API IP white listing (might make HTTP access more palatable)

Notes

Basic Auth is better than munging a body to pass authentication but I cannot switch completely for backward compatibility reasons.

Tags:

#pfsense #mudmap

WGD Fri 2022-04-29

Fri, 29 Apr 2022 00:00:00 +0000

WGD Fri 2022-04-29

Mudmap

I added some extra notifications, namely webhooks, whenever a user increments or decrements their device numbers in Stripe. I.e. when a payment intent is sent for a subscription.

Also, spent some time investigating firewall connection issues which a user notified me about. Ultimately, I could not replicate the issue. What they provided me works but I look forward to their response as it could be affecting other users as most don’t report issues.

I started mulling over the idea of switching to using HTTP, or providing it in addition to SSH for Mudmap connections. I understand that this means exposing the firewall’s user interface to the internet but it would be locked down to a few IP’s. Which is what happens with SSH currently - though SSH is more secure than a web server. Needs more thought but what makes this even more appealing is the API client I use is soon releasing a OpenAPI document so I should be able to auto-generate HTTP endpoints.

ds

I integrated zet-cmd into ds and updated the underlying dependencies.

zet-cmd

I have too many notebooks. My writings and scribbles go into a book and get lost, forever gone and never looked upon again. So I am kinda re-inventing the wheel yet again and trying something I have failed at before, taking my notes online.

So, I decieded to create a zettelkasten tool for cataloguing my various thoughts in a way that is publicly accessible. Unlike my blog which I tend to take a little more time curating, these entries should be more raw with less care for grammar or correctness.

And, because I’m a developer, I created a tool to do this. It uses bonzai to create a simple (easy to use for me) CLI tool for writing markdown files. This is written in my zet repo and copy pasted into What Got Done.

So far, zet-cmd can do the following:

create new zet’s
edit zet’s
search by title and tags
commit and push to GitHub

Conclusion

Work has been pretty interesting and fun this week. I’ve been pulling longer hours but the days have been good. A lot of k8s work which I really enjoy.

Golang URL Params

Thu, 28 Apr 2022 00:00:00 +0000

Golang URL Params

To create URL encoded parameters on a URL in Go.

// example from a Zulip webhook
params := url.Values{}
params.Add("to", "stream-name")
params.Add("topic", "topic")
params.Add("type", "stream")
params.Add("content", "here is the content")
encoded := params.Encode()

u := fmt.Sprintf("https://example.com?%s",encoded)

fmt.Println(u)

This will create the URL as var u with all the parameters correctly encoded. This could then be used in http.NewRequest like so:

// skip err handling for brevity
req, _ := http.NewRequest("POST", u, nil)

Tags:

#golang #web

Initial Zettelkasten

Tue, 26 Apr 2022 00:00:00 +0000

Initial Zettelkasten

This marks the very first Zettelkasten entry in this repo. To do this I’ve created my own Zet tool called zet-cmd written in Golang. It uses Bonzai by rwxrob and is heavily inspired by his own zet repo.

I needed a way to capture small, digestible nuggets of wisdom and other thoughts or learnings in a manner that let’s me do it without needing a full blog post. It also needs to be searchable publicly and using GitHub makes that pretty simple.

So far zet-cmd can:

Create new Zet’s
Edit existing ones
Search across all Zet’s
Generate a url with a search term for the repo on GitHub
Retrieve the latest zet for easier editing

In the future more commands will be added which make life easier. More helper commands to make piping data into Vim will be added in the future.

Some example usages:

Creating a GitHub.com search query with zet q "search term" would return: https://github.com/danielmichaels/zet/search?q=search+term

Tags:

#zettelkasten #golang #bonzai #rwxrob

Vim Filters

Tue, 26 Apr 2022 00:00:00 +0000

Vim Filters

Today I learnt about (and showed some vim-hater colleagues) vim filters.

I am still at the point of consciously thinking about them. So, I’ve already noticed a few spots where I could of used a filter but instead dropped back to what I know.

A few contrived and simple examples.

Get the first zet with a #golang tag

:r !ds zet t golang | awk '{print $1} | head -n 1
# 20220424000235

Get unix time

:r !ds uniq second
# 1650971184

Line count for this document

:r !wc -l #
# 16 zet/20220426105644/README.md

These are so basic its not funny but gaining an awareness is the start. Conscious thought when writing leads to the mental leaps which turn things like this into muscle memory.

Tags:

#vim

ANZAC Day COVID Bullshit

Mon, 25 Apr 2022 00:00:00 +0000

ANZAC Day COVID Bullshit

ANZAC Day Dawn Service in Perth 2022 is invite only with a maximum attendance capacity of 500 due to COVID restrictions.

The local football team - The West Coast Eagles - home attendance’s for 2022:

Round 1: 20,932
Round 3: 38,920
Round 5: 42,888

What a load of absolute bullshit. We all know why. There is no money made from a memorial service but football, cha-ching.

Risk versus Reward in capitalist society will forsaken those who made the ultimate sacrifice to pursue a few Shekels.

Tags:

#covid #rant

How to check Go module versions

Sun, 24 Apr 2022 00:00:00 +0000

How to check Go module versions

When creating zet-cmd and using go install to pull the latest version, I was having some issues where it would not grab the most recent version.

Here are some methods to find and then retrieve the version you want.

To check all existing versions:

go list -m -version github.com/danielmichaels/zet-cmd

A list of compatible operators when using go install or go get:

Specific version: @v.1.3.4
Specific commit: @abcdefgh
Specific branch: @main
Version prefix: @v2
Comparison: @>=2.1.4
Latest: @latest (most common)

Tags:

#golang #modules

Github Vigilant Mode Setup

Sun, 17 Apr 2022 00:00:00 +0000

Setting up Github’s vigilant mode

create GPG key

gpg --full-generate-key
Pick RSA
Enter a key of at least 4096
Default expiration
Enter git config --global user.name value
Enter git config --global user.email value
Add a comment describing it as Github or similar

Add the key to Github

Retrieve the key ID and copy it into the clipboard.

Run the follwing and grab the key ID on the ssb line.

❯ gpg --list-secret-keys --keyid-format=long dan@danielms.site
sec rsa4096/XXXXXXXXXXXXXXXX 2022-04-15 [SC]
 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
uid [ultimate] Daniel Michaels <dan@danielms.site>
ssb rsa4096/XXXXXXXXXXXXXX8B 2022-04-15 [E] <-- copy after rsa4096/

Github expects the Private key file so next retrieve it using gpg --armor --export XXXXXXXXXXXXXX8B. Copy the output to the clipboard.

Paste it into the GPG keys page inside Github using their guide

Ensure your commits (and tags) are signed

The easiest way is to setup every commit as signed lest you forget to sign a commit.

git config --global commit.gpgsign true
git config --global user.signingkey ABCDEF01 (where ABCDEF01 is the fingerprint of the key to use)
git config --global alias.logs "log --show-signature" (now available as $ git logs)

Copying GPG key between devices

If you have more than one device and do not want to have several keys but instead use the same key across all devices, there are a few steps.

gpg --list-secret-keys user@example.com

gpg --list-secret-keys user@example.com > private.key

Copy the key to the device using scp or similar and then install it into the gpg keychain.

gpg --import private.key is touted to work however this did not work for me on Ubuntu 20.04.Instead, I found gpg --batch --import private.key worked as expected.

Deleting tags

If you’re like me and push tags without signing them first (fixed by following the above guide) how do you delete them?


# delete local tag '12345'
git tag -d 12345
# delete remote tag 
git push --delete origin tagName

This is a quick down and dirty on setting up GPG keys, git and GitHub.

March 2022 Retrospective

Wed, 06 Apr 2022 00:00:00 +0000

Summary

After taking some time off to have fun and experiment, I got back into the swing of Mudmap development.

Highlights

Released several updates to Mudmap
Got Covid (not really a highlight but noteworthy)

Goal Performance

A review of last months three goals. See Feb’s Retrospective

I did not set any goals for this month. My partner has been quite unwell, and I decided to take some pressure off and be more available for her.

Month overview

I discovered the project Alpine.js this month and was really intrigued by how user-friendly it is. I am pretty comfortable using React these days but the added complexity of running a separate frontend can get a little annoying, especially for simple app’s. Setting up Alpine is a cinch and can easily be integrated into any web application by either dropping in a link to the CDN, or by creating a bundle for it and linking it directly.

Where Alpine shines is when coupled with traditional web applications such as Django, or Go templates. To get more familiar with it, I refactored an existing app of mine - tars.run. It was originally built with a Next.js frontend but was overkill for what it does. Now, instead of needing two applications, the entire thing including database runs inside a single binary. It is refreshingly easy building app’s this way.

Mudmap updates

I have received a number of requests from customers asking Mudmap to support devices which do not have static IP addresses. I finally added this feature and it was so simple that I regret not doing it sooner.

Mudmap now shows each device’s DHCP lease table too. This feature is not fully finished as you cannot update any of the leases, only view them. I chose to release it in stages so that customers can start using it straight away, rather than having to wait for all aspects of that feature to be completed first. This is probably a theme I will continue with, and it allows for a faster capture of feedback too.

The application user interface also received a bit of love. Most of the tables now support pagination by default, have smaller margins and where appropriate have tool-tips to reduce ambiguity. Many of the table buttons are now smaller, or represented by a more familiar icon - the play button for example. These small fixes do a lot to make the user interface feel more polished.

Recommendations

Re-watch Seinfeld. It’s been such a brilliant way to decompress before bed and even after 30 years it is still hilariously relevant. I am about half-way through the entire show after nearly a month of watching an episode or four each evening. So good.

I have completely moved from Arch (actually, Manjaro) to Ubuntu and couldn’t be happier. Honestly, I’m not sure if I’m missing anything from the move - any package that I had on Arch I probably have on Ubuntu. And, I actually like snap packages - I said it! I’ve had a couple of little issues, but they’ve all been super easy fixes.

fighting words: Ubuntu is the closest thing we have to linux desktop, and I’m now a recommender of the distro.

Wrap up

I expected this month to unproductive but even after getting Covid, it still ended up being quite good. I was able to use my time to experiment with things and just have some fun which gave me the break I needed to get back into Mudmap development. It was also good just coming home and relaxing some nights, without the guilt I’ve put on myself to keep working on things like Mudmap. This has been especially important as my day job’s workload has increased. In all, March was a far better month than February.

What can I do better?

Write in my journal every day, even if only a couple of words
Read more often before bed

What have I done well?

I Wrote two short blog posts this month
Didn’t take myself too seriously!

Next month’s goals

Push two Mudmap features to production, no matter how small
Walk with my daughter to the telegraph station whilst we’re in Alice Springs in April

Analytics

mudmap.io

check-redirects.com

danielms.site

Citrix Workspaces SSL Error on Linux Fix

Fri, 01 Apr 2022 00:00:00 +0000

TIL: Citrix on linux does not work out of the box.

I don’t know why it’s like this and honestly don’t really care what the excuse it - it feels like a perfect example of how much care Citrix puts into its products.

This quick post goes over how to get it working.

Overview

You need to download Citrix’s certificate and put it in the ICACClient’s keystore - which is ridiculous.

Download the Certificate

Head to Citrix’s website.
In your browser’s navigation bar there is likely a lock symbol. Clicking on this should give some info about the certificate. Click around here until you end up at a page with the cert’s information.
In the certificate’s about page, scroll down until you hit a PEM download button and click it.

Place certificate in Citrix keystore

Change the PEM file to a .crt using cp cert.pem cert.crt.
sudo up using sudo su
Change the permission to 644 with chmod 644 cert.crt
Make owner root with chown root:root cert.crt
Move this file into the keystore by running cp cert.crt /opt/Citrix/ICAClient/keystore/cacerts/
Run /opt/Citrix/ICAClient/util/ctx_rehash to make Citrix recognise the new cert

Doing these steps should resolve the SSL error and allow you to use Citrix as expected.

NOTE: These steps assume Ubuntu 20.04 and icaclient_22.3.0.24_amd64.deb

Reference: https://support.citrix.com/article/CTX231524

Alpine.js and Tailwind HTML setup

Sat, 12 Mar 2022 00:00:00 +0000

Plain HTML with Alpine and Tailwind CSS

Today I set up a Go project using Tailwind and Alpine.js. This web application uses Go templates so there was no need for any React or other javascript frontend. I did want dropdowns and to use tailwind without a CDN. This meant I needed to install some NPM packages and include some javascript to facilitate the dropdowns.

This is the minimum requirement to get a basic Tailwind and Alpine system set up for a simple Go web application - it could extend to any other language but is not tested.

Pre-amble

This project is laid out with the static files and html inside a directory named ui within their own directories.

Before the setup and initialisation of Alpine and Tailwind, the directory looks something like this.

ui
├── html
│   ├── base.layout.tmpl
│   ├── footer.partial.tmpl
│   ├── home.page.tmpl
│   └── navbar.partial.tmpl
├── static
│   ├── css
│   │   └── main.css
│   └── js
└── ui.go

Installing and compiling Tailwind

First thing to do is install Tailwind. I use yarn but I’ve used npm here because its more widely used.

npm install -D tailwindcss 
# I also installed these tailwind plugins which I needed for the layout
#npm install -D @tailwindcss/forms @tailwind/aspect-ration
npx tailwindcss init

This creates a tailwind.config.js file which needs to be updated to reflect our directory structure and template file types.

module.exports = {
 content: ["./ui/**/*tmpl"], <-- THIS IS IMPORTANT
 theme: {
 extend: {},
 },
 plugins: [
 require('@tailwindcss/aspect-ratio'),
 require('@tailwindcss/forms'),
 ],
}

What tripped me up was the content: [] block. My Go templates use the following naming convention; <name>.<type>.tmpl. The types are layout,page and partial but it always ends with tmpl. Initially, I was using .html because I wasn’t paying attention and copying other guides. This time I am catching all tmpl files inside the ui folder.

I then created a npm script block entry into the package.json file.

{
 "scripts": {
 "build-css": "tailwindcss -i ui/static/css/main.css -o ui/static/css/theme.css -m",
 },
 "devDependencies": {
 "@tailwindcss/aspect-ratio": "^0.4.0",
 "@tailwindcss/forms": "^0.5.0",
 }
}

build-css will create a file called theme.css in the ui/static/css directory which is then referenced inside my base.layout.tmpl template. The -m minifies it, which I do for prod and dev because this file is never touched.

The above step is predicated on the existence of a file called main.css inside the ui/static/css directory. For clarity my main.css looks like this:

" https://tailwindcss.com/docs/installation - Step #3
@tailwind base;
@tailwind components;
@tailwind utilities;

Alternatively, you could run just run the compiler (build-css) in a bash script, CI pipeline or Dockerfile.

Alpine.js Set up

Next, bundle up Alpine inside the application without needed to use the CDN.

npm install alpinejs esbuild

Alpine needs to be instantiated, so I created an index.js at the root of the directory and called Alpine.

import Alpine from 'alpinejs';

window.Alpine = Alpine;

Alpine.start();

Then I add an extra script to the package.json which will compile Alpine using esbuild and drop it into the ui/static/js directory. Alpine, like tailwind is minified because I am not altering it in any way.

{
 "scripts": {
 "build-css": "tailwindcss -i ui/static/css/main.css -o ui/static/css/theme.css -m",
 "alpine": "npx esbuild index.js --outfile=ui/static/js/bundle.js --bundle --minify" # New
 },
 "devDependencies": {
 "@tailwindcss/aspect-ratio": "^0.4.0",
 "@tailwindcss/forms": "^0.5.0",
 "tailwindcss": "^3.0.23"
 },
 "dependencies": {
 "alpinejs": "^3.9.1", # New
 "esbuild": "^0.14.25" # New
 }
}

The final directory structure should look something like this

ui
├── html
│   ├── base.layout.tmpl
│   ├── footer.partial.tmpl
│   ├── home.page.tmpl
│   └── navbar.partial.tmpl
├── static
│   ├── css
│   │   ├── main.css
│   │   └── theme.css # New
│   └── js
│   └── bundle.js # New
└── ui.go

Testing it out

Remove any references to Tailwind or Alpine’s CDN and replace them with the location of theme. css and bundle.js respectively. Refreshing the application page should render everything correctly and any Alpine widgets should work.

The best part is the application is completely portable. For clarity, this application is using Go embedded files so that it is self-contained. After running go build you will be able to run it completely isolated and have the CSS and JS work as expected.

TIL: Interview Questions Worth Thinking About

Fri, 11 Mar 2022 00:00:00 +0000

Interview Questions Worth Considering

Today I stumbled across a job advert that had a section called The Chat. Here is the snippet of things they might ask you about during the interview.

Language: Why? What do you like and dislike about it?
Domain Modelling: What’s it good for?
Databases - Querying and Transactions: what makes them challenging?
API Design: What’s taken into consideration?
Application Architecture: Microservices, monoliths, event-driven, app servers and serverless; when and why?
Testing and Code Quality: Where and how does it add value?
SCM: Branching and merging flows - how do you like them?
Logging, Metrics, and Monitoring: What’s the difference?
Error handling and Debugging: Describe your process and what’s got you stuck in the past?
SDLC and CI/CD: Benefits and downfalls?
Security: What does that mean to you?

I like this because I immediately get the sense that this company is thinking about these issues, and potentially looking to see how you could fit in.

Most importantly, I realised that some of these points where weak spots, or rather I would need to collect my thoughts to answer effectively. How much of what I know is based on previous training/conditioning versus good practices. There is no right or wrong but having a well thought out response to these questions is valuable - not just in an interview but at any time.

Feb 2022 Retrospective

Sun, 06 Mar 2022 00:00:00 +0000

Summary

A number of external influences impeded my ability to do get much work done this month. As a result very little was achieved on Mudmap.

Highlights

Small fixes for Mudmap installer
Built a Cobra CLI for interacting with an Australia crypto exchange

Goal Performance

A review of last months three goals. See January’s Retrospective

Deliver the Interfaces feature for Mudmap

Appraisal: Failed
Rating: F

I didn’t even get close to completing this.

A bit of background. A number of things have been happening this month in my personal and work life - which obviously share resources with my development time. My entire family got quite sick though I suffered the worst, and the kicker, it was plain old flu. I actually had someone tell me I am wrong here and that it must have been COVID but just a false negative. Give it up people, the cold and flu still exist!

Additionally, I’ve been moved to a senior position in another team at work. The team is at the point of complete atrophy with everyone jumping ship or taking jobs elsewhere, and unfortunately, I am the best candidate to help out. I cannot begin to get started on how badly everything has fallen apart over the last 6 months - I feel like I’m working at Parts Unlimited from the Phoenix Project. We even have our own Brent!

Needless to say, my mind has been quite occupied over the last few weeks meaning time for Mudmap has been low.

Write at least one blog post for Mudmap

Appraisal: Failed
Rating: F

For the same reasons above I have had no time to do this task either.

Recommendations

I am about two years late to the party but Ted Lasso has to be the most wholesome and feel-good show I’ve watched in a long time. Some of it really hits home for me, in the past I have been in a high performance team with high stakes. I’d deal with the team dynamics of highly talented, motivated and sometimes abrasive individuals, and it is tough sometimes. So this series made me happy, sad, reflective and proud all at the same time. The soundbite that I’ll take away from it is be curious, not judgemental. It is something I need to remind myself daily.

After 6 weeks of training I competed in my first open water swim. It was a tough two kilometers in some decent swell; 3m at 10 second intervals. But, it feels like I’ve accomplished something worth being proud of and most importantly it aligns with my true self. The points in my life that I look back upon with the most fondness where also the most painful and required a lot of commitment and dedication.

“In some ways suffering ceases to be suffering at the moment it finds a meaning, such as the meaning of a sacrifice.” - Viktor Frankl

I recommend doing something hard and painful every now and again. Get out of your chair, don’t take the umbrella and feel rain on your skin or challenge yourself by going for a Park Run. Just do something where all you can think about for a few minutes is survival - most problems aren’t that bad when you have such a contrast.

Wrap up

A short retro after a month of very little in the way of accomplishments. I did create a small Go CLI using Cobra for a popular Australian crypto exchange (still a WIP). This was mostly to enable my friend’s analysis of trade conditions but also to get more acquainted with writing Go CLI’s. This is an area where I believe Go really shines, especially with its ease of producing cross-platform binaries. Whilst I may have accomplished little on Mudmap I did create something useful for a friend - a silver lining for the month.

In March, I hope to be a bit more productive on Mudmap, or if I am still unable to produce much assess my ability to keep going with the project. This is that is weighing on my mind, especially in light of Netgate’s recent announcements. Effectively, they are looking to transition pfSense CE from development to maintenance mode. At least that is the feeling amongst the community, and whilst I do understand their desire to move in this direction, I do see that as a potential threat to Mudmap’s long term viability.

What can I do better?

Write my weekly What Got Done on Friday instead of delaying until the weekend or later

What have I done well?

I’ve done a lot to help out a friend with their own project, mostly on the deployment side
Allowed myself to take time off when needed

Next month’s goals

This month I’m setting no goal other than to help my sick partner recover. As much as it feels cheap from a product standpoint family trumps penny stocks.

Analytics

mudmap.io

check-redirects.com

danielms.site

Jan 2022 Retrospective

Tue, 01 Feb 2022 00:00:00 +0000

Summary

This month I deployed the new version of Mudmap - a complete re-write using Golang.

Highlights

Deployed version 2 of Mudmap

Goal Performance

A review of last months three goals. See Decembers’s Retrospective

Push Mudmap version 2 to production

Appraisal: Completed
Rating: A+

After checking my git history, I was surprised to see that my first commit for the new version of Mudmap was on the 5th of November. On January 10, I officially pushed v2 to production. Considering I was maintaining an existing product and rewriting it in another language - which is still relatively new to me - this feels like quite an achievement.

Did it improve performance, and what was your reasoning?

Here is a twitter thread with visuals. But the tl;dr is yes, memory consumption alone is remarkable. With my Django app, even a small load on the server could run the risk of pushing the application above the plan limits. Now, I am nowhere near it.

The difference between container image sizes are ridiculous as well. Somewhere around 15mb for the Go image and 1GB for Django. Build times feel days apart - from ~10 minutes using python to under a minute using Go. A pet peeve of mine is how slow python image building is, and I am glad to be rid of it. Another thing I’m happy with is the cost reduction in terms of the number of services needed. I save at least $14 per month by not needing redis or celery. I’m even considering removing Postgres for Litestream as I now only store basic user and device data.

There is a notion document with some of my reasons for switching from Django to Go. Of course, in almost any of those statements, a counter-point could be constructed. That is fair, but regardless, they are my reasons for such an undertaking.

Add at least one more core feature to Mudmap

Appraisal: Completed
Rating: B

Mudmap now supports three helpful but not core features; shutdown, reboot and shell command execution (see them in action). Admittedly, these are useful, but I did not deliver the bigger feature I’ve been working on; the ability to create, read, update and delete interfaces.

I did update the documentation, landing page and recorded five YouTube explainer videos. It is the first time I have recorded and uploaded anything to YouTube, and it shows in my delivery. I opted to push what I recorded rather than waste time finessing and re-recording it again and again. Reaching for pragmatism not perfection.

As I’ve transitioned (but still support version 1) to version 2, the documentation needed updating. This meant updating all the screenshots, links and explanations. I also added the videos to the docs as some people prefer videos over a wall of text.

Mudmap’s landing page also got a little better. Previously, I was using a placeholder image in the main section but replaced it with a screenshot of the dashboard. The overview video is now embedded into the page as well.

Recommendations

My First Million episode on Kevin Van Trump and FarmCon - its incredible. This guy runs a daily agricultural investing/analysis newsletter which brings in ~30M USD a year. He’s been writing it for 17 years and now runs FarmCon. I’ve been binge-watching all the videos on YouTube and cannot praise this enough - talk about finger on the pulse.

OpenFaaS for serverless functions. I’ve never been much interested in serverless, mostly because I don’t like using AWS (etc) for small workloads/one-off tasks. And, when I’ve looked at it before it didn’t really gel with me - another skill to learn. Then I found OpenFaaS - an open source serverless technology. Using OpenFaaS, I can write functions in almost any language with my two preferred options being python (Flask) and Go. It also supports NET, C#, Node, Java and more (I think). I am using it to render the analytics charts below. It supports Kubernetes and a standalone binary called faasd for deployment to a small server - I run it on a $5 droplet.

Wrap up

A good start to the year. I took the Christmas and New Year period off and also took a four-day long weekend at the end of this month. Only during the four-day break by the sea with just my little family did I start to feel rested. If I can, I will try to schedule more of these little breaks to ward off burn out.

What can I do better?

Not get as distracted by other things; OpenFaaS took some time away from Mumdap this month
I need to read more books and not look at my phone before bed

What have I done well?

Integrated Stripe and deployed Mudmap’s new version early in the month
Wrote several emails to customers, a blog post and newsletter announcing it

Next month’s goals

Deliver the Interfaces feature for Mudmap
Write at least one blog post for Mudmap

Analytics

mudmap.io

check-redirects.com

danielms.site

Tweets

December 2021 Retrospective

Sun, 02 Jan 2022 00:00:00 +0000

Summary

Mudmap is now completely rebuilt in Go (with extra features) and ready for official (re)launch in January 2022.

Highlights

Mudmap version 2 exceeds version 1 in features
New authentication backend makes life a lot simpler for me (and will allow MFA for users)
After a big year, I’ve taken a couple of weeks off and feel great

Goal Performance

A review of last months three goals. See Novembers’s Retrospective

Plug in the user authentication to Mudmap

Appraisal: Completed
Rating: A

Mudmap’s current iteration uses third party packages and my own implementation to handle the login, logout, password reset flow for the application. It took me a long time and added complexity, especially to the frontend. This complexity meant that sometimes users could be booted from a session randomly, which isn’t a great user experience. It also took a lot of my development time to create and maintain which is time better served elsewhere. Hindsight is always crystal clear, and I knew that if I had my time again, I would simply use an existing third party service instead. Enter Auth0.

Off the bat, it has been a breeze. Setting up the frontend to authenticate a user with Auth0 and then allow them to access the application was very simple. Using a React based frontend (Next.js) has its perks, namely the first class support most big orgs provide for it. Auth0 was no exception here providing excellent libraries for both Next.js and React.

Where I did hit some issues was the secure communication between the (now authenticated) frontend and my backend server. When a user gains access (logs in) via your frontend, that does not give them immediate access to anything else, they need to then request an access token. That access token must then be sent as a header with each request to the server and verified. The process is well understood and should have been easy enough, but if you want to use Auth0’s auth0-react module and axios well you’re probably going to struggle a little.

Why? The library expects you to request an access token from Auth0 via a react hook. Unfortunately, this effectively means I needed to either create an axios request (object) per component, or use fetch.

In the end I removed axios completely and replaced it with fetch. On record, I much prefer axios - fetch has pitiful error handling.

At the same time, my backend implementation also hit a problem causing a two pronged debugging session whereby I had two errors but thought there was only one. Go is a typed language and when you say you’re only sending a string but instead send a []string, you’re going to get an error. What tripped me up the most was the documentations' description of what should be getting send to the server, did not match what was actually getting sent to it. After trawling the Auth0 forums and decoding the token I found the solution and implemented it. What I love here is that it was a fix I could easily implement into a third party module. In the past, I’ve found this process much more difficult using python.

Email customers about upcoming changes to Mudmap

Appraisal: I emailed a few but not all (I actually forgot)
Rating: C

Embarrassingly, I wrote a big email to let customers know about the upcoming changes and forgot to send it before Christmas. It’s going out this week.

I did however write a blog post stating much the same things which are in the email.

I’m giving myself a C because I did write it and I also wrote a blog post about it. Cutting myself some slack because I just drank a litre of sangria with my wife at the beach tonight too.

Recommendations

If you’re into mountaineering or other sports which push the limits of human potential, you will absolutely love the movie 14 Peaks. Two things I loved about it; the team is entirely nepalese, and Nims is ex SB. Having served with some SB blokes over the years makes it feel closer to home. Also, I love to see former warfighters do big things with their lives once they’ve left the service.

Write your successes and failures throughout the year. I’ve now written a retrospective for each month of this year, and reading back over them has been hugely rewarding for me. Each week, I write about what I did over at What Got Done, too. Without doing these I really would not remember what I’ve been up to at all over the last year. Cataloging my year is the single best thing I have to beat out imposter syndrome, and also to keep me accountable.

Wrap up

It’s been a big year with lots ups and downs, wasted efforts, and successes.

What can I do better?

Remember to send emails after I’ve drafted them up!
Step away from the keyboard on those really nice days and just take the family out somewhere instead - the keyboard won’t miss me

What have I done well?

I’ve learnt a lot, and have been able to implement things now that only 12 months ago would have taken me months - keep growing and stay the course
Exercising - I’ve done a lot more this month, especially since signing up for a 2k ocean swim in Feb

Next month’s goals

Push Mudmap version 2 to production
Add at least one more core feature to Mudmap

Analytics

mudmap.io

check-redirects.com

danielms.site

November 2021 Retrospective

Tue, 07 Dec 2021 00:00:00 +0000

Summary

Screw it, I’m rebuilding Mudmap’s backend.

Highlights

80% feature parity between Mudmap version 1 and now
Decided to withdraw from MSFT interviews
Started planning new user interface enhancements

Goal Performance

A review of last months three goals. See October’s Retrospective

Mudmap feature proof-of-concept

Appraisal: A PoC turned into a rewrite, and it’s working out great
Rating: A

Firstly, I want to acknowledge two things.

Everyone says don’t rewrite, it is a time sink or that it’s chasing waterfalls
There is nothing wrong with Django - it’s great

That out of the way, let’s continue. This started as a prototype to see if I could use Go for sending SSH traffic more efficiently. It pretty quickly devolved into a bunch of HTTP handlers with SSH connections inside them.

Somewhere within the first half of the month, I started to feel a lot more productive and in control by re-implementing sections of Mudmap in Go. It was at this point that I realised it’s time to commit and just do it. Despite this, I did have a number of self-conscious thoughts about undertaking this task. But, ultimately they all boiled down to me caring about what others think which is an indicator that you’re doing, or not doing, something for the wrong reasons.

After a month of work, Mudmap V2 is about 80% feature parity with my in production V1 codebase. It also has more tests and better coverage than the current application and some components even have additional features. Something I’m really pleased with is how easy it is to add new features using Go - its type system and interfaces are great for this.

This is a personal thing and not slight on Django, but I’ve found that Go has forced me to think more critically about how the application should be built. Django’s beauty can also be its curse as there is a lot of magic. For a CRUD application (and especially one without DRF), it’s wonderful. For me, some of that magic didn’t work well for what I’m trying to achieve with Mudmap. For instance, my endpoints make calls to another API using SSH. This is not something Django expects and takes away the brevity of Class Based Views. I’m also not a huge fan of the serializers used by DRF (which may as well be a Django core module these days), whereas I am a big champion of Pydantic/FastAPI’s type system. Go feels like a step above both options but it’s a subject comparison.

In hindsight, Go feels like the better option for a couple of other reasons which are mostly related to the developer experience. It has made me think more critically about what I am trying to accomplish at every step along the way. How? Firstly, in Go, a little copying is better than a little dependency. This has made me do a lot more research where in the past I may have just pip install’d something, and it has an added benefit of making me read a lot more source code, a powerful educator. I also love seeing how changing a functions’ signature can ripple across the entire app, which has also made me plan further ahead. Lastly, Go is a concurrent language. Mudmap sends emails and executes things in background workers without Redis or Celery, and it can handle a boatload of requests. I’m still coming to grip with the fact that I don’t need Gunicorn, its dev server is its prod server, and it’s fast.

In all, I think this has been a good move and feel more confident about feature development and maintaining this codebase than I do with the current app.

Study at least 2 pomodoro’s each day

Appraisal: This or rewrite Mudmap. This lost.
Rating: C

This spawned from interviewing with Microsoft for a position within Azure. I put in a number of hours at the start of the month, working through the typical developer interview type questions. After about 10 days, I gave it away to spend more time on Mudmap but in the process did brush up on some of the basics.

I guess, this is a two-part reason as to why I stopped studying; I don’t really want that job, mostly because they want me to stay local and work in the office. That’s not something my family, or I want anymore. We’re keen to head back to the coast where we lived for ten years before coming here. Also, that crap is utterly boring and saps my energy - if it means I never work for a Big Corp, so be it.

Recommendations

We started watching The Expanse on Prime and after three seasons am totally hooked. If you like space, it’s actually really good.

I like to listen to music when working but get distracted by certain genre’s. It means I have to pick music that is rather mellow. As much as I love 90’s/2000’s hip hop, it is too stimulating for me and makes it hard to concentrate. This month I found a really chill playlist which has replaced my Interstellar and Tron soundtracks.

Wrap up

I feel like I am back this month after a sluggish August-September-October. It might be the end of lockdown, the increasingly nicer weather or just because I took some time to slow down but whatever it is I feel much more motivated.

What can I do better?

More exercise
More engagement with customers

What have I done well?

Focused on completing one task well before starting the next
Prioritised what matters and brushed off meaningless tasks

Next month’s goals

Plug in the user authentication to Mudmap
Email customers about upcoming changes to Mudmap

Analytics

mudmap.io

check-redirects.com

danielms.site

Twitter

October 2021 Retrospective

Fri, 05 Nov 2021 00:00:00 +0000

Summary

A month of learning and exploring, free from self-imposed pressure.

Highlights

Focused on learning Go
Deployed my first Go app
Interviewed with Microsoft

Goal Performance

A snippet from last month’s goal setting in the September retro

No goals, or rather I am taking the month to freestyle the next thing to do. Not setting goals is setting a goal to fail, or something like that, but I need a month to relax my mind. My wife has had some health issues, COVID garbage is constant and working from home whilst homeschooling ain’t easy. But, that should all ease off towards the end of the month, so I’m taking until then to just experiment.

Instead of reviewing and assessing my goals, I’ll cover off on what I have done.

Focused on learning Go

After several on and off attempts at learning Go over the years, I’ve actually committed to it over the last month or so. When it comes to learning new things - regardless of topic - I like to use the four stages of competence model as my frame of reference.

Stage	Description
Unconscious incompetence	The individual does not understand or know how to do something and does not necessarily recognize the deficit. They may deny the usefulness of the skill. The individual must recognize their own incompetence, and the value of the new skill, before moving on to the next stage. The length of time an individual spends in this stage depends on the strength of the stimulus to learn.
Conscious incompetence	Though the individual does not understand or know how to do something, they recognize the deficit, as well as the value of a new skill in addressing the deficit. The making of mistakes can be integral to the learning process at this stage.
Conscious competence	The individual understands or knows how to do something. However, demonstrating the skill or knowledge requires concentration. It may be broken down into steps, and there is heavy conscious involvement in executing the new skill.
Unconscious competence	The individual has had so much practice with a skill that it has become “second nature” and can be performed easily. As a result, the skill can be performed while executing another task. The individual may be able to teach it to others, depending upon how and when it was learned.

sourced from wikipedia as it much more eloquent than anything I could write.

Initially, I set about the fumbling and foolish unconscious incompetent phase. This is when I just start and know that anything I write will be bad, of poor quality and embarrassing in the future. In the past, I’ve spent way too long researching the best way and otherwise convincing myself that I could prevent this by just reading more. You can’t, just like you can’t theorise yourself into an NBA contract - you have to do the work.

When I trained new guys on their way into the special forces, you’d see all kinds of wacky stuff. The old way of teaching was to punish them for silly mistakes. Thankfully, times are changing and we (eventually) came to realise that learning is a curve. Something akin to 60% of time spent in a learning continuum is spent in this first phase, with the next stages accelerating off the back of a solid foundation. This tangent is a poor attempt to say; embrace the wacky and uncomfortable start because beginners are allowed to make mistakes.

Eventually, I started to hit some rough spots where experience taught me that I was missing some fundamental Go building blocks. Instead of searching around, I just pulled the trigger on a book by Alex Edwards called Let’s Go Further. It has been a wonderful resource which really helped to teach me how to set out my projects and make them extensible.

I think I am halfway between consciously incompetent and competent now. So I know when I’m making the right decision about 50 percent of the time now.

Deployed my first Go app

Off the back of Alex’s book, I was able to refactor my URL shortener application that I had been writing in Go. I decided to not only refactor it but also deploy it to my Caprover instance. This way I’d get the experience of deploying a dockerized Go app with a next frontend (hosted by Vercel). In addition to that, I tried out Litestream as the SQL backend and was able to replicate the database using S3 as the storage layer.

I use Github for personal stuff, such as this but professionally use Gitlab (and Gitea in the past). Github is moving with some serious speed these days but my familiarity and productivity is firmly with Gitlab. This did trip me up during the deployment where I retrospectively added a CI pipeline. I stumbled upon some issues with secrets and contributors which I ultimately solved by refactoring my code. Nonetheless, as I understand it, secret management in public repo’s can be a security issue which is why I opted to refactor rather than sort out GH actions.

The process of building and deploying my first Go app was a breeze compared to any python project. I’m still coming to terms with Go having a webserver that’s production ready after having spent a lot of time setting up uvicorn and gunicorn to serve python apps.

Check it out at https://tars.run.

Interviewed with Microsoft

During the month I was approached by a Microsoft recruiter about a couple of potential jobs they think I might be suitable for. Flattered, I took the call and ended up having a couple of first round interviews. Quickly I discovered that my technical interview skills and knowledge are not up to scratch. Despite being recommended to continue along the pipeline, I elected to postpone any further interviews. In the meantime, I’ve decided to slowly brush up and in some spots actually learn new skills.

But, I’m an Indie Hacker now right? Yes, and no. I work full-time for someone else and still need to provide for my family. Until I can swing from one branch to the next without my kids skipping dinner, I’ll need to be ready for new job opportunities. Getting slightly humbled by some easy questions really put my future employability in the spotlight.

I think this was a great wake-up call too. I’ve been ignoring the fact that knowing how solve the types of problems used during these interviews will at worst make me learn some more fundamentals and at best get me a job. I’ve nothing to lose by studying a little each day.

Recommendations

I am seriously late to the party but I’ve started watching some Twitch streamers. Here are a couple I enjoy listening to:

rwxrob for his Linux, Go and k8s rants
Jordan Lewis for his cockroach labs Go programming
Anthony Sottile because he writes python open source and sounds like a real person when streaming

Wrap up

This month was a great unwind and refresh from working on Mudmap. It gave me the intellectual and emotional space to take stock of a few things.

Next month’s goals

Mudmap feature proof-of-concept
Study at least 2 pomo’s each day

Analytics

mudmap.io

check-redirects.com

danielms.site

Twitter

I really haven’t been playing the Twitter game much lately. Struggle thinking of things to say!

September 2021 Retrospective

Mon, 04 Oct 2021 00:00:00 +0000

Summary

Finally, I’ve released my firewall feature to Mudmap. It’s a small step towards having a more fully featured application for pfSense users.

Highlights

Deployed a big feature to prod
Took on a lot of learning for my day job - cutting into Mudmap’s time

Goal Performance

A review of last months three goals. See August’s Retrospective

Release firewall rules page

Appraisal: I’ve done it, finally!
Rating: A

After what feels like a lifetime I’ve pushed a big feature to Mudmap. Even saying that makes me shudder, it’s a bad thing - features shouldn’t take so long to deploy. Regardless, I am happy its out and glad to move on to the next thing. I keep asking myself why it took so long. I think it comes down to two main things; time and complexity.

Time

It is just me working on Mudmap, and I already work a full-time job. On one hand, it’s amazing what you can accomplish over time as your small increments compound. On the other, if it was my full-time job those increments would compound much faster. Acknowledging this fact is important but it’s only half the battle. I also spend more time than I should on things which do not add value to the project. Battling with tooling issues or lack of foresight for issues that would cost me precious time later, are especially damaging. As I won’t be quitting full-time employment any time soon, time or the lack thereof will be a continued factor in the velocity of release.

complexity

I have definitely made things more complex than they need be in certain aspects. In hindsight, it is obvious but only now do I possess such clarity. Further, pfSense is a complex beast in itself and building an API plus management dashboard for it isn’t trivial. I cannot decrease the system complexity of pfSense but I can reduce my own systems. It starts with more time allotted to design and proof of concept work instead of ploughing headfirst into things I think can work.

Write a Mudmap blog post

Appraisal: I have it written just not published
Rating: C

I can deploy what I have written to prod, but it feels half-baked or even nebulous in hindsight. I’ve decided to sit on it and maybe leave that post as a forever draft instead. As a result, this is unfinished.

Recommendations

After years of pestering by Amazon, I finally trialled Prime. Before I mention the show I really enjoyed I want to say two things about Prime Video which suck.

Paying for movies and shows (in addition to the yearly subscription)
Search absolutely sucks

When I searched for a show that my daughter was begging me to watch, all I could remember is that it had “princess” in it. So I searched for “princess”. No results. Tears ensued. A week later, my wife found it - “princess and the dragon”. Now what crappy search engine cannot return “princess and the dragon” when searching “princess”.

Onto my recommendation. Clarksons Farm is hilarious and well worth a watch. I’d also recommend watching all episodes from season four of The Grand Tour - I haven’t laughed like that in a while.

Wrap up

The first half of September was me working really hard to push out the firewall feature with the second half mostly devoted to things unrelated to it. My job - the one that pays my bills - is still important to me and must take priority when needed. We’re moving to a new platform, have several big features to produce and need to upskill all at the same time. For me personally, as soon as I deliver my current scope of work, I could be seconded to another team to aid their work on the container platform. So this has required me to commit some of my in and out-of-hours work to Kubernetes education. In between that, time has also been spent increasing my proficiency in Go for potential job opportunities that (have) and may come up in the future.

What can I do better?

Plan and prove that features or ideas are going work/fail before jumping in. Patience

What have I done well?

Deployed without issue a long awaited feature

Next month’s goals

No goals, or rather I am taking the month to freestyle the next thing to do. Not setting goals is setting a goal to fail, or something like that, but I need a month to relax my mind. My wife has had some health issues, COVID garbage is constant and working from home whilst homeschooling ain’t easy. But, that should all ease off towards the end of the month, so I’m taking until then to just experiment.

Analytics

mudmap.io

check-redirects.com

danielms.site

Twitter

I really haven’t been playing the Twitter game much lately. Struggle thinking of things to say!

August 2021 Retrospective

Wed, 01 Sep 2021 00:00:00 +0000

Summary

Productivity took a nosedive this month, as did my own mental health. Thankfully it’s rebounding and I’m mostly back on my own schedule and feeling good.

Highlights

Setup my own Sentry server
Enabled Upptime monitoring for Mudmap
Took a week off to learn some Golang

Goal Performance

A review of last months goals. See July’s Retrospective

Release firewall rules page

Appraisal: Did not complete this
Rating: C (it’s looking great though…)

This page is really starting to take shape and works well but is still a work in progress. The amount of work in creating nice forms was underestimated.

An example of what the rules page overview will look like. The styling and fields for the table are still in flux but the framework is there. The nice thing about building it in javascript is how snappy it feels when jumping between interfaces.

The edit rule form is also coming along. The slowness has come mostly from me figuring out the conditionals and react-hook-form. For instance, if the source is any then no source address or CIDR ranges should be applied. However, a single host source address requires a source address but not a CIDR range, and a source network requires both an IP source address and CIDR. The form is littered with those sorts of distinctions and its sometimes a bit of grind working through those and other edge cases - ICMP, I’m looking at you.

Write a Mudmap blog post

Appraisal: Did not publish this
Rating: D

The post is pretty big but not finished yet. I could release it over a few parts but would prefer to make it a long-form post. Still a bit to do and finding time for it was hard this month.

A rough patch

No denying it, this month was tough on me mentally. A number of things that have been
background noise over the last year suddenly came to the forefront of my mind. It brought my productivity to halt and put me in a bad place. This can happen to anyone for any reason in their life, and as a solo developer, there isn’t anyone else to pick up the slack.

The two biggest pieces that affected me were:

Australia’s dip into oppression over Covid
Afghanistan and the treatment of men like me

Lockdown

The first couple of weeks of being reduced to one hour of outside activity hit me harder than expected. Sounds easy at first but having young children and a relatively small house means there is no escape from the chaos (and joy) that kids bring. On paper, it seems fine, just a routine; wake up, walk to the coffee shop as a family, walk home, find things to entertain a child all day without going outside or anywhere. Except both parents also have full-time work. After the second week we have settled into a bit of a routine - it’s not perfect, but it’s working. Still, it’s not great for my girl who loves her friends and is very active - kids and adolescents are paying a high price, so much that child suicide is now being normalised.

Afghanistan

If you’ve turned on the news (hopefully you didn’t watch it for too long! mind-rot that stuff) you’ve likely seen the Afghan coverage. I did my service in the country and am proud of it - perhaps the best times of my life were there. For a professional soldier, fighting is something you want to do, not something you endure. The hardest pill to swallow with the withdrawal has been the number of conversations with people who are overnight geopolitical experts. Asking me things like “how does it feel to know it was all a waste of time” or “would you say that it was worth it?”. These questions betray the asker’s naivety about the nature of combat and soldiers or dare I say, warrior culture.

The fall of Afghanistan was not if but when. Not only that but it is what the media and public have wanted since the late 2000s. Let them stand up on their own two feet and hand over power - Hamid Karzai was inserted with the Green Berets in 2001 to start the removal and handover of power. It was not unexpected yet the media went to town on the predicament, blissfully ignorant to the many factors at play. As long as they get some clicks and outrage - the end justifies the means, right?

There is more to this story but that is all I am comfortable sharing today. At the end of the day, a few things compounded over a couple of weeks that had a real effect on me, and it shows up in this retrospective and my performance.

Recommendations

A couple choice things I’ve discovered this month.

Fantastic fungi

A really interesting documentary - easy watching yet intriguing. Perhaps the most interesting to me was the fusion of mushroom spore with termite insecticide. Maybe I need to increase my variety of mushrooms instead of just the woolies button mushroom!

Dune

A book that everyone has heard of. After reading The Three Body Problem I found that reading science fiction before bed was really relaxing and helped me to unwind. So, I deceided to tick off a few of the must read before you die type sci-fi titles. I’m about half way through the first book and look forward to reading it every night. I’m unlikely to watch the movie as every movie adaptation of a good book has been terrible, Ready Player One - I want my money back please. But, I might have watched it if I hadn’t read it first and am glad I finally got around to it.

Wrap up

A bit of write off a month. Hopefully, this month is better!

What can I do better?

Be more grateful; get want you want and want what you get.

What have I done well?

I rebounded quite well at the backend of this month by letting go and venting with some friends over issues that plagued me. This release was more beneficial than I expected in hindsight.

Next months goals

Repeating this month’s goals.

Release firewall rules page
Write a Mudmap blog post

Analytics

mudmap.io

check-redirects.com

danielms.site

Twitter

July 2021 Retrospective

Wed, 04 Aug 2021 00:00:00 +0000

Summary

This month really got away from me. Mudmap’s development has not slowed, but it has not been hitting the goals I’ve set this month. Instead, I’ve adapted to a changing landscape and user needs.

Highlights

Mudmap gets a free tier
Static IP addresses are now standard for Mudmap’s servers
A shift in focus and support for pfSense versions

Goal Performance

A review of last months three goals. See June’s Retrospective

Cold email at least 15 people

Appraisal: Probably set the bar too low here; only a few replies.
Rating: A

I did this, and got probably the market expected percentage of responses. All the responses mentioned a need for locking down their SSH port with some static IP addresses.

It was from this feedback that I chose to prioritise the deployment of a proxy in front of Mudmap. Setting the proxy allowed for a pair of static IP’s to be placed in front of Mudmap. The proxy is completely transparent to users even with SSH tunnelling. By doing this, users can now set source addresses for SSH - a huge security increase.

Some might wonder why my servers don’t have a static IP already? The hosting platform, Render.com, does not provide static addresses for any of its containers. A third party service, QuotaGuard integrates well with Render and hooking it into my application took about 2 hours. I am pretty happy with the service and whilst they don’t offer a free tier, I feel the starter plan is quite generous.

Add firewall rules to Mudmap

Appraisal: C
Rating: I have built the read-only pages but paused development due to important issues that arose.

So far, Mudmap’s feature set is limited to a only a portion of pfSense’s. The vision is grand and will eventually cover the majority of its features. Firewall rules, including the ability to read, create, update and delete them is a high priority. Unfortunately, other issues came up that forced this into the paused, or blocked state. I could deploy it as a read-only copy but have chosen to instead deploy it once its finished properly.

A work in progress look.

Record some videos for onboarding new clients

Appraisal: Did not start this.
Rating: F

I did not even get started on this.

I did create a landing page for new users inside the application itself. Initially, when a user register an account and logged in they were presented an empty table. That’s not very welcoming and makes the assumption that users know how to use Mudmap. Now they get presented with a few helpful links to get started - this will eventually be upgraded again, likely with a welcome video.

The welcome page. Unfortunately, I didn’t get screenshot of the empty tables as a comparison.

Free Tier

Mudmap is new, lacks social proof and is chasing a market full of security minded folks. Initially, to get started using the platform I was asking for a subscription. But, this was, as you would expect, not working too well. Before someone could even evaluate the performance, reliability and security of Mudmap they were asked for a payment method. I wanted to reduce this friction and increase user uptake.

So I decided to offer a free tier for all users.

Every user can now add their first two devices for free. No credit card needed and no time limit. It also lets small businesses or hobbyists who have a couple of devices test it out.

In hindsight this should have been released from the start. I’m hoping this will bring in some more feedback of the system too.

This month I spent some time upping Mudmap’s social media presence. You can now find it at:

It’s always that fine line between development of the product and marketing. The developer in me always thinks it is not good enough to spruik yet. This, I feel, is a problem all developers have.

A problem and shift in Mudmap’s offering

Late this month, I started to see a uptick of users and a correlating spike of errors. The investigation into this lead me to a resolution I always felt was probable but hopefully unlikely.

tl;dr Mudmap is ceasing support for pfSense+

At the start of this year, Netgate elected to split pfSense into a closed and open-source model. The newer and closed source pfSense+ is for the time being largely the same as pfSense Community Edition (CE). Unfortunately for Mudmap (yet, fortunate for everyone else) they’ve made changes that have effected the product.

Initially, I attempted to reverse engineer a workaround to the most recent breaking change between pfSense+ and Mudmap. But after careful consideration, I decided this course of action is not in the best interested of customers.

Why? Providing a reliable and safe platform from which customers can manage multiple pfSense devices remotely is Mudmap’s core mission statement. This cannot be achieved without a stable platform to build from. Providing an interface that could potentially cause disruption or worse for customers is a risk I will not take.

Naturally, I am disappointed, and it will reduce Mudmap’s market appeal - possibly quite significantly - but I’m not losing hope.

So where to from here?

Mudmap will continue to support pfSense CE and develop functionality to meet the needs of that user base. I’ll also be keeping a close eye on the developments of pfSense+. I think it is important to disclose how excited I am for it and that I fully support Netgate’s decision to close source the project. I expect pfSense+ to be a cut above the rest when it re-launches after its rewrite in Golang. This should also deploy with an API very similar to TNSR’s. As both a user and developer, this makes me really happy. It should also allow Mudmap to push back into the market of supporting pfSense+ clients as I will (hopefully) be able to hook into the new API.

I have written a page in the documentation explaining the change in support, updated the homepage and made a toggle in the application itself ensuring that users are aware of this. I will be writing a blog post, newsletter and LinkedIn post in response as well.

Recommendations

Interesting topics

Whilst I am not a pentester or bug bounty hunter, I thoroughly enjoy watching videos on the subjects. Having binge watched most of ippsec’s videos, I recently started on John Hammond’s stuff. If you’re a developer, there is a lot of information to glean from how these guys go about exploiting your servers.

Worth a mention, ippsec has a handy search page where you can filter topics and find which video to watch based on it. For instance, if you searched jwt, it will show you a selection of boxes he’s pwned with JWT’s in there somewhere.

React development cheat code

If you aren’t using swr from Vercel, maybe you should be! react-query is apparently just as good - and I believe it, it’s from Tanner Linsley. I can only talk about swr and say that its made my Next.js app blazing fast - it literally feels like a cheat code for client side web. It integrates perfectly with axios too. As a heavy user of axios interceptors for JWT refreshing, I initially saw all the docs using fetch and thought maybe interceptors wouldn’t work. Thankfully it worked flawlessly.

Podcast of the month:

Everything on the How to Take Over The World Podcast. My favourite is the Putin series and Caesar.

Wrap up

This month was a bit of test. It started quite well but sadly ended on a sour note. Nonetheless, I’m looking forward to August and producing more content for users of Mudmap to enjoy.

What can I do better?

Market Mudmap’s potential rather than internally focus on what it’s not providing right now

What have I done well?

Adapted to changing circumstances and took action when needed.

Next months goals

Actually release the firewall rules pages
Publish a Mudmap blog post

Analytics

mudmap.io

check-redirects.com

danielms.site

Twitter

June 2021 Retrospective

Fri, 02 Jul 2021 00:00:00 +0000

Summary

I spent less time on Mudmap this month and relaunched an old project, Check-Redirects.com. Taking a breather from Mudmap and switching gears worked well. I feel much more motivated about the project and in the last few days have done a lot of solid work on it. Nothing comes for free and the price I paid was failing to fulfill one of my goals from last month.

Highlights

Dropped a new product.
Next.js and I are becoming good friends.
Mudmap had a small spike of interest at the beginning of the month but it petered out.

Goal Performance

A review of last month’s three goals. See May’s Retrospective

Add firewall rules to Mudmap

Appraisal: Time got away from me and this didn’t get done
Rating: D

I did spend a couple of hours writing out some ideas of how this will get done. They were rough designs and scribbles so nothing of real substance. I’m not even sure I know where the sketches would be. Nonetheless, I do have an intuition of how it’s going to look and feel for the user.

The amount of work needed to make this isn’t much and would offer another great feature for Mudmap. Prioritizing this for July.

Email 5 prospects

Appraisal: Emailed with some minor correspondence but mostly crickets
Rating: B

Several users have signed up for access to Mudmap though none have committed to paying for the service. I emailed every single sign-up with extra details and asking for feedback if they feel so inclined. It is a very small sample size but so far only one replied with a feature request.

The feature is not unwarranted and something I will eventually offer all customers as standard; static IPs for SSH targets. Unfortunately, my hosting platform, render.com, does not offer this, yet. The workaround is to place a load balancer in front of my server and point users at that. This option isn’t free, add latency, and I have not tested the effectiveness of SSH communications through it. More customer data is needed to ascertain how high of a priority it is before I drop other things to complete this.

Although I’ve sent out a few emails those emails were to my warm network - anyone who has signed up has already found Mudmap. I need to go after a cold network and start planting Mudmap in people’s minds. So far, my lack of conversion from sign up to paid is because I’m not reaching out to more people.

Reinvigorate an old project that I plan to monetize

Appraisal: Successfully re-launched check-redirects.com
Rating: A

I’ve come to terms with it. I like building products around network analysis and security. So after chatting with @Ruurtjan and @andreas_zettl about our love for these tools, I decided to take a breather from Mudmap and refactor check-redirects.com. We have a discord for network tool builders - DM or email me for the link.

What is it?

It is a tool that lets you plug in a URL and quickly get the final URL without having to traverse all the redirects. In doing that, you can analyze each hop along the path including the location data, headers, response body, and latency.

A few use cases:

Checking a shortened links legitimacy
Bypassing ad networks
Circumventing network security policies that prevent certain URL’s from being opened, for instance, trackers from newsletter sign up pages and the like
Ascertaining how many redirects you’re introducing into your application (Google cares about too many 301’s for instance)
Determining referral links, and if they’re working as expected

The original site was built using Vuejs, though this time I opted to use Next.js as I am now more familiar with React. I figured this was a good opportunity to build another application with it - I build Mudmap’s landing page with it last month. This app needed to interface with an API so it felt like a natural stair-step into bigger apps in the future.

Honestly, so glad I used Next. It’s just a great platform and after about 6 weeks of playing around with it, I am convinced it’s better than plain React. I grok the file-based router whereas react-router-dom feels weird to me and got me with a couple of foot guns when writing Mudmap’s dashboard. As a solo developer, I value good Developer Experience and Next has a great one. They also released Next11 whilst writing it. I upgraded from version 10 to 11 and got their awesome Image updates without a hiccup, which for a javascript framework is often not the case.

Why divert time from Mudmap?

I want to place multiple bets and explore many different avenues of income generation. I am still employed full-time so these projects are not my sole source of income, which gives me the space to try things and spread myself out a bit. I tend to get a little task fixated which can cause issues so switching to another objective for a short period lets my motivation banks recharge for the thing that’s not being worked on.

Having two different projects on the go is also a means to explore two different models; subscription and affiliate/ad streams. Mudmap is a typical SaaS subscription-based application but check-redirects.com is not, letting me branch out creatively. This fits better with my self-image as I believe I am a generalist - not excelling at one particular thing but being familiar with many different modalities.

This is something I embraced (but also at times disliked) about my old job in the army. We were across so many tasks but at any one moment not particularly great at any of them. You can’t be an expert diver, driver, sniper, demolitionist, parachutist, or whatever all at the same time. You have to re-train the skills when needed. The time between average to good decreases the more you practice this style of rapid upskilling. I am a big believer in T-shaped skills but I also know that the leg of the T is in constant flux. At least for those that actively adopt this approach to life.

Recommendations

I like listening to podcasts and audiobooks on my commute to the office.

This months book:

Company of One

Worth a listen if you’re aren’t looking to scale into a massive business. I couldn’t do it justice in a short paragraph but so far its an enjoyable listen.

Sidenote: I want to listen to Phil Jacksons’ book, Eleven Rings on Audible. Unfortunately, it seems to be unavailable to Australian’s which kind of pisses me off.

Podcast of the month:

Confidence by My First Million podcast

10 minutes or so by @theSamParr on how he became so confident. Spoiler, it was a choice - a learned behavior. I loved how he explained it and have been telling a lot of friends to give it a listen.

Also, my mate just dropped a couple of clips on youtube from his holiday to New Zealand before our recent (silly) lockdowns here in Australia. He’s got a quirky style but makes me laugh and sometimes cringe. PeskyPas if you want to watch.

Wrap up

I can’t believe its July already 🙃

What can I do better?

Get out of my developer comfort zone and engage with potential customers

If I don’t do this my product will die. That’s it.

Stick to the month’s goals

I can get carried away and miss the objective of the month as I’ve set it out here. To fix that, I’m just writing them down and sticking them to a post-it note on my monitor.

What have I done well?

Re-launched an old product and recharged the Mudmap motivation banks
I’ve been pretty good at listening to my body and mind this month.

When I’ve needed extra rest, I’ve taken it. Also increased my exercise and committed to a gym by using money as a psychological weapon - I’m a tight arse and hate parting with money. So, knowing how much each day I don’t train costs me is a great motivator.

Analytics

Twitter

I put this here for my future reference.

mudmap.io

check-redirects.com

danielms.site

Next months goals

Cold email at least 15 people
Add Firewall rules to Mudmap
Record some videos for onboarding new clients

May 2021 Retrospective

Sun, 06 Jun 2021 00:00:00 +0000

Summary

A month of ups and downs, at least mentally and emotionally. At the start of May, I was days from launching Mudmap. That turned into a mental hell hole as I encountered several bugs that weren’t apparent until in production - bad programmer, bad! Deployment woes with providers and other frustrations. After a lot of persistence and some back-to-the-drawing-board thinking, I eventually deployed the app. The month ended on a bit of high with some things falling into place outside of Mudmap which I might talk about in the next Retro.

Highlights

Deployed Mudmap Beta into the wild
Discovered Render and am now a convert
Stood up the docs using Docusaurus and started flirting with Next.js

Goal Performance

A review of last months three goals. See April’s Retrospective

Launch Mudmap

Appraisal: It’s live but still got a few kinks to iron out
Rating: B

I think this has been on my list for the last two months. Perhaps, I was a little hasty or overconfident in the beginning, but I’m happy to say its now live. There are still some issues to sort out but users can sign up and look around.

A few issues presented during the deployment, which lead to a lot of trial and error testing. My original plan of a digital ocean backend and netlify frontend proved painful. This could be equated to my lack of skill in this area though. Eventually, I tried out Render after several recommendations.

Almost immediately things improved. Having the ability to push new builds and see the output is super helpful. My old method was to build on the VM (using a CI Runner) after tests passed. It works, but it’s a lot of setup, and it was this initial setup that was punishing. Cross-origin requests were also beating me up as well.

Eventually with Render I was able to stand up all the services and static pages, including domains inside a single yaml file. Render’s Infrastructure as Code capability is really sweet and should help to prevent redeploys missing some toggle that was clicked in a GUI and long since forgotten about.

As always, if interested, check it out Mudmap here

Start the documentation pages

Appraisal: Live and being indexed by Algolia
Rating: A

Before the end of the month I was able to stand up Mudmap’s documentation pages using Docusaurus. I highly recommend the framework, it’s the easiest tool I have used by far for this type of thing. The setup took about an hour or maybe two (optimistic memory aside) which for a javascript framework is really good! Writing the detailed documentation with images, much longer.

Mudmap’s documentation isn’t complete, but the most important pages are. I am making an effort to write a portion of a page every day as I feel Mudmap needs good docs. It’s a complex system with several moving parts for users to configure before they get started using the application.

You can read more here to see what I mean.

It is also hosted on Render as a static site for free. 💰

Check them out at docs.mudmap.io

If looking for a knowledge base tool that you can be hosted, I would consider Docusaurus one of the best. Really simple but elegant and has support for search through either self-hosted or managed Algolia.

I think it looks great 👇

Replace the pricing page, and if possible, embed it in Hugo as a React component

Appraisal: I’ll go one better! Replace the landing page
Rating: C+

I ended up looking at rewriting the entire landing page rather than trying to fit React into Hugo. Now, I do love Hugo - this is a Hugo blog though most of my work is between python and javascript. So I wanted to decrease any possible mental context switching between languages and frameworks. It’s also a great opportunity to make the site more aesthetically pleasing.

Technically, I didn’t complete this goal. I did find another way to achieve a better result, so I’m being favourable to myself in the rating.

End state, love Hugo but want to unify the language stack I am using, so I’ll be switching to React/Next.js

Recommendations

If you’re on Discord and looking for a great Maker/Indie Hacker like community, I recommend Lunadio. I recently participated in one their weekly Mastermind calls and it was really interesting and great to hear others in a similar position as me. The Discord channel is also full of interesting people solving problems and sharing what they’ve learnt with everyone else. Admittedly, I could be more active on there but even in a passive capacity (which I don’t recommend; get involved 💪) its super useful.

Twitter

I’ve really started to get involved in sharing my journey on Twitter. It’s been really helpful for my personal development - you meet and talk to soo many great folks. Several times I’ve reached out and asked questions of those around me, and it has ended up being more valuable to me than I’d ever expected.

My “stats” are down in everything but followers, though I don’t really care because I’m having fun and making friends.

May saw me join in a Twitter experiment created by @MeetKevon called #30DaysinPublic. It’s been great. Every second day Kevon sends an email with a Twitter task, which is to write a themed tweet. Having a task and then having to think of things to tweet about that has been great for my ability to think of sharable items. Usually, I think everything I do isn’t worth sharing - its probably boring, or been done many times over. Being held to account and committing to a cause has really helped overcome that - who cares if what I tweet is boring. If people don’t want to read it, or don’t like me that’s fine and welcomed, after all, getting people to like you is not your task.

Wrap up

What can I do better?

Project/Task Management

I am bit all over the shop with this. I have Notion but its a bit heavy and clicky for me sometimes. I’m a Vim guy, mice scare me.

Kill off distractions.

I’ve read Cal Newport’s books - an aside, I think So Good They Can’t Ignore You is far superior to Deep Work. Distractions are ever present, but I really need to switch off that time-sink/mind-rot machine called a phone when I actually want to get work done. Days where I replace the phone in my pocket with a notebook and pen are far more productive.

Getting real off-topic. I had an idea a few months back about a product which is basically a box you put all the families phones in. It has a timer and once set, locks the box until that time (or an emergency override is hit). If I could be arsed, I make it and call it Family Time.

What have I done well?

I pushed through and got Mudmap into the wild
Segmented my time quite well between development, writing docs, and planning the next items

Google ranking

Nearly forgot but checking my Plausible stats, I noticed there is some google search terms that pop up frequently. I did a search using a VPN and in incognito mode (just to be sure) and Mudmap is Second!

Next month’s goals

Add a Rule set feature to Mudmap
Email 5 prospects
Reinvigorate an old project that I plan to monetize (long term SEO play now for reward later)

Analytics

mudmap.io

check-redirects.com

danielms.site

April 2021 Retrospective

Sun, 02 May 2021 00:00:00 +0000

Summary

Wrote a lot of Mudmap’s new frontend, integrated it with the backend, and published another blog post for the site. I did not get all that I wanted to be done within the month, though.

Made the transition from using Stripe to Paddle after a wonderful exchange of Tweets which led to a phone call with Paddle. They were incredibly helpful and offered some points of refinement for the business. It cemented my commitment to Paddle as I was treated like a valuable addition to their cause, instead of a faceless and numbered money-generating object.

Highlights

Integrated the new dashboard
Published a post and sent out a newsletter for Mudmap
Finished the MVP feature-set for Mudmap minus the payment system

Goal Performance

1. Deploy Mudmap’s application for beta testing

Appraisal: Did not complete all the tasks necessary to do this 😞
Rating: D

Why a D if I failed to launch it? Seems pretty binary.

I accomplished a lot during this month and although the goal wasn’t completed, I am reluctant to call it a failure. Having a time box around this really helped to keep me focused, made it easy to cut distractions, and in general, pushed me hard to achieve it. That is not a failure in my eyes.

So what did I get done?

Added structlog for better logging management (also have Sentry)
Improved the SSH Client error and timeout handling
Reworked the device registration flow in the front and back end
Built the React frontend sign in/out, registration, etc from scratch
Integrated dj-rest-auth after wasting time rolling my own cookie auth
- This took a lot of time out of the month but it gave me so much for free that it was worth it
Replaced fetch with axios on the frontend to make JWT’s so much easier to manage
Began the Paddle integration

More importantly, what didn’t I get done that prevented this from being a success (i.e. launching the MVP on time):

Payment integration
Underlying API tweaks (namely around the installer redundancy)
Deployment testing (I see some initial difficulty getting Traefik and netlify to play nice)

Will I knock it out this month? I really do hope so!

Demonstrating functionality

Here is a short clip of the dashboard’s device detail in action. It’s a bit neater now, but the rough functionality is the same. It will at a glance give you up-to-date information about the pfSense system, as well as let you start, stop or restart system services.

2. Write a blog post about pfSense

Appraisal: Done!
Rating: A

What is pfSense?

A pretty easy goal but important. If I did not set this, I would not have done any content marketing for Mudmap because I spend all my time down in weeds writing code instead.

I like writing, but it is definitely not something that I am gifted at. It is very humbling when you think you know a subject but struggle to write about it in a coherent and concise manner. Though, that is why you have to keep producing and perfecting it as a craft.

3. Increase Mudmap’s marketing

Appraisal: Average to poor - didn’t prioritise this as I should have
Rating: C

I wrote a blog post, publicised that on Twitter, and pushed out a newsletter to my audience of zero! Can only go up from here. Now as I reflect, I see that focusing so hard on knocking out code, so I can launch the MVP is also a bit of a trap. The more marketing I do, the more awareness, the more feedback and, the more validation I will get.

On the topic of emails, I did find a great email template generator; mjml. I have not used it for any of my emails just yet, but it looks brilliant. Trying to style an email (in my opinion) is a complete tyre-fire.

I am, however, working on my Twitter game. It’s been a lot of fun so far. Trying to write short 280 character messages that convey meaning is a lot harder than I originally thought. Having to word things carefully has been great for my communication skills - especially as our society’s attention spans continue to rapidly decrease. It serves as my primary marketing platform for Mudmap at the moment as well - something that will need to be adjusted as time goes on. This makes it an important skill to refine and work on which is why I am investing in it.

Recommendations

I listened to Lex Friedman’s podcast; 176 Robert Breedlove. Even if you aren’t into bitcoin, it is a wonderful podcast with some interesting views on the current state of the financial world. I highly recommend it, if you’re only slightly interested in finance.

Not a recommendation. A sad note. I am getting older! I am now working with a lot of people that have never seen Home Alone or Jurassic Park. Thankfully, the guy next to me is my age and we can reminisce about the glory days of movies 🤣

I’ve nearly finished The Three Body Problem. It’s a great book, even if most of the physics is above my pay grade (it is still understandable). I am already looking forward to the next book in the trilogy. I usually read non-fiction but have opted for fiction before bed these days as I need to unwind.

This month I learned of litestream. I think it’s a project that has real potential - not every application needs Postgres or some other server-based SQL solution. For a great introduction, see this Tweet.

Wrap up

What can I do better?

Marketing for Mudmap,
Documentation - I am going to have a documentation fest once Mudmap launches, but it would be much easier to have done parts of it as I developed instead.

What have I done well?

I’ve knuckled down and accomplished a lot this month. So much that it wasn’t until I started this retro and looked back that I realised how much I have done,
Went from having never used React to implementing a fully functional application with authentication.

Next months goals

Launch Mudmap
Start the documentation pages
Replace the pricing page, and if possible, embed it in Hugo as a React component

Analytics

mudmap.io

check-redirects.com

danielms.site

March 2021 Retrospective

Fri, 02 Apr 2021 00:00:00 +0000

Summary

Launched Mudmap’s landing page and blog! So far it’s been getting very minor traffic but that is okay - it is still in development. I did however get an email from a potential client asking to join in a beta in the meantime. That sort of feedback really fuels the fire and has me working hard to get the MVP out the door as fast as reasonably possible.

Highlights

Deployed Mudmap’s landing page!
Started the implementation of a React frontend for the application.

Goal Performance

Each month I write up how I went in the pursuit of last month’s goals. It is also somewhere I can expand on the things I’ve done. Helpful for historical review and analysis.

Finish the copywriting on mudmap’s landing page

Appraisal: Deployed the entire landing page!
Rating: A+

The copy is not perfect, and I will need to refine it continually, but it is out there and getting read by potential customers.

I had originally written all the copy in Django templates, and it seemed to take a lot longer, and be more prone to ugliness than I was happy with. Most of this is due to my lack of design skills. Nonetheless, I started a quick google for some Hugo themes and found a great one that suited my style.

Within 48 hours, I had cloned the theme repo, created my own website repo and deployed it to Netlify. It is now being indexed by google and something I can work with to produce content that might attract more users in the future.

I listened to a podcast that featured Pierre De Wulf of ScrapingBee.com and was inspired by his experience running a SaaS business that failed. He was able to sell it because it was generating a lot of traffic. Pierre, said that their content marketing was what made it sellable.

It made me realize that everything that holds value is an asset, and valuable things can be created anywhere and from the seemingly inconsequential. That’s why I launched the site prematurely, it’s a way to start raising awareness, keep me motivated and perfect my craft while no one really cares.

Integrate Stripe as the payment processor

Appraisal: Abandoned it for now as I am ripping out templates and replacing them with a React and Django SPA for the dashboard
Rating: F

I spent a lot of time on this, and it was one of those nail in the coffin moments for the template system. I was getting frustrated with all the inline javascript in Mudmap already, and it was made worse with Stripe. Not Stripe’s fault, just the way it has to be.

So, in a way this F score has actually been a blessing - part of the reason I never look back on things with regret, only as a learning opportunity. This helped set me down the path of React for the dashboard, and once I knocked out a quick Proof-of-Concept I knew that it was the way forward for the sort of user experience I need to offer my future users.

Stripe seems to have a superb React package as well which will hopefully made the integration (hopefully this month - April) much easier.

Write a blog post for danielms and Mudmap

Appraisal: I wrote Mudmap’s first blog post but did not write one for this blog. I have been prioritising Mudmap and with so little time between that, family and my day job, something has to give.
Rating: B

I like writing. But, I also like making things, and the fire under my feet to deliver Mudmap is so strong right now. The only things I will not budge on are my monthly Retro’s and weekly WhatGotDone’s.

However, I did produce a blog for Mudmap. It explains the rough genesis and intent of Mudmap and is a bit of taste of blog deployment for me too.

In the future, blog post’s need to follow a deployment flow. The flow might look something like:

Write and deploy to the site
Send out a newsletter to subscribers to let them know
Send a Twitter post from the Mudmap Twitter account

At the moment it is all very manual which will lead to problems in the future. How I automate this, I am not sure just yet but possibly via a GitHub action workflow.

Recommendations

I found KneesOverToesGuy the other day. His story and achievements in the field of human performance are both contrarian and extraordinary. Worth a look if you want to see a cool story.

A show worth watching is the documentary, Sherpa. It’s currently on Netflix, at least in Australia (I won’t drop a link because it’ll be gone or different for you, no doubt). It highlights the difficulties and conditions the Sherpa face during Everest expeditions. Personally, the whole everest scene has become a shambles - pictures of people attempting to summit with the helmets on backwards and climbers only learning what a jumar is at basecamp indicate its now just a tourism operation.

Also, if you’re an indie hacker or thinking about starting an online business, check out Arvid Khal’s book Zero to Sold. I’m listening to the audible version and have found it to be a fantastic resource.

Wrap up

What can I do better?

Research topics before getting straight into coding, for example, I was going to integrate Auth0 when I already have 90% of session authentication already handled
Set Twitter boundaries, and use that scroll time for development of ideas instead

What have I done well?

Deployed the landing page (without fear of what people think)
Be open about my past work history online - something for years I have hidden away
Managed my time between Mudmap and my family

Side Notes

Last month, I assessed a goal from January - to gain five Twitter followers. Well this month, rather unexpectedly, I grew that quite significantly.

I think my follower count went from 30 to 89 with engagements and impressions skyrocketing. I attribute this to one thing; engaging with others. In the past I have treated Twitter like I have treated my general life, that is to say, avoiding people.

Mostly, I have avoided talking about myself, or things I have done yet hoping people would just notice what I have done. This just isn’t how the world works, not on Twitter, not in life, not anywhere. Two things come to mind here:

Do you need to be disagreeable to get by in the world, not really but you definitely need to stand from something and have opinions. My main point here, is that silently consuming and never producing, and never engaging is not an effective strategy.

So, that is why I am committing more of my mental reserves to opening myself up to world. This also means breaking a conditioned fear drummed into me from my years in the special forces; going public about who you are, and what you’ve done or are doing.

In my past life, nearly ten years, I ran a cover story for meeting people. I played in a Hockey team for 5 years and none of them knew what I really did. My parents didn’t really know. It’s just the culture of that system - working in the shadows. It has its place and is a necessity but eventually you leave that world and need to remove those protective walls. For me, it starts with Twitter and Mudmap.

Next months goals

Deploy Mudmap’s application for beta testing
Write a blog post about pfSense
Increase Mudmap’s marketing

Analytics

mudmap.io

danielms.site

Feb 2021 Retrospective

Sun, 28 Feb 2021 00:00:00 +0000

Summary

Added a blog to mudmap but am still not ready to launch. 😞

Highlights

Got the blog working at mudmap
Backend code works as well but still has some quirks

Goal Performance

Each month I write up how I went in the pursuit of last month’s goals. It is also somewhere I can expand on the things I’ve done. Helpful for historical review and analysis.

Add a blog to Mudmap

Appraisal: Happy that Wagtail is so easy to introduce and that the blog lives inside my app
Rating: A

After looking around about how to deal with this I decieded on Wagtail.

It met my requirements:

static rendering
supports markdown and rich text
hosting under my control
routable within the application

Initially I considered a few options:

Ghost
Hugo
Pelican
Plain HTML

None of those were a great fit, I really wanted to use Hugo because its a brilliant product (this site is built on Hugo) but I just couldn’t get it to work how i wanted it to. In the end I chose Wagtail because its built for Django and I’ve used it before, so I don’t need to use up one of my precious innovation tokens.

In the end, its working and let’s me write in both markdown and rich text. It’s easy to make changes to the underlying HTML and CSS. It uses Jinja as the templating language and integrates seemlessly into the Django ecosystem.

My routing is simple but works exactly like a typical Django url and it is hosted on my infrastructure. Supports SEO optimization out of the box as well. Nothing is perfect but this seems pretty great so far. Only complaint is my shabby formatting and design!

Complete Mudmap’s backend PoC

Appraisal: It works but there’s a lot of polish (and bugs) to iron out
Rating: B

I am happy with the proof-of-concept. By month’s end the backend and dashboard could:

Register a new device in the database
Make the initial connection
Install the API package
Install a service account user
Retrieve system information and display it in the dashboard
Update a registered device, or delete it

Gain 5 Twitter followers

Appraisal: I need to actually provide useful content to readers!
Rating: F

In hindsight, its kind of a crap metric. I do believe in having an audience, hence why I am electing to publicise my work (good and bad). But, I don’t know that holding myself to a follower count improvement is a good goal. It should probably be the by-product of other goals such as write a long form blog post or some other value add for any readers.

Recommendations

I really enjoy listening to Nick Janetakis’ Running in Production podcast. In particular, I throughly enjoyed his interview with Justin from ButtonDown. It’s good advertising too because I pretty much immediately signed up for his service - I will always support the indie creator in favor of the big tech companies.

Otherwise, I watched a documentary on YouTube about animals returning to Chernobyl. I highly recommend it, family friendly too - it’s like Milo and Otis for those that have now grown up!

Wildlife Takeover: How Animals Reclaimed Chernobyl

Wrap up

What can I do better?

If I am going to publish things to twitter, make sure they’re actually of value to any readers
Be more focused; I need to prioritize features that will deliver the MVP in the least amount of time

What have I done well?

Used my previous experience with a product (Wagtail) to make a quick integration instead of pursuing the traditional developer move of trying something new
Learnt a great deal about Django Rest Framework; pretty happy with its performance.

Next months goals

Finish the copywriting on mudmap’s landing page
Integrate Stripe as the payment processor
Write a blog post for danielms and mudmap

Jan 2021 Retrospective

Sun, 31 Jan 2021 00:00:00 +0000

Summary

Thought of a potential business to create and its target market.

Highlights

Purchased a domain; mudmap.io
Created the proof of concept backend code
Researched the domain for market fit, competition and general sense of whether its viable

Goal Performance

Given this is my first retro, no goals were set last month. So I will talk to the product and highlights above.

Finding a product to create

Late last year, I discovered a niche in something I have found annoying in the past; managing multiple pfSense firewalls. This usually means logging in to each device one at a time using either SSH or a VPN. Its laborious and I hated it.

Then I found a few custom API’s that interface with pfSense. This meant I could now make remote calls to each device and script my tasks. Anything that can be scripted can also be turned into a web application, and thus the idea of mudmap was born.

Before I started writing any code, I did some research to see if the idea was viable, or needed. I must admit it did take me too long to find one of the competitors in this space. Originally, mudmap was going to be called pfmon but pfmonitor already exists. The naming similarities are just too close for me to be comfortable with, even though I still really like the name.

I asked questions on Reddit, searched pfSense’s own forums and even poked around on Twitter.

A crude summary from a Reddit post:

I can’t upvote this enough. I love pfSense and consider it top of the stack, but there are so many options who offer central management and that makes it tough to recommend it to customers.

It’s pretty much this mentality that makes me feel this product is viable. Time will tell

Recommendations

If you live in Australia, or want to travel to Australia checkout Newcastle. I used to live there but moved away to take a job and I miss it. I was there over Australia Day and its just magical. For a West Australian to say that an east coast beach is brilliant is a big deal; west coast beaches are on a whole unequivocally better.

Wrap up

What can I do better?

Catalogue my research, particularly product market and feasibility information into notion for future reference

What have I done well?

Started on the path from worker to creator. I follow dvassallo’s school of thought here; optimize life over money.

Next months goals

Add a blog to mudmap
Complete PoC of mudmap backend; API install, account creation, device registration, view properties
Gain 5 twitter followers

2020 In Review ✨

Thu, 31 Dec 2020 00:00:00 +0000

What a time to be alive.

This year has been as weird as it has been amazing. In Australia, we’ve gone from burning to the ground to Covid to a downright wintery Christmas. Alice Springs recorded its lowest Christmas high temperature of 20 degrees Celsius. The stock market has tanked, only to rebound and in some sectors flourish. China and Australia are starting to engage in economic warfare. Most notably, people were beating each other up over toilet paper.

Thankfully, none of these are within my locus of control. This is actually a good thing because I can focus on what matters to me.

So what has happened in my life that is noteworthy.

Personal

Quite a lot has happened in my life this year, but I am highlighting some of the best parts.

Home and away 🇦🇺

I survived my first Canberra winter. Canberra is one of those places where as soon as you mention you live there to a non-Canberran, they remark about how cold it is.

It is not cold here. Not real cold, not you will die if locked out your house in boardies and t-shirt cold. It would suck but you won’t die.

And, I loved it - it is beautiful here. Snow-covered mountains, foggy mornings, and sunny clear days (for the most part).

We traveled to the south coast a few times this year. I have traveled a lot in Australia but seen little of the south coast. It is a magical place with some beautiful beaches, rainforests and lovely weather. In 2021, I hope to get out there more often and explore with the family. Highly recommend Merimbula and Narooma.

One bummer was hitting the snowfields this year. We got down there just in time for the fields to be declared shut for the entire weekend we were there. Still, I was able to experience some actual cold and do a little bit of hiking with my buddy. Out of all the traveling we’ve done this year it was the only true Covid effected situation we experienced.

Work

Work has been busy but rewarding. I have learned so much as developed a lot as a person. Definitely the best working year I’ve had since 2016. The highlights have been the increase in my skillset: Django Angular Docker CI/CD Ansible, and Packer Microservices Design and architecture I feel grateful to have been able to learn all this, and greater still to have kept my job during the pandemic.

2021

I don’t do resolutions, I do try to make habits and sometimes I fail, or forget so this should keep me accountable.

Things I need to do to achieve all that I am capable of:

I plan on do more in public; building tools, more blogging and I aim to speak at a local tech conference/meetup.
I’ve started a SaaS business idea that I think has legs. If it’s a flop, fine, but I will use it as a platform to level-up my dev and business skills. Checkout pftrace.com.
My execise commitment wanes and waxes throughout the year, I need to make it more consistent - min 15 minutes of activity per day that isn’t walking (I do 20-40 mins of that already)
Save money; I have calculated that it is the small frequent purchases that have the biggest effect on my bank balance. None of those things make me healthy, wealthy or happy.

🍻 Here’s to internet accountability!

Reflection

Another year completed and one of the best I have had in the last few. Whilst everyone is in the midst of Covid despair, I have tried to focus on what I have and what makes my life great. I have my family, a place to call home, and purpose in life - that’s what makes me feel wealthy, and this year has added to that wealth.

From requirements.txt to poetry's pyproject.toml

Tue, 27 Oct 2020 00:00:00 +0000

Poetry and Requirements.txt

I am now a poetry convert, opting for it in any reasonably large projects. Personally, I have found the ability to pin pacakges and produce a lock file invaluable for getting a complete picture of an applications dependencies. It also prevets needless docker build time spikes if a package is updated.

Unfortunately, converting requirements.txt to pyproject.toml is not yet integrated natively by poetry.

The problem

I needed to change a requirements.txt file of this structure:

-r base.txt
django-debug-toolbar==3.1.1 # https://github.com/jazzband/django-debug-toolbar
django-extensions==3.0.9 # https://github.com/django-extensions/django-extensions
django-coverage-plugin==1.8.0 # https://github.com/nedbat/django_coverage_plugin
pytest-django==4.1.0 # https://github.com/pytest-dev/pytest-django
... truncated ...

to a pyproject.toml file like this:

[tool.poetry.dependencies]
python = "^3.8"
gunicorn = "^20.0.4"
psycopg2 = "^2.8.6"
sentry-sdk = "^0.19.3"
django-storages = {extras = ["boto3"], version = "^1.10.1"}
django-anymail = {extras = ["sendgrid"], version = "^8.1"}
pytz = "^2020.4"
python-slugify = "^4.0.1"
... truncated ...

The solution

All it took was one line:

cat requirements.txt | grep -E '^[^# ]' | cut -d= -f1 | xargs -n 1 poetry add

It will loop over each line and strip out any # then call poetry add followed by the name of each package. Test it by calling cat requirements.txt | grep -E '^[^# ]' | cut -d= -f1 to see what it will output before trying to poetry add it.

If it encounters something like -r base.txt which is not a package, poetry will throw an error to stdout but continue looping over the file. If you’re really keen, you could strip lines starting with -r but why bother?

The caveats, this will install the latest version of each package because xargs will only return the package name and not any accompanying ==n.n.n. Again, this could be extended if needed.

To install development dependancies the line can be amended with a -D or -dev.

Poetry for the win

If you are considering switching to poetry but have many legacy requirements.txt files for various build states, its now almost trivial to switch. If you’re on docker, poetry or other tools such as pipenv or pip-tools are going to potentially save you a lot of lost time from your pip layer caches being busted when an unpinned subpackage gets updated. Github’s dependabot can also interpret pyproject.toml files meaning you will get automated pull requests for package updates just like you would with a requirements.txt file.

Get poetry here.

The Slight Edge, A Summary

Sun, 20 Sep 2020 00:00:00 +0000

The Slight Edge in my Words

Do the thing, and you shall have the power.

The Slight Edge is a fantastic book that explains the power of continual small improvements that compound overtime. These 1% gains each day go on to make you appear to be an overnight success. I have experienced this as a self taught software developer.

To bystanders and friends my journey appears as though I learned everything I know about software development during my university degree (which was in arts, not science). Sadly, this train of thought is because people don’t see the slight edge at work - I spent everynight learning and practicing my craft to suddenly become an overnight success.

I have written this as a homage to one of my top five favourite and most profound books.

Two takeaways are:

You have complete control over the direction that the rest of your life takes.
“This is Football” or why we overcomplicate everything.

Beginning

Two Mindsets

Entitled: “What have you done for me lately”, “Pay me more and I’ll work harder”

Value-driven: “What can I do to help you?”, “I’ll work harder, and then I expect you’ll pay me more”

The Water Hyacinth:
- A story of the plant that doubles itself each day - starting at a single cell.
- It takes many days of seemingly no growth before it compounds monumentally to eventually over grow the entire lily pond
- This story demonstrates that the little changes compounded over time aren’t noticed, they are so slight that its not noticed for a long period. When it is noticed the onlookers think “it happened overnight” which it surely did not!
The Millionaires Sons:
- Another story, this time about two boys each given a choice; receive 1 million dollars 30 days after their fathers death, or get 1 penny but have its value double everyday for 30 days.
- 1 penny compounded each day for 30 days is 10 million dollars…

“It’s never too late to start” & “It’s always too late to wait”

Parkinson’s Law states:
- “Work expands to fill the time available for its completion.”

Spare a thought for what junk you put in your body:
- It’s not a trash can
- “Many of us are literally digging our graves with our teeth.”

Consistently repeated daily actions + time = unconquerable results

Happiness

Happiness is more important to our success than we realise. These are some “happy habits”:

Each morning, write down three things you are grateful for:

Not the same three; new ones that force you to think of new tings to be grateful for.

Journal for two minutes each day about something positive:

Write down this positive moment in great detail - this causes the brain to relive that happy experience, doubling the positive impact. I also find it gives me time to reflect, and think back on my day often seeing many moments worth writing about.

Mediate daily

Just stop and do nothing for two minutes. Breathe in and out letting it all wash over you. Relax your body from bottom to top through visualisations such as “making your legs concrete and heavy”.

Do one random act of kindness each day

Praise someone, or thank them when it is not expected. Offer time, or assistance when no one expects it.

Exercise for 15 minutes daily

Simple cardio; a brisk walk, or hardcore session, its this stimulation for the body that triggers a strong antidepressant response. 15 minutes is 1% of your day; you have the time!

Read at least ten pages of a good book each day!

This list is only 30 minutes of each day but the key is not to over commit and start doing it all. Begin slowly, developing that slight edge by creating lifelong habits. Once you habitualise one, add another. This is a lifelong journey.

Are You Your Own Cause?

The upper half take full responsibility for their actions. They view all forces that brought them to this point, to be under their control - parents, teachers, friends, family, and so on. These influencers are viewed through the lens of gratitude and appreciation, not blame. They view themselves as the cause of what comes next in their life, no matter whether those influences have been negative or positive.
Negative people dwell on the past and positive, successful people look to their future. Your past can bring you down, its what people on the failure curve look at - I failed at this, so and so did this to me, I was X at school. People on the success curve don’t shun the past, they use it as a tool to shape their future for the better. Failure curvers use their past as weapon of self-flagellation and to hurt those around them.

Honestly look at your your level of happiness; do you take the time to see things in a postive light, rather than a negative? Are you savouring the moment, and expressing your appreciation of others? Are you engaged in activies you want to do, rather than those you feel you have to do?

Problem Solving

The size of the problem determines the size of the person.

You can quickly determine the limitations of someone’s life by the size of the problems getting them down. But you can equally determine the impact a person’s life has by the size of the problems they solve.

If the biggest problems you are solving is should I put the bread, or cans into the shopping bag first? Then that is the level of your problem solving, and that level will be commensurate with your income.

A problem is just a gap between point A and point B; an open space to be filled.
When there is a gap - a problem - it will create tension. The tension can be overcome in only two ways:
- We either move headlong into the tension to slowly overcome the gap and solve the problem (as the top 95% do), or we remove the tension by extracting ourselves from the problem - in a detrimental to our future way.
- Doing nothing, is a form of tension relief as time marches on the problems will erode but they will likely erode negatively. For instance, if you want to make exercise a habit, there will be a tension each time you want to train. By sleeping in, or missing that session today you have removed todays tension - its only one day! But overtime, you have trained less, are more habitually likely to skip sessions and therefore be highly likely to not be that person you wanted to be. The tension is gone but the problem leading you to wanting to start exercising still remains.

Master the Slight Edge

“Be not afraid of moving slowly; be afraid only of standing still.” - Chinese proverb

Momentum preveils:
- Everything has an optimal rate of growth; never do long standing things have a high rate of growth that can maintain that rate over a long period. We harness gradual and seemingly inconsequential growth through the small and insignificant habits done everyday over years.
Completion:
- A lack of completion drains us; all the little things we still have to do, all compound to suck our will and life from us. The only way to tame this beast is to actively address as many of these problems to make the list as small as possible.
Reflect, and do so in public:
- do not hide your actions; catch yourself doing something right.
- Be open with the public about what you are doing; let the masses hold you to account, and be “shocked” about your sudden “sudden” overnight success.
- Holding the past you to account serves as a reminder that you are growing - all we need is the be better than our past self.

Three “you’s”

“Three words. Exercise and books.” - /u/ryans01

We have three “you’s”; past, present and future. And this ain’t from the slight edge but sums up a lot of it and is ghetto gospel for the youth. Straight from ryans01:

La deuxieme regle - yeah i learnt french. its a canadian thing. please excuse the lack of accent graves, but lemme get into rule number 2. BE GRATEFUL TO THE 3 YOU’S. Uh what? 3 me’s? That sounds like mumbo jumbo bullshit. News flash, there are three you’s homeslice. There’s the past you, the present you, and the future you. If you wanna love someone and have someone love you back, you gotta learn to love yourself, and the 3 you’s are the key. Be GRATEFUL to the past you for the positive things you’ve done. And do favours for the future you like you would for your best bro. Feeling like shit today? Stop a second, think of a good decision you made yesterday. Salad and tuna instead of Big Mac? THANK YOU YOUNGER ME. Was yesterday a nonzero day because you wrote 200 words (hey, that’s all you could muster)? THANK YOU YOUNGER ME. Saved up some coin over time to buy that sweet thing you wanted? THANK YOU. Second part of the 3 me’s is you gotta do your future self a favour, just like you would for your best fucking friend (no best friend? you do now. You got 2. It’s future and past you). Tired as hell and can’t get off reddit/videogames/interwebs? fuck you present self, this one’s for future me, i’m gonna rock out p90x Ab Ripper X for 17 minutes. I’m doing this one for future me. Alarm clock goes off and bed is too comfy? fuck you present self, this one’s for my best friend, the future me. I’m up and going for a 5 km run (or 25 meter run, it’s gotta be non zero). MAKE SURE YOU THANK YOUR OLD SELF for rocking out at the end of every.single.thing. that makes your life better. The cycle of doing something for someone else (future you) and thanking someone for the good in your life (past you) is key to building gratitude and productivity. Do not doubt me. Over time you should spread the gratitude to others who help you on your path.

Cultivating the Slight Edge

Your Philosophy => Your Attitude, Actions, Results => Your Life

Its habits and philosophy done repeatedly, over time that will make you the person you want to be.

Habits come from daily activities compounded over time. Those activities are choices you make in the moment (get out of bed early, read rather than scroll your socials). Your choices come from habits of thought, which come from your thinking, which are a representation of how you view the world and your place in it - this is your philosophy.
In other words, its your everyday habits and beliefs that will shape who you are in the future. Simple little actions will make the difference over the long history of your life.

Habit 1: Show up

Most people are struck down with hesitation, the cornerstone of mediocrity.
The greatest achievers when asked how they did it often say something to the effect of: I just decided to do it
do the thing, and you shall have the power

Habit 2: Be Consistent

80 percent of success is showing up every day
Do the work, every day and you will have beaten more than 50% of the competition

Habit 3: Have a Positive Outlook

People who see the glass more than half full in every circumstance are:
- more successful,
- happier,
- more creative,
- live longer,
- make more money,
- have better immune response,
- less heart disease and strokes,
- have better, and longer lasting relationships, and
- are more successful in their careers.

Does this mean you have to jubilantly happy all the time? No, we all have bad days. But its how we manage those. When feeling blue, take time to reflect on your blessings; what you’ve achieved, your family and friends, your health and so.

Will this take you out of your funk immediately? Probably not, and that’s when the importance of understanding contrast is powerful.

Contrast:

How can I know love, if I’ve never felt loss,
I cannot understand what good feels like, if I have never felt bad,
Without feeling the funk, I can never know what happiness is,
Life is ebb and flow - we must experience both spectrum’s to appreciate the side we want to live on.

Habit 4: Be Committed to the Long Haul

It takes a long time to grow and old friend
Plant, cultivate, harvest - the comma between cultivate and harvest often represents a looooooong period of time.
10,000 hours doesn’t happen in a weekend - it takes years of sustained, and prudent effort to achieve the success we dream of. Commit to it.

Habit 5: Cultivate a Burning Desire Backed by Faith

No, not religious faith; faith in your self!
Desire by itself is a weak binding by itself - it is not the sole marker, that desire must come with a burning desire - that which keeps you up at night and gets you out of bed early in the morning.
desire backed by faith is the deep, passionate wanting to get somewhere and knowing - not wanting or wishing, but knowing that you will get there.
There will be problems encountered along this path but remember: you can determine the size of a person by the size of the problems they overcome.
A burning desire is what helps you confront them, rather than turning an running away.

Habit 6: Be Willing to Pay the Price

We can only do so much in our life, some times we must make a choice between greatness and pleasure.
Very few billionaires are playing in the NHL, or whatever sport you follow - they made a sacrifice in one area, to be great in another.
Whatever price you pay, there’s a bigger price to pay for not doing it than there is for the price of doing it.
The price of neglect is higher than the price of discipline!

Habit 7: Practice Slight Edge Integrity

Be committed to doing, even when no one is watching.
Its the choices you make when you’re tired, or broke, or in a funk that define whether or not you have the integrity of the slight edge.
You need to do what you set out to do, every day even when no one could ever find out that you didn’t do it because someone will always know the truth; you.

How to Achieve a Goal

Write it down: give it a what and a when.
Look at it everyday: keep it in your face; soak your subconscious in it - auto-suggestion.
Start with a plan: make the plan simple. The plan won’t get you there, but it will get you started.

Summary

Do one small thing to advance yourself every day in key areas of your life - health, happiness, career, finances, impact, personal development
Review these actions daily, and track them. You can journal them, write them down on a post-it, share them with your friends or mentor. It makes you both consistent and aware of your growth.
Spend high-quality time with men and women who have achieved goals similar to those which you desire. Model those people and surround yourself with like minded persons, after all you are the average of your five closest friends.

Successful people own their problems, they do not blame. They know the path between success and failure are razor thin, and that by doing nothing we automatically fall down the failure curve.
It is the “insignificant” actions done day in day out which set us apart between the 95% and the 5%.
The top 5% acquire three kinds of knowledge required to succeed:
- Book and street smarts,
- Learning through study and doing,
- Catalysing and accelerating that knowledge by finding mentor and modelling their behaviour.
successful people always ask:
- “Who am I spending time with? Are they people who best represent where I want to be headed?”
They surround themselves with, and develop strong relationships with positive people.
They read at least ten pages of a good book each day, or 15 minutes of high quality audio.
They work on their philosophy first because they know it creates their values, actions and results in life.

To understand the slight edge you must know:

The power of simple things,
The power of daily disciplines,
How the water hyacinth works, and how to use it,
When you are being offered the choice of wisdom,
How to put the slight edge to work for you, rather than against you.

Getting Fedora 32 and Docker to Play Nice

Sat, 29 Aug 2020 00:00:00 +0000

Fedora 32 and Docker Are Incompatible

Fedora is a great distribution for development, its very customizable, and supports a great number of packages and applications out of the box. If you’re going to use a linux environment thats not Debian based, it ought to be fedora.

Sadly nothing is ever perfect, meaning Fedora too, has its downsides. Currently, at least to me, is its inability to run Docker out of the box.

Its actually dockers fault

To be fair its not Fedora’s fault but Dockers. Fedora no longer supports control group version 1 and has moved onto the much better version 2. Unfortunately, Docker only supports version 1 and they aren’t backwards compatible.

This effectively makes using Fedora with docker untenable without modification.

Podman to the rescue

Sure, you can use podman but the world still uses docker and more specifically docker-compose.

Your work environment might even negate both by just using kubernetes or openshift. But in the open source and hobby project world that is overkill.

So, let’s instead support both.

Why not have both?

While the current version of docker (19.03 as of writing) doesn’t support cgroupv2, it does support it on its moby branches.

Unstable? Well, its either that or no docker for you! It appears docker will merge support and make a release sometime this year - 2020. But until then you need to use the binaries supplied from compiling the docker branches.

Thankfully, some legend has already done it. Go to the github page, download the latest release and follow the installation instructions - which is just dnf install. Once its installed, you will now have docker with cgroupv2 support.

Your new docker setup will still need to be configured as per the normal docker instructions

Profit!

Docker and Podman in harmony

Now you can safely run both docker and podman without downgrading anything and still get all the benefits of fedora and cgroupv2.

I personally think Podman is great, but so is docker. For my outside-of-work humble purposes, I only need docker and am thankful there is a workaround. Because, unpopular opinion, macosx sucks and so does ubuntu.

Docker has a rich ecosystem, massive mind share and wonderful support in both vscode and pycharm. Something most other container platforms lack. Until podman has a better integration I think its uptake in the developer community at large will be slow.

Recap

Fedora 31+ cant run docker unless you install the precompiled binaries from the Moby branch.

After installing these, you’ll have docker and podman working seamlessly.

No excuse to not use fedora as a docker acolyte.

Also if you want more information as to why we should be switching to cgroup version 2 see here (this dude is legit, probably the authority on linux), here and here.

2020 So Far, A Brag Doc

Mon, 13 Jul 2020 00:00:00 +0000

Brag Documents

A reminder to yourself to review what you’ve done this year. Catalogue your wins and reinforce good behaviour.

2020 In Review

Here is what I have to brag about in 2020 so far:

Technologies:

I have developed, integrated or otherwise taken the following technologies beyond “beginner”, or enhanced these skills with the following examples.

Angular
- Gone from knowing zero Angular to being the lead (sole) developer UI developer in a complex microservices application.
- Lead the UI redesign including initial mock up and wireframes
- Produced the scope of work within sprints
- On boarded another UI developer and helped other experienced member re familiarise themselves with the code base
- Deployed the redesign to production
Vue.js
- Built two separate web application with vue, both using Typescript
- Both integrated into Python application backend’s using FastAPI
Ansible
- Identified that creating the infrastructure for deployments is tedious, error prone and not easily repeatable made learning ansible a high priority.
- Successfully created a playbook to create a consistent deploy environment for side projects;
  - update server security and packages
  - install load balancer
  - ensure small servers have sufficient swap space (to compile javascript)
  - git operations on private and public repositories
  - encrypt secrets using vault
  - development with Vagrant and live instances
Flask
- Developed a bespoke application which could interface with other custom devices on a large network
- Utilised plain javascript and server side rendered templates to create a UX that users found more familiar than the tool it replaced
- Integrated stripe into Flask
- Used a third party websockets service to facilitate realtime chat between users
FastAPI
- Deployed two application with the framework, replacing it with Flask in one instance
Docker (and swarm)
- Went from having a rudimentary understanding to being able to create detailed docker-compose files and deploy to a swarm
- Now use docker as my development environment
- All side projects have been ported to swarm with load balancing and HTTPS included
- Utilised kubernetes for a short notice project and learnt the basics of the platform

Skills:

Leadership
- Lead the UI design and implementation after being assigned to short staffed, over worked close knit team on a complex microservice architecture.
- Integrated well with the team and was able to provide subject matter expertise in Angular within a short period of time
- Became the lead developer of another project requiring the integration of two systems (both in python) requiring facilitation of planning and design sessions between the teams
- Speaking role in presentations selling our new product to senior management, including live demonstration of our MVP
Design
- High-level planning sessions for the integration of two platforms
- Mock up and wireframing the UI for another team’s platform
CI/CD
- From being relatively inexperienced in CI/CD to developing pipelines for projects ranging from established to newly created
- Learnt how to establish shell and docker runners in using Gitlab and AWS
- Created segregated pipelines for separate deploys based on tags
- Successfully integrated Github actions in all side project web applications, including deployment of containers to docker swarm
Bash | Shell scripting
- Took over established and complicated deployment shell scripts for production assets
- Learnt a lot about debugging leading to the identification of several fixes resulting in the eradication of pipelines passing when they should have failed

Accomplishments:

Deployed lic-investing.online
- An Australian financial web application which visually shows whether certain stocks are trading at a premium or discount.
- Uses FastAPI and Vue.js
- Scrapes web resources using various techniques to gather the data, this includes punting that raw data into Postgres.
Deployed check-redirects.com
- Provides a service for tracing the redirection chain of a URL including status codes and response headers.
- Includes an API for third parties
- Built using FastAPI and Vue.js
Replaced a rotting internal legacy tool with a web application:
- Took an internal tool which was not being actively maintained and replaced it with Flask.
  - The tool needed to be either fixed or replaced to meet a looming deadline and our team was assigned to aid in unblocking the task
  - I first attempted to learn the code base and improve upon the design, however there were several critical issues which were cost prohibitive
  - After proposing a porting of the tool into a web application the design, implementation and handover of the task was completed in four weeks
  - This lead to the successful redeployment of a much needed tool, ultimately saving future development resources by rewriting it with a framework almost all developers in the organisation are familiar with.

After writing all this, its plain to see that I have achieved more than I realised.

The value in taking the time to write out what you have achieved over a period of time cannot be under estimated.

I did this in one sitting, unrefined and (mostly) unedited. This is for you to look back on and polish later when you need it.

what gets measured, gets managed. - Peter Drucker

Flask's jsonify in 2020

Thu, 14 May 2020 00:00:00 +0000

Flask jsonify, do I still need it?

For a long time, returning JSON in flask required using the flask.jsonify API. However, since this PR Flask will by default call jsonify under the hood on any dictionary it receives on a make_response call.

How does it work

When you need to return json in a response, you can simply use a plain python dictionary instead of the jsonify API.

# plain dictionary
@app.route('/plaindict')
def testplain():
 test = 'testing'
 name = 'roger'
 age = 30
 _list = [1, 2, 3]
 return {
 "development": test,
 "name": name,
 "age": age,
 "list": _list,
 }

The above code will return the following:

# response
HTTP/1.0 200 OK
Content-Length: 105
Content-Type: application/json
Date: Thu, 14 May 2020 21:20:06 GMT
Server: Werkzeug/1.0.1 Python/3.8.2

{
 "age": 30,
 "development": "testing",
 "list": [
 1,
 2,
 3
 ],
 "name": "roger"
}

And the same code but with jsonify (for comparison) will return the exact same thing.

@app.route('/jsonify')
def testjsonify():
 test = 'testing'
 name = 'roger'
 age = 30
 lst = [1, 2, 3]
 return jsonify({
 "development": test,
 "name": name,
 "age": age,
 "list": lst,
 })

How does it work

As stated above, Flask now does an isinstance check during the make_response API call. In the conditional, Flask checks if the body is of type dict and if so calls jsonify. To see this in action refer to the previously mentioned line in the core Flask app.py.

So, while we no longer need to explicitly call jsonify it is still very much being used by the application itself.

Whats the catch?

There is one small gotcha, as seen from pgjones pull request.

This doesn’t support returning anything other than an associate array at the top level in the JSON response. I’m ok with this as in practice APIs are only extensible if the top level is an associate array.

An example of when you will still need to user jsonify is when the response is not of type dict. As an example, doing a list comprehension and returning that directly will be required to be wrapped in jsonify.

@app.route("/users")
def users_api():
 users = get_all_users()
 return jsonify([user.to_json() for user in users])

Fin

While its not a complete replacement for flask.jsonify, it will probably reduce its explicit usage by a huge proportion. Personally, I removed jsonify from most of my applications and now just drop in plain dictionaries.

Starting Your Side Hustle

Sun, 12 Apr 2020 00:00:00 +0000

Starting Your Side Hustle

This is a collection of notes from one of the world’s expert Side Hustlers, Chris Guillebeau. If you have not read his book The $100 Startup, you really should. It also includes important takeaways from other bloggers and authors. Wherever applicable references have been added.

Its also pretty important to define a side hustle. We are talking about small enterprises which take little to no time for setup, maintenance and daily running. We aren’t looking to start the next Facebook - though it could evolve into that but its not the intent from day one.

Notes From a Hustler

Side Hustle Questions to Ask Yourself

There is three (3) things you need to ask yourself about any chosen side hustle:

Does this solve a clear and urgent problem, or, does this thing make someone’s life easier in a specific way?
Will the idea generate income over a long time scale?
Is this something you like to do, or are you just doing it for the money?

If you can answer those questions in the positive sense then you have a potential option to explore.

Are There Bad Side Hustle Ideas?

Yep, and here is a list of things to look out for:

Part-time jobs dressed up as side hustles (freelancing),
Anything where you give up control of the income potential,
Something that exists solely on one platform or network,
If its not replicable (each iteration is a one-time deal),
Anything where you are simply trading time for money.

Examples:

Rideshare
Gig economy options

The Three categories

Every hustle can be put into one of three baskets:

Products,
Services, and
Everything else.

Products

Something you give people,
Build once and sell once,
One, or multiple items for sale,
Books, gifts and other non-recurring items sold to a consumer

Services

Something you do for people,
Often ongoing for either short or long periods,
Customer facing,
Courses, coaching and management are good examples.

Everything else

Anything where you are not selling an item or dealing with customers,
Affiliate commissions, arbitraging items between stores or in currency are examples.

Each of the three types will require slightly difference avenues of management and marketing. Knowing where your idea sits is important.

Idea Curation and Selection

This is the hardest part for someone like me, and probably most people. How do you first find a good idea, and if you have several, select the right one.

Patrick McKenzie said it best.

Some people profess difficulty at finding applications to write. I have never understood this: talk to people. People have problems — lots of problems, more than you could enumerate in a hundred lifetimes. Talk to a carpenter, ask him what about carpentry sucks. Talk to the receptionist at your dentist’s office — ask her what about her job sucks. Talk to a teacher — ask her what she spends time that she thinks adds the least value to her day. (I’ll bet you the answer is “Prep!” or “Paperwork!”)

That’s what it really comes down to, find a problem someone is having that you can solve. It could even be something you find annoying. Whatever it is, your next step is to assess it again the following criteria:

Feasibility: can I even do this thing, do I have the skills?
Profitability: will people buy this, or will my outlay be recovered by purchases. Include both time and financial outlay.
Urgency: is this a good idea in today’s world. Do people care about solving this problem today - many things have launched way before the masses respected the problem being solved.
Efficiency: can I achieve this quickly? What might hold me back?
Motivation: simply, do I really care about this? If not, you won’t have the gumption to stick out the tough moments.

Think hard on each point, write it out. Score it honestly and even discus it with close family or friends. They might see problems from a view which you are blinded to right now.

Finding Your Customer

This was a big takeaway; identify your perfect client in exacting detail and be ruthless in designing the product around them.

Ask yourself:

How old are they?
Where do they live?
Gender,
Race,
Country or region,
Unique characteristics, and
What challenges do they have.

Once you hone in on your target, design your product or services around them. Speak to them. Act as if the product/service is tailored (and it is) exactly for them.

You need to now speak to them, and if you have bracketed this person or persons correctly they should be hearing your loud and clear.

Idea Selected, Now What?

Time for action. What do I now need to get the ball rolling? Here is simple list for getting something out there.

Website with Sales, Product/Service, Contact and About pages,
Email list,
Social media,
Initial inventory (id applicable)
Payment processor

Focus only on the bare essentials. Don’t build out excess “functionality” based on what you think the customer wants. They will tell you.

A realy important part is designing your payment workflow.

How does your custome make a purchase - think this through carefully. Any hiccups on this road may lead to a lost sale, we are most likely to reject a product just before buying it.

Successful offers

At its core, each category is making an offering to its potential customer. Every successful offer has three things:

Promise - why should you care about my widget,
Pitch - why you need it right now,
Price - and you can get it for n Now

How to craft the offer. Remember you are writing to a person, not a group or large swath of people. Each sale is form one person at a time, so speak to that person! You action words to describe the product, why its needed and why its needed right now. Always use numbers to explain statistics or prices. And, use stories that elicit happy emotions.

Pricing your Yourself

Simple, know what you are worth.

If its a service, work out what your time is worth and add a margin on that. And if its a product, calculate your expense for each item, time to create and add a profit margin.

Don’t be coy, a profit margin of less than 20% is probably going to be a profit of 0% in the long run.

Testing the Hustle

Rehearse, rehearse, rehearse. Double check your work in every aspect: does my website work on desktop and mobile, does my payment workflow work (have I manually tested in several times), does my landing page have spelling or grammatical errors?

No doubt after so long reading the same script over and over, you aren’t seeing simple mistakes. This is when its paramount we get friends or family to proof our work.

Does your offering have the following:

A clear offer,
Pitch, Promise and Price,
Can it be explained in one sentence (and understood),
How are your customers going to find you (don’t rely on one (1) method),
A simple payment workflow
And, do you as the creator have goal for this project for at least first 30 days.

Side Hustler

Creating a side hustle doesn’t have to be massive, or change the world but it needs to have several characteristics if its going to survive. Contrast each idea against this and see where you fall short, and if you can fix it.

Web Scraping Javascript with Python

Fri, 13 Mar 2020 00:00:00 +0000

Scraping Dynamic Pages with Python

Web scraping

Python is a great tool for web scraping tasks, it is efficient, easy to read and fast. Whenever looking to grab data from a site, the canonical packages are BeautifulSoup and Requests. Unfortunately, when our target site is dynamically rendered, BeautifulSoup can’t “see” those parts leading to a lot of head scratching. The often touted answer to this is Selenium which spins up a browser thereby rendering the JavaScript making it possible to scrape data from it.

While selenium does work it kinda sucks, can be a pain to setup and introduces more complexity. On big projects where scrolling, pagination or link traversal is required you may be best served by using Selenium. Though I would recommend Scrapy, which is excellent and is built for scraping - Selenium is not.

In some cases, there is another way.

An alternative

Many modern web applications render data from third party API’s or other backend services. When a site uses JavaScript to do this, then we can get a lot of data from it by using only Requests, Postman and cURL.

The basic steps:

find the API endpoint using the browser’s development tools,
copy the URL as a curl command,
import that command into postman, checking it works,
get postman to auto generate the request into Python code, and
plug that into your script, and profit!

This will provide you repeatable python code which will always return a response with the data you require.

Step 1: Find the URL

Let’s use an example website which uses an XMLHttpRequest pull data from another server and populate a table with stock data.

After navigating to the site, open the development tools and click on the network pane. You may need to refresh the page to populate the network pane with its requests.

Searching through the requests, we find what we are looking for, in this case the requests are named after the stock symbols. There is no standard but you can filter by XHR in the network pane to make finding juicy targets easier.

For the uninitiated, when clicking on the XHR request, select the response pane and look at the JSON data it has returned.

Step 2: Copy as cURL

After finding the appropriate XHR endpoint, we now need to replicate the GET request.

In the network tab, as seen above, right click and hover over Copy which will show you an option to Copy as cURL. Selecting this will output something like the following:

curl 'https://lic-investing.online/api/stocks/bki' \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Accept-Language: en-US,en;q=0.5' --compressed \
-H 'DNT: 1' -H 'Connection: keep-alive' \
-H 'Referer: https://lic-investing.online/' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache' \
-H 'TE: Trailers'
# formatted for easier reading

You can test that this works by running the command in your console on MacOS or Linux. On Windows? 🤷 soz, I don’t know.

Step 3: Import into Postman

After copying as cURL above, open up Postman.

In the top left will be a orange box titled New, and to the right of it will be Import. Select Import and then click on the Paste Raw Text tab. Paste in the cURL command and hit Import. This process should resemble the image below.

After importing the command, Postman will populate all the GET parameters needed to make a request in its main window. Clicking on the blue Send button will fire the request and once Postman receives the response you will see it in the main body of the screen.

Right now we have returned a JSON object from our target without needing to be on their website proving we can simulate a request from their frontend to the backend. This is how we will scrape the site in a repeatable and reliable way.

Step 4: Get the python code

So far, we’ve created a request to retrieve data from the server. But now we need to turn this request into something we can replicate using Python.

Let’s use Postman to automatically generate some Python code for us.

To get this auto generated code simply select the Code text block which is directly below the Save drop-down on the right hand side of Postman’s main screen.

Scroll down to find Python code nicely generated for us. In this example I have chosen Requests, though it also offers http.client as well.

Step 5: Win

Now we have fully functional Python code which can make a request to the endpoint we are targeting and it will return a response as if we were the website. What we do now is pure business logic but the main thing is we did not need to reach for Selenium, and that is worth its weight in code.

Closing points

Be wary of hitting the endpoint repeatedly, you may get black listed by the site. To prevent this, use a library such as Requests-cache. Give them a star if you can!

This is also helpful when playing with API’s which have request or rate limits imposed.

Not related to XHR but always look over the source of a webpage in a browser, and then contrast that with BeautifulSoup’s response data. I have had success on React sites by grabbing their Props data that I couldn’t even see rendered in the browser. To get at custom, hard to scrape data like this, Regex is your friend.

If its on the internet, its possible to scrape (within legal limits of course).

Wagtail embeded YouTube videos

Sat, 25 Jan 2020 00:00:00 +0000

Wagtail Embed Video

Wagtail is a brilliant content management service built atop of Django. It comes with all of Django’s functionality and Django Rest Framework built in for headless work making life a lot easier.

In all, I really love it, especially its StreamFields. But I did stumble on one component, getting embedded videos to work correctly.

Embedding videos

This is harder than I feel it should be and poorly explained by the Wagtail documentation. Perhaps too harsh, in a sense - the documentation is actually really good. Except for in the circumstance whereby you want to render an embedded video inside your own HTML and CSS.

Before, covering that, again I should mention that wagtail’s documentation is excellent and so too is their implementation of this Framework. For instance, you can render an embed video ‘block’ by calling a template tag and creating the component in the model.

Except, it will always render this block inside a custom CSS div which apparently cannot be overridden. I’d wager 8 out of 10 times this might mess with your sites style.

Custom embedding of video content

There is a solution to this issue; just call the video.url within your own implementation of an iframe element. Except, if its YouTube, and your are using a URL such as https://www.youtube.com/watch?v=xUWd3o6z2bk, it will likely not run. Why? It’s the watch url endpoint. YouTube, wants embedded videos to use https://www.youtube.com/embed?v=xUWd3o6z2bk instead. Note the use of embed in place of watch.

And, wagtail will not accept YouTube urls containing embed causing a template error. So what’s a person to do; custom template tags to override this blocker!

Code or GTFO

Lets create a VideoBlock that will embed a video component into our template.

# blocks.py
class VideoBlock(blocks.StructBlock):
 """Only used for Video Card modals."""
 video = EmbedBlock() # <-- the part we need

 class Meta:
 template = "streams/video_card_block.html"
 icon = "media"
 label = "Embed Video"

This will now give us a StreamField which we can then use to enter the url we wish to embed into the template. In this example the url we are entering into our StreamField video_card_block is https://www.youtube.com/watch?v=xUWd3o6z2bk

Next, in the video block template we’ll use Bootstrap’s embedded video component and inject the url into the src attribute.

<!-- video_card_block.html -->
{% load wagtailcore_tags wagtailembeds_tags %}

{% block content %}
 <div class="embed-responsive embed-responsive-16by9">
 <iframe class="embed-responsive-item" src="{{ self.video.url }}?rel=0" allowfullscreen></iframe>
</div>
{% endblock %}

Unfortunately, this will render it as seen below, and be blocked by YouTube. In fact when it is blocked, you will need to consult dev tools to see that wagtail did indeed render the HTML.

<div class="embed-responsive embed-responsive-16by9">
 <iframe class="embed-responsive-item" src="https://www.youtube.com/watch/zpOULjyy-n8?rel=0" allowfullscreen></iframe>
</div>

To fix this we need to create a custom template tag which will intercept the watch url and inject the embed url into the template. This will make wagtail’s embed classifiers happy and us too.

import re
from django import template

register = template.Library()

@register.filter(name="embedurl")
def get_embed_url_with_parameters(url):
 if "youtube.com" in url or "youtu.be" in url:
 regex = r"(?:https:\/\/)?(?:www\.)?(?:youtube\.com|youtu\.be)\/(?:watch\?v=)?(.+)" # Get video id from URL
 embed_url = re.sub(
 regex, r"https://www.youtube.com/embed/\1", url
 ) # Append video id to desired URL
 print(embed_url)
 embed_url_with_parameters = embed_url + "?rel=0" # Add additional parameters
 return embed_url_with_parameters
 else:
 return None

This will now render the embed url as expected and upon inspection with dev tools should look like so:

<div class="embed-responsive embed-responsive-16by9">
 <iframe class="embed-responsive-item" src="https://www.youtube.com/embed/zpOULjyy-n8?rel=0" allowfullscreen></iframe>
</div>

fixed

I Searched the internet for hours and only after piecing it together over time did I eventually unlock this riddle! This is still an open ticket on the tail issue tracker!

Ecosia

Sat, 28 Dec 2019 00:00:00 +0000

Ecosia

Building awareness

The search engine that plants trees.

As we welcome 2020, lets acknowledge that doing something to help make our planet more sustainable is worth our time and effort.

Not everyone need agree on this topic, but those that do and have yet to hear about Ecosia let me give you some background.

The tree planting search engine

79 million trees have been planted to date and every 0.8 seconds a new tree is planted - December 2019

Ecosia was founded in 2009 by Christian Kroll who after graduating university travelled searching for a business idea with positive social impact. It was founded as a means to fund forest regeneration through ad revenue derived from search. It is also a B Corporation.

The business is very transparent with its finances publishing its balance sheet each month. It doesnt just say it uses 80% of its profits to plant trees, it proves it. Further, the owner has two publicly stated core beliefs regarding the future and funding of the mission; to never sell and never take profits out of the company. Ecosia is a steward-owned company making a breach of either commitment illegal.

In 2019, the company became completely carbon neutral through the construction of its own solar plant. It offsets all of its servers carbon footprint through the use solar energy.

Surely the results are rubbish

The search provider backing Ecosia is Bing, though they do tweak the algorithm somewhat. Anecdotally, I have not found it to be much different than either DuckDuckGo or Google. In fact it is my mobile browser and search engine of choice.

Bing is also carbon neutral.

Features

All the usual search engine features are available such as images, maps, videos, news with filtering options. They also offer direct searches into YouTube and Wikipedia to name a few.

If you are used to DuckDuckGo’s ‘shebangs’, its pleasing to know that Ecosia offers this feature also - not as many I should add.

The main ones to remember are:

#w wikipedia
#yt YouTube
#g Google
#a Amazon

Unique to Ecosia is its “green leaf” icon and Ecosia travel.

Ecosia travel lets you search and make accommodate bookings directly from the browser. For every booking, they pledge to plant 25 trees, though the actual number is derived from total cost of booking so it may be more, or less.

On some sites a “green leaf” may appear next to it. This denotes a business that shares the values of Ecosia. To be awarded the icon, each business must have demonstrated ecologically friendly or supply sustainable products or services. The leaf icon does not increase visibility in search rankings, nor can it be bought, only earned.

Thankfully, Ecosia respects privacy offering a service without trackers, analytics, onselling to advertisers and actually respects Do Not Track headers. Trust but verify here.

Where can I find more info

Checkout Ecosia by giving it a try below.

Search page

Blog

Swag

learn docker in one month

Sun, 24 Nov 2019 00:00:00 +0000

Learning Docker in one month

Yep, one month.

Docker basics

What

docker is a product that uses operating system virtualisation to created packages and applications which live within a “container”. These containers bundle together applications and dependancies without the need for installation on the host operating system. Containerisation is meant to empower developers by providing a “written once and run anywhere” methodology.

Why

Some of the core reasons to use docker.

The ability to have standalone applications and packages with dependencies that are isolated for the base operating system and other applications is a boon for developers and users,
Linux hosts can have several docker containers running without fear of contaminated package versions or dependancies causing errors,
Sharing consistent environments between developers is possible as each docker container image is idempotent, meaning the result of building a docker image will be the same whether built once, or one thousand times,
Version pinning is possible making compatibility issues on new release cycles less problematic, and testing of packages easier to implement,
Continious Integration is a perfectly paired with docker allowing each new release, or push to the repository to be built automatically with a new docker image, and tested to ensure against regression.

Virtual Machines aren’t docker

Docker and virtual machines differ in their implementation and uses cases. They are complimentary rather than competitors and service different needs through each use-case.

Containers leverage the base operating systems kernel whereas virtual machines initialise each machine with its own hardware virtualised operating system. Although docker utilises the host’s operating system, its application are seperated from the kernel ensuring isolation via the docker engine.

Regardless, each technology should be utilised as a method to achieve an outcome rather than be a dogmatic position for any one developer to live and die by. In fact, both methods can be used in conjunction, it is possible to run docker contianers within virtual machines - this being something I do often in Vagrant.

Docker processes

An important distinction with docker is its process orientation. Containers are designed to execute one process and then quit - the length of the running process arbitary but once its is complete the container will effectively exit.

this is an important point for new docker users. It contrasts heavily from a virtual machine which will live forever, sitting idly doing nothing until instructed.

For example, a contianer may run a command line application that fetches the weather from your local area. Once the container has finished this process it will stop and cease to draw any resources until manually called again by the user or application.

How does docker work

docker container run -d -p 8080:80 --name myapache httpd

There is two main concepts to grasp with docker; images and containers.

images

Images can be thought of as virtual machine snapshots - a point in time that contains an application, its dependancies and files. Unlike a virtual machine this “snapshot”, or image is immutable and static. It serves as a blueprint, rather than a dormant virtual appliance waiting to be brought back to life.

To sum, images are what we then create containers from and ensure that each launched container will be identical to the base image.

containers

Containers are the running applications brought to life from a docker image. This is the process which does some work, be it short or long lived. At their essence, containers are meant to be disposable, existing only to complete their task and then exiting.

Principles of containers:

Immutable
Disposable
One process per container

Demo #1

Let’s start a docker application.

docker run --rm danielmichaels/http-tracer http://nyti.ms/1QETHgV

You should see something similar to the following:

Unable to find image 'danielmichaels/http-tracer:latest' locally
latest: Pulling from danielmichaels/http-tracer
c87736221ed0: Already exists
9dc197b2c846: Already exists
02f2755d81e6: Already exists
012e932ae3a4: Pull complete
Digest: sha256:b463068aec6a475854277d8e0a485281166d1fa18eb7c9c8e09cb858e3031684
Status: Downloaded newer image for danielmichaels/http-tracer:latest

Docker always checks the local machine for any images matching the request. If it does not find any it will check its upstream repository for any matches, in this case, Docker Hub.

If found, docker will then start to pull down the container’s image. Once its finished downloading, docker executes the docker run command. Try it out.

In this example, we’ve downloaded a script that executes once - printing information to the terminal and then exits. The next time the container is called, it finds the cached image and spins up the container much faster. The only slow down being the scripts IO bound process.

Let’s look at a long lived process, such as a webserver.

Demo #2

docker run --rm -it -p 8081:80 nginx

Again, an image will be downloaded (don’t worry we’ll go over deleting images) before it finally runs the container.

This time, nothing should happen and it may appear to have hung. In a browser, navigate to localhost:8081 and observe the terminal where the container is running. Requests will be printed to the screen.

If you are wondering about the -p flag, it is shorthand for --publish and is the mechanism for mapping the host and container port. The host port is first, container second or -p <host>:<container>. For nginx, it expects port 80 so we set the container to that however the host port could be set to any realistic port.

So we should have the nginx default page in the browser, and the request for that page in our terminal. This is a good example of how docker processes can be long running - the webserver is a process and it’s still running.

We can also make this process run without printing anything to the screen by omitting the -it and replacing it with -d or --detach. The terminal should output a long alphanumeric string indicating its container id. Confirm this with docker ps.

To stop detached containers use docker stop <container id|container name>. Again, the id or name used to kill the process should be echoed to the terminal. A docker ps will confirm this.

Lastly, the --rm flag simply removes the non-dead container from the host filesystem. For our purposes, we do not need or want to keep copies of our containers on the system. Generally, its a good idea to keep them for debugging but can really start to absorb a lot of disk space on the host during testing. To check for this run docker ps -a which will print out any old containers that can be cleaned up. To clean away these container use the prune command: docker container prune.

Bind Mounts

So far the utility has lacked any real substance. I’ve got a web app, how does my code get on here?

There are two methods to mount data to contianers; using --volume/-v or --mount. Docker recommends using the --mount method for a few reasons, namely the ability to share mounts between contianers.

Using `-v`

Short running container

docker run \  # run docker
--rm \  # remove container once done
-it \  # interactive mode
-v $(pwd):/src python:3 \ # volume to mount with container
python hello.py # process to run; python and script located in the volume - /src

Long running container

docker run \  # run docker
--rm \  # remove container once done
-it \  # interactive mode
# replace -it with -d to detach and run in background
-v $(pwd):/usr/share/nginx/html \ # volume to mount with container
-p 8080:80 \  # port host:container
nginx:latest \ # image to run with tag (latest version)

bind routes -v put area on local to area on container. docker run -d -p 8080:80 --name nginx-bind -v $(pwd):/var/share/html nginx

and `--mount`

This method is more verbose than -v but much more powerful. Because we are using key:value pairs the order is not important.

From the documentation:

The type of the mount, which can be bind, volume, or tmpfs. This topic discusses bind mounts, so the type is always bind.

The source of the mount. For bind mounts, this is the path to the file or directory on the Docker daemon host. May be specified as source or src.

The destination takes as its value the path where the file or directory is mounted in the container. May be specified as destination, dst, or target.

The readonly option, if present, causes the bind mount to be mounted into the container as read-only.

The bind-propagation option, if present, changes the bind propagation. May be one of rprivate, private, rshared, shared, rslave, slave.

The consistency option, if present, may be one of consistent, delegated, or cached. This setting only applies to Docker Desktop for Mac, and is ignored on all other platforms.

To illustrate we are creating our volume (a bind mount) in our current working directory. Inside the directory we have a python script which runs print('--mount!') and nothing more. The file is called mount.py. The structure should look like this:

# tree .
.
└── mount.py

0 directories, 1 file

To mount this file and make it executable via docker we use the --mount option.

docker run --rm --mount source="$(pwd)"/,destination=/apps,type=bind python python /apps/mount.py

Breaking down the --mount starting with source. This is the location of the data to be “loaded” into the container, most often from a local machine. The unix command $(pwd) returns an absolute path, which mount requires—it cannot accept relative paths. This effectively tells docker to put anything within that directory into a volume and attach itot the container.

Next is destination which can also be called as target. This tells docker to place the source data inside this location within the container. So if I was to set destination=/shazwazza, then the data would be located in the / as /shazawazza.

Lastly, in this example we are setting type=bind. This is the most simple method, though less extensible than volume mounting. For small independant applications it is often best to at least start with a bind type and then when ready move onto volume. See here for detailed information regarding the volume mounting, and when you should use it.

Useful docker commands

docker info

Displays system-wide information

docker inspect <image/contianer>

Return low-level information about a container or image

docker container ls

Returns currently running containers

docker ps

Shorthand for docker container ls. Appending -a will show containers that have exited.

docker image ls

Prints all images on the local machine that have been pulled down from docker hub.

docker pull

Will download a docker image onto the local machine

docker run

Runs a docker container, and if not found on the local machine will attempt to download it.

docker flags

--rm: remove container from list once process exits
prune: remove any old or “dangling” images, containers or networks. Good for deleting old cached builds no longer in use. E.g. docker container prune
exec: docker exec -it myapache bash executes the bash shell for an interactive session.

Dockerfile

So far, we have only pulled and ran docker images, or added a mount to a local container. That is useful but dockers biggest selling point is being able to create and bundle our own applications into a container and allowing others to use them on other systems. To do this, we use a Dockerfile.

From Docker:

Docker can build images automatically by reading the instructions from a Dockerfile. A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession.

When creating a docker image, we most often want to pull from an existing image within a registry such as Docker hub and extend it.

Example

# simple example
FROM jfloff/alpine-python:3.6-slim

ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

RUN pip install http-tracer

ENTRYPOINT ["http-tracer"]

CMD ["pfsense.org"]

The above Dockerfile, has some keyword’s denoted by capitals, followed by an argument. This is the general structure of a Dockerfile and the layout is very important as we will see below. There are many more keywords but that’s reserved for your own experimentation.

All dockerfile’s start with FROM - which basically means “pull this image and extend it with the following..”

In this particular application, we need to ensure the languages are set correctly. Dockerfile’s allow users to set Environment Variables with the ENV keyword.

Perhaps the most important and well known command is RUN which tells docker to call the following commands within the container. In this case, as we are building atop of an alpine-linux minimal python version, pip comes bundled with it. If the image was ubuntu based we could call apt-get and so on.

The next two lines (layers) are special to this commandline application. ENTRYPOINT defines the command we want to execute within the container and because our app, http-tracer is installed in the PATH as apart of its pip installation, it is executable. Typically, ENTRYPOINT is used in contianers that are designed to be run as an executable, which this application is.

CMD is similar to ENTRYPOINT except we can only use it once and it is used to set default commands or parameters. the argument that follows a CMD keyword will run anytime the container is executed without specifying any arguments with it. So in this case, out http-tracer container will be executed with "pfense.org" as its default argument. If the container runs with a command, this default will be overwritten.

More simply, to allow user input;

The ENTRYPOINT specifies a command that will always be executed when the container starts. The CMD specifies arguments that will be fed to the ENTRYPOINT

confused?

Calling our docker image without any arguments will result in the following:

Whereas if we set a argument of “cisco.com” it will overwrite the default and return data relating to that parameter.

Another example worth looking at is the ping command. You can execute that with a dockerfile containing the following:

FROM debian:wheezy
ENTRYPOINT ["/bin/ping"] # executes when container is run
CMD ["localhost"] # executes unless user input supplied

layers

In a Dockerfile, each line is a layer.

layers cannot shrink the size of the image
layers are cached (checks the Dockerfile lines for differences)
caching means it does not need to be rebuilt, only new layers
container is a layer, one that exists only while the container lives:
- the build layers are read only - cannot edit them and are shared across many containers
- if the image needs updating, all layers must be rebuilt
container layer:
- read write
- modifications of source, or volumes within the container exist only within that container unless rebuilt into a new image
image layer:
- read only

docker networks

Docker also provides a networking layer, allowing many contianers to be on a local area network. If this is not setup, each container is essentially within its own virtual LAN and cannot communicate with other containers. This is a big topic so read up on the docs [here].

The basics are as follows:

to create a network for a mysql database and node application to communitcate run: docker network create <name>
see that your network has been created with docker network ls
inspect your network with docker network inspect <name>

To add containers to a network, and test they have connectivity with ping is just a matter of adding them to the network with --net.

# create mysql on docker network named 'test'
docker run --rm -d --net test --name test_mysql -e MYSQL_ROOT_PASSWORD='root' mysql:5.6
# create node linking to the 'test' network and spawn a shell
docker run --rm -it --net test --name test_node node:8 /bin/bash
# once they've been built it will drop into a shell
# test the network connection by trying to ping test_mysql
ping test_mysql # from node /bin/bash shell
# unless explicitly denied, this gives access to internet
ping 1.1.1.1

Docker enough to get by

That is really enough docker to start playing around and learning to be comfortable in it. This hasn’t covered docker-compose which is another powerful tool, but having a baseline understanding of docker is paramount. The whole point is to learn slower than you think, and be exposed to new technologies over a longer period of time - rather than glossing of all the new shinies and not really learning anything. It’s okay to be slow - its all time under tension and repeated small exposures that eventualy lead to true understanding.

Defending your apps

Sat, 19 Oct 2019 00:00:00 +0000

Web Application Defence Mechanisms

one rule; do not trust user input.

Core defences

The methods by which we can defend against attackers.

Handling user access
Handling user input
Handling attacks or ensuring that the application works as intended when being directly targeted, and
Handling the application itself so everything can be monitored, and configured correctly

User Access

In most applications that have users, there will be several privileges or roles within the user space. For instance, a social-networking site might have the base user, a group administrator and overall application administrator accounts. Likewise, the owner of an account should be able to view their own data but not that of others. Mitigation’s, or security mechanisms in place within this spectrum are:

Authentication,
Session Management,
Access Control.

Authentication

The simplest to understand; a method to check the validity of a user - username and password. Authentication can scale appropriately with the applications need, such as MFA. Many authentication mechanisms also have password resets, account recovery, and self-registrations. All of which add to the attack surface.

Session Management

On busy web applications there can be several other users sending requests simultaneously to the server requesting identical things. For instance, hundreds of people may be viewing the same page on amazon and adding an item to their basket whilst browsing without being logged in. The application must keep track of theses requests and be able to distinguish each unique visitor.

This is done via tokens, or sessions that track each user and are stored on the web server. The server then forwards the unique token to the client, which then returns that token in each subsequent HTTP request. This allows both sides to session track. Cookies are one such method for session management.

Common attacks against session management is stealing tokens, guessing tokens or exploiting defects in how the token is used to authenticate the owner. Stealing a session token could allow an attacker to use an authenticated users token effectively allowing them to masquerade as that user.

Access Control

Simply put, the applications ability to assess whether a user has the required Bona-Fides to gain access to the system resources they are requesting. This often requires some fine-grained logic and thus can be exploited if developers make too many assumptions about how a “user” will access the resources.

User Input

All user input is untrustworthy. Therefore, applications must handle all input with strict enforcement of this rule. Sanitation and restriction of inputs and reducing assumptions about what a “user” would submit are necessary precautions. But, protecting against all forms of malicious input is a difficult task and many applications are vulnerable to this.

Varieties of Input

Limiting what a user can input is common practice. For example, a user sign up page may use an email account that requires the use of an @ symbol and must only contain alphanumeric letters with a minimum length of four characters. Other examples may be the requirement to submit a textual paragraph that allows all alphabetical characters plus special characters but will not allow HTML tags. Many other possibilities exist such as markup languages and how they are delimited so as not to allow arbitrary code to be run.

Approaches to Input Handling

Reject Known Bad

Blacklisting known bad things. The weakest form of defence, as spoofing or encoding arbitrary code blocks could slip past this measure.

For example, if SELECT is blocked, SeLeCt might not be! or if alert('xss') is blocked maybe prompt('xsss') isn’t. In addition, some tokensing parsers which detect these blacklisted words may stop searching when the detect a NULL Byte. %00<script>alert(1)</script> may get through.

Accept Known Good

This method is very effective against stopping code injection as it is based off only allowing accepted elements to be input. The flaw in this is where accepting a known good can conflict with the requirements of the application itself. For instance, a name field in a form will likely necessitate the use of -, as some people legitimately have hyphenated names. But, - are commonly used to attack databases on the backend, so it may not be used in all cases where it could detract from the user experience.

Sanitisation

A common defence mechanism that relies on the fact that users will try malicious code, so before processing the application will sanitise the input in various ways. Methods such as removing dangerous characters or escaping them before processing. This is something all applications should do, and their are many libraries in every language to support this.

Boundary Validation

Simplistically, this is the process of validating, or checking user input on the periphrases first prior to the input reaching the backend. Think taking dirty input on the web side and cleaning it before sending to the server side, and therefore trusting all server side code to be “clean”.

This has some serious faults. Often attackers can exploit such a scheme by chaining exploits in a way that no validation method could feasibly defend against without serious detriment to the overall functionality of the application itself.

The best solution to defending against this, is boundary validation where for each layer of processing, a validation check is conducted appropriate to that layer; defence in depth.

Handling Attackers

All applications should be of the mindset that they will be targeted and attacked by dedicated and skilled attackers. With this mindset, the developers must have a methodology for handling and monitoring such incidents. These measures should include:

Error Handling,
Logging,
Alerts,
Reacting to attackers.

Handling Errors

Robust testing of the application prior to launch should be conducted, and if done properly identify many bugs and errors in the design and code base. But, not all issues can be counted for in testing and as such there must be a way to capture and respond to errors in the application.

Debug and system messages should never be presented to the outer layers - they greatly assist the attackers in identifying weaknesses in the application. All errors should be gracefully raised, and catered for so as to not give away too much information to the attacker.

Audit Logs

All activities taken on the system should be logged for analysis later. Items that definitely require logging are:

All authentication actions; failed, successful, and forgotten or changed passwords,
Transactions such as payments,
Activities blocked by access control mechanisms,
Any attack strings such as malicious input.

It is important that these logs are protected against unauthorised read and writes. This can sometimes be achieved by the use of autonomous systems that only accept input data from the servers.

Alerting Administrators

Audit logs are retroactive; they allow a detailed look into the past for forensic analysis but provide little real time information. Alert mechanisms that should be brought up to the administrators attention are things like:

Anomalies, such as a large number of requests originating from a single IP,
business anomalies like unusual funds transfers or out of hours actions on a machine,
requests containing attack strings,
requests which data is hidden with the intent to obfuscate the true intent of the data, or request.

Reacting to Attacks

Alerts signifying an attack is nothing without a response. Reactive approaches will should be unique to the application in question but many are generic. Things such as slowing or putting up things to block simple attacks will not defeat a prudent and patient attacker, but should stop a lot of automated broad brush attacks.

Summary

The above methods for protecting web applications are nothing new, in fact, much of this content is taken from old resources such as the web application hackers handbook. That is because the advice still holds true even in today’s “modern” web app’s. Attacks and methods may change slightly, or evolve to overcome new frameworks securities mechanisms but the vast majority of attacks simply exploit what could be defended against by implementing whats contained in this post.

Things don’t change that much.

Dropbear and AWS

Tue, 10 Sep 2019 00:00:00 +0000

BusyBox, SSH and EC2

Accessing an EC2 instance from BusyBox’s Dropbear SSH client isn’t easy. Firstly, .pem files are not compatible with dropbear, nor can you convert them to dropbear’s key format with the built-in dropbearconvert. Secondly, depending on your version of openssh it may not be immediately apparent that your private keys are incompatible with the conversion application either. Thankfully, workarounds are possible.

Dropbear SSH

Dropbear is a lightweight client and server application mostly seen on embedded devices. It is designed to replace OpenSSH in low memory footprint systems as it can be compiled down to 110kb. It is compatible with .ssh/authorized_keys, however it does have limitations.

Those limitations

Amazon Web Services such as Elastic Cloud Compute hand out private keys for passwordless login. Unfortunately for dropbear users, the format is .pem which is incompatible. And, dropbearconvert - the program provided for turning openssh keys into dropbear keys - does not accept .pem files.

Dropbear’s conversion tool will also not accept private keys from openssh, as seen below.

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEAtNY9pkKAhGYD/qDIThmRd7Y8kMo7QvbSQj+tv5huz4hTqdgALmxP
....
FBuu7Rt2qQdJPBAAAADGRhbmllbEBBbm9hdAECAwQFBg==
-----END OPENSSH PRIVATE KEY-----

Instead, it must be an rsa key like so.

The workaround

I found the most reliable means to gain access to EC2 from dropbear on busybox was to create the keys elsewhere and then move them to the device manually.

The steps

1. Create a public and private key on either the AWS EC2 instance, or your local machine with:

ssh-keygen -m PEM -t rsa -b 4096 -f <new_privkey>

2. Copy the private key to BusyBox:

scp <priv_key_filename> <user>@<busybox>:/<user>/.ssh/

3. Copy the private key to the cloud - only if doing these steps on your local system:

scp -i <aws-ec2-private-key>.pem <new_privkey> ec2-user@<ec2-dns-address>:/home/<ubuntu>/.ssh/

4. On the system you are running these steps, inside `.ssh/authorized_hosts` append the public key:

from local machine:

cat <new_pubkey>.pub | ssh -i <aws-ec2-private_key> | "cat >> /home/ubuntu/.ssh/authorized_keys"

inside ec2:

cat <new_pubkey>.pub >> .ssh/authorized_keys

5. This will convert our rsa key into a dropbear compatible key

Login to busybox
dropbearconvert openssh dropbear <new_privkey> <new_privkey_dropbear_compat>

ssh -i <new_privkey_dropbear_compat> <ec2-user>@<ec2-dns-address>
profit

Get to work

We can now access our AWS server from our busybox device. From here we can do whatever we need to do such as establish a reverse tunnel for administration 😉

Curl you an email for great good

Mon, 12 Aug 2019 00:00:00 +0000

Curl

Today I learnt that Curl can also be used to send emails. One of my projects is to manage a fleet of embedded devices that communicate information back to our servers. Email is one part of this process. And, curl is installed on these devices, greatly simplifying things.

Syntax

# gmail specific
curl --url 'smtps://smtp.gmail.com:587' --ssl-reqd \
--mail-from 'username@gmail.com' --mail-rcpt 'rcpt@email.com' \
--user 'username@gmail.com:password' --insecure \
--upload-file fileToBeUploaded.txt

Let’s break down each component:

--url - specifies the SMTP server address and port
--ssl-reqd - mandates SSL/TLS must be used
--mail-from - who sent the email
--mail-rcpt - the addressee
--user - credentials for the mail server
--insecure - a curl idiom that tells curl to ignore CA errors and continue
upload-file - curl cannot send a body, but will accept an RFC 5322 compliant text file as its message. Json, HTML and text data can be sent by adding a Content-Type header appropriate to the type of data being sent.

Curl documentation.

Bash script

A simple example bash script to illustrate some of the potential that this curl functionality provides us. The code can be found here.

#!/bin/bash
# A simple email script that takes a recipient and
# RFC5322 compliant text file as the message body.

echo "Curl auto-emailer"
echo "SUPPORTS GMAIL ONLY"

RCPT=$1 # 'username@example.com'
FILE=$2 # takes a filename, relative or abs path
FROM=server@gmail.com # 'me@me.com'
EMAIL=server@gmail.com:p@ssword # 'username@gmail.com:password'

if [ -z "$1" ]
 then
 echo "Missing argument 1; recipient address\
 eg username@gmail.com"
 exit
elif [ -z "$2" ]
then
 echo "Missing argument; File to be uploaded"
 exit
else
 curl --url 'smtp://smtp.gmail.com:587' --ssl-reqd \
 --mail-from $FROM --mail-rcpt $RCPT \
 --user $EMAIL --upload-file $FILE --insecure
fi

This is for demonstration purposes only. Also, its worth mentioning that entering credentials in the commandline without the appropriate safe guards can be a security issue!

HTTP Methods Primer

Thu, 11 Jul 2019 00:00:00 +0000

HTTP 101

Prefetch

What happens when you type an address into the search bar and hit enter:

The browser extracts the domain name from the search bar.
The users Domain Name System (DNS) cache is queried. If no previous entries are found a DNS request is sent via your configured DNS server.
After getting the IP address via DNS, the host will initiate a TCP connection with that address.
If the connection is successful, a response will be sent from the server and its data will be rendered using HTTP.

For a more detailed explanation of DNS resolution in step 2, check out my post on the subject.

HTTP

Hyper-Text Transfer Protocol (HTTP) is an application layer protocol for communications between a client and web server. It operates on a request-response model with the client initiating any connection to the server.

HTTP Methods

Each request starts with a HTTP method which defines the request semantics, indicating the purpose of the clients request and what the client expects in a successful response. More succinctly, the method tells the server what sort of data it is sending or wants to receive. The server can then respond appropriately to the request.

The nine methods available are listed below.

GET

The most common request, it broadly asks the remote host to send the requested data back to the client. In general, GET requests will retrieve the data from a Uniform Request Identifier (URI).

Responses to a GET request are cacheable, meaning a cache can elect to use the it on subsequent GET and HEAD requests.

HEAD

Very similar to the GET request except that it only retrieves the header information. All body content is omitted.

Responses to a HEAD request are cacheable, meaning a cache can elect to use the it on subsequent GET and HEAD requests.

POST

A POST request will typically be sent via a HTML form and will effect some change on the server. According the specification it is indicated by the Content-Type header.

Common POST actions are things like adding new users, or payment processing. The POST sends that information to the server for it to be processed and declares the scheme it will use to send the data. The server needs this information so it knows how to handle the request.

Schemes:

application/x-www-form-urlencoded
multipart/form-data
application/json

Example:

# Request
POST /payments HTTP/1.1
Host: bank.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 34

accountNumber=123&amount=9999&toAccount=789

Successful POST requests should get a 201 Created response and a Location header pointing to the data’s new URI.

# Response
HTTP/1.1 201 Created
Location: /payments/<ID>/<transactionID>

PUT

Similar to a POST request, a PUT references an already existing entity, and requests a update to it. The server must verify the validity of the data to be updated, and is not obligated to action the request.

Every PUT is idempotent whereas POST requests are not. Each POST will have an updating effect on the data. For instance, repeatedly sending a POST on endpoints like payment processing could see that payment processed as many times as the request was sent. This is one of the differences between the two at the HTTP level. But, the most important difference is how the each request interacts with the URI it is requesting.

A PUT request must know the exact URI it intends to influence.

For example:

# Updates an existing record 
# OR adds a record to that exact URI
PUT /api/v1/books/3452 HTTP/1.1
Host: https://library-store.com/

Should return a 200 OK or in some cases a 201 Created.

REST API’s usually adhere to the correct use of PUT and POST requests. A URI such as above is a good indication of a REST conforming web application.

PATCH

Unlike PUT which will replace an existing entity, PATCH only modifies a portion of it. This makes it an non-idempotent method. Similar to POST, it will make alterations or add data for each request. However, a PATCH can be made idempotent - it depends on context and the data being modified.

This method is relatively new in comparison to its siblings, having been introduced as early as RFC 2616 but only being ratified in RFC 5789 circa 2010.

# Request
PATCH /file.txt HTTP/1.1
Host: www.example.com
Content-Type: application/example
If-Match: "e0023aa4e"
Content-Length: 100

A successful response generally yields a 204 No Content as no message body is present. A 200 OK requires a message body. Other codes can be configured by the server if required.

# Response
HTTP/1.1 204 No Content
Content-Location: /file.txt
ETag: "e0023aa4f"

Note how the If-Match and ETag data match. This will be covered in a later post.

DELETE

Is a request to the server that calls a function capable of deleting a specified entity. The server is responsible for checking and carrying out the function.

# Request
DELETE /user/2341 HTTP/1.1

Should return a 202 Accepted, 204 No Content or 200 OK.

TRACE

An uncommon request that is often seen during diagnostics such as Traceroute. Can see what the server was delivered.

Often this method is disabled as its an attack vector for Cross Site Tracing.

CONNECT

For use with a proxy to initiate the connection.

OPTIONS

A utility call that asks the server which HTTP methods are supported.

The allow header shows which options are authorized. Some applications prohibit the OPTIONS method and will return a 4XX status code.

HTTP/2 200 
allow: OPTIONS, GET, HEAD, POST
cache-control: max-age=604800
content-type: text/html; charset=UTF-8
date: Sun, 14 Jul 2019 05:17:14 GMT
expires: Sun, 21 Jul 2019 05:17:14 GMT
server: EOS (vny006/044F)
content-length: 0

Summary

Digging into the HTTP specifications and RFC’s has not been wasted effort. They underpin the Internet and all associated technologies that power it. Learn some for great good.

HTTP Security Basics

Sat, 06 Jul 2019 00:00:00 +0000

HTTP Basics

HTTP is an application level message based model where the client sends a request and the server returns a response. HTTP uses a stateful protocol - TCP - but is stateless, where each request is connectionless avoiding the need for servers to hold an open connection.

HTTP Requests

The first line of every HTTP request has three (3) items, separated by spaces:

The verb - GET,PUT,DELETE,POST,TRACE,CONNECT,OPTIONS, or HEAD
The requested URL, typically the name of the resource plus an optional query string. Query strings are noted by the ? eg. /auth/448/yourdetails.ashx?uid=345
The HTTP version in use. Most common is 1.1 and will look like this HTTP/1.1

# Example request
 
GET /Protocols/rfc2616/rfc2616-sec5.html HTTP/1.1
Host: www.w3.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://duckduckgo.com/
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

HTTP Responses

Like the request, each responses first line will return three items:

The HTTP version in use,
The status code - 200, 302 and 404 being most common,
The response “phrase” that further describes the status of the response - legacy and not required by browsers.

Responses can contain many more items after the first line, such as:

Server: This may contain information about the server such as what engine it is running or modules installed. It may or may not be correct.
Set-Cookie: Will set a cookie for future use, and it will be submitted via the Cookie header in future requests.
Pragma: Can be used to instruct the browser not the store the response in its cache. Likewise the Expires header says when the is set to cache expire, and if expired it won’t load from the cache.
Content-Type: Almost all responses will contain a message body after the headers (separated by a single blank line). The Content-Type header indicates what is in the message body.
Content-Length: The total size of the response in bytes.

# Example response

HTTP/1.1 200 OK
Date: Mon, 18 Jun 2018 04:29:32 GMT
Server: Apache/2.2.22 (Debian)
Content-Location: rfc7230.html
Vary: negotiate,Accept-Encoding
TCN: choice
Last-Modified: Sun, 17 Jun 2018 07:19:43 GMT
ETag: "3c9fe1-417e4-56ed145dec1c0;56ee301a89430"
Accept-Ranges: bytes
Cache-Control: max-age=604800
Expires: Mon, 25 Jun 2018 04:29:32 GMT
Content-Encoding: gzip
Strict-Transport-Security: max-age=3600
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Clacks-Overhead: GNU Terry Pratchett
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

HTTP Methods

GET

The most common method. When clicking on a hyperlink, you are likely sending a GET request. GET requests can also be used to send information via a URL query string such as ?.

http://example.com/over/there?name=ferret

URL’s are displayed on screen and also logged in places such as browser history and web access logs. They are also transmitted via the Referrer header when clicking on an link that takes you an external resource. In such an example sensitive information in a query string would be logged in an external servers logs. Further, these strings (if unencrypted, or SSLstripped) could be sniffed by a malicious actor. Never use GET requests for transmission of sensitive data.

POST

POST is used to submit data to a resource. A POST can submit this information via the URL query string, or via the message body (preferred method, imo). If the POST is coupled with URL query strings, any bookmarking would remove the query from the URL, in an attempt to provide security. Using this method also ensures that only one request is sent with that information at a time. That is, if a user submits a POST request and then hits the ‘back’ button, they will likely see a prompt stating something along the lines of ‘having to resubmit’. This ensures that the user can only submit the POST once per action, as the browser will not automatically reissue the request - unlike a GET.

HEAD

This functions in the same manner as a GET except it only returns the header information.

TRACE

Used for diagnostic purposes only. The server should return in the response body the exact contents that it received. Often used to diagnose issues and check if proxies are working correctly/ intercepting data unexpectedly. Commonly disallowed option returning 405 Method Not Allowed

OPTIONS

Returns what HTTP methods are available for a particular resource.

PUT

Allows uploading of resources to the server. Used by API’s to update existing records or data. Can provide the attacker with an avenue to upload malicious scripts to the server for execution.

Uniform Resource Locator (URL)

URL’s are unique identifiers for web resources.

# Example URI specification from RFC 3986

 foo://example.com:8042/over/there?name=ferret#nose
 \_/ \______________/\_________/ \_________/ \__/
 | | | | |
 scheme authority path query fragment
 | _____________________|__
 / \ / \
 urn:example:animal:ferret:nose

Within the authority a port number is delimited by :, this is optional and generally only included if not default for the protocol in use.

REST

Representational State Transfer (ReST) is a style of architecture for distributed systems in which requests and responses contain representations of the current state of the systems resources. What is means is that RESTful applications will have their parameters within the path, rather than in the query string.

Non-RESTful

https://example.com/search?manuf=samsung&model=galaxy

RESTful

https://example.com/search/samsung/galaxy

See here for more information.

HTTP Headers

Some interesting and important headers:

General Headers

Connection: Lets the other end know whether to shutdown communication or keep it open for future messages. Defaults to keep-alive in HTTP/1.1 but must not be used in HTTP/2 as header metadata is handled differently due to HPACK.
Content-Encoding: Specifies what encoding is used. gzip being very common as it compresses the data for faster transmission.
Content-Length: Provides the message body length in bytes.
Content-Type: What type of contents are in the message body such as text/html, or application/json. Very important during POST requests in determining the type of data that will be sent in the body, and how the server should expect to parse it. One type is multipart/form-data which will set a boundary= string, allowing the server to accept each field as a new part and separate them according to the boundary defined in the Content-Type. Another method is the application/x-www-form-urlencode type which will place the data from the form into the body of the message, rather than in the query string.
Transfer-Encoding: If any encoding has been done on the message body, typically seen when chunking has been performed.

Request Headers

Accept: Tells the server what kind of content the client is willing to accept, such as image types etc.
Accept-Encoding: What encoding types the server will accept.
Authorization: Submission of credentials to the server for one the built-in HTTP authentication types.
Cookie: Submits cookies that the server has previously issued.
Host: Hostname that appeared in the requested URL.
If-Modified-Since: When the browser last received the requested resource. If no changes have happened since then the server may issue a 304 status code telling it to render from its cached copy. Compared against Last-Modified and determines the cached variant is older or newer than the current resource.
If-None-Match: A sort of hash that the server issues to the browser which it checks to see if is valid prior to telling the browser to render from cache. This specifies an entity tag and it is the ETag that is validated. It cares only if the information is identical or not and takes precedence or If-Modified-Since as the ETag is considered stronger validator. See RFC 7232 for more information.
Referer: From which URL this request has originated.
User-Agent: Information about the browser or application making this request.

Response Headers

Access-Control-Allow-Origin: Whether this resource can be retrieved via a cross-domain request.
Cache-Control: Passes caching directives to the browser.
ETag: The entity tag submitted to server. A special header that specifies the version of a resource. Used to verify if the server needs to send a full response, or if the client can render a cached copy. Each time a resource changes, and new ETag value must be generated.
Expires: How long the contents of the message body are valid for. Cached copies may be used until this time.
Location: Used in redirect responses (3XX) to specify the target of the redirect.
Pragma: Passes caching directives to the browser.
Server: Information about the server’s software.
Set-Cookie: Issues cookies to the browser which it will send back to the server in subsequent requests.
WWW-Authenticate: Provides details on types of authentication that server supports after 401 status code.
X-Frame-Options: Whether responses can be loaded within browsers frame and how to do it.

Cookies

Many web applications rely on cookies as allow servers to send data to the client, and then have the client return that data in its transmissions within the server. Since HTTP is stateless, cookies allow functionality to persist over time. Login data, shopping carts and site settings all rely on cookies for a better user experience.

# Example of setting a cookie

Cookie: tracking=dw34tad23590ddfsawWcsTs

Cookies are usually made up of a key value pair but may consist of any string that does not contain a space. A server can send more than one Set-Cookie per response by using a semi-colon to separate the cookies.

Set-Cookie can also include optional attributes such as:

expires: The date at which the cookie will become invalid. This means that cookies are stored by the browser on disk and may be reused repeatedly until that date. If no expires is set, it will only remain valid for the current browser session.
domain: Which domain the cookie is valid for.
secure: Whether that cookie must be sent via HTTPS or not.
HttpOnly: if this is it set, it prevents javascript from accessing it.

Status Codes

Five (5) groupings:

1xx - Informational.
2xx - Request successful.
3xx - Redirection.
4xx - Request error of some type.
5xx- Server error of some type.

The most common status codes are:

100 continue: Sometimes sent when a server receives a request containing a body. It means that the headers were received and the client should continue sending the message. Once all is received a further response will be sent.
200 OK: Successfully received.
201 Created: The request was successful and a new resource was created. Typically the result of POST and PUT requests.
301 Moved Permanently: Redirection to the new URL of the requested resource. A Location header will specify the new resource location and that it should use that in future.
302 Found: Redirection to a temporary URL, again specified in the Location header.
304 Not Modified: Instruction to browser saying it should use its cached copy of the requested resource. If-Modified-Since and If-None-Match headers are sent to determine if client has most recent copy.
400 Bad Request: Invalid HTTP request was received.
401 Unauthorized: Invalid credentials or none were supplied. A reply with WWW-Authenticate headers will be sent stating what authentication types are supported.
403 Forbidden: No one is allowed access.
404 Not Found: Resource does not exist.
405 Method Not Allowed: Method used is not supported by the specified URL. Both HEAD and GET must never be disabled, and should not return this code.
413 Request Entity Too Large: Some endpoints or browsers will limit the string length - usually seen in buffer overflow attacks.
414 Request URI Too Long: Same as above.
500 Internal Server Error: The server hit some error processing your request. A good hint that you are trying to exploit something it is not configured to handle properly. Always try and identify the nature of this error.
503 Service Unavailable: An indication that the server is responding but the application is not.

State and Sessions

Although HTTP is stateless there is a requirement for the client and server to process long running requests. An example of this is a sites Shopping Cart. A user can browse and add items to their cart whilst making several requests over a period of time. To make this possible the server maintains a User Session, which allows it to track the users actions across the site.

In some cases, state information is stored on the client side. Instead of the server storing the session, now the data is sent to the client in each server response and the client will send it back in each request. It is important that the server does not trust the client, as it may alter the session information for nefarious reasons. One mitigation is using a hash of the state, which the server validates before accepting the clients session data.

Given the stateless nature of HTTP, applications need a method in which to re-identify clients. Typically, this is achieved through the use of tokens - a unique identifier for each user such as cookie.

Encoding Schemes

HTTP has several encoding schemes to ensure the safe delivery of data. The following is some of the most common.

URL Encoding

URL’s can only contain characters from the US-ASCII set; 0x20 through to 0x7e. Several characters within this set are restricted as they have special meaning in the scheme or HTTP. All URL encoded characters are prefixed by % followed by their hexadecimal representation of the ASCII character.

Examples:

%3d — =
%25 — %
%0a — New Line (\n)
%00 — Null Byte

note: + represents the ‘space’ in a URL.

Unicode Encoding

Unicode is the system that allows us to use many different character sets. For English speakers, utf-8 will be the most common, but many more exists. Encoding 16-bit unicode in a URL requires the prefix %u followed by the hexadecimal.

Example unicode

%u2215 — /
u00e9 — é

UTF-8 is a variable length encoding that uses one or more bytes for each character. To send this using URL encoding, each byte is delimited by a %.

For instance:

%c2%a9 — ©
%e2%89%a0 — ≠

Unicode is an important part of attacking web applications as it can defeat input validation schemes.

HTML Encoding

Safe incorporation of HTML within web applications is important. Several HTML characters have special meaning and must be encoded correctly. When attacking an application, HTML encoding will be most evident when probing for XSS vulnerabilities. If an application returns user input unmodified, it is likely vulnerable.

Base64

Base64 allows any binary data to be represented as using only ASCII characters. It permits the sending of ASCII strings over the wire for safe reassembly on the other side in the original format. It is also used heavily in basic HTTP user authentication.

Base64 splits the bytes into 6bit streams, allowing for 64 possible permutations - each chunk of 6 bits allows for 64 possible characters. It allows only the following characters:

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789/+

If the final block results in fewer than three chunks (16 bits or 2 bytes) then it will be marked by = or == which says that their may be one or two trailing blocks of zero’s.

Base64 is prevalent across the web and is often used to transmit binary data within cookies and other parameters. It is often used to obfuscate data in transit through security by obscurity. Always decode any intercepted Base64 data, it could be a goldmine. Base64 can often be identified quickly by the = trail, or if it is JSON it will start with ey which represents {.

Fin

HTTP is an important protocol not just for security and network engineers but for developers too. Having a cursory understanding aides every one who uses the web. Next time you look at the network tab in your browsers developer tools be sure to look at the headers, you might notice something worth exploring.

User Agents 101

Wed, 15 May 2019 00:00:00 +0000

User agents

What

Every browser sends with its request, a string that says who it is. That string will contain information on its application type, operating system, and software version currently in use.

Heres mine: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0 but it could just as easily be Mozilla/5.0 (PlayStation 4 4.71) AppleWebKit/601.2 (KHTML, like Gecko).

If you’ve ever been surfing the net on your mobile and selected “Request Desktop Site” then you have changed your phone browsers user agent. And, if you have done that you might have noticed a significant change in that sites resolution. Its all to do with that string of data, and what the website does with that information.

Not only will each browser have its own user agent but so will any program used to query a web site. Google uses a series of “spiders” which crawl the internet and index it giving us Google search. Googlebot, cURL, wget, Postman and your latest smart light bulb all have user agents.

You can find yours by typing “user agent” into DuckDuckGo. Or, you could open your browsers Development Tools with Ctrl + Shift I and then select the Network tab. Once inside look for a subsection named Headers. Your user agent will be located in the request headers section.

Why do I need to know this

Knowing is half the battle. My bank once wouldn’t let me login unless I used Chrome or Edge which is ridiculous. I choose my browser not the web application, so I just changed the user agent, and wouldn’t you know, …I got in.

Maybe you are an app developer, or for some reason need to verify something by assessing the user agent. At least now you know that it isn’t a fool-proof way to confirm a device - you will need more than one method to ascertain the true origin. Don’t rely on user agents alone as you can see below.

As a developer, or user if you repeatedly request a website it is possible to be served a HTTP/1.1 429 Too Many Requests. It’s probable that your user agent in conjunction with your IP address has triggered that.

Web scraping is a common programming task, and allows developers to access information from sites that do not offer an application programming interface (API). Some pages offer API’s but a programmer may choose not to use it. Usually this is to circumvent having to pay for it, or maybe the API does not offer a certain, wanted piece of information. More often then not, scraping is in violation of the web applications terms of service. It’s best to utilise an API whenever possible.

So how do people get around this?

It’s just a string

Thankfully, HTTP is stateless (yes, even HTTP/2) and thus we can influence each request and its headers.

We can demonstrate this with a simple script which will choose a user agent at random from a list and then send it with its request. Every language has the ability to get information from the internet and will likely already have a way to do the same thing. This example will be in python using the requests library.

Step 1. Set the user agents to be selected randomly.

import random
import requests

def user_agent():
 user_agents = ['Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)',
 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',
 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)',
 'Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko',
 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
 'Mozilla/5.0 (Windows NT 5.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
 'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36']
 agent = random.choice(user_agents)

 return agent

Step 2. Call the URL with our randomly selected user agents.

def check_ua():
 url = 'https://www.whatsmyua.info/api/v1/ua?='
 resp = requests.get(url, headers={'User-Agent': user_agent()})
 ua = resp.json()[0]['ua']['rawUa']
 return ua

We override the default headers with our user_agent function and are returned a response. In this case we’ve hit an endpoint that will return our user agent as a JSON object.

The source for this script can be found here

So changing a user agent is that easy. Firefox has extensions, Chrome would, Opera has it built-in. PC, laptop or mobile handset its all the same after all its just HTTP.

LessPass: A Primer

Mon, 15 Apr 2019 00:00:00 +0000

LessPass

The manager-less password manager

LessPass, unlike other password managers does not store any information in a database or local cache. It simply takes in three pieces of information and then hashes that to return a password. If all three pieces of information haven’t changed LessPass will always return the same password.

Why use it over other choices

Straight from the creator himself:

How do I synchronize this file on all my devices?
How do I access a password on my parents’ computer without installing my password manager?
How do I access a password on my phone, without any installed app?

For some of these I have my own workarounds but they are hacks and not very robust. For instance, I upload my password database to a cloud service and then pull it down on to my phone. It’s a pain, not user friendly and exposes that database to a third-party vendor.

LessPass is also Free as in free beer, open source and secure. It has a command line implementation, a web application, android app (sorry iOS) and browser extensions.

I would also add another reason to use it, its easy and what is easy gets used. Good luck getting your dad to download an application that requires you to go to github, clone the repo and then run an install script or bat file. Even worse, good luck explaining to dad why 1Password is worth the 5 bucks a month or whatever it is. The smallest barriers to entry can be enough to be prohibitive when the user isn’t computer savvy.

How does it work

LessPass returns a unique password that it derives from three (or four, more on that later) pieces of information; site, login and a master password. If they are same it will always return the same output, a great example of idempotence or a pure function.

An image from the creator’s Medium post giving a great visual representation of what is happening under the hood.

The ‘how’ of LessPass and it’s ease of use.

It also allows the user to set password rules such as using capitalised letters, special characters and numbers. So if certain websites require that the user not use special characters, LessPass can be configured for this. It defaults to 16 characters but can be set to a maximum of 32.

You are also able to set whats called a counter meaning if you do not want to change your master password but are required to update your password you can issue a counter. It is an effective way to keep the three main pieces of information the same yet result in a new hash. Granted, this requires another level of cognitive load if you need to track several of these counters across sites.

There must be a better way

LessPass Profiles

The application also offers something called a profile which allows you to store such information. This requires a sign up to LessPass which is free, or you can host your own docker image for extra security. Each site gets its own profile containing everything except the master password and the generated password.

Each profile is stored as a json object like so:

{
 "login": "danielmichaels",
 "site": "www.github.com",
 "lowercase": true,
 "uppercase": false,
 "symbols": true,
 "numbers": true,
 "counter": 1,
 "length": 20
}

Command Line Interface

I really like the CLI for LessPass - it suits my workflow by allowing me to set profiles on my machine via alias. I refer to this as a ‘hardcoded’ approach - it helps me none when I’m not on my box but I can just as easily hit lesspass.com for it in such a case so it’s hardly a show stopper. As example this is how I grab my github password.

alias ghpw="lesspass -c github.com danielmichaels"

This drops me straight into a prompt for my master password and then copies it to my clipboard. I could go further and set an environment variable but I have my dotfiles on github and would rather not accidentally expose such information.

I Don’t Need A Password Manager

Said someone just waiting to get owned by a credential dump. We humans just simply cannot remember suitably difficult passwords that are hard for a computer to guess and easy for us to remember.

Password manager public enemy number one is; if my master gets cracked they own everything. Yep thats true. If my house key gets cloned they’ve got unrestricted access to my house too. The mitigation lies in defeating the most likely avenue of exploitation and that is password attacks from credential dumps hoping to strike gold from someone that reuses their password on several sites. The bad guys are looking for the ‘path of least resistance’, just don’t be that path! Using a password manager and some common sense will go a long way to protecting yourself online. It’s not a panacea but its better than the alternative.

On teaching

Sat, 15 Dec 2018 00:00:00 +0000

on teaching

Recently it has befallen upon me to tutor my peers in programming concepts and fundamentals. It struck me that I don’t know how to properly impart my knowledge. Where do I start, with strings, or variables. When do I talk about types and equality. Is it better to teach while or for loops first? Classes, OOP, recursion, comprehensions, async! What is important now versus what is the minimum necessary to give them the basics required to go away and learn on their own. In the end, I found it does not matter. Start, and progress from there. Listen to them, and discuss things; dont’t dictate. Pay attention to what they are asking, and try your best to make the experience relevant for them.

They are there to learn from you because you know something they don’t. Questioning whether you are ready to teach is just self-doubt. Feeling like you are a fraud, or that you will be outted as a pretender is that doubt manifesting itself within you. We have to let go of these feelings and be ready to engage with the community. No one knows everything, and you aren’t teaching someone who alreadys knows what you know. It’s also okay for a “student” to teach you something too, they aren’t trying to make you feel foolish so you shouldn’t take offence, or let it feed that monster called self-doubt. We are always learning, both student and teacher, mentor and mentee.

Python's bytes and strings

Tue, 23 Oct 2018 00:00:00 +0000

Python: strings and bytes

Python 3 is the new python, and python 2 should be regarded as legacy. But, it is still being used heavily and will likely live beyond its end of life. Recently, a project forced me to use python 2.7 within a Centos environment and I hit some small encoding issues.

Character Sequences

In python there are two types that represent sequences of characters:

Python 3

Bytes: raw 8 bit values
Strings: unicode characters

Python 2

Strings: raw 8 bit values
Unicode: unicode characters

Python 3’s strings and python 2’s unicode instances do not have binary encoding. Therefore, the user must implement the conversion manually by encoding and decoding the between binary and unicode.

It is important that when constructing our program to deal with conversion between these data types that it is done at the furthest boundaries of our interfaces. This is sometimes referred to as the ‘unicode sandwich’.

This guarantee’s a level of compliance when accepting alternate encodings and a much stricter output encoding scheme - generally utf-8. We should not be doing any encoding or decoding in the middle of our applications.

This is of particular importance when dealing with file operations that are not text. In python 3, file operations do much of the encoding and decoding for you, returning str instances on read and write. But, python 3 will use the system default encoding where an encoding scheme is not specified. Much of the time we will receive utf-8 but on some machines this is not guaranteed.

If your application is expected to work across multiple operating systems and/or versions, you should set the encoding=<your-encoding type> keyword argument manually to prevent hidden bugs in the future.

Subtle differences exist between python 2 and 3.

with open('encoding.txt', 'w') as f:
 f.write(os.urandom(10))

In python 2 this will work as by default open use binary encoding. However, python 3 uses utf-8 instead. If we tried to run this it will raise a TypeError: must be str not bytes.

To make this work in both 2 and 3, we need to specify that it is binary.

with open('encoding.txt', 'wb') as f:
 f.write(os.urandom(10))

Writing binary to a file must also be specified with rb instead of r.

with open('encoding.txt', 'rb') as f:
 for line in f:
 print(line)

Notable mentions

Other interesting points between python 2’s str and unicode, and python 3’s str and bytes are how they are compared, or typed. In python 2, if a str is ASCII (7-bits) it will pass an equality check with unicode characters. They can also be combined using the + operator, and can be formatted using %s string formatting.

a = 'a'
b = u'a'

a == b # true
c = a + b # u'aa'
print 'as str a,b =%s%s' % (a,b) # as str a,b=aa

This means that in python 2 we could pass unicode or a str instance and expect the same result - if 7-bit ASCII. In python 3 str and bytes never cannot pass equality checks, not even empty strings.

Summary

Legacy python is still a thing, so having an understanding of how the encoding paradigms differ between each version is important. The differences in file operations can have significant effects within your applications, and is quite prominent in network level communications. Wherever possible work with str types inside your main business logic to alleviate any nasty surprises further down the line.

And have a read over some these gems regarding utf-8, which I deliberately did not go over in any detail.

references

Ned Batchelder’s How Do I Stop The Pain

Joel Spolsky’s The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

Sqlite3: A Primer

Sun, 09 Sep 2018 00:00:00 +0000

SQL Reference

SQLite3 Specific

Transactions

SQLite transactions are ACID (Atomic, Consistent, Isolated and Durable). This means that data is not lost in the event of crashes, power failures or operating system dumps.

Atomic - Meaning changes cannot be broken down into smaller parts; either the entire transaction is commit or nothing at all.

Consistent - A transaction must ensure to change the database from one valid state to another. If a transaction results in invalid data, the database will revert its previous state.

Isolated - Transactions are isolated from one client to the next. When changes are made to the database the changes are only visible to that client until it is committed. If two people concurrently make changes to a database, they will be transacted sequentially. It does not ensure the order of transactions.

Durability - If a transaction is successfully committed, the changes are permanent going forward, if it is interrupted during the commit it will revert to its previous state.

Please check out Julia Evans work at https://jvns.ca

SQLite3 uses auto-commit mode by default.

Select

Used to query data from one or more tables. Example below:

SELECT DISTINCT column_list
FROM table_list
 JOIN table ON join_condition
WHERE row_filter
ORDER BY column
LIMIT count OFFSET offset
GROUP BY column
HAVING group_filter;

A breakdown of each clause:

ORDER BY sorts the result set,
DISTINCT gets unique rows in a table,
WHERE is a filter such as, WHERE employees IS NULL,
LIMIT OFFSET will return a set number of rows,
GROUP BY sorts columns into groups and applies functions to each group,
HAVING is another filter that checks for presence of the operator.

SELECT * FROM database_table is acceptable usage in testing or development but in production the query should be explicit.

Insert

Used to insert new data be it single row, multiple rows, and default data into tables.

INSERT INTO table1 (
 column1,
 column2 ,..)
VALUES
 (
 value1,
 value2 ,...),
 (
 value1,
 value2 ,...),
 ...
 (
 value1,
 value2 ,...);

The above example shows how to insert multiple rows into a table.

INSERT INTO artists (name)
VALUES
 ("Buddy Rich"),
 ("Candido"),
 ("Charlie Byrd");

An example of how to insert into column name within table artists.

Data can also be inserted via a SELECT statement.

Update

Update already existing data in a table.

UPDATE table
SET column_1 = new_value_1,
 column_2 = new_value_2
WHERE
 search_condition 
ORDER column_or_expression
LIMIT row_count OFFSET offset;

the UPDATE clause informs SQL that some part of the table is going to be amended. the SET clause signifies and allows the updating of the column/s that precede it. On the left of the assignment operator (=) is the column to be updated and on the right the new value, expression or data. Whilst the WHERE operator is optional if it is omitted all rows will be updated.

Delete

The delete statement allows for deletion of one, multiple or all rows within a table.

DELETE
FROM
 table
WHERE
 search_condition;

Like updating a table, WHERE, ORDER BY and LIMIT clauses can be applied to a delete operation. In addition deletes can be orchestrated through the SELECT operator.

JOINS

Inner Joins

In relation databases, data is often distributed amongst many related tables. Foreign Keys are used to associate these tables.

INNER JOIN clause will combine columns from correlated tables.

In fig 1.1 Table A’s ‘f’ column is compared with table B’s ‘f’ column. If the value of the ‘f’ column in the A table equals that of B’s ‘f’ column it will return the match. More simply, INNER JOIN clauses return rows from table A that have corresponding rows in the B table.

Fig 1.1 Inner Join

INNER JOIN may connect more than two tables. This will require two inner join clauses form the SELECT statement.

SELECT
 trackid,
 tracks.name AS Track,
 albums.title AS Album,
 artists.name AS Artist
FROM
 tracks
INNER JOIN albums ON albums.albumid = tracks.albumid
INNER JOIN artists ON artists.artistid = albums.artistid;

Example code of a query joining three tables by their Artist. (from Chinook.db found at SQLite’s website tutorial.)

Fig. 1.2 is the result set of the above query.

Full Outer Join

this command is not in SQLite3

Theoretically, the result of a Right Join and Left Join with NULL values for every column of the table that does not have a matching row.

This is technically not available in SQLite3, however they do offer a workaround.

Given that SQLite3 does not have RIGHT JOIN or FULL OUTER JOIN it uses both UNION and LEFT JOIN to emulate it.

It achieves this by switching the position of the LEFT JOIN clause over both columns. SQLite also uses UNION ALL to duplicate rows from the result set of both queries. Finally, the use of a WHERE clause will remove rows already included in the result set of the first SELECT statement. See the below code for an example.


SELECT d.type,
 d.color,
 c.type,
 c.color
FROM dogs d
LEFT JOIN cats c USING(color)
UNION ALL
SELECT d.type,
 d.color,
 c.type,
 c.color
FROM cats c
LEFT JOIN dogs d USING(color)
WHERE d.color IS NULL;

Terms/ Definitions

Primary Key

In order to qualify as a relational table, it must have a primary key.

The primary key consists of one or more columns whose data contained within is used to uniquely identify each row in the table. metaphor If rows were mailboxes the primary key would be the street address.

To be a true primary key it must be:

Unique (data, not the name),
Must not be NULL or ""

Tables must have primary keys, and these are stored in an index which is used to enforce the uniqueness requirement. Being indexed, accessing it does not necessitate scanning the entire table.

Foreign Key

A foreign key is one or more columns in a table that refers to the primary key or unique identifier in another table.

They unlike primary keys can contain NULL, blank or duplicate values. As they allow duplication it is best not to use a foreign key as a primary key, unless it is a one-to-one relationship.

In the above picture we have a Students and Cities table. PERSON_ID and CITY_ID are unique, and therefore make good Primary keys (which they are). Inside the Students table it is possible for several students to be born in the same city. As such the BIRTH_PLACE column is the foreign key to the Cities table. This ensures that there is relationship between the tables and that relation points to something unique in the other table.

Union’s

BLUF: combines result set of two or more queries into a single result set.

UNION by default removes duplicate rows whereas UNION ALL does not. As UNION ALL does not remove duplicates it will process faster. Both statements have the following rules:

Must have the same number of columns,
Corresponding columns must be same data type,
The column names of the first query will determine the column of the combined result set,
Any GROUP BY and HAVING clauses are applied to each individual query, not the final result set,
The ORDER BY is applied to the combined result set, not the individual result sets.

Transaction

The basis of all interactions with the database. Inserts, updates, deletes, commits and table creation and deletion are all transactions.

Linux Environment Variables & PATH

Sat, 11 Aug 2018 00:00:00 +0000

Environment Variables

Simply put, environment variables are a collection of key value strings held by the current shell or process. Each process has access to an array of environment variables held in its user-space memory.

The environment variables are available to all applications, and allow these programs or scripts use them within the shell environment. They can allow custom modification to defaults that permeate throughout the user-space rather than be homed within each applications configuration.

Each process that is created via fork(), will inherit a copy of its parent’s environment. This ensures the communication and transfer of variables from the parent is a once-only one-way transaction. This is important to prevent updated environment variables from one child polluting the global user-space should it go wrong.

PATH

Each environment will have access to several default or standard variables such as:

PATH
HOME
LOGNAME
SHELL
EDITOR
MAIL

By typing echo $HOME the users home location will be printed to the terminal. Environmental variables are case sensitive, and by convention the key is written in capitals.

echoing $PATH on a unix platform will output a : separated list of file paths such as:

/usr/local/bin:/usr/local/sbin:/usr/bin

The PATH is set by system and user start-up scripts - the exact process can change between implementations. Its function is to search in the PATH locations for the name of any program being called for execution. In linux everything is file, and when a file is set with the execution bit, that file can be called with the exec() method. What the PATH does is search recursively through each location, from left to right as its printed, looking for that file if its not given with an absolute path.

Example

$ which echo
/bin/echo
$ ls -l /bin/echo
-rwxr-xr-x 1 root root 30k Dec 29 2017 /bin/echo
$ echo $PATH
/home/userName/bin:/usr/local/bin:/usr/bin:/bin # /bin in PATH

Above we see that calling which echo returns the location of echo (in my distro it returns echo: shell built-in command but /bin/echo will call the file nonetheless). Now when we call echo which is not an absolute path, the PATH variable is consulted. Moving left to right the system checks for an executable file named echo, which is finds inside /bin. Once found it executes and ceases the search. If we called /bin/echo the PATH would be searched.

Example Two

In a real world case, I use Go’s /bin location as an exemplar of why you may need to add to the PATH variable.

Dave Cheney has a pretty sweet Go knock off of an awesome package by the same name httpstat. The TL;DR is that it gives you statistic on a site by simply calling httpstat https://reorx.com. Astute readers will see that this requires httpstat to be on the PATH or it will generate an error.

Fixing this was simple as we see below.

Creation & Deletion

Environment variables are key value pairs, case sensitive and by convention have the key written in caps.

Creation

Adding to your current shell environments list of variables is simple.

# method 1
$ VAR=foo # create key:value
$ export VAR # export is called on VAR
$ echo $VAR # echo VAR to terminal
foo

# method 2
$ export VAR=foo # combine export and variable assignment
$ echo $VAR
foo

Bourne shells accept the second method, and its probably the most common and convenient method. export is unix command to set the attribute for a variable. See man export for more information.

Deletion

$ unset VAR
$ echo $VAR
# will echo empty string

Continuing with the httpstat example we will append the location of Go’s package binaries to our PATH.

$ export PATH=~/go/bin:$PATH # now added to PATH

Once the variable is exported, it will only live within the current shell. If we close it down, that updated PATH is now lost.

To make the variables persistent we add them to our shell’s .profile or .bashrc if using bash. In my case, I use Zsh so its added to .zshrc. Calling source .zshrc will reload .zshrc and make the new variables available if not already. And, because each process inherits form the parent, removing it form the file will prevent it being added to the environment list if needed in the future.

Security tidbits

Of note linux implements a security measure within the Superusers PATH. The current working directory is normally omitted from the PATH meaning files must be called using ./ as the prefix or as an absolute path. This prevents malicious users from placing an executable file with the same name as a system-wide executable such as ls being run as root.

Conclusion

Environment variables allows the system to access a list of values for use in applications
Users can set and unset variables
PATH is important when you need to access an executable without calling its absolute path

Python Object References

Mon, 09 Jul 2018 00:00:00 +0000

Python Object Reference ELI5

I first learnt about variables through the analogy of “variables are boxes” and that we assign things to those boxes Turns out, this isn’t particularly helpful in objected orientated programming. This post is about how python treats object assignment and some of the hidden gotcha’s that can cause unintended errors along the way. Instead of “boxes” it is better to think of variables as “labels” that we attach to objects. And, as everything in python is an object its important to remember that all objects have three things; identity, type and values. Values are the only things that change once an object is created, and it values that we often care about, and hence label.

Labels not boxes

Extending the “labels” metaphor a little we look at the assignment of variables.

a = 2 # we label the integer 2 as 'a'
b = a # 'a' is now labelled as 'b'
c = b # and 'b' is now labelled as 'c'

Above we can see that the object 2} is assigned to the variable ‘a’. Each subsequent assignment thereafter is simply a reference to the same object. When viewed through this lense you can start to see how objects have labels. It is not feasible that the 2 can exist in three different boxes rather we visualise 2 having three sticky notes attached to it. If we changed a like this a = 20 then it is just a matter of peeling off the sticky note with a written on it from 2 and attaching it to 20. To further aid in this thinking, always read assignments from right to left. The right side is where the object is created or retrieved and the left is what we bind to it (the label.. Enough you get it already!)

When an object like 2 has many labels we called this aliasing. Aliasing is an important concept to grasp, and to illustrate why we will examine the identity of a, b, and c.

print(f'a id: {id(a)}') # original object') # a id: 139886603774600
print(f'b id: {id(b)}') # alias of a') # b id: 139886603774600
print(f'c id: {id(c)}') # copy of a') # c id: 139886603774600

All aliases of a have the same identity which in python is unique integer representing its C memory address. If any change were to be made the identity integer would also change to reflect that.

When is == true?

Let’s check out Equality and Identity (and aliases, too)

An object’s identity never changes once it has been created. However its values might, and generally this is what we care about more. Python gives us the option to check either like so:

a == b # compares the values
a is b # compares the identities

Lets extend this using a more complex example using some dictionaries.

batman = {'name': 'Bruce Wayne', 'job': 'crime fighter'}
bruce = batman
print(batman == bruce) # True
print(batman is bruce) # True

Both batman and bruce are equal in identity, and their values. Suppose we have a vigilante crime fighter out there pretending to be batman, named manbat, does he have the same equality?

manbat = {'name': 'Bruce Wayne', 'job': 'crime fighter'}
print(batman == manbat) # True
print(batman is manbat) # False

In this case, both manbat and batman share equal values but not the same identity. manbat is not an alias of bruce or batman, and thus has his own unique identity. This is because we created an entirely new identity albeit with the same values as batman.

Much of the time we care mostly about the values an object holds not its identity but you will see is in a lot during conditionals such as:

if x is None:
 do something
if x is not None:
 do something else

Alias Issues

Something I didn’t realise until it came back to haunt me much later is that aliases can have unintended side effects with mutable types. Let’s say we have two lists, the original and its alias. The alias will have items added to it but we want the original untouched for whatever reason.

orig = [10, 20, 30, [100, 200]]
new = orig

Looks good, we can now make changes to new.

new.append('FizzBuzz')
print(orig) # [10, 20, 30, [100, 200], 'FizzBuzz']
print(new) # [10, 20, 30, [100, 200], 'FizzBuzz']

After appending to new it becomes apparent that this change has affected both lists. This happens because the alias works two way with mutable types. I think this is really important to know - aliases are not copies!

Copies

If aliases aren’t copies then how do we copy?

orig = [10, 20, 30, [100, 200]]
new = list(orig)
# dict(x) also works this way
print('orig id:', id(orig)) # orig id: 140443406513496
print('new id:', id(new)) # new id: 140443402343535

By using the list() class we successfully create two new objects. Now if we append or remove items from either list it does not propagate through. Except, it does sometimes.

In this case we are only making a new copy of the overall object but not any mutable nested types within the copy. So while any changes made within the first layer of the object are contained within the copy, any mutable objects nested more deeply will be aliases.

Confused, an example.

orig = [10, 20, 30, [100, 200]]
new = list(orig)
new.append('not nested')
print(orig) # [10, 20, 30, [100, 200]]
print(new) # [10, 20, 30, [100, 200], 'not nested']
# first layer is not affected as it is a copy, not an alias
orig[-1].append('i am aliased to orig')
print(orig) # [10, 20, 30, [100, 200, 'i am aliased to a']]
print(new) # [10, 20, 30, [100, 200, 'i am aliased to a'], 'not nested']

While the orig and new are independent of each other when making changes to the first layer of abstraction, any mutable types within that are simply aliases of the copies source.

Another example to check this out.

# before we started making alterations to the lists
print(id(orig)) # 140443390926984
print(id(new)) # 140443392352593
print(id(orig[-1])) # 140443395483400
print(id(new[-1])) # 140443395483400

Inspecting the identities reveals that only the overall object's were initialised as new objects but the nested types within were bound to the original nested type - an alias!

This is something to take into consideration when passing variables around that have nested types. To circumvent this immutable types such as tuples can be used in place.

Python can do deep copies which will take care of this issue, but it has its own drawbacks. Of which we not be discussed here as this post is already quite long. See Dan Bader's excellent post for more information.

Wrapping Up

In python all objects have a type, identity and values. Only the values can change after it is created and knowing a little bit more about how this works can help us prevent unintended bugs.

Notes:

assignment does not create copies
nested mutable types within shallow copies are aliases
equality has two different checks; identity, and values

Public Key Crypto: The Basics

Thu, 14 Jun 2018 00:00:00 +0000

Public Key Cryptography

My simplest explanation

TL;DR: Public key cryptography is a method of using a different key for the encryption and decryption of a file, message or other medium.

Public Key Cryptography

What problem does it solve?

If two parties wish to exchange information to one another but the medium which they use to communicate is compromised, how do they share a secret key to unlock their encrypted messages? Symmetric key cryptography would make this untenable for the most part. But, if the parties use two keys; one for encryption and one for decryption they can circumvent the interception as having only one of the keys is not enough to unlock the messages contents. This is also known as asymmetric cryptography.

Public key cryptography uses two keys; a public and private key. The public key can be transferred or published globally without worry, as it requires the private key to unlock the cipher.

how does it work?

We have two actors whom wish to send a secure message; Bob and Alice. If Alice wants to send a message to Bob, first she finds Bob’s public key either by looking it up globally, or getting it directly from Bob. Alice now encrypts her message using Bob’s public key. This is safe because Bob’s public key cannot decrypt this message, meaning it does not matter if anyone else has Bob’s public key.

When Bob receives Alice’s encrypted message, he uses his private key -which only Bob has access to - and decrypts the message. Should Bob wish to reply, he simply repeats the process, now using Alice’s public key to encrypt his message to her.

At it’s core, this is the principle of asymmetric or public key encryption.

Digital Signatures

In the physical world, a signature is a form of validation. It certifies that the signing party agrees to the document, and that they are who they say they are. Unfortunately, it is easy to spoof or refute a signature over the internet - how does one prove it was them and not someone else?

Using public key cryptography we are able to digitally watermark an object in a way that verifies the signature is from the party in question. If Alice wants to digitally verify a document with her name, she can sign it using public key cryptography.

When Bob receives Alice’s encrypted message everyone can decrypt this message as they can get her public key, but no-one else has her private key. This means that no-one could of altered the message, or forged it entirely because Alice is the sole owner of the private key.

This is often referred to non-repudiation. Meaning that at a later time the author cannot refute the fact that they are the owner of the message.

Can Someone Use My Key?

The entire principle of public key cryptography relies on trusting the source of the keys. If a third party can steal or intercept the private and public keys then they can masquerade as the legitimate recipient or sender of the message.

Two common examples are possible; interception, and theft.

If a third party can intercept the public key before the legitimate recipient and then forward on their own in its place, they will effect a man-in-the-middle (MITM) attack. In this case, it would be possible for the third party to read the messages, and reply on their behalf.

Another possibility is theft. If a belligerent were able to break into a key-store or server which has the private key stored on file, it is possible to now read, write and intercept messages. More often this type of intrusion will use that key to bypass other servers authentication which in the case of passwordless-ssh may give an attacker access to another server in the network.

This issue of authentication is outside this post, but it is essential to understand that asymmetric cryptography provides confidentially, rather than authentication.

Base64 ELI5

Tue, 22 May 2018 00:00:00 +0000

Base64

Base64 is a common encoding scheme. But what is it, why 64 bits and how does it work?

TL;DR

Base64 is a binary to ASCII encoding that takes a byte (8 bits) and chunks its down into segments of 6 bits - six ones in binary equates to 64 which is where it derives its name.

first, a demo

Let’s look at an example using the three letter word “The”.

This tabulated data below shows how the word ‘The’ is represented in both binary and base64. Each letter corresponds with an integer. For instance, ‘T’ is mapped to the number 84. This is then converted into binary, and the entire word ‘The’ can be expressed in three bytes.

#ascii T (84) h (104) e (101) <-- ASCII plus base10 number
#binary 01010100 01101000 01100101 <-- three bytes
#base64 010101 000110 100001 100101 <-- four segments of six bits

While binary takes 8 bits, base64 only accepts 6. This means that the last two bits are clipped from the end and become the first two bits of the next segment.

Now that we have base64 binary we can calculate the base10 integer. This is done so we can map that integer to its base64 character.

#base64 010101 000110 100001 100101
#total 21 06 33 37

Or, a graphical representation for mobile users.

Character to value conversion chart for base64. By cross referencing the binary representation of the four segments (21, 06, 33, 37) we get the ASCII characters; VGhl.

Gotcha’s

In the example we used a three byte word which can neatly be broken down into four segments of six. What if the word is four characters equalling 32 bits. Dividing 32 by 6 gives us 5 with a remaining two bits.

We can see this clearly using python (because, why not.)

32 / 6
>> 5.33333333333
32 % 6
>> 2

In that case we need to pad out the base64 to indicate that the last segment is not complete.

The decimal representation of “Them” is 84 104 101 109 or VGhlbQ== in base64. As you can see there are two = symbols tacked on to the end. This indicates that there is two empty segments of 6 bits at the end of the encoding.

#ascii T (84) h (104) e (101) m (109)
#binary 01010100 01101000 01100101 01101101
#base64 010101 000110 100001 100101 001101 010000
#total 21 06 33 37 13 16

Looks like that works out perfectly, so where does the extra two segments of zero’s come in that generates the =’s?

The segmenting of eight bits into six bits cannot delimit or end half way between a byte. Meaning the base64 encoding must continue until there is no remainder. Let’s explore this again using just one character.

In text format

#ascii M (77)
#binary 01011101 00000000 00000000
#base64 010111 010000 000000 000000
#total 19 16 00 00

A pretty picture, too.

I chose the letter ‘M’ as it is easier to explain than ‘T’. Looking at it you can see that even with only one byte (read: character), it takes three bytes for base64’s segments of six to equally divide - (18 / 6 = 3). This is why the letter ‘M’ in base64 would be padded with ==.

This example uses two equals signs but base64 can also be padded with just one as seen below.

If you are wondering about the -1 in the columns, ignore them, its just how the JavaScript that powers the encoder has been developed. See Ty Lewis’ CodePen to see it in action.

Base64 increases the size of the data transmission as more packets are required to be sent. So why use it? Base64’s genesis was in MIME (Multipurpose Internet Mail Extensions) where large streams of text are used to send emails and attachments.

In the early days of email, everything was plain text but the inclusion of HTML and attachments created problems. Email is sent as a stream and streaming binary, or hexadecimal could of had unintended consequences as some systems or programs may interpret binary incorrectly. For instance, a null byte in some systems may indicate that the message has ended when in fact it has not.

Today base64 is used everywhere on the net, mostly as a obfuscation technique. A poor one at that. Here is some more information on base64 in the modern web skewed towards infosec.

LiveOverFlow highlighting base64 patterns in hiding JSON data.
SANS whitepaper on how base64 can get you pwned.
MalwareBytes looking at malware obfuscation techniques.

More information?

As always Wikipedia has an excellent page on Base64.

Shout out to Ty Lewis for his nice encoder which I have used for demonstration.

Flask, forms and http requests

Mon, 23 Apr 2018 00:00:00 +0000

Flask, Forms and HTTP Requests

Preface

This week I have been building a small webapp that leverage’s an external API to populate values in the user pages. And it pains me to say that I spent a silly length of time debugging what was a rather easy problem.

The webapp uses Flask, jinja2 templates and the fuelwatcher API. The issue started when trying to institute a search bar that allows the user to select a series of parameters.

In the end the following issues were present:

My lack of understanding in regards to HTTP requests
Not knowing how the Flask Request context functions
Improper formatting of my HTML forms

Learn You Some HTTP For Great Good

Foolishly, I thought I understood HTTP requests. Sometimes you need to use your theoretical knowledge practically to fully grasp a concept. It turns out I was mixing GET and POST when submitting data to the server in conjunction with using HTML form attributes incorrectly.

Get	Post
1.	Sends Parameters in URL	Sends Parameters in body
2.	Used for fetching documents	Used for updating data
3.	Has max length URL limitations	No max length (theoretically)
4.	Often Cached	Server does not cache
5.	Idempotent	Not idempotent
6.	Should not change server data	Can change server data

A key point for said webapp is item number 1; I was sending parameters in the URL when they really needed to be sent via the body. Expanding on this a little, in a GET request the parameters are transmitted using URI schema friendly syntax (see wiki here). It is common to see query delimiters in the URL such as &,=, + and # and should of been a clue in the debugging process. Transmitting a GET request like this fetches data from another resource, where in contrast my data was needed to execute a function within the server application to pull information from a third party.

Flask Requests

This is deep subject but is tightly coupled to the previous point. The request object in flask gives access to the global request object. Meaning it parses the incoming request data for you. This is important because it is checking for a method attribute within that request. Depending on which method is sent will affect how it/ you should check for the request object.

TL;DR: If you incorrectly set the method to GET when its actually a POST you will have problems.

To receive the request object and parse for the data you want the following should be used:

# displaying both POST and GET for clarity

# POST request // Use 'form'
@app.route("/test" , methods=['GET', 'POST'])
def test():
 select = request.form.get('attribute to be parsed')
 return select # or render_template with select=select etc

# GET request // Use 'args'
@app.route('/data')
def data():
 # here we want to get the value of user (i.e. ?user=some-value)
 user = request.args.get('user')
 return str(user) # or how ever you want to use that data

Given these snippets you can see how mixing GET with POST will lead to calling the wrong request method.

HTML form tags for Dummies

HTML, basic right? Don’t let hubris fool you, not learning HTML deeply is foolish. To be fair, this is the first app I have created that has not required the use of flask-wtf, which kindly generates forms and their tags for you.

<!-- standard form tags -->
<form action="{{ url_for('app.function')}}" method="post">

Simple, right? Not if you:

Mix up the HTTP request
Use the wrong flask request method
Don’t use an attribute which you can parse in using request.form.get('attribute')

Tightly coupled (I think there is a SOLID principle about this…) and frustrating when you don’t connect the dots.

The last piece of the puzzle was the simplest of all, I had not set an attribute that the request object could get. In this example a <select> tag was used and simply appending name=attribute_to_be_parsed worked.

For clarity, first the html.

<!-- html -->
<select name='item1'> <-- attribute flask looks for
 {% for item in items %}
 <option>
 {{ item }} <-- what we get in the response
 </option>
 {% endfor}
</select>

And the flask part.

# python: flask
@app.route('/test')
def test():
 item_to_get = request.form.get('item1') <-- flask request parsing the response body
 ... snip ...

… My face when it worked.

Learn from your mistakes

Everyone makes mistakes, learn from it, be humble and don’t do it again.

Uploading to PyPI

Sat, 07 Apr 2018 00:00:00 +0000

UPDATE 27/01/2019: This area of python and the PyPI is under going rapid development and as such the following may have parts which are no longer in date. In time I will update this post to reflect these changes.

Python Package Index

Python has a wonderful community and package ecosystem. It currently has over 130,000 packages for download and a large variety to choose from. To download a python package via the pip command, the package must be uploaded to the Python Package Index, or PyPI. Going forward it may be referred to as the “warehouse” as PyPI is going through an upgrade of its infrastructure and for the better.

Have application, now what?

PyPI offers two servers for the uploading of python packages; testing and production.

Sending the package to the test server is a great idea as it allows you to download your tarball onto any system for testing. This allows you to do a few novel things like sharing it with friends or co-workers, spinning up virtual machine’s with different operating systems or installing it into separate virtual environments with different versions of python.

This article does assume you are using a version control system, and in particular GitHub but this is not a requirement for PyPI.

Productionise your code

Before looking at how to upload your modules, first it must be made ready for release into the wild.

Directory Structure

root-dir/ # The directory which all your files live.
 setup.py # covered below (Required)
 setup.cfg # if using markdown rather than ReStructuredText
 LICENSE.txt # should be required!
 README.md # Also should be required!
 tests/ # tests are a good idea
 test.py
 your-package/
 __init__.py
 awesome.py
 wicked.py

If you look at any great package such as Requests or Glances you will see a similar (although much more intricate) structure. The key files we need are setup.py and setup.cfg if using Markdown.

Setup.py

# setup.py

from codecs import open
# ensure consistent encoding
from setuptools import setup
# always prefer over distutils
from os import path

VERSION = '0.1.0'
URL = 'https://github.com/username/package'
DOWNLOAD_URL = (URL + '/tarball/' + VERSION)

here = path.abspath(path.dirname(__file__))

with open(path.join(here, 'README.md'), encoding='utf-8') as f:
 long_description = f.read()

setup(
 name='yourpackage',
 packages=['yourpackage'],
 version=VERSION,
 description='blurb that users first see to decide if interesting',
 long_description=long_description,
 long_description_content_type='text/markdown',
 author='Optional',
 author_email='Optional',
 url=URL,
 download_url=DOWNLOAD_URL,
 classifiers=[
 # 3 - Alpha
 # 4 - Beta
 # 5 - Production/Stable
 'Development Status :: 3 - Alpha',
 'Intended Audience :: Developers',
 'License :: OSI Approved :: MIT License',
 'Programming Language :: Python :: 3',
 'Programming Language :: Python :: 3.5',
 'Programming Language :: Python :: 3.6', ]
)

The information contained within setup.py and in particular its setup() function is what creates the package’s metadata for parsing by PyPI once uploaded.

This is a stripped down version of my own setup.py. Many tutorials which are older will use distutils but PyPI’s example structure explicitly states to favour the newer setuptools.

By using a context manager and the codecs.open method we can read the README.md file for use in the long_description parameter within setup(). This is a requirement for the PyPI server as it by default only parses ReStructuredText. If using README.rst, this can be ignored. Further, the long_description_content_type='text/markdown' must be included or it will not format the content correctly. This is a very recent addition to PyPI - March 2018.

Also, the version is what sets the tarball filename, so creating an easy to adjust global variable makes it a lot easier to amend this file when updating your package. As you can see there is three locations that need to be updated for it to function correctly. (Thanks to [Dan Bader] (https://dbader.org) for the idea.)

Please refer to the example page for more information, particularly if your package is more complex than just a few modules.

Setup.cfg

This is just required for Markdown parsing. As you may see it might just be easier to utilise the default supported .rst files. Something I may do in the future.

[metadata]
description-file = README.md

Python setup.py sdist

Running this command will invoke the setup.py and create a folder called dist/ inside your root directory. This is where your application’s tarball will now live. It is also a good time to create or update your git tags for your repository.

FYI, once you create a local tag it must be pushed to the remote.

git tag X.Y.Z -m "Add a message such as; First!"
git push X.Y.Z # preferred option
git push --tags # less preferred as it pushs ALL tags to the remote server

Upload: y u no easy

In theory uploading to PyPI is just that simple. Unfortunately it isn’t that easy and a lot of the helpful blogs and references out there aren’t current with the new standards. As always the official user guide (here) is the holy grail but isn’t the easiest reading when completely unsure!

1. Register

To upload anything to PyPI you must first register with it. And if you want to make use of the testing server, you must register with it separately. Although they use the same software, each server uses a separate database and this is why two different sign up’s are required. They can be found here for Live and Test.

2. Create .pypirc

This file allows your development machine to talk to the PyPI servers. It should look something like this.

# ~/.pypirc

[distutils]
index-servers = 
 pypi
 pypitest

[pypi]
repository=https://upload.pypi.org/legacy/
username=username

[pypitest]
repository=https://test.pypi.org/legacy/
username=username

This is current to today’s date but the repository url may change as the PyPI warehouse continues its evolution. The file must be located in the home directory. Both username and password can be set in this file, or in environment variables.

3. Install Twine

What is it and why use it? Twine is a package written by the PyPI maintainers that uses SSL by default when sending information to their endpoint. Python versions before 2.7.9 and 3.2 do not use this by default and spill user credentials over the air. Also twine separates the creation of the package tarball and uploading into two logical commands; setuptools does not - they are done in the same invocation.

Tarball? To send your package to the server it first must be compressed into a single file. So basically, you zip your files to send and when pip install xxxx is called your tarball is downloaded and unzipped at the end user. This is an important point because any changes you make after creating your tarball are not included in the package so you will need to rezip it to include them.

Personally, I install twine on the system interpreter and update it along with setup tools frequently.

4. Upload!

twine upload -r pypitest dist/package-version-i-select-explicitly.0.1.0.tar.gz
>>> Uploading distributions to https://test.pypi.org/legacy/
>>> Enter your password:
>>> Uploading package-version-i-select-explicitly.0.1.0.tar.gz

Using the -r flag allows you to set which server to send the file too. This file name is setup in the .pypirc file and if you have setup a username it will not prompt you for it. Likewise, it will not prompt for a password should you choose to enter that, and if you do consider chmod 600 on that file for security reasons.

In many examples you may see something like this:

twine upload -r pypitest dist/*

This will upload all of your tarball’s located in the dist/ directory. I personally choose which distribution to upload. Either, or. Once done, goto the test PyPI and check to see that it looks as expected, or as previously stated download the test file and check its functioning as expected.

This can be done like so:

pip install --index-url https://test.pypi.org/simple/ yourPackageName

Once happy send that baby to the production PyPI by repeating the commands but this time specifying the pypi server like so:

twine upload -r pypi dist/package-version-i-select-explicitly.0.1.0.tar.gz

You now have a production package in the wild. Check out your code at Libraries.io.

5. Help PyPI

None of this would be possible with the tireless work of the Python Software Foundation and the handful of volunteers that make PyPI a reality. Join the PSF and maybe consider a donation, or convince your employer to contribute if they rely on python software! Without PSF and PyPI we wouldn’t have python as we know it today. Please visit and sign up here: PSF

Mentoring at CoderDojo

Mon, 26 Mar 2018 00:00:00 +0000

CoderDojo

Today I volunteered my time at a local CoderDojo as a Mentor. The basic premise of a CoderDojo is it’s a place where children between 7 and 17 can learn programming, robotics or basically any technology that might inspire them. The kids are called ’ninjas’ whilst the program co-ordinators are known as champions, and people who help out in their area of expertise are the mentors.

What did we do?

There was around 10-15 kids there and an almost fifty percent split between boys and girls. Most of them were coding away on their laptops using Scratch. Some of them had pretty good little programs; games, stories and creative landscapes being the general theme. They also had a Makeblock robot which the kids could build into ten different configurations. It reminded me of the K’nex toys I had as a kid only way cooler.

Windows sucks… when licences expire

It was great to see that the library (where this dojo is housed) were able to secure some retired laptops from the local government. But also so painful to be a part of because they, as expected, were all Windows based. Almost all had expired licences, missing product keys, or were locked down with administrator access for which no password was supplied. Thankfully, they will all now become the proud new emissaries of linux - although one old laptop was struggling to run with Ubuntu 16. Arch, anyone? :smirk:

Also, I tip my hat to those who donate to charitable causes such as CoderDojo and give away their old laptops. Especially in suburbs where being gifted a functional laptop can be like winning the lottery.

CoderDojo’s one rule: Be Cool!

Click the link for a few minute video that summarises the ethos and idea’s behind the program.

TIL How Network Time Protocol Works

Fri, 23 Mar 2018 00:00:00 +0000

Network Time Protocol

So today I watched a talk on NTP and it was amazing. I will do my best to summarise the core parts in this post.

What is it?

Network Time Protocol allows clocks between computers to be synchronised over the internet or local area network. It was developed by [David L. Mills] (https://en.wikipedia.org/wiki/David_L._Mills) in 1985 and is currently in its fourth version. Accurate to a few milliseconds of Coordinated Universal Time, NTP is a fundamental instrument in modern networking.

Why do we need it?

Its pretty important that we all agree on what time it is. It is even more important that banks and other financial institutions agree on a universal time stamp when conducting transactions, especially if they are ordered chronologically. Things like SSL certificates could be spoofed or bypassed if NTP did not ensure a universal time. There is probably many other security related issues that NTP solves but I think the point is clear, its a necessary protocol.

How does it get the “right” time?

Most people have heard of [atomic clocks] (https://en.wikipedia.org/wiki/Atomic_clock) or GPS time. Basically, these methods are considered as accurate as humans can calculate to ‘real’ time. It would be simplistic to say that NTP polls these sources for the current time - but nonetheless that’s the basic premise.

Diving deeper, NTP actually uses things called a ‘Stratum’. An atomic clock is a Stratum 0, also known as a reference clock. The Stratum’s go from 0 all the way to 16. Stratum 1 is the closest an NTP server can get to the reference clock, and is usually within a few milliseconds. These are referred to as primary time servers. Each Stratum refers to the preceding number, and polls that server for its time. Stratum 3 will synchronise to a Stratum 2 server for example. They can and often do peer with servers in their own Stratum as a sanity check and backup. Of note Stratum 16 means ‘unsynchronised’.

The Process

Roughly the process follows something like this:

ask for the time,
get the roundtrip times,
figure out if you trust the response,
make any adjustments,
repeat every 64 seconds, forever.

Roundtrip Times

After asking the time from a server, NTP needs to factor in how long it took to get the response back because the time from when it sent the response to receiving it will now be out of sync.

# The four timestamps needed for calculating the time
t1 = timestamp of request packet
t2 = timestamp when server received packet
t3 = timestamp of servers reply transmission
t4 = clients response packet reception timestamp

Example of NTP packet

13:11:58.155997 IP (tos 0x0, ttl 56, id 42684, offset 0, flags [none], proto UDP (17), length 76)
 cpe-110-141-196-84.vic.asp.telstra.net.ntp > client.local.lan.ntp: [udp sum ok] NTPv4, length 48
 Server, Leap indicator: (0), Stratum 1 (primary reference), poll 10 (1024s), precision -23
 Root Delay: 0.000000, Root dispersion: 0.001953, Reference-ID: PPS^@
 Reference Timestamp: 3730684254.163282214 (2018/03/22 13:10:54) # t1
 Originator Timestamp: 3730684318.091658531 (2018/03/22 13:11:58) # t2
 Receive Timestamp: 3730684318.125586175 (2018/03/22 13:11:58) # t3
 Transmit Timestamp: 3730684318.125623665 (2018/03/22 13:11:58) # t4
 Originator - Receive Timestamp: +0.033927643
 Originator - Transmit Timestamp: +0.033965133

To calculate the current time from the servers response NTP does the following calculation:

t4 - t1 = roundtrip time
roundtrip time / 2 = one-way latency
t3 + one-way latency = current time

The next part of this is whether or not the client trusts the NTP server. This is done in a few ways. Firstly, by sending out several queries to several servers rather than trusting that the response from one server is correct. NTP then favours the lowest latency and discards any outliers. Secondly, NTP uses some statistical analysis of its responses over a period of minutes to determines who is accurate and who isn’t based off those statistics.

Make Adjustments

What NTP tries to never do is go backwards in time. Sometimes it has to and we will get to that. But for the most part what it does it ‘slew’ the clock. Simply it just slows or speeds up the clock to match the correct time and does this in a gradual way with small increments.

At its max adjustment speed of 500ppm it would take 2000 seconds to make an adjustment of just one second!

Given the slow slew rate, slewing is capped to 128 milliseconds. Anything above that cannot be slewed and must be ‘stepped’ or jumped to the correct time, be it forward or backward. This does not happen often except in cases such as bringing a machine back online after maintenance or during initial setup. Any machine that is over 1000 seconds out must be manually configured within that threshold or it will not be able to receive adjustments.

Conclusion

That’s Network Time Protocol in a nutshell. I had never paid much heed to NTP prior to Joel Potischman’s talk at !!Con. He gave a great talk and it only goes for ten minutes and uses some good graphs and visualisations that are missing from this post. If you want to see NTP in action on your computer you can use tcpdump -vv port 123 or check it out in wireshark. Whilst writing this I found a bad response from one server that was +1023 seconds out and thus dropped as an outlier - so it does happen.

Scapy Snippets

Wed, 28 Feb 2018 00:00:00 +0000

How to get RSSI from WLAN packet

from scapy.layers.dot11 import RadioTap, Dot11

def get_rssi(packet):
 if packet.haslayer(RadioTap):
 return packet.dbm_antsignal

caveat: currently only tested on Ralink: RT5370 chipset

Linux file permissions

Sun, 04 Feb 2018 00:00:00 +0000

Linux: Chmod and Chown

This is a short introduction to basic Linux file permissions and how to set them correctly. As a beginner, junior or experienced developer and/or administrator, you are going to need this!

Chmod: Change File Mode Bits

15:45:18 user@example temporary 
total 12K
drwxr-xr-x 2 test users 4.0K Feb 1 14:29 .
drwx------ 33 test users 4.0K Feb 3 15:45 ..
-rw-r--r-- 1 test users 0 Jan 21 12:14 1
-rw-r--r-- 1 test users 0 Jan 21 12:14 3
-rw-r--r-- 1 test users 16 Feb 1 14:29 file.txt

In the above example output from left to right we have: filemode, number of links, user, group, size, date last modified, name of file.

We will focus on the filemode for now.

drwxr-xr-x
0123456789 <-- this is our reference marker.

0: is either - meaning a file, or d signifying that it is a directory.

1-3: represents the user or owner of the file.

4-6: is reference to the group that owns the file.

7-9: everybody else, or other.

Change the filemodes:

The command chmod is used to change the mode files of files. Its basic syntax is as follows:

In chmod u+x file.txt the u signifies user and the + means add and the x refers to execute. This can be stacked, and used to takeaway permissions.

chmod -R g+rwx directory would recursively (-R) change all group files within directory to read, write and execute.

Importantly, if the user or group does not own the file or directory then the command sudo must be used. Otherwise it is not needed.

Octal:

In the above examples we used rwx to note what permissions we wanted to add or remove from a file. Another method is to use the octal notation; numeral.

chmod 777 file.txt would mean that we are giving rwx to user, group and other. How does 7 equal rwx?

This is because 4 gives read access, 2 is write and 1, execute. When using octal we add the numbers together, so if we wanted read, write and execute we simply add 4,2,1 which equals 7.

chmod 765 xxx.txt would mean: - user: read, write and execute -group: read and write - other: read and execute

Using this is very simple, but how do I remove permissions? Previously, we would chmod u-x i.e. we used the - to signify removal of privilege. Using octal we just set new filemode to what we want and it will add or subtract the mode accordingly. An example below:

chmod a+rwx == give read, write and execute to all (user,group a)nd other) or 777.

But if we only wanted the user to have rwx and everyone else read and write we could call: chmod 755. To use non-octal here would be chmod go-x.

Important note: As with all things linux there are many more advanced features. We are just touching the surface here.

Chown: Change File Owner and Group

The chown command deals with changing the ownership of files and directories.

drwxrwxr-x 1 test users 14.0K Jan 21 12:14 dir_1

-rw-r--r-- 1 admin root 16 Feb 1 14:29 file.txt
[filemode] [owner] [grp]

Above we have a break down of the important parts of our ls -la output.

In the faked output we have admin in the first position which is representing the owner/ creator of file.txt. In postilion two we have the group that file.txt belongs to, in this case its root.

The owner of a file can make changes to the filemode and ownership of a file without super user privileges. However, if another user wanted to change the ownership they would require this access.

To make a change is as simple as chown [user]:[group] [file/s].

$ sudo chown root file.txt # 1.
$ ls -l 
-rw-r--r-- 1 root root 16 Feb 1 14:29 file.txt

The simplest usage of chown is the command plus the new owner and file or directory to be affected.

$ sudo chown -R admin dir_1 # 2.
$ ls -l
drwxrwxr-x 1 admin users 14.0K Jan 21 12:14 dir_1

If we wanted to make the changes to all files and directories inside a directory we can add -R. This is a recursive function with the same syntax as chmod.

$ sudo chown admin:admin file.txt 
$ ls -l
-rw-r--r-- 1 admin admin 16 Feb 1 14:29 file.txt

To specify a new owner and group the use of : between the owner and group can be used. admin:users, root:root and so on.

SSH in a nutshell

Sun, 28 Jan 2018 00:00:00 +0000

SSH

Definition

Straight from the man pages:

SSH (SSH client) is a program for logging into a remote machine and for executing commands on a remote machine. It is intended to provide secure encrypted communications between two untrusted hosts over an insecure network. X11 connections, arbitrary TCP ports and UNIX-domain sockets can also be forwarded over the secure channel.

Setup (Old School Method)

On a new machine we need to setup up ssh. This includes installation if its not installed by default (i.e. Arch Linux). It also includes the creation of our private and public keys (maybe write a post about the basics of pgp).

To create a new key we use ssh-keygen. Taken straight from the man pages:

ssh-keygen generates, manages and converts authentication keys for ssh. It can create keys for use by SSH protocols versions one and two. Protocol one is depreciated and should not be used.

When creating a key we need to specify the type. To do this we use the -t flag followed by one of the following specifications:

RSA, DSA, ECDSA, ED25519

If no flag is present it will default to RSA which is what we want.

Also important is the number of bits in the key to generate. By default it will generate a key length of 2048 bits. In this article we will not discuss non-RSA configurations which require different lengths (refer to the man pages for more info). Generally it is recommended to use 4096 bits of entropy, to do that we use the -b followed by the number, in this case 4096.

Running ssh-keygen -t rsa -b 4096 will create the ssh key.

It will ask for a password, this is up to the user but it does not limit your ability to login without entering password as we discuss below. It does however, offer protection should your device be stolen or compromised. The password can be bruteforced but it will give you time to manually for to your administered servers and delete the public key that matches the now stolen private key.

Key Based Authentication

Key based authentication allows seamless interaction between disparate hosts over the network. Generally, to ssh into a remote server the client will need to enter the servers password. Using ssh keys (unless they too are configured with a password) negates this.

Below we see how to transfer the clients public key to the remote server.

scp ~/.ssh/id_rsa.pub user@remote.server.org:/home/user/.ssh/clients_public_key.pub

scp is the Secure Copy program. It uses ssh for the data transfer and provides the same security. The basic syntax is to call scp then append the file to be copied, followed by the destination to send it to. The destination is punctuated by a colon : coupled with the location on the remote server in which the file should be saved to (as seen above).

Next we need to authorize the key we have sent to the remote server. If we do not authorize it, it will not allow us to login using the key, i.e. we will be prompted for a password each time. To do that we can input the following:

cat ~/.ssh/name_of_new_key.pub >> ~/.ssh/authorized_keys

Firstly, in linux we do not need to append a file name at the end so this works as expected. And secondly, the >> will create the file for us if it does not already exist.

Finally, it is important to set permissions on the new directory and its keys. Permissions is another topic, but we want the following:

chmod 700 /home/user
chmod 700 /home/user/.ssh
chmod 600 /home/user/.ssh/authorized_keys

We want our /home and /home/.ssh to have read, write and execute only by the user and ../authorized_keys only read and writable by the user. This is on the remote server. We can also now delete the public key from within ~/.ssh/ as it is saved to the authorized keys although this is optional.

Setup V2 (New School)

After explaining the old way or more traditional (sometimes better) way we turn to ssh-copy-id.

To do all of that in one command we can just enter:

ssh-copy-id user@remote_server

And, it will do everything in about 2 seconds (after you authenticate with the remotes password). To test it without committing anything use the -n flag (dry run). It is important to know what is happening in the background though, so do the old school method to get a better feel for linux – if you are old hat, just use this.

Further Security Measures

Disable Password Authentication

After we have setup key based authentication on the remote server we may want to close off all non-key authenticated access via ssh. This means that only those with authorized keys will be able to access the server. In a home network this may be overkill (dependent on the situation or security posture/threat model you need, or face) but if you are using ssh to administer a VPS this should be the minimum standard.

To do this we need to edit our /etc/ssh/sshd_config back it up first!.

To do this we just change PasswordAuthentication to no inside the file. After the change we need to restart the ssh daemon to make the change immediate.

Change SSH Default Port

Many of the automated botnets automatically search the web looking for open ports, and exploiting them by entering common default credentials. Whilst changing the ssh port from 22 to something else is security by obscurity it does enough to make elevate you above the ’low hanging fruit’ category.

To alter the default port the sshd_config needs to be edited.

#Port 22 <---- Uncomment this and change port
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

You need to be root to do this. And the ssh daemon must be restarted. systemctl restart sshd.service on systemd.

Auto Deny/Fail2Ban

To access a host via ssh it needs to be open over the network. This is adds risk and creates an attack vector. fail2ban is a service that attempts to mitigate this by altering your firewalls configuration after a set number of unsuccessful login attempts.

Port Knocking

A method to surreptitiously open the port to ssh. By making connection attempts (“knocking”) on ports in a predefined manner the firewall allows access to ssh. Again, this is security through obscurity but coupled with several strategies does help to harden the system. Nmap can ascertain if a host has port knocking enabled.

Rsync cheatsheet

Wed, 24 Jan 2018 00:00:00 +0000

This is a short primer on the most simple of rsync’s capabilities.

Rsync is a fast and extraordinarily versatile file copying tool. It uses a delta transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source and destination files. It can contact remote systems via SSH or through a rsync daemon over TCP.

Basic Syntax

-r: The ‘r’ flag denotes that the operation is recursive. It will copy across all files and folders inside the source directory. However, if the source destination does not have a trailing slash it will copy the folder, rather than the files inside the folder.

-a Archive mode is superior to recursive mode as it will sustain symbolic links, special and device files, modification times, group, owner and permissions.

--dry-run or -n As a sanity check it is worth checking that your command is going to do what you think it is going to do. The dry run will not execute the command. It can (read: should) be coupled with the next command.

-v: Verbose will print out the what actions were undertaken by the command. If -n is not coupled with a verbose flag it will print nothing to the screen.

--delete: Will remove extra items in the destination folder that do not exist in the source directory. Caution: This can lead to complete deletion of the destination folder if incorrectly implemented.

-z: When transferring across the network rsync provides a compression option to save on bandwidth.

-P: Outputs a progress bar to the terminal.

-H: Preserves hard links - very useful for back ups and snap shots as files that don’t change are often hard linked to the original file.

--stats: will output a summary of what was sent. Use --stats -h to get a human readable output as seen below.

# rsync with --stats -h
sending incremental file list

Number of files: 2,265 (reg: 2,264, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 100.20M bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 65.50K
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 108.52K
Total bytes received: 17

sent 108.52K bytes received 17 bytes 19.73K bytes/sec
total size is 100.20M speedup is 923.17

--exclude=important_file.txt: Can be used to omit files or directories from being synced.

--exclude=backups/ --include=backups/most_recent: Inside the exclusion we can explicitly include certain file, folders or patterns that fall inside the broader exclude.

example:

rsync -azvnP source_dir/ destination_dir/

Syncing Across Remote Systems

Syncing can be either a “pull” or “push”.

In this example we are copying the directory, not just its contents and sending it from the local system to a remote system.

To copy the entire folder and not just the files within, we must omit the trailing slash.

The “push”:

rsync -a ~/local_source_dir username@remote_host:/home/username/destination_dir

Here we are retrieving data from a remote server and pulling it down to our local system. Again, we are taking the entire directory so there’s no trailing slash.

The “pull”:

rsync -a username@remote_host:/home/username/directory_we_want where_we_want_it_locally

SSH

Syncing between systems is made much easier if key based authentication is enabled. If not, the user will be prompted with a password.

To use rsync over a ssh connection we need to specify the protocol rsync needs to use.

rsync -aznvPH -e "ssh -i ~/ec2_keyfile.pem" user@remote:/home/folder /tmp/local_system/

The -e flag above, tells rsync to use a command, or in this case, another protocol to tunnel over.

We wrap the -e flag command in quotations to encapsulate our private key - this is particularly useful when accessing AWS. Any .ssh/config settings are also respected by rsync making commonly accessed systems far easier to use.