Blog
I’ve migrated most of my writing to my Medium account. I previous wrote extensively for the Red Hat Enable Sysadmin blog until the community was shut down in 2023.
Below are the technical articles that have been exclusively published on my personal website. I also maintain a separate section of this website for my personal thoughts.
Built-In Locks for Kubernetes Workloads
Jul 31 2023
I’ve been working on a project that interacts with an external web endpoint to query a list of available resources, select a free resource, and then mark it as reserved. The upstream system doesn’t have any type of built-in read locking, so this process is naturally prone to the following race condition: User A looks up a list of available resources While User A is still reviewing the list, User B also looks up a list of available resources. Read more...
Referencing one subnet per AZ with Terraform
Apr 21 2022
I recently had a Terraform use case in AWS where I needed to obtain a list of subnets from a VPC, but I only wanted one subnet per availability zone. Some quick searching around on DuckDuckGo didn’t yield an immediate solution to this problem, so I’m sharing my findings in the hope that it will help someone else. If you’re interested in the solution, feel free to skip directly to “The Solution” below. Read more...
Introducing Rucksack: A place to store your one-liners
Dec 23 2021
I’ve been administering Linux systems for several years across a few jobs, and one thing has remained fairly constant: I always keep a file called useful_stuff.txt on my desktop. Invariably, this contains some crazy one-liners that I’ve come up with. Sometimes they’re fairly generic (such as how to parse an Apache log file and find the top IP addresses), while other times they’re very specialized to the applications run by the company. Read more...
Signals and the "kubectl delete" command
Oct 18 2021
Some colleagues and I were recently implementing a Chaos Monkey style test against a Kubernetes deployment. The goal was to forcibly kill an application to understand how it behaved. Specifically, we were looking to see if the application engaged in some atomic I/O operations that were safe, even if they ungracefully terminated while data was being processed. To do this, we needed to make sure the process (and, by extension, the container running the process) was forcibly terminated without an opportunity to gracefully run any shutdown routines. Read more...
Using OPA to block malicious annotations in Kubernetes
Aug 21 2021
My friend Jared Stroud recently wrote a great article about abusing annotations in Kubernetes as storage space for malicious payloads. It’s a great read and a clever idea, especially since most K8s admins probably pay little attention to what is going on with their annotations. As I was reading his article, I kept thinking to myself: this seems like something that a ValidatingAdmissionWebhook could prevent. If you’re unfamiliar with admission controllers in Kubernetes, I’d recommend a look over the official docs. Read more...
Automated OS Qualification with Ansible
Dec 1 2020
Upgrading thousands of servers is challenging and filled with uncertainty. This article describes how we leveraged Ansible at Datto to build automation that increases confidence in our upgrade process. This was a project that I personally championed, designed a technical solution for, and then led my team through implementation. The article can be found on the Datto Engineering Blog.
Read more...Automating Vault and Consul Template Management
Mar 18 2020
One of my favorite projects at Datto was architecting and implementing our Vault environment. This article discusses some of the harder problems that we solved, including how to make Vault work within our Puppet environment. The article can be found on the Datto Engineering Blog.
Read more...Using Vault as a CA for Graylog
Feb 25 2019
Overview Graylog is a pretty sweet log management solution that allows you to quickly get up and running with centralized log collection and analysis. One common way to get your logs into Graylog is to use Filebeat, which can be further secured using TLS. Graylog even includes a handy Collector Sidecar for handling configuration. Vault is an excellent secrets management tool created by Hashicorp. It includes the ability to easily set up a public key infrastructure, right out of the box. Read more...
Monitoring VMware with Icinga
Nov 29 2018
One of my coworkers and I have been working on building a monitoring environment using Icinga, and I began to consider some options for monitoring the VMware environments that we support. PowerCLI is great for programmatically interacting with VMware environments, and PowerShell for Linux removes the barrier for integrating PowerShell and PowerCLI scripts in a Linux monitoring environment. My goal was to run our check scripts directly on one of our Icinga masters, without the need for a separate Windows satellite that would only be used for running scripts. Read more...
Connecting to systemd-nspawn SSH containers in Ansible
Aug 3 2018
I’ve recently been working on using Ansible to deploy some test services, one of which is an open source IAM server called Gluu. Gluu is unique in that it runs in a systemd-nspawn container. Management and installation of Gluu requires dropping into the container namespace using /sbin/gluu-serverd-3.1.3 login. While this is all well and good for manual configuration, it makes it a bit tricky to deploy using automation. There’s no real official support in Ansible for systemd containers, although there was some discussion on this pull request. Read more...
Persistent Password SSH on AWS AMIs
Feb 14 2017
If you use AWS EC2, you’re definitely familiar with the concept of using a key pair for SSH authentication. Recently, I had a use case that required password SSH login. I set PasswordAuthentication yes in /etc/ssh/sshd_config and created an AMI, but was surprised to discover that PasswordAuthentication no quickly reappeared in my sshd_config when launching an image from the AMI. I spent some time troubleshooting this (more than I care to admit, to be honest), and eventually found that most AMIs use cloud-init to accomplish their provisioning steps. Read more...
Deploying certificate-based SSH with Ansible
Dec 25 2016
A few months ago, I read “Scalable and secure access with SSH” by Marlon Dutra on the Facebook Engineering blog. It’s an informative look into how an organization of Facebook’s size is able to keep authentication manageable across a very large, dynamic, and scalable environment without a single point of failure. If you haven’t read the article, do that before reading mine. Otherwise, nothing below is going to make any sense. Read more...
A Packet Look at Cisco FabricPath
Nov 6 2016
Spanning-tree protocol was one of the first network control plane protocols that I learned about back in my Intro to Routing and Switching class during college. At the time, it seemed pretty obvious: network loops are bad at layer 2, and should be indiscriminately avoided in an effort to prevent broadcast storms. However, real-life networks really aren’t that simple, as any data center engineer will gladly tell you. Specifically, modern data centers face a few important issues: Read more...
Flannels n sh*t is Making my WiFi Slow
Jul 24 2016
I’ve recently been working on renewing the Certified Wireless Network Administrator (CWNA) certification.The CWNA focuses on a deep, technical, and vendor-agnostic understanding of the foundational principles underlying 802.11 WLANs. One day, in between flipping through flash cards, I decided to take a look at the wireless traffic in my own home environment. I was interested to see quite a few Request to Send/Clear to Send (RTS/CTS) exchanges on the same channel as mine, so I decided to dig a bit deeper to “diagnose” the issue. Read more...
Peer Routing and VLT with Dell FTOS Switches
May 26 2016
I recently wrapped up a Dell networking deployment consisting of both Dell S-series switches running the Force10 Operating System (FTOS) and N-series switches running the Dell Network Operating System (DNOS). Both boasted straightforward configuration and were pleasant to work with. The FTOS switches in particular offered a powerful and Dell-recommended feature called Peer Routing that could be used in conjunction with the Virtual Link Trunking (VLT) capabilities. VLT is similar to Cisco’s Virtual Port Channel (vPC) feature, and allows for a single port channel to be multihomed to two Dell FTOS switches. Read more...
Path MTU Discovery
Mar 20 2016
The maximum transmission unit (MTU) is the largest packet that can be transmitted on a link. It naturally follows that the MTU of a given path is the smallest MTU that would be experienced along any given hop on a packet’s journey to its destination. While many of us have become accustomed to the default Ethernet LAN MTU of 1500 bytes, different transmission technologies may have a more constrained MTU. With IPv4, a host didn’t necessarily have to know the MTU of a given path. Read more...
Neighbor Discovery
Sep 27 2015
The next topic that we’ll be covering is fairly straightforward (allowing for a quick article and providing some forward momentum on this blog series). Neighbor discovery is a crucial element of host communication on a local network. While it’s not a particularly complex topic, it is a fundamental IPv6 networking concept that should be understood by any administrator of an IPv6 network. This article will build on our previous discussion of Stateless Autoconfigurationin some ways, and it is recommended that you read that article first. Read more...
Stateless Autoconfiguration
Aug 22 2015
A packet series about IPv6 If you perform nearly any role in the information technology world, you’re no doubt familiar with the issue of IPv4 exhaustion and the challenges of IPv6 adoption. While it doesn’t mean that every business, small and large, will be renumbering their entire networks anytime soon, it does mean that every IT professional should have a familiarity (I’d argue comfortability) with this new addressing scheme that will be directing our packets to and from their destinations in the future. Read more...
Pockethernet: A review of the net admin Swiss Army Knife
Mar 2 2015
About a year ago, I decided to fund a nifty project called Pockethernet on IndieGoGo.With the product being marketed as “The Swiss Army knife for network administrators,” I was really looking forward to the final product being released sometime in July of 2014. When the team started missing target dates due to various setbacks, I shrugged it off as the business learning experience of a few hard working guys, and I largely forgot about it. Read more...
Hacking VoIP: Decrypting SDES Protected SRTP Phone Calls
Jun 22 2014
VoIP security is a fairly complex topic, rife with acronyms, competing solutions, and enough implementation challenges to make any administrator pull their hair out. The Session Description Protocol Security Descriptions (SDES)provide one method for exchanging the keys that are used to encrypt RTP media. Essentially, SDES allows for key exchange within the SDP portion of a SIP packet. Remember that SDP provides parameters, such as media encoding, for a connection. Also remember that SIP is usually unencrypted by default. Read more...
Voice and XMPP: Integrating Asterisk with ejabberd
Apr 2 2014
My current home voice system consists of an Asterisk virtual machine, two Cisco 7940 IP Phones running SIP firmware, and a Google Voice number that is handled by Asterisk. Clearly, I was lacking in the home telephony department, so I decided to try finding some neat things to do with my setup. That’s when I decided to learn about XMPP, unified communications, presence, synergy, communications enabled business practices, agile, methodologies, eXtreme programm…Oops, my bad. Read more...
Upgrading Cisco 7940 Firmware to SIP
Feb 28 2014
So, you picked up a few cheap Cisco 7940s on eBay with the hopes of using SIP and Asterisk, but you don’t really feel like using Call Manager Express (mainly because you don’t want to drop more money on a router). You Google around for firmware upgrade instructions, only to find that the vast majority of “tutorials” are completely unhelpful, wrong, or missing some critical component of the process. Yeah, I did that too. Read more...
Getting around paid in-flight Wi-Fi
Sep 2 2013
Note: The following article is theoretical and based on lab testing. I do not condone or suggest that you attempt to bypass filtering mechanisms for paid Wi-Fi access. This is merely an academic exercise performed in a lab setting. I was flying back to New York recently, and my flight (like most) had a paid Wi-Fi access option. Naturally, as a student with an interest in wireless networking, I started to wonder if there was a way to bypass the payment option get some free access. Read more...
Book and Cert Review: Certified Wireless Network Administrator
Jul 29 2013
I used to hate 802.11. Wireless networking was some form of black magic that broke often, was impossible to troubleshoot, and made me convinced that everything should be hardwired if a reliable connection was desired. I found that others shared my sentiments toward WLANs, and complaining about some broken wireless device was a frequent occurrence among my peers. Overall, wireless networking was this mystical creature that I could never quite understand, troubleshoot, or control. Read more...